Image retrieval method and system based on cross-X semantic hypergraph learning

Abstract

The invention discloses an image retrieval method and system based on cross-X semantic hypergraph learning. The method comprises the following steps: importing an image to be detected; extracting global features and local region features in a cross-layer and multi-scale manner through a pre-trained convolutional neural network, and outputting feature spectrums of the last two layers of the convolutional neural network; inputting the feature spectrum into a pre-constructed hypergraph neural network to generate a semantic hypergraph so as to simulate high-order correlation between global features and local region features in different layers, different scales and different images, and outputting a multi-layer logic element vector; and fusing the multiple layers of logic element vectors, inputting the fused multiple layers of logic element vectors into a multi-codebook quantizer, carrying out calculation through a quantization codebook in the multi-codebook quantizer, finding a quantization code corresponding to a database image closest to the quantization embedding of the image to be detected, and taking the corresponding database image as a retrieval result. According to the method, the precision of fine-grained image retrieval can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning fine-grained image retrieval, and in particular to an optimal transmission quantization fine-grained image retrieval method and system based on cross-X semantic hypergraph learning. Background Art

[0002] In recent years, we have witnessed the rapid development of deep learning techniques and their successful applications in the field of fine-grained image classification and retrieval. Compared with coarse-grained classification tasks, fine-grained retrieval tasks can be extended to an open world with unlimited subcategories. Some fine-grained image retrieval techniques have been widely used in the fields of conservation and commerce. Among them, compared with coarse-grained image retrieval, large-scale fine-grained image retrieval is still a challenging task due to the subtle differences between visually similar objects. Unlike coarse-grained images, highly similar subcategories in fine-grained images lead to small inter-class variations, making it difficult to capture subtle visual differences. In addition, large intra-class variations due to changes in illumination, pose, and scale pose great challenges to extracting robust discriminative features. Therefore, locating the discriminative information of local details of fine-grained objects and solving the robustness problem of complex personalized differences are the key to alleviating large intra-class variance and small inter-class variance. In addition, memory consumption and query efficiency are another major challenge facing large-scale fine-grained image retrieval. Summary of the invention

[0003] The main purpose of the present invention is to provide an optimal transmission quantization fine-grained image retrieval method and system based on cross-X semantic hypergraph learning, which can extract high-order relations between global features and local features, reduce quantization errors and maintain semantic similarity in the quantization process, improve the utilization efficiency of codewords, and reduce redundant information.

[0004] The technical solution adopted by the present invention is:

[0005] An optimal transmission quantization fine-grained image retrieval method based on cross-X semantic hypergraph learning is provided, comprising the following steps:

[0006] S1. Import the image to be tested;

[0007] S2, extracting global features and local area features across layers and at multiple scales through a pre-trained convolutional neural network, and outputting the feature spectra of the last two layers of the convolutional neural network;

[0008] S3, input the feature spectrum into the pre-built hypergraph neural network to generate a semantic hypergraph to simulate the high-order correlation between global features and local region features from different layers, different scales and different images, and output a multi-layer logical element vector;

[0009] S4, after fusing the multi-layer logical element vectors, input them into the multi-codebook quantizer, and introduce the error regularization term to guide the quantization code learning;

[0010] S5. Calculate through the quantization codebook in the multi-codebook quantizer to find the quantization code corresponding to the database image that is closest to the quantization embedding of the image to be tested, and use the corresponding database image as the retrieval result.

[0011] Following the above technical solution, step S3 also includes the steps of: pre-enhancing the feature spectrum, and obtaining aggregated features by using a pooling operation to reduce storage consumption and computational burden.

[0012] Following the above technical solution, the specific process of feature spectrum enhancement is as follows: extract feature maps from the feature spectra of the last two layers, and use 1×1 convolution to align the channel dimension for subsequent feature interaction; use three levels of spatial pyramids to obtain multiple regions from feature maps of different scales, and perform pooling operations in multiple regions to obtain regional aggregated feature embedding vectors; connect the regional aggregated feature embedding vectors from different images to obtain enhanced feature representation.

[0013] Following the above technical solution, step S3 specifically includes:

[0014] S31: constructing the multilateral semantic relations in the regional features using the preset semantic relation matrix to obtain a semantic hypergraph;

[0015] S32: Update regional features in semantic hypergraph using two-layer hypergraph neural network.

[0016] Following the above technical solution, in step S5, during the training process, the multi-codebook quantizer introduces a class-guided filter mask to weight the output of the hypergraph neural network, and generates embedded features through the fully connected layer for the weighted output; the LSQ algorithm is used to optimize the embedded features, and code balance and irrelevant constraints are introduced in the quantization process to further optimize the quantization process.

[0017] Following the above technical solution, the hypergraph neural network is trained specifically through course training loss.

[0018] Following the above technical solution, in step S2, the global features and local area features in the feature spectrum are extracted specifically through three-scale spatial pyramids, including a 1×1 spatial pyramid, a 2×2 spatial pyramid, and a 3×3 spatial pyramid.

[0019] The present invention also provides an optimal transmission quantization fine-grained image retrieval system based on cross-X semantic hypergraph learning, comprising:

[0020] Image acquisition module, used to import the image to be tested;

[0021] A feature spectrum extraction module is used to extract global features and local area features across layers and at multiple scales through a pre-trained convolutional neural network, and output the feature spectra of the last two layers of the convolutional neural network;

[0022] A hypergraph neural network processing module is used to input the feature spectrum into a pre-built hypergraph neural network to generate a semantic hypergraph to simulate the high-order correlation between global features and local region features from different layers, different scales and different images, and output a multi-layer logical element vector;

[0023] The quantization codebook retrieval module is used to fuse the multi-layer logical element vectors and input them into the multi-codebook quantizer, introduce the error regularization term to guide the quantization code learning; calculate through the quantization codebook in the multi-codebook quantizer, find the quantization code corresponding to the database image that is closest to the quantization embedding of the image to be tested, and use the corresponding database image as the retrieval result.

[0024] In accordance with the above technical solution, it also includes a feature spectrum enhancement module, which is used to pre-enhance the feature spectrum and simultaneously obtain aggregated features using a pooling operation to reduce storage consumption and computational burden.

[0025] The present invention also provides a computer storage medium storing a computer program executable by a processor, wherein the computer program executes the optimal transmission quantization fine-grained image retrieval method based on cross-X semantic hypergraph learning described in the above technical solution.

[0026] The beneficial effect of the present invention is that the present invention generates a semantic hypergraph by extracting high-order relationships between global view features and local view features from different layers, different scales and different images through a hypergraph neural network, so as to reduce the impact of complex personalized differences on subtle details.

[0027] Furthermore, a class-guided filter mask is used to control the quantization error between continuous eigenvalues ​​and their quantized outputs, thereby enhancing the discrimination granularity of quantized features.

[0028] Furthermore, in order to maintain semantic similarity and reduce quantization error, the present invention introduces a regularization term for the progressive asymmetric quantization loss by adopting the LSQ algorithm during the quantization process. Finally, code equalization and irrelevant constraints are introduced in the multi-codebook quantization framework to improve the utilization efficiency of codewords, eliminate redundant information, and approximately eliminate redundant information by solving the optimal transmission problem. Experimental results on multiple fine-grained image datasets show that this method outperforms the most advanced fine-grained image retrieval methods.

[0029] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 is a flow chart of an optimal transmission quantization fine-grained image retrieval method based on cross-X semantic hypergraph learning according to an embodiment of the present invention;

[0032] Figure 2 Schematic diagram of the training process of an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0034] It should be noted that the illustrations provided in the embodiments of the present invention are only used to illustrate the basic concept of the present invention in a schematic manner. Therefore, the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout type may also be more complicated.

[0035] In the present invention, it is also necessary to explain that, if the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, if the terms "first" and "second" appear, they are only used for description and distinction purposes, and cannot be understood as indicating or implying relative importance.

[0036] In addition, it should be noted that the features of the various embodiments of the present invention may be combined or combined in part or in whole, and may interact and operate in different ways as will be appreciated by those skilled in the art. Each embodiment may be implemented independently of one another, or in an associated relationship.

[0037] Example 1

[0038] like Figure 1 As shown, the optimal transmission quantization fine-grained image retrieval method based on cross-X semantic hypergraph learning in an embodiment of the present invention includes the following steps:

[0039] S1. Import the image to be tested;

[0040] S2, extracting global features and local area features across layers and at multiple scales through a pre-trained convolutional neural network, and outputting the feature spectra of the last two layers of the convolutional neural network;

[0041] S3, input the feature spectrum into the pre-built hypergraph neural network to generate a semantic hypergraph to simulate the high-order correlation between global features and local region features from different layers, different scales and different images, and output a multi-layer logical element vector;

[0042] S4, after fusing the multi-layer logical element vectors, input them into the multi-codebook quantizer, and introduce the error regularization term to guide the quantization code learning;

[0043] S5. Calculate through the quantization codebook in the multi-codebook quantizer to find the quantization code corresponding to the database image that is closest to the quantization embedding of the image to be tested, and use the corresponding database image as the retrieval result.

[0044] Among them, step S3 also includes the steps of: pre-enhancing the feature spectrum, and using pooling operation to obtain aggregated features to reduce storage consumption and computing burden.

[0045] Furthermore, the specific process of feature spectrum enhancement is as follows: extract feature maps from the feature spectra of the last two layers, and use 1×1 convolution to align the channel dimension for subsequent feature interaction; use three levels of spatial pyramids to obtain multiple regions from feature maps of different scales, and perform pooling operations in multiple regions to obtain regional aggregated feature embedding vectors; connect the regional aggregated feature embedding vectors from different images to obtain enhanced feature representation.

[0046] In step S2, the global features and local region features in the feature spectrum may be extracted specifically through three-scale spatial pyramids, including a 1×1 spatial pyramid, a 2×2 spatial pyramid, and a 3×3 spatial pyramid.

[0047] Furthermore, step S3 specifically includes:

[0048] S31: constructing the multilateral semantic relations in the regional features using the preset semantic relation matrix to obtain a semantic hypergraph;

[0049] S32: Update regional features in semantic hypergraph using two-layer hypergraph neural network.

[0050] In step S4, during the training process, the multi-codebook quantizer introduces a class-guided filter mask to weight the output of the hypergraph neural network, and generates embedded features through the fully connected layer for the weighted output; the LSQ algorithm is used to optimize the embedded features, and code balance and irrelevant constraints are introduced during the quantization process to further optimize the quantization process.

[0051] In step S5, the multi-codebook quantizer includes a quantization layer, a loss layer and a search layer. During the model training process, the embedded vector generated by the sample image is quantized by the soft quantization layer to generate a compact and discriminative quantization code; the feature representation ability of the quantization code is enhanced by the loss layer, and a quantization code book is generated; the loss layer is constructed by introducing an error regularization term in the progressive asymmetric quantization loss, which is used to constrain the generation of the quantization code, explicitly reduce the quantization error and maintain the semantic similarity in the quantization process, so as to learn the optimal quantization code and codebook with the smallest quantization error; during retrieval, the search layer calculates through the quantization code book to find the quantization code corresponding to the database image that is closest to the logical element vector of the image to be tested, and takes the corresponding database image as the retrieval result.

[0052] The present invention uses high-order relationships between global view features and local view features extracted from different layers, different scales and different images across X (i.e., across stages, layers, and pictures in multiple ways). In addition, high-order relationships can reflect the data manifold, effectively overcome the negative impact of complex personalized differences in fine-grained images by using the inherent geometric and topological structures of the data manifold, and generate smooth feature embeddings to improve the generalization of the model to unseen data sampling from the same distribution. On this basis, a semantic hypergraph model is constructed to model the high-order relationship between the two to reduce the impact of complex personalized differences on subtle details. In addition, a class-guided filter mask is used to control the quantization error between continuous eigenvalues ​​and their quantized outputs, thereby enhancing the discriminant granularity of quantized features. In order to maintain semantic similarity and reduce quantization errors, the present invention introduces regularization terms to the progressive asymmetric quantization loss during the quantization process. Finally, code balance and irrelevant constraints are introduced in the multi-codebook quantization framework to improve the utilization efficiency of codewords, eliminate redundant information, and approximately eliminate redundant information by solving the optimal transmission problem. Experimental results on multiple fine-grained image datasets demonstrate that our method outperforms the state-of-the-art fine-grained image retrieval methods.

[0053] Example 2

[0054] This embodiment is based on Embodiment 1 and provides further specific solutions on this basis.

[0055] Wherein, step S2 specifically includes:

[0056] S21: Since the dimensions of the feature maps extracted from the last two stages of the convolutional neural network are different, a 1×1 convolution is first used to align the channel dimension for the following feature interaction.

[0057] S22: Using three scales of spatial pyramids (1×1, 2×2 and 3×3), 14 regions are obtained from each feature map of different scales. In addition, GeM pooling operation is performed in the divided regions to obtain regional aggregation features. Finally, a total of 28 embedding vectors can be obtained in the two stages. And concatenate these embedding vectors from different images to get regional feature representation from a batch of images

[0058]

[0059] In order to learn diverse representations at different levels, a curriculum training strategy can be introduced during training to improve the generalization ability of the convolutional neural network model. The global features of the last two layers are aggregated and input into the classifier head through the first layer (1×1) to obtain the prediction vector Incorporating this into the smoothed curriculum training objective looks like this:

[0060]

[0061]

[0062] in is a smooth label with a smoothing factor of ε in the s layer. t∈[1,2,...,C] is a classification index, I y The index corresponding to the ground truth classification label y. cat Represents the prediction vector generated by the fusion of the last two layers of logit.

[0063] Furthermore, step S3 is specifically as follows:

[0064] S31: First, perform multi-level feature aggregation on the regional features Z to construct a semantic hypergraph. Specifically, we use a fully connected layer to Projection to C-dimensional logical element representation (preprocessing before inputting the hypergraph neural network), where C is the number of classes and |B| is the number of mini-batches. The softmax function is used to Normalize and define the semantic relationship matrix H∈[0,1] |B|(N+1)×C The regional features across the mini-batch, where the ijth element is:

[0065]

[0066] Among them, i represents the regional feature index of the mini-batch, and j represents the category semantics. Here, we use the semantic relationship matrix H to model the high-order relations in the regional features Z extracted from the mini-batch images. According to the definition of the hypergraph, each hyperedge can connect multiple nodes, thus providing multilateral semantic relations between mini-batch regional features.

[0067] S32: After the hypergraph is constructed, we use a two-layer hypergraph neural network (HGNN) to update these regional features from small batches of samples as follows:

[0068]

[0069] Where l = 0, 1 and Ψ l Represents the trainable weight matrix of the first layer of HGNN. HGNN outputs the updated regional embedding features Right now Where LayerNorm is layer normalization. Function σ(·) represents nonlinear activation. The node degree is defined as The hyperedge is defined as where v i represents the i-th node in the node set, e j represents the jth hyperedge in the hyperedge set. and are diagonal matrices representing node degree and hyperedge degree respectively.

[0070] When the hypergraph neural network is trained, the global features updated in the last two layers are aggregated and fed into the classification head through the hypergraph neural network HGNN to obtain the corresponding prediction scores. HGNN is trained using the curriculum training loss as follows:

[0071]

[0072] Furthermore, step S4 is specifically as follows:

[0073] S41: Introduced a class-guided filtering mask The logit output is weighted to reduce the direct influence of the previous classification task and increase the between-class variance as follows:

[0074]

[0075] in For all combinations of logit outputs, including ⊙ represents the element-by-element multiplication operation. The adaptive class-guided self-entropy filter mask M is defined as follows:

[0076]

[0077] where β is a predefined inhibitory factor.

[0078] S42: In the weighted logarithm This is followed by a fully connected layer to generate an e-dimensional embedding feature E for the subsequent multi-codebook quantization. The quantization code of the entire database sample is recorded as B∈{0,1} DK×M , where D is the number of codebooks, K is the number of codewords in each codebook, and M is the number of database samples. We use S∈{-1,1} M×M and Denote the semantic similarity matrix and codebook respectively. In order to avoid time-consuming retraining of models with different quantization code lengths, the following progressive asymmetric quantization loss function is introduced to train only one quantization model and generate quantization codes with different code lengths at the same time.

[0079]

[0080] sB d ∈{0,1} K×N ,(B d ) T 1 K =1 N ,

[0081] 1 of them * represents an all-ones vector, and the second constraint is applied to generate one-hot codes in each codebook. This function can fully preserve the semantic similarity between the embedded features and the quantized database features.

[0082] During the training process of the multi-codebook quantizer, considering that the quantized features should be as close as possible to the original normalized logit features, in order to reduce the quantization error between the embedded features and the mean of the quantized features during the quantization process, an additional additive quantization loss l2 is introduced:

[0083]

[0084] Therefore, the overall objective function can be expressed as:

[0085]

[0086] Where λ is the balancing factor and θ is the parameter set of the deep network.

[0087] The above loss function can be solved by an alternating optimization strategy. The parameters θ and C can be updated by PyTorch's automatic differentiation engine. The objective function to minimize B can be re-expressed in matrix form:

[0088]

[0089] sB d ∈{0,1}K×N ,(B d ) T 1 K =1 N .

[0090] In order to solve B, a local search process is adopted to iteratively change the strategy for B. d For optimization, the strategy is as follows:

[0091]

[0092] sB d ∈{0,1} K×N ,(B d ) T 1 K =1 N .

[0093] In the formula, I e is the unit matrix, and G is:

[0094]

[0095] The whole optimization process is called LSQ algorithm.

[0096] Although the one-hot encoding constraint in LSQ can cluster image embeddings around the nearest quantization centroid (codeword), it tends to assign unquantized feature vectors to a few main codewords, which limits the model's generalization ability to unknown data. Therefore, code balance and irrelevant constraints are introduced in the quantization process to encourage the quantizer to assign feature vectors equally to all quantization codewords and promote the quantization model to use codewords more efficiently. Therefore, after the objective function of LSQ introduces irrelevant constraints, the quantization code can be restated as:

[0097]

[0098] sB d ∈{0,1} K×N ,(B d ) T 1 K =1 N , B d (B d ) T =NI K ,

[0099] Among them I K Represents the identity matrix. corresponds to the code balance constraint, which enforces that each index assignment in the sub-codebook quantization has an equal probability of occurring in the entire dataset. d (Bd ) T =NI K Corresponding to the code irrelevant constraint, this requires that the quantization codes are orthogonal to remove redundant information.

[0100] The existence of discrete constraints and orthogonal constraints leads to a difficult combinatorial optimization problem. d Relax to continuous values ​​in [0, 1] to derive the approximation; then use alternating optimization to optimize B d Optimize. d After relaxation, the orthogonality constraint can be approximated as an entropy regularization constraint:

[0101]

[0102] Therefore, the objective function B d The sub-optimization problem can be rewritten as:

[0103]

[0104] sB d ∈[0,1] K×N ,(B d ) T 1 K =1 N ,

[0105] in and:

[0106]

[0107] In this way, B d The sub-optimization problem is transformed into an optimal transport problem, and the entire optimization process is called the OTQ algorithm.

[0108] The embodiment of the present invention tests the proposed algorithm on three public datasets: CUB-200-2011, FGCV-Aircraft and StanfordDogs. The CUB-200-2011 dataset is a fine-grained dataset containing 11,788 bird images in 200 categories, of which the training set contains 5,994 images and the test set contains 5,794 images. The FGCV-Aircraft dataset is a fine-grained dataset containing 10,000 aircraft images in 100 categories, of which the training set contains 6,667 images and the test set contains 3,333 images. The Stanford Dogs dataset is a fine-grained dataset containing 20,580 dog images in 120 categories, of which the training set contains 12,000 images and the test set contains 8,580 images.

[0109] The proposed LSQ and OTQ methods are compared with state-of-the-art fine-grained image retrieval methods on CUB-200-2011, Stanford Dogs, and FGVC-Aircraft datasets with encoding lengths ranging from 16 to 64 bits, including ExchNet, CFH, SRLH, OPQN, FISH, and SwinFGHash. It is worth noting that the proposed method uses ResNet18 and ResNet50 as feature extractors, and uses an NVIDIA RTXA6000 GPU and the open source machine learning library Pytorch to implement the proposed method. During the training phase, the input images are resized to 256×256 and randomly cropped to 224×224. In addition, random horizontal flipping and normalization are used for preprocessing during training. During the testing phase, the input images are resized to 256×256 and cropped to 224×224 by center cropping. The model is optimized using the stochastic gradient descent (SGD) optimizer with initial learning rates set to 5e-4, 1e-4, and 7e-4 and momentum to 0.9 on the Cub-200-2011, Stanford-Dog, and FGVC-Aircraft datasets, respectively. The batch sizes on the CUB-200-2011, Stanford-Dog, and FGVC-Aircraft datasets are set to 64, 64, and 32, respectively. The number of iterations is set to 90. The number of codewords K in each codebook is set to 256 = 2 8 . {ε s The smoothing coefficients of} are set to {0.9, 1.0}; λ is set to 0.1, 1, and 100 on the CUB-200-2011, Stanford-Dog, and FGVC-Aircraft datasets, respectively. The predefined suppression factor β is set to 0.7.

[0110] The widely adopted Average Precision (MAP) will be used to measure the accuracy of all retrieval models for code lengths of 16, 32, 48, and 64. The larger the MAP value, the higher the retrieval accuracy.

[0111] Table 1 Comparison results of the present invention with other methods

[0112]

[0113] As shown in Table 1, compared with other state-of-the-art methods, the proposed method shows better performance. The results show that the proposed optimal transmission quantization fine-grained image retrieval method based on cross-X semantic hypergraph learning can reduce the influence of complex individual differences on the discrimination of subtle details and the interference of complex background, improve codeword utilization and remove redundant information, and can more effectively distinguish fine-grained quantitative features. The proposed method performs better on CUB-200-2011, Stanford Dogs and FGVC-Aircraft datasets. The results show that this method is superior to other methods.

[0114] Example 3

[0115] In order to implement the above method embodiment, the present invention also provides an optimal transmission quantization fine-grained image retrieval system based on cross-X semantic hypergraph learning, which includes:

[0116] Image acquisition module, used to import the image to be tested;

[0117] A feature spectrum extraction module is used to extract global features and local area features across layers and at multiple scales through a pre-trained convolutional neural network, and output the feature spectra of the last two layers of the convolutional neural network;

[0118] A hypergraph neural network processing module is used to input the feature spectrum into a pre-built hypergraph neural network to generate a semantic hypergraph to simulate the high-order correlation between global features and local region features from different layers, different scales and different images, and output a multi-layer logical element vector;

[0119] The quantization codebook retrieval module is used to fuse the multi-layer logical element vectors and input them into the multi-codebook quantizer, introduce the error regularization term to guide the quantization code learning; calculate through the quantization codebook in the multi-codebook quantizer, find the quantization code corresponding to the database image that is closest to the quantization embedding of the image to be tested, and use the corresponding database image as the retrieval result.

[0120] Furthermore, the system also includes a feature spectrum enhancement module, which is used to pre-enhance the feature spectrum and simultaneously utilize a pooling operation to obtain aggregated features to reduce storage consumption and computational burden.

[0121] Each module is mainly used to implement each step of the method embodiment, which will not be described in detail here.

[0122] Example 4

[0123] The present application also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, a server, an App application store, etc., on which a computer program is stored, and when the program is executed by a processor, a corresponding function is realized. When the computer-readable storage medium of this embodiment is executed by a processor, the optimal transmission quantization fine-grained image retrieval method based on cross-X semantic hypergraph learning of the method embodiment is realized.

[0124] In summary, the present invention can model the high-order relationship between global features and local features extracted from different levels, scales, and images, suppress sample level and background noise, and alleviate the overfitting problem of complex personalized differences; by encouraging quantizers to evenly distribute feature vectors to all quantization codewords and make the quantization codes orthogonal respectively, the utilization efficiency of codewords can be improved, redundant information can be reduced, and the accuracy of fine-grained image retrieval can be improved.

[0125] It should be pointed out that, according to the needs of implementation, the various steps / components described in this application can be split into more steps / components, and two or more steps / components or partial operations of steps / components can be combined into new steps / components to achieve the purpose of the present invention.

[0126] The order of execution of each step in the above embodiment does not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0127] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. An optimal transmission quantization fine-grained image retrieval method based on cross-X semantic hypergraph learning, characterized in that: The following steps are involved: S1. Import the image to be tested; S2, extracting global features and local area features across layers and at multiple scales through a pre-trained convolutional neural network, and outputting the feature spectra of the last two layers of the convolutional neural network; S3, input the feature spectrum into the pre-built hypergraph neural network to generate a semantic hypergraph to simulate the high-order correlation between global features and local region features from different layers, different scales and different images, and output a multi-layer logical element vector; S4, after fusing the multi-layer logical element vectors, input them into the multi-codebook quantizer, and introduce the error regularization term to guide the quantization code learning; S5. Calculate through the quantization codebook in the multi-codebook quantizer to find the quantization code corresponding to the database image that is closest to the quantization embedding of the image to be tested, and use the corresponding database image as the retrieval result.

2. The optimal transmission quantization fine-grained image retrieval method based on cross-X semantic hypergraph learning according to claim 1 is characterized in that: Step S3 also includes the steps of: pre-enhancing the feature spectrum, and obtaining aggregated features by using a pooling operation to reduce storage consumption and computational burden.

3. The optimal transmission quantization fine-grained image retrieval method based on cross-X semantic hypergraph learning according to claim 2 is characterized in that: The specific process of feature spectrum enhancement is as follows: extract feature maps from the feature spectra of the last two layers, use 1×1 convolution to align the channel dimension for subsequent feature interaction; use three levels of spatial pyramids to obtain multiple regions from feature maps of different scales, and perform pooling operations in multiple regions to obtain regional aggregated feature embedding vectors; The region-aggregated feature embedding vectors from different images are concatenated to obtain an enhanced feature representation.

4. The optimal transmission quantization fine-grained image retrieval method based on cross-X semantic hypergraph learning according to claim 1 is characterized in that: Step S3 specifically includes: S31: constructing the multilateral semantic relations in the feature spectrum using a preset semantic relation matrix to obtain a semantic hypergraph; S32: Updating feature spectra in semantic hypergraph using a two-layer hypergraph neural network.

5. The optimal transmission quantization fine-grained image retrieval method based on cross-X semantic hypergraph learning according to claim 1 is characterized in that: In step S4, during the training process, the multi-codebook quantizer introduces a class-guided filter mask to weight the output of the hypergraph neural network, and generates embedded features through the fully connected layer after the weighted output; The LSQ algorithm is used to optimize the embedded features, and code balance and irrelevant constraints are introduced in the quantization process to further optimize the quantization process.

6. The optimal transmission quantization fine-grained image retrieval method based on cross-X semantic hypergraph learning according to claim 1 is characterized in that: Specifically, the hypergraph neural network is trained through course training loss.

7. The optimal transmission quantization fine-grained image retrieval method based on cross-X semantic hypergraph learning according to claim 1 is characterized in that: In step S2, global features and local region features are extracted through spatial pyramids of three scales, including a 1×1 spatial pyramid, a 2×2 spatial pyramid, and a 3×3 spatial pyramid.

8. An optimal transmission quantization fine-grained image retrieval system based on cross-X semantic hypergraph learning, characterized in that: include: Image acquisition module, used to import the image to be tested; A feature spectrum extraction module is used to extract global features and local area features across layers and at multiple scales through a pre-trained convolutional neural network, and output the feature spectra of the last two layers of the convolutional neural network; A hypergraph neural network processing module is used to input the feature spectrum into a pre-built hypergraph neural network to generate a semantic hypergraph to simulate the high-order correlation between global features and local region features from different layers, different scales and different images, and output a multi-layer logical element vector; The quantization codebook retrieval module is used to fuse the multi-layer logical element vectors and input them into the multi-codebook quantizer, and introduce the error regularization term to guide the quantization code learning; The quantization codebook in the multi-codebook quantizer is used for calculation to find the quantization code corresponding to the database image that is closest to the quantization embedding of the image to be tested, and the corresponding database image is used as the retrieval result.

9. The optimal transmission quantization fine-grained image retrieval system based on cross-X semantic hypergraph learning according to claim 8, characterized in that: It also includes a feature spectrum enhancement module, which is used to pre-enhance the feature spectrum and use pooling operations to obtain aggregated features to reduce storage consumption and computing burden.

10. A computer storage medium, characterized in that: A computer program executable by a processor is stored therein, and the computer program executes the optimal transmission quantization fine-grained image retrieval method based on cross-X semantic hypergraph learning as described in any one of claims 1 to 7.

Images