Image retrieval method, system, device and storage medium

Abstract

The present invention provides an image retrieval method, system, device and storage medium, which belongs to the field of retrieval. The method includes the following steps: obtaining an image to be detected and an image database; based on a residual network model, adding a feature fusion module after each bottleneck block of the residual network model to obtain a feature extraction network, adding a hash coding layer after the feature extraction network to obtain an image retrieval model; inputting the image to be detected into the image retrieval model, using the feature extraction network to extract local information and global information of different layers of the image to be detected; the hash coding layer calculates a hash code between the image to be detected and the image in the image database according to the local information and the global information, and performs image retrieval according to the hash code. The present invention can improve the accuracy and efficiency of image retrieval. The present invention can enhance the network's perception of important features, and can reduce the errors generated by features during quantization, thereby improving the accuracy of image retrieval.

Description

Technical Field

[0001] The present invention belongs to the field of retrieval, and in particular relates to an image retrieval method, system, device and storage medium. Background Art

[0002] With the rapid development of digital technology, image data has exploded, and the requirements for image retrieval technology have also increased. Traditional content-based image retrieval methods often rely on manually designed feature extractors. The performance of these methods is limited by the sophistication of feature design and the diversity of scenes, and it is difficult to meet the needs of large-scale and efficient retrieval.

[0003] In recent years, the efficiency of image retrieval has been improved by using deep learning combined with hashing technology to retrieve images. However, most deep hashing image retrieval methods only consider the output of the last layer of the network as a feature representation representing the semantic information of the image. When these features are used in subsequent retrieval processes, the advantages of neural networks in feature representation are limited. In fact, as the depth of deep neural networks increases, features become increasingly abstract and difficult to interpret. If only deep features are considered, some important details in the image may be ignored, which limits the feature extraction capabilities of traditional deep learning models in the semantic dimension, resulting in unsatisfactory image retrieval results in complex scenarios. Summary of the invention

[0004] In order to overcome the above-mentioned shortcomings of the prior art, the present invention provides an image retrieval method, comprising the following steps:

[0005] Obtaining an image to be detected and an image database;

[0006] Based on the residual network model, a feature fusion module is added after each bottleneck block of the residual network model to obtain a feature extraction network, and a hash coding layer is added after the feature extraction network to obtain an image retrieval model;

[0007] The image to be detected is input into the image retrieval model, and the feature extraction network is used to extract the local information and global information of different layers of the image to be detected, and the local information and global information of different layers are fused; the fused features are spliced ​​and used as the input of the hash coding layer, the hash coding layer converts the spliced ​​features into hash codes, and uses the hash codes to calculate the similarity between the image to be detected and the images in the image database, and then the images are retrieved based on the similarity.

[0008] Preferably, before inputting the image to be detected into the image retrieval model, the method further includes preprocessing the image in the image database, specifically: resizing, rotating, flipping, enhancing and denoising the image in the image database.

[0009] Preferably, before inputting the image to be detected into the image retrieval model, the following steps are also included:

[0010] Design similarity loss function, quantization loss function, independent loss function, balanced loss function and classification loss function;

[0011] Construct the objective function based on the similarity loss function, quantization loss function, independence loss function, balance loss function and classification loss function;

[0012] The preprocessed images in the image database are input into the image retrieval model, and the image retrieval model is trained according to the objective function.

[0013] Preferably, the method uses hash codes to calculate the similarity between the image to be detected and the images in the image database, and then retrieves the images based on the similarity, specifically: compares the distance between the hash codes of the image to be detected and the images in the image database to obtain the image similarity, sorts the candidate images in the image database according to the image similarity, and selects the most similar image as the retrieval result.

[0014] The present invention also provides an image retrieval system, comprising:

[0015] An image acquisition module is used to acquire the image to be detected and the image database;

[0016] A model building module is used to add a feature fusion module after each bottleneck block of the residual network model based on the residual network model to obtain a feature extraction network, and add a hash coding layer after the feature extraction network to obtain an image retrieval model;

[0017] The image retrieval module is used to input the image to be detected into the image retrieval model, use the feature extraction network to extract the local information and global information of different layers of the image to be detected, and fuse the local information and global information of different layers; the fused features are spliced ​​and used as the input of the hash coding layer, the hash coding layer converts the spliced ​​features into hash codes, and uses the hash codes to calculate the similarity between the image to be detected and the images in the image database, and then retrieve the image based on the similarity.

[0018] The present invention also provides a computer device, comprising a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the image retrieval method.

[0019] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the image retrieval method.

[0020] The image retrieval method, system, device and storage medium provided by the present invention have the following beneficial effects:

[0021] The present invention is based on a residual network model. A feature fusion module is added after each bottleneck block of the residual network model to obtain a feature extraction network. Through the feature extraction network, local information and global information of different layers of an image to be detected can be fused, the network's perception of important features can be enhanced, and the network can learn effective feature representations. Features fused from different layers are spliced ​​as inputs to a hash coding layer, the hash coding layer converts the spliced ​​features into hash codes, and the hash codes are used to calculate the similarity between the image to be detected and the image in an image database, and then the image is retrieved according to the similarity. This process can reduce the error generated by the feature quantization process and improve the accuracy of image retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiment of the present invention and its design scheme, the following briefly introduces the drawings required for this embodiment. The drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 is a flow chart of an image retrieval method according to an embodiment of the present invention;

[0024] Figure 2 It is the structural diagram of feature fusion module;

[0025] Figure 3 Figure 2 is a diagram of the selective kernel convolution structure. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the technical solution of the present invention and implement it, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the scope of protection of the present invention.

[0027] Example

[0028] The present invention provides an image retrieval method, specifically Figure 1 As shown, the following steps are included:

[0029] Step 1: Obtain the image to be detected and the image database.

[0030] The image database selected by the present invention is the CIFAR-10 dataset and the NUS-WIDE dataset.

[0031] Step 2: Based on the residual network model, a feature fusion module is added after each bottleneck block of the residual network model to obtain a feature extraction network, and a hash coding layer is added after the feature extraction network to obtain an image retrieval model.

[0032] The residual network model adopted in the present invention is ResNet50, which is the abbreviation of Residual Network 50 and its Chinese name is "Residual Network-50". The feature fusion module in the feature extraction network of the present invention includes a local feature extraction part and a global feature extraction part.

[0033] Local feature extraction part corresponds to Figure 2 In the right column, selective kernel convolution is used to extract local details of the feature map. The structure is as follows Figure 3 As shown in the figure, the selective kernel convolution contains two main convolution layers, one of which is the main convolution layer and the other is the mask convolution layer. The main convolution layer is used to extract the features of the input data, and the mask convolution layer is used to generate a mask map with the same output size as the main convolution layer. The mask map is normalized by the Sigmoid activation function so that the weight value of each position is between 0 and 1, so that the network can adaptively adjust the degree of attention to different positions in the feature map. By multiplying the output of the main convolution layer by the mask map element by element, selective thinking is introduced in the convolution operation, thereby realizing selective weighting of the input features, so that the model can selectively focus on the content at different positions in the input feature map, and improve the model's perception of important features.

[0034] After the features are extracted through the selective kernel convolution, the batch normalization layer (BatchNormalization, BN) and the ReLU activation function are connected. Finally, the selective kernel convolution layer is passed again to make the number of output channels the same as the input channels, and the same normalization layer (BN) and ReLU activation function are used to improve the robustness and nonlinearity of the model. Finally, the features processed by the local feature extraction part will contain more local details.

[0035] Specifically, for a given input feature map , the shape is , the output after the local feature extraction module is defined as:

[0036] ,

[0037] In the formula, and The convolution kernel sizes are and The optional kernel convolution operation, represents the ReLU activation function, BRepresents a batch normalization operation, where the input feature map is the image to be detected or the image in the image library. If it is the training stage, it refers to the image in the image library. If it is the detection stage, it refers to the image to be detected. C , W and H Both represent the shape dimension of the input feature map. r Indicates the channel reduction ratio.

[0038] Global feature extraction part corresponds to Figure 2 In the left column of the figure, the adaptive feature fusion module first uses the global average pooling (GAP) layer to obtain the global context information of the input feature map and reduce the spatial dimension of the feature map. The global average pooling is used instead of the global maximum pooling because the latter may ignore some important detail information because the global maximum pooling focuses on the local maximum. Finally, two selective kernel convolution operations and ReLU+BN operations are performed to enhance the model's perception and nonlinearity of important features, and the final output feature map has the same dimension as the input.

[0039] Assume that the input feature map is , the local information and global information extracted by the adaptive feature fusion module are and , the total output of the adaptive selection feature fusion module is defined as follows:

[0040] ,

[0041] In the formula, To adaptively select the attention weights extracted by the feature fusion model, Represents the ReLU activation function.

[0042] In the feature extraction network part of the present invention, the global information and local information of the feature maps of different layers of the original image data are feature-fused, and then the feature splicing is converted to the same dimension through the maximum pooling operation to obtain high-dimensional image features, and the hash layer quantization coding is connected to obtain a low-dimensional hash code. When designing the hash layer, the method is used to Function Replacement The function enables the model to learn parameters through back propagation during the training phase. The number of nodes in the hash layer is the number of hash code bits, so that the hash code obtained in the hash layer is an approximate hash code, and the final retrieval is done with The function obtains a strictly discrete hash code. Since the category information of the data set plays a supervisory role in the training model, a classification layer is added after the hash layer to supervise the training of the model.

[0043] Step 3: Input the image to be detected into the image retrieval model, use the feature extraction network to extract the local information and global information of different layers of the image to be detected, and fuse the local information and global information of different layers; concatenate the fused features and use them as the input of the hash coding layer. The hash coding layer converts the concatenated features into hash codes, and uses the hash codes to calculate the similarity between the image to be detected and the images in the image database, and then retrieves the image based on the similarity.

[0044] 1. Before inputting the image to be detected into the image retrieval model, the following steps are also included:

[0045] (1) Design similarity loss function, quantization loss function, independence loss function, balance loss function and classification loss function.

[0046] (a) Considering the correlation between various features in the feature space, a similarity loss function is introduced, as follows:

[0047] ,

[0048] In the formula, represents the hash code generated for a given sample, S represents the similarity matrix of pairwise labels, Represents the similarity information between paired labels. If the input sample and If the labels are the same, ,on the contrary , and Represents the hash code corresponding to the input sample, Represents paired labels The likelihood.

[0049] ,

[0050] In the formula, express and The inner product between express Function, that is .

[0051] (b) Design the information loss generated by the quantization process and transform the quantization process into a maximum likelihood problem, using the variance as and the standard deviation is Gaussian distribution estimator for to represent the binary likelihood of the hash code.

[0052] ,

[0053] In the formula, is the output of the hash layer, exp represents the exponential function, and in use When and , corresponding to the two cases of (1, -1) in the hash code: when using hour, , and when using hour, , and use the same , the final binary cross entropy loss function is:

[0054] ,

[0055] In the formula, K is the number of hash code bits, , , and Both represent binary cross entropy loss, and Both represent the likelihood of the hash code, namely: , ,and and Represents the binary likelihood labels in two cases, and the specific form is as follows:

[0056] ,

[0057] (c) The design balance loss is:

[0058] ,

[0059] In the formula, , It means taking the average value of the hash code. Indicates i The hash code generated by each sample.

[0060] (d) The design independent loss is:

[0061] ,

[0062] In the formula, represents the identity matrix, represents the hash code generated for a given sample, K Indicates the number of hash code digits, N Indicates the number of samples.

[0063] (e) The classification loss is designed as:

[0064] ,

[0065] In the formula, and Respectively represent the c Class i The class labels of samples and the corresponding prediction results, M Indicates the number of categories in the dataset, N Indicates the number of samples.

[0066] (2) Construct the objective function based on the similarity loss function, quantization loss function, independence loss function, balance loss function and classification loss function.

[0067] The overall objective function of the training phase is:

[0068] ,

[0069] In the formula, , , and It is an influencing factor added to control the influence of each item in the objective function.

[0070] The image retrieval model is trained according to the objective function.

[0071] The images in the image database are preprocessed, specifically: resizing, rotating, flipping, enhancing and denoising the images in the image database.

[0072] The preprocessed images in the image database are input into the image retrieval model, and the image retrieval model is trained according to the objective function.

[0073] The data sets selected in the experiment of the present invention are CIFAR-10 and NUS-WIDE data sets. The model is trained using stochastic gradient descent (STD), with an initial learning rate of 0.001, and the model is trained for 100 rounds. The mean average precision mAP of the method of the present invention on the two image data sets reached 90.2% and 87.8% respectively. In the experiment of the method of the present invention, the number of hash code bits is set to 48 bits.

[0074] 2. Input the image to be detected into the image retrieval model, and use the feature extraction network to extract the local and global information of different layers of the image to be detected.

[0075] The image to be detected first passes through the convolution layer to extract the preliminary features of the image, then is normalized through the BN layer, the ReLU activation function is applied to increase the nonlinearity, and finally downsampled through the maximum pooling layer to reduce the size of the feature map. After that, the feature map processed by four layers of feature fusion modules of different scales will contain more important global semantic information and local details in the original feature map, and then retain the enhanced feature map generated by each branch of the network. Finally, the fused features of different layers are converted to the same dimension through the maximum pooling operation and then feature splicing is performed, so that the image features to be detected finally extracted by the network contain more semantic information at different levels of the image.

[0076] 3. Image retrieval based on hash code.

[0077] Specifically, the distance between the hash code of the image to be detected and the image in the image database is compared to obtain the image similarity, the candidate images in the image database are sorted according to the image similarity, and the most similar image is selected as the retrieval result.

[0078] The present invention also provides an image retrieval system, including an image acquisition module, a model construction module and an image retrieval module. The image acquisition module is used to acquire an image to be detected and an image database; the model construction module is used to add a feature fusion module after each bottleneck block of the residual network model based on a residual network model to obtain a feature extraction network, and add a hash coding layer after the feature extraction network to obtain an image retrieval model; the image retrieval module is used to input the image to be detected into the image retrieval model, and use the feature extraction network to extract local information and global information of different layers of the image to be detected; the hash coding layer calculates a hash code between the image to be detected and the image in the image database according to the local information and the global information, and performs image retrieval according to the hash code.

[0079] The present invention also provides a computer device, comprising a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the image retrieval method.

[0080] The present invention also provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for being loaded by a processor to execute the image retrieval method.

[0081] The embodiments described above are only preferred specific implementation modes of the present invention, and the protection scope of the present invention is not limited thereto. Any simple changes or equivalent replacements of the technical solutions that can be obviously obtained by any technician familiar with the field within the technical scope disclosed in the present invention belong to the protection scope of the present invention.

Claims

1. An image retrieval method, characterized in that: The steps include: Obtaining an image to be detected and an image database; Based on the residual network model, a feature fusion module is added after each bottleneck block of the residual network model to obtain a feature extraction network, and a hash coding layer is added after the feature extraction network to obtain an image retrieval model; The image to be detected is input into the image retrieval model, and the local information and global information of different layers of the image to be detected are extracted using the feature extraction network, and the local information and global information of different layers are fused; the fused features are spliced ​​and used as the input of the hash coding layer, which converts the spliced ​​features into hash codes, and uses the hash codes to calculate the similarity between the image to be detected and the images in the image database, and then the images are retrieved based on the similarity; The feature fusion module includes a local feature extraction part and a global feature extraction part; the local feature extraction part uses a selective kernel convolution to extract local details of the feature map, then connects a batch normalization layer BN and a ReLU activation function, and then passes a selective kernel convolution layer to make the number of output channels the same as the input channels, and finally extracts features containing more local details through the same BN and ReLU activation function; The global feature extraction part first uses the global average pooling layer GAP to obtain the global context information of the input feature map and reduce the spatial dimension of the feature map, and then extracts important features and nonlinear features through two selective kernel convolutions and ReLU+BN; The selective kernel convolution includes two convolution layers, one is a main convolution layer and the other is a mask convolution layer. The main convolution layer is used to extract the features of the input data, and the mask convolution layer is used to generate a mask map with the same output size as the main convolution layer. The mask map is normalized by the Sigmoid activation function to adaptively adjust the network's attention to different positions in the feature map.

2. The image retrieval method according to claim 1, characterized in that: Before inputting the image to be detected into the image retrieval model, the method also includes preprocessing the image in the image database, specifically: resizing, rotating, flipping, enhancing and denoising the image in the image database.

3. The image retrieval method according to claim 2, characterized in that: Before inputting the image to be detected into the image retrieval model, the method further includes: Design similarity loss function, quantization loss function, independent loss function, balanced loss function and classification loss function; Construct the objective function based on the similarity loss function, quantization loss function, independence loss function, balance loss function and classification loss function; The preprocessed images in the image database are input into the image retrieval model, and the image retrieval model is trained according to the objective function.

4. The image retrieval method according to claim 1, characterized in that: The method uses hash codes to calculate the similarity between the image to be detected and the images in the image database, and then retrieves the images based on the similarity. Specifically, the distance between the hash codes of the image to be detected and the images in the image database is compared to obtain the image similarity, and the candidate images in the image database are sorted according to the image similarity, and the most similar image is selected as the retrieval result.

5. An image retrieval system, characterized in that: include: An image acquisition module is used to acquire the image to be detected and the image database; A model building module is used to add a feature fusion module after each bottleneck block of the residual network model based on the residual network model to obtain a feature extraction network, and add a hash coding layer after the feature extraction network to obtain an image retrieval model; The image retrieval module is used to input the image to be detected into the image retrieval model, use the feature extraction network to extract the local information and global information of different layers of the image to be detected, and fuse the local information and global information of different layers; perform feature splicing on the fused features and use them as the input of the hash coding layer, the hash coding layer converts the spliced ​​features into hash codes, and uses the hash codes to calculate the similarity between the image to be detected and the images in the image database, and then retrieve the image according to the similarity; The feature fusion module includes a local feature extraction part and a global feature extraction part; the local feature extraction part uses a selective kernel convolution to extract local details of the feature map, then connects a batch normalization layer BN and a ReLU activation function, and then passes a selective kernel convolution layer to make the number of output channels the same as the input channels, and finally extracts features containing more local details through the same BN and ReLU activation function; The global feature extraction part first uses the global average pooling layer GAP to obtain the global context information of the input feature map and reduce the spatial dimension of the feature map, and then extracts important features and nonlinear features through two selective kernel convolutions and ReLU+BN; The selective kernel convolution includes two convolution layers, one is a main convolution layer and the other is a mask convolution layer. The main convolution layer is used to extract the features of the input data, and the mask convolution layer is used to generate a mask map with the same output size as the main convolution layer. The mask map is normalized by the Sigmoid activation function to adaptively adjust the network's attention to different positions in the feature map.

6. A computer device, characterized in that: It comprises a memory and a processor; the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the image retrieval method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the image retrieval method according to any one of claims 1 to 4.

Images