Image multi-modal similarity identification method based on local feature fusion
By fusing local features and global features in image multimodal similarity recognition, using attention mechanism and clustering algorithm, the problems of feature information fusion deviation and multimodal data difference in image similarity learning are solved, and similarity is improved accuracy and flexibility of degree recognition.
Patent Information
- Application Number
- CN202410925852.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-11
AI Technical Summary
The prior art is difficult to effectively fuse different feature information in image similarity learning, resulting in bias in learned feature representations, affecting the accuracy of similarity calculations, and the differences and scarcity of multimodal data limit the application scope of the method. .
Using the multimodal similarity recognition method based on local feature fusion, the multimodal similarity network is constructed, combined with global features and regional features, and the attention mechanism and clustering algorithm are used to dynamically adjust the attention weight and integrate the area of the image. Block information and overall information, build a multimodal similarity matrix.
It improves the accuracy and flexibility of image similarity recognition, can consider overall similarity and local detail similarity at the same time, adapt to different scenario needs, and enhances the accuracy and interpretability of similarity evaluation.
Smart Images

Figure CN118941901A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of image similarity learning, and mainly relates to an image multimodal similarity recognition method based on local feature fusion. Background Art
[0002] With the advent of the big data era, people are facing the huge challenge of processing and analyzing massive data. Among these data, how to quickly and accurately identify the similarities and commonalities between image pairs has become an urgent problem to be solved. Similarity learning provides an effective solution. It can achieve accurate matching and retrieval of data by learning the intrinsic representation of data and similarity measurement methods.
[0003] To solve the similarity learning problem, researchers use multi-feature fusion and representation learning methods to enrich the sample dimensions by integrating multiple feature information, thereby improving the accuracy of similarity learning. However, how to effectively fuse the information between different features and design appropriate fusion strategies for different application scenarios is an important issue. Improper fusion methods may lead to deviations in the learned feature representation, which directly affects the accuracy of similarity calculation. In addition, in the process of fusing multiple features, noise and redundant information may also interfere with the results of similarity calculation, causing adverse effects.
[0004] In addition, researchers are also exploring the use of multimodal similarity methods, committed to fusing information from different modalities to break the limitations of a single modality and enable similarity learning to span different fields and modalities. Nevertheless, data from different modalities may differ greatly in their representation and intrinsic characteristics, and it is a challenge to effectively fuse these different information. At the same time, data from certain modalities may be relatively scarce or difficult to obtain, which limits the scope of application of this method to a certain extent. Summary of the invention
[0005] The present invention aims to solve the deficiencies of the above-mentioned prior art and proposes an image multimodal similarity recognition method based on local feature fusion, which regards the block information and the whole image as different modes, so as to consider both the complex relationship between different blocks of the image itself and the different relationships between images, thereby improving the accuracy of similarity recognition between images.
[0006] In order to solve the technical problem, the present invention adopts the following technical solution:
[0007] The method for multimodal similarity recognition of images based on local feature fusion of the present invention is characterized in that it is performed according to the following steps:
[0008] Step 1: Get the preprocessed image dataset I and construct a similarity matrix S of size n×n, where n represents the number of images in the image dataset I and S[i,j] represents the visual or semantic similarity between the i-th image and the j-th image in I;
[0009] Step 2: Construct a multimodal similarity network based on local feature fusion, and process the image pair consisting of the i-th image and the j-th image in I to obtain the multimodal similarity s between the i-th image and the j-th image in I ij ;
[0010] Step 3: Model training:
[0011] Step 3.1: Use formula (8) to construct the loss function L of the multimodal similarity network:
[0012]
[0013] In formula (8), M represents the number of image pairs with visual or semantic similarity in I;
[0014] Step 3.2: Use the back-propagation algorithm and optimizer to minimize the loss function L, so as to train the multimodal similarity network to update the network parameters. When the preset maximum number of training rounds is reached or the loss function L converges, the training is stopped and the optimal modal similarity model is obtained for similarity recognition of any pair of input images.
[0015] The image multimodal similarity recognition method based on local feature fusion described in the present invention is also characterized in that the multimodal similarity network in step 2 includes: a global feature module, a regional feature module, a fusion feature module, a clustering module, an attention module, and a similarity module;
[0016] Step 2.1: The global feature module uses a convolutional neural network to extract the global features of each image in the similarity dataset I, and flattens each global feature into a one-dimensional vector form and combines them to construct a global feature matrix X1 of size n×n1, where n1 represents the length of the flattened one-dimensional vector;
[0017] Step 2.2: The regional feature module divides the regional features of each image in I into m image blocks of the same size, extracts the features of each image block and then performs linear mapping of n1 dimensions, thereby constructing a regional feature matrix X2 of size n×m×n1;
[0018] Step 2.3: The fusion feature module expands the dimension of feature X1 from n×n1 to n×1×n1, and then concatenates the dimension-enhanced feature X1 with feature X2 in the second dimension, thereby constructing a fusion feature set with a dimension of n×(m+1)×n1, denoted as p={p1,p2,...,p i ,...,p n}, where p i represents the fusion features of the i-th image, i∈n; m+1 represents the number of blocks in each image;
[0019] Step 2.4: The clustering module uses the k-means clustering algorithm to cluster the fused feature set p, and obtains k cluster centers as the relationship vector set between the image pairs in I, denoted as in, Represents the rth relationship vector between the i-th image and the j-th image, j∈n, i≠j;
[0020] Step 2.5: The attention module uses formula (1) to obtain the self-attention weight υ of the i-th image in I i , and use formula (2) to get the non-negative self-attention weight of the x-th block in the i-th image in I
[0021] υ i =muti-attention(p i ,p i ,p i ) (1)
[0022]
[0023] In formula (1), muti-attention means multi-head attention;
[0024] In formula (2), σ represents the sigmoid function, Represents the features of the xth block in the i-th image, x∈m+1;
[0025] The attention module uses formula (3) to obtain the non-negative attention weight between the xth block in the i-th image and the yth block in the j-th image in I
[0026]
[0027] In formula (3), υ j represents the self-attention weight of the jth image; Represents the features of the yth block in the jth image in I;
[0028] Step 2.6: The similarity module constructs the relationship vector between the i-th image and the j-th image in I at the r-th position using equations (4), (5) and (6): The similarity of
[0029]
[0030] In formula (4)-formula (6), α and β represent two parameters to be trained respectively; represents the block similarity between the i-th image and the j-th image in I under the r-th relationship vector, represents the mixed similarity between the i-th image and the j-th image in I under the r-th relationship vector, λ1 and λ2 are two balance coefficients; T represents transposition;
[0031] The similarity module uses formula (7) to obtain the sum of multimodal similarities s between the i-th image and the j-th image: ij :
[0032]
[0033] In formula (7), k represents the number of relationship vectors.
[0034] The step 3.2 comprises:
[0035] Step 3.2.1: Use the back propagation algorithm and optimizer to minimize the loss function L, so as to train the global feature module, the regional feature module, and the fusion feature module, and supervise the change of the loss function L. When the preset maximum number of training rounds is reached, the training is stopped to obtain the trained global feature module, the regional feature module, and the fusion feature module;
[0036] Step 3.2.2: Fix the parameters of the trained global feature module, regional feature module, and fusion feature module, and train the attention module and similarity module, and monitor the change of the loss function L. When the preset maximum number of training rounds is reached, stop the training to obtain the trained attention module and similarity module;
[0037] Step 3.2.3: After merging the trained global feature module, regional feature module, fusion feature module, trained attention module and similarity module, perform multiple rounds of global training and parameter fine-tuning to obtain the optimal modal similarity model.
[0038] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the image multimodal similarity recognition method, and the processor is configured to execute the program stored in the memory.
[0039] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the image multimodal similarity recognition method when executed by a processor.
[0040] Compared with the prior art, the present invention has the following beneficial effects:
[0041] 1 The present invention uses the PVT network to extract multi-scale information in the image, and fuses this information to obtain rich global information, providing sufficient overall image information for subsequent similarity learning. At the same time, the autoencoder is used to extract the local features of the image blocks, and the global information and local information of the image are fused together, providing rich features, so that the image similarity calculation can simultaneously consider the overall similarity and the local detail similarity, thereby obtaining a more accurate and comprehensive similarity evaluation and improving the accuracy of similarity recognition.
[0042] 2 The present invention dynamically gives different attention weights to different blocks within the image, and can adjust these attention weights in real time according to actual needs. In particular, the present invention also regards the entire sample as a special block, and weights it together with the blocks within the image. This design fully takes into account the actual situation, sometimes a certain part of the block of the sample is more important, and sometimes the overall information of the sample occupies a dominant position, so that it can more flexibly and accurately capture the complex relationship within the image, adapt to the needs of different scenes, and effectively deal with complex image similarity calculations in real life;
[0043] 3 The present invention takes into account that the network structure is relatively complex and it is difficult to perform global training. Therefore, the method of step-by-step training is used to fine-tune the model parameters. First, the feature extraction part is trained, and then the trained feature extraction results are fixed and input into the similarity learning part, and the parameters of the similarity learning part are further trained. After completing the step-by-step training, the feature extraction and similarity learning parts are unified under the overall framework, and global training and parameter fine-tuning are performed to obtain the global optimal result. During the training process, the attention mechanism can be used to adjust the attention weights of each block in the sample in real time according to the actual scene requirements. In this way, not only the interpretability of the training results can be ensured, but also the effect of similarity learning can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 It is a schematic diagram of the process of the present invention;
[0045] Figure 2 It is a structural diagram of the model of the present invention. DETAILED DESCRIPTION
[0046] In the present embodiment, a method for multimodal similarity recognition of images based on local feature fusion is intended to solve the problem of interference in similarity learning caused by different elements in the sample in the image retrieval task. When elements that do not need too much attention receive too much attention, it will significantly affect the accuracy and effect of similarity learning. Therefore, the method of the present invention first uses a convolutional neural network to extract the overall features of the input image; then combines the results of multi-feature fusion to guide the learning of the convolutional neural network; then divides the input image into blocks of the same size, extracts the features of each block, and obtains local features; secondly, fuses the overall features of the image and the block features into a comprehensive feature; then forms multiple relationship vectors based on the comprehensive features, constructs a multimodal similarity network based on local feature fusion, and designs a loss function; finally, uses distributed training and global training to obtain the optimal model. Specifically, if Figure 1 As shown, the steps are as follows:
[0047] Step 1: Get the preprocessed image dataset I and construct a similarity matrix S of size n×n, where n represents the number of images in the image dataset I, and S[i,j] represents the similarity between the i-th image and the j-th image in I. The value of the similarity depends on whether there is visual or semantic similarity between the image pairs.
[0048] The labels of image pairs that show visual or semantic similarity are recorded as 1, and the labels of image pairs that have no significant common features or semantics are recorded as 0, providing labels for subsequent loss calculations.
[0049] Step 2: Use convolutional neural network to extract the overall features of the image:
[0050] Step 2.1: For each image in the image dataset I, perform transformation operations such as rotation with a 50% random probability, and then resize the processed images to a uniform size of 512×512;
[0051] Step 2.2: Use the pre-trained PVT network to extract the features of the image, and obtain features with sizes of 1 / 8, 1 / 16, and 1 / 32 of the original image, denoted as F1, F2, and F3. Scale the sizes of F1, F2, and F3 to 1 / 8 of the original image by upsampling to obtain C1, C2, and C3. Add C1, C2, and C3 element by element to form a global feature, and then convert each feature into a one-dimensional vector form. Combine the feature vectors of all images to construct an overall feature matrix X1 of size n×n1, where n represents the total number of images in the data set I and n1 represents the length of the flattened one-dimensional vector.
[0052] The PVT network can extract global information at different levels in the image, and then upsample the features and fuse them to extract the overall information at different levels in the image. By extracting the overall features of the image, it ensures that more accurate and comprehensive results are obtained when the overall information of the image dominates, providing sufficient information for subsequent fusion with regional features.
[0053] Step 3: Get the regional features of the image:
[0054] Each image in dataset I is divided into m image blocks of the same size, and the features of each block are extracted using the autoencoder technology. Then, the features of the block are mapped to n1 dimensions through two layers of MLP layers using the ReLU activation function and a flattening layer to construct a regional feature matrix X2 of size n×m×n1;
[0055] By dividing the input data, the significance of different blocks within the input data is fully considered, which is similar to the human perception system, so as to calculate the similarity between image pairs from a regional perspective.
[0056] Step 4: Comprehensive image overall features and regional features to form fusion features:
[0057] The overall image feature X1 is dimensionally expanded from n×n1 to n×1×n1, and then the dimension-enhanced feature X1 is concatenated with the feature X2 in the second dimension to construct a fused feature set with a dimension of n×(m+1)×n1, denoted as p={p1,p2,...,p i ,...,p n}, where p i represents the fusion features of the i-th image, i∈n; m+1 represents the number of blocks in each image;
[0058] By fusing the overall features and regional features of the image, the entire sample is also regarded as a special block, and the attention weight is weighted together with the internal blocks of the image. This design fully takes into account that in practical applications, sometimes a certain characteristic block of the image carries key information, while sometimes the overall information of the image dominates, making it possible to capture the complex relationships within the image more flexibly and accurately, thereby adapting to the needs of different scenarios.
[0059] Step 5: Construct relationship vector:
[0060] The k-means clustering algorithm is used to cluster the fusion features p corresponding to the i-th image and the j-th image in I. i and p jThe feature pairs are clustered and k samples are randomly initialized as cluster centers C = {c1, c2, c3..c k}, calculate the Euclidean distance between each sample and the k cluster centers, divide the sample into the cluster corresponding to the cluster with the closest distance, and recalculate the cluster centers of different clusters, repeat the above operation until the maximum number of iterations is reached; finally, k cluster centers are obtained and used as the relationship vector set between the image pairs, denoted as Among them, p j represents the fusion features of the jth image, Represents the rth relationship vector between the i-th image and the j-th image, j∈n, i≠j; the cluster center represents the image structure.
[0061] Step 6: Construct a multimodal similarity network based on local feature fusion:
[0062] Step 6.1: Input the fusion features of the image into the attention module, and obtain the self-attention weight υ of the i-th image in I by formula (1): i , the non-negative attention weight of the x-th block in the i-th image in I is obtained by formula (2):
[0063] υ i =muti-attention(p i ,p i ,p i ) (1)
[0064]
[0065] In formula (1), muti-attention means multi-head attention;
[0066] In formula (2), σ represents the sigmoid function, Represents the feature of the xth block in the ith image, x∈m+1, and the non-negative attention weight between the xth block in the ith image and the yth block in the jth image in I is obtained by formula (3):
[0067]
[0068] Using equations (4), (5) and (6), we construct the similarity equation between the i-th image and the j-th image in I at the r-th relationship:
[0069]
[0070]
[0071] In formula (4)-formula (6), α and β represent two parameters to be trained respectively; represents the block similarity between the i-th image and the j-th image in I under the r-th relationship vector, represents the mixed similarity between the i-th image and the j-th image in I under the r-th relationship vector, λ1 and λ2 are two balance coefficients; T represents transposition;
[0072] Through the multi-head self-attention mechanism, multiple self-attention heads are used in parallel to capture different contextual information in the input image, so as to more accurately determine the salient areas in the image and improve the effect of similarity learning.
[0073] Step 6.2: According to formula (7), the multimodal similarity s between the i-th image and the j-th image in I is obtained by summing the similarities between the i-th image and the j-th image under k relationship vectors: ij :
[0074]
[0075] In formula (7), k represents the number of relationship vectors.
[0076] By calculating the similarity between image pairs under different relationship vectors, the expressive power of the model is enhanced, and complex problems can be solved more accurately. This process shows the similarity between samples under diversified relationships, giving the calculation process extremely high interpretability.
[0077] Step 7: Use formula (8) to construct the loss function L:
[0078]
[0079] In formula (8), M represents the number of image pairs that can be formed in I;
[0080] By minimizing the loss function, that is, letting the model continuously learn the similarities between image pairs, it aims to maximize the similarity between image pairs with label 1, that is, to enhance the correlation between them, and minimize the similarity between image pairs with label 0 to weaken the association between them, so that the model can identify similar and dissimilar image pairs.
[0081] Step 8: Model training:
[0082] Step 8.1: Use the back-propagation algorithm and optimizer to minimize the loss function and train the feature extraction part. During the training process, monitor the change of the loss function and evaluate it on the validation set to prevent overfitting. Stop training when the preset maximum number of training rounds is reached or other stopping conditions are met.
[0083] Step 8.2: After the feature extraction part is trained, its parameters are fixed and input into the similarity learning part. When the preset maximum number of training rounds is reached or other stopping conditions are met, the training is stopped;
[0084] Step 8.3: Combine the feature extraction part and similarity learning part trained step by step into a complete model, perform multiple rounds of global training and parameter fine-tuning, and complete the training;
[0085] Through step-by-step training, the loss function can be effectively trained to complete the training of networks with complex and long network structures, thereby obtaining good training results and effectively completing the similarity calculation between image pairs.
[0086] Step 9: Given the image I whose similarity is to be calculated x and I y , resize the two images to 512*512 pixels, use Normalize provided by transforms to normalize the images, and divide the images into image blocks of the same size, input the unsegmented image and the segmented image blocks into the trained similarity network to obtain image I x and I y The similarity between
[0087] The present invention ensures that the model can effectively handle the similarity calculation of image pairs under various abnormal conditions through preprocessing operations such as resizing and normalization, helping the model to accurately capture similar features between images to cope with various unforeseen situations that may occur in practical applications. The original image and the corresponding image blocks after segmentation are input into the model together, realizing the combination of the overall features and local features of the image, so that when the model predicts the similarity of the image pair, it not only measures the similarity between the two images from an overall perspective, but also continuously learns the "interesting" local areas in the image pair, thereby evaluating the similarity between the images from a local perspective;
[0088] like Figure 2 As shown, the pre-processed I x and I y First, it is input into the multi-scale feature processing module and the local feature processing module to form the overall and local features of the image. Then, through further integration in the feature fusion module, the fusion features F x and F y. Next, the two fused features are input into the clustering module, and the obtained k cluster centers are regarded as relationship vectors. Finally, the similarity network is used to calculate the similarity of the image pairs under k relationship vectors, and the similarities under different relationship vectors are accumulated to finally obtain the comprehensive measurement result of the multimodal similarity between images. In this process, the block information and the whole image are regarded as different modes, which not only takes into account the complex relationship between different blocks within the image, but also takes into account the relationship between different images, which can capture the local features and overall structure of the image more carefully, so as to fully understand the image content, help the calculation of similarity recognition, and improve its accuracy; in addition, the correlation between image pairs is measured from multiple relationships between images, which provides an effective basis for similarity calculation, which not only improves the accuracy of image pair similarity recognition, but also enhances the algorithm's adaptability to the diversity and complexity of image content.
[0089] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.
[0090] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program executes the steps of the above method when executed by a processor.
Claims
1. A method for multimodal similarity recognition of images based on local feature fusion, characterized in that: The steps are as follows: Step 1: Get the preprocessed image dataset I and construct a similarity matrix S of size n×n, where n represents the number of images in the image dataset I and S[i,j] represents the visual or semantic similarity between the i-th image and the j-th image in I; Step 2: Construct a multimodal similarity network based on local feature fusion, and process the image pair consisting of the i-th image and the j-th image in I to obtain the multimodal similarity s between the i-th image and the j-th image in I ij ; Step 3: Model training: Step 3.1: Use formula (8) to construct the loss function L of the multimodal similarity network: In formula (8), M represents the number of image pairs with visual or semantic similarity in I; Step 3.2: Use the back-propagation algorithm and optimizer to minimize the loss function L, so as to train the multimodal similarity network to update the network parameters. When the preset maximum number of training rounds is reached or the loss function L converges, the training is stopped and the optimal modal similarity model is obtained for similarity recognition of any pair of input images.
2. The method for multimodal similarity recognition of images based on local feature fusion according to claim 1, characterized in that: The multimodal similarity network in step 2 includes: a global feature module, a regional feature module, a fusion feature module, a clustering module, an attention module, and a similarity module; Step 2.1: The global feature module uses a convolutional neural network to extract the global features of each image in the similarity dataset I, and flattens each global feature into a one-dimensional vector form and combines them to construct a global feature matrix X1 of size n×n1, where n1 represents the length of the flattened one-dimensional vector; Step 2.2: The regional feature module divides the regional features of each image in I into m image blocks of the same size, extracts the features of each image block and then performs linear mapping of n1 dimensions, thereby constructing a regional feature matrix X2 of size n×m×n1; Step 2.3: The fusion feature module expands the dimension of feature X1 from n×n1 to n×1×n1, and then concatenates the dimension-enhanced feature X1 with feature X2 in the second dimension, thereby constructing a fusion feature set with a dimension of n×(m+1)×n1, denoted as p={p1,p2,...,p i ,...,p n }, where p i represents the fusion features of the i-th image, i∈n; m+1 represents the number of blocks in each image; Step 2.4: The clustering module uses the k-means clustering algorithm to cluster the fused feature set p, and obtains k cluster centers as the relationship vector set between the image pairs in I, denoted as in, Represents the rth relationship vector between the i-th image and the j-th image, j∈n, i≠j; Step 2.5: The attention module uses formula (1) to obtain the self-attention weight υ of the i-th image in I i , and use formula (2) to get the non-negative self-attention weight of the x-th block in the i-th image in I u i =multi-attention(p i ,p i ,p i ) (1) In formula (1), muti-attention means multi-head attention; In formula (2), σ represents the sigmoid function, Represents the features of the xth block in the i-th image, x∈m+1; The attention module uses formula (3) to obtain the non-negative attention weight between the xth block in the i-th image and the yth block in the j-th image in I In formula (3), υ j represents the self-attention weight of the jth image; Represents the features of the yth block in the jth image in I; Step 2.6: The similarity module constructs the relationship vector between the i-th image and the j-th image in I at the r-th position using equations (4), (5) and (6): The similarity of In formula (4)-formula (6), α and β represent two parameters to be trained respectively; represents the block similarity between the i-th image and the j-th image in I under the r-th relationship vector, represents the mixed similarity between the i-th image and the j-th image in I under the r-th relationship vector, λ1 and λ2 are two balance coefficients; T represents transposition; The similarity module uses formula (7) to obtain the sum of multimodal similarities s between the i-th image and the j-th image: ij : In formula (7), k represents the number of relationship vectors.
3. The method for multimodal similarity recognition of images based on local feature fusion according to claim 2, characterized in that: The step 3.2 comprises: Step 3.2.1: Use the back propagation algorithm and optimizer to minimize the loss function L, so as to train the global feature module, the regional feature module, and the fusion feature module, and supervise the change of the loss function L. When the preset maximum number of training rounds is reached, the training is stopped to obtain the trained global feature module, the regional feature module, and the fusion feature module; Step 3.2.2: Fix the parameters of the trained global feature module, regional feature module, and fusion feature module, and train the attention module and similarity module, and monitor the change of the loss function L. When the preset maximum number of training rounds is reached, stop the training to obtain the trained attention module and similarity module; Step 3.2.3: After merging the trained global feature module, regional feature module, fusion feature module, trained attention module and similarity module, perform multiple rounds of global training and parameter fine-tuning to obtain the optimal modal similarity model.
4. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the image multimodal similarity recognition method described in any one of claims 1 to 3, and the processor is configured to execute the program stored in the memory.
5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image multimodal similarity recognition method described in any one of claims 1 to 3 are executed.
Citation Information
Patent Citations
Visual saliency and visual phrase combined image retrieval method
CN103838864A
Similarity image retrieval method and system based on multi-feature fusion
CN111914921A
Shielding pedestrian re-identification method and device based on double-similarity segmentation
CN117315716A
Multiple instance learner for tissue image classification
US20220237788A1
