Image similarity matching method based on hierarchical structure of multi-scale and multi-modal features

Abstract

The present invention discloses an image similarity matching method based on a hierarchical structure of multi-scale and multi-modal features, comprising the following steps: 1. obtaining original multi-modal features of image samples; 2. performing multi-scale transformations on the multi-modal features and fusing them into comprehensive features; 3. performing hierarchical clustering on the samples to obtain the hierarchical structure of the samples, and determining the number of neurons and the connection mode of each layer; 4. constructing a similarity equation based on the comprehensive features and the hierarchical structure; 5. constructing a target equation, training the model using a global optimization method, iteratively updating each relational neuron, and obtaining an optimal training model; 6. using the trained model to perform image similarity matching on input samples, obtaining the image pair with the highest similarity as the most similar image pair. The present invention can better learn the comprehensive features of images, thereby improving the effect of image similarity learning and making the image matching results more interpretable.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning and computer vision, and primarily to an image similarity matching method based on a hierarchical structure of multi-scale and multi-modal features. Background Art

[0002] In image retrieval applications, the image representation vector determines the performance of the entire retrieval system. Therefore, research on efficient description of image content presents a significant challenge. Currently, content-based image retrieval (CBIR) is widely used, such as pathology detection in medical images, searching for similar product information in online shopping, object localization in remote sensing images, and video surveillance in the security field. As image search applications gradually expand, the requirements for image classification and retrieval accuracy and timeliness are becoming increasingly stringent. Therefore, effectively describing image content to achieve efficient retrieval is a challenging problem.

[0003] Recent research on image content description has shown that a single feature type often only describes one aspect of an image while ignoring other characteristics. This results in an inability to effectively describe content differences between images in a database, significantly limiting the accuracy of image classification and retrieval. Multiple complementary features can be used to jointly describe an image. However, the pervasive "semantic gap" between an image's underlying visual features and its semantics has long been a challenge in image retrieval. In the field of computer vision, deep learning has achieved significant success in addressing this "semantic gap." Image retrieval based on multi-feature fusion can effectively overcome the limitations of single-feature-based image retrieval in its ability to accurately describe image descriptions. Consequently, researchers are increasingly using multiple features to jointly describe image content. Leveraging complementary features to express image information can effectively improve retrieval accuracy. Consequently, researchers have begun focusing on how to fuse data from multiple domains. The fusion of multimodal data can provide more information for model decisions, thereby improving the overall accuracy of decision-making. The goal is to develop models capable of processing and correlating information from multiple modalities. This is a typical multidisciplinary field and has gradually become a hot research topic. In many practical tasks, the relationship between samples is important semantic information, such as the relationship between objects in an image or the relationship between sentences in a text. Ignoring these relationships will cause the model to be unable to learn more complex patterns, unable to capture complex semantic information, and more susceptible to noise and interference. Summary of the Invention

[0004] In order to address the shortcomings of the above-mentioned prior art, the present invention proposes an image similarity matching method based on a hierarchical structure of multi-scale and multi-modal features, in order to better learn the comprehensive features of images, thereby improving the effect of image similarity learning and making the image matching results more interpretable.

[0005] In order to solve the technical problem, the present invention adopts the following technical solution:

[0006] The image similarity matching method based on a hierarchical structure of multi-scale and multi-modal features of the present invention is characterized in that it is performed according to the following steps:

[0007] Step 1: Get an image dataset I={I1,I2,…,I i ,,I t}, where I i represents the i-th sample; t represents the number of samples in the image dataset;

[0008] Extract the i-th sample I i The features of k different modes are obtained, and the feature vector set {F i (1) ,F i (2) ,…,F i (l) ,…,F i (k)}, where F i (l) Indicates the i-th sample I i The lth modal feature of;

[0009] Step 2: Build a feature fusion model and use it to obtain the i-th sample I i The multi-scale and multi-modal features G i :

[0010] Step 3: Obtain the hierarchical structure between samples through the hierarchical clustering method and build a hierarchical similarity model:

[0011] Step 3.1: Use Euclidean distance to calculate the distance between any two sample features and compare them to find the two samples with the closest distance and merge them into one cluster. Then, merge all samples in I to obtain a hierarchical structure between samples.

[0012] Step 3.2: Use the hierarchical structure obtained by clustering as the hierarchical network link structure element, use each cluster center as the relationship neuron, and regard the cluster center of the first layer as the relationship vector;

[0013] Step 4: Construct a similarity equation and use it to obtain the i-th sample I i The similarity γ with the jth sample ij :

[0014] Step 5: Use formula (1) to construct the loss function L, and train the feature fusion model and the hierarchical similarity model through the global optimization method, and use the back propagation algorithm to iteratively update each relationship neuron to obtain the optimal image similarity matching model;

[0015]

[0016] In formula (1), I + is a set of positive sample images with the same label in the image dataset I, I - is a set of negative sample images with different labels in the image dataset I; ij- Indicates that the i-th sample I i The jth sample I with different labels j The similarity of γ ij+ Indicates that the i-th sample I i The jth sample I with the same label j , θ is the set of all parameters to be trained in similarity learning, (i, j+) represents a positive sample pair, and (i, j-) represents a negative sample pair;

[0017] Step 6: Set the i-th sample I i For index samples, the best image similarity matching model is used to detect the index samples, and the image pair with the highest similarity to the index samples is obtained and used as the most similar image pair result.

[0018] The image similarity matching method based on a hierarchical structure of multi-scale and multi-modal features according to the present invention is also characterized in that step 2 is performed as follows:

[0019] Step 2.1: Generate the i-th sample I using formula (2) i The initial eigenvector f i1 :

[0020] f i1 =concat(F i (1) ,F i (2) ,…,F i (l) ,…,F i (k) ) (2)

[0021] In formula (1), Concat is a cascade operation along the channel direction;

[0022] Step 2.2: Use convolution kernels of different sizes to calibrate the feature vector f i1 Perform convolution operation to obtain the i-th sample I iThe J different scale feature vectors {f ij |j=1,2,…,J}, where f ij Indicates the i-th sample I i The j-th feature of different scales;

[0023] Step 2.3: The i-th sample I i The J different scale feature vectors {f ij |j=1,2,…,J} are respectively input into J multi-head self-attention modules MHSA for multimodal interaction to obtain the feature vector {f ij ′j=1,2,…,J}, where f ij ′ represents the i-th sample I i The feature after the j-th feature fusion;

[0024] Step 2.4: Use formula (3) and formula (4) to get the i-th sample I i The j-th low-dimensional feature vector g ij And the j-th scale multimodal feature g i ' j :

[0025] g ij =Reshape(FC(f ij ′)) (3)

[0026] g i ' j =MLP(Concat(g ij ,f i1 )) (4)

[0027] In formula (5), FC is a fully connected layer, Reshape is a reshaping operation to restore the length of the feature vector; Concat is a cascade operation along the channel direction, and MLP is a multi-layer perceptron;

[0028] Step 2.5: Use formula (5) to get the i-th sample I i The multi-scale and multi-modal features G i , thus obtaining a multi-scale multi-modal feature set {G1,G2,…,G i ,…,G t};

[0029] G i =MLP(Concat(f′ i1 ,g′ i1 ,…,g′ ij ,…,g′ iJ )) (5).

[0030] Described step 4 is carried out as follows:

[0031] Step 3.1 defines the current hierarchical structure as h and initializes h = 1;

[0032] Step 3.2: Use formula (6) to calculate the i-th sample I in the h-1-th hierarchical structure i and the jth sample I j Similarity under the t-th relationship vector Thus we get the i-th sample I i and the jth sample I j Similarity in the h-1 hierarchy

[0033]

[0034] In formula (5), G j Indicates the jth sample I j Multi-scale and multi-modal features of , T represents transposition; e t is the t-th relationship vector in the h-1-th hierarchical structure, α and β are two control parameters, and Softplus is the activation function; d h-1 is the number of relation neurons in the h-1th level structure;

[0035] Step 3.3: Calculate the i-th sample I using formula (7) i and the jth sample I j The hierarchical similarity at the hth level Thus we get the i-th sample I i The similarity with the j-th sample output at the last level is denoted as γ ij ;

[0036]

[0037] In formula (7), is the hth level structure h ×d h-1 dimensional weight matrix, d h represents the number of relation neurons in the h-th level structure, represents the weight linking the i-th neuron in the h-1-th layer to the j-th neuron in the h-th layer, b h is d in the hth level structure h dimensional bias vector, sig is the activation function; when h=1, let H represents the total number of layers in the hierarchical structure.

[0038] The electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the image similarity matching method, and the processor is configured to execute the program stored in the memory.

[0039] The present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program executes the steps of the image similarity matching method when the computer program is executed by a processor.

[0040] Compared with the prior art, the beneficial effects of the present invention are embodied in:

[0041] This method takes into account the various features of an image, such as color, texture, edges, and local texture features, and accurately reflects the similarity of images across these features, thereby improving the accuracy of sample similarity learning. This method significantly improves the accuracy of sample similarity learning, enabling more accurate and reliable results in practical applications, whether for image recognition or image retrieval.

[0042] 2. This invention uses convolutional neural networks and multiple multi-head attention mechanisms to learn the comprehensive feature expression of images, which can not only mine the effective information in the comprehensive features of images, but also reduce the dimension of the data and speed up the training of feature fusion models;

[0043] 3 This invention extends the calculation of single-layer similarity to semantic-based hierarchical similarity calculation, defines the hierarchical relationship between samples through the hierarchical clustering method, and balances the similarity equation's trade-off between expressiveness and the number of optimized parameters by adjusting the rank of the relationship matrix. This further enhances the interpretability of the algorithm, enables more accurate understanding and classification of image data, and thus provides more insightful results in complex image retrieval tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Schematic diagram of the process of the present invention;

[0045] Figure 2 This is a structural diagram of the model of the present invention. DETAILED DESCRIPTION

[0046] In this embodiment, in the process of image similarity matching, the sample feature fusion method and the hierarchical semantic relationship between samples are studied, and an image similarity matching method with a hierarchical structure of multi-scale and multi-modal features is proposed. First, different features of the image are extracted using an image feature extraction method; then, the extracted different image features are fused using a multi-head attention mechanism; and finally, different relationship neurons are obtained using a hierarchical clustering method, such as Figure 2The single-layer similarity calculation is extended to semantic-based hierarchical similarity calculation, making the hierarchical neurons and the links between them more explainable. A target equation is designed; finally, the BP algorithm is used to iteratively update the relationship neurons to complete the learning. Specifically, Figure 1 As shown, it is done in the following steps:

[0047] Step 1: Get an image dataset I={I1,I2,…,I i ,…,I t}, where I i represents the i-th sample; t represents the number of samples in the image dataset;

[0048] Extract the i-th sample I i The features of k different modes are obtained, and the feature vector set {F i (1) ,F i (2) ,…,F i (l) ,…F i (k)}, where F i (l) Indicates the i-th sample I i The lth modal feature of;

[0049] By extracting different features from the image, the image retrieval improves the retrieval accuracy on the one hand, and ensures the richness of the retrieval results on the other hand; in this embodiment, the data set has a total of four original features, and LBP (Local Binary Pattern) feature, CH (Colour Histogram) feature, HoG (Histogram of Gradients) feature, WT (Wavelet Texture) feature are extracted respectively, and the features are normalized to obtain feature vectors.

[0050] Step 2: Build a feature fusion model and use it to obtain the i-th sample I i The multi-scale and multi-modal features G i :

[0051] Step 2.1: Generate the i-th sample I using formula (1) i The initial eigenvector f i1 :

[0052] f i1 =concat(F i (1) ,F i (2) ,…,F i(l) ,…,F i (k) ) (1)

[0053] In formula (1), Concat is a cascade operation along the channel direction. By fusing multiple features, on the one hand, the representative information in each feature is mined, and at the same time, the dimension of the feature is reduced, which speeds up the calculation.

[0054] Step 2.2: Use convolution kernels of different sizes to calibrate the feature vector f i1 Perform convolution operation to obtain the i-th sample I i The J different scale feature vectors {f ij |j=1,2,…,J}, where f ij Indicates the i-th sample I i The j-th feature of different scales;

[0055] Step 2.3: The i-th sample I i The J different scale feature vectors {f ij |j=1,2,…,J} are respectively input into J multi-head self-attention modules MHSA for multimodal interaction to obtain the feature vector {f ij ′j=1,2,…,J}, where f ij ′ represents the i-th sample I i The feature after the j-th feature is fused; in this embodiment, a two-layer MHSA structure is adopted to realize the fusion of multimodal features.

[0056] Step 2.4: Use formula (2) and formula (3) to get the i-th sample I i The j-th low-dimensional feature vector g ij And the j-th scale multimodal feature g i ' j :

[0057] g ij =Reshape(FC(f ij ′)) (2)

[0058] g i ' j =MLP(Concat(g ij ,f i1 )) (3)

[0059] In formula (5), FC is a fully connected layer, Reshape is a reshaping operation to restore the length of the feature vector; Concat is a cascade operation along the channel direction, and MLP is a multi-layer perceptron;

[0060] Step 2.5: Use formula (4) to get the i-th sample I i The multi-scale and multi-modal features G i , thus obtaining a multi-scale multi-modal feature set {G1,G2,…,G i ,,G t};

[0061] G i =MLP(Concat(f i ′1,g i ′1,…,g i ' j ,…,g i ' J )) (4)

[0062] The fusion of modal features can better capture the relationships and dependencies between different modalities, improving the richness and robustness of feature expression. Multi-scale features can more comprehensively describe the information in the image. Through multi-modal interaction and step-by-step fusion, more expressive feature vectors are ultimately obtained, which helps improve model performance.

[0063] Step 3: Obtain the hierarchical structure between samples through the hierarchical clustering method and build a hierarchical similarity model:

[0064] Step 3.1: Use Euclidean distance to calculate the distance between any two sample features and compare them to find the two samples with the closest distance and merge them into one cluster. Then, merge all samples in I to obtain a hierarchical structure between samples.

[0065] Step 3.2: Use the hierarchical structure obtained by clustering as the hierarchical network link structure element, use each cluster center as the relationship neuron, and regard the cluster center of the first layer as the relationship vector;

[0066] To better reflect the hierarchical relationships between samples, this model does not fully link all neurons. Instead, hidden layer neurons are linked using hierarchical clustering. This method uses five 5x5 convolution kernels. The resulting hierarchical structure controls the number of neurons in each layer and the state of their connections. When neurons in a layer are related, they are linked; otherwise, their weights are set to 0, making the neurons in the layer and their connections more interpretable.

[0067] Step 4: Construct similarity equation:

[0068] Step 4.1 defines the current hierarchical structure as h and initializes h = 1;

[0069] Step 4.2: Use formula (5) to calculate the i-th sample I in the h-1-th hierarchical structure i and the jth sample Ij Similarity under the t-th relationship vector Thus we get the i-th sample I i and the jth sample I j Similarity in the h-1 hierarchy

[0070]

[0071] In formula (5), G j Indicates the jth sample I j Multi-scale and multi-modal features of , T represents transposition; e t is the t-th relationship vector in the h-1-th hierarchical structure, α and β are two control parameters, and Softplus is the activation function; d h-1 is the number of relation neurons in the h-1th level structure;

[0072] This similarity calculation equation first measures the similarity between sample i and sample j separately, then calculates the final similarity by multiplying the two similarities. In practical tasks, the relationship between samples is often determined by only a few relationships. Therefore, the softplus equation is used here to force the similarity calculation equation to maintain sparsity across different relationships, meaning that only a few relationship vectors contribute to the final similarity calculation.

[0073] Step 4.3: Calculate the i-th sample I using formula (6) i and the jth sample I j The hierarchical similarity equation in the h-th hierarchical structure like Figure 2 As shown, the i-th sample I is obtained i The similarity γ with the j-th sample output at the last level ij ;

[0074]

[0075] In formula (6), is the hth level structure h ×d h-1 dimensional weight matrix, d h represents the number of relation neurons in the h-th level structure, represents the weight linking the i-th neuron in the h-1-th layer to the j-th neuron in the h-th layer, b h is d in the hth level structure h dimensional bias vector, sig is the activation function; when h=1, let H represents the total number of layers in the hierarchical structure;

[0076] Through this layer-by-layer approach, the output of each layer depends on the similarity score of the previous layer. Linear and nonlinear transformations are performed layer by layer, gradually extracting and fusing multi-level feature information, ultimately achieving a more accurate and robust similarity score. This process combines hierarchical feature representations of multi-scale and multi-modal data, making image similarity matching more accurate.

[0077] Step 5: Use formula (6) to calculate the i-th sample I i The similarity γ with the jth sample ij , and use formula (7) to construct the loss function L, and use the global optimization method to train the feature fusion model and the hierarchical similarity model, and use the back propagation algorithm to iteratively update each relationship neuron to obtain the optimal image similarity matching model;

[0078]

[0079] In formula (7), I + is a set of positive sample images with the same label in the image dataset I, I - is a set of negative sample images with different labels in the image dataset I; ij- Indicates that the i-th sample I i The jth sample I with different labels j The similarity of γ ij+ Indicates that the i-th sample I i The jth sample I with the same label j , θ is the set of all parameters to be trained in similarity learning, (i, j+) represents a positive sample pair, and (i, j-) represents a negative sample pair;

[0080] Step 6: Use formula (7) to construct the loss function L, and train the feature fusion model and the hierarchical similarity model through the global optimization method, and use the back propagation algorithm to iteratively update each relationship neuron to obtain the optimal image similarity matching model;

[0081]

[0082] In formula (7), I + is a set of positive sample images with the same label in the image dataset I, I - is a set of negative sample images with different labels in the image dataset I; ij- Indicates that the i-th sample I i The jth sample I with different labels j The similarity of γ ij+ Indicates that the i-th sample I i The jth sample I with the same labelj , θ is the set of all parameters to be trained in similarity learning, (i, j+) represents a positive sample pair, and (i, j-) represents a negative sample pair;

[0083] To train hierarchical similarity, we first fully connect all neurons for training as a pre-training step. Then, based on the results of hierarchical clustering, we set the weights of neurons that should not be connected to zero. Through this optimization process, the similarity of positive image pairs increases while the similarity of negative image pairs decreases. An error margin of 1 is added to the loss function to prevent overfitting during optimization. Stochastic gradient descent is used to optimize the parameters of the loss function.

[0084] Step 7: Set the i-th sample I i For index samples, the best image similarity matching model is used to detect the index samples, and the image pair with the highest similarity to the index samples is obtained and used as the most similar image pair result.

[0085] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0086] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are executed.

Claims

1. A hierarchical image similarity matching method based on multi-scale and multi-modal features, characterized in that: The steps are as follows: Step 1: Get an image dataset I={I1,I2,…,I i ,…,I t }, where I i represents the i-th sample; t represents the number of samples in the image dataset; Extract the i-th sample I i The features of k different modes are obtained, and the feature vector set {F i (1) ,F i (2) ,...,F i (l) ,…,F i (k) }, where F i (l) Indicates the i-th sample I i The lth modal feature of; Step 2: Build a feature fusion model and use it to obtain the i-th sample I i The multi-scale and multi-modal features G i : Step 3: Obtain the hierarchical structure between samples through the hierarchical clustering method and build a hierarchical similarity model: Step 3.1: Use Euclidean distance to calculate the distance between any two sample features and compare them to find the two samples with the closest distance and merge them into one cluster. Then, merge all samples in I to obtain a hierarchical structure between samples. Step 3.2: Use the hierarchical structure obtained by clustering as the hierarchical network link structure element, use each cluster center as the relationship neuron, and regard the cluster center of the first layer as the relationship vector; Step 4: Construct a similarity equation and use it to obtain the i-th sample I i The similarity γ with the jth sample ij : Step 5: Use formula (1) to construct the loss function L, and train the feature fusion model and the hierarchical similarity model through the global optimization method, and use the back propagation algorithm to iteratively update each relationship neuron to obtain the optimal image similarity matching model; In formula (1), I + is a set of positive sample images with the same label in the image dataset I, I - is a set of negative sample images with different labels in the image dataset I; ij- Indicates that the i-th sample I i The jth sample I with different labels j The similarity of γ ij+ Indicates that the i-th sample I i The jth sample I with the same label j , θ is the set of all parameters to be trained in similarity learning, (i, j+) represents a positive sample pair, and (i, j-) represents a negative sample pair; Step 6: Set the i-th sample I i For index samples, the best image similarity matching model is used to detect the index samples, and the image pair with the highest similarity to the index samples is obtained and used as the most similar image pair result.

2. The image similarity matching method based on a hierarchical structure of multi-scale and multi-modal features according to claim 1, characterized in that: Described step 2 is carried out as follows: Step 2.1: Generate the i-th sample I using formula (2) i The initial eigenvector f i1 : f i1 =concat(F i (1) ,F i (2) ,...,F i (l) ,…,F i (k) ) (2) In formula (1), Concat is a cascade operation along the channel direction; Step 2.2: Use convolution kernels of different sizes to calibrate the feature vector f i1 Perform convolution operation to obtain the i-th sample I i The J different scale feature vectors {f ij |j=1,2,…,J}, where f ij Indicates the i-th sample I i The j-th feature of different scales; Step 2.3: The i-th sample I i The J different scale feature vectors {f ij |j=1,2,...,J} are respectively input into J multi-head self-attention modules MHSA for multimodal interaction to obtain the feature vector {f ij ′j=1,2,...,J}, where f ij ′ represents the i-th sample I i The feature after the j-th feature fusion; Step 2.4: Use formula (3) and formula (4) to get the i-th sample I i The j-th low-dimensional feature vector g ij And the j-th scale multimodal feature g i ' j : g ij =Reshape(FC(f ij ′)) (3) g i ′ j =MLP(Concat(g ij ,f i1 )) (4) In formula (5), FC is a fully connected layer, Reshape is a reshaping operation to restore the length of the feature vector; Concat is a cascade operation along the channel direction, and MLP is a multi-layer perceptron; Step 2.5: Use formula (5) to get the i-th sample I i The multi-scale and multi-modal features G i , thus obtaining a multi-scale multi-modal feature set {G1,G2,…,G i ,…,G t }; G i =MLP(Concat(f′ i1 ,g′ i1 ,…,g′ ij ,…,g′ iJ )) (5)。 3. The image similarity matching method based on a hierarchical structure of multi-scale and multi-modal features according to claim 2, characterized in that: Described step 4 is carried out as follows: Step 3.1 defines the current hierarchical structure as h and initializes h = 1; Step 3.2: Use formula (6) to calculate the i-th sample I in the h-1-th hierarchical structure i and the jth sample I j Similarity under the t-th relationship vector Thus we get the i-th sample I i and the jth sample I j Similarity in the h-1 hierarchy In formula (5), G j Indicates the jth sample I j Multi-scale and multi-modal features of , T represents transposition; e t is the t-th relationship vector in the h-1-th hierarchical structure, α and β are two control parameters, and Softplus is the activation function; d h-1 is the number of relation neurons in the h-1th level structure; Step 3.3: Calculate the i-th sample I using formula (7) i and the jth sample I j The hierarchical similarity at the hth level Thus we get the i-th sample I i The similarity with the j-th sample output at the last level is denoted as γ ij ; In formula (7), is the hth level structure h ×d h-1 dimensional weight matrix, d h represents the number of relation neurons in the h-th level structure, represents the weight linking the i-th neuron in the h-1-th layer to the j-th neuron in the h-th layer, b h is d in the hth level structure h dimensional bias vector, sig is the activation function; when h=1, let H represents the total number of layers in the hierarchical structure.

4. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the image similarity matching method according to any one of claims 1 to 3, and the processor is configured to execute the program stored in the memory.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image similarity matching method according to any one of claims 1 to 3 are executed.

Images