An image similarity matching method based on external memory attention weight distribution

Abstract

The application discloses an image similarity matching method based on attention weight distribution of external memory, comprising the following steps: 1, obtaining image original features of an image data set; 2, replacing the image original features with the result of representation learning to obtain high-level representation thereof; 3, performing simple similarity calculation on input samples to obtain external initial memory thereof; 4, constructing two external selection models to distribute external memory selection weights, including an external sample selection model and an external relationship selection model; 5, optimizing and training the external selection models by using two training methods of selection optimization and global optimization to obtain an optimal training model; and 6, detecting the input samples by using the trained external selection model to obtain an image pair with the highest external memory selection weight as the most similar image pair. The application can improve the similarity learning effect, thereby obtaining the globally most similar image, and the image matching result has better interpretability.

Description

Technical Field

[0001] This invention belongs to the field of image retrieval and mainly relates to an image similarity matching method based on attention weight allocation of external memory. Background Technology

[0002] In today's era, the massive growth of image data has created a demand for efficient image data management and utilization methods. Image retrieval technology has emerged to address this need, aiming to solve the challenge of quickly locating the images a user needs from vast image databases.

[0003] In the field of image retrieval, common retrieval algorithms include feature-based methods, content-based image retrieval (CBIR), and advanced methods based on deep learning. However, existing traditional algorithms often evaluate the relevance between input samples from a single modality perspective, limiting their ability to handle complex scenes and object relationships in images. For example, when a user seeks images similar to images containing multiple objects, traditional algorithms may rely solely on single modalities such as pixel values ​​or feature vectors for similarity matching, ignoring the relationships between objects and their semantic information. This can lead to inaccurate or incomplete retrieval results.

[0004] Therefore, traditional single-modal image similarity matching methods often fail to achieve satisfactory relevance assessments when faced with complex scenarios in real-world applications. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, this invention provides a novel image similarity matching method. This method is based on an external memory mechanism, which uses attention weight allocation to obtain different similarities between samples under different relationships, thereby obtaining the image pair with the highest weight selected by external memory as the most similar image pair.

[0006] To solve the technical problem, the present invention adopts the following technical solution:

[0007] The image similarity matching method based on external memory attention weight allocation of this invention is characterized by the following steps:

[0008] Step 1: Obtain an image dataset containing m samples. Where, x p This represents the p-th sample; m represents the number of samples in the dataset.

[0009] Using convolutional neural networks on image datasets The p-th sample x p Feature extraction is performed to obtain the high-level feature representation Φ of the p-th sample with feature dimension k. p =Φ(x p, ω), wherein ω is a set of parameters to be learned in the convolutional neural network;

[0010] Step 2: obtaining n relationship vectors with a dimension of k wherein a y represents the yth relationship vector; n represents the number of relationship vectors;

[0011] Step 3: constructing the pth sample x p and the qth sample x q under the yth relationship vector a y as an external initial memory:

[0012]

[0013] In formula (1), sig represents a sigmoid function; T represents a transpose; Φ q is a high-level feature representation of the qth sample x q processed by the convolutional neural network;

[0014] Step 4: constructing two external selection models, including an external sample selection model and an external relationship selection model, for assigning weights to the external memory to obtain an external sample memory selection weight and an external relationship memory selection weight;

[0015] Step 4.1: the external sample selection model calculates the external sample memory selection weight of the ith nearest neighbor sample of the pth sample x y and the jth nearest neighbor sample of the qth sample x p under the yth relationship vector a q

[0016]

[0017] In formula (2), represents the difference in similarity between the ith nearest neighbor sample of the pth sample x y and the jth nearest neighbor sample of the qth sample x p under the yth relationship vector a q and the pth sample and the qth sample, which is obtained by formula (3);

[0018]

[0019] In formula (3), softplus represents a Softplus function; Φ p (i) represents the ith nearest neighbor sample of the pth sample x p , and Φ q (j) represents the jth nearest neighbor sample of the qth sample x​​q the jth neighbor sample of the pth sample x

[0020] Step 4.2: the external relation selection model calculates the external relation memory selection weight α y between the pth sample x p and the qth input sample x q under the yth relation vector a y (pq) by using formula (4):

[0021]

[0022] In formula (4), exp represents an exponential function;

[0023] Step 5: the external selection model is optimized by using a selection optimization method and a global optimization method, and a loss function L is constructed to measure the difference between the output of each external selection model and the actual label, so as to obtain the best external selection model;

[0024] Step 6: the pth sample x p is set as an index sample, and the qth sample x q is set as a test sample; the index sample and the test sample are detected by using the trained best external selection model, so as to obtain the image pairs composed of the index sample and its neighbor samples and the test sample and its neighbor samples under various relation vectors and the corresponding external memory selection weights, and then the image pairs are sorted according to the size of the external memory selection weights, so as to obtain the image pair with the highest external memory selection weight and as the most similar image pair.

[0025] The image similarity matching method based on the attention weight distribution of external memory according to the application has the characteristics that step 5 is performed as follows:

[0026] Step 5.1: the external sample selection model is optimized first, and then the external relation selection model is optimized, so as to obtain more accurate similarity values;

[0027] Step 5.1.1: the comprehensive similarity s p of the pth sample x q and the qth sample x s under the external sample memory is obtained by using formula (5): pq

[0028]

[0029] In formula (5), is the neighbor sample of the pth sample x y and the qth sample x p under the yth relation vector a qThe weighted similarity of the pth sample x

[0030]

[0031] Step 5.1.2: the pth sample x p and the qth sample x q The comprehensive similarity s r pq :

[0032]

[0033] Step 5.2: simultaneously optimize the external sample memory model and the external relationship memory model:

[0034] The pth sample x p and the qth sample x q The comprehensive similarity s pq :

[0035]

[0036] Step 5.3: construct the loss function L by using formula (9):

[0037]

[0038] In formula (9), I + is a positive sample image set with the same label in the image data set , I - is a negative sample image set with different labels in the image data set ; s pq- represents the similarity of the qth input sample x p with different labels from the pth sample x q under the yth relationship vector a y , and s pq+ represents the similarity of the qth input sample x p with the same label as the pth sample x q under the yth relationship vector a y .

[0039] The electronic device comprises a memory and a processor, and is characterized in that the memory is used for storing a program supporting the processor to execute the image similarity matching method, and the processor is configured to execute the program stored in the memory.

[0040] The computer readable storage medium stores a computer program, and the computer program is executed by a processor to perform the steps of the image similarity matching method.

[0041] Compared with the prior art, the beneficial effects of the present application are reflected in:

[0042] 1、The method of the present application utilizes a multi-feature fusion method, which comprehensively mines the relationships within and between features, making the representation learning effect significantly better than directly using the original data features. This fusion strategy not only improves the expression ability of the model, but also enhances the abstract and understanding ability of the image features;

[0043] 2、The method of the present application provides two external memory selection mechanisms based on samples and relationships, which can flexibly select the characteristics of input samples and the relationships between samples according to specific needs. By giving high weight to highlight the sample characteristics and relationships that meet the current needs, and giving low weight to ignore irrelevant information, the similarity learning effect is effectively improved. This mechanism enables the model to more intelligently adapt to different application scenarios and task requirements, enhancing the generalization ability and applicability of the model;

[0044] 3、The image similarity matching model trained by the method of the present application can bring more accurate image retrieval and matching results in actual application scenarios, providing strong support for image processing and application. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The figure is a structural diagram of the method of the present application. DETAILED DESCRIPTION

[0046] In this embodiment, a similarity learning method using external memory for attention weight distribution is first to obtain the external initial memory by high-level representation and simple similarity calculation on the image dataset; then two external memory selection models are constructed to allocate external memory selection weights, including an external sample selection model and an external relationship selection model; the external sample memory selection model selects the given sample by finding different neighboring samples of the given sample, describes the different characteristics of the given sample by using different neighboring samples, and selects the given sample by using the initial memory between the given samples, which gives a higher weight to the sample pair that is more consistent with the actual conditions; the external relationship memory selection model selects the different relationships between the samples by the initial memory, which gives a higher weight to the relationship vector that is more consistent with the actual conditions; then the external selection model is optimized and trained by two training methods of selection optimization and global optimization; finally, the input samples (index samples and test samples) are detected on the trained model to obtain the pair with the highest external memory selection weight of the index sample and its neighboring samples and the test sample and its neighboring samples under various relationship vectors, as the most similar image pair. When the test sample changes, the external memory selection weight will change with the change of the test sample, which reflects the commonality and hidden relationship between the index sample and the test sample. Specifically, the following steps are taken:

[0047] Step 1: Obtain an image dataset containing m samples where x p represents the pth sample; m represents the number of samples in the dataset;

[0048] Use a convolutional neural network to extract features from the pth sample x p in the image dataset p , and obtain the high-level feature representation of the pth sample with feature dimension k Φ p = Φ(x y , ω), where ω is the parameter set to be learned in the convolutional neural network; the image original features of the sample are used as input to obtain its high-level representation.

[0049] Step 2: Obtain n relationship vectors with dimension k where a p represents the yth relationship vector; n represents the number of relationship vectors;

[0050] Step 3: Use formula (1) to construct the similarity of the pth sample x q and the qth sample x y under the yth relationship vector a q as the external initial memory:

[0051] ​

[0052] In formula (1), sig represents a sigmoid function; T represents a transpose; Φ q is the qth sample x q is a high-level feature representation processed by a convolutional neural network; an initial memory outside is generated by a most basic multi-modal similarity equation.

[0053] Step 4: Two external selection models are constructed, including an external sample selection model and an external relationship selection model, for assigning weights to external memories, to obtain an external sample memory selection weight and an external relationship memory selection weight.

[0054] Step 4.1: The external sample selection model calculates the external sample memory selection weight of the ith neighbor sample of the pth sample x y and the jth neighbor sample of the qth sample x p under the yth relationship vector a q

[0055]

[0056] In formula (2), represents the similarity difference between the ith neighbor sample of the pth sample x y and the jth neighbor sample of the qth sample x p and the pth sample and the qth sample under the yth relationship vector a q , which is obtained by formula (3);

[0057]

[0058] In formula (3), softplus represents a Softplus function; Φ p (i) represents the ith neighbor sample of the pth sample x p , and Φ q (j) represents the jth neighbor sample of the qth sample x q ; by finding different neighbor samples of a given sample, using different neighbor samples to describe different characteristics of the given sample, and selecting the given sample through the initial memory between the given samples, a sample pair that is more in line with the actual conditions will be given a higher weight.

[0059] Step 4.2: The external relationship selection model calculates the external relationship memory selection weight α y (pq) between the pth sample x p and the qth input sample x q under the yth relationship vector a y

[0060] ​​

[0061] In formula (4), exp represents an exponential function; the different relations between samples and samples are selected by initial memory, and the more significant sample pairs and relation vectors under a given training requirement are highlighted while ensuring that the sum of weights is 1.

[0062] Step 5: Optimize the training of the external selection model by selecting optimization methods and global optimization methods, and construct a loss function L to measure the difference between the output of each model and the actual label to obtain the best external selection model.

[0063] Step 5.1: Select optimization, that is, first optimize the external sample selection model separately, and then optimize the external relation selection model separately to obtain more accurate similarity values.

[0064] Step 5.1.1: According to the external sample memory selection weight obtained from formula (3), construct a similarity calculation formula as shown in formula (5) to obtain the comprehensive similarity s p of the pth sample x q and the qth sample x s under the external sample memory. pq :

[0065]

[0066] In formula (5), is the similarity of the pth sample x y and the qth sample x p under the yth relation vector a q , and the similarity obtained by weighted sum of the external sample memory selection weight, which is obtained by formula (6):

[0067]

[0068] The optimization first optimizes the external sample selection model separately by formula (5), that is, the external sample memory selection weight obtained by formula (3) and the similarity between each neighbor sample obtained by formula (6) are used for weighted sum to obtain the comprehensive similarity s p of the pth sample and the qth sample under the external sample memory as a more accurate similarity value.

[0069] Step 5.1.2: According to the external relation memory selection weight obtained from formula (4), construct a similarity calculation formula as shown in formula (7) to obtain the comprehensive similarity s q of the pth sample x r and the qth sample x pq under the external relation memory. :

[0070]

[0071] Then, the external relation memory selection model is optimized separately by Equation (7), that is, by using the external relation memory selection weight obtained by Equation (4) and the external initial memory obtained by Equation (1) to perform a weighted summation, and obtain the comprehensive similarity between the p-th sample and the q-th sample under the external relation memory, as a more accurate similarity value.

[0072] Step 5.2: Global optimization, which involves simultaneously optimizing both the external sample memory model and the external relation memory model:

[0073] Based on the external sample memory selection weights obtained from equation (3) and the external relation memory selection weights obtained from equation (4), the similarity calculation formula shown in equation (8) is constructed to obtain the p-th sample x. p and the qth sample x q The combined similarity s under external sample memory and external relational memory pq :

[0074]

[0075] Equation (8) is used to simultaneously optimize the external sample memory selection model and the external relation memory selection model. Specifically, the external relation memory selection weight obtained from Equation (4) and the similarity between each nearest neighbor sample obtained from Equation (6) are weighted and summed to obtain the comprehensive similarity between the p-th sample and the q-th sample under external sample memory and external relation memory, which serves as a more accurate similarity value.

[0076] Step 5.3: Using the loss function L shown in Equation (9), calculate the comprehensive similarity corresponding to the above optimized external selection model to measure the difference between the output of each external selection model and the actual label, and compare them to obtain the best external selection model:

[0077]

[0078] In equation (9), I + For image datasets A set of positive sample images with the same label, I - For image datasets A collection of negative example images with different labels; s pq- Indicates the relationship with the p-th sample x p The qth input sample x with different labels q The y-th relation vector a y Similarity under s pq+ Indicates the relationship with the p-th sample x pThe qth input sample x with the same label q The similarity under the yth relation vector a y The similarity under the yth relation vector a

[0079] Step 6: Set the pth sample x p As an index sample, the qth sample x q As a test sample; using the trained optimal external selection model to detect the index sample and the test sample, obtaining the image pairs composed of the index sample and its nearest neighbor samples and the test sample and its nearest neighbor samples under various relation vectors and their corresponding external memory selection weights, and then sorting according to the size of the external memory selection weight, thereby obtaining the image pair with the highest external memory selection weight, and taking it as the most similar image pair.

[0080] By sorting the size relationship of the external memory selection weights of the obtained index sample and its nearest neighbor samples and the test sample and its nearest neighbor samples under different relation vectors, the image pair with the highest external memory selection weight can be obtained, which realizes the allocation of different attention weights according to the different characteristics of the samples and the different relations between the samples under specific tasks. In addition, when the test sample changes, the external memory selection weight changes accordingly. It embodies the commonality and hidden relationship between the index sample and the test sample.

[0081] As Figure 1 shown, the red line and the blue line respectively represent the link with the highest memory selection weight, and the black dashed line represents the link without enhancement. The triangle represents the input sample whose similarity needs to be calculated, the square represents different nearest neighbor samples of each sample, and the circle represents different relation vectors.

[0082] In this embodiment, an electronic device includes a memory for storing a program supporting a processor to execute the above method, and a processor configured to execute the program stored in the memory.

[0083] In this embodiment, a computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to perform the steps of the above method.

Claims

1. An image similarity matching method based on external memory attention weight allocation, characterized in that, The procedure is as follows: Step 1: Obtain an image dataset containing m samples. Where, x p Let m represent the p-th sample; m represents the number of samples in the dataset. Using convolutional neural networks on image datasets The p-th sample x p Feature extraction is performed to obtain the high-level feature representation Φ of the p-th sample with feature dimension k. p =Φ(x p ,ω), where ω is the set of parameters to be learned in the convolutional neural network; Step 2: Obtain n relation vectors with dimension k. Among them, a y This represents the y-th relation vector; n represents the number of relation vectors. Step 3: Construct the p-th sample x using equation (1) p and the qth sample x q The y-th relation vector a y Similarity As an external initial memory: In equation (1), sig represents the sigmoid function; T represents the transpose; Φ q For the q-th sample x q High-level feature representation processed by a convolutional neural network; Step 4: Construct two external selection models, including an external sample selection model and an external relation selection model, to allocate external memory selection weights and obtain external sample memory selection weights and external relation memory selection weights; Step 4.1: The external sample selection model uses equation (2) to calculate the y-th relation vector a. y The p-th sample x p The i-th nearest neighbor sample and the q-th sample x q External sample memory selection weight of the j-th nearest neighbor sample In equation (2), This represents the y-th relation vector a y The p-th sample x p The i-th nearest neighbor sample and the q-th sample x q The similarity difference between the j-th nearest neighbor sample and the p-th and q-th samples is obtained by equation (3); In equation (3), softplus represents the Softplus function; Φ p (i) represents the p-th sample x p The i-th nearest neighbor sample, Φ q (j) represents the q-th sample x q The j-th nearest neighbor sample; Step 4.2: External Relationship Selection Model. Calculate the y-th relation vector a using equation (4). y The p-th sample x p and the qth input sample x q External Relationship Memory Selection Weight α y (pq): In equation (4), exp represents the exponential function; Step 5: Optimize the external selection model using selection optimization and global optimization methods, and construct a loss function L to measure the difference between the output of each external selection model and the actual label, so as to obtain the best external selection model; Step 6: Set the p-th sample x p For the index sample, the q-th sample x q The test samples are used to detect the index samples and test samples using the trained best external selection model. This results in image pairs consisting of the index samples and their nearest neighbors, and the test samples and their nearest neighbors, under various relation vectors, along with their corresponding external memory selection weights. The images are then sorted according to the magnitude of the external memory selection weights to obtain the image pairs with the highest external memory selection weights, which are then used as the most similar image pairs.

2. The image similarity matching method based on external memory attention weight allocation according to claim 1, characterized in that, Step 5 is performed as follows: Step 5.1: First optimize the external sample selection model, then optimize the external relationship selection model to obtain a more accurate similarity value; Step 5.1.1: Use equation (5) to obtain the p-th sample x. p and the qth sample x q Comprehensive similarity s under external sample memory s pq : In equation (5), For the y-th relation vector a y The p-th sample x p The nearest neighbor samples and the qth sample x q The weighted similarity is obtained from equation (6): Step 5.1.2: Use equation (7) to obtain the p-th sample x. p and the qth sample x q Comprehensive similarity s under external relational memory r pq : Step 5.2: Simultaneously optimize the external sample memory model and the external relation memory model: The p-th sample x is obtained using equation (8). p and the qth sample x q The combined similarity s under external sample memory and external relational memory pq : Step 5.3: Construct the loss function L using equation (9): In equation (9), I + For image datasets A set of positive sample images with the same label, I - For image datasets A collection of negative example images with different labels; Indicates the relationship with the p-th sample x p The qth input sample x with different labels q The y-th relation vector a y Similarity under Indicates the relationship with the p-th sample x p The qth input sample x with the same label q The y-th relation vector a y The similarity is as follows.

3. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the image similarity matching method of claim 1 or 2, and the processor is configured to execute the program stored in the memory.

4. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program is executed by the processor to perform the steps of the image similarity matching method according to claim 1 or 2.

Images