Image retrieval method based on difficult negative sample representation synthesis

Abstract

The invention discloses an image retrieval method based on difficult-to-negative sample representation synthesis. The method comprises the following steps: constructing a difficult-to-negative sample representation generation network capable of synthesizing rich information; a batch image constructed by a category equalization sampling strategy is used as network input, batch image representation is extracted through an image representation extraction module, and then the batch image representation is input into an inter-sample global association learning module for learning; inputting the mutually negative sample pairs into a correlation perception channel diversity interpolation module to synthesize difficult-to-negative sample characterization; training the difficult-to-negative sample representation generation network capable of synthesizing rich information, and jointly training an image representation extraction module by using synthesized difficult-to-negative sample representation and real sample representation; performing image retrieval by using the trained image representation extraction module; by combining the global correlation learning ability between samples, difficult-to-load sample representation with more information amount is synthesized, and the image representation extraction module is guided to extract image representation with higher discrimination so as to enhance the retrieval performance.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and artificial intelligence technology, and in particular relates to an image retrieval method based on hard negative sample representation synthesis. Background Art

[0002] In the digital age, image retrieval technology combines traditional visual understanding with modern deep learning methods, providing an efficient way to quickly retrieve target images similar to query images from huge image databases. The core of the image retrieval task is to design a metric method that can accurately quantify sample similarity in the metric space, so that similar images are close to each other in the space, while dissimilar images are far away from each other. In order to improve the discriminative ability of image retrieval models, this task aims to synthesize information-rich representations of hard negative samples to enhance the model's discrimination on category boundaries in the metric space. This plays an important role in achieving efficient and accurate image retrieval in complex category distribution scenarios.

[0003] Current research on hard negative representation synthesis usually synthesizes hard negative representations by fusing a small number of sample pairs or triplets through generative adversarial networks (GANs) or auto-encoders. These methods mainly focus on the local correlation between a few selected samples, and therefore may have the problem of inappropriate difficulty levels in the generated negative representations. This is because they lack consideration of the potential impact of other categories. In this case, hard negative representation synthesis methods that only consider local sample correlations may synthesize negative representations that do not match the target category distribution, thereby interfering with the training process of the image retrieval model and resulting in limited retrieval performance. This highlights the necessity of considering a wider range of negative sample category correlations in the process of hard negative representation synthesis in order to more accurately grasp the similarities between different categories, thereby generating negative representations with appropriate difficulty and category correlation to facilitate the retrieval model to distinguish category boundaries.

[0004] The main challenge of the image retrieval task based on hard negative sample representation synthesis is how to learn and utilize the associated information of a set of sample pairs in the global category distribution to synthesize informative hard negative sample representations and use them to enhance the accuracy of the image retrieval model. Summary of the invention

[0005] In view of this, it is necessary to address the above technical problems. The present invention provides an image retrieval method based on hard negative sample representation synthesis, comprising the following steps:

[0006] Step 1: construct a network capable of synthesizing information-rich hard negative sample representation generation, including an image representation extraction module, a global association learning module between samples, and a correlation-aware channel diversity interpolation module;

[0007] Step 2: The batch images constructed by the class-balanced sampling strategy are used as network input, and the batch image representations are extracted by the image representation extraction module. The batch image representations are then input into the global association learning module between samples to learn the correlation between each anchor sample and all other negative samples. The sample pairs that are mutually negative are input into the association-aware channel diversity interpolation module, and the hard negative sample representations are interpolated and synthesized according to the correlation of the sample pairs.

[0008] Step 3, training the network capable of synthesizing information-rich hard negative sample representation generation, and using the synthesized hard negative sample representation and the real sample representation to jointly train the image representation extraction module;

[0009] Step 4: Use the trained image representation extraction module to perform image retrieval.

[0010] Furthermore, the image representation extraction module includes a convolutional neural network and a fully connected layer; the batch image representation extraction steps are specifically as follows:

[0011] Step 20201: After the batch images are input into the convolutional neural network, they are compressed by two-dimensional maximum pooling to obtain a one-dimensional tensor;

[0012] Step 20202: The one-dimensional tensor is processed by batch normalization, a fully connected layer, and L2 norm normalization to obtain the batch image representation;

[0013] The batch image representation includes the representation of all samples in the batch image.

[0014] Furthermore, the inter-sample global association learning module includes a node message propagation network and an edge message propagation network. The specific steps of building the inter-sample global association learning module are as follows:

[0015] Step 20301: The sample global association learning module constructs the batch image representation into a graph structure, wherein the graph structure includes a plurality of nodes and edges, each of the nodes is a representation of a single sample, and each of the edges is a dot product of the representations of the mutually negative sample pairs;

[0016] Step 20302: the node message propagation network acts on all the nodes in the entire graph structure; wherein each node fuses the adjacent nodes and edges through the Tranformer mechanism, so that the node learns the global sample association information;

[0017] Step 20303: the edge message propagation network acts between each pair of negative class samples to integrate the global sample association information learned by the node into the edge;

[0018] Step 20304: After two iterations of the node message propagation network and the edge message propagation network, the construction of the global association learning module between samples is completed.

[0019] Furthermore, the association-aware channel diversity interpolation module includes an association-aware sample pair interpolation module and an intra-class diversity interpolation module. The steps for building the association-aware channel diversity interpolation module are as follows:

[0020] Step 20401: The graph structure obtains the diversity interpolation weight of each negative class sample pair, the training loss of the image representation extraction module, and the distance relationship interpolation between samples in the batch image through the association-aware sample pair interpolation module, and synthesizes the difficult negative sample representation between the negative class sample pairs;

[0021] Step 20402: The intra-class diversity interpolation module randomly weights all the hard negative sample representations of the same negative class corresponding to each anchor sample to mine potential hard negative sample representations.

[0022] Furthermore, the specific steps of step 3 are as follows:

[0023] Step 20501: Use a training function to train the network capable of synthesizing information-rich difficult negative sample representation generation network;

[0024] Step 20502: Optimize the image representation extraction module using an optimization function;

[0025] Step 20503: During the training of the network capable of synthesizing information-rich hard negative sample representation generation, overlapping training is performed on step 20501 and step 20502.

[0026] Further, the training function includes classification loss, similarity loss and diversity loss;

[0027] The classification loss ensures that the corresponding negative class distribution is maintained between the synthesized representations of the hard negative samples through a classification head network;

[0028] The similarity loss makes the representation of the hard negative sample similar to the representation of the anchor sample under the premise that the classification loss maintains the corresponding negative class distribution between the representations of the synthetic hard negative samples;

[0029] The diversity loss improves the standard deviation of the interpolation weights to expand the search space for synthesizing the hard negative sample representation within a reasonable constraint.

[0030] Further, the optimization function includes global association learning loss, synthetic hard negative sample representation optimization loss and real sample representation optimization loss;

[0031] The global association learning loss enables the node to maintain category semantic information by performing category constraints on the node;

[0032] The synthetic hard negative sample representation optimizes the loss, and optimizes the metric space by combining the similarity relationship between the sample pairs of the real sample representation and the hard negative sample representation;

[0033] The real sample representation optimizes the loss, and the similarity relationship between sample pairs represented by the real samples is combined to optimize the metric space.

[0034] Furthermore, the specific steps of step 4 are as follows:

[0035] The image representation extraction module extracts database image representations from all images in the database and saves them. After the query image is input into the image representation extraction module to extract the query image representation, the cosine similarity between the query image representation and the database image representation is calculated, and the top K results in descending order of similarity are selected as output.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention effectively considers the correlation between samples in the global category distribution, and can synthesize difficult negative sample representations with appropriate difficulty and category correlation, which helps the image retrieval model to efficiently distinguish category boundaries and improve the image retrieval accuracy;

[0038] The present invention also extracts image representation through an image representation extraction module. The proposed global association learning module between samples can learn the correlation between each sample and all other negative samples in the batch in the global category distribution; the proposed association-aware channel diversity interpolation module interpolates and synthesizes difficult negative sample representations according to the learned correlation between samples; the described difficult negative sample representation generation network capable of synthesizing information-rich difficult negative sample representations is alternately trained, and the image representation extraction module is jointly trained using the synthesized difficult negative sample representations and the real sample representations, so as to guide the image representation extraction module to efficiently distinguish category boundaries and continuously synthesize more difficult difficult negative sample representations to enhance the discriminability of the model in the metric space, thereby achieving accurate image retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic flow chart of an implementation method of the present invention is shown;

[0040] Figure 2 A schematic diagram showing the structure of each module working in an embodiment of the present invention is shown;

[0041] Figure 3 A structural diagram of the class-balanced sampling strategy is shown. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] For the purpose of reference and clarity, the technical terms, abbreviations or acronyms used below are summarized and explained as follows:

[0044] Sigmoid / ReLU: non-linear activation function.

[0045] ResNet50: A convolutional neural network that belongs to the residual network family and consists of 50 layers, including convolutional layers, batch normalization layers, ReLU activation functions, and fully connected layers.

[0046] Transformer: A neural network architecture for processing sequence data.

[0047] Batch Normalization: A method for accelerating neural network training and improving model stability. The core idea is to normalize the batch image representation of each batch of images to alleviate the gradient vanishing and gradient exploding problems and make the network training more stable and converge faster.

[0048] Metric space: Metric space refers to the geometric space in which samples are embedded as feature representations, and the distance between samples can be measured by some metric function (such as Euclidean distance, cosine similarity, etc.).

[0049] Anchor samples: Anchor samples refer to reference samples used to learn the distance or similarity between samples. Their main function is to compare with other samples and calculate the similarity difference between positive samples and negative samples.

[0050] Positive samples: Positive samples refer to samples that belong to the same category as the anchor samples. The model expects them to be as close as possible to the anchor samples in the metric space, that is, the similarity is high.

[0051] Negative samples: Negative samples refer to samples that do not belong to the same class as the anchor samples. The model expects that the distance between them and the anchor samples in the metric space should be as far as possible, that is, the similarity is low.

[0052] Positive sample pair: A positive sample pair refers to a combination of two similar samples, usually including an anchor sample and a positive sample.

[0053] Negative sample pair: A negative sample pair refers to a combination of samples of two different classes, usually including an anchor sample and a negative sample.

[0054] Hard negative samples: Hard negative samples refer to those negative samples that come from different categories than anchor samples but are close in the metric space. Such samples are challenging to train the model because the model needs to distinguish them as much as possible rather than mistakenly believe that they belong to the same category.

[0055] L2 norm normalization: L2 norm normalization refers to scaling a vector by its L2 norm so that the L2 norm of the normalized vector is equal to 1. This is often used in machine learning to make data more stable or easier to calculate, especially when dealing with feature vectors.

[0056] The invention discloses an image retrieval method based on hard negative sample representation synthesis to solve many problems existing in the prior art.

[0057] Figure 1 A flow chart of an embodiment of the present invention is shown, which is an image retrieval method based on hard negative sample representation synthesis, comprising the following steps:

[0058] Step 1: construct a network capable of synthesizing information-rich hard negative sample representation generation, including an image representation extraction module, a global association learning module between samples, and a correlation-aware channel diversity interpolation module;

[0059] Step 2: The batch images constructed by the class-balanced sampling strategy are used as network input, and the batch image representations are extracted by the image representation extraction module. The batch image representations are then input into the global association learning module between samples to learn the correlation between each anchor sample and all other negative samples. The sample pairs that are mutually negative are input into the association-aware channel diversity interpolation module and the hard negative sample representations are interpolated and synthesized according to the correlation of the sample pairs.

[0060] Step 3, training the network capable of synthesizing information-rich hard negative sample representation generation, and using the synthesized hard negative sample representation and the real sample representation to jointly train the image representation extraction module;

[0061] Step 4: Use the trained image representation extraction module to perform image retrieval.

[0062] Specifically, Figure 2As shown, the inter-sample global association learning module and the association-aware channel diversity interpolation module in step 2 are only used to synthesize difficult negative sample representations during the model training process. The difficult negative sample representations are then combined with the real sample representations to jointly train the image representation extraction module for image retrieval, so that the image representation extraction module can extract more discriminative real sample representations from the input batch images, thereby improving its retrieval accuracy in the metric space. During the test process, only the image representation extraction module is needed to extract image representations from the query image (input image) and the database image for similarity measurement to achieve image retrieval.

[0063] Furthermore, the specific operation steps of step 2 in this embodiment are as follows:

[0064] Step 201. A batch of images constructed using a class-balanced sampling strategy is used as network input;

[0065] The steps of the class-balanced sampling strategy are as follows:

[0066] Each time a batch of images is constructed, 27 categories are sampled from the entire data set, 3 samples are sampled for each category, and a total of 81 samples are sampled; among them, the first 27 samples in the batch image are 1 sample sampled in each category, and the subsequent samples after the 27 samples are also arranged in the same category order; Figure 3 As shown, sampling There are m categories, and m samples are sampled in each category. Therefore, The samples are x 1,1 to The second group The samples are x 2,1 to Subsequent group m The samples are x m,1 to Total samples; the class balanced sampling strategy is to avoid the lack of attention to the class with a small number of samples due to random sampling;

[0067] Next, the batch images are randomly cropped and deformed to a resolution of 224*224, and then the batch images are subjected to random horizontal flip enhancement and used as network input.

[0068] Step 202: construct an image representation extraction module, including a ResNet50 convolutional neural network and a fully connected layer; wherein the batch image representation extraction step is specifically as follows:

[0069] Step 20201: Input the batch images into the ResNet50 convolutional neural network and downsample by 32 times to obtain a tensor of 7×7×2048 dimensions, and then compress it to a one-dimensional tensor of 2048 dimensions through two-dimensional maximum pooling;

[0070] Step 20202: After performing batch normalization on the one-dimensional tensor, the one-dimensional tensor is input into a fully connected layer and compressed into a one-dimensional tensor of 512 dimensions, and finally subjected to L2 norm normalization processing to obtain the batch image representation z; wherein the batch image representation z includes the representations of all samples in the batch image;

[0071] Step 203: Building a global association learning module between samples, wherein the global association learning module between samples includes a node message propagation network and an edge message propagation network;

[0072] Step 20301, the sample global association learning module constructs batch image representation into a graph structure, in which the representation of each sample in the batch image representation is connected with the representations of all mutually negative samples, and the graph structure includes a number of nodes and edges, each node is defined as the representation of a single sample, and each edge is defined as the dot product of the representations of the mutually negative sample pairs;

[0073] Step 20302: The node message propagation network acts on all nodes in the entire graph structure, so that each sample in the batch of images can learn a more accurate similarity relationship with all negative samples in the batch of images from a global perspective; wherein each node fuses the adjacent nodes and edges through the Tranformer mechanism, thereby effectively learning global sample association information to enhance the perceived global association of the node; the mathematical expression of the node message propagation network is as follows:

[0074]

[0075] represents the representation of the i-th node after the k-th node message propagation network action, SA(V k ) i It means that the self-attention mechanism is used to act on all nodes for the kth time and then the representation of the i-th node is taken out. represents the number of sample sets of the batch image, LN represents layer normalization, and FFN represents a feed-forward network; It represents the representation of the adjacent edges between the i-th node and the j-th node after the k-th edge message propagation network is applied. Represents the intermediate variable updated by the i-th node in the k+1-th node message propagation network action;

[0076] Step 20303: an edge message propagation network acts between each pair of negative samples to integrate the global sample association information learned by the node into the edge, so as to help the association-aware channel diversity interpolation module generate the hard negative sample representation; wherein the mathematical expression of the edge message propagation network is as follows:

[0077]

[0078] represents the representation of the adjacent edges between the i-th node and the j-th node after the k-th edge message propagation network, and CA represents the cross attention mechanism; It represents the representation of the i-th node after the k+1-th node message propagation network action, Represents the intermediate variable updated by the adjacent edge between the i-th node and the j-th node in the k+1-th edge message propagation network action;

[0079] Step 20304: After two iterations of the node message propagation network and the edge message propagation network, the global association learning between samples is completed, so that each edge in the graph structure can effectively perceive the relevance of the global samples and complete the construction of the global association learning module between samples;

[0080] Step 204: constructing a correlation-aware channel diversity interpolation module, wherein the correlation-aware channel diversity interpolation module includes a correlation-aware sample pair interpolation module and an intra-class diversity interpolation module;

[0081] Step 20401, the steps of the association-aware sample pair interpolation module are to first pass the representations of all edges in the graph structure through a fully connected layer and then normalize them using a Sigmoid function, provide channel-level diversity interpolation weights for each negative sample pair, and then interpolate based on the interpolation weights, the training loss of the image representation extraction module, and the distance relationship between the positive and negative samples in the batch image to synthesize the difficult negative sample representations between the negative sample pairs, and the mathematical expression is as follows:

[0082]

[0083]

[0084] in, represents the representation of the adjacent edges between the i-th node and the j-th node after the K-th edge message propagation network. FC represents a fully connected layer. The Sigmoid function is used for numerical normalization. In a negative sample pair, the anchor sample representation is defined as z i , the negative sample representation relative to the anchor point is defined as z j , d - is the distance between the anchor sample and the negative sample, d+ is the distance between the anchor point and the positive sample, α is a hyperparameter, J avg Represents the metric loss value of the current image representation extraction module training; λ ij Represents the correlation interpolation of sample pairs, that is, in z i As anchor point samples, z j As the interpolated vector representation of negative samples, Indicates that in z i As anchor point samples, z j As negative samples, they are represented by synthetic hard negative samples, and e represents a natural constant;

[0085] Step 20402: The intra-class diversity interpolation module randomly weights all synthetic hard negative sample representations of the same negative class corresponding to each anchor sample to mine as many potential hard negative sample representations as possible. The mathematical expression is as follows:

[0086]

[0087] Indicates that the category is the nth category and the anchor sample is z i The set of all synthetic hard negative sample representations; l j represents the category of the jth class), RW represents the fusion of two difficult negative sample representations in the set by random weighting to obtain a new interpolation vector representation, and then repeats the above operation between the new interpolation vector representation and the next difficult negative sample representation until all difficult negative sample representations of the set are traversed, thereby obtaining each anchor point sample z i Representation of hard negative samples between the nth category

[0088] Step 205: training the network capable of synthesizing information-rich hard negative sample representation generation, and using the synthesized hard negative sample representation and the real sample representation to jointly train the image representation extraction module;

[0089] Step 20501, training the network capable of synthesizing information-rich difficult negative sample representation generation, including three parts of the loss function: classification loss, through a classification head network to ensure that the synthesized difficult negative sample representation accurately maintains the negative class distribution corresponding to the difficult negative sample representation, that is, to guide the network capable of synthesizing information-rich difficult negative sample representation generation to synthesize another difficult negative sample representation as close as possible to the negative class distribution corresponding to a difficult negative sample representation; similarity loss, under the premise that the classification loss maintains the category relevance of the synthesized difficult negative sample representation, that is, the corresponding negative class distribution, the difficult negative sample representation is made as similar as possible to the anchor sample representation; diversity loss, which is used to increase the standard deviation of the interpolation weight to expand the search space of the synthesized difficult negative sample representation within a reasonable constraint range; the mathematical expression of the training function composed of these three together is as follows:

[0090]

[0091] J div =1-σ(λ i. )

[0092]

[0093] Among them, C z is a classification head network, CE represents the cross entropy loss, l' n represents the label of category n, σ represents the standard deviation function, and γ s is the weight of similarity loss, γ d is the weight of diversity loss, represents the number of categories sampled in a batch of images, G represents the global association learning module between the samples; J ce is the classification loss, J sim is the similarity loss, J div is the diversity loss, λ i. Represents the anchor point sample z i and the interpolation vector representation set of all negative samples, l i represents the category of the i-th category, represents the number of sample sets of the batch of images, It means minimizing the optimization function to optimize the global association learning module G between samples, J gen A loss generated by the hard negative sample representation is optimized for the training function.

[0094] The classification head network C z It is also necessary to optimize the weights of the classification head network through real sample representation, which is mathematically expressed as follows:

[0095]

[0096] J cz Represents the classification loss of the real sample representation;

[0097] Step 20502, using the synthetic hard negative sample representation and the real sample representation to jointly optimize the image representation extraction module, including three parts of the loss function: global association learning loss, by constraining the category of the node, so that the node can still maintain the category semantic information of the node after the perception of the global association in step 203; synthetic hard negative sample representation optimization loss, combining the similarity relationship between the sample pairs of the real sample representation and the synthetic hard negative sample representation to optimize the metric space; real sample representation optimization loss, combining the similarity relationship between the sample pairs of the real sample representation to optimize the metric space; these three together constitute the optimization function, which is mathematically expressed as follows:

[0098]

[0099] represents the representation of the i-th node after K times of node message propagation network action, C v Represents another classification head, the C v and C z Although the structures are the same, the weights are different, so the two have different targets; Represents the anchor sample z in the batch image i A corresponding positive sample, m represents the number of samples sampled for each category in the batch image, represents the number of categories sampled in a batch of images, F represents the weight of the image representation extraction module, β is a hyperparameter, and J gca is the global correlation learning loss, J syn Optimizing loss for synthetic hard negative representation, J r (z) is the optimization loss for the real sample representation, J m is the loss of the optimization function, represents the inner product of the positive sample pair, represents the inner product of negative sample pairs, q represents the qth class, for That is, the number of negative samples, It means minimizing the optimization function to optimize the image representation extraction module F, the global association learning module G between samples and the other classification head C v ;

[0100] The optimization process of the optimization function is as J gen The loss is reduced by gradually introducing the synthetic hard negative sample representation into the optimization process of the image representation extraction module;

[0101] Step 20503: During the training of the network capable of synthesizing information-rich hard negative sample representation generation, overlapping training is performed on step 20501 and step 20502;

[0102] Step 206. Use the trained image representation extraction module for image retrieval; the image representation extraction module extracts and saves the database image representations from all the images in the candidate database, inputs the query image into the image representation extraction module to extract the query image representation, calculates the cosine similarity between the query image representation and the database image representation, and selects the Top-K, i.e., the first K results in descending order of similarity, as output.

[0103] In the present invention, the AdamW optimizer is used in the training process, the weight decay factor is 0.0001, the network layer learning rate of the image representation extraction module is 0.00015, the network layer learning rate of the global association learning module between samples is 0.0003, the network layer learning rate of the classification head is 0.001, and the hyperparameters α is 5, β is 2, and γ is 0.0001. s is 1, γ d The value is 0.01, the batch size is set to 81, and 50 epochs are trained on 1 RTX3090 GPU.

[0104] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0105] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0106] The technical features of the above embodiments can be combined arbitrarily. At the same time, the number of each step is not used to constrain the order of the steps. As long as there is no strict order constraint between the steps, the order is allowed to be adjusted and changed. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0107] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An image retrieval method based on hard negative sample representation synthesis, characterized in that: The following steps are involved: Step 1: construct a network capable of synthesizing information-rich hard negative sample representation generation, including an image representation extraction module, a global association learning module between samples, and a correlation-aware channel diversity interpolation module; Step 2: The batch images constructed by the class-balanced sampling strategy are used as network input, and the batch image representations are extracted by the image representation extraction module. The batch image representations are then input into the global association learning module between samples to learn the correlation between each anchor sample and all other negative samples. The sample pairs that are mutually negative are input into the association-aware channel diversity interpolation module, and the hard negative sample representations are interpolated and synthesized according to the correlation of the sample pairs. Step 3, training the network capable of synthesizing information-rich hard negative sample representation generation, and using the synthesized hard negative sample representation and the real sample representation to jointly train the image representation extraction module; Step 4: Use the trained image representation extraction module to perform image retrieval.

2. The image retrieval method based on hard negative sample representation synthesis according to claim 1, characterized in that: The image representation extraction module includes a convolutional neural network and a fully connected layer; the batch image representation extraction steps are as follows: Step 20201: After inputting the batch images into the convolutional neural network, a one-dimensional tensor is obtained by performing two-dimensional maximum pooling compression; Step 20202: The one-dimensional tensor is subjected to batch normalization, a fully connected layer, and L2 norm normalization to obtain the batch image representation; The batch image representation includes the representation of all samples in the batch image.

3. The image retrieval method based on hard negative sample representation synthesis according to claim 1, characterized in that: The inter-sample global association learning module includes a node message propagation network and an edge message propagation network. The specific steps of building the inter-sample global association learning module are as follows: Step 20301: The sample global association learning module constructs the batch image representation into a graph structure, wherein the graph structure includes a plurality of nodes and edges, each of the nodes is a representation of a single sample, and each of the edges is a dot product of the representations of the mutually negative sample pairs; Step 20302: the node message propagation network acts on all the nodes in the entire graph structure; wherein each node fuses the adjacent nodes and edges through the Tranformer mechanism, so that the node learns the global sample association information; Step 20303: the edge message propagation network acts between each pair of negative class samples to integrate the global sample association information learned by the node into the edge; Step 20304: After two iterations of the node message propagation network and the edge message propagation network, the construction of the global association learning module between samples is completed.

4. The image retrieval method based on hard negative sample representation synthesis according to claim 3, characterized in that: The association-aware channel diversity interpolation module includes an association-aware sample pair interpolation module and an intra-class diversity interpolation module. The steps for building the association-aware channel diversity interpolation module are as follows: Step 20401: The graph structure obtains the diversity interpolation weight of each negative class sample pair, the training loss of the image representation extraction module, and the distance relationship interpolation between samples in the batch image through the association-aware sample pair interpolation module, and synthesizes the difficult negative sample representation between the negative class sample pairs; Step 20402: The intra-class diversity interpolation module randomly weights all the hard negative sample representations of the same negative class corresponding to each anchor sample to mine potential hard negative sample representations.

5. The image retrieval method based on hard negative sample representation synthesis according to claim 4, characterized in that: The specific steps of step 3 are as follows: Step 20501: Use a training function to train the network capable of synthesizing information-rich difficult negative sample representation generation network; Step 20502: Optimize the image representation extraction module using an optimization function; Step 20503: During the training of the network capable of synthesizing information-rich hard negative sample representation generation, overlapping training is performed on step 20501 and step 20502.

6. The image retrieval method based on hard negative sample representation synthesis according to claim 5, characterized in that: The training function includes classification loss, similarity loss and diversity loss; The classification loss ensures that the corresponding negative class distribution is maintained between the synthesized representations of the hard negative samples through a classification head network; The similarity loss makes the representation of the hard negative sample similar to the representation of the anchor sample under the premise that the classification loss maintains the corresponding negative class distribution between the representations of the synthetic hard negative samples; The diversity loss improves the standard deviation of the interpolation weights to expand the search space for synthesizing the hard negative sample representation within a reasonable constraint.

7. The image retrieval method based on hard negative sample representation synthesis according to claim 5, characterized in that: The optimization function includes global association learning loss, synthetic hard negative sample representation optimization loss and real sample representation optimization loss; The global association learning loss enables the node to maintain category semantic information by performing category constraints on the node; The synthetic hard negative sample representation optimizes the loss, and optimizes the metric space by combining the similarity relationship between the sample pairs of the real sample representation and the hard negative sample representation; The real sample representation optimizes the loss, and the similarity relationship between sample pairs represented by the real samples is combined to optimize the metric space.

8. The image retrieval method based on hard negative sample representation synthesis according to claim 1, characterized in that: The specific steps of step 4 are as follows: The image representation extraction module extracts database image representations from all images in the database and saves them. After the query image is input into the image representation extraction module to extract the query image representation, the cosine similarity between the query image representation and the database image representation is calculated, and the top K results in descending order of similarity are selected as output.

Images