Training methods, devices, electronic equipment, and storage media for image-text matching models

Abstract

This disclosure relates to the field of computer technology and provides a training method, apparatus, electronic device, and storage medium for an image-text matching model. The method includes: inputting the current batch of training text and the current batch of training images into the image-text matching model for encoding to obtain initial feature vectors for the current batch of training text and the current batch of training images; retrieving feature vectors for historical batches of training text and historical batches of training images from a feature storage queue; obtaining a loss value based on the current batch feature vectors and historical batch feature vectors, and updating the parameters of the image-text matching model based on the loss value to obtain a trained image-text matching model; inputting the current batch of training text and the current batch of training images into the trained image-text matching model for encoding to obtain feature vectors for the current batch of training text and the current batch of training images, and updating the feature storage queue accordingly, thereby improving the performance and robustness of the image-text matching model.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a training method, apparatus, electronic device, and storage medium for a text-image matching model. Background Technology

[0002] Searching based on the similarity between images or text is a common multimodal task. The most popular approach is to use Contrastive Language–Image Pre-training (CLIP) for image-text matching. CLIP is a multimodal visual and text learning method that jointly trains image feature extraction networks and text feature extraction networks, enabling them to extract highly correlated feature vectors. In CLIP, images and text are mapped to the same feature space, and the similarity between text and images is measured by calculating the cosine similarity between feature vectors. However, this method also has some limitations. One major problem is that the source of negative samples is limited by the data in the current training batch. Due to the limitations of graphics processing unit memory (GPU), only a limited amount of data can be processed per training session, meaning the number of samples that can be used as negative samples is also limited. Furthermore, the training effect of contrastive loss typically improves with the increase in the number of negative samples, thus significantly limiting the model's performance. Summary of the Invention

[0003] In view of this, the present disclosure provides a training method, apparatus, electronic device and storage medium for a text-image matching model, in order to solve the problem that the number of negative samples is insufficient due to memory limitations during the training process of the prior art, thereby affecting the performance of the text-image matching model.

[0004] A first aspect of this disclosure provides a method for training a text-image matching model, comprising:

[0005] Get the current batch training set, which includes multiple current batch training samples. Each current batch training sample includes current batch training text and current batch training image.

[0006] Each current batch of training text is input into the text encoder of the image-text matching model for text encoding to obtain the initial feature vector of each current batch of training text. Similarly, each current batch of training image is input into the image encoder of the image-text matching model for image encoding to obtain the initial feature vector of each current batch of training image.

[0007] Obtain the feature vectors of multiple historical batches of training text and multiple historical batches of training images;

[0008] Based on the initial feature vectors of each current batch of training text, the initial feature vectors of each current batch of training images, the feature vectors of each historical batch of training text, and the feature vectors of each historical batch of training images, the loss value corresponding to the contrastive loss function is obtained.

[0009] The parameters of the original text encoder, text momentum encoder, original image encoder, and image momentum encoder of the image-text matching model are updated based on the loss value until the loss value is less than or equal to a preset threshold, thus obtaining the trained image-text matching model.

[0010] Each current batch of training text is input into a text momentum encoder for text encoding to obtain the feature vector of each current batch of training text, and each current batch of training image is input into an image momentum encoder for image encoding to obtain the feature vector of each current batch of training image.

[0011] The text feature storage queue is updated based on the feature vectors of each current batch of training texts, and the image feature storage queue is updated based on the feature vectors of each current batch of training images; wherein the feature vectors of multiple historical batches of training texts are obtained from the text feature storage queue, and the feature vectors of multiple historical batches of training images are obtained from the image feature storage queue.

[0012] A second aspect of this disclosure provides a training apparatus for a text-image matching model, comprising:

[0013] The first acquisition module is configured to acquire the current batch training set, which includes multiple current batch training samples, and each current batch training sample includes current batch training text and current batch training image.

[0014] The first encoding module is configured to input each current batch of training text into the text encoder of the image-text matching model for text encoding to obtain the initial feature vector of each current batch of training text, and to input each current batch of training image into the image encoder of the image-text matching model for image encoding to obtain the initial feature vector of each current batch of training image.

[0015] The second acquisition module is configured to acquire feature vectors of multiple historical batch training texts and feature vectors of multiple historical batch training images.

[0016] The calculation module is configured to obtain the loss value corresponding to the contrastive loss function based on the initial feature vectors of each current batch of training texts, the initial feature vectors of each current batch of training images, the feature vectors of each historical batch of training texts, and the feature vectors of each historical batch of training images.

[0017] The model update module is configured to update the parameters of the image-text matching model based on the loss value until the loss value is less than or equal to a preset threshold, thus obtaining the trained image-text matching model.

[0018] The second encoding module is configured to input each current batch of training text into a text momentum encoder for text encoding to obtain the feature vector of each current batch of training text, and to input each current batch of training image into an image momentum encoder for image encoding to obtain the feature vector of each current batch of training image.

[0019] The queue update module is configured to update the text feature storage queue based on the feature vectors of each current batch of training texts, and to update the image feature storage queue based on the feature vectors of each current batch of training images; wherein the feature vectors of multiple historical batches of training texts are obtained from the text feature storage queue, and the feature vectors of multiple historical batches of training images are obtained from the image feature storage queue.

[0020] A third aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0021] A fourth aspect of this disclosure provides a readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0022] The beneficial effects of this disclosed embodiment compared to the prior art are as follows: By acquiring the current batch training set, each current batch training text is input into the original text encoder of the image-text matching model for text encoding to obtain the initial feature vector of each current batch training text. Similarly, each current batch training image is input into the original image encoder of the image-text matching model for image encoding to obtain the initial feature vector of each current batch training image. This conversion of text and image data into feature vectors facilitates subsequent feature matching and loss calculation. Furthermore, by acquiring feature vectors from multiple historical batch training texts and multiple historical batch training images from the text feature storage queue and the image feature storage queue, the number of negative samples is increased for the current batch samples, thereby improving the effect of contrastive learning. By increasing negative samples, positive and negative samples can be better distinguished, enhancing the discriminative ability of the image-text matching model. Based on the initial feature vectors of the current batch and the feature vectors of historical batches, the loss value corresponding to the contrastive loss function is obtained. The contrastive loss measures the difference between the similarity between positive sample pairs and the similarity between negative sample pairs. By calculating the loss value, the current matching effect of the image-text matching model can be evaluated, and the encoder parameters in the image-text matching model (including the original text encoder, text momentum encoder, original image encoder, and image momentum encoder) can be updated accordingly. By continuously adjusting the parameters, the image-text matching model can better learn the correspondence between text and images, resulting in a trained image-text matching model. The text momentum encoder and image momentum encoder encode the training text and training images of the current batch. By accumulating historical information, a smoother update path is provided, generating more stable feature vectors for each training text and each training image in the current batch, which will be used to update the feature storage queue. The feature vectors of the current batch are updated in the feature storage queue, ensuring that the feature vectors in the storage queue are up-to-date, and the queue length remains unchanged. By establishing separate storage queues for image and text features, storing the image and text features extracted from previous training batches, the number of negative samples available for contrastive learning can be increased under limited GPU memory conditions, thereby improving training quality. The momentum encoder, through momentum updates using accumulated historical information, makes parameter updates smoother and generates more stable feature vectors than the original encoder. By establishing storage queues to increase the number of negative samples and smoothing the training process through a momentum network, the resulting image-text matching model can extract higher-quality feature representations. This addresses the problem of insufficient negative samples due to GPU memory limitations in existing image-text matching models, thus affecting model performance and improving the robustness of image-text matching models in practical applications. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this disclosure;

[0025] Figure 2 This is a flowchart illustrating a training method for a text-image matching model provided in an embodiment of this disclosure;

[0026] Figure 3 This is a schematic diagram of the training process of another image-text matching model provided in this embodiment of the disclosure;

[0027] Figure 4 This is a schematic diagram of the structure of a training device for a text-image matching model provided in an embodiment of this disclosure;

[0028] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0029] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of the embodiments of this disclosure. However, those skilled in the art will understand that this disclosure may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this disclosure with unnecessary detail.

[0030] The following will describe in detail, with reference to the accompanying drawings, a method and apparatus for training a text-image matching model according to an embodiment of the present disclosure.

[0031] Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this disclosure. The application scenario may include terminal devices 1, 2, and 3, server 4, and network 5.

[0032] Terminal devices 1, 2, and 3 can be hardware or software. When terminal devices 1, 2, and 3 are hardware, they can be various electronic devices with displays and supporting communication with server 4, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 1, 2, and 3 are software, they can be installed in the aforementioned electronic devices. Terminal devices 1, 2, and 3 can be implemented as multiple software programs or software modules, or as a single software program or software module; this disclosure does not limit this. Furthermore, various applications can be installed on terminal devices 1, 2, and 3, such as data processing applications, instant messaging tools, social platform software, search applications, shopping applications, etc.

[0033] Server 4 can be a server that provides various services, such as a backend server that receives requests sent by terminal devices with which it has established communication connections. This backend server can receive and analyze the requests sent by the terminal devices and generate processing results. Server 4 can be a single server, a server cluster consisting of several servers, or a cloud computing service center. This disclosure embodiment does not limit this.

[0034] It should be noted that server 4 can be either hardware or software. When server 4 is hardware, it can be various electronic devices that provide various services to terminal devices 1, 2, and 3. When server 4 is software, it can be multiple software programs or software modules that provide various services to terminal devices 1, 2, and 3, or it can be a single software program or software module that provides various services to terminal devices 1, 2, and 3. This disclosure does not limit the scope of the embodiments.

[0035] Network 5 can be a wired network using coaxial cable, twisted pair, and fiber optic connection, or it can be a wireless network that enables interconnection of various communication devices without wiring, such as Bluetooth, Near Field Communication (NFC), and Infrared. This disclosure does not limit the scope of the network.

[0036] Users can establish a communication connection with server 4 via network 5 through terminal devices 1, 2, and 3 to receive or send information, etc. Specifically, the following steps are taken: The current batch training set is obtained, which includes multiple current batch training samples, each including current batch training text and current batch training images; each current batch training text is input into the text encoder of the image-text matching model for text encoding to obtain the initial feature vector of each current batch training text; and each current batch training image is input into the image encoder of the image-text matching model for image encoding to obtain the initial feature vector of each current batch training image; feature vectors of multiple historical batch training texts and multiple historical batch training images are obtained; based on the initial feature vectors of each current batch training text, the initial feature vectors of each current batch training image, the feature vectors of each historical batch training text, and the feature vectors of each historical batch training image, the loss value corresponding to the contrastive loss function is obtained; based on... The loss value is used to update the parameters of the original text encoder, text momentum encoder, original image encoder, and image momentum encoder of the image-text matching model until the loss value is less than or equal to a preset threshold, thus obtaining the trained image-text matching model. Each current batch of training text is input into the text momentum encoder for text encoding to obtain the feature vector of each current batch of training text, and each current batch of training image is input into the image momentum encoder for image encoding to obtain the feature vector of each current batch of training image. The text feature storage queue is updated based on the feature vector of each current batch of training text, and the image feature storage queue is updated based on the feature vector of each current batch of training image. Among them, the feature vectors of multiple historical batches of training text are obtained from the text feature storage queue, and the feature vectors of multiple historical batches of training images are obtained from the image feature storage queue.

[0037] It should be noted that the specific types, quantities, and combinations of terminal devices 1, 2, and 3, server 4, and network 5 can be adjusted according to the actual needs of the application scenario, and this disclosure embodiment does not impose any restrictions on this.

[0038] Figure 2 This is a flowchart illustrating a training method for a text-image matching model provided in this embodiment. Figure 2 The training method for the image-text matching model can be derived from... Figure 1 Server 4 executes. For example... Figure 2 As shown, the training method for this image-text matching model includes:

[0039] Step 101: Obtain the current batch training set, which includes multiple current batch training samples. Each current batch training sample includes current batch training text and current batch training image.

[0040] In some embodiments, during the training of the image-text matching model, due to the limited GPU memory, only a limited number of samples can be processed at a time. Therefore, the training data is typically divided into multiple batches for processing. The current batch training set is the data currently being used to train the image-text matching model. The current batch training set includes multiple current batch training samples, each consisting of a current batch training text and a corresponding current batch training image. This pairing data can be used to learn the association between images and text. The current batch training text is the text data in the current batch training samples, used to learn text features, and the current batch training images are the image data in the current batch training samples, used to learn image features. When processing large-scale datasets during training, due to GPU memory limitations, loading all data into GPU memory at once may lead to memory overflow. By dividing the training data into multiple batches for processing, GPU computing resources can be utilized more effectively for parallel computation, thereby accelerating the training process.

[0041] Step 102: Input each current batch of training text into the text encoder of the image-text matching model for text encoding to obtain the initial feature vector of each current batch of training text, and input each current batch of training image into the image encoder of the image-text matching model for image encoding to obtain the initial feature vector of each current batch of training image.

[0042] In some embodiments, each text in the current batch (i.e., the current batch training text) is input into the text encoder of the image-text matching model. The text encoder converts each current batch training text into a digital feature vector, capturing the semantic information in the text and obtaining the initial feature vector of each current batch training text. The text encoder can be a pre-trained language model such as Bidirectional Encoder Representations from Transformers (BERT) or Generative Pre-trained Transformer (GPT). Each training image in the current batch (i.e., the current batch training image) is input into the image encoder. The image encoder converts each current batch training image into a digital feature vector, capturing the visual features in the current batch training images and obtaining the initial feature vector of each current batch training image. The image encoder can be a Residual Network (ResNet), a Vision Transformer (ViT), or similar model. The text and image encoders of the image-text matching model transform the text and image data of the current batch of training data into numerical forms that the model can process. This converts unstructured text and image data into structured vector representations, facilitating further processing by the model. This helps the image-text matching model learn the correlation between text and images in the current batch of training samples, capture key information in the text and images of the current batch of training samples, and learn how to match them during training, thus better learning the correspondence between text and images.

[0043] Step 103: Obtain feature vectors of multiple historical batch training texts and feature vectors of multiple historical batch training images.

[0044] In some embodiments, a text feature storage queue is set up to store feature vectors of historical batch training texts extracted from previous batches, and an image feature storage queue is set up to store feature vectors of historical batch training images extracted from previous batches. During training, in addition to using the data of the current batch (i.e., the current batch training set), feature vectors of multiple historical batch training texts and feature vectors of multiple historical batch training images are obtained from the feature storage queues. These feature vectors are used to perform contrastive loss calculations with the initial feature vectors of each current batch training text and each current batch training image. By using the feature vectors of multiple historical batch training texts and feature vectors of multiple historical batch training images, the number of negative samples in the current batch training can be increased. Even under conditions of limited GPU memory, more negative sample pairs can be simulated using the feature vectors of historical batch training texts and feature vectors of historical batch training images, thereby improving the effect of contrastive learning. More negative samples can improve the ability of the image-text matching model to distinguish between positive and negative samples, enabling the image-text matching model to better learn the correspondence between text and images.

[0045] Step 104: Based on the initial feature vectors of each current batch of training text, the initial feature vectors of each current batch of training images, the feature vectors of each historical batch of training text, and the feature vectors of each historical batch of training images, obtain the loss value corresponding to the contrastive loss function.

[0046] In some embodiments, for each text-image pair in the current batch (i.e., the initial feature vector of the training text in the current batch versus the initial feature vector of the training image in the current batch), they constitute a positive sample pair. Feature vectors from historical batches combined with feature vectors in the current batch can form a large number of negative sample pairs. The loss value of the contrastive loss function is calculated using both positive and negative sample pairs. During training, the similarity between positive sample pairs is maximized while the similarity between negative sample pairs is minimized. By penalizing cases where the similarity of negative sample pairs is higher than that of positive sample pairs, the image-text matching model is encouraged to learn the correct text-image matching relationship, bringing positive sample pairs closer together and pushing negative sample pairs further apart. By calculating the loss value corresponding to the contrastive loss function and updating the image-text matching model parameters accordingly, the model can learn to distinguish between matching and non-matching text-image pairs, thus improving its accuracy and robustness.

[0047] Step 105: Update the parameters of the original text encoder, text momentum encoder, original image encoder, and image momentum encoder of the image-text matching model based on the loss value until the loss value is less than or equal to a preset threshold, and obtain the trained image-text matching model.

[0048] The image-text matching model includes a raw text encoder, a raw text encoder, a raw image encoder, and a raw image encoder. The raw text encoder converts the training text into an initial feature vector, and the raw image encoder converts the training image into an initial feature vector, capturing the characteristics of the current batch of training data. The underlying structure of the text momentum encoder is the same as that of the raw text encoder, and the underlying structure of the image momentum encoder is the same as that of the raw image encoder. By accumulating historical information, the text momentum encoder and the image momentum encoder can provide more stable and robust feature representations.

[0049] In some embodiments, the gradient is calculated based on the loss value of the contrastive loss function using the backpropagation algorithm. The gradient represents the sensitivity of the loss value to the model parameters, i.e., how the parameters should be adjusted to minimize the loss value. An optimization algorithm is then used to update the parameters of each encoder. For the text momentum encoder and image momentum encoder, a momentum factor is used to smooth the parameter updates until the updated loss value is less than or equal to a preset threshold. The image-text matching model training is then complete, resulting in a trained image-text matching model. By continuously updating the parameters of the original text encoder, text momentum encoder, original image encoder, and image momentum encoder, the image-text matching model can better learn the correspondence between text and images. The introduction of the momentum encoder improves the robustness of the image-text matching model, making its performance more stable under different batches of training data distributions. The momentum update mechanism, by accumulating historical information, makes parameter updates smoother, reduces oscillations during training, helps accelerate convergence, and improves the generalization ability of the image-text matching model.

[0050] Step 106: Input each current batch of training text into the text momentum encoder for text encoding to obtain the feature vector of each current batch of training text, and input each current batch of training image into the image momentum encoder for image encoding to obtain the feature vector of each current batch of training image.

[0051] In some embodiments, each current batch of training text is input into a text momentum encoder, converting each current batch of training text into feature vectors to obtain more stable and robust feature representations, i.e., feature vectors of each current batch of training text. These feature vectors are used to update the text feature storage queue, thereby increasing the number of negative samples. Similarly, each current batch of training images is input into an image momentum encoder, converting each current batch of training images into feature vectors to obtain more stable and robust feature representations, i.e., feature vectors of each current batch of training images. These feature vectors are used to update the image feature storage queue, thereby increasing the number of negative samples. Updating the feature storage queue with feature vectors extracted by the momentum encoder ensures that the features in the feature storage queue are up-to-date while maintaining the same queue length. This facilitates the continued use of these feature vectors as negative samples in the next training batch, thus increasing the number of negative samples.

[0052] Step 107: Update the text feature storage queue based on the feature vectors of each current batch of training texts, and update the image feature storage queue based on the feature vectors of each current batch of training images; wherein the feature vectors of multiple historical batches of training texts are obtained from the text feature storage queue, and the feature vectors of multiple historical batches of training images are obtained from the image feature storage queue.

[0053] In some embodiments, the feature vector of the current batch of training text is the representation obtained by encoding the current batch of training text using a text momentum encoder, which can capture the key information of the current batch of training text; the feature vector of the current batch of training images is the representation obtained by encoding the current batch of training images using an image momentum encoder, which can capture the key information of the current batch of training images. In each training batch, the feature vectors of each current batch of training text are added to the text feature storage queue, and the oldest feature vector may be removed according to the queue length limit. Correspondingly, the feature vectors of each current batch of training images are added to the image feature storage queue, and the oldest feature vector may be removed.

[0054] The feature storage queues (text feature storage queue and image feature storage queue) store the feature vectors of historical training texts and images extracted by the momentum encoder from previous batches. These feature vectors are stored in memory, not GPU memory, thus effectively utilizing more memory space to store more feature vectors. In each training batch, the feature vectors of historical training texts and images can be retrieved from the storage queues and used together with the initial feature vectors of the current batch of training texts and images. This increases the number of negative samples and improves the contrastive learning effect. Increasing the number of negative samples enhances the image-text matching model's ability to distinguish between positive and negative samples, allowing it to better learn the correspondence between text and images.

[0055] The training method for the image-text matching model proposed in this disclosure involves obtaining the current batch training set, inputting each current batch training text into the text encoder of the image-text matching model for text encoding to obtain the initial feature vector of each current batch training text, and inputting each current batch training image into the image encoder of the image-text matching model for image encoding to obtain the initial feature vector of each current batch training image. This conversion of text and image data into feature vectors facilitates subsequent feature matching and loss calculation. Feature vectors from multiple historical batch training texts and multiple historical batch training images are obtained from the text feature storage queue and the image feature storage queue, increasing the number of negative samples in the current batch samples. This improves the contrastive learning effect; by increasing negative samples, positive and negative samples can be better distinguished, thus enhancing the discriminative ability of the image-text matching model. Based on the initial feature vectors of the current batch and the feature vectors of historical batches, the loss value corresponding to the contrastive loss function is obtained. The contrastive loss measures the difference between the similarity between positive sample pairs and the similarity between negative sample pairs. By calculating the loss value, the current matching effect of the image-text matching model can be evaluated, and the encoder parameters in the image-text matching model (including the original text encoder, text momentum encoder, original image encoder, and image momentum encoder) can be updated accordingly. By continuously adjusting the parameters, the image-text matching model can better learn the correspondence between text and images, resulting in a trained image-text matching model. The text momentum encoder and image momentum encoder encode the training text and training images of the current batch. By accumulating historical information, a smoother update path is provided, generating more stable feature vectors for each training text and each training image in the current batch, which will be used to update the feature storage queue. The feature vectors of the current batch are updated in the feature storage queue, ensuring that the feature vectors in the storage queue are up-to-date, and the queue length remains unchanged. By establishing separate storage queues for image and text features, storing the image and text features extracted from previous training batches, the number of negative samples available for contrastive learning can be increased under limited GPU memory conditions, thereby improving training quality. The momentum encoder, through momentum updates using accumulated historical information, makes parameter updates smoother and generates more stable feature vectors than the original encoder. By establishing storage queues to increase the number of negative samples and smoothing the training process through a momentum network, the resulting image-text matching model can extract higher-quality feature representations. This addresses the problem of insufficient negative samples due to GPU memory limitations in existing image-text matching models, thus affecting model performance and improving the robustness of image-text matching models in practical applications.

[0056] In some embodiments, the parameters of the original text encoder, text momentum encoder, original image encoder, and image momentum encoder of the image-text matching model are updated based on the loss value until the loss value is less than or equal to a preset threshold, thereby obtaining the trained image-text matching model, including:

[0057] Based on the loss value, update the parameters of the original text encoder and the original image encoder to obtain the updated original text encoder and the updated original image encoder.

[0058] Based on the updated original text encoder and the first preset momentum factor, determine the first momentum update function, and update the parameters of the text momentum encoder according to the first momentum update function;

[0059] Based on the updated original image encoder and the second preset momentum factor, a second momentum update function is determined, and the parameters of the image momentum encoder are updated according to the second momentum update function to obtain the trained image-text matching model.

[0060] Specifically, the backpropagation algorithm can be used to calculate the gradient of the loss value with respect to the model parameters, and the gradient can be used to update the parameters of the original text encoder and the original image encoder, so that the original text encoder and the original image encoder can better fit the current batch training set.

[0061] Based on the updated original text encoder and the first preset momentum factor, a first momentum update function is determined, and the parameters of the text momentum encoder are updated according to the first momentum update function, which can be expressed by the following formula:

[0062]

[0063] in, Here are the parameters of the text momentum encoder at time step t, where α is the first preset momentum factor and θ is the value of θ. text (t) represents the parameters of the original text encoder at time step t. These are the parameters of the text momentum encoder at time step (t-1).

[0064] Based on the updated original image encoder and the second preset momentum factor, a second momentum update function is determined, and the parameters of the image momentum encoder are updated according to the second momentum update function, which can be expressed by the following formula:

[0065]

[0066] in, Here are the parameters of the image momentum encoder at time step t, β is the second preset momentum factor, and θ is... image (t) represents the parameters of the original image encoder at time step t. These are the parameters of the image momentum encoder at time step (t-1).

[0067] By combining the current gradient with the accumulation of previous gradients (i.e., momentum) through the momentum update mechanism, it is helpful to maintain a certain stability during the parameter update process. By introducing momentum factors (first preset momentum factor and second preset momentum factor), the influence of historical gradients on the current update can be controlled. Updating the parameters of the text momentum encoder and the image momentum encoder aims to make them converge to the optimal solution faster, while reducing oscillations during the training process.

[0068] By continuously updating the parameters of the text and image encoders, the image-text matching model can gradually learn the matching relationship between paired text and images, reducing matching errors. The introduction of the momentum update function further accelerates the convergence speed of the image-text matching model and improves the stability and efficiency of training. When the loss value is less than or equal to a preset threshold, the image-text matching model is considered to have completed training, resulting in a fully trained model with sufficient generalization ability to handle unseen data. At this point, the image-text matching model can be applied to real-world scenarios to achieve accurate matching between text and images.

[0069] In some embodiments, updating the text feature storage queue based on the feature vectors of each current batch of training texts includes:

[0070] If the text feature storage queue is not full, add the feature vectors of each current batch of training texts to the text feature storage queue to obtain the updated text feature storage queue.

[0071] If the text feature storage queue is full, replace the feature vectors of the n oldest historical batches of training texts in the text feature storage queue with the feature vectors of each current batch of training texts, and obtain the updated text feature storage queue. The number of feature vectors of the current batch of training texts is n.

[0072] In some embodiments, if the text feature storage queue is not full, the feature vectors of each current batch of training texts are added to the text feature storage queue, resulting in an updated text feature storage queue. This ensures that when the text feature storage queue is not full, the feature vectors of all new training texts can be saved for subsequent use as training data (negative samples), helping to gradually increase the number of feature vectors in the text feature storage queue and provide more diverse data for subsequent training. If the text feature storage queue is full, the feature vectors of each current batch of training texts replace the feature vectors of the n oldest historical batches of training texts in the text feature storage queue, resulting in an updated text feature storage queue, where the number of feature vectors of the current batch of training texts is n. This step ensures that when the text feature storage queue is full, the feature vectors of new training texts can replace the feature vectors of the oldest training texts. This maintains the stability of the text feature storage queue's capacity while allowing the feature vectors in the text feature storage queue to be continuously updated, reflecting the latest data distribution. Simultaneously, the introduction of new feature vectors helps increase the diversity of negative samples, thereby improving the generalization ability of the image-text matching model.

[0073] In some embodiments, updating the image feature storage queue based on the feature vectors of each current batch of training images includes:

[0074] If the capacity of the image feature storage queue is not full, add the feature vectors of each current batch of training images to the image feature storage queue to obtain the updated image feature storage queue.

[0075] If the image feature storage queue is full, replace the feature vectors of the n oldest historical batches of training images in the image feature storage queue with the feature vectors of each current batch of training images to obtain the updated image feature storage queue. The number of feature vectors of the current batch of training images is n.

[0076] In some embodiments, if the image feature storage queue is not full, the feature vectors of each current batch of training images are added to the image feature storage queue, resulting in an updated image feature storage queue. This ensures that when the image feature storage queue is not full, the feature vectors of all new training images can be saved for subsequent use as training data (negative samples), helping to gradually increase the number of feature vectors in the image feature storage queue and providing more diverse data for subsequent training. If the image feature storage queue is full, the feature vectors of each current batch of training images replace the feature vectors of the n oldest historical batches of training images stored in the image feature storage queue, resulting in an updated image feature storage queue, where the number of feature vectors of the current batch of training images is n. This step ensures that when the image feature storage queue is full, the feature vectors of new training images can replace the feature vectors of the oldest training images. This maintains the stability of the image feature storage queue's capacity while allowing the feature vectors in the image feature storage queue to be continuously updated, reflecting the latest data distribution. Simultaneously, the introduction of new feature vectors helps increase the diversity of negative samples, thereby improving the generalization ability of the image-text matching model.

[0077] In some embodiments, the results of the training process of the image-text matching model are as follows: Figure 3As shown, it includes a text-image matching model 301, a text feature storage queue 306, an image feature storage queue 307, and a contrast loss calculation module 308. The text-image matching model 301 includes a text raw encoder 302, an image raw encoder 303, a text momentum encoder 304, and an image momentum encoder 305. Each current batch of training text is input into the text encoder 302 of the image-text matching model 301 for text encoding to obtain the initial feature vector of each current batch of training text. Similarly, each current batch of training images is input into the image encoder 303 of the image-text matching model 301 for image encoding to obtain the initial feature vector of each current batch of training images. Feature vectors of multiple historical batches of training text are obtained from the text feature storage queue 306, and feature vectors of multiple historical batches of training images are obtained from the image feature storage queue 307. The initial feature vectors of each current batch of training text, the initial feature vectors of each current batch of training images, the feature vectors of each historical batch of training text, and the feature vectors of each historical batch of training images are input into the contrast loss calculation module 308 for contrast loss calculation to obtain the loss value corresponding to the contrast loss function. The parameters of the text encoder 302, text momentum encoder 304, image encoder 303, and image momentum encoder 305 of the image-text matching model 301 are updated based on the loss value until the loss value is less than or equal to a preset threshold, thus obtaining the trained image-text matching model 301. Each current batch of training text is input into the text momentum encoder 304 for text encoding to obtain the feature vector of each current batch of training text, and each current batch of training image is input into the image momentum encoder 305 for image encoding to obtain the feature vector of each current batch of training image; the text feature storage queue 306 is updated according to the feature vector of each current batch of training text, and the image feature storage queue 307 is updated according to the feature vector of each current batch of training image.

[0078] In some embodiments, based on the initial feature vectors of each current batch of training text, the initial feature vectors of each current batch of training images, the feature vectors of each historical batch of training text, and the feature vectors of each historical batch of training images, the loss value corresponding to the contrastive loss function is obtained, including:

[0079] The loss value corresponding to the contrastive loss function can be obtained using the following formula:

[0080]

[0081] Where q is the initial feature vector of the current batch of training images, and k + k is the initial feature vector of the current batch of training text paired with the initial feature vector of the current batch of training images. i Let q be the initial feature vector of the i-th current batch of training text or the feature vector of the i-th historical batch of training text, where k is the initial feature vector of the current batch of training text, and q is the initial feature vector of the current batch of training text.+ The initial feature vector of the training image is paired with the initial feature vector of the current batch of training text, where qi is the initial feature vector of the i-th training image or the feature vector of the i-th historical batch of training images, and L q τ is the loss value, and τ is the temperature coefficient.

[0082] In some embodiments, the initial feature vector q of the current batch training image and the initial feature vector k of the current batch training text are... + The positive sample pairs are the initial feature vector q of the current batch of training images and the initial feature vector k of the i-th current batch of training text. i The negative sample pair consists of the initial feature vector q of the current batch of training images and the feature vector k of the i-th historical batch of training text. i This forms a negative sample pair. (Formula Part 1) Calculate the relative similarity between the initial feature vector of the current batch of training images and the initial feature vector of the current batch of training texts that are correctly matched, relative to the feature vectors of all other texts.

[0083] The initial feature vector k of the current batch of training text and the initial feature vector q of the current batch of training images mentioned above. + Positive sample pairs are formed by the initial feature vector k of the current batch of training text and the initial feature vector qi of the i-th training image, and negative sample pairs are formed by the initial feature vector k of the current batch of training text and the feature vector qi of the i-th historical batch of training image. (Formula Part 2) Calculate the relative similarity between the initial feature vector of the current batch of training text and the initial feature vector of the currently correctly matched training image in the current batch, relative to the feature vectors of all other images.

[0084] The goal of the contrastive loss function is to ensure that the similarity between correctly matched text-image pairs is significantly greater than that between any incorrectly matched pairs. By penalizing correctly matched pairs with low similarity and rewarding incorrectly matched pairs with high similarity, the image-text matching model learns better matching relationships. During training, this is achieved by minimizing the loss value L. q We continuously adjust the parameters of the image-text matching model to better match similar image and text pairs and distinguish dissimilar pairs.

[0085] In some embodiments, each current batch of training text is input into the original text encoder of the image-text matching model for text encoding to obtain the initial feature vector of each current batch of training text, including:

[0086] Perform word embedding on each current batch of training texts, and generate multiple word embedding vectors for each current batch of training texts;

[0087] Self-attention processing is performed on each word embedding vector of each current batch of training texts to obtain the initial feature vector of each current batch of training texts.

[0088] In some embodiments, word embeddings are performed on each current batch of training texts, converting each word in the current batch of training texts into a real-valued vector. This captures the meaning of the words and reflects the relationships between them, transforming the originally discrete text data into a continuous spatial representation, resulting in multiple word embedding vectors for each current batch of training texts. Self-attention processing is then applied to each word embedding vector in each current batch of training texts, calculating the correlation between each word (or word embedding vector) and other words. A weighted sum is generated based on the correlation to represent a new feature vector for each word, resulting in the initial feature vector for each current batch of training texts. During the self-attention process, long-distance dependencies in the current batch of training texts can be captured, improving the understanding of the current batch of training texts. By assigning different weights to different word embedding vectors, important information in the current batch of training texts can be captured more accurately, and more representative feature vectors can be generated. Therefore, the generated initial feature vectors of the current batch of training texts better reflect the overall semantics and structure of the current batch of training texts.

[0089] In addition, the original text encoder can be a BERT model. The current batch of training text input to the BERT model is preprocessed text, which may include word segmentation, removal of stop words, punctuation, etc.

[0090] In some embodiments, each current batch of training images is input into the image raw encoder of the image-text matching model for image encoding to obtain the initial feature vector of each current batch of training images, including:

[0091] Perform object detection on each current batch of training images to obtain the detection candidate boxes for each current batch of training images;

[0092] Convolutional processing is performed on the detection candidate boxes of each current batch of training images to obtain the convolutional feature vectors of the detection candidate boxes of each current batch of training images.

[0093] Feature extraction is performed on each current batch of training images to obtain feature vectors of multiple sub-images of each current batch of training images and category feature vectors of each current batch of training images.

[0094] Self-attention processing is performed on the convolutional feature vectors of the detection candidate boxes of each current batch of training images, the feature vectors of each sub-image of each current batch of training images, and the category feature vectors of each current batch of training images to obtain the initial feature vectors of each current batch of training images.

[0095] In some embodiments, each current batch of training images can be input into the Grounding DINO (Detectors with Noisy Instance-level Objectives) model for object detection, locating regions of interest (ROIs) within each current batch of training images, focusing on the most important parts of each current batch of training images, and obtaining detection candidate boxes for each current batch of training images. Convolutional neural networks can be used to convolve the detection candidate boxes of each current batch of training images to extract their local features, capturing details of the image content within the detection candidate boxes, thereby generating discriminative feature vectors, i.e., the convolutional feature vectors of the detection candidate boxes of each current batch of training images. In addition to convolutional processing of the detection candidate boxes, feature extraction is also performed on the entire image of each current batch of training images, extracting more features from the entire image, including features of sub-images and category features, fully utilizing various information from the entire image to obtain feature vectors of multiple sub-images and category feature vectors for each current batch of training images. The convolutional feature vectors of the detection candidate boxes of each current batch of training images, the feature vectors of each sub-image of each current batch of training images, and the category feature vectors of each current batch of training images are processed with self-attention to establish relationships between feature vectors. Different weights are assigned according to the importance of each input feature vector to capture the correlation between feature vectors and generate more representative feature vectors, namely the initial feature vectors of each current batch of training images. The initial feature vectors of the current batch of training images not only contain local and global information in the current batch of training images, but also consider the interaction between different features.

[0096] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0097] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0098] Figure 4 This is a schematic diagram of a training device for a text-image matching model provided in an embodiment of this disclosure. Figure 4 As shown, the training device for this image-text matching model includes:

[0099] The first acquisition module 401 is configured to acquire the current batch training set, which includes multiple current batch training samples, and each current batch training sample includes current batch training text and current batch training image.

[0100] The first encoding module 402 is configured to input each current batch of training text into the text encoder of the image-text matching model for text encoding to obtain the initial feature vector of each current batch of training text, and to input each current batch of training image into the image encoder of the image-text matching model for image encoding to obtain the initial feature vector of each current batch of training image.

[0101] The second acquisition module 403 is configured to acquire feature vectors of multiple historical batch training texts and feature vectors of multiple historical batch training images.

[0102] The calculation module 404 is configured to obtain the loss value corresponding to the contrastive loss function based on the initial feature vectors of each current batch of training texts, the initial feature vectors of each current batch of training images, the feature vectors of each historical batch of training texts, and the feature vectors of each historical batch of training images.

[0103] The model update module 405 is configured to update the parameters of the image-text matching model based on the loss value until the loss value is less than or equal to a preset threshold, thus obtaining the trained image-text matching model.

[0104] The second encoding module 406 is configured to input each current batch of training text into a text momentum encoder for text encoding to obtain the feature vector of each current batch of training text, and to input each current batch of training image into an image momentum encoder for image encoding to obtain the feature vector of each current batch of training image.

[0105] The queue update module 407 is configured to update the text feature storage queue based on the feature vectors of each current batch of training texts, and to update the image feature storage queue based on the feature vectors of each current batch of training images; wherein the feature vectors of multiple historical batches of training texts are obtained from the text feature storage queue, and the feature vectors of multiple historical batches of training images are obtained from the image feature storage queue.

[0106] According to the technical solution provided in this disclosure, by acquiring the current batch training set, each current batch training text is input into the text encoder of the image-text matching model for text encoding to obtain the initial feature vector of each current batch training text. Similarly, each current batch training image is input into the image encoder of the image-text matching model for image encoding to obtain the initial feature vector of each current batch training image. The text and image data are converted into feature vectors for subsequent feature matching and loss calculation. Feature vectors from multiple historical batch training texts and multiple historical batch training images are obtained from the text feature storage queue and the image feature storage queue, increasing the number of negative samples in the current batch samples. This improves the effect of contrastive learning; by increasing negative samples, positive and negative samples can be better distinguished, thus improving the discriminative ability of the image-text matching model. Based on the initial feature vectors of the current batch and the feature vectors of historical batches, the loss value corresponding to the contrastive loss function is obtained. The contrastive loss measures the difference between the similarity between positive sample pairs and the similarity between negative sample pairs. By calculating the loss value, the current matching effect of the image-text matching model can be evaluated, and the encoder parameters in the image-text matching model (including the original text encoder, text momentum encoder, original image encoder, and image momentum encoder) can be updated accordingly. By continuously adjusting the parameters, the image-text matching model can better learn the correspondence between text and images, resulting in a trained image-text matching model. The text momentum encoder and image momentum encoder encode the training text and training images of the current batch. By accumulating historical information, a smoother update path is provided, generating more stable feature vectors for each training text and each training image in the current batch, which will be used to update the feature storage queue. The feature vectors of the current batch are updated in the feature storage queue, ensuring that the feature vectors in the storage queue are up-to-date, and the queue length remains unchanged. By establishing separate storage queues for image and text features, storing the image and text features extracted from previous training batches, the number of negative samples available for contrastive learning can be increased under limited GPU memory conditions, thereby improving training quality. The momentum encoder, through momentum updates using accumulated historical information, makes parameter updates smoother and generates more stable feature vectors than the original encoder. By establishing storage queues to increase the number of negative samples and smoothing the training process through a momentum network, the resulting image-text matching model can extract higher-quality feature representations. This addresses the problem of insufficient negative samples due to GPU memory limitations in existing image-text matching models, thus affecting model performance and improving the robustness of image-text matching models in practical applications.

[0107] In some embodiments, the model update module 405 is configured to update the parameters of the original text encoder and the original image encoder according to the loss value to obtain the updated original text encoder and the updated original image encoder; determine a first momentum update function according to the updated original text encoder and a first preset momentum factor, and update the parameters of the text momentum encoder according to the first momentum update function; determine a second momentum update function according to the updated original image encoder and a second preset momentum factor, and update the parameters of the image momentum encoder according to the second momentum update function to obtain the trained image-text matching model.

[0108] In some embodiments, the queue update module 407 is configured to, if the capacity of the text feature storage queue is not full, add the feature vectors of each current batch of training texts to the text feature storage queue to obtain an updated text feature storage queue; if the capacity of the text feature storage queue is full, replace the feature vectors of the n historical batches of training texts with the feature vectors of each current batch of training texts in the text feature storage queue with the feature vectors of each current batch of training texts to obtain an updated text feature storage queue, wherein the number of feature vectors of the current batch of training texts is n.

[0109] In some embodiments, the queue update module 407 is configured to, if the capacity of the image feature storage queue is not full, add the feature vectors of each current batch of training images to the image feature storage queue to obtain an updated image feature storage queue; if the capacity of the image feature storage queue is full, replace the feature vectors of the n historical batches of training images with the feature vectors of each current batch of training images in the image feature storage queue with the feature vectors of each current batch of training images to obtain an updated image feature storage queue, wherein the number of feature vectors of the current batch of training images is n.

[0110] In some embodiments, the calculation module 404 is configured to obtain the loss value corresponding to the contrast loss function according to the following formula: Where q is the initial feature vector of the current batch of training images, and k + k is the initial feature vector of the current batch of training text paired with the initial feature vector of the current batch of training images. i Let q be the initial feature vector of the i-th current batch of training text or the feature vector of the i-th historical batch of training text, where k is the initial feature vector of the current batch of training text, and q is the initial feature vector of the current batch of training text. + The initial feature vector of the training image paired with the initial feature vector of the current batch of training text, q i Let Lq be the initial feature vector of the i-th training image or the feature vector of the i-th historical batch training image, Lq be the loss value, and τ be the temperature coefficient.

[0111] In some embodiments, the first encoding module 402 is configured to perform word embedding on each current batch of training texts and to process multiple word embedding vectors of each current batch of training texts; and to perform self-attention processing on each word embedding vector of each current batch of training texts to obtain the initial feature vector of each current batch of training texts.

[0112] In some embodiments, the first encoding module 402 is configured to perform object detection on each current batch of training images to obtain detection candidate boxes for each current batch of training images; perform convolution processing on the detection candidate boxes for each current batch of training images to obtain convolutional feature vectors of the detection candidate boxes for each current batch of training images; perform feature extraction on each current batch of training images to obtain feature vectors of multiple sub-images of each current batch of training images and category feature vectors of each current batch of training images; and perform self-attention processing on the convolutional feature vectors of the detection candidate boxes of each current batch of training images, the feature vectors of each sub-image of each current batch of training images, and the category feature vectors of each current batch of training images to obtain initial feature vectors of each current batch of training images.

[0113] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.

[0114] Figure 5 This is a schematic diagram of the electronic device 5 provided in an embodiment of this disclosure. Figure 5 As shown, the electronic device 5 of this embodiment includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, it implements the steps in the various method embodiments described above. Alternatively, when the processor 501 executes the computer program 503, it implements the functions of each module / unit in the various device embodiments described above.

[0115] Electronic device 5 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 5 may include, but is not limited to, processor 501 and memory 502. Those skilled in the art will understand that... Figure 5 This is merely an example of electronic device 5 and does not constitute a limitation on electronic device 5. It may include more or fewer components than shown, or different components.

[0116] The processor 501 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0117] The memory 502 can be an internal storage unit of the electronic device 5, such as a hard disk or RAM of the electronic device 5. The memory 502 can also be an external storage device of the electronic device 5, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 5. The memory 502 can also include both internal and external storage units of the electronic device 5. The memory 502 is used to store computer programs and other programs and data required by the electronic device.

[0118] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0119] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0120] The above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit it. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be included within the protection scope of this disclosure.

Claims

1. A training method for an image-text matching model, characterized in that, include: Obtain the current batch training set, which includes multiple current batch training samples, each of which includes current batch training text and current batch training image; Each of the current batch training texts is input into the text encoder of the image-text matching model for text encoding to obtain the initial feature vector of each current batch training text, and each of the current batch training images is input into the image encoder of the image-text matching model for image encoding to obtain the initial feature vector of each current batch training image. Obtain the feature vectors of multiple historical batches of training text and multiple historical batches of training images; Based on the initial feature vectors of each current batch of training text, the initial feature vectors of each current batch of training image, the feature vectors of each historical batch of training text, and the feature vectors of each historical batch of training image, the loss value corresponding to the contrastive loss function is obtained. The parameters of the original text encoder, text momentum encoder, original image encoder, and image momentum encoder of the image-text matching model are updated based on the loss value until the loss value is less than or equal to a preset threshold, thus obtaining the trained image-text matching model. Each of the current batch training texts is input into the text momentum encoder for text encoding to obtain the feature vector of each current batch training text, and each of the current batch training images is input into the image momentum encoder for image encoding to obtain the feature vector of each current batch training image. The text feature storage queue is updated based on the feature vectors of each current batch of training texts, and the image feature storage queue is updated based on the feature vectors of each current batch of training images; wherein the feature vectors of the plurality of historical batches of training texts are obtained from the text feature storage queue, and the feature vectors of the plurality of historical batches of training images are obtained from the image feature storage queue. The step of updating the text feature storage queue based on the feature vectors of each of the current batch of training texts includes: If the capacity of the text feature storage queue is not full, the feature vectors of each of the current batch of training texts are added to the text feature storage queue to obtain an updated text feature storage queue. If the capacity of the text feature storage queue is full, the feature vectors of each current batch of training text are used to replace the feature vectors of the n historical batches of training texts with the longest storage time in the text feature storage queue, to obtain an updated text feature storage queue, wherein the number of feature vectors of the current batch of training texts is n.

2. The method according to claim 1, characterized in that, The process of updating the parameters of the original text encoder, text momentum encoder, original image encoder, and image momentum encoder of the image-text matching model based on the loss value until the loss value is less than or equal to a preset threshold, to obtain the trained image-text matching model, includes: Based on the loss value, update the parameters of the original text encoder and the original image encoder to obtain the updated original text encoder and the updated original image encoder. Based on the updated original text encoder and the first preset momentum factor, determine the first momentum update function, and update the parameters of the text momentum encoder based on the first momentum update function; Based on the updated original image encoder and the second preset momentum factor, a second momentum update function is determined, and the parameters of the image momentum encoder are updated according to the second momentum update function to obtain the trained image-text matching model.

3. The method according to claim 1, characterized in that, The step of updating the image feature storage queue based on the feature vectors of each of the current batch training images includes: If the capacity of the image feature storage queue is not full, the feature vectors of each of the current batch training images are added to the image feature storage queue to obtain an updated image feature storage queue. If the capacity of the image feature storage queue is full, the feature vectors of each of the current batch training images are used to replace the feature vectors of the n historical batch training images with the longest storage time in the image feature storage queue to obtain an updated image feature storage queue, wherein the number of feature vectors of the current batch training images is n.

4. The method according to claim 1, characterized in that, The step of obtaining the loss value corresponding to the contrastive loss function based on the initial feature vectors of each current batch of training texts, the initial feature vectors of each current batch of training images, the feature vectors of each historical batch of training texts, and the feature vectors of each historical batch of training images includes: The loss value corresponding to the contrastive loss function is obtained according to the following formula: Among them, the The initial feature vector of the current batch of training images, The initial feature vector of the current batch of training text is paired with the initial feature vector of the current batch of training images. The initial feature vector of the i-th current batch of training text or the feature vector of the i-th historical batch of training text, wherein... The initial feature vector of the current batch of training text, the The initial feature vector of the current batch of training images is paired with the initial feature vector of the current batch of training text. L is the initial feature vector of the i-th current batch training image or the feature vector of the i-th historical batch training image. q For the loss value, the This is the temperature coefficient.

5. The method according to claim 1, characterized in that, The step of inputting each of the current batch training texts into the original text encoder of the image-text matching model for text encoding to obtain the initial feature vector of each current batch training text includes: Word embedding is performed on each of the current batch training texts, and multiple word embedding vectors of each current batch training text are generated. Self-attention processing is performed on each word embedding vector of each current batch of training texts to obtain the initial feature vector of each current batch of training texts.

6. The method according to claim 1, characterized in that, The step of inputting each of the current batch training images into the image-text matching model's original encoder for image encoding to obtain the initial feature vector of each current batch training image includes: Target detection is performed on each of the current batch training images to obtain the detection candidate boxes for each current batch training image; Convolution processing is performed on the detection candidate boxes of each current batch of training images to obtain the convolution feature vector of the detection candidate boxes of each current batch of training images. Feature extraction is performed on each of the current batch training images to obtain feature vectors of multiple sub-images of each current batch training image and category feature vectors of each current batch training image; Self-attention processing is performed on the convolutional feature vectors of the detection candidate boxes of each of the current batch training images, the feature vectors of each sub-image of each of the current batch training images, and the category feature vectors of each of the current batch training images to obtain the initial feature vectors of each of the current batch training images.

7. A training device for an image-text matching model, characterized in that, include: The first acquisition module is configured to acquire the current batch training set, which includes multiple current batch training samples, each of which includes current batch training text and current batch training image. The first encoding module is configured to input each of the current batch training texts into the text encoder of the image-text matching model for text encoding to obtain the initial feature vector of each current batch training text, and to input each of the current batch training images into the image encoder of the image-text matching model for image encoding to obtain the initial feature vector of each current batch training image. The second acquisition module is configured to acquire feature vectors of multiple historical batch training texts and feature vectors of multiple historical batch training images. The calculation module is configured to obtain the loss value corresponding to the contrast loss function based on the initial feature vectors of each current batch of training text, the initial feature vectors of each current batch of training images, the feature vectors of each historical batch of training text, and the feature vectors of each historical batch of training images. The model update module is configured to update the parameters of the text raw encoder, text momentum encoder, image raw encoder and image momentum encoder of the image-text matching model based on the loss value, until the loss value is less than or equal to a preset threshold, and the trained image-text matching model is obtained. The second encoding module is configured to input each of the current batch training texts into the text momentum encoder for text encoding to obtain the feature vectors of each current batch training text, and to input each of the current batch training images into the image momentum encoder for image encoding to obtain the feature vectors of each current batch training image. The queue update module is configured to update the text feature storage queue based on the feature vectors of each current batch of training texts, and to update the image feature storage queue based on the feature vectors of each current batch of training images; wherein the feature vectors of the plurality of historical batches of training texts are obtained from the text feature storage queue, and the feature vectors of the plurality of historical batches of training images are obtained from the image feature storage queue. The queue update module is further configured to: if the capacity of the text feature storage queue is not full, add the feature vectors of each of the current batch training texts to the text feature storage queue to obtain an updated text feature storage queue; if the capacity of the text feature storage queue is full, replace the feature vectors of the n historical batch training texts with the feature vectors of each of the current batch training texts in the text feature storage queue with the feature vectors of each of the current batch training texts to obtain an updated text feature storage queue, wherein the number of feature vectors of the current batch training texts is n.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images