Image-text matching model training method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computers, and provides an image-text matching model training method and device, electronic equipment and a storage medium. The method comprises the following steps: inputting a current batch of training texts and a current batch of training images into an image-text matching model for encoding to obtain an initial feature vector of the current batch of training texts and an initial feature vector of the current batch of training images; obtaining feature vectors of historical batch training texts and feature vectors of historical batch training images from a feature storage queue; obtaining a loss value based on the current batch of feature vectors and the historical batch of feature vectors, updating parameters of the image-text matching model based on the loss value, and obtaining a trained image-text matching model; and inputting the current batch of training texts and the current batch of training images into the trained image-text matching model for coding to obtain a feature vector of the current batch of training texts and a feature vector of the current batch of training images, updating a feature storage queue according to the feature vectors, improving the performance of the image-text matching model and enhancing the robustness of the image-text matching model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a training method, device, electronic device and storage medium for a graph-text matching model. Background Art

[0002] Searching based on the similarity between images or texts is a common multimodal task. The most popular solution is to use the Contrastive Language–Image Pre-training (CLIP) model for image-text matching. CLIP is a multimodal vision and text learning method that jointly trains image feature extraction networks and text feature extraction networks to extract highly correlated feature vectors. In CLIP, images and texts are mapped to the same feature space, and the similarity between text and images is measured by calculating the cosine similarity between feature vectors. However, this method also has some limitations. One of the main problems is that the source of negative samples is limited to the data in the current training batch. Due to the limitation of the graphics processing unit memory (GPU), only a limited amount of data can be processed each time for training, which means that the number of negative samples that can be used is also limited. The training effect of contrast loss usually becomes better as the number of negative samples increases, which greatly limits the performance of the model. Summary of the invention

[0003] In view of this, the embodiments of the present disclosure provide a training method, device, electronic device and storage medium for a graph-text matching model to solve the problem in the prior art that the number of negative samples is insufficient during the training process of the graph-text matching model due to video memory limitations, thereby affecting the performance of the graph-text matching model.

[0004] A first aspect of the embodiments of the present disclosure provides a method for training a graph-text matching model, including:

[0005] Obtaining a current batch training set, where the current batch training set includes multiple current batch training samples, each current batch training sample includes a current batch training text and a current batch training image;

[0006] Input each current batch of training texts into the text original encoder of the image-text matching model for text encoding to obtain the initial feature vector of each current batch of training texts, and input each current batch of training images into the image original encoder of the image-text matching model for image encoding to obtain the initial feature vector of each current batch of training images;

[0007] Obtain feature vectors of multiple historical batch training texts and feature vectors of multiple historical batch training images;

[0008] Based on the initial feature vectors of each current batch of training texts, the initial feature vectors of each current batch of training images, the feature vectors of each historical batch of training texts, and the feature vectors of each historical batch of training images, a loss value corresponding to the contrast loss function is obtained;

[0009] Based on the loss value, the parameters of the text original encoder, the text momentum encoder, the image original encoder, and the image momentum encoder of the image-text matching model are updated until the loss value is less than or equal to a preset threshold, thereby obtaining a trained image-text matching model;

[0010] Input each current batch of training texts into a text momentum encoder for text encoding to obtain a feature vector of each current batch of training texts, and input each current batch of training images into an image momentum encoder for image encoding to obtain a feature vector of each current batch of training images;

[0011] The text feature storage queue is updated according to the feature vectors of each current batch of training texts, and the image feature storage queue is updated according to the feature vectors of each current batch of training images; wherein the feature vectors of multiple historical batch training texts are obtained from the text feature storage queue, and the feature vectors of multiple historical batch training images are obtained from the image feature storage queue.

[0012] A second aspect of the embodiments of the present disclosure provides a training device for a graph-text matching model, including:

[0013] A first acquisition module is configured to acquire a current batch training set, where the current batch training set includes a plurality of current batch training samples, each of which includes a current batch training text and a current batch training image;

[0014] The first encoding module is configured to input each current batch of training texts into the text original encoder of the image-text matching model for text encoding to obtain the initial feature vector of each current batch of training texts, and input each current batch of training images into the image original encoder of the image-text matching model for image encoding to obtain the initial feature vector of each current batch of training images;

[0015] A second acquisition module is configured to acquire feature vectors of a plurality of historical batch training texts and feature vectors of a plurality of historical batch training images;

[0016] A calculation module is configured to obtain a loss value corresponding to the contrast loss function based on an initial feature vector of each current batch of training texts, an initial feature vector of each current batch of training images, a feature vector of each historical batch of training texts, and a feature vector of each historical batch of training images;

[0017] A model updating module is configured to update parameters of the image-text matching model based on the loss value until the loss value is less than or equal to a preset threshold, thereby obtaining a trained image-text matching model;

[0018] The second encoding module is configured to input each current batch of training texts into a text momentum encoder for text encoding to obtain a feature vector of each current batch of training texts, and input each current batch of training images into an image momentum encoder for image encoding to obtain a feature vector of each current batch of training images;

[0019] The queue update module is configured to update the text feature storage queue according to the feature vectors of each current batch of training texts, and to update the image feature storage queue according to the feature vectors of each current batch of training images; wherein the feature vectors of multiple historical batch training texts are obtained from the text feature storage queue, and the feature vectors of multiple historical batch training images are obtained from the image feature storage queue.

[0020] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0021] According to a fourth aspect of an embodiment of the present disclosure, a readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0022] Compared with the prior art, the disclosed embodiment has the following beneficial effects: by obtaining the current batch training set, inputting each current batch training text into the text original encoder of the image-text matching model for text encoding, obtaining the initial feature vector of each current batch training text, inputting each current batch training image into the image original encoder of the image-text matching model for image encoding, obtaining the initial feature vector of each current batch training image, converting the text and image data into feature vectors for subsequent feature matching and loss calculation. The feature vectors of multiple historical batch training texts and the feature vectors of multiple historical batch training images are obtained from the text feature storage queue and the image feature storage queue, and the number of negative samples is increased for the current batch samples, thereby improving the effect of contrastive learning. By increasing the negative samples, positive samples and negative samples can be better distinguished, and the distinguishing ability of the image-text matching model is improved. Based on the initial feature vector of the current batch and the feature vector of the historical batch, the loss value corresponding to the contrast loss function is obtained. The contrast loss measures the difference between the similarity between the positive sample pairs and the similarity between the negative sample pairs. By calculating the loss value, the current matching effect of the image-text matching model can be evaluated, and the encoder parameters in the image-text matching model (including the original text encoder, the text momentum encoder, the original image encoder, and the image momentum encoder) are updated accordingly. By continuously adjusting the parameters, the image-text matching model can better learn the correspondence between text and image, and obtain the trained image-text matching model. The text momentum encoder and the image momentum encoder are used to encode the training text and training image of the current batch, and a smoother update path is provided by accumulating historical information to generate more stable feature vectors of each current batch of training texts and each current batch of training images, which will be used to update the feature storage queue. The feature vectors of the current batch are updated to the feature storage queue, so that the feature vectors in the storage queue are the latest and the queue length remains unchanged. By establishing storage queues for image and text features respectively, the storage queues store image features and text features extracted from previous training batches, which can increase the number of negative samples that can be used for comparative learning under limited video memory conditions, thereby improving the quality of training. The momentum encoder accumulates historical information and updates the parameters through momentum, making it smoother, and can generate feature vectors that are more stable than the original encoder. By establishing a storage queue to increase the number of negative samples and smoothing the training process through the momentum network, the image-text matching model finally trained can extract higher-quality feature representations, solving the problem of insufficient number of negative samples caused by video memory limitations during the training process of the image-text matching model in the prior art, thereby affecting the performance of the image-text matching model, improving the performance of the image-text matching model, and enhancing the robustness of the image-text matching model in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 is a schematic diagram of an application scenario of an embodiment of the present disclosure;

[0025] Figure 2 It is a flowchart of a training method for a graph-text matching model provided by an embodiment of the present disclosure;

[0026] Figure 3 It is a structural schematic diagram of the training process of another image-text matching model provided by an embodiment of the present disclosure;

[0027] Figure 4 It is a structural schematic diagram of a training device for a picture-text matching model provided by an embodiment of the present disclosure;

[0028] Figure 5 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0030] A method and device for training a graph-text matching model according to an embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0031] Figure 1 Schematic diagram of an application scenario of an embodiment of the present disclosure. The application scenario may include terminal devices 1, 2 and 3, a server 4 and a network 5.

[0032] Terminal devices 1, 2 and 3 can be hardware or software. When terminal devices 1, 2 and 3 are hardware, they can be various electronic devices with display screens and supporting communication with server 4, including but not limited to smart phones, tablet computers, laptop portable computers and desktop computers, etc.; when terminal devices 1, 2 and 3 are software, they can be installed in the above electronic devices. Terminal devices 1, 2 and 3 can be implemented as multiple software or software modules, or as a single software or software module, and the embodiments of the present disclosure are not limited to this. Furthermore, various applications can be installed on terminal devices 1, 2 and 3, such as data processing applications, instant messaging tools, social platform software, search applications, shopping applications, etc.

[0033] The server 4 may be a server that provides various services, for example, a background server that receives a request sent by a terminal device that establishes a communication connection with the server, and the background server may receive and analyze the request sent by the terminal device, and generate a processing result. The server 4 may be a single server, or a server cluster composed of several servers, or a cloud computing service center, which is not limited in the embodiments of the present disclosure.

[0034] It should be noted that the server 4 can be hardware or software. When the server 4 is hardware, it can be various electronic devices that provide various services for the terminal devices 1, 2, and 3. When the server 4 is software, it can be multiple software or software modules that provide various services for the terminal devices 1, 2, and 3, or it can be a single software or software module that provides various services for the terminal devices 1, 2, and 3, and the embodiments of the present disclosure are not limited to this.

[0035] The network 5 can be a wired network connected by coaxial cable, twisted pair and optical fiber, or it can be a wireless network that can interconnect various communication devices without wiring, such as Bluetooth, Near Field Communication (NFC), infrared, etc., which is not limited in the embodiments of the present disclosure.

[0036] The user can establish a communication connection with the server 4 via the network 5 through the terminal devices 1, 2 and 3 to receive or send information, etc. Specifically, obtain a current batch training set, the current batch training set includes multiple current batch training samples, each current batch training sample includes a current batch training text and a current batch training image; input each current batch training text into the text original encoder of the image-text matching model for text encoding to obtain an initial feature vector of each current batch training text, and input each current batch training image into the image original encoder of the image-text matching model for image encoding to obtain an initial feature vector of each current batch training image; obtain feature vectors of multiple historical batch training texts and feature vectors of multiple historical batch training images; based on the initial feature vectors of each current batch training text, the initial feature vectors of each current batch training image, the feature vectors of each historical batch training text and the feature vectors of each historical batch training image, obtain the loss value corresponding to the contrast loss function; based on The loss value updates the parameters of the text original encoder, text momentum encoder, image original encoder and image momentum encoder of the image-text matching model until the loss value is less than or equal to a preset threshold value, thereby obtaining a trained image-text matching model; each current batch of training texts is input into the text momentum encoder for text encoding to obtain a feature vector of each current batch of training texts, and each current batch of training images is input into the image momentum encoder for image encoding to obtain a feature vector of each current batch of training images; the text feature storage queue is updated according to the feature vector of each current batch of training texts, and the image feature storage queue is updated according to the feature vector of each current batch of training images; wherein the feature vectors of multiple historical batch training texts are obtained from the text feature storage queue, and the feature vectors of multiple historical batch training images are obtained from the image feature storage queue.

[0037] It should be noted that the specific types, quantities and combinations of the terminal devices 1, 2 and 3, the server 4 and the network 5 can be adjusted according to the actual needs of the application scenario, and the embodiments of the present disclosure are not limited to this.

[0038] Figure 2 It is a flowchart of a training method for a graph-text matching model provided in an embodiment of the present disclosure. Figure 2 The training method of the image-text matching model can be Figure 1 Server 4 executes. Figure 2 As shown, the training method of the image-text matching model includes:

[0039] Step 101 : obtaining a current batch training set, where the current batch training set includes a plurality of current batch training samples, each of which includes a current batch training text and a current batch training image.

[0040] In some embodiments, in the training of the image-text matching model, due to the limited GPU video memory, only a limited number of samples can be processed at a time, and the training data is usually divided into multiple batches for processing. The current batch training set is the data of the current batch that is being used for the training of the image-text matching model. The current batch training set includes multiple current batch training samples, each of which consists of a current batch training text and a corresponding current batch training image. Such paired data can be used to learn the association between images and texts. The current batch training text is the text data in the current batch training sample, which is used to learn text features, and the current batch training image is the image data in the current batch training sample, which is used to learn image features. When processing large-scale data sets during training, due to the limitation of video memory, if all data is loaded into the video memory at one time, it may cause video memory overflow. By dividing the training data into multiple batches for processing, the GPU computing resources can be more effectively utilized for parallel computing, thereby accelerating the training process.

[0041] Step 102, input each current batch of training texts into the text original encoder of the image-text matching model for text encoding to obtain the initial feature vector of each current batch of training texts, and input each current batch of training images into the image original encoder of the image-text matching model for image encoding to obtain the initial feature vector of each current batch of training images.

[0042] In some embodiments, each text in the current batch (i.e., the current batch of training texts) is input into the text original encoder (Text Encoder) of the image-text matching model, and the text original encoder converts each current batch of training texts into a digital feature vector, captures the semantic information in the text, and obtains the initial feature vector of each current batch of training texts. The text original encoder can be a pre-trained language model such as a bidirectional encoder representation from Transformers (BERT) based on a transformer model, a generative pre-trained transformer (GPT), etc. Each training image in the current batch (i.e., the current batch of training images) is input into the image original encoder (Image Encoder), and the image original encoder converts each current batch of training images into a digital feature vector, captures the visual features in the current batch of training images, and obtains the initial feature vector of each current batch of training images. The image original encoder can be a residual neural network (Residual Network, ResNet), a visual transformer model (ViT), etc. The text raw encoder and image raw encoder of the image-text matching model are used to convert the text data and image data of the current batch training set into numerical forms that can be processed by the image-text matching model, and to convert the unstructured text and image data into structured vector representations to facilitate further processing by the model, thereby helping the subsequent image-text matching model to learn the association between the text and images in the current batch training samples, capture the key information in the current batch training text and the current batch training image text and images in the current batch training samples, and then learn how to match during the training process, and better learn the correspondence between text and images.

[0043] Step 103, obtaining feature vectors of multiple historical batch training texts and feature vectors of multiple historical batch training images.

[0044] In some embodiments, a text feature storage queue is set to store feature vectors of historical batch training texts extracted from previous batches, and an image feature storage queue is set to store feature vectors of historical batch training images extracted from previous batches. During the training process, in addition to using the data of the current batch (i.e., the current batch training set), feature vectors of multiple historical batch training texts and feature vectors of multiple historical batch training images are obtained from the feature storage queue. The feature vectors of the acquired historical batch training texts and feature vectors of multiple historical batch training images will be used to perform contrast loss calculation with the initial feature vectors of each current batch training text and the initial feature vectors of each current batch training image. Through the feature vectors of multiple historical batch training texts and feature vectors of multiple historical batch training images, the number of negative samples can be increased in the training of the current batch. Even under the condition of limited video memory, more negative sample pairs can be simulated through the feature vectors of the historical batch training texts and the feature vectors of the historical batch training images, thereby improving the effect of contrast learning. More negative samples can improve the ability of the image-text matching model to distinguish between positive samples and negative samples, so that the image-text matching model can better learn the correspondence between text and image.

[0045] Step 104, based on the initial feature vectors of each current batch of training texts, the initial feature vectors of each current batch of training images, the feature vectors of each historical batch of training texts, and the feature vectors of each historical batch of training images, a loss value corresponding to the contrast loss function is obtained.

[0046] In some embodiments, for each text-image pair in the current batch, that is, the initial feature vector of the current batch training text-the initial feature vector of the current batch training image, they constitute a positive sample pair, and the feature vectors in the historical batch are combined with the feature vectors in the current batch to form a large number of negative sample pairs. The loss value of the contrast loss function is calculated using the positive sample pairs and the negative sample pairs. During the training process, the similarity between the positive sample pairs is maximized, while the similarity between the negative sample pairs is minimized. By penalizing the situation where the similarity of the negative sample pairs is higher than that of the positive sample pairs, the image-text matching model is prompted to learn the correct text-image matching relationship, and the image-text matching model is encouraged to bring the positive sample pairs closer and push the negative sample pairs further away. By calculating the loss value corresponding to the contrast loss function and updating the image-text matching model parameters accordingly, the image-text matching model can learn to distinguish between matching and non-matching text-image pairs, which helps to improve the accuracy and robustness of the image-text matching model.

[0047] Step 105, based on the loss value, update the parameters of the text original encoder, text momentum encoder, image original encoder and image momentum encoder of the image-text matching model until the loss value is less than or equal to a preset threshold, thereby obtaining a trained image-text matching model.

[0048] The image-text matching model includes a text original encoder, a text momentum encoder, an image original encoder, and an image momentum encoder. The text original encoder can convert the training text into the initial feature vector of the training text, and the image original encoder can convert the training image into the initial feature vector of the training image, capturing the data characteristics of the current batch training set. The underlying structure of the text momentum encoder is the same as that of the text original encoder, and the underlying structure of the image momentum encoder is the same as that of the image original encoder. The text momentum encoder and the image momentum encoder can provide more stable and robust feature representations by accumulating historical information.

[0049] In some embodiments, the gradient is calculated based on the loss value of the contrast loss function through the back-propagation algorithm. The gradient can represent the sensitivity of the loss value to the model parameters, that is, how the parameters should be adjusted to minimize the loss value. The parameters of each encoder are updated using an optimization algorithm. For the text momentum encoder and the image momentum encoder, the momentum factor is used to smooth the parameter update until the updated loss value is less than or equal to the preset threshold. The training of the image-text matching model is completed, and a trained image-text matching model is obtained. By continuously updating the parameters of the text original encoder, the text momentum encoder, the image original encoder, and the image momentum encoder, the image-text matching model can better learn the correspondence between text and image. The introduction of the momentum encoder can improve the robustness of the image-text matching model, making the image-text matching model more stable under the distribution of different batches of training data. The momentum update mechanism makes the parameter update smoother by accumulating historical information, reduces oscillations during training, helps to accelerate the convergence speed, and improves the generalization ability of the image-text matching model.

[0050] Step 106, input each current batch of training texts into a text momentum encoder for text encoding to obtain a feature vector of each current batch of training texts, and input each current batch of training images into an image momentum encoder for image encoding to obtain a feature vector of each current batch of training images.

[0051] In some embodiments, each current batch of training text is input into a text momentum encoder, each current batch of training text is converted into a feature vector, and a more stable and robust feature representation is obtained, that is, a feature vector of each current batch of training text. The feature vector of each current batch of training text will be used to update the text feature storage queue, thereby increasing the number of negative samples. Each current batch of training images is input into an image momentum encoder, each current batch of training images is converted into a feature vector, and a more stable and robust feature representation is obtained, that is, a feature vector of each current batch of training images. The feature vector of each current batch of training images will be used to update the image feature storage queue, thereby increasing the number of negative samples. Updating the feature storage queue with the feature vector extracted by the momentum encoder can make the features in the feature storage queue the latest, while keeping the length of the feature storage queue unchanged, which helps to continue to use these feature vectors as negative samples in the next training batch, thereby increasing the number of negative samples.

[0052] Step 107, updating the text feature storage queue according to the feature vectors of each current batch of training texts, and updating the image feature storage queue according to the feature vectors of each current batch of training images; wherein the feature vectors of multiple historical batch training texts are obtained from the text feature storage queue, and the feature vectors of multiple historical batch training images are obtained from the image feature storage queue.

[0053] In some embodiments, the feature vector of the current batch of training texts is a representation obtained after the current batch of training texts are encoded by a text momentum encoder, which can capture the key information of the current batch of training texts; the feature vector of the current batch of training images is a representation obtained after the current batch of training images are encoded by an image momentum encoder, which can capture the key information of the current batch of training images. In each training batch, the feature vectors of each current batch of training texts are added to the text feature storage queue, and the earliest feature vectors can be removed according to the length limit of the queue. Accordingly, the feature vectors of each current batch of training images are added to the image feature storage queue, and the earliest feature vectors may be removed.

[0054] The feature storage queues (text feature storage queues and image feature storage queues) can be used to store feature vectors of historical batch training texts and feature vectors of historical batch training images extracted by momentum encoders in historical batches. The feature vectors of historical batch training texts and feature vectors of historical batch training images are stored in memory instead of video memory, so a larger memory space can be effectively used to store more feature vectors. In each training batch, feature vectors of historical batch training texts and feature vectors of historical batch training images can be obtained from the storage queue and used together with the initial feature vectors of the current batch training texts and the initial feature vectors of the current batch training images, thereby increasing the number of negative samples and improving the effect of contrastive learning. By increasing the number of negative samples, the ability of the image-text matching model to distinguish between positive and negative samples can be improved, so that the image-text matching model can better learn the correspondence between text and image.

[0055] Based on the training method of the image-text matching model proposed in the present disclosure, by obtaining the current batch training set, inputting each current batch training text into the text original encoder of the image-text matching model for text encoding, obtaining the initial feature vector of each current batch training text, inputting each current batch training image into the image original encoder of the image-text matching model for image encoding, obtaining the initial feature vector of each current batch training image, converting the text and image data into feature vectors for subsequent feature matching and loss calculation. The feature vectors of multiple historical batch training texts and the feature vectors of multiple historical batch training images are obtained from the text feature storage queue and the image feature storage queue, and the number of negative samples is increased for the current batch samples, thereby improving the effect of contrastive learning. By increasing the negative samples, positive samples and negative samples can be better distinguished, and the distinguishing ability of the image-text matching model is improved. Based on the initial feature vector of the current batch and the feature vector of the historical batch, the loss value corresponding to the contrast loss function is obtained. The contrast loss measures the difference between the similarity between the positive sample pairs and the similarity between the negative sample pairs. By calculating the loss value, the current matching effect of the image-text matching model can be evaluated, and the encoder parameters in the image-text matching model (including the original text encoder, the text momentum encoder, the original image encoder, and the image momentum encoder) are updated accordingly. By continuously adjusting the parameters, the image-text matching model can better learn the correspondence between text and image, and obtain the trained image-text matching model. The text momentum encoder and the image momentum encoder are used to encode the training text and training image of the current batch, and a smoother update path is provided by accumulating historical information to generate more stable feature vectors of each current batch of training texts and each current batch of training images, which will be used to update the feature storage queue. The feature vectors of the current batch are updated to the feature storage queue, so that the feature vectors in the storage queue are the latest and the queue length remains unchanged. By establishing storage queues for image and text features respectively, the storage queues store image features and text features extracted from previous training batches, which can increase the number of negative samples that can be used for comparative learning under limited video memory conditions, thereby improving the quality of training. The momentum encoder accumulates historical information and updates the parameters through momentum, making it smoother, and can generate feature vectors that are more stable than the original encoder. By establishing a storage queue to increase the number of negative samples and smoothing the training process through the momentum network, the image-text matching model finally trained can extract higher-quality feature representations, solving the problem of insufficient number of negative samples caused by video memory limitations during the training process of the image-text matching model in the prior art, thereby affecting the performance of the image-text matching model, improving the performance of the image-text matching model, and enhancing the robustness of the image-text matching model in practical applications.

[0056] In some embodiments, the parameters of the text original encoder, the text momentum encoder, the image original encoder, and the image momentum encoder of the image-text matching model are updated based on the loss value until the loss value is less than or equal to a preset threshold, thereby obtaining a trained image-text matching model, including:

[0057] According to the loss value, update the parameters of the text original encoder and the parameters of the image original encoder to obtain an updated text original encoder and an updated image original encoder;

[0058] Determine a first momentum update function according to the updated text original encoder and the first preset momentum factor, and update the parameters of the text momentum encoder according to the first momentum update function;

[0059] A second momentum update function is determined according to the updated image original encoder and the second preset momentum factor, and the parameters of the image momentum encoder are updated according to the second momentum update function to obtain a trained image-text matching model.

[0060] Specifically, the back-propagation algorithm can be used to calculate the gradient of the loss value to the model parameters, and the gradient can be used to update the parameters of the text original encoder and the image original encoder so that the text original encoder and the image original encoder can better fit the current batch training set.

[0061] According to the updated text original encoder and the first preset momentum factor, a first momentum update function is determined, and according to the first momentum update function, the parameters of the text momentum encoder are updated, which can be expressed by the following formula:

[0062]

[0063] in, is the parameter of the text momentum encoder at time step t, α is the first preset momentum factor, θ text (t) is the parameter of the original text encoder at time step t, are the parameters of the text momentum encoder at time step (t-1).

[0064] According to the updated image original encoder and the second preset momentum factor, a second momentum update function is determined, and according to the second momentum update function, the parameters of the image momentum encoder are updated, which can be expressed by the following formula:

[0065]

[0066] in, is the parameter of the image momentum encoder at time step t, β is the second preset momentum factor, θ image (t) is the parameter of the original image encoder at time step t, are the parameters of the image momentum encoder at time step (t-1).

[0067] Combining the current gradient with the accumulation of previous gradients (i.e., momentum) through the momentum update mechanism helps to maintain a certain stability during the parameter update process. By introducing the momentum factor (the first preset momentum factor and the second preset momentum factor), the influence of the historical gradient on the current update can be controlled. The parameters of the text momentum encoder and the image momentum encoder are updated to make them converge to the optimal solution faster while reducing oscillations during training.

[0068] By continuously updating the parameters of the text and image encoders, the image-text matching model can gradually learn the matching relationship between paired text and images and reduce matching errors. The introduction of the momentum update function further accelerates the convergence speed of the image-text matching model and improves the stability and efficiency of training. When the loss value is less than or equal to the preset threshold, the image-text matching model is considered to be trained, and a trained image-text matching model is obtained, which has sufficient generalization ability to process unseen data. At this point, the image-text matching model can be applied to actual scenarios to achieve accurate matching between text and images.

[0069] In some embodiments, updating the text feature storage queue according to the feature vectors of each current batch of training texts includes:

[0070] If the capacity of the text feature storage queue is not full, the feature vectors of each current batch of training texts are added to the text feature storage queue to obtain an updated text feature storage queue;

[0071] If the capacity of the text feature storage queue is full, the feature vectors of each current batch of training texts are used to replace the feature vectors of the n longest-stored historical batch of training texts in the text feature storage queue to obtain an updated text feature storage queue, in which the number of feature vectors of the current batch of training texts is n.

[0072] In some embodiments, if the capacity of the text feature storage queue is not full, the feature vectors of each current batch of training texts are added to the text feature storage queue to obtain an updated text feature storage queue. When the text feature storage queue is not full, the feature vectors of all new training texts can be saved so that they can be used as training data (negative samples) later, which helps to gradually increase the number of feature vectors stored in the text feature storage queue and provide more diverse data for subsequent training. If the capacity of the text feature storage queue is full, the feature vectors of each current batch of training texts replace the feature vectors of the n historical batch training texts with the longest storage time in the text feature storage queue to obtain an updated text feature storage queue, in which the number of feature vectors of the current batch of training texts is n. This step allows the feature vectors of the new training text to replace the feature vectors of the oldest training text when the text feature storage queue is full. It not only keeps the capacity of the text feature storage queue stable, but also allows the feature vectors in the text feature storage queue to be continuously updated to reflect the latest data distribution. At the same time, due to the introduction of new feature vectors, it helps to increase the diversity of negative samples, thereby improving the generalization ability of the image-text matching model.

[0073] In some embodiments, updating the image feature storage queue according to the feature vectors of each current batch of training images includes:

[0074] If the capacity of the image feature storage queue is not full, the feature vectors of each current batch of training images are added to the image feature storage queue to obtain an updated image feature storage queue;

[0075] If the capacity of the image feature storage queue is full, the feature vectors of each current batch of training images replace the feature vectors of the n longest-stored historical batch of training images in the image feature storage queue to obtain an updated image feature storage queue, in which the number of feature vectors of the current batch of training images is n.

[0076] In some embodiments, if the capacity of the image feature storage queue is not full, the feature vectors of each current batch of training images are added to the image feature storage queue to obtain an updated image feature storage queue. When the image feature storage queue is not full, the feature vectors of all new training images can be saved so that they can be used as training data (negative samples) later, which helps to gradually increase the number of feature vectors in the image feature storage queue and provide more diverse data for subsequent training. If the capacity of the image feature storage queue is full, the feature vectors of each current batch of training images replace the feature vectors of the n historical batch training images with the longest storage time in the image feature storage queue to obtain an updated image feature storage queue, in which the number of feature vectors of the current batch of training images is n. This step allows the feature vectors of the new training images to replace the feature vectors of the oldest training images when the image feature storage queue is full. It not only keeps the capacity of the image feature storage queue stable, but also allows the feature vectors in the image feature storage queue to be continuously updated to reflect the latest data distribution. At the same time, due to the introduction of new feature vectors, it helps to increase the diversity of negative samples, thereby improving the generalization ability of the image-text matching model.

[0077] In some embodiments, the results of the training process of the image-text matching model are as follows: Figure 3As shown, it includes a picture-text matching model 301, a text feature storage queue 306, an image feature storage queue 307, and a contrast loss calculation module 308. The picture-text matching model 301 includes a text original encoder 302, an image original encoder 303, a text momentum encoder 304 and an image momentum encoder 305. Input each current batch of training texts into the text original encoder 302 of the image-text matching model 301 for text encoding to obtain the initial feature vector of each current batch of training texts, and input each current batch of training images into the image original encoder 303 of the image-text matching model 301 for image encoding to obtain the initial feature vector of each current batch of training images; obtain the feature vectors of multiple historical batch training texts from the text feature storage queue 306, and obtain the feature vectors of multiple historical batch training images from the image feature storage queue 307; input the initial feature vectors of each current batch of training texts, the initial feature vectors of each current batch of training images, the feature vectors of each historical batch of training texts, and the feature vectors of each historical batch of training images into the contrast loss calculation module 308 for contrast loss calculation to obtain the loss value corresponding to the contrast loss function; update the parameters of the text original encoder 302, the text momentum encoder 304, the image original encoder 303, and the image momentum encoder 305 of the image-text matching model 301 based on the loss value until the loss value is less than or equal to the preset threshold, and obtain the trained image-text matching model 301. Input each current batch of training texts into the text momentum encoder 304 for text encoding to obtain the feature vector of each current batch of training texts, and input each current batch of training images into the image momentum encoder 305 for image encoding to obtain the feature vector of each current batch of training images; update the text feature storage queue 306 according to the feature vector of each current batch of training texts, and update the image feature storage queue 307 according to the feature vector of each current batch of training images.

[0078] In some embodiments, based on the initial feature vectors of each current batch of training texts, the initial feature vectors of each current batch of training images, the feature vectors of each historical batch of training texts, and the feature vectors of each historical batch of training images, a loss value corresponding to the contrast loss function is obtained, including:

[0079] According to the following formula, the loss value corresponding to the contrast loss function is obtained:

[0080]

[0081] Among them, q is the initial feature vector of the current batch of training images, k + is the initial feature vector of the current batch of training texts paired with the initial feature vector of the current batch of training images, k i is the initial feature vector of the i-th current batch of training text or the feature vector of the i-th historical batch of training text, k is the initial feature vector of the current batch of training text, q+ is the initial feature vector of the training image paired with the initial feature vector of the current batch of training text, qi is the initial feature vector of the i-th training image or the feature vector of the i-th historical batch of training images, L q is the loss value and τ is the temperature coefficient.

[0082] In some embodiments, the initial feature vector q of the current batch of training images and the initial feature vector k of the current batch of training texts are + Constitute a positive sample pair, the initial feature vector q of the current batch of training images and the initial feature vector k of the i-th current batch of training texts i Constitute a negative sample pair, the initial feature vector q of the current batch of training images and the feature vector k of the i-th historical batch of training texts i Constitute a negative sample pair. The first part of the formula Calculate the relative size of the similarity between the initial feature vector of the current batch of training images and the initial feature vector of the current batch of training texts that they correctly match relative to all other text feature vectors.

[0083] The initial feature vector k of the current batch of training text and the initial feature vector q of the current batch of training images are + The initial feature vector k of the current batch of training text and the initial feature vector qi of the i-th training image constitute a positive sample pair. The initial feature vector k of the current batch of training text and the feature vector qi of the i-th historical batch of training images constitute a negative sample pair. The second part of the formula Calculate the relative size of the similarity between the initial feature vector of the current batch of training texts and the initial feature vector of the current batch of training images that it correctly matches with respect to all other image feature vectors.

[0084] The goal of the contrast loss function is to make the similarity between correctly matched text-image pairs much greater than any incorrectly matched pairs. By penalizing correctly matched pairs with lower similarity and rewarding incorrectly matched pairs with higher similarity, the image-text matching model can learn better matching relationships. During the training process, the loss value L is minimized. q , continuously adjust the image-text matching model parameters to better match similar image and text pairs and distinguish dissimilar pairs.

[0085] In some embodiments, each current batch of training texts is input into the text original encoder of the image-text matching model for text encoding to obtain the initial feature vector of each current batch of training texts, including:

[0086] Perform word embedding on each current batch of training texts, and embed multiple words of each current batch of training texts into vectors;

[0087] Self-attention processing is performed on each word embedding vector of each current batch of training texts to obtain the initial feature vector of each current batch of training texts.

[0088] In some embodiments, each current batch of training texts is word-embedded, each word in the current batch of training texts is converted into a real vector, the meaning of the word is captured, and the relationship between words can also be reflected, and the originally discrete text data is converted into a continuous spatial representation, and multiple word embedding vectors of each current batch of training texts are obtained. Each word embedding vector of each current batch of training texts is self-attention processed, the correlation between each word (or word embedding vector) and other words is calculated, and a weighted sum is generated based on the correlation to represent a new feature vector of each word, and the initial feature vector of each current batch of training texts is obtained. In the process of self-attention processing, the long-distance dependency in the current batch of training texts can be captured, and the ability to understand the current batch of training texts can be improved. By giving different weights to different word embedding vectors, the important information in the current batch of training texts can be captured more accurately, and a more representative feature vector can be generated, so that the initial feature vector of the current batch of training texts generated better reflects the overall semantics and structure of the current batch of training texts.

[0089] In addition, the original text encoder can be a BERT model. The current batch training text input to the BERT model is a preprocessed text, and the preprocessing may include word segmentation, removal of stop words, punctuation marks, etc.

[0090] In some embodiments, each current batch of training images is input into the image original encoder of the image-text matching model for image encoding to obtain the initial feature vector of each current batch of training images, including:

[0091] Perform target detection on each current batch of training images to obtain detection candidate boxes for each current batch of training images;

[0092] Perform convolution processing on the detection candidate boxes of each current batch of training images to obtain the convolution feature vectors of the detection candidate boxes of each current batch of training images;

[0093] Performing feature extraction on each current batch of training images to obtain feature vectors of multiple sub-images of each current batch of training images and category feature vectors of each current batch of training images;

[0094] Self-attention processing is performed on the convolution feature vectors of the detection candidate boxes of each current batch of training images, the feature vectors of each sub-image of each current batch of training images, and the category feature vectors of each current batch of training images to obtain the initial feature vectors of each current batch of training images.

[0095] In some embodiments, each current batch of training images can be input into the Grounding DINO (Detectors with Noisy Instance-level Objectives) model for target detection, and the region of interest in each current batch of training images can be located, focusing on the most important part of each current batch of training images, and obtaining the detection candidate frame of each current batch of training images. A convolutional neural network can be used to perform convolution processing on the detection candidate frame of each current batch of training images to extract its local features, capture the details of the image content in the detection candidate frame of each current batch of training images, and thus generate a distinguishing feature vector, that is, a convolution feature vector of the detection candidate frame of each current batch of training images. In addition to performing convolution processing on the detection candidate frame, feature extraction is also performed on the entire image of each current batch of training images, and more features are extracted from the entire image, including sub-image features and category features, and various information of the entire image is fully utilized to obtain feature vectors of multiple sub-images of each current batch of training images and category feature vectors of each current batch of training images. The convolution feature vectors of the detection candidate boxes of each current batch of training images, the feature vectors of each sub-image of each current batch of training images, and the category feature vectors of each current batch of training images are processed by self-attention, and connections are established between the feature vectors. Different weights are given according to the importance of each input feature vector to capture the correlation between the feature vectors and generate more representative feature vectors, namely, the initial feature vectors of each current batch of training images. The initial feature vectors of the current batch of training images not only contain the local and global information in the current batch of training images, but also consider the interaction between different features.

[0096] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0097] The following are embodiments of the device disclosed herein, which can be used to execute the method embodiments disclosed herein. For details not disclosed in the device embodiments disclosed herein, please refer to the method embodiments disclosed herein.

[0098] Figure 4 Schematic diagram of a training device for a graph-text matching model provided by an embodiment of the present disclosure. Figure 4 As shown, the training device of the image-text matching model includes:

[0099] A first acquisition module 401 is configured to acquire a current batch training set, where the current batch training set includes a plurality of current batch training samples, each of which includes a current batch training text and a current batch training image;

[0100] The first encoding module 402 is configured to input each current batch of training texts into the text original encoder of the image-text matching model for text encoding to obtain the initial feature vector of each current batch of training texts, and input each current batch of training images into the image original encoder of the image-text matching model for image encoding to obtain the initial feature vector of each current batch of training images;

[0101] The second acquisition module 403 is configured to acquire feature vectors of a plurality of historical batch training texts and feature vectors of a plurality of historical batch training images;

[0102] A calculation module 404 is configured to obtain a loss value corresponding to the contrast loss function based on the initial feature vectors of each current batch of training texts, the initial feature vectors of each current batch of training images, the feature vectors of each historical batch of training texts, and the feature vectors of each historical batch of training images;

[0103] The model updating module 405 is configured to update the parameters of the image-text matching model based on the loss value until the loss value is less than or equal to a preset threshold, thereby obtaining a trained image-text matching model;

[0104] The second encoding module 406 is configured to input each current batch of training texts into a text momentum encoder for text encoding to obtain a feature vector of each current batch of training texts, and input each current batch of training images into an image momentum encoder for image encoding to obtain a feature vector of each current batch of training images;

[0105] The queue update module 407 is configured to update the text feature storage queue according to the feature vectors of each current batch of training texts, and to update the image feature storage queue according to the feature vectors of each current batch of training images; wherein the feature vectors of multiple historical batch training texts are obtained from the text feature storage queue, and the feature vectors of multiple historical batch training images are obtained from the image feature storage queue.

[0106] According to the technical solution provided by the embodiment of the present disclosure, by obtaining the current batch training set, inputting each current batch training text into the text original encoder of the image-text matching model for text encoding, obtaining the initial feature vector of each current batch training text, inputting each current batch training image into the image original encoder of the image-text matching model for image encoding, obtaining the initial feature vector of each current batch training image, and converting the text and image data into feature vectors for subsequent feature matching and loss calculation. The feature vectors of multiple historical batch training texts and multiple historical batch training images are obtained from the text feature storage queue and the image feature storage queue, and the number of negative samples is increased for the current batch samples, thereby improving the effect of contrastive learning. By increasing the negative samples, positive samples and negative samples can be better distinguished, and the distinguishing ability of the image-text matching model can be improved. Based on the initial feature vector of the current batch and the feature vector of the historical batch, the loss value corresponding to the contrast loss function is obtained. The contrast loss measures the difference between the similarity between the positive sample pairs and the similarity between the negative sample pairs. By calculating the loss value, the current matching effect of the image-text matching model can be evaluated, and the encoder parameters in the image-text matching model (including the original text encoder, the text momentum encoder, the original image encoder, and the image momentum encoder) are updated accordingly. By continuously adjusting the parameters, the image-text matching model can better learn the correspondence between text and image, and obtain the trained image-text matching model. The text momentum encoder and the image momentum encoder are used to encode the training text and training image of the current batch, and a smoother update path is provided by accumulating historical information to generate more stable feature vectors of each current batch of training texts and each current batch of training images, which will be used to update the feature storage queue. The feature vectors of the current batch are updated to the feature storage queue, so that the feature vectors in the storage queue are the latest and the queue length remains unchanged. By establishing storage queues for image and text features respectively, the storage queues store image features and text features extracted from previous training batches, which can increase the number of negative samples that can be used for comparative learning under limited video memory conditions, thereby improving the quality of training. The momentum encoder accumulates historical information and updates the parameters through momentum, making it smoother, and can generate feature vectors that are more stable than the original encoder. By establishing a storage queue to increase the number of negative samples and smoothing the training process through the momentum network, the image-text matching model finally trained can extract higher-quality feature representations, solving the problem of insufficient number of negative samples caused by video memory limitations during the training process of the image-text matching model in the prior art, thereby affecting the performance of the image-text matching model, improving the performance of the image-text matching model, and enhancing the robustness of the image-text matching model in practical applications.

[0107] In some embodiments, the model update module 405 is configured to update the parameters of the text original encoder and the parameters of the image original encoder according to the loss value to obtain an updated text original encoder and an updated image original encoder; determine a first momentum update function according to the updated text original encoder and a first preset momentum factor, and update the parameters of the text momentum encoder according to the first momentum update function; determine a second momentum update function according to the updated image original encoder and a second preset momentum factor, and update the parameters of the image momentum encoder according to the second momentum update function to obtain a trained image-text matching model.

[0108] In some embodiments, the queue update module 407 is configured to add the feature vectors of each current batch of training texts to the text feature storage queue if the capacity of the text feature storage queue is not full, so as to obtain an updated text feature storage queue; if the capacity of the text feature storage queue is full, replace the feature vectors of each current batch of training texts with the feature vectors of the n historical batch training texts with the longest storage time in the text feature storage queue, so as to obtain an updated text feature storage queue, in which the number of feature vectors of the current batch of training texts is n.

[0109] In some embodiments, the queue update module 407 is configured to add the feature vectors of each current batch of training images to the image feature storage queue if the capacity of the image feature storage queue is not full, so as to obtain an updated image feature storage queue; if the capacity of the image feature storage queue is full, replace the feature vectors of the n historical batch training images with the longest storage time in the image feature storage queue with the feature vectors of each current batch of training images, so as to obtain an updated image feature storage queue, in which the number of feature vectors of the current batch of training images is n.

[0110] In some embodiments, the calculation module 404 is configured to obtain the loss value corresponding to the contrast loss function according to the following formula: Among them, q is the initial feature vector of the current batch of training images, k + is the initial feature vector of the current batch of training texts paired with the initial feature vector of the current batch of training images, k i is the initial feature vector of the i-th current batch of training text or the feature vector of the i-th historical batch of training text, k is the initial feature vector of the current batch of training text, q + is the initial feature vector of the training image paired with the initial feature vector of the current batch of training text, q i is the initial feature vector of the i-th training image or the feature vector of the i-th historical batch training image, Lq is the loss value, and τ is the temperature coefficient.

[0111] In some embodiments, the first encoding module 402 is configured to perform word embedding on each current batch of training texts, and embed multiple words of each current batch of training texts into vectors; perform self-attention processing on each word embedding vector of each current batch of training texts to obtain an initial feature vector of each current batch of training texts.

[0112] In some embodiments, the first encoding module 402 is configured to perform target detection on each current batch of training images to obtain a detection candidate box for each current batch of training images; perform convolution processing on the detection candidate box of each current batch of training images to obtain a convolution feature vector of the detection candidate box of each current batch of training images; perform feature extraction on each current batch of training images to obtain feature vectors of multiple sub-images of each current batch of training images and a category feature vector of each current batch of training images; perform self-attention processing on the convolution feature vectors of the detection candidate boxes of each current batch of training images, the feature vectors of each sub-image of each current batch of training images, and the category feature vector of each current batch of training images to obtain an initial feature vector of each current batch of training images.

[0113] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure.

[0114] Figure 5 Schematic diagram of an electronic device 5 provided in an embodiment of the present disclosure. Figure 5 As shown, the electronic device 5 of this embodiment includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 501 executes the computer program 503, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0115] The electronic device 5 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 5 may include, but is not limited to, a processor 501 and a memory 502. Those skilled in the art will appreciate that Figure 5 The electronic device 5 is merely an example and does not limit the electronic device 5 , and may include more or less components than those shown in the figure, or different components.

[0116] The processor 501 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0117] The memory 502 may be an internal storage unit of the electronic device 5, for example, a hard disk or memory of the electronic device 5. The memory 502 may also be an external storage device of the electronic device 5, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 5. The memory 502 may also include both an internal storage unit of the electronic device 5 and an external storage device. The memory 502 is used to store computer programs and other programs and data required by the electronic device.

[0118] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0119] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, the present disclosure implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0120] The above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should all be included in the protection scope of the present disclosure.

Claims

1. A training method for a picture-text matching model, characterized in that: include: Acquire a current batch training set, wherein the current batch training set includes a plurality of current batch training samples, each of the current batch training samples includes a current batch training text and a current batch training image; Inputting each of the current batch of training texts into the text original encoder of the image-text matching model for text encoding to obtain an initial feature vector of each of the current batch of training texts, and inputting each of the current batch of training images into the image original encoder of the image-text matching model for image encoding to obtain an initial feature vector of each of the current batch of training images; Obtain feature vectors of multiple historical batch training texts and feature vectors of multiple historical batch training images; Based on the initial feature vectors of each current batch of training texts, the initial feature vectors of each current batch of training images, the feature vectors of each historical batch of training texts, and the feature vectors of each historical batch of training images, a loss value corresponding to the contrast loss function is obtained; Based on the loss value, the parameters of the text original encoder, the text momentum encoder, the image original encoder and the image momentum encoder of the image-text matching model are updated until the loss value is less than or equal to a preset threshold, thereby obtaining a trained image-text matching model; Inputting each of the current batch training texts into the text momentum encoder for text encoding to obtain feature vectors of each current batch training text, and inputting each of the current batch training images into the image momentum encoder for image encoding to obtain feature vectors of each current batch training image; The text feature storage queue is updated according to the feature vectors of each current batch of training texts, and the image feature storage queue is updated according to the feature vectors of each current batch of training images; wherein the feature vectors of the multiple historical batch training texts are obtained from the text feature storage queue, and the feature vectors of the multiple historical batch training images are obtained from the image feature storage queue.

2. The method according to claim 1, characterized in that The method updates the parameters of the text original encoder, the text momentum encoder, the image original encoder, and the image momentum encoder of the image-text matching model based on the loss value until the loss value is less than or equal to a preset threshold value, thereby obtaining a trained image-text matching model, including: According to the loss value, update the parameters of the text original encoder and the parameters of the image original encoder to obtain an updated text original encoder and an updated image original encoder; Determine a first momentum update function according to the updated text original encoder and the first preset momentum factor, and update the parameters of the text momentum encoder according to the first momentum update function; A second momentum update function is determined according to the updated image original encoder and the second preset momentum factor, and the parameters of the image momentum encoder are updated according to the second momentum update function to obtain a trained image-text matching model.

3. The method according to claim 1, characterized in that The updating of the text feature storage queue according to the feature vectors of each current batch of training texts comprises: If the capacity of the text feature storage queue is not full, adding the feature vectors of each of the current batch training texts to the text feature storage queue to obtain an updated text feature storage queue; If the capacity of the text feature storage queue is full, the feature vectors of each current batch of training texts are used to replace the feature vectors of the n longest-stored historical batch of training texts in the text feature storage queue to obtain an updated text feature storage queue, wherein the number of feature vectors of the current batch of training texts is n.

4. The method according to claim 1, characterized in that: The updating of the image feature storage queue according to the feature vectors of each current batch of training images comprises: If the capacity of the image feature storage queue is not full, adding the feature vectors of each of the current batch training images to the image feature storage queue to obtain an updated image feature storage queue; If the capacity of the image feature storage queue is full, the feature vectors of each current batch of training images replace the feature vectors of the n historical batch of training images with the longest storage time in the image feature storage queue to obtain an updated image feature storage queue, and the number of feature vectors of the current batch of training images is n.

5. The method according to claim 1, characterized in that The step of obtaining a loss value corresponding to a contrast loss function based on an initial feature vector of each current batch of training texts, an initial feature vector of each current batch of training images, a feature vector of each historical batch of training texts, and a feature vector of each historical batch of training images comprises: According to the following formula, the loss value corresponding to the contrast loss function is obtained: Wherein, q is the initial feature vector of the current batch of training images, and k + is the initial feature vector of the current batch of training texts paired with the initial feature vector of the current batch of training images, the k i is the initial feature vector of the i-th current batch of training text or the feature vector of the i-th historical batch of training text, k is the initial feature vector of the current batch of training text, q + is the initial feature vector of the current batch of training images paired with the initial feature vector of the current batch of training texts, qi is the initial feature vector of the ith training image or the feature vector of the ith historical batch of training images, and L q is the loss value, and τ is the temperature coefficient.

6. The method according to claim 1, characterized in that The step of inputting each of the current batch training texts into the text original encoder of the image-text matching model for text encoding to obtain the initial feature vector of each current batch training text comprises: Performing word embedding on each of the current batch training texts, and embedding multiple words of each current batch training text into vectors; Self-attention processing is performed on each word embedding vector of each current batch of training texts to obtain an initial feature vector of each current batch of training texts.

7. The method according to claim 1, characterized in that The step of inputting each of the current batch training images into the image original encoder of the image-text matching model for image encoding to obtain the initial feature vector of each of the current batch training images comprises: Performing target detection on each of the current batch of training images to obtain detection candidate boxes for each of the current batch of training images; Performing convolution processing on each of the detection candidate boxes of the current batch of training images to obtain a convolution feature vector of each of the detection candidate boxes of the current batch of training images; Performing feature extraction on each of the current batch training images to obtain feature vectors of a plurality of sub-images of each of the current batch training images and a category feature vector of each of the current batch training images; Self-attention processing is performed on the convolution feature vectors of the detection candidate boxes of each of the current batch of training images, the feature vectors of each of the sub-images of each of the current batch of training images, and the category feature vectors of each of the current batch of training images to obtain the initial feature vectors of each of the current batch of training images.

8. A training device for a picture-text matching model, characterized in that: include: A first acquisition module is configured to acquire a current batch training set, wherein the current batch training set includes a plurality of current batch training samples, each of which includes a current batch training text and a current batch training image; The first encoding module is configured to input each of the current batch training texts into the text original encoder of the image-text matching model for text encoding to obtain the initial feature vector of each current batch training text, and input each of the current batch training images into the image original encoder of the image-text matching model for image encoding to obtain the initial feature vector of each current batch training image; A second acquisition module is configured to acquire feature vectors of a plurality of historical batch training texts and feature vectors of a plurality of historical batch training images; A calculation module is configured to obtain a loss value corresponding to a contrast loss function based on an initial feature vector of each current batch of training texts, an initial feature vector of each current batch of training images, a feature vector of each historical batch of training texts, and a feature vector of each historical batch of training images; A model updating module is configured to update the parameters of the image-text matching model based on the loss value until the loss value is less than or equal to a preset threshold, thereby obtaining a trained image-text matching model; A second encoding module is configured to input each of the current batch training texts into the text momentum encoder for text encoding to obtain a feature vector of each current batch training text, and input each of the current batch training images into the image momentum encoder for image encoding to obtain a feature vector of each current batch training image; The queue update module is configured to update the text feature storage queue according to the feature vectors of each current batch of training texts, and to update the image feature storage queue according to the feature vectors of each current batch of training images; wherein the feature vectors of the multiple historical batch training texts are obtained from the text feature storage queue, and the feature vectors of the multiple historical batch training images are obtained from the image feature storage queue.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images