Training method of retrieval model, retrieval method, electronic device, medium and product

Abstract

This application discloses a training method for a retrieval model, a retrieval method, an electronic device, a medium, and a product, comprising: acquiring an original dataset, wherein the original dataset includes multiple original video-text pairs and corresponding classification labels, the classification labels being used to indicate whether the video content and text content in the original video-text pairs are consistent; performing enhancement processing on the original video-text pairs in the original dataset to obtain corresponding enhanced video-text pairs; training the retrieval model using the original video-text pairs and the corresponding enhanced video-text pairs as samples, and using the classification labels corresponding to the original video-text pairs as labels; constructing a loss function for the retrieval model based on the inter-modal similarity and intra-modal similarity calculated from the extracted features of the video-text pairs; and training the retrieval model until the loss function converges to obtain the trained retrieval model.

Description

Technical Field

[0001] This application relates to the field of retrieval technology, and in particular to a training method for a retrieval model, a retrieval method, an electronic device, a medium, and a product. Background Technology

[0002] With the rapid development of mobile internet and the widespread adoption of smartphones, video—a vivid, intuitive, information-rich, and highly effective form of information—is gradually becoming the mainstream in the mobile internet era, playing an increasingly important role in social life. Video-text cross-modal retrieval models can effectively bridge the gap between video and text data of different modalities, allowing for the retrieval of relevant text through video data or vice versa, which is of great significance for information management and digital development.

[0003] The current mainstream video-to-text retrieval models employ a two-stream structure, using different encoders to extract features from video and text. The retrieval model calculates the similarity between the video and text data based on the features extracted by the two encoders to achieve the retrieval. However, the limited information contained in the training samples during model training leads to poor retrieval performance and low accuracy of the results. Summary of the Invention

[0004] The purpose of this application is to provide a training method for a retrieval model, a retrieval method, an electronic device, a medium, and a product to solve the problem of poor retrieval performance of retrieval models used for video text retrieval.

[0005] To solve the above-mentioned technical problems, this specification is implemented as follows:

[0006] Firstly, a method for training a retrieval model is provided, including:

[0007] Obtain the raw data set, which includes multiple raw video-text pairs and corresponding classification tags. The classification tags are used to indicate whether the video content and text content in the raw video-text pairs are consistent.

[0008] Enhancement processing is performed on the original video-text pairs in the original dataset to obtain the corresponding enhanced video-text pairs;

[0009] Using original video-text pairs and corresponding enhanced video-text pairs as samples, and the classification tags corresponding to the original video-text pairs as labels, a retrieval model is trained. The retrieval model includes a video encoder, a text encoder, and a similarity module. The video encoder is used to extract video features from each input original video-text pair and corresponding enhanced video-text pair. The text encoder is used to extract text features from each input original video-text pair and corresponding enhanced video-text pair. The similarity module is used to calculate the intermodal similarity between the extracted text features and video features of each original video-text pair and corresponding enhanced video-text pair, the first intramodal similarity between the extracted text features and the text features of the corresponding enhanced video-text pair, and the second intramodal similarity between the extracted video features and the video features of the corresponding enhanced video-text pair.

[0010] The loss function of the retrieval model is constructed based on the inter-modal similarity, the first intra-modal similarity, and the second intra-modal similarity. The loss function represents the overall loss between the inter-modal similarity, the first intra-modal similarity, the second intra-modal similarity, and the corresponding label.

[0011] The parameters of the video encoder and the text encoder are adjusted based on the overall loss, and the original video-text pairs and the enhanced video-text pairs after enhancement processing in the original dataset are used as samples. The corresponding classification labels are used as labels to train the retrieval model until the loss function converges to obtain the trained retrieval model.

[0012] Optionally, constructing the loss function of the retrieval model based on the inter-modal similarity, the first intra-modal similarity, and the second intra-modal similarity includes:

[0013] Calculate the cross-entropy loss of inter-modal similarity, the cross-entropy loss of intra-modal similarity, and the cross-entropy loss of intra-modal similarity for each input original video-text pair and the corresponding enhanced video-text pair.

[0014] Based on the cross-entropy loss, the loss function of the retrieval model is constructed.

[0015] Optionally, the video encoder extracts text features from a first number of original video-text pairs and a corresponding first number of enhanced video-text pairs from the batch input, respectively, to obtain the first number of original text features and the first number of enhanced text features;

[0016] The text encoder extracts video features from a first number of original video-text pairs and a corresponding first number of enhanced video-text pairs from the batch input, to obtain the first number of original video features and the first number of enhanced video features.

[0017] Optionally, the similarity module,

[0018] Based on the first number of original text features, the first number of enhanced text features, the first number of original video features, and the first number of enhanced video features, a first intermodal similarity matrix between each original video feature and each original text feature, a second intermodal similarity matrix between each enhanced video feature and each enhanced text feature, a third intermodal similarity matrix between each enhanced video feature and each original text feature, and a fourth intermodal similarity matrix between each original video feature and each enhanced text feature, and a first intramodal similarity matrix between each original text feature and each enhanced text feature, and a second intramodal similarity matrix between each original video feature and each enhanced video feature, are calculated respectively.

[0019] Optionally, the calculation of cross-entropy loss for inter-modal similarity, cross-entropy loss for intra-modal similarity, and cross-entropy loss for intra-modal similarity of the input video-text pairs includes:

[0020] Calculate the cross-entropy loss in the i-th row and i-th column of the target similarity matrix, where the target similarity matrix is ​​an N*N matrix, i is between 1 and N, and N represents the first quantity; wherein, the target similarity matrix is ​​any one of the first inter-modal similarity matrix, the second inter-modal similarity matrix, the third inter-modal similarity matrix, the fourth inter-modal similarity matrix, the first intra-modal similarity matrix, and the second intra-modal similarity matrix, and the target similarity matrix is ​​the similarity between the first quantity of original video-text pairs and the first quantity of enhanced video-text pairs input in batches.

[0021] Optionally, the cross-entropy loss of the target similarity matrix can be calculated using the following formula:

[0022]

[0023]

[0024]

[0025]

[0026]

[0027] in, This represents the cross-entropy loss. This represents the first value of the similarity calculated between the text features in the i-th row of the target similarity matrix and the video features in the N columns of the target similarity matrix, after softmax normalization. Represents the target similarity matrix. This represents the second value of the similarity calculated between the video features in the j-th column of the target similarity matrix and the text features in the N columns of the target similarity matrix, after being normalized by softmax. Indicates based on The first cross-entropy loss of the calculated target similarity matrix, Indicates based on The second cross-entropy loss of the target similarity matrix is ​​calculated. This represents the similarity in the i-th row and N-th column of the target similarity matrix. This represents the similarity score in the Nth row and jth column of the target similarity matrix. This represents an exponential function.

[0028] Optionally, the step of constructing the loss function of the retrieval model based on the inter-modal similarity, the first intra-modal similarity, and the second intra-modal similarity further includes:

[0029] Calculate the InfoNCE loss for the i-th row and i-th column of the target similarity matrix, respectively;

[0030] The loss function for constructing the retrieval model based on the cross-entropy loss includes:

[0031] The overall loss of the target similarity matrix is ​​determined by linearly superimposing the cross-entropy loss and InfoNCE loss of the target similarity matrix;

[0032] The loss function of the retrieval model is constructed based on the overall losses of the first inter-modal similarity matrix, the second inter-modal similarity matrix, the third inter-modal similarity matrix, the fourth inter-modal similarity matrix, the first intra-modal similarity matrix, and the second intra-modal similarity matrix.

[0033] Optionally, constructing the loss function of the retrieval model based on the overall losses of the first inter-modal similarity matrix, the second inter-modal similarity matrix, the third inter-modal similarity matrix, the fourth inter-modal similarity matrix, the first intra-modal similarity matrix, and the second intra-modal similarity matrix includes:

[0034] The loss function of the retrieval model is constructed based on the average of the overall losses of the first inter-modal similarity matrix, the second inter-modal similarity matrix, the third inter-modal similarity matrix, the fourth inter-modal similarity matrix, the first intra-modal similarity matrix, and the second intra-modal similarity matrix.

[0035] Optionally, the enhancement processing based on the target original video-text pairs in the original dataset to obtain target enhanced video-text pairs includes:

[0036] By performing operations including cropping, rotating, scaling, color transformation, or Gaussian blurring on the videos in the target original video-text pairs in the original dataset, the target original video-text pairs are obtained as video-enhanced target video-text pairs;

[0037] By performing operations including synonym replacement, noise addition, or machine translation on the text of the target original video-text pair in the original dataset, a target enhanced video-text pair is obtained, wherein the target original video-text pair is any one or more original video-text pairs in the original dataset.

[0038] Secondly, a method for retrieving video text is provided, including:

[0039] The target text and multiple videos are input into a retrieval model for searching, and the similarity between the target text and each video is output. Videos related to the target text are retrieved based on the highest similarity score; or...

[0040] The target video and multiple texts are input into the retrieval model for retrieval, and the similarity between the target video and each text is output. The texts related to the target video are retrieved based on the highest similarity.

[0041] The retrieval model is trained according to the method described in the first aspect.

[0042] Thirdly, an electronic device is provided, including a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the method as described in the first or second aspect.

[0043] Fourthly, a readable storage medium is provided, characterized in that a program or instructions are stored on the readable storage medium, which, when executed by a processor, implement the steps of the method described in the first or second aspect.

[0044] Fifthly, a computer program product is provided, comprising a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform the steps of the method as described in the first or second aspect.

[0045] In this embodiment, an original dataset is obtained, comprising multiple original video-text pairs and corresponding classification tags. The classification tags indicate whether the video content and text content in the original video-text pairs are consistent. Enhancement processing is performed on the original video-text pairs in the original dataset to obtain corresponding enhanced video-text pairs. Using the original video-text pairs and their corresponding enhanced video-text pairs as samples and the classification tags corresponding to the original video-text pairs as labels, a retrieval model is trained. The retrieval model includes a video encoder, a text encoder, and a similarity module. The video encoder extracts video features from each input original video-text pair and its corresponding enhanced video-text pair. The text encoder extracts text features from each input original video-text pair and its corresponding enhanced video-text pair. The similarity module calculates the intermodal similarity between the extracted text features and video features of each original video-text pair and its corresponding enhanced video-text pair, and the intermodal similarity between the extracted text features and the corresponding enhanced video features of each original video-text pair. The retrieval model is trained by considering the first intra-modal similarity between text features of video-text pairs and the second intra-modal similarity between video features of each original video-text pair and the corresponding video features of the enhanced video-text pair. Based on these inter-modal similarities, the first intra-modal similarity, and the second intra-modal similarity, a loss function is constructed for the retrieval model. This loss function represents the overall loss between the inter-modal similarity, the first intra-modal similarity, the second intra-modal similarity, and the corresponding label. The parameters of the video encoder and the text encoder are adjusted based on this overall loss. The original video-text pairs and the enhanced video-text pairs from the original dataset are then used as samples, and the corresponding classification labels are used as labels to train the retrieval model until the loss function converges to obtain the trained retrieval model. This allows the enhanced data to be combined with the original data. By constructing a loss function based on the inter-modal and intra-modal losses of the combined video and text modal data, the information contained in the enhanced data can be fully explored, resulting in better retrieval performance for the trained retrieval model. Attached Figure Description

[0046] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0047] Figure 1 This is a flowchart illustrating the training method of the video text retrieval model according to an embodiment of this application.

[0048] Figure 2 This is an example of the intermodal similarity calculation principle of the video text retrieval model in this application embodiment.

[0049] Figure 3 This is a schematic diagram of the consistency distribution of intermodal similarity in the video text retrieval model of this application embodiment.

[0050] Figure 4 This is an example of the intramodal similarity calculation principle of the video text retrieval model in this application embodiment.

[0051] Figure 5 This is a schematic diagram of the consistency distribution of intramodal similarity of the video text retrieval model in an embodiment of this application.

[0052] Figure 6 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0053] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. The drawing numbers in this application are only used to distinguish the various steps in the solution and are not used to limit the execution order of the various steps. The specific execution order is subject to the description in the specification.

[0054] To address the problems existing in the prior art, embodiments of this application provide a method for training a video text retrieval model, such as... Figure 1 As shown, it includes the following steps 102 to 110.

[0055] Step 102: Obtain the original data set, which includes multiple original video-text pairs and corresponding classification tags. The classification tags are used to indicate whether the video content and text content in the original video-text pairs are consistent.

[0056] The original video-text pairs can be referenced. Figure 3 In this context, videos 12 and 14 are the original videos, and texts 22 and 24 are the original texts. Video 12 and text 22 form one original video-text pair, and video 14 and text 24 form another original video-text pair. The classification tags for each original video-text pair indicate whether the video content and text content in the corresponding pair are consistent. For example... Figure 3 In the original video-text pair consisting of video 12 and text 22, the content of text 22 is "Dancers perform traditional dance on stage," which is consistent with the content shown in video 12. If the original video-text pair consisting of video 12 and text 24 contains the content "A man walks in a rainy forest," then it is inconsistent with the content shown in video 12. Classification labels can be represented, for example, by 0 and 1.

[0057] Step 104: Perform enhancement processing on the original video-text pairs in the original data set to obtain the corresponding enhanced video-text pairs.

[0058] In one embodiment, optionally, the enhancement processing based on the target original video-text pairs in the original dataset to obtain target enhanced video-text pairs includes: performing operations including cropping, rotating, scaling, color transformation, or Gaussian blurring on the videos in the target original video-text pairs in the original dataset to obtain video-enhanced target enhanced video-text pairs; and performing operations including synonym replacement, noise addition, or machine translation on the text in the target original video-text pairs in the original dataset to obtain text-enhanced target enhanced video-text pairs, wherein the target original video-text pairs are any one or more original video-text pairs in the original dataset.

[0059] In this embodiment, data augmentation can be performed on all or part of the original video-text pairs in the original dataset. For the original video, the augmentation process can be any one of cropping, rotation, scaling, color transformation, and Gaussian blurring; for the original text, the augmentation process can be any one of synonym replacement, noise addition, or machine translation. By augmenting the original video or original text, augmented videos or augmented text similar to the original video or original text can be obtained, thus expanding the size of the original dataset.

[0060] Enhanced video-text pairs can be referenced. Figure 3 Videos 12' and 14' are enhanced videos, while texts 22' and 24' are enhanced texts. Videos 12' and 14' can be obtained by enhancing the color transformation of the corresponding videos 12 and 14, and texts 22' and 24' can be obtained by enhancing the corresponding texts 22 and 24 by adding noise.

[0061] Video 12' and text 22' constitute an enhanced video-text pair, corresponding to the original video-text pair that includes video 12 and text 22. Similarly, video 14' and text 24' constitute an enhanced video-text pair, corresponding to the original video-text pair that includes video 14 and text 24. The classification label for each enhanced video-text pair is the same as the classification label for the corresponding original video-text pair.

[0062] Step 106: Using the original video-text pairs and their corresponding enhanced video-text pairs as samples, and the classification tags corresponding to the original video-text pairs as labels, the retrieval model is trained. The retrieval model includes a video encoder, a text encoder, and a similarity module. The video encoder is used to extract text features from each input original video-text pair and its corresponding enhanced video-text pair. The text encoder is used to extract video features from each input original video-text pair and its corresponding enhanced video-text pair. The similarity module is used to calculate the intermodal similarity between the extracted text features and video features of each original video-text pair and its corresponding enhanced video-text pair, the first intramodal similarity between the extracted text features and the text features of the corresponding enhanced video-text pair, and the second intramodal similarity between the extracted video features and the video features of the corresponding enhanced video-text pair.

[0063] In this embodiment of the application, the original video-text pairs and the corresponding enhanced video-text pairs in the original dataset are used as samples, and the classification labels corresponding to each video-text pair are used as labels to train the retrieval model. The retrieval model is used to retrieve the similarity between the input video-text pairs and the input video-text pairs.

[0064] Specifically, the retrieval model includes a dual-stream structure of a video encoder and a text encoder, which are used to extract video features and text features of the input video-text pairs one-to-one. Here, the input video-text pairs include the original video-text pairs and the corresponding enhanced video-text pairs. The similarity module calculates the similarity between the video data and text data of the corresponding video-text pairs based on the video features and text features extracted by the video encoder and text encoder to achieve retrieval.

[0065] In this embodiment of the application, during the training of the retrieval model, an original video-text pair is input into the retrieval model along with its corresponding enhanced video-text pair. The extracted features include the original text features in the original video-text pair, the enhanced text features in the enhanced video-text pair, the original video features in the original video-text pair, and the enhanced video features in the enhanced video-text pair.

[0066] The input video-text pairs are batch inputs, including positive and negative samples. A single sample corresponds to a video-text pair from which both text and video features are extracted. Multiple samples input simultaneously in batches will yield a corresponding number of text and video features.

[0067] Optionally, the video encoder extracts text features from a first number of batch input original video-text pairs and a first number of corresponding enhanced video-text pairs to obtain the first number of original text features and the first number of enhanced text features; the text encoder extracts video features from a first number of batch input original video-text pairs and a first number of corresponding enhanced video-text pairs to obtain the first number of original video features and the first number of enhanced video features.

[0068] like Figure 2 As shown, for example, if four original video-text pairs and four corresponding enhanced video-text pairs are input in batches, then the original video features 10 extracted by the video encoder from the corresponding original videos include four features, namely... Figure 2 The enhanced video features extracted from the corresponding enhanced videos, namely V1, V2, V3, and V4, include four features: Figure 2 V1', V2', V3', and V4'. The text encoder extracts four original text features from the corresponding original text: V1', V2', V3', and V4'. Figure 2 The enhanced text features extracted from the corresponding enhanced text, T1, T2, T3, and T4, include four features: Figure 2 T1', T2', T3', T4'.

[0069] Accordingly, the similarity module, based on the first number of original text features, the first number of enhanced text features, the first number of original video features, and the first number of enhanced video features, calculates the first inter-modal similarity matrix between each original video feature and each original text feature, the second inter-modal similarity matrix between each enhanced video feature and each enhanced text feature, the third inter-modal similarity matrix between each enhanced video feature and each original text feature, and the fourth inter-modal similarity matrix between each original video feature and each enhanced text feature, and calculates the first intra-modal similarity matrix between each original text feature and each enhanced text feature, and the second intra-modal similarity matrix between each original video feature and each enhanced video feature.

[0070] The similarity module is used to calculate the intermodal similarity between the extracted text features and video features of the input video-text pairs, including the original text features and original video features, the original text features and enhanced video features, the original video features and enhanced text features, and the enhanced video features and enhanced text features. The similarity module is also used to calculate the intramodal similarity between the text features of the input original video-text pairs and the text features of the enhanced video-text pairs, as well as the intramodal similarity between the video features of the input original video-text pairs and the video features of the enhanced video-text pairs.

[0071] by Figure 2 Taking the four original video-text pairs and four enhanced video-text pairs as an example, the similarity module calculates the intermodal similarity between each pair of the extracted original text features (T1, T2, T3, T4) and the original video features (V1, V2, V3, V4). By calculating the similarity between the original text feature T1 and each of the four original video features (V1, V2, V3, V4), four similarities are obtained. Similarly, the original text features T2, T3, and T4 are calculated with each of the four original video features, resulting in a 4*4 first intermodal similarity matrix.

[0072] Similarly, the similarity module can calculate a 4x4 inter-modal similarity matrix, a 4x4 inter-modal similarity matrix, a 4x4 inter-modal similarity matrix, a 4x4 intra-modal similarity matrix, and a 4x4 intra-modal similarity matrix. The specific calculation method is the same as that for the inter-modal similarity between the original text features and the original video features, and will not be repeated here.

[0073] In practical implementation, we can first perform a softmax function normalization operation on each row and column of each similarity matrix to transform the distribution of each similarity value into a probability distribution. Then, based on the probability distribution values ​​included in the similarity matrix, we can perform cross-entropy loss between the probability distributions of the corresponding rows and columns.

[0074] Step 108: Construct a loss function for the retrieval model based on the inter-modal similarity, the first intra-modal similarity, and the second intra-modal similarity. The loss function represents the overall loss between the inter-modal similarity, the first intra-modal similarity, the second intra-modal similarity, and the corresponding label.

[0075] Optionally, constructing the loss function of the retrieval model based on the inter-modal similarity, the first intra-modal similarity, and the second intra-modal similarity includes: calculating the cross-entropy loss of the inter-modal similarity, the cross-entropy loss of the first intra-modal similarity, and the cross-entropy loss of the second intra-modal similarity for each input original video-text pair and the corresponding enhanced video-text pair; and constructing the loss function of the retrieval model based on each cross-entropy loss.

[0076] In this embodiment, based on the four inter-modal similarity matrices and two intra-modal similarity matrices of the input original video-text pairs and enhanced video-text pairs, the cross-entropy loss between the features of each similarity matrix is ​​calculated. The cross-entropy loss can be based on... Formula calculation, Indicates calculation As the truth value, calculate and Cross-entropy loss between two probability distributions. In the embodiments of this application, Let represent the probability distribution corresponding to the similarity in the i-th row of each similarity matrix. This represents the probability distribution corresponding to the similarity in the i-th column of the similarity matrix.

[0077] Optionally, the step of calculating the cross-entropy loss of inter-modal similarity, the cross-entropy loss of first intra-modal similarity, and the cross-entropy loss of second intra-modal similarity for the input video-text pairs includes: calculating the cross-entropy loss of the i-th row and the i-th column in the target similarity matrix, wherein the target similarity matrix is ​​an N*N matrix, i is between 1 and N, and N represents the first quantity; wherein the target similarity matrix is ​​any one of the first inter-modal similarity matrix, the second inter-modal similarity matrix, the third inter-modal similarity matrix, the fourth inter-modal similarity matrix, the first intra-modal similarity matrix, and the second intra-modal similarity matrix, and the target similarity matrix is ​​the similarity between the first quantity of original video-text pairs and the first quantity of enhanced video-text pairs input in batches.

[0078] Taking the above batch input of 4 original video-text pairs and 4 enhanced video-text pairs as an example, it can be seen that 6 4*4 similarity matrices of 100 are calculated accordingly. Combined with... Figure 2 For the original text features T1, T2, T3, and T4, the similarity is calculated with the four original video features V1, V2, V3, and V4 respectively, resulting in a 4x4 first-modal similarity matrix A. Similarly, 4x4 second-modal similarity matrices B, C1, and C2 can be obtained. Figure 4 The first modal similarity matrix F and the second modal similarity matrix E are shown.

[0079] according to Figure 2 The similarity matrix 100 on the right shows the method and formula. The cross-entropy loss of the first modality similarity matrix A can be calculated separately for the first row and first column, the second row and second column, the third row and third column, and the fourth row and fourth column. The cross-entropy loss of the first modality similarity matrix A can be obtained by averaging the sum of these cross-entropy losses.

[0080] Through the formula When calculating the cross-entropy loss of each similarity matrix, and The calculated cross-entropy loss will differ depending on the position of the elements. Therefore, in one embodiment, the cross-entropy loss is calculated separately. and The cross-entropy loss before and after the position swap is calculated twice, using the probability distributions of the rows and columns of the similarity matrix as the predicted probability and the true value, respectively.

[0081] Optionally, the similarity module calculates the cross-entropy loss of the target similarity matrix using the following formula:

[0082]

[0083]

[0084]

[0085]

[0086]

[0087] in, This represents the cross-entropy loss. This represents the first value of the similarity calculated between the text features in the i-th row of the target similarity matrix and the video features in the N columns of the target similarity matrix, after softmax normalization. Represents the target similarity matrix. This represents the second value of the similarity calculated between the video features in the j-th column of the target similarity matrix and the text features in the N columns of the target similarity matrix, after being normalized by softmax. Indicates based on The first cross-entropy loss of the calculated target similarity matrix, Indicates based on The second cross-entropy loss of the target similarity matrix is ​​calculated. This represents the similarity in the i-th row and N-th column of the target similarity matrix. This represents the similarity score in the Nth row and jth column of the target similarity matrix. This represents an exponential function.

[0088] As can be seen from the above formula, the cross-entropy loss of the similarity matrix... Let the i-th row and i-th column of the corresponding similarity matrix be used as... and The average cross-entropy loss before and after the position swap.

[0089] The cross-entropy loss of the six similarity matrices can be calculated using the above method. Cross-entropy loss effectively measures the difference between two probability distributions. When the similarity difference between a row and a column in the similarity matrix is ​​large, the cross-entropy loss between the two probability distributions will also be large. The gradient of the loss with respect to the retrieval model parameters is then calculated, and optimization algorithms are used to optimize the model parameters. Ultimately, the retrieval model can continuously narrow the distance between the similarity of that row and column, thus achieving better symmetry of the similarity matrix along the main diagonal. This indicates better consistency of the feature space for data from two different modalities. By calculating the symmetric consistency loss of the cross-entropy loss between the rows and columns of the similarity matrix, the discriminative power between negative samples can be introduced. Effectively constraining this discriminative power helps the retrieval model fully utilize and learn the large amount of information containing the relationships between negative samples during training, thereby improving the retrieval performance of the trained retrieval model.

[0090] refer to Figure 3 The line segment connecting the video and the text represents the distance between them. The shorter the line segment, the closer the distance and the higher the similarity. Figure 3 It presents an ideal scenario for enhancing the symmetric consistency between data modalities. Figure 3 The solid line represents the original data, including original video and original text; the dashed line represents the augmented data, including augmented video and augmented text. Taking the two original video-text pairs in the upper left corner as an example, from... Figure 3 It can be seen that the distance between video 12 and text 24 in text 22 is basically the same as the distance between video 14 and text 22 in text 24. Taking the two video-text pairs in the upper right corner as an example, from... Figure 3 It can be seen that the distance between video 12' of text 22 and text 24 is basically the same as the distance between video 14' of text 24 and text 22. Other details will not be described in detail.

[0091] refer to Figure 5 The line segments connecting videos and text represent the distance between them. The shorter the line segment, the closer the distance and the higher the similarity. Figure 5 It presents an ideal scenario for enhancing symmetric consistency within data modalities, from Figure 5It can be seen that the distance between video 12 and video 14' corresponding to video 12' is basically the same as the distance between video 14 and video 12' corresponding to video 14'; and the distance between text 22' and text 24' corresponding to text 22' is basically the same as the distance between text 24 and text 22' corresponding to text 24'.

[0092] Using cross-entropy loss to construct the loss function of the retrieval model for training can effectively help the retrieval model learn the consistency of different modalities in the feature space.

[0093] In one embodiment, constructing the loss function of the retrieval model based on the inter-modal similarity, the first intra-modal similarity, and the second intra-modal similarity further includes: calculating the InfoNCE loss of the i-th row and the i-th column of the target similarity matrix respectively; constructing the loss function of the retrieval model based on the cross-entropy loss includes: determining the overall loss of the target similarity matrix by linearly superimposing the cross-entropy loss and the InfoNCE loss of the target similarity matrix; and constructing the loss function of the retrieval model based on the overall losses of the first inter-modal similarity matrix, the second inter-modal similarity matrix, the third inter-modal similarity matrix, the fourth inter-modal similarity matrix, the first intra-modal similarity matrix, and the second intra-modal similarity matrix.

[0094] The above embodiments further incorporate InfoNCE loss on top of cross-entropy loss. InfoNCE loss is a loss function used for contrastive learning. It maximizes the similarity of similar sample pairs and minimizes the similarity of dissimilar sample pairs. In this embodiment, for the similarity module, six similarity matrices are calculated as described above, and the InfoNCE loss is calculated between the i-th row and i-th column of each similarity matrix. For example, using... Figure 2 Taking a 4x4 similarity matrix 100 as an example, we calculate the InfoNCE loss between corresponding rows and columns of similarity matrix 100 and the preset matrix 200. The preset matrix 200 is a similarity matrix that is close to the ideal one. For example, the probability distribution of similarity at the diagonal positions of the preset matrix is ​​1, and at other positions it is 0.

[0095] The InfoNCE loss of the first row of similarity matrix 100 and the first column of the preset matrix 200, the second row of similarity matrix 100 and the second column of the preset matrix 200, the third row of similarity matrix 100 and the third column of the preset matrix 200, and the fourth row of similarity matrix 100 and the fourth column of the preset matrix 200 are calculated separately. The InfoNCE loss of similarity matrix 100 is obtained by averaging these InfoNCE losses.

[0096] After obtaining the InfoNCE loss, the overall loss of the target similarity matrix is ​​determined by combining it with the cross-entropy loss of the target similarity matrix. For example, the target similarity matrix... The overall loss is:

[0097]

[0098] in, Represents cross-entropy loss, Indicates InfoNCE loss, and These are preset coefficients, and the overall loss of each similarity matrix can be represented by linearly superimposed cross-entropy loss and InfoNCE loss. The target similarity matrix here is any one of the following: inter-modal similarity matrix, inter-modal similarity matrix, inter-modal similarity matrix, inter-modal similarity matrix, inter-modal similarity matrix, intra-modal similarity matrix, and intra-modal similarity matrix.

[0099] Based on the six overall losses corresponding to the six similarity matrices, the final loss function can be constructed.

[0100] Specifically, constructing the loss function of the retrieval model based on the overall losses of the first inter-modal similarity matrix, the second inter-modal similarity matrix, the third inter-modal similarity matrix, the fourth inter-modal similarity matrix, the first intra-modal similarity matrix, and the second intra-modal similarity matrix includes: constructing the loss function of the retrieval model based on the average of the overall losses of the first inter-modal similarity matrix, the second inter-modal similarity matrix, the third inter-modal similarity matrix, the fourth inter-modal similarity matrix, the first intra-modal similarity matrix, and the second intra-modal similarity matrix.

[0101] For example, Let the similarity matrices between the first mode, second mode, third mode, and fourth modes, the similarity matrix within the first mode, and the similarity matrix within the second mode be represented one-to-one. Then, the loss function of the retrieval model is... It is expressed as follows:

[0102]

[0103] By The loss function is obtained by summing the overall losses corresponding to these six similarity matrices and averaging them. .

[0104] Step 110: Adjust the parameters of the video encoder and the text encoder based on the overall loss, and continue to use the original video-text pairs and the enhanced video-text pairs after enhancement processing in the original dataset as samples, and use the corresponding classification labels as labels to train the retrieval model until the loss function converges to obtain the trained retrieval model.

[0105] In one embodiment, the training can be stopped when the loss function converges to complete the retrieval model training, or it can be stopped when the training iterations reach a preset number.

[0106] In this embodiment, an original dataset is obtained, comprising multiple original video-text pairs and corresponding classification tags. The classification tags indicate whether the video content and text content in the original video-text pairs are consistent. Enhancement processing is performed on the original video-text pairs in the original dataset to obtain corresponding enhanced video-text pairs. Using the original video-text pairs and their corresponding enhanced video-text pairs as samples and the classification tags corresponding to the original video-text pairs as labels, a retrieval model is trained. The retrieval model includes a video encoder, a text encoder, and a similarity module. The video encoder extracts video features from each input original video-text pair and its corresponding enhanced video-text pair. The text encoder extracts text features from each input original video-text pair and its corresponding enhanced video-text pair. The similarity module calculates the intermodal similarity between the extracted text features and video features of each original video-text pair and its corresponding enhanced video-text pair, and the intermodal similarity between the extracted text features and the corresponding enhanced video features of each original video-text pair. The retrieval model is trained by considering the first intra-modal similarity between text features of video-text pairs and the second intra-modal similarity between video features of each original video-text pair and the corresponding video features of the enhanced video-text pair. Based on these inter-modal similarities, the first intra-modal similarity, and the second intra-modal similarity, a loss function is constructed for the retrieval model. This loss function represents the overall loss between the inter-modal similarity, the first intra-modal similarity, the second intra-modal similarity, and the corresponding label. The parameters of the video encoder and the text encoder are adjusted based on this overall loss. The original video-text pairs and the enhanced video-text pairs from the original dataset are then used as samples, and the corresponding classification labels are used as labels to train the retrieval model until the loss function converges to obtain the trained retrieval model. This allows the enhanced data to be combined with the original data. By constructing a loss function based on the inter-modal and intra-modal losses of the combined video and text modal data, the information contained in the enhanced data can be fully explored, resulting in better retrieval performance for the trained retrieval model.

[0107] Optionally, embodiments of this application also provide a video text retrieval method, comprising: inputting target text and multiple videos into a retrieval model for retrieval, outputting the similarity scores between the target text and each video respectively, and retrieving videos related to the target text based on the highest similarity score; or inputting target videos and multiple texts into a retrieval model for retrieval, outputting the similarity scores between the target video and each text respectively, and retrieving text related to the target video based on the highest similarity score; wherein the retrieval model is trained according to the training method of any of the above video text retrieval models.

[0108] Optionally, such as Figure 6 As shown, this application embodiment also provides an electronic device 2000, including a processor 2400 and a memory 2200. The memory 2200 stores a program or instructions that can run on the processor 2400. When the program or instructions are executed by the processor 2400, they implement the various steps of the above-mentioned training method embodiment of the video text retrieval model or the video text retrieval method embodiment, and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0109] This application also provides a readable storage medium storing a program or instructions. When executed by a processor, the program or instructions implement the various processes of any of the above-described video text retrieval model training method embodiments or video text retrieval method embodiments, achieving the same technical effect. To avoid repetition, further details are omitted here. The readable storage medium includes computer-readable storage media, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0110] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. The computer program is operable to enable a computer to implement the various processes of any of the above-described video text retrieval model training method embodiments or video text retrieval method embodiments when executed, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0111] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0112] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0113] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for training a retrieval model, characterized in that, include: Obtain the raw data set, which includes multiple raw video-text pairs and corresponding classification tags. The classification tags are used to indicate whether the video content and text content in the raw video-text pairs are consistent. Enhancement processing is performed on the original video-text pairs in the original dataset to obtain the corresponding enhanced video-text pairs; Using original video-text pairs and corresponding enhanced video-text pairs as samples, and the classification tags corresponding to the original video-text pairs as labels, a retrieval model is trained. The retrieval model includes a video encoder, a text encoder, and a similarity module. The video encoder is used to extract video features from each input original video-text pair and corresponding enhanced video-text pair. The text encoder is used to extract text features from each input original video-text pair and corresponding enhanced video-text pair. The similarity module is used to calculate the intermodal similarity between the extracted text features and video features of each original video-text pair and corresponding enhanced video-text pair, the first intramodal similarity between the extracted text features and the text features of the corresponding enhanced video-text pair, and the second intramodal similarity between the extracted video features and the video features of the corresponding enhanced video-text pair. The loss function of the retrieval model is constructed based on the inter-modal similarity, the first intra-modal similarity, and the second intra-modal similarity. The loss function represents the overall loss between the inter-modal similarity, the first intra-modal similarity, the second intra-modal similarity, and the corresponding label. The parameters of the video encoder and the text encoder are adjusted based on the overall loss, and the original video-text pairs and the enhanced video-text pairs after enhancement processing in the original dataset are used as samples. The corresponding classification labels are used as labels to train the retrieval model until the loss function converges to obtain the trained retrieval model.

2. The method according to claim 1, characterized in that, The loss function for constructing the retrieval model based on the inter-modal similarity, the first intra-modal similarity, and the second intra-modal similarity includes: Calculate the cross-entropy loss of inter-modal similarity, the cross-entropy loss of intra-modal similarity, and the cross-entropy loss of intra-modal similarity for each input original video-text pair and the corresponding enhanced video-text pair. Based on the cross-entropy loss, the loss function of the retrieval model is constructed.

3. The method according to claim 2, characterized in that, The text encoder extracts text features from a first number of original video-text pairs and a corresponding first number of enhanced video-text pairs from the batch input, respectively, to obtain the first number of original text features and the first number of enhanced text features; The video encoder extracts video features from a first number of original video-text pairs and a corresponding first number of enhanced video-text pairs from the batch input, to obtain the first number of original video features and the first number of enhanced video features.

4. The method according to claim 3, characterized in that, The similarity module, Based on the first number of original text features, the first number of enhanced text features, the first number of original video features, and the first number of enhanced video features, a first intermodal similarity matrix between each original video feature and each original text feature, a second intermodal similarity matrix between each enhanced video feature and each enhanced text feature, a third intermodal similarity matrix between each enhanced video feature and each original text feature, and a fourth intermodal similarity matrix between each original video feature and each enhanced text feature, and a first intramodal similarity matrix between each original text feature and each enhanced text feature, and a second intramodal similarity matrix between each original video feature and each enhanced video feature, are calculated respectively.

5. The method according to claim 4, characterized in that, The calculation of cross-entropy loss for inter-modal similarity, cross-entropy loss for intra-modal similarity, and cross-entropy loss for intra-modal similarity for each input original video-text pair and corresponding enhanced video-text pair includes: Calculate the cross-entropy loss in the i-th row and i-th column of the target similarity matrix, where the target similarity matrix is ​​an N*N matrix, i is between 1 and N, and N represents the first quantity; wherein, the target similarity matrix is ​​any one of the first inter-modal similarity matrix, the second inter-modal similarity matrix, the third inter-modal similarity matrix, the fourth inter-modal similarity matrix, the first intra-modal similarity matrix, and the second intra-modal similarity matrix, and the target similarity matrix is ​​the similarity between the first quantity of original video-text pairs and the first quantity of enhanced video-text pairs input in batches.

6. The method according to claim 5, characterized in that, The cross-entropy loss of the target similarity matrix is ​​calculated using the following formula: in, This represents the cross-entropy loss. This represents the first value of the similarity calculated between the text features in the i-th row of the target similarity matrix and the video features in the N columns of the target similarity matrix, after softmax normalization. Represents the target similarity matrix. This represents the second value of the similarity calculated between the video features in the j-th column of the target similarity matrix and the text features in the N columns of the target similarity matrix, after being normalized by softmax. Indicates based on The first cross-entropy loss of the calculated target similarity matrix, Indicates based on The second cross-entropy loss of the target similarity matrix is ​​calculated. This represents the similarity in the i-th row and N-th column of the target similarity matrix. This represents the similarity score in the Nth row and jth column of the target similarity matrix. This represents an exponential function.

7. The method according to claim 5 or 6, characterized in that, The loss function for constructing the retrieval model based on the inter-modal similarity, the first intra-modal similarity, and the second intra-modal similarity further includes: Calculate the InfoNCE loss for the i-th row and i-th column of the target similarity matrix, respectively; The loss function for constructing the retrieval model based on each cross-entropy loss includes: The overall loss of the target similarity matrix is ​​determined by linearly superimposing the cross-entropy loss and InfoNCE loss of the target similarity matrix; The loss function of the retrieval model is constructed based on the overall losses of the first inter-modal similarity matrix, the second inter-modal similarity matrix, the third inter-modal similarity matrix, the fourth inter-modal similarity matrix, the first intra-modal similarity matrix, and the second intra-modal similarity matrix.

8. The method according to claim 7, characterized in that, The loss function of the retrieval model is constructed based on the overall losses of the first inter-modal similarity matrix, the second inter-modal similarity matrix, the third inter-modal similarity matrix, the fourth inter-modal similarity matrix, the first intra-modal similarity matrix, and the second intra-modal similarity matrix, including: The loss function of the retrieval model is constructed based on the average of the overall losses of the first inter-modal similarity matrix, the second inter-modal similarity matrix, the third inter-modal similarity matrix, the fourth inter-modal similarity matrix, the first intra-modal similarity matrix, and the second intra-modal similarity matrix.

9. The method according to claim 1, characterized in that, The enhancement processing based on the original video-text pairs in the original dataset to obtain corresponding enhanced video-text pairs includes: By performing operations including cropping, rotating, scaling, color transformation, or Gaussian blurring on the videos in the target original video-text pairs in the original dataset, the target original video-text pairs are obtained as video-enhanced target video-text pairs; By performing operations including synonym replacement, noise addition, or machine translation on the text of the target original video-text pair in the original dataset, a target enhanced video-text pair is obtained, wherein the target original video-text pair is any one or more original video-text pairs in the original dataset.

10. A method for retrieving video text, characterized in that, include: The target text and multiple videos are input into the retrieval model for retrieval, and the similarity between the target text and each video is output. Based on the highest similarity, the video related to the target text is retrieved. or The target video and multiple texts are input into the retrieval model for retrieval, and the similarity between the target video and each text is output. The texts related to the target video are retrieved based on the highest similarity. The retrieval model is trained using the method described in any one of claims 1-9.

11. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the method as described in any one of claims 1-9 or claim 10.

12. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-9 or claim 10.

13. A computer program product, characterized in that, The computer program product includes a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1-9 or claim 10.

Images