Video text matching model training, video text matching method and device

Abstract

The application relates to a video-text matching model training method, a video-text matching method, a device, computer equipment, a storage medium and a computer program product. The application relates to artificial intelligence technology. The method comprises the following steps: inputting video features corresponding to a training video pair, reference features and training text features corresponding to a training text pair in a training sample pair set into an initial video-text matching model; the reference features comprise at least one of audio features and action features; the corresponding video features are subjected to feature enhancement based on the reference features corresponding to the same training video, so as to obtain reference enhanced video features corresponding to the training video; the training text features corresponding to the training text are subjected to similarity calculation with the video features and the reference enhanced video features corresponding to the training video respectively; and the initial video-text matching model is trained based on the similarity set corresponding to each training sample pair, so as to obtain a target video-text matching model. The method can improve the prediction accuracy of the model.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a video text matching model training, video text matching method, apparatus, computer device, storage medium, and computer program product. Background Technology

[0002] With the development of computer technology, video text matching models have emerged. Based on video text matching models, matching videos and texts can be identified from a large number of videos and texts. Video text matching models can be applied to scenarios such as video text retrieval, video content recommendation, and video content understanding.

[0003] Traditional techniques typically train video-text matching models based on video and text features. However, video features only provide image information and cannot accurately represent the rich information within the video. Consequently, video-text matching models trained using current methods suffer from low prediction accuracy. Summary of the Invention

[0004] Therefore, it is necessary to provide a video text matching model training, video text matching method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the accuracy of model prediction in order to address the above-mentioned technical problems.

[0005] This application provides a method for training a video text matching model. The method includes:

[0006] Obtain a set of training sample pairs; the training sample pairs in the set of training sample pairs include positive sample pairs and negative sample pairs that match the positive sample pairs, the training sample pairs include training videos and training text, and the training videos and training text in the positive sample pairs match each other;

[0007] The video features, reference features, and training text features corresponding to the training videos in the training sample pair set are input into the initial video-text matching model; the reference features include at least one of the audio features and motion features corresponding to the training videos.

[0008] Based on the reference features corresponding to the same training video, feature enhancement is performed on the corresponding video features to obtain reference enhanced video features corresponding to the training video; the reference enhanced video features include at least one of motion enhanced video features and audio enhanced video features.

[0009] For the same training sample pair, the similarity of the training text features corresponding to the training text with the video features corresponding to the training video and the reference enhanced video features is calculated to obtain the similarity set corresponding to each training sample pair.

[0010] The training loss is calculated based on the similarity set corresponding to the positive sample pairs and the similarity set corresponding to the matching negative sample pairs. The model parameters of the initial video-text matching model are adjusted based on the training loss until the convergence condition is met, thus obtaining the target video-text matching model. The target video-text matching model is used to determine the matching result between the video and the text.

[0011] This application also provides a video text matching model training device. The device includes:

[0012] The training sample pair set acquisition module is used to acquire a training sample pair set; the training sample pairs in the training sample pair set include positive sample pairs and negative sample pairs that match the positive sample pairs, the training sample pairs include training videos and training text, and the training videos and training text in the positive sample pairs match each other;

[0013] The feature input module is used to input the video features, reference features, and training text features corresponding to the training videos in the training sample pair set into the initial video-text matching model; the reference features include at least one of the audio features and motion features corresponding to the training videos.

[0014] The feature enhancement module is used to enhance the corresponding video features based on the reference features corresponding to the same training video to obtain the reference enhanced video features corresponding to the training video; the reference enhanced video features include at least one of motion enhanced video features and audio enhanced video features.

[0015] The similarity calculation module is used to calculate the similarity between the training text features corresponding to the training text and the video features and reference enhanced video features corresponding to the training video for the same training sample pair, so as to obtain the similarity set corresponding to each training sample pair.

[0016] The model adjustment module is used to calculate the training loss based on the similarity set corresponding to the positive sample pairs and the similarity set corresponding to the matching negative sample pairs, and adjust the model parameters of the initial video-text matching model based on the training loss until the convergence condition is met to obtain the target video-text matching model; the target video-text matching model is used to determine the matching result between the video and the text.

[0017] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps described in the video text matching model training method above.

[0018] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps described in the above-described video text matching model training method.

[0019] A computer program product includes a computer program that, when executed by a processor, implements the steps described in the video text matching model training method above.

[0020] The aforementioned video-text matching model training method, apparatus, computer equipment, storage medium, and computer program product acquire a training sample pair set. The training sample pairs in the set include positive sample pairs and negative sample pairs that match the positive sample pairs. Each training sample pair includes a training video and training text, with the training video and training text in the positive sample pairs matching each other. The video features, reference features, and training text features corresponding to the training videos in the training sample pair set are input into an initial video-text matching model. The reference features include at least one of audio features and motion features corresponding to the training videos. Based on the reference features corresponding to the same training video, feature enhancement is performed on the corresponding video features to obtain the training model. The video is used to obtain reference enhanced video features, which include at least one of action-enhanced video features and audio-enhanced video features. For the same training sample pair, the training text features corresponding to the training text are compared with the video features corresponding to the training video and the reference enhanced video features to obtain similarity sets for each training sample pair. The training loss is calculated based on the similarity sets corresponding to positive sample pairs and the similarity sets corresponding to matching negative sample pairs. The model parameters of the initial video-text matching model are adjusted based on the training loss until the convergence condition is met to obtain the target video-text matching model. The target video-text matching model is used to determine the matching result between the video and the text. In this way, video features can provide image information of the video, audio features can provide sound information of the video, and action features can provide motion information of the video. By training the video-text matching model based on the video features corresponding to the training video, the reference features, and the training text features corresponding to the training text, the rich modal information in the video can be used to improve the model's understanding of the video content, thereby improving the model's prediction accuracy. Furthermore, feature enhancement and feature guidance based on audio or motion features can highlight important information in the video. By calculating similarity between the video features and the reference enhanced video features and the training text features, and adjusting the model parameters based on the training loss generated from the calculated similarity set, the model can better establish the relationship between the video and the text, thereby further improving the model's prediction accuracy.

[0021] This application provides a video text matching method. The method includes:

[0022] Obtain the video features and reference features corresponding to the video to be matched, and obtain the text features corresponding to the text to be matched; the reference features to be matched include at least one of the audio features and motion features corresponding to the video to be matched.

[0023] Based on the reference features to be matched, feature enhancement is performed on the video features to be matched to obtain reference enhanced video features corresponding to the video to be matched; the reference enhanced video features include at least one of motion enhanced video features and audio enhanced video features;

[0024] The similarity between the text feature to be matched and the video feature to be matched and the reference enhanced video feature is calculated to obtain the similarity set between the video to be matched and the text to be matched.

[0025] Based on the similarity set corresponding to the video to be matched and the text to be matched, the matching result between the video to be matched and the text to be matched is determined.

[0026] In one embodiment, the step of calculating the similarity between the video text features corresponding to the video to be matched and the target text features corresponding to the text to be matched, to obtain a reference similarity between the video text features and the target text features, includes:

[0027] Calculate the initial similarity between the video text features and the target text features to obtain an initial similarity matrix;

[0028] The number of matrix elements in the initial similarity matrix whose values ​​are greater than a preset threshold is counted to obtain a first count.

[0029] The number of text words corresponding to the text to be matched and the video text are fused to obtain a second number; the video text refers to the text corresponding to the video text feature.

[0030] Based on the first quantity and the second quantity, a reference similarity is obtained between the video text features and the target text features.

[0031] In one embodiment, the current text is any one of audio text, image text, and text to be matched, and the text feature corresponding to the current text is any one of audio text feature, image text feature, or target text feature. The process of generating the text feature corresponding to the current text includes the following steps:

[0032] Noun extraction is performed on the current text to obtain text nouns;

[0033] Feature extraction is performed on the text nouns to obtain the text features corresponding to the current text.

[0034] In one embodiment, based on the reference features to be matched, feature enhancement is performed on the video features to be matched to obtain reference enhanced video features corresponding to the video to be matched. Then, the text features to be matched are compared with the video features to be matched and the reference enhanced video features to obtain a similarity set between the video to be matched and the text to be matched, including:

[0035] The features of the video to be matched, the reference features to be matched, and the features of the text to be matched are input into the target video-text matching model to obtain the similarity set corresponding to the video to be matched and the text to be matched; the target video-text matching model is used for feature enhancement and similarity calculation.

[0036] This application also provides a video text matching device. The device includes:

[0037] The feature acquisition module is used to acquire the video features to be matched and the reference features to be matched corresponding to the video to be matched, and to acquire the text features to be matched corresponding to the text to be matched; the reference features to be matched include at least one of the audio features and motion features corresponding to the video to be matched;

[0038] The feature enhancement module is used to enhance the features of the video to be matched based on the reference features to be matched, so as to obtain the reference enhanced video features corresponding to the video to be matched; the reference enhanced video features include at least one of motion enhanced video features and audio enhanced video features;

[0039] The similarity calculation module is used to calculate the similarity between the text feature to be matched and the video feature to be matched and the reference enhanced video feature, respectively, to obtain the similarity set between the video to be matched and the text to be matched;

[0040] The matching result determination module is used to determine the matching result between the video to be matched and the text to be matched based on the similarity set corresponding to the video to be matched and the text to be matched.

[0041] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps described in the video text matching method above.

[0042] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps described in the video text matching method above.

[0043] A computer program product includes a computer program that, when executed by a processor, implements the steps described in the video text matching method above.

[0044] The aforementioned video-text matching method, apparatus, computer equipment, storage medium, and computer program product acquires the video features to be matched and the reference features to be matched corresponding to the video to be matched, and then acquires the text features to be matched corresponding to the text to be matched. The reference features to be matched include at least one of the audio features and motion features corresponding to the video to be matched. Based on the reference features to be matched, feature enhancement is performed on the video features to be matched to obtain the reference enhanced video features corresponding to the video to be matched. The reference enhanced video features include at least one of the motion enhanced video features and the audio enhanced video features. The similarity between the text features to be matched and the video features to be matched and the reference enhanced video features is calculated to obtain a similarity set between the video to be matched and the text to be matched. Based on the similarity set between the video to be matched and the text to be matched, the matching result between the video to be matched and the text to be matched is determined. In this way, video features can provide image information of the video, audio features can provide sound information of the video, and motion features can provide motion information of the video. By determining the matching result between the video to be matched and the text to be matched based on the video features, reference features, and text features corresponding to the video to be matched, the rich modal information in the video can be used to improve the understanding of the video content, thereby improving the matching accuracy. Furthermore, feature enhancement and guidance based on audio or motion features can highlight important information in the video, further improving the understanding of the video content. Calculating similarity between the video features and the reference enhanced video features, and then determining the matching result based on the calculated similarity set, can further improve matching accuracy. Attached Figure Description

[0045] Figure 1 This is a diagram illustrating the video text matching model training method and the application environment of the video text matching method in one embodiment.

[0046] Figure 2 This is a flowchart illustrating a video-text matching model training method in one embodiment;

[0047] Figure 3 This is a schematic diagram of a video enhancement video feature and text feature matching network in one embodiment;

[0048] Figure 4 This is a schematic diagram of an audio-enhanced video feature and text feature matching network in one embodiment;

[0049] Figure 5 This is a flowchart illustrating the similarity calculation and loss calculation process in one embodiment;

[0050] Figure 6 This is a flowchart illustrating the process of determining the predicted matching ranking corresponding to a test sample pair in one embodiment.

[0051] Figure 7 This is a schematic diagram illustrating the sorting of the similarity matrix in one embodiment;

[0052] Figure 8 This is a flowchart illustrating a video text matching method in one embodiment;

[0053] Figure 9 This is a flowchart illustrating the video text matching method in another embodiment;

[0054] Figure 10 This is a schematic diagram of a cosine similarity matrix calculated based on the text to be matched and the video text in one embodiment;

[0055] Figure 11 This is a flowchart illustrating the process of calculating the similarity between the text to be matched and the video text in one embodiment;

[0056] Figure 12 This is a schematic diagram of the architecture of a video text matching method in one embodiment;

[0057] Figure 13 This is a schematic diagram of the text to be matched and the video text in another embodiment;

[0058] Figure 14 This is a structural block diagram of a video text matching model training device in one embodiment;

[0059] Figure 15 This is a structural block diagram of a video text matching device in one embodiment;

[0060] Figure 16 This is an internal structural diagram of a computer device in one embodiment;

[0061] Figure 17 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] The solutions provided in this application involve artificial intelligence technologies such as computer vision, speech technology, natural language processing, and machine learning, which are specifically illustrated through the following embodiments:

[0064] The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, data retrieval, and data recommendation.

[0065] The video text matching model training method and video text matching method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on other servers. Terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server, a server cluster consisting of multiple servers, or a cloud server.

[0066] Both the terminal and the server can be used independently to execute the video text matching model training method and the video text matching method provided in the embodiments of this application.

[0067] For example, the server acquires a set of training sample pairs. These training sample pairs include positive sample pairs and negative sample pairs that match the positive ones. Each training sample pair includes training videos and training text, with the training videos and training text in the positive sample pairs matching each other. The server inputs the video features, reference features, and training text features corresponding to the training videos from the training sample pair set into the initial video-text matching model. The reference features include at least one of the audio features and motion features corresponding to the training videos. In the model, based on the reference features corresponding to the same training video, feature enhancement is performed on the corresponding video features to obtain reference enhanced video features corresponding to the training videos. These reference enhanced video features include at least one of the motion enhanced video features and audio enhanced video features. For the same training sample pair, the similarity between the training text features corresponding to the training text and the video features and reference enhanced video features corresponding to the training videos is calculated to obtain similarity sets corresponding to each training sample pair. The server calculates the training loss based on the similarity sets corresponding to the positive sample pairs and the similarity sets corresponding to the matching negative sample pairs. Based on the training loss, the model parameters of the initial video-text matching model are adjusted until the convergence condition is met, resulting in the target video-text matching model.

[0068] The server obtains the video features and reference features corresponding to the video to be matched, and the text features corresponding to the text to be matched. The reference features include at least one of the audio features and motion features corresponding to the video to be matched. Based on the reference features, the server enhances the video features to be matched, obtaining reference enhanced video features corresponding to the video to be matched. The reference enhanced video features include at least one of the motion enhanced video features and audio enhanced video features. The server calculates the similarity between the text features to be matched and the video features and reference enhanced video features, respectively, to obtain a similarity set between the video and the text to be matched. Based on the similarity set between the video and the text to be matched, the server determines the matching result between the video and the text to be matched.

[0069] The terminal and server can also work together to execute the video text matching model training method and video text matching method provided in the embodiments of this application.

[0070] For example, the server obtains a set of training sample pairs from the terminal, and trains the initial video-text matching model based on the video features, reference features, and training text features corresponding to the training videos and training texts in the set of training sample pairs, thereby obtaining the target video-text matching model.

[0071] The terminal sends a data matching request to the server. This request carries data identifiers corresponding to the video and text to be matched, respectively. The server obtains the video features and reference features corresponding to the video based on these identifiers. The server then enhances the video features based on the reference features, resulting in enhanced reference video features. These enhanced reference features include at least one of motion-enhanced and audio-enhanced video features. The server calculates the similarity between the text features and both the video features and the enhanced reference features, obtaining a similarity set between the video and text. Based on this similarity set, the server determines the matching result between the video and text. Finally, the server sends the matching result to the terminal.

[0072] In one embodiment, such as Figure 2 As shown, a video text matching model training method is provided. Taking the application of this method to a computer device as an example, the computer device can be the one described above. Figure 1 Terminal 102 or server 104 in the reference. Figure 2 The training method for the video-text matching model includes the following steps:

[0073] Step S202: Obtain a training sample pair set; the training sample pairs in the training sample pair set include positive sample pairs and negative sample pairs that match the positive sample pairs. The training sample pairs include training videos and training texts, and the training videos and training texts in the positive sample pairs match each other.

[0074] The training sample pair set includes multiple training sample pairs. Each training sample pair consists of one training video and one training text. Training sample pairs can be divided into positive and negative sample pairs. In a positive sample pair, the training video and training text match each other; that is, the content and information expressed by the training video and training text in a positive sample pair are consistent, and the training text in a positive sample pair can be used to describe and explain the training video in the positive sample pair. In a negative sample pair, the training video and training text do not match; that is, the content and information expressed by the training video and training text in a negative sample pair are inconsistent.

[0075] The training sample pairs in the training sample pair set include positive sample pairs and negative sample pairs that match the positive sample pairs. Negative sample pairs that match the positive sample pairs include at least one of the following: negative sample pairs that contain the same training video but different training text as the positive sample pairs, and negative sample pairs that contain the same training text but different training videos as the positive sample pairs.

[0076] Specifically, the computer device can obtain a set of training sample pairs locally or from other devices, and train the model based on the set of training sample pairs to obtain a trained video-text matching model.

[0077] Step S204: Input the video features, reference features, and training text features corresponding to the training videos in the training sample pair set into the initial video-text matching model; the reference features include at least one of the audio features and motion features corresponding to the training videos.

[0078] Here, video features refer to the image features of the video, used to represent the image modal information in the video. Audio features are used to represent the audio modal information in the video. Motion features are used to represent the motion modal information (i.e., motion modal information) in the video. Reference features include at least one of the audio features and motion features corresponding to the training video. Feature extraction from the training video yields the corresponding video features, motion features, and audio features. Feature extraction from the training video can be performed based on machine learning models. Specifically, video features can be extracted based on a video feature extraction model, motion features can be extracted based on a motion feature extraction model, and audio features can be extracted based on an audio feature extraction model. For example, motion features can be extracted from the training video using the S3D model, and audio features can be extracted based on the VGGish model.

[0079] Text features are used to represent the textual modal information of a text. Training text features refer to the text features corresponding to the training text. Features can be extracted from the training text based on machine learning models; specifically, training text features can be extracted based on text feature extraction models.

[0080] In one embodiment, the video features corresponding to the training video and the text features corresponding to the training text can be obtained based on the same machine learning model. The video features corresponding to the training video and the text features corresponding to the training text can be extracted based on a video text processing model, which includes a video encoder and a text encoder. The video features corresponding to the training video are extracted based on the video encoder, and the text features corresponding to the training text are extracted based on the text encoder. In one embodiment, the video text processing model is used to classify videos based on their descriptive text. The input data of the video text processing model are the video and its corresponding descriptive text, and the output data is the classification result of the video. For example, the video features corresponding to the training video and the text features corresponding to the training text can be extracted based on the CLIP (Contrastive Language-Image Pre-training) model.

[0081] The initial video-text matching model refers to the video-text matching model to be trained. The input data of the video-text matching model consists of the video features corresponding to the training video, the reference features, and the training text features corresponding to the training text.

[0082] Specifically, the computer device can input the video features, reference features, and training text features corresponding to the training videos in the training sample pair set into the initial video-text matching model. In the initial video-text matching model, the video features, reference features, and training text features are processed to obtain the similarity set corresponding to each training sample pair. The initial video-text matching model can output the similarity set corresponding to each training sample pair.

[0083] Step S206: Based on the reference features corresponding to the same training video, feature enhancement is performed on the corresponding video features to obtain reference enhanced video features corresponding to the training video; the reference enhanced video features include at least one of action enhanced video features and audio enhanced video features.

[0084] Feature enhancement is used to strengthen information in video features that is related to reference features, while weakening other information in video features. For example, based on the action features corresponding to the training video, feature enhancement is performed on the video features corresponding to the training video to obtain action-enhanced video features. Action-enhanced video features strengthen the feature representation of moving objects in the video, while the feature representation of other background and noise is weakened.

[0085] Motion-enhanced video features are obtained by enhancing the video features corresponding to the motion features of the training video. Audio-enhanced video features are obtained by enhancing the video features corresponding to the audio features of the training video.

[0086] It is understandable that the processes of feature enhancement based on action features and feature enhancement based on audio features can be the same or different.

[0087] Specifically, in the initial video-text matching model, feature enhancement can be performed on the corresponding video features based on the reference features corresponding to the same training video to obtain reference-enhanced video features corresponding to the training video. If the reference features include action features, then feature enhancement is performed on the corresponding video features based on the action features corresponding to the same training video to obtain action-enhanced video features corresponding to the training video, and the reference-enhanced video features include action-enhanced video features. If the reference features include audio features, then feature enhancement is performed on the corresponding video features based on the audio features corresponding to the same training video to obtain audio-enhanced video features corresponding to the training video, and the reference-enhanced video features include audio-enhanced video features.

[0088] In one embodiment, reference features and video features are fused to obtain reference video features, which incorporate both image and motion information from the video. Channel attention processing is applied to the reference video features to obtain reference channel attention weights. Based on these reference channel attention weights, feature enhancement is performed on the video features to obtain reference enhanced video features. Feature enhancement based on the reference channel attention weights helps establish associations between objects with motion relationships within the video features, highlighting moving objects in the video features.

[0089] Step S208: For the same training sample pair, calculate the similarity between the training text features corresponding to the training text and the video features and reference enhanced video features corresponding to the training video, respectively, to obtain the similarity set corresponding to each training sample pair.

[0090] In this system, a similarity set corresponds to a training sample pair. A similarity set includes the target similarity calculated by performing similarity calculations on the training text features corresponding to the training text and the video features corresponding to the training video, and the target similarity calculated by performing similarity calculations on the training text features corresponding to the training text and the reference augmented video features corresponding to the training video.

[0091] Specifically, in the initial video-text matching model, for the same training sample pair, the training text features corresponding to the training text are compared with the video features corresponding to the training video and the reference enhanced video features to calculate similarity. The calculated target similarities are then used to form the similarity set corresponding to the training sample pair. Since there are multiple training sample pairs, the final similarity set corresponding to each training sample pair can be obtained.

[0092] If the reference enhanced video features include action-enhanced video features, then the similarity set includes the target similarity calculated by performing similarity calculations on the training text features and action-enhanced video features. If the reference enhanced video features include audio-enhanced video features, then the similarity set includes the target similarity calculated by performing similarity calculations on the training text features and audio-enhanced video features.

[0093] It is understandable that when calculating similarity, one can calculate the Euclidean distance or cosine similarity between two features, or calculate the similarity between two features based on a custom formula or algorithm.

[0094] Step S210: Calculate the training loss based on the similarity set corresponding to the positive sample pairs and the similarity set corresponding to the matching negative sample pairs. Adjust the model parameters of the initial video-text matching model based on the training loss until the convergence condition is met to obtain the target video-text matching model. The target video-text matching model is used to determine the matching result between the video and the text.

[0095] The target video-text matching model refers to the trained video-text matching model. The convergence condition can be at least one of the following: training loss is less than a preset threshold, the number of model iterations is greater than a preset number, etc.

[0096] Specifically, after determining the similarity sets corresponding to each training sample pair, the computer device can calculate the training loss based on the similarity sets corresponding to positive sample pairs and the similarity sets corresponding to matching negative sample pairs. The computer device can calculate the training loss based on a loss function, which can be a commonly used loss function in model training or a custom loss function. Furthermore, the computer device can adjust the model parameters of the initial video-text matching model based on the training loss, and perform backpropagation based on the training loss to adjust the model parameters of the initial video-text matching model until the convergence condition is met, thus obtaining the target video-text matching model.

[0097] The target video-to-text matching model is used to determine the matching result between any video and any text. For example, the video features, reference features, and text features of the text to be matched can be input into the target video-to-text matching model. The model outputs a similarity set between the video and the text, and the matching result is determined based on this similarity set. Alternatively, the matching result can be determined internally within the target video-to-text matching model based on the similarity set, and the model then outputs the matching result.

[0098] In one embodiment, the training loss includes training sub-losses corresponding to various similarities, that is, the training loss includes training sub-losses corresponding to each similarity category. The computer device can first calculate the training sub-losses based on the target similarities belonging to the same similarity category in positive sample pairs and matching negative sample pairs, obtaining the training sub-losses corresponding to each similarity category, and then obtain the training loss based on these training sub-losses. It can be understood that target similarities calculated based on data of the same type belong to the same similarity category.

[0099] In one embodiment, when calculating any training sub-loss, the computer device can fuse the target similarities corresponding to positive sample pairs and matching negative sample pairs to obtain the fused similarity for each positive sample pair. For example, the sum of the target similarities corresponding to the positive sample pairs and matching negative sample pairs can be used as the fused similarity; the target similarities corresponding to the positive sample pairs and matching negative sample pairs can be exponentially processed to the base of the natural constant e, and the sum of the results of each exponential processing can be used as the fused similarity; and so on. Furthermore, the computer device calculates the sample loss based on the difference between the target similarity and the fused similarity corresponding to the same positive sample pair to obtain the sample loss for each positive sample pair. For example, the ratio of the target similarity to the fused similarity can be used as the sample loss; the target similarity corresponding to the positive sample pair can be exponentially processed to the base of the natural constant e, and the ratio of the exponential processing result corresponding to the positive sample pair to the fused similarity can be used as the sample loss; and so on. Finally, the computer device calculates the training sub-loss based on each sample loss. For example, the average of each sample loss can be calculated as the training sub-loss; the median of each sample loss can be used as the training sub-loss; and so on.

[0100] In one embodiment, the computer device can acquire multiple sets of training sample pairs and randomly select one set as the current set. The computer device inputs the video features, reference features, and training text features corresponding to the training video and training text in the current set into an initial video-text matching model. After data processing by the initial model, similarity sets are obtained for each training sample pair in the current set. Training loss is calculated based on these similarity sets, and the model parameters of the initial model are adjusted based on the training loss to obtain an intermediate video-text matching model. The computer device then uses the next set of training sample pairs as the new current set, the intermediate model as the new initial model, and returns to the step of inputting the video features, reference features, and training text features corresponding to the training video and training text in the current set into the initial model for iterative training. This process is repeated multiple times, continuously adjusting the model parameters until the convergence condition is met, resulting in the target video-text matching model. For example, if in a certain training round, the training loss calculated based on the similarity sets corresponding to the current sample pair set is less than a preset threshold, then the model parameters are stopped from being adjusted, and the latest adjusted video-text matching model is used as the target video-text matching model. If the number of model iterations after a certain training round is greater than a preset number, then the latest adjusted video-text matching model is used as the target video-text matching model.

[0101] It is understandable that adjusting the model parameters based on a current set of sample pairs constitutes one round of model training or one model iteration. Different sets of training sample pairs may or may not contain duplicate training sample pairs.

[0102] In the above video-text matching model training method, a training sample pair set is obtained. The training sample pairs in the set include positive sample pairs and negative sample pairs that match the positive sample pairs. Each training sample pair includes a training video and training text, and the training video and training text in the positive sample pairs are mutually matched. The video features, reference features, and training text features corresponding to the training videos in the training sample pair set are input into the initial video-text matching model. The reference features include at least one of the audio features and motion features corresponding to the training videos. Based on the reference features corresponding to the same training video, feature enhancement is performed on the corresponding video features to obtain the reference enhanced video corresponding to the training video. The video text matching model employs a combination of video and audio features. The reference enhanced video features include at least one of motion-enhanced video features and audio-enhanced video features. For the same training sample pair, the training text features corresponding to the training text are compared with the video features corresponding to the training video and the reference enhanced video features to calculate similarity sets for each training sample pair. The training loss is calculated based on the similarity sets corresponding to positive sample pairs and the similarity sets corresponding to matching negative sample pairs. The model parameters of the initial video text matching model are adjusted based on the training loss until the convergence condition is met, resulting in the target video text matching model. The target video text matching model is used to determine the matching result between the video and the text. In this way, video features can provide image information, audio features can provide sound information, and motion features can provide motion information. By training the video text matching model based on the video features corresponding to the training video, the reference features, and the training text features corresponding to the training text, the rich modal information in the video can be utilized to improve the model's understanding of the video content, thereby improving the model's prediction accuracy. Furthermore, feature enhancement and feature guidance based on audio or motion features can highlight important information in the video. By calculating similarity between the video features and the reference enhanced video features and the training text features, and adjusting the model parameters based on the training loss generated from the calculated similarity set, the model can better establish the relationship between the video and the text, thereby further improving the model's prediction accuracy.

[0103] In one embodiment, step S202 includes:

[0104] Obtain multiple positive sample pairs; reorganize the data of each positive sample pair to obtain multiple negative sample pairs; take the negative sample pairs that have overlapping data with the positive sample pairs as the negative sample pairs that are matched with the positive sample pairs; based on each positive sample pair and each matched negative sample pair, obtain the training sample pair set.

[0105] Data recombination refers to exchanging training videos or training texts between different positive sample pairs to reassemble the training sample pairs. For example, exchanging the training videos between two positive sample pairs can yield two negative sample pairs, and exchanging the training texts between two positive sample pairs can yield two more negative sample pairs.

[0106] Specifically, the computer device can acquire multiple positive sample pairs locally or from other devices, and recombine the data of each positive sample pair to obtain multiple negative sample pairs. Among each negative sample pair, the computer device can identify negative sample pairs that overlap with the positive sample pairs as matching negative sample pairs, thus obtaining the negative sample pairs corresponding to each positive sample pair. Finally, the positive sample pairs and their matching negative sample pairs are combined to form a training sample pair set.

[0107] For example, positive sample pair A includes training video a1 and training text a2, positive sample pair B includes training video b1 and training text b2, and positive sample pair C includes training video c1 and training text c2. Recombining three positive sample pairs yields a maximum of six negative sample pairs: negative sample pair D1 includes training video a1 and training text b2, negative sample pair D2 includes training video a1 and training text c2, negative sample pair D3 includes training video b1 and training text a2, negative sample pair D4 includes training video b1 and training text c2, negative sample pair D5 includes training video c1 and training text a2, and negative sample pair D6 includes training video c1 and training text b2. The positive sample pair A, positive sample pair B, positive sample pair C, and negative sample pairs D1 through D6 form the training sample pair set.

[0108] In the above embodiments, the positive sample pairs are recombined to obtain multiple negative sample pairs. Training sample pairs with overlapping data are used as mutually matched training sample pairs. Each positive sample pair and each matched negative sample pair form a training sample pair set. In this way, a training sample pair set containing both positive and negative sample pairs can be quickly obtained based on the positive sample pairs.

[0109] In one embodiment, the generation of motion-enhanced video features includes the following steps:

[0110] Intramodal attention processing is performed on the video features and action features corresponding to the current training video to obtain self-attention video features and self-attention action features corresponding to the current training video. Intermodal attention processing is performed on the video features and self-attention action features corresponding to the current training video to obtain cross-attention video features corresponding to the current training video. Intermodal attention processing is also performed on the action features and self-attention video features corresponding to the current training video to obtain cross-attention action features corresponding to the current training video. The cross-attention action features and cross-attention video features corresponding to the current training video are fused to obtain the action-video fusion features corresponding to the current training video. Channel attention processing is performed on the action-video fusion features corresponding to the current training video to obtain the first channel attention weight. Based on the first channel attention weight, feature enhancement is performed on the video features corresponding to the current training video to obtain the action-enhanced video features corresponding to the current training video.

[0111] In this context, modality refers to the source or form of information. For example, video features are one type of modal data, motion features are another, and audio features are yet another.

[0112] Intramodal attention processing involves applying attention to data from a single modality to highlight key or important information within that modality. Intermodal attention processing involves applying attention to data from at least two modalities to facilitate information exchange between them, thereby emphasizing the correlation between the two modalities.

[0113] Channel attention processing involves applying attention to the channel dimension of a feature to highlight the information represented by the most important channels. It can be understood that the data features obtained from feature extraction typically include multiple sub-features corresponding to different channels. Different channels focus on different information, and the sub-features corresponding to different channels can represent different semantic information within the data.

[0114] Specifically, when enhancing video features based on video action features, various attention processing methods can be used to process video features and action features to obtain first channel attention weights. Based on the first channel attention weights, video features can be enhanced to obtain action-enhanced video features.

[0115] The current training video refers to the training video being processed, and it can be any training video. The computer first performs intra-modal attention processing on the video features corresponding to the current training video to obtain self-attention video features. Then, it performs intra-modal attention processing on the action features corresponding to the current training video to obtain self-attention action features. Through intra-modal attention processing, features from each modality can be weighted, highlighting the key content within each modality.

[0116] Then, the computer device performs intermodal attention processing on the video features and self-attention action features corresponding to the current training video to obtain cross-attention video features, and performs intermodal attention processing on the action features and self-attention video features corresponding to the current training video to obtain cross-attention action features. By performing intermodal attention processing on the video features and self-attention action features, and weighting the video features based on the self-attention action features, action-weighted video features, i.e., cross-attention video features, can be obtained. Action-weighted video features are used to highlight motion information in video features. By performing intermodal attention processing on the action features and self-attention video features, and weighting the action features based on the self-attention video features, video-weighted action features, i.e., cross-attention action features, can be obtained. Video-weighted action features are used to highlight image information in action features.

[0117] Furthermore, the computer device fuses the cross-attention action features and cross-attention video features corresponding to the current training video to obtain the action-video fusion features corresponding to the current training video. By fusing the cross-attention action features and cross-attention video features, the information about the relationship between the action features and video features can be further highlighted.

[0118] Finally, the computer device performs channel attention processing on the motion video fusion features corresponding to the current training video to obtain the first channel attention weight. The first channel attention weight helps establish associations between objects with motion relationships in the video features. Based on the first channel attention weight, the computer device performs feature enhancement on the video features corresponding to the current training video to obtain the motion-enhanced video features corresponding to the current training video. The motion-enhanced video features emphasize the feature representation of moving objects in the video while weakening the feature representation of other information such as background and noise in the video.

[0119] It is understandable that audio-enhanced video features can also be obtained using the methods described above.

[0120] In one embodiment, intra-modal attention processing and inter-modal attention processing can be implemented based on commonly used attention mechanisms, such as key-value pair attention mechanisms and multi-head attention mechanisms. Alternatively, they can be implemented based on custom formulas or algorithms. The difference between intra-modal and inter-modal attention processing lies in the input information: intra-modal attention processing uses data from one modality, while inter-modal attention processing uses data from different modalities. Similarly, channel attention processing can be implemented based on commonly used channel attention mechanisms or based on custom formulas or algorithms.

[0121] In one embodiment, channel attention processing can be applied to the motion video fusion features using formula (1) to obtain motion-enhanced video features:

[0122] V t M =W t C ⊙V t W t C =σ(W2δ(W1MV) t )) Formula (1)

[0123] in, and These are two linear transformations. δ and σ represent the activation operations of ReLU (Rectified Linear Unit, an activation function) and sigmoid (an activation function), respectively. v denoted by , where d represents the feature dimension corresponding to the motion video fusion feature MV, and d represents the feature dimension of the data obtained after linearly transforming the motion video fusion feature MV based on W1. MV t This represents the motion-enhanced video feature corresponding to the t-th video frame in the video. W t C V represents the attention weight of the first channel corresponding to the t-th video frame in the video. t M This represents the motion-enhanced video feature corresponding to the t-th video frame in the video.

[0124] In the above embodiments, by organically combining intramodal attention processing, intermodal attention processing, and channel attention processing, important information related to each other can be extracted from video features and action features to obtain the first channel attention weight. Based on the first channel attention weight, the video features are enhanced to obtain action-enhanced video features. Action-enhanced video features strengthen the feature representation of moving objects in the video, which can better express the semantic information of the video, thereby helping to improve the prediction accuracy of the model during model training.

[0125] In one embodiment, intra-modal attention processing is performed on the video features and action features corresponding to the current training video to obtain self-attention video features and self-attention action features corresponding to the current training video, including:

[0126] The video features corresponding to the current training video are fused intramodally to obtain the first self-attention weight. The video features corresponding to the current training video and the first self-attention weight are fused to obtain the first attention feature. Based on the video features corresponding to the current training video and the first attention feature, the self-attention video feature is obtained. The action features corresponding to the current training video are fused intramodally to obtain the second self-attention weight. The action features corresponding to the current training video and the second self-attention weight are fused to obtain the second attention feature. Based on the action features corresponding to the current training video and the second attention feature, the self-attention action feature is obtained.

[0127] Intramodal fusion involves fusing data from single modalities. For example, intramodal fusion can be achieved by multiplying video features together and then performing a softmax (normalized exponential function) operation; alternatively, video features can be linearly transformed, and the transformed video features can be multiplied together and then subjected to a softmax operation; and so on.

[0128] Specifically, when performing intra-modal attention processing on the video features corresponding to the current training video, the computer device can perform intra-modal fusion of the video features corresponding to the current training video to obtain a first self-attention weight, and then fuse the video features corresponding to the current training video and the first self-attention weight to obtain a first attention feature. The first attention feature emphasizes important information in the video features, particularly information with important semantics. Furthermore, based on the video features corresponding to the current training video and the first attention feature, the computer device obtains a self-attention video feature. For example, the average of the video features and the first attention feature can be used as the self-attention video feature; or the weighted average of the video features and the first attention feature can be used as the self-attention video feature; and so on.

[0129] Similarly, when performing intramodal attention processing on the action features corresponding to the current training video, the computer device can perform intramodal fusion of the action features corresponding to the current training video to obtain a second self-attention weight, and then fuse the action features corresponding to the current training video and the second self-attention weight to obtain a second attention feature. The second attention feature emphasizes important information in the action features, particularly information with important semantics. Furthermore, based on the action features corresponding to the current training video and the second attention feature, the computer device obtains self-attention action features.

[0130] In the above embodiments, based on the first self-attention weights obtained by intramodal fusion of video features, weighted fusion of video features can strengthen the feature representation of important information in the video features. Then, based on the video features and the first attention features obtained through weighted fusion, accurate self-attention video features can be obtained. Similarly, based on the second self-attention weights obtained by intramodal fusion of action features, weighted fusion of action features can strengthen the feature representation of important information in the action features. Then, based on the action features and the second attention features obtained through weighted fusion, accurate self-attention action features can be obtained.

[0131] In one embodiment, intermodal attention processing is performed on the video features and self-attention action features corresponding to the current training video to obtain cross-attention video features corresponding to the current training video, and intermodal attention processing is performed on the action features and self-attention video features corresponding to the current training video to obtain cross-attention action features corresponding to the current training video, including:

[0132] The video features and self-attention action features corresponding to the current training video are concatenated to obtain the first concatenated feature. The video features and the first concatenated feature are then fused between modalities to obtain the first cross-attention weight. The first concatenated feature and the first cross-attention weight are then fused to obtain the first cross-attention feature. Based on the first concatenated feature and the first cross-attention feature, a cross-attention video feature is obtained. The action features and self-attention video features corresponding to the current training video are concatenated to obtain the second concatenated feature. The action features and the second concatenated feature are then fused between modalities to obtain the second cross-attention weight. The second concatenated feature and the second cross-attention weight are then fused to obtain the second cross-attention feature. Based on the second concatenated feature and the second cross-attention feature, a cross-attention action feature is obtained.

[0133] Intermodal fusion involves fusing data from different modalities. For example, intermodal fusion can be achieved by multiplying video features and the first concatenated features, followed by a softmax operation; alternatively, a linear transformation can be performed on the video features and the first concatenated features, followed by multiplying the transformed video features and the first concatenated features, and then performing a softmax operation; and so on.

[0134] Specifically, when performing intermodal attention processing on the video features and self-attention action features corresponding to the current training video, the computer device concatenates the video features and self-attention action features to obtain a first concatenated feature. This first concatenated feature integrates image and action information from the video and is a new feature distinct from the video features. Next, the computer device performs intermodal fusion on the video features and the first concatenated feature to obtain a first cross-attention weight. The first concatenated feature and the first cross-attention weight are then fused to obtain a first cross-attention feature. This first cross-attention feature emphasizes important information and semantically significant information within the first concatenated feature. Finally, based on the first concatenated feature and the first cross-attention feature, the computer device obtains the cross-attention video feature.

[0135] Similarly, during intermodal attention processing of action features and self-attention video features, the computer device concatenates the action features and self-attention video features corresponding to the current training video to obtain a second concatenated feature. This second concatenated feature integrates image and action information from the video, representing a new feature distinct from the action features. Furthermore, the computer device performs intermodal fusion of the action features and the second concatenated feature corresponding to the current training video to obtain a second cross-attention weight. This second concatenated feature is then fused with the second cross-attention weight to obtain a second cross-attention feature. The second cross-attention feature emphasizes important information and semantically significant information within the second concatenated feature. Finally, based on the second concatenated feature and the second cross-attention feature, the computer device obtains the cross-attention action feature.

[0136] In the above embodiments, video features and self-attention action features are concatenated to obtain a first concatenated feature. Based on the first cross-attention weight obtained by intermodal fusion of the video features and the first concatenated feature, the first concatenated feature is weighted and fused, which can strengthen the feature expression of important information in the first concatenated feature. Then, based on the first concatenated feature and the first cross-attention feature obtained by weighted fusion, accurate cross-attention video features can be obtained. Action features and self-attention video features are concatenated to obtain a second concatenated feature. Based on the second cross-attention weight obtained by intermodal fusion of the action features and the second concatenated feature, the second concatenated feature is weighted and fused, which can strengthen the feature expression of important information in the second concatenated feature. Then, based on the second concatenated feature and the second cross-attention feature obtained by weighted fusion, accurate cross-attention action features can be obtained.

[0137] In one embodiment, the cross-attention action features and cross-attention video features corresponding to the current training video are fused to obtain the action-video fusion features corresponding to the current training video, including:

[0138] The cross-attention action features and cross-attention video features corresponding to the current training video are concatenated to obtain cross-attention concatenated features; the cross-attention action features and cross-attention video features corresponding to the current training video are fused to obtain cross-attention fused features; the cross-attention concatenated features and cross-attention fused features are fused between modalities to obtain third cross-attention weights; the cross-attention concatenated features and third cross-attention weights are fused to obtain third cross-attention features; and the action-video fused features are obtained based on the cross-attention concatenated features and third cross-attention features.

[0139] Specifically, when fusing cross-attention action features and cross-attention video features, the computer device can concatenate the cross-attention action features and cross-attention video features corresponding to the current training video to obtain cross-attention concatenated features. The computer device can then fuse the cross-attention action features and cross-attention video features corresponding to the current training video to obtain cross-attention fused features. For example, multiplying the cross-attention action features and cross-attention video features and then scaling them yields the cross-attention fused features; and so on. Furthermore, the computer device performs intermodal fusion of the cross-attention concatenated features and cross-attention fused features to obtain a third cross-attention weight. The cross-attention concatenated features and the third cross-attention weight are then fused to obtain a third cross-attention feature. The third cross-attention feature emphasizes important information in the cross-attention concatenated features, particularly semantically important information. Finally, based on the cross-attention concatenated features and the third cross-attention feature, the action-video fusion feature is obtained.

[0140] In the above embodiments, cross-attention action features and cross-attention video features are respectively spliced ​​and fused to obtain cross-attention spliced ​​features and cross-attention fused features. Based on the third cross-attention weight obtained by intermodal fusion of cross-attention spliced ​​features and cross-attention fused features, the cross-attention spliced ​​features are weighted and fused to enhance the feature expression of important information in the cross-attention spliced ​​features. Then, based on the cross-attention spliced ​​features and the third cross-attention features obtained by weighted fusion, accurate action video fused features can be obtained.

[0141] In one embodiment, attention processing can be performed based on a dual-stream fusion strategy of the Transformer encoder. The Transformer encoder is shown in Equation (2):

[0142]

[0143]

[0144] In this equation, MHA represents Multi-Head Attention, which works by multiplying matrices Q and K, performing a softmax operation to obtain weight W, and then multiplying W and V to obtain the weighted result. FFN represents Feed Forward Network, LN represents Layer Normalization, and Encoder represents the encoder of the Transformer. In equation (2), Q, K, and V are the features of the input encoder. This represents a linear transformation, where d is the feature dimension.

[0145] Self-attention video features V obtained by performing intra-modal attention processing on video features V self =Encoder(V,V,V).

[0146] Self-attentional action features M obtained by performing intramodal attention processing on action features M self =Encoder(M,M,M).

[0147] For video features V and self-attention action features M self Cross-attention video features V obtained by intermodal attention processing cross =Encoder(V,cat(V,M) self ),cat(V,M self ()). Here, cat concatenates features along the time dimension.

[0148] Action features M and self-attention video features V self Cross-attention action features M obtained by intermodal attention processing cross =Encoder(M,cat(M,V) self ),,cat(M,V self )).

[0149] Fusion of cross-attention video features V cross and cross-attention action features M cross The obtained motion video fusion feature MV = Encoder(V cross ·M cross ,cat(V cross M cross ),cat(V cross M cross )).

[0150] The motion video fusion feature MV is subjected to channel-level attention operations to obtain the first channel attention weight. Based on the first channel attention weight, the video feature V is enhanced and guided to obtain the motion-enhanced video feature V. M .

[0151] In one embodiment, the initial video-text matching model includes an action-enhanced video feature and text feature matching network, which includes a first intramodal attention layer, a second intramodal attention layer, a first intermodal attention layer, a second intermodal attention layer, an action-video fusion attention layer, a first channel attention layer, and a first similarity calculation layer.

[0152] The first intramodal attention layer performs intramodal attention processing on video features, and the second intramodal attention layer performs intramodal attention processing on action features. The first intermodal attention layer performs intermodal attention processing on video features and self-attention action features, and the second intermodal attention layer performs intermodal attention processing on action features and self-attention video features. The action-video fusion attention layer fuses cross-attention action features and cross-attention video features corresponding to the same training video. The first channel attention layer performs channel attention processing on the action-video fusion features. The first similarity calculation layer calculates the similarity between action-enhanced video features and training text features.

[0153] Among them, the motion-enhanced video feature and text feature matching network is used to enhance the video features corresponding to the training video based on the motion features corresponding to the training video, and to calculate the target similarity between the motion-enhanced video features and the training text features corresponding to the training text.

[0154] refer to Figure 3 The motion-enhanced video feature and text feature matching network includes a first intramodal attention layer, a second intramodal attention layer, a first intermodal attention layer, a second intermodal attention layer, an motion-video fusion attention layer, a first channel attention layer, and a first similarity calculation layer. The intramodal attention layer can also be called an intramodal attention module, the intermodal attention layer can also be called an intermodal fusion attention module, and the motion-video fusion attention layer can also be called an motion-video fusion attention module.

[0155] The video features corresponding to the current training video are input into the first intramodal attention layer for intramodal attention processing, thereby exploring the modality. The first intramodal attention layer outputs self-attention video features. The action features corresponding to the current training video are input into the second intramodal attention layer for intramodal attention processing, thereby exploring the modality. The second intramodal attention layer outputs self-attention action features. The video features and self-attention action features corresponding to the current training video are input into the first intermodal attention layer for intermodal attention processing, thereby modeling intermodal relationships. The first intermodal attention layer outputs cross-attention video features. The action features and self-attention video features corresponding to the current training video are input into the second intermodal attention layer for intermodal attention processing, thereby modeling intermodal relationships. The second intermodal attention layer outputs cross-attention action features. The cross-attention action features and cross-attention video features corresponding to the current training video are input into the action-video fusion attention layer for further fusion of action features and video features, resulting in action-video fusion features (also called action-video fusion features). The action-video fusion features are input into the first channel attention layer for channel attention processing, and the first channel attention layer outputs first channel attention weights. Based on the first channel attention weight, feature enhancement is performed on the video features corresponding to the current training video to obtain action-enhanced video features (also known as action-enhanced video features). The action-enhanced video features corresponding to the current training video and the training text features corresponding to the corresponding training text are input into the first similarity calculation layer for similarity calculation and similarity matching. The first similarity calculation layer outputs a target similarity between the current training video and the corresponding training text.

[0156] In the above embodiments, the initial video-text matching model includes an action-enhanced video feature and a text feature matching network. The action-enhanced video features are obtained by performing feature enhancement through the dedicated action-enhanced video feature and text feature matching network. The similarity between the action-enhanced video features and the text features is calculated through the action-enhanced video feature and text feature matching network, which can reduce the risk of confusion with other modalities and help improve the training quality of the model.

[0157] In one embodiment, the process of generating audio-enhanced video features includes the following steps:

[0158] The video and audio features corresponding to the current training video are fused to obtain the initial audio-video fusion features. The initial audio-video fusion features are then subjected to random deactivation and pooling to obtain the intermediate audio-video fusion features. The intermediate audio-video fusion features are then normalized to obtain the target audio-video fusion features. The target audio-video fusion features are then subjected to channel attention processing to obtain the second channel attention weights. Based on the second channel attention weights, the video features corresponding to the current training video are then augmented to obtain the audio-enhanced video features corresponding to the current training video.

[0159] The random deactivation process involves randomly changing some sub-features in the initial audio-video fusion features. For example, the feature values ​​of some sub-features in the initial audio-video fusion features are randomly set to preset values.

[0160] Pooling is a statistical process that performs statistical analysis on the feature values ​​of sub-features obtained after random deactivation. For example, pooling can specifically be sum pooling, which divides the features obtained after random deactivation into multiple feature regions, each containing multiple sub-features. The feature values ​​of the sub-features in each feature region are summed to obtain the feature statistics corresponding to each feature region, and the pooling result is obtained based on these feature statistics.

[0161] Normalization is the process of mapping data to a preset range of values.

[0162] Specifically, when enhancing video features based on audio features, the second channel attention weight can be obtained by processing the video and audio features. The video features are then enhanced based on the second channel attention weight to obtain audio-enhanced video features.

[0163] The computer device can fuse video and audio features corresponding to the current training video to obtain initial audio-video fusion features. For example, it can expand the video and audio features to the same dimension and then multiply them to obtain the initial audio-video fusion features; or it can perform a linear transformation on the video and audio features and then perform a dot product on the transformed video and audio features to obtain the initial audio-video fusion features. Next, the computer device performs random deactivation processing on the initial audio-video fusion features, then pooling processing on the randomly deactivated initial audio-video fusion features to obtain intermediate audio-video fusion features, and then normalizing the intermediate audio-video fusion features to obtain the target audio-video fusion features. Through random deactivation, pooling, and normalization processing, video and audio features can be effectively and accurately fused to obtain highly semantic target audio-video fusion features. Finally, the computer device performs channel attention processing on the target audio-video fusion features to obtain second channel attention weights. Based on the second channel attention weights, feature enhancement is performed on the video features corresponding to the current training video to obtain audio-enhanced video features corresponding to the current training video.

[0164] It is understandable that motion-enhanced video features can also be obtained using the methods described above.

[0165] In one embodiment, the normalization process includes square normalization and L2 normalization. The intermediate audio-video fusion features are subjected to square normalization, and then L2 normalization is applied to the square-normalized intermediate audio-video fusion features.

[0166] In the above embodiments, through a series of fusion, random deactivation, pooling, and normalization processes, video and audio features can be fully fused to obtain accurate target audio-video fusion features. Based on the second channel attention weight obtained by channel attention processing of the target audio-video fusion features, feature enhancement can be performed on the video features to obtain accurate audio-enhanced video features.

[0167] In one embodiment, the initial video-text matching model includes an audio-enhanced video feature and text feature matching network, which includes an audio-video fusion layer, a random deactivation layer, a pooling layer, a normalization layer, a second channel attention layer, and a second similarity calculation layer.

[0168] The audio-video fusion layer is used to fuse video features and audio features; the random deactivation layer is used to randomly deactivate the input data; the pooling layer is used to pool the input data; the normalization layer is used to normalize the input data; the second channel attention layer is used to perform channel attention processing on the input data; and the second similarity calculation layer is used to calculate the similarity between the audio-enhanced video features and the training text features.

[0169] Among them, the audio-enhanced video feature and text feature matching network is used to enhance the video features corresponding to the training video based on the audio features corresponding to the training video, and to calculate the target similarity between the audio-enhanced video features and the training text features corresponding to the training text.

[0170] refer to Figure 4 The audio-enhanced video feature and text feature matching network includes an audio-video fusion layer, a random deactivation layer, a pooling layer, a normalization layer, a second channel attention layer, and a second similarity calculation layer.

[0171] The video and audio features corresponding to the current training video are input into the audio-video fusion layer. The audio features A are then processed using a ReLU-activated fully connected layer (also known as an FC layer). t and video features V t Expand to the same dimension kd o The initial audio-video fusion features are then fused together. These features are then input into a dropout layer for random deactivation, and the output is fed into a pooling layer for pooling to obtain intermediate audio-video fusion features. These intermediate features are then input into a normalization layer for normalization to obtain the target audio-video fusion features. The target features are then input into a second-channel attention layer for channel attention processing to obtain second-channel attention weights. Based on these second-channel attention weights, feature enhancement is performed on the video features corresponding to the current training video to obtain audio-enhanced video features (also known as audio-enhanced video features). Finally, the audio-enhanced video features corresponding to the current training video and the corresponding training text features are input into a second similarity calculation layer for similarity calculation and matching. The second similarity calculation layer outputs a target similarity between the current training video and the corresponding training text.

[0172] In the above embodiments, the initial video-text matching model includes an audio-enhanced video feature and a text feature matching network. The audio-enhanced video features are obtained by performing feature enhancement through a dedicated audio-enhanced video feature and text feature matching network. The similarity between the audio-enhanced video features and the text features is calculated through the audio-enhanced video feature and text feature matching network, which can reduce the risk of confusion with other modalities and help improve the training quality of the model.

[0173] In one specific embodiment, audio features A are processed using a ReLU-activated fully connected layer. t and video features V t Expand to the same dimension kd oThe expanded features are then input into the MFB (Multi-modal Factorized Bilinear Decomposition) module to fuse the video and audio features to obtain the audiovisual fusion feature (AV). t (i.e., target audio and video fusion features).

[0174] AV t =SP(D(φ) T A t ⊙Ψ T V t Formula (3)

[0175] in, and These are two learnable matrix parameters: SP(f,k) represents a sum pooling operation with a kernel and step size of k, and D(·) represents a dropout layer used to prevent overfitting.

[0176] In addition, squared normalization and L2 normalization are introduced to stabilize model training.

[0177]

[0178] Referring to formula (4), first perform a square normalization, and then perform an L2 normalization. This represents square normalization. This indicates L2 normalization. `sign()` represents the sign function.

[0179] AV with audiovisual fusion features t The input is fed into the second-channel attention layer, where channel-level attention operations are performed to obtain the second-channel attention weights. Based on these weights, the video features are enhanced and guided to obtain the audio-enhanced video feature V. t A .

[0180]

[0181] in, and For two linear transformations, δ and σ represent the activation operations of ReLU and sigmoid, respectively.

[0182] In one embodiment, in addition to the motion-enhanced video feature and text feature matching network and the audio-enhanced video feature and text feature matching network, the initial video-text matching model may also include a video feature and text feature matching network. The video feature and text feature matching network is used to calculate the similarity between video features and text features.

[0183] In one embodiment, the video-text matching model training method further includes:

[0184] When audio is missing in the training video, the preset features or the video features corresponding to the training video are obtained as the audio features corresponding to the training video.

[0185] Specifically, some videos may have missing audio. For these videos, alignment can be performed based on preset features or video features. If audio is missing in a training video, the computer can obtain preset features or the video features corresponding to the training video as the audio features for that training video. Preset features are features that are pre-defined and can be set according to actual needs. For example, if audio features are missing in some samples, samples with missing audio features can be aligned using a value of 1.0.

[0186] The purpose of enhancing and guiding video features based on audio features is to increase the weight of the sound source. However, when some videos are missing audio, fixing the audio feature to 1 or replacing the audio feature with a video feature can both achieve the goal of using video features to guide the video feature when the audio feature is missing.

[0187] In the above embodiments, when audio is missing in the training video, the preset features or the video features corresponding to the training video are obtained as the audio features corresponding to the training video. This enables modality alignment when modality is missing, thereby helping to improve the robustness of the model.

[0188] In one embodiment, the similarity set includes at least two target similarities, the current video feature is either the video feature corresponding to the training video in the current training sample pair or the reference enhanced video feature, the current text feature is the training text feature corresponding to the training text in the current training sample pair, the training video includes multiple video frames, and the training text includes multiple text words.

[0189] The process of calculating the target similarity between current video features and current text features includes the following steps;

[0190] Calculate the initial similarity between the current video features and the current text features; the initial similarity includes the sub-similarity between multiple text words in the training text and the same video frame, and the sub-similarity between multiple video frames in the training video and the same text word; obtain text weights based on the current text features, and obtain video weights based on the current video features; for the initial similarity, take the maximum value from multiple sub-similarity values ​​corresponding to the same video frame as the first sub-similarity, and take the maximum value from multiple sub-similarity values ​​corresponding to the same text word as the second sub-similarity, to obtain the first sub-similarity corresponding to each video frame and the second sub-similarity corresponding to each text word; obtain the first similarity based on each first sub-similarity, and obtain the second similarity based on each second sub-similarity; fuse the first similarity and text weights to obtain the first fused data, and fuse the second similarity and video weights to obtain the second fused data; obtain the target similarity between the current video features and the current text features based on the first fused data and the second fused data.

[0191] The similarity set includes at least two target similarities. For example, the similarity set includes one target similarity calculated based on training text features and video features, another target similarity calculated based on training text features and motion-enhanced video features, and yet another target similarity calculated based on training text features and audio-enhanced video features.

[0192] The current training sample pair refers to the training sample pair currently being processed. The current video feature is any one of the video features corresponding to the training video in the current training sample pair and the reference enhanced video features. It can be understood that if the reference enhanced video features include action-enhanced video features and audio-enhanced video features, the current video feature is any one of the video features corresponding to the training video in the current training sample pair, action-enhanced video features, and audio-enhanced video features. The current text feature is the training text feature corresponding to the training text in the current training sample pair.

[0193] As can be understood, the training video consists of multiple video frames, and the training text consists of multiple text words. The features corresponding to the training video include sub-features corresponding to each video frame, and the features corresponding to the training text include sub-features corresponding to each text word.

[0194] Specifically, the calculation method for various target similarities can be the same. Taking the calculation of target similarity between current video features and current text features as an example, the calculation process is explained.

[0195] First, the computer device calculates the initial similarity between the current video features and the current text features. For example, the cosine similarity between the current video features and the current text features can be used as the initial similarity. The initial similarity includes the sub-similarity between multiple text words in the training text and the same video frame, and the sub-similarity between multiple video frames in the training video and the same text word. For example, the initial similarity can be represented by a matrix, where the horizontal axis represents multiple text words in the training text, specifically the sub-similarity between multiple text words and the same video frame, with different rows corresponding to different video frames. The vertical axis represents multiple video frames in the training video, specifically the sub-similarity between multiple video frames and the same text word, with different columns corresponding to different text words. For the initial similarity, the computer device takes the maximum value from the multiple sub-similarity values ​​corresponding to the same video frame as the first sub-similarity, obtains the first sub-similarity for each video frame, and combines these first sub-similarity values ​​into a first similarity. Similarly, for the initial similarity, the computer device takes the maximum value from multiple sub-similarity values ​​corresponding to the same text word as the second sub-similarity, obtaining the second sub-similarity values ​​for each text word, and combining these second sub-similarity values ​​to form the second similarity. For example, the initial similarity, first similarity, and second similarity can be represented by matrices. The horizontal axis of the initial similarity matrix represents the sub-similarity between multiple text words in the training text and the same video frame. The maximum value of each row of the initial similarity matrix is ​​taken to form the first similarity matrix. The vertical axis of the initial similarity matrix represents the sub-similarity between multiple video frames in the training video and the same text word. The maximum value of each column of the initial similarity matrix is ​​taken to form the second similarity matrix.

[0196] The computer device can also calculate weights for current text features and current video features, obtaining text weights based on the current text features and video weights based on the current video features. Text words with important semantic meaning have higher weights in the text weights, and video frames with important semantic meaning have higher weights in the video weights. The first similarity score includes a first sub-similarity score representing the sub-similarity between multiple text words and the same video frame. The text weight includes the text sub-weights corresponding to each text word. The first similarity score and text weights are fused to obtain the first fused data, which emphasizes the sub-similarity between text words with important semantic meaning and video frames. The second similarity score includes a second sub-similarity score representing the sub-similarity between multiple video frames and the same text word. The video weight includes the video sub-weights corresponding to each video frame. The second similarity score and video weights are fused to obtain the second fused data, which emphasizes the sub-similarity between video frames with important semantic meaning and text words.

[0197] Finally, the computer device calculates the target similarity between the current video features and the current text features based on the first and second fused data. For example, the average of the first and second fused data can be used as the target similarity.

[0198] In the above embodiments, the first and second similarities obtained by data processing of the initial similarity integrate the sub-similarity between the most matching video frames and text words. Then, the first similarity is weighted and fused based on the text weights determined by the text features, and the second similarity is weighted and fused based on the video weights determined by the video features. The target similarity obtained based on the weighted fusion result is the weighted similarity, which has higher accuracy and helps to improve the training quality of the model.

[0199] In one specific embodiment, the target similarity (WTI) between video features and training text features can be calculated using the following formula.

[0200] WTI(V i C i ) = (c2v_logits + v2c_logits) / 2

[0201]

[0202]

[0203] Among them, WTI(V) i C i ) represents the video feature V corresponding to the training video in the i-th positive sample. i Training text features C corresponding to the training text i The similarity between the target samples is calculated as follows: `c2v_logits` represents the first similarity corresponding to the i-th positive sample, which can be considered as the similarity in the text-to-video (caption to video, C2V) task. `v2c_logits` represents the second similarity corresponding to the i-th positive sample, which can be considered as the similarity in the video-to-text (video to caption, V2C) task. The C2V task (also known as the T2C task) involves identifying the video described by each of the multiple texts. The V2C task (also known as the V2T task) involves identifying the video description text corresponding to each of the multiple videos.

[0204] f cw,θ and f vw,θ It is a classic MLP (Multilayer Perceptron) structure with SoftMax, f cw,θ Used to calculate the weights of text features, f vw,θUsed to calculate weights for video features. L C and L V represents the number of tokens (i.e., text words) in the training text and the number of frames (video frames) in the training video, respectively. p and q represent the token number in the training text and the frame number in the training video, respectively.

[0205] Indicates that based on the training text features C i and video features V i The matrix calculation yields an initial similarity matrix, where the horizontal axis represents the tokens of the training text and the vertical axis represents the frames of the training video. Indicates to Perform matrix transpose. Indicates to Normalize.

[0206] This means finding the maximum value for each row of the initial similarity matrix. This means finding the maximum value for each column of the initial similarity matrix.

[0207] In one embodiment, the similarity set includes at least two target similarities. The training loss is calculated based on the similarity set corresponding to positive sample pairs and the similarity set corresponding to matching negative sample pairs, including:

[0208] The target category is determined from each similarity category. Based on the target similarity of positive sample pairs and each negative sample pair containing the same training text as the positive sample pairs in the target category, the first similarity weight corresponding to each training sample pair in the target category is obtained. Based on the target similarity of positive sample pairs and each negative sample pair containing the same training video as the positive sample pairs in the target category, the second similarity weight corresponding to each training sample pair in the target category is obtained. The target similarity and the first similarity weight of the same training sample pair in the target category are fused to obtain the first updated similarity corresponding to each training sample pair. The target similarity and the second similarity weight of the same training sample pair in the target category are fused to obtain the second updated similarity corresponding to each training sample pair. Similarity; based on the first updated similarity corresponding to each positive sample pair and each negative sample pair containing the same training video as the positive sample pair, a first loss is obtained; based on the second updated similarity corresponding to each positive sample pair and each negative sample pair containing the same training text as the positive sample pair, a second loss is obtained; based on the first loss and the second loss, the training sub-loss corresponding to the target category is obtained; the next similarity category is obtained as the target category, and the steps of obtaining the first similarity weight corresponding to each training sample pair on the target category based on the target similarity of each positive sample pair and each negative sample pair containing the same training text as the positive sample pair are executed until the training sub-loss corresponding to each similarity category is determined; the training loss is obtained based on the various training sub-losses.

[0209] Specifically, the similarity set includes at least two target similarities, and the training loss includes training sub-losses corresponding to various similarity categories, with each training sub-loss calculated in the same way.

[0210] The computer device can randomly select a similarity category from various similarity categories as the target category. Based on the target similarity of positive sample pairs and each negative sample pair matching the positive sample pair on the target category, it calculates the training sub-loss corresponding to the target category. The computer device can then select the next similarity category from the various similarity categories as a new target category. Based on the target similarity of positive sample pairs and each negative sample pair matching the positive sample pair on the new target category, it calculates the training sub-loss corresponding to the new target category. This process continues until the computer device can calculate the training sub-loss corresponding to each similarity category, and then derive the training loss based on these training sub-losses.

[0211] It is understandable that computer devices can also compute the training sub-losses corresponding to various similarity categories in parallel.

[0212] For any given similarity category, the computer device calculates the first similarity weight for each training sample pair in the target category based on the target similarity between the positive sample pair and each negative sample pair containing the same training text as the positive sample pair. For example, by forming subsets of training sample pairs with each positive sample pair and each negative sample pair containing the same training text as the positive sample pair, the first similarity weight for each training sample pair in the subset is determined based on the difference in target similarity between the positive sample pair and each training sample pair in the subset in the target category. The first similarity weight is generated based on the target similarity of each training sample pair containing the same training text in the target category, and can be considered to incorporate relevant information for the C2V task.

[0213] Similarly, the computer device calculates the second similarity weight for each training sample pair in the target category based on the target similarity between the positive sample pairs and each negative sample pair containing the same training video as the positive sample pairs. The second similarity weight is generated based on the target similarity of each training sample pair containing the same training video in the target category, and can be considered to incorporate relevant information for the v2c task.

[0214] Furthermore, the computer device fuses the target similarity and first similarity weights of the same training sample pair in the target category to obtain the first updated similarity corresponding to each training sample pair. For example, the first updated similarity is obtained by multiplying the target similarity and the first similarity weights. Based on the first updated similarities corresponding to the positive sample pairs and each negative sample pair containing the same training video as the positive sample pairs, the first loss is obtained. Specifically, the first sub-loss can be calculated based on the first updated similarities corresponding to the positive sample pairs and each negative sample pair containing the same training video as the positive sample pairs, resulting in the first sub-loss corresponding to each positive sample pair. The first loss is obtained based on the first sub-loss corresponding to each positive sample pair. The first similarity weights incorporate relevant information for the C2V task, and the first updated similarity calculated based on the target similarity and the first similarity weights can also be considered to incorporate relevant information for the C2V task. The data calculated based on the relevant data of the positive sample pairs and each negative sample pair containing the same training video as the positive sample pairs can be considered to incorporate relevant information for the V2C task. The first loss is calculated based on the first update similarity corresponding to each positive sample pair and each negative sample pair containing the same training video as the positive sample pair. This can be considered as comprehensively considering relevant information for both C2V and V2C tasks. Adjusting the model based on this loss can enable the model to perform well on both C2V and V2C tasks.

[0215] Similarly, the computer device fuses the target similarity and second similarity weights of the same training sample pairs in the target category to obtain the second updated similarity corresponding to each training sample pair. Based on the second updated similarity corresponding to each positive sample pair and each negative sample pair containing the same training text as the positive sample pair, a second loss is obtained. The second similarity weights incorporate relevant information for the v2c task, and the first updated similarity calculated based on the target similarity and the first similarity weights can also be considered to incorporate relevant information for the v2c task. The data calculated based on the relevant data of the positive sample pairs and each negative sample pair containing the same training video as the positive sample pairs can be considered to incorporate relevant information for the c2v task. The first loss calculated based on the first updated similarity corresponding to each positive sample pair and each negative sample pair containing the same training video as the positive sample pairs can be considered to comprehensively consider relevant information for both the c2v and v2c tasks. Adjusting the model based on this loss allows the model to perform well on both c2v and v2c tasks.

[0216] Ultimately, the computer device obtains the training sub-loss corresponding to the target category based on the first loss and the second loss. For example, the average of the first loss and the second loss can be used as the training sub-loss; the weighted average of the first loss and the second loss can be used as the training sub-loss; and so on.

[0217] In the above embodiments, the training sub-loss corresponding to each similarity category is calculated separately, and the accurate training loss can be obtained based on each training sub-loss. Based on the target similarity of positive sample pairs and each negative sample pair containing the same training text as the positive sample pairs in the target category, a first similarity weight is calculated. The target similarity and the first similarity weight are fused to obtain a first updated similarity. The first loss calculated based on the first updated similarity helps the model perform well on both c2v and v2c tasks during model training. Based on the target similarity of positive sample pairs and each negative sample pair containing the same training video as the positive sample pairs in the target category, a second similarity weight is calculated. The target similarity and the second similarity weight are fused to obtain a second updated similarity. The second loss calculated based on the second updated similarity helps the model perform well on both c2v and v2c tasks during model training.

[0218] In one embodiment, based on the target similarity of the positive sample pair and each negative sample pair containing the same training text as the positive sample pair on the target category, a first similarity weight corresponding to each training sample pair on the target category is obtained; based on the target similarity of the positive sample pair and each negative sample pair containing the same training video as the positive sample pair on the target category, a second similarity weight corresponding to each training sample pair on the target category is obtained, including:

[0219] Based on the target similarity of each training sample pair in the target category, a first similarity matrix is ​​obtained. The first dimension of the first similarity matrix represents the target similarity of each training sample pair containing the same training video in the target category. The second dimension of the target similarity matrix represents the target similarity of each training sample pair containing the same training text in the target category. The diagonal of the target similarity matrix represents the target similarity of each positive sample pair in the target category. Based on the current matrix element in the first similarity matrix and the forward matrix element of the current matrix element in the second dimension, the second matrix element corresponding to the current matrix element is generated. Based on the second matrix elements corresponding to each matrix element in the first similarity matrix, a second similarity matrix is ​​generated. The current matrix element and its forward matrix element in the first dimension are used to generate the third matrix element corresponding to the current matrix element. Based on the third matrix elements corresponding to each matrix element in the first similarity matrix, a third similarity matrix is ​​generated. Based on the target similarity of positive sample pairs in the target category, each matrix element in the second similarity matrix is ​​adjusted to obtain the fourth similarity matrix. Based on the target similarity of positive sample pairs in the target category, each matrix element in the third similarity matrix is ​​adjusted to obtain the fifth similarity matrix. The fourth similarity matrix represents the first similarity weight corresponding to each training sample pair in the target category, and the fifth similarity matrix represents the second similarity weight corresponding to each training sample pair in the target category.

[0220] In this context, the preceding matrix element of the current matrix element in the second dimension refers to the matrix element in the first similarity matrix that precedes the current matrix element in the second dimension. The preceding matrix element of the current matrix element in the first dimension refers to the matrix element in the first similarity matrix that precedes the current matrix element in the first dimension.

[0221] Specifically, when calculating the first similarity weight and the second similarity weight, a first similarity matrix is ​​generated based on the target similarity of each training sample pair in the target category. The matrix elements are then processed in an ordered manner to quickly obtain the first similarity weight and the second similarity weight.

[0222] The first dimension of the first similarity matrix generated based on the target similarity of each training sample pair in the target category represents the target similarity of each training sample pair containing the same training video in the target category. The second dimension of the target similarity matrix represents the target similarity of each training sample pair containing the same training text in the target category. The diagonal of the target similarity matrix represents the target similarity of each positive sample pair in the target category.

[0223] For example, suppose the training sample pair set includes three positive sample pairs and six negative sample pairs. The three positive sample pairs are positive sample pair A containing training video 1 and training text 1, positive sample pair B containing training video 2 and training text 2, and positive sample pair C containing training video 3 and training text 3. The six negative sample pairs are obtained by recombining the data from the three positive sample pairs.

[0224] The first similarity matrix is:

[0225] Among them, S 11 S 22 S 33 S represents the target similarity of three positive sample pairs in the target category. 12 S 13 S represents the target similarity in target category among each negative sample pair containing the same training video 1 as the positive sample pair A. 21 S 23 S represents the target similarity in target category among each negative sample pair that, like the positive sample pair B, contains training video 2. 31 S 32 This represents the target similarity in target category for each negative sample pair that also contains training video 3, just like the positive sample pair C.

[0226] When calculating the first similarity weight, a second matrix element corresponding to the current matrix element is generated based on the current matrix element and its forward matrix element in the second dimension of the first similarity matrix. For example, the current matrix element and its forward matrix element in the second dimension can be exponentially processed with the natural constant e as the base, and then added together to obtain the second matrix element corresponding to the current matrix element; or the current matrix element and its forward matrix element in the second dimension can be scaled up proportionally, and then the scaled data can be exponentially processed with the natural constant e as the base, and then added together to obtain the second matrix element corresponding to the current matrix element; and so on. After determining the second matrix element corresponding to each matrix element, the various second matrix elements are combined to form the second similarity matrix. Based on the target similarity of the positive sample pairs in the target category, the matrix elements in the second similarity matrix are adjusted to obtain the fourth similarity matrix. For example, the target similarity of the positive sample pairs in the target category is divided by each matrix element to obtain the fourth similarity matrix. The fourth similarity matrix represents the first similarity weight corresponding to each training sample pair in the target category.

[0227] For example, the first similarity matrix is The first dimension of the first similarity matrix is ​​rows, and the second dimension of the matrix is ​​columns.

[0228] The second similarity matrix is:

[0229]

[0230] The fourth similarity matrix is:

[0231]

[0232] Here, exp() represents an exponential function with the natural constant e as its base.

[0233] The first row of the fourth similarity matrix contains the first similarity weights of positive sample pair A, negative sample pair containing training video 1 and training text 2, and negative sample pair containing training video 1 and training text 3 in the target category. The second row contains the first similarity weights of negative sample pair containing training video 2 and training text 1, positive sample pair B, and negative sample pair containing training video 2 and training text 3 in the target category. The third row contains the first similarity weights of negative sample pair containing training video 3 and training text 1, negative sample pair containing training video 3 and training text 2, and positive sample pair C in the target category.

[0234] Similar to calculating the first similarity weight, when calculating the second similarity weight, a third matrix element corresponding to the current matrix element is generated based on the current matrix element and its forward matrix element in the first dimension of the first similarity matrix. A third similarity matrix is ​​then generated based on the third matrix elements corresponding to each matrix element in the first similarity matrix. Based on the target similarity of the positive sample pairs in the target category, the matrix elements in the third similarity matrix are adjusted to obtain the fifth similarity matrix. The fifth similarity matrix represents the second similarity weight corresponding to each training sample pair in the target category.

[0235] For example, the first similarity matrix is The first dimension of the first similarity matrix is ​​rows, and the second dimension of the matrix is ​​columns.

[0236] The third similarity matrix is:

[0237]

[0238] The fifth similarity matrix is:

[0239]

[0240] The first row of the fifth similarity matrix contains the second similarity weights for positive sample pair A, negative sample pair containing training video 1 and training text 2, and negative sample pair containing training video 1 and training text 3, respectively, on the target category. The second row contains the second similarity weights for negative sample pair containing training video 2 and training text 1, positive sample pair B, and negative sample pair containing training video 2 and training text 3, respectively, on the target category. The third row contains the second similarity weights for negative sample pair containing training video 3 and training text 1, negative sample pair containing training video 3 and training text 2, and positive sample pair C, respectively, on the target category.

[0241] In the above embodiments, a first similarity matrix is ​​obtained based on the target similarity of each training sample pair in the target category. A second similarity matrix is ​​generated based on the matrix elements in the second dimension of the first similarity matrix. Based on the target similarity of positive sample pairs in the target category, the matrix elements in the second similarity matrix are adjusted to obtain a fourth similarity matrix. The fourth similarity matrix incorporates relevant information for the C2V task, which helps improve the model's ability to handle C2V tasks during model training. A third similarity matrix is ​​generated based on the matrix elements in the first dimension of the first similarity matrix. Based on the target similarity of positive sample pairs in the target category, the matrix elements in the third similarity matrix are adjusted to obtain a fifth similarity matrix. The fifth similarity matrix incorporates relevant information for the V2C task, which helps improve the model's ability to handle V2C tasks during model training.

[0242] In one embodiment, a first loss is obtained based on the first updated similarity corresponding to each positive sample pair and each negative sample pair containing the same training video as the positive sample pair, and a second loss is obtained based on the second updated similarity corresponding to each positive sample pair and each negative sample pair containing the same training text as the positive sample pair, including:

[0243] The first update similarity of each positive sample pair and each negative sample pair containing the same training video as the positive sample pair is fused to obtain the first similarity statistics of each positive sample pair. Based on the first update similarity and the first similarity statistics of the same positive sample pair, the first sub-loss of each positive sample pair is obtained. The first loss is obtained based on each first sub-loss. The second update similarity of each positive sample pair and each negative sample pair containing the same training text as the positive sample pair is fused to obtain the second similarity statistics of each positive sample pair. Based on the second update similarity and the second similarity statistics of the same positive sample pair, the second sub-loss of each positive sample pair is obtained. The second loss is obtained based on each second sub-loss.

[0244] Specifically, when calculating the first loss, the first updated similarity corresponding to each positive sample pair and each negative sample pair containing the same training video as the positive sample pair is fused to obtain the first similarity statistic for each positive sample pair. For example, the sum of the first updated similarity corresponding to each positive sample pair and each negative sample pair containing the same training video as the positive sample pair is used as the first similarity statistic. Based on the first updated similarity and the first similarity statistic corresponding to the same positive sample pair, the first sub-loss corresponding to each positive sample pair is obtained. For example, the ratio of the first updated similarity to the first similarity statistic corresponding to the positive sample pair is used as the first sub-loss. Finally, the first loss is obtained based on each first sub-loss. For example, the average of each first sub-loss is calculated as the first loss.

[0245] Similar to calculating the first loss, when calculating the second loss, the second update similarity corresponding to each positive sample pair and each negative sample pair containing the same training text as the positive sample pair is fused to obtain the second similarity statistics corresponding to each positive sample pair. Based on the second update similarity and the second similarity statistics corresponding to the same positive sample pair, the second sub-loss corresponding to each positive sample pair is obtained. The second loss is obtained based on each second sub-loss.

[0246] In the above embodiments, the first updated similarity corresponding to each positive sample pair and each negative sample pair containing the same training video as the positive sample pair is fused. Based on the first updated similarity and the first similarity statistics corresponding to the same positive sample pair, a first sub-loss corresponding to each positive sample pair is obtained. A first loss is obtained based on each first sub-loss, reflecting a normalization concept. The training sub-loss generated based on this first loss helps improve the model training quality. Similarly, the second updated similarity corresponding to each positive sample pair and each negative sample pair containing the same training text as the positive sample pair is fused. Based on the second updated similarity and the second similarity statistics corresponding to the same positive sample pair, a second sub-loss corresponding to each positive sample pair is obtained. A second loss is obtained based on each second sub-loss, reflecting a normalization concept. The training sub-loss generated based on this second loss helps improve the model training quality.

[0247] In a specific embodiment, for the target similarity calculated based on video features and training text features, the training sub-loss corresponding to the similarity category can be calculated using the following formula (7).

[0248]

[0249]

[0250]

[0251] Among them, WTI(V)i C i ) represents the target similarity corresponding to the training sample pair consisting of training video i and training text j, which is calculated based on video feature V and training text feature C. This represents the first similarity weight corresponding to the training sample pair consisting of training video i and training text j. Pr represents the second similarity weight corresponding to the training sample pair consisting of training video i and training text j. v2c This represents the fourth similarity matrix, which incorporates relevant information from the C2V task. (Pr) c2v This represents the fifth similarity matrix, which incorporates relevant information from the v2c task. (Pr) v2c and Pr c2v It can also be called a priority matrix.

[0252] L v2c Indicates the first loss, L c2v This indicates the second loss. This represents the training sub-loss corresponding to the similarity category to which the target similarity is calculated based on video features and training text features. It can also represent the loss information corresponding to the video feature and text feature matching network.

[0253] `temp` represents the temperature hyperparameter, used to smooth the gradient; it can be set according to actual needs. `L` represents the scaling parameter, which can also be set according to actual needs. `B` represents the number of positive sample pairs.

[0254] Based on the motion-enhanced video features and training text features, WTI similarity matching can be performed between the motion-enhanced video features and the training text features to obtain the similarity matrix Sim. c_m =WTI(C,V M Similar to formula (7), the loss function can be obtained: Loss m =L m2c +L c2m , This represents the training sub-loss corresponding to the similarity category to which the target similarity is calculated based on action-enhanced video features and training text features. It can also represent the loss information corresponding to the action-enhanced video feature and text feature matching network.

[0255] Based on the audio-enhanced video features and training text features, WTI similarity matching can be performed between the action-enhanced video features and the training text features to obtain the similarity matrix Sim. c_a =WTI(C,V A Similar to formula (7), the loss function can be obtained: Loss a =L a2c +Lc2a , This represents the training sub-loss corresponding to the similarity category to which the target similarity is calculated based on audio-enhanced video features and training text features. It can also represent the loss information corresponding to the audio-enhanced video feature and text feature matching network.

[0256] In one specific embodiment, reference Figure 5 This section uses video features and text features as examples to illustrate the similarity calculation process and the loss calculation process. The video features corresponding to the training video can be obtained by inputting the training video into a video encoder, and the training text features corresponding to the training text can be obtained by inputting the training text into a text encoder. When calculating the target similarity between video features and training text features, the initial similarity between the video features and training text features is first calculated (see reference...). Then, perform a max operation on the initial similarity, specifically by taking the maximum value of each row of the initial similarity to obtain a 3*1 matrix (see reference). The initial similarity is then maximized column-wise to obtain a 1x3 matrix (see reference). Also, calculate the text weights corresponding to the training texts (see reference). ), calculate the video weights corresponding to the training videos (refer to...). Finally, the 3*1 matrix and text weights are weighted and fused (refer to c2v_logits), and the 1*3 matrix and video weights are weighted and fused (refer to v2c_logits). The target similarity between video features and training text features is obtained based on the weighted fusion results.

[0257] Since there are multiple training sample pairs, each training sample pair can have its corresponding target similarity calculated based on its own video features and training text features. The target similarity values ​​of each training sample pair form a target similarity matrix, with the diagonal of the target similarity matrix representing the target similarity values ​​of each positive sample pair. Based on the target similarity values ​​of the positive sample pairs, a softmax operation is performed on the target similarity matrix column-wise to obtain the similarity weight matrix (refer to Pr). v2c Based on the target similarity corresponding to the positive sample pairs, the similarity weight matrix is ​​obtained by performing softmax calculation on the target similarity matrix row by row (refer to Pr). c2v The target similarity matrix is ​​multiplied element-wise by the two similarity weight matrices to obtain two updated similarity matrices (see reference). and Based on the target similarity corresponding to the positive sample pairs, the loss is obtained by performing softmax calculation on the updated similarity matrix (refer to L). v2c and L c2v(i.e., the first loss and the second loss). Finally, the training sub-loss is obtained based on the first loss and the second loss (see reference). ).

[0258] In one embodiment, the initial video-text matching model includes a video feature and text feature matching network and a reference enhanced video feature and text feature matching network. The reference enhanced video feature and text feature matching network includes at least one of an action enhanced video feature and text feature matching network and an audio enhanced video feature and text feature matching network. The training loss includes the training sub-losses corresponding to each network.

[0259] The model parameters of the initial video-text matching model are adjusted based on the training loss until the convergence condition is met, resulting in the target video-text matching model, including:

[0260] Adjust the model parameters of the corresponding networks in the initial video-text matching model based on each training sub-loss until the convergence conditions of each network are met, thus obtaining the target video-text matching model.

[0261] Specifically, the model can be trained using an ensemble training model. Ensemble training refers to training each network in the initial video-text matching model separately, with each network training based on its own loss information. Once all networks have completed training, the trained networks are combined to form the target video-text matching model.

[0262] The training loss, calculated based on the similarity sets corresponding to positive sample pairs and the similarity sets corresponding to matched negative sample pairs, includes training sub-losses for various networks. The computer can adjust the model parameters of the corresponding networks in the initial video-text matching model using each training sub-loss until the convergence conditions for each network are met, thus obtaining the target video-text matching model. Specifically, the model parameters of the video and text feature matching network can be adjusted based on the training sub-losses corresponding to the video and text feature matching network until the convergence conditions for that network are met, resulting in a trained video and text feature matching network. If an action-enhanced video and text feature matching network exists, its model parameters are adjusted based on the training sub-losses until the convergence conditions for that network are met, resulting in a trained action-enhanced video and text feature matching network. If an audio-enhanced video feature and text feature matching network exists, adjust the model parameters of the audio-enhanced video feature and text feature matching network based on the training sub-loss corresponding to the audio-enhanced video feature and text feature matching network until the convergence condition corresponding to the audio-enhanced video feature and text feature matching network is met, thus obtaining the trained audio-enhanced video feature and text feature matching network.

[0263] When applying the model, the prediction results of various networks in the target video-text matching model are integrated to obtain the final prediction result. For example, the maximum similarity among the outputs of the three networks in the target video-text matching model is taken as the target value. If the target value is greater than a preset threshold, the final matching result is determined to be a successful match; the similarity of the outputs of the three networks in the target video-text matching model is obtained, and the average value is calculated as the target value. If the target value is greater than a preset threshold, the final matching result is determined to be a successful match; and so on.

[0264] In the above embodiments, by adjusting the model parameters of the corresponding networks in the initial video text matching model based on each training sub-loss, each network in the video text matching model can reach its own optimal state, ensuring that the final target video text matching model has high prediction accuracy.

[0265] In one embodiment, the training loss includes a first training sub-loss corresponding to each similarity category. The model parameters of the initial video-text matching model are adjusted based on the training loss until a convergence condition is met, resulting in the target video-text matching model, including:

[0266] The minimum value among the first training sub-losses is obtained as the first target sub-loss. Based on the difference between the first target sub-loss and each remaining training sub-loss, the loss contribution of each remaining training sub-loss is obtained. Based on the loss contribution of each remaining training sub-loss, the loss weight of each remaining training sub-loss is obtained. Based on the loss weight of each training sub-loss, the training sub-losses are fused to obtain the first target loss. The loss weight of the first target sub-loss is a preset weight. The model parameters of the initial video text matching model are adjusted based on the first target loss until the convergence condition is met, thus obtaining the target video text matching model.

[0267] The training loss includes the first training sub-loss corresponding to each similarity category. If the initial video-text matching model includes multiple networks, then the first training sub-loss corresponding to a certain similarity category is the training sub-loss corresponding to that network.

[0268] Specifically, the training mode of the model can adopt the end-to-end training mode (also known as the E2E training mode). The final loss function of the end-to-end training mode can adopt the multimodal balanced loss function. The various first training sub-losses are fused according to their contribution to obtain the final target loss. The model parameters of the entire model are adjusted based on the final target loss until the convergence condition is met, and the target video text matching model is obtained.

[0269] First, the computer device selects the minimum value from each of the first training sub-losses as the first target sub-loss, and uses the other first training sub-losses as the remaining training sub-losses. Since the first target sub-loss has the smallest value, it can be considered the sub-loss with the largest contribution. Next, the computer device calculates the loss contribution of each remaining training sub-loss based on the difference between the first target sub-loss and each remaining training sub-loss. The loss contribution decreases as the value of the remaining training sub-loss increases; the larger the remaining training sub-loss, the less accurate the corresponding network, and the smaller its corresponding loss contribution. Then, the computer device calculates the loss weight of each remaining training sub-loss based on its respective loss contribution. The loss weight increases as the loss contribution increases; the larger the loss contribution of a training sub-loss, the larger its corresponding loss weight. Finally, the computer device fuses the training sub-losses based on their respective loss weights to obtain the first target loss. The loss weight corresponding to the first target sub-loss is a preset weight. In one embodiment, the preset weight is greater than the loss weights corresponding to each remaining training sub-loss.

[0270] The first objective loss can be considered a comprehensive loss of the model. The computer equipment uses backpropagation based on the first objective loss to adjust the model parameters of the initial video-text matching model, and performs end-to-end training on the model until the convergence condition is met, thus obtaining the target video-text matching model.

[0271] In one embodiment, three modes are designed as i,j,k∈{v,a,m}, and the contribution rate between each mode is defined as... Where i represents the modality with the largest contribution, specifically the modality with the smallest loss. v represents the modality corresponding to the video feature, or it can also represent the similarity category corresponding to the video feature; a represents the modality corresponding to the audio-enhanced video feature, or it can also represent the similarity category corresponding to the audio-enhanced video feature; m represents the modality corresponding to the action-enhanced video feature, or it can also represent the similarity category corresponding to the action-enhanced video feature.

[0272] First, the mode with the minimum loss is selected as i. This mode must satisfy the following condition: The other modes are designated as j and k, respectively. Then, referring to the following formula (8), the contribution of each mode can be dynamically adjusted and balanced using the contribution rate to obtain the balance coefficient (i.e., loss weight) of each mode. Referring to the following formula (9), the final first target loss is calculated using the balance coefficient.

[0273]

[0274] Loss1=theta i Loss i +theta j Loss j +theta k Loss k Formula (9)

[0275] Among them, theta i Theta represents the balance coefficient corresponding to i. j Theta represents the balance coefficient corresponding to j. k This represents the balance coefficient corresponding to k. tanh() represents the hyperbolic tangent function. α represents an adjustment parameter, which can be set according to actual needs, for example, set to 1.0. Loss i Let i represent the training sub-loss, and Loss be the loss value. j Let j represent the training sub-loss, and Loss. k Let k represent the training sub-loss, and Loss1 represent the first objective loss.

[0276] In the above embodiments, the first target loss is obtained by weighted fusion of the first training sub-losses based on their contribution to the loss of each first training sub-loss. The model is then trained end-to-end based on the first target loss, which can effectively improve the training quality of the model and ensure that the final target video text matching model has high prediction accuracy.

[0277] In one embodiment, the training loss includes a second training sub-loss corresponding to each positive sample pair in each similarity category. The model parameters of the initial video-text matching model are adjusted based on the training loss until the convergence condition is met, resulting in the target video-text matching model, including:

[0278] For the same positive sample pair, the minimum value is obtained from each second training sub-loss as the second target sub-loss, and the second target sub-loss corresponding to each positive sample pair is obtained. Based on the statistical value of each second target sub-loss, the second target loss is obtained. Based on the second target loss, the model parameters of the initial video text matching model are adjusted until the convergence condition is met, and the target video text matching model is obtained.

[0279] The training loss includes the second training sub-loss for each positive sample pair across each similarity category. For example, refer to Loss... v Loss a and Loss m Each positive sample pair has its own corresponding loss. v Loss a and Loss m Loss v Loss a and Loss m It is the second training sub-loss corresponding to each similarity category for positive sample pairs.

[0280] Specifically, when using an end-to-end training model, the final loss function can be the optimal loss function. The optimal loss function is obtained by integrating the optimal loss of the same positive sample pair on each similarity category as the final loss corresponding to the positive sample pair, and then obtaining the final loss of the model based on the final loss corresponding to each positive sample pair.

[0281] For the same positive sample pair, the computer device can obtain the minimum value from each of the second training sub-losses as the second target sub-loss. For each positive sample pair, the computer device can finally obtain the second target sub-loss corresponding to each positive sample pair. Then, the computer device obtains the second target loss based on the statistical values ​​of each second target sub-loss, for example, by calculating the average value of each second target sub-loss as the second target loss; calculating the median value of each second target sub-loss as the second target loss; and so on.

[0282] The second objective loss can also be considered as a comprehensive loss of the model. The computer device uses backpropagation based on the second objective loss to adjust the model parameters of the initial video-text matching model, and performs end-to-end training on the model until the convergence condition is met, thus obtaining the target video-text matching model.

[0283] In one embodiment, the final target loss is calculated using the balance coefficient with reference to the following formula (10).

[0284]

[0285] Loss corresponding to the same positive sample pair v Loss m Loss a Find the minimum value as the loss, obtain the loss for each positive sample pair, and calculate the average of all losses as the second objective loss.

[0286] In the above embodiments, for each positive sample pair, the optimal second training sub-loss of each branch is taken as the final training sub-loss to calculate the final second target loss. The model is trained end-to-end based on the second target loss, which can effectively improve the training quality of the model and ensure that the final target video text matching model has high prediction accuracy.

[0287] In one embodiment, the video-text matching model training method further includes:

[0288] The video features, reference features, and test text features corresponding to the test videos and test texts in the test sample pair set are input into the target video-text matching model to obtain the similarity set corresponding to each test sample pair in the test sample pair set. Based on the similarity set corresponding to each test sample pair, a similarity test matrix corresponding to each similarity category is generated. Based on the similarity test matrix corresponding to each similarity category, the predicted matching sub-ranking corresponding to each test sample pair in each similarity category is determined. Based on the predicted matching sub-rankings corresponding to the same test sample pair, the predicted matching ranking is determined to obtain the predicted matching ranking corresponding to each test sample pair. Based on the predicted matching ranking corresponding to the matching sample pairs in each test sample pair, the prediction accuracy corresponding to the target video-text matching model is determined.

[0289] The test sample pair set includes multiple test sample pairs. The test sample pair set includes matching test sample pairs and non-matching test sample pairs. A matching test sample pair means that the test video and test text in the test sample pair match, while a non-matching test sample pair means that the test video and test text in the test sample pair do not match.

[0290] Specifically, after training the target video text matching model, the model can be further tested to evaluate its performance. Only after passing the tests can the model be deployed. During model testing, a multimodal post-fusion processing strategy can be employed. This strategy involves first calculating the similarity for each modality, and then integrating these similarities to obtain the final matching result.

[0291] During testing, the video features, reference features, and test text features corresponding to the test video and test text in the test sample pair set need to be input into the target video-text matching model. After data processing by the model, a similarity set corresponding to each test sample pair in the test sample pair set can be obtained. A similarity set includes the target similarity corresponding to each similarity category. The computer device can generate a similarity test matrix corresponding to each similarity category based on the similarity set corresponding to each test sample pair. A similarity test matrix includes the target similarity corresponding to each test sample pair in the same similarity category. Furthermore, the computer device can determine the predicted matching sub-ranking of each test sample pair in each similarity category based on the similarity test matrix corresponding to each similarity category. Specifically, the values ​​of the matrix elements in a certain similarity test matrix can be sorted from largest to smallest by row or column to obtain the predicted matching sub-ranking of each test sample pair in the corresponding similarity category. The computer equipment then determines the predicted matching ranking of each test sample pair based on the predicted matching sub-rankings corresponding to each test sample pair. For example, the highest ranking among the predicted matching sub-rankings can be used as the predicted matching ranking; the average ranking can be used as the predicted matching ranking; and so on. Finally, the computer equipment determines the prediction accuracy of the target video text matching model based on the predicted matching rankings of the matched sample pairs in each test sample pair.

[0292] If the prediction accuracy is greater than the preset accuracy, the target video text matching model passes the test and can be deployed. If the prediction accuracy is less than or equal to the preset accuracy, the target video text matching model fails the test. If the target video text matching model fails the test, a new set of training samples can be obtained to further train the target video text matching model and readjust the model parameters. The preset accuracy refers to a pre-set accuracy that can be set according to actual needs.

[0293] In one embodiment, the prediction accuracy of the target video text matching model is determined based on at least one of the following data: R@1, R@5, R@10, Median Score, or Mean Score, corresponding to the predicted matching ranking of the matched sample pairs in each test sample pair, to evaluate the model performance. R@1 is obtained based on the proportion of the first-ranked matched sample pair among all matched sample pairs; R@5 is obtained based on the proportion of the top five ranked matched sample pairs; R@10 is obtained based on the proportion of the top ten ranked matched sample pairs; Median Score is obtained based on the proportion of the middle-ranked matched sample pairs; and Mean Score is obtained based on the average of the predicted matching rankings of each matched sample pair. Higher values ​​for R@1, R@5, and R@10 indicate higher model prediction accuracy and better model performance. Lower values ​​for Median Score and Mean Score indicate higher model prediction accuracy and better model performance.

[0294] In one embodiment, a regularization term can also be introduced to assist model testing. Video text features can be extracted from the video. The target similarity between the video text features of the test video and the test text features of the test text in the test sample pair is calculated, resulting in the target similarity for each test sample pair in the new similarity category. Based on the target similarity for each test sample pair in the new similarity category, the predicted matching sub-ranking for each test sample pair in the new similarity category is determined. The predicted matching ranking is determined based on the predicted matching sub-rankings corresponding to the same test sample pair, resulting in the predicted matching ranking for each test sample pair. Based on the predicted matching ranking of the matched sample pairs in each test sample pair, the prediction accuracy of the target video text matching model is determined.

[0295] refer to Figure 6The video-to-text matching model comprises four branches. These branches are used to calculate the similarity between video features corresponding to the video and text features corresponding to the text in the test sample pair, the similarity between video and text features, the similarity between motion-guided video features (i.e., motion-enhanced video features) and text features, and the similarity between audio-guided video features (i.e., audio-enhanced video features) and text features, respectively. The video features, video and text features, motion-guided video features, audio-guided video features, and test text features corresponding to the test text in the test sample pair set are input into the target video-to-text matching model, and the similarity matrix for each branch is calculated. The similarity matrices are sorted by row or column to obtain the predicted matching sub-ranking for each test sample pair. Then, the optimal ranking score of each branch is taken as the final predicted matching ranking for each test sample pair. The model performance is evaluated using R@1, R@5, R@10, Median Score, and Mean Score statistics for the final predicted matching ranking of each matching sample pair. (Reference) Figure 6 The predicted matching sub-rankings of test sample pair 1 in each branch are 7, 6, 0, and 3, respectively. The best ranking among the predicted matching sub-rankings is taken as the final predicted matching ranking. The final predicted matching ranking of test sample pair 1 is 0.

[0296] refer to Figure 7 When sorting the similarity matrix, if the horizontal axis of the similarity matrix represents test sample pairs containing the same video and the vertical axis represents test sample pairs containing the same text, then the ranking obtained by sorting the similarity matrix by row is the ranking result on the c2v task, and the ranking obtained by sorting the similarity matrix by column is the ranking result on the v2c task.

[0297] It is understandable that determining the predicted match sub-ranking of a test sample pair on a similarity category can include ranking results on at least one of the c2v or v2c tasks.

[0298] In the above embodiments, the trained video-text matching model is tested based on a set of test samples. The model testing employs a multimodal post-fusion strategy, taking the optimal prediction result from each branch as the final prediction result, which helps ensure the effectiveness of the model testing. Correspondingly, the multimodal post-fusion strategy can also be used when applying the model, which can significantly improve the model's prediction performance.

[0299] In one embodiment, such as Figure 8 As shown, a video text matching method is provided. Taking the application of this method to a computer device as an example, the computer device can be the one described above. Figure 1 Terminal 102 or server 104 in the reference. Figure 8The video text matching method includes the following steps:

[0300] Step S802: Obtain the video features to be matched and the reference features to be matched corresponding to the video to be matched, and obtain the text features to be matched corresponding to the text to be matched; the reference features to be matched include at least one of the audio features and motion features corresponding to the video to be matched.

[0301] In this context, "video to be matched" and "text to be matched" refer to the video and text for which a match is to be determined. "Video features to be matched" refers to the video features corresponding to the video to be matched, and "reference features to be matched" refers to the reference features corresponding to the video to be matched. The process of obtaining the video features and reference features to be matched can refer to the process of obtaining the video features and reference features corresponding to the training video. Similarly, "text features to be matched" refers to the text features corresponding to the text to be matched, and the process of obtaining the text features to be matched can refer to the process of obtaining the training text features corresponding to the training text.

[0302] Step S804: Based on the reference features to be matched, feature enhancement is performed on the video features to be matched to obtain the reference enhanced video features corresponding to the video to be matched; the reference enhanced video features include at least one of motion enhanced video features and audio enhanced video features.

[0303] Specifically, the computer device can acquire the video features to be matched and the reference features to be matched corresponding to the video to be matched. Based on the reference features to be matched, the video features to be matched are enhanced to obtain the reference enhanced video features corresponding to the video to be matched. If the reference features to be matched include action features, then the video features to be matched are enhanced based on the action features to obtain action-enhanced video features, and the reference enhanced video features include the action-enhanced video features. If the reference features to be matched include audio features, then the video features to be matched are enhanced based on the audio features to obtain audio-enhanced video features, and the reference enhanced video features include the audio-enhanced video features.

[0304] It is understood that the specific process of feature enhancement can refer to the content of the various relevant embodiments of the aforementioned image generation model training method. For example, intramodal attention processing is performed on the video features and action features to be matched, respectively, to obtain self-attention video features and self-attention action features; intermodal attention processing is performed on the video features to be matched and self-attention action features to obtain cross-attention video features; intermodal attention processing is performed on the action features and self-attention video features to obtain cross-attention action features; the cross-attention action features and cross-attention video features are fused to obtain action video fusion features; channel attention processing is performed on the action video fusion features to obtain first channel attention weights; based on the first channel attention weights, feature enhancement is performed on the video features to be matched to obtain action-enhanced video features.

[0305] Step S806: Calculate the similarity between the text features to be matched and the video features to be matched and the reference enhanced video features, respectively, to obtain the similarity set between the video and the text to be matched.

[0306] The similarity set includes the similarity between the text features to be matched and the video features to be matched, as well as the similarity between the text features to be matched and the reference enhanced video features.

[0307] It is understood that the specific process of similarity calculation can refer to the content of the various relevant embodiments of the aforementioned image generation model training method. For example, the initial similarity between the text features to be matched and the text features to be matched is calculated; the initial similarity includes the sub-similarity between multiple text words in the text to be matched and the same video frame, and the sub-similarity between multiple video frames in the video to be matched and the same text word; text weights are obtained based on the text features to be matched, and video weights are obtained based on the video features to be matched; for the initial similarity, the maximum value is obtained from multiple sub-similarity corresponding to the same video frame as the first sub-similarity, and the maximum value is obtained from multiple sub-similarity corresponding to the same text word as the second sub-similarity, thus obtaining the first sub-similarity corresponding to each video frame and the second sub-similarity corresponding to each text word; a first similarity is obtained based on each first sub-similarity, and a second similarity is obtained based on each second sub-similarity; the first similarity and text weights are fused to obtain the first fused data, and the second similarity and video weights are fused to obtain the second fused data; based on the first fused data and the second fused data, the target similarity between the video features to be matched and the text features to be matched is obtained.

[0308] Step S808: Based on the similarity sets corresponding to the video to be matched and the text to be matched, determine the matching result between the video to be matched and the text to be matched.

[0309] Specifically, the computer device can acquire the video features to be matched and the reference features to be matched corresponding to the video to be matched, either locally or from other devices, and acquire the text features to be matched corresponding to the text to be matched. Then, based on the reference features to be matched, the video features to be matched are enhanced to obtain the reference enhanced video features corresponding to the video to be matched. For example, models for generating motion-enhanced video features and audio-enhanced video features can be trained separately. The video features and motion features of the video to be matched are input into the model for generating motion-enhanced video features to obtain motion-enhanced video features, and the video features and audio features of the video to be matched are input into the model for generating audio-enhanced video features to obtain audio-enhanced video features. Furthermore, the computer device calculates the similarity between the text features to be matched and the video features to be matched and the reference enhanced video features, respectively, to obtain the similarity set between the video to be matched and the text to be matched. For example, the cosine similarity between the text features to be matched and the video features to be matched and the reference enhanced video features, respectively, is calculated to obtain the similarity set. Finally, the computer device determines the matching result between the video and the text to be matched based on the similarity set corresponding to them. For example, if all similarities in the similarity set are greater than a preset threshold, the matching result between the video and the text is determined to be a successful match; otherwise, the matching result is a failed match. If at least two similarities in the similarity set are greater than the preset threshold, the matching result between the video and the text is determined to be a successful match; otherwise, the matching result is a failed match. The maximum value is taken from the similarity set as the target value. If the target value is greater than the preset threshold, the matching result is determined to be a successful match; otherwise, the matching result is a failed match. And so on.

[0310] It is understood that the specific processes of feature enhancement and similarity calculation can refer to the content of the various relevant embodiments of the aforementioned image generation model training method. The methods described in the various relevant embodiments of the aforementioned video text matching model training method can be implemented not only by a single model but also by multiple models. For example, models for generating action-enhanced video features and audio-enhanced video features can be trained separately. The methods described in the various relevant embodiments of the aforementioned video text matching model training method can also be implemented independently of a model, by designing corresponding algorithms or formulas.

[0311] In one embodiment, there can be multiple instances of either the video to be matched or the text to be matched, and each pair of videos and texts can be combined to obtain multiple video-text pairs. Similar to the testing model, when determining the matching result, a similarity prediction matrix can be generated for each similarity category based on the similarity set corresponding to each video-text pair. Based on the similarity prediction matrix for each similarity category, the predicted matching sub-ranking for each video-text pair in each similarity category is determined. The predicted matching ranking is then determined based on the predicted matching sub-rankings for the same video-text pair, resulting in the predicted matching ranking for each video-text pair. Finally, the matching result for each video-text pair is determined based on its predicted matching ranking. For example, the matching result for video-text pairs with a predicted matching ranking in the top 10% can be considered a successful match.

[0312] The video text matching method of this application can be applied in data retrieval and data recommendation scenarios. For example, in a video retrieval scenario, the user's search query can be used as the text to be matched, and the videos in the candidate video set can be used as the videos to be matched. Based on the target video text matching model, the predicted matching ranking of each video text pair is determined, and the candidate videos in the top 10 video text pairs with the predicted matching ranking are used as the target videos. Each target video is then used as the video search result corresponding to the search query.

[0313] The aforementioned video-to-text matching method obtains the video features and reference features corresponding to the video to be matched, and then obtains the text features corresponding to the text to be matched. The reference features include at least one of the audio features and motion features corresponding to the video to be matched. Based on the reference features, the video features to be matched are enhanced to obtain the reference enhanced video features corresponding to the video to be matched. The reference enhanced video features include at least one of the motion enhanced video features and audio enhanced video features. The similarity between the text features to be matched and the video features and the reference enhanced video features is calculated to obtain the similarity set between the video and the text to be matched. Based on the similarity set between the video and the text to be matched, the matching result between the video and the text to be matched is determined. In this way, video features can provide image information of the video, audio features can provide sound information of the video, and motion features can provide motion information of the video. By determining the matching result between the video and the text to be matched based on the video features, reference features, and text features corresponding to the video to be matched, the rich modal information in the video can be used to improve the understanding of the video content, thereby improving the matching accuracy. Furthermore, feature enhancement and guidance based on audio or motion features can highlight important information in the video, further improving the understanding of the video content. Calculating similarity between the video features and the reference enhanced video features, and then determining the matching result based on the calculated similarity set, can further improve matching accuracy.

[0314] In one embodiment, such as Figure 9 As shown, steps S804 and S806 include:

[0315] Step S902: Input the features of the video to be matched, the reference features to be matched, and the features of the text to be matched into the target video-text matching model to obtain the similarity set corresponding to the video to be matched and the text to be matched.

[0316] The training process of the target video text matching model includes the following steps:

[0317] Obtain a training sample pair set; the training sample pairs in the training sample pair set include positive sample pairs and negative sample pairs that match the positive sample pairs. The training sample pairs include training videos and training text, and the training videos and training texts in the positive sample pairs match each other; input the video features, reference features, and training text features corresponding to the training videos in the training sample pair set into the initial video-text matching model; the reference features include at least one of the audio features and motion features corresponding to the training videos; based on the reference features corresponding to the same training video, perform feature enhancement on the corresponding video features to obtain the reference enhanced video features corresponding to the training videos; the reference enhanced video features include at least one of the motion enhanced video features and audio enhanced video features; for the same training sample pair, calculate the similarity between the training text features corresponding to the training text and the video features and reference enhanced video features corresponding to the training videos, respectively, to obtain the similarity sets corresponding to each training sample pair; calculate the training loss based on the similarity sets corresponding to the positive sample pairs and the similarity sets corresponding to the matching negative sample pairs, and adjust the model parameters of the initial video-text matching model based on the training loss until the convergence condition is met to obtain the target video-text matching model.

[0318] It is understood that the training process of the target video text matching model can refer to the content of the various embodiments in the aforementioned video text matching model training method, and will not be repeated here.

[0319] Specifically, the computer device can acquire the video features and reference features corresponding to the video to be matched, and the text features corresponding to the text to be matched. These features are then input into a target video-text matching model. After data processing by the model, a similarity set between the video and the text to be matched is obtained. Within the target video-text matching model, feature enhancement is performed on the video features based on the reference features, resulting in enhanced reference video features. Finally, similarity calculations are performed between the text features and the reference and enhanced reference video features to obtain the similarity set between the video and the text to be matched.

[0320] In the above embodiments, by obtaining the video features and reference features corresponding to the video to be matched, the text features corresponding to the text to be matched are obtained. The reference features include at least one of the audio features and motion features corresponding to the video to be matched. The video features, reference features, and text features are input into the target video-text matching model to obtain a similarity set between the video and the text to be matched. Based on the similarity set, the matching result between the video and the text is determined. Thus, by using a target video-text matching model for feature enhancement and similarity calculation, data processing efficiency can be effectively improved, the similarity set between the video and the text to be matched can be obtained quickly, and the matching result between the video and the text to be matched can be determined quickly.

[0321] In one embodiment, the video text matching method further includes:

[0322] The similarity between the video text features corresponding to the video to be matched and the target text features corresponding to the text to be matched is calculated to obtain a reference similarity between the video text features and the target text features. The video text features include at least one of audio text features and image text features. The audio text features are obtained by extracting features from the audio of the video to be matched, and the image text features are obtained by extracting features from the image of the video to be matched. Based on the similarity set and the reference similarity between the video to be matched and the text to be matched, the matching result between the video to be matched and the text to be matched is determined.

[0323] Specifically, when determining the matching result between the video to be matched and the text to be matched, the similarity between the video text features corresponding to the video to be matched and the target text features corresponding to the text to be matched can be further considered to improve the matching accuracy.

[0324] Computer equipment can acquire the video text features corresponding to the video to be matched and the target text features corresponding to the text to be matched. It then calculates the similarity between these two features, using the result as a reference similarity. Finally, based on the similarity set and the reference similarity, a comprehensive judgment is made regarding the matching result between the video and text to be matched. For example, if all similarities in the similarity set and the reference similarity are greater than a preset threshold, the matching result is considered successful; otherwise, it is considered a failed match. Similarly, if at least two similarities in the similarity set and the reference similarity are greater than the preset threshold, the matching result is considered successful; otherwise, it is considered a failed match.

[0325] The video text features include at least one of audio text features and image text features. Audio text features are obtained by extracting features from the audio of the video to be matched, while image text features are obtained by extracting features from the image of the video to be matched. Audio text can be extracted from the audio of the video to be matched using ASR (Automatic Speech Recognition), and image text can be extracted from the image of the video to be matched using OCR (Optical Character Recognition). Furthermore, audio text features can be obtained by extracting features from the audio text using a machine learning model, and image text features can be obtained by extracting features from the image text using a machine learning model.

[0326] It is understandable that the target text features and the text features to be matched can be the same or different.

[0327] If the video text features include audio text features and image text features, a first reference similarity can be obtained by calculating the similarity between the audio text features and the target text features, and a second reference similarity can be obtained by calculating the similarity between the image text features and the target text features. Based on the similarity sets corresponding to the video and the text to be matched, the first reference similarity, and the second reference similarity, the matching result between the video and the text to be matched is determined.

[0328] In the above embodiments, by comprehensively considering the similarity set and reference similarity when determining the matching result between the video to be matched and the text to be matched, the matching accuracy can be further improved.

[0329] In one embodiment, the similarity between the video text features corresponding to the video to be matched and the target text features corresponding to the text to be matched is calculated to obtain a reference similarity between the video text features and the target text features, including:

[0330] Calculate the initial similarity between video text features and target text features to obtain an initial similarity matrix; count the number of matrix elements in the initial similarity matrix whose values ​​are greater than a preset threshold to obtain a first count; fuse the number of text words corresponding to the text to be matched and the video text respectively to obtain a second count; video text refers to the text corresponding to video text features; based on the first and second counts, obtain the reference similarity between video text features and target text features.

[0331] Specifically, the computer device can first calculate an initial similarity based on the video text features and the target text features to obtain an initial similarity matrix. For example, the cosine similarity between the video text features and the target text features can be used as the initial similarity. The larger the value of the matrix element in the initial similarity matrix, the higher the similarity between the corresponding text words in the text to be matched and the video text. Therefore, the computer device can count the number of matrix elements in the initial similarity matrix whose values ​​are greater than a preset threshold, and use the count result as the first quantity. At the same time, the computer device can fuse the number of text words corresponding to the text to be matched and the video text respectively to obtain a second quantity. For example, the sum of the number of text words corresponding to the text to be matched and the video text can be used as the second quantity; the product of the number of text words corresponding to the text to be matched and the video text can be used as the second quantity; and so on. Finally, the computer device obtains a reference similarity between the video text features and the target text features based on the first and second quantities. For example, the ratio of the first and second quantities can be used as the reference similarity.

[0332] In one embodiment, a cosine similarity matrix is ​​calculated between the video text features and the target text features. Then, the keyword score is calculated using the cosine similarity matrix, and the keyword score is used as a reference similarity. The keyword score can be calculated using the following formula (11).

[0333] N key_word =L(Sim_maxtirx>=threshold)

[0334]

[0335] Where, N key_word `Score` represents the number of elements in the cosine similarity matrix that are greater than or equal to the threshold. `Sim` represents the cosine similarity matrix, and `threshold` represents the threshold. caption_text L represents the keyword score. C L represents the number of words in the text to be matched. T This indicates the number of words in the video text.

[0336] Figure 10 This is a schematic diagram of the cosine similarity matrix. The horizontal axis of the cosine similarity matrix represents each text word in the text to be matched, and the vertical axis represents each text word in the video text. Figure 10 One square in the matrix represents a matrix element of the cosine similarity matrix. The matrix elements of the cosine similarity matrix represent the similarity between corresponding words in the text to be matched and the video text. The higher the similarity, the darker the color of the square.

[0337] In the above embodiments, when calculating the reference similarity, there is no need to perform complex data processing on the video text features and the target text features. Based on the first number obtained by statistically analyzing the matrix elements of the initial similarity matrix and the second number obtained by fusing the number of text words, the reference similarity between the video text features and the target text features can be calculated quickly.

[0338] In one embodiment, the current text is any one of audio text, image text, and text to be matched, and the text feature corresponding to the current text is any one of audio text feature, image text feature, or target text feature. The process of generating the text feature corresponding to the current text includes the following steps:

[0339] Noun extraction is performed on the current text to obtain text nouns; feature extraction is performed on the text nouns to obtain the text features corresponding to the current text.

[0340] Specifically, video text features and target text features can be generated using the same method. Data analysis reveals that abstract nouns such as names of people and places in text are difficult to represent in video features, action features, and audio features, but these names play a crucial role in video text matching tasks. Therefore, when generating video text features and target text features, the computer device can extract nouns from the current text, filter out nouns to obtain text nouns, and extract features only from these text nouns to obtain the text features corresponding to the current text.

[0341] In one embodiment, reference Figure 11 To further improve the accuracy of text features corresponding to the current text, preprocessing can be performed on the text to be matched and the video text. Then, feature extraction can be performed on the preprocessed text to be matched and the video text to obtain target text features and video text features. Keyword scores are calculated based on the target text features and video text features as reference similarity. The preprocessing includes filtering out nouns, restoring words to their original parts of speech and lowering their letter cases. Specifically, nouns are first filtered out from the text, then the nouns are restored to their original parts of speech, and then the nouns with the original parts of speech are lowered.

[0342] In the above embodiments, video text features are generated based on nouns in the video text, and target text features are generated based on nouns in the text to be matched. Such video text features and target text features help to take into account noun information in the video and text that is difficult to express in other modalities when matching video text, thereby helping to improve the matching accuracy of video text.

[0343] In one specific embodiment, the video text matching method of this application is a multi-level, multi-modal hybrid fusion method. (See reference...) Figure 12The target video-text matching model includes four branches: video feature and text feature matching branch (also known as video feature and text feature matching network), video text feature and target text feature matching branch (also known as video text feature and target text feature matching network), action-guided video feature and text feature matching branch (also known as action-enhanced video feature and text feature matching network), and audio-guided video feature and text feature matching branch (also known as audio-enhanced video feature and text feature matching network).

[0344] The input data for the video feature and text feature matching branch are the video features of the video to be matched and the text features of the text to be matched. The output data is the target similarity calculated based on the video features and the text features to be matched. The input data for the video text feature and target text feature matching branch are the video text features of the video to be matched and the target text features of the text to be matched. The output data is the target similarity calculated based on the video text features and the target text features. The input data for the action-guided video feature and text feature matching branch are the video features, action features, and target text features of the video to be matched. The output data is the target similarity calculated based on the action-guided video features and the text features to be matched. The input data for the audio-guided video feature and text feature matching branch are the video features, audio features, and target text features of the video to be matched. The output data is the target similarity calculated based on the audio-guided video features and the text features to be matched.

[0345] The target video-text matching model uses a forward fusion approach, which guides the video features with audio features and motion features respectively. Then, it calculates the similarity between the video features, audio-guided video features, motion-guided video features, and target text features and the text features of the text to be matched.

[0346] The target video text matching model uses a backward fusion approach to integrate the prediction results of the four branches to obtain the final prediction result.

[0347] Additionally, video features can be used to fill misaligned audio features to align multimodal features. This allows for guidance using video features when modalities are misaligned, which can improve the performance of video-text matching to some extent.

[0348] The target video text matching model utilizes a hybrid forward and backward approach to fully mine and leverage the detection capabilities of various modal features, ultimately significantly improving the performance of video text matching. This multi-level, multi-modal hybrid fusion video text matching method has achieved the highest detection performance on relevant datasets.

[0349] The video text matching method of this application utilizes multimodal information, which can provide video text matching with much information that image features cannot provide, thus helping to improve the performance of video text matching. For example, see reference... Figure 13 Text information can provide abstract noun information such as names of people and places. Figure 13 The video text and the text to be matched for the medium video include the nouns "public" and "education," respectively.

[0350] This application's video-to-text matching method employs a multi-level modal fusion strategy, which establishes hierarchical relationships between the multi-modal features of the video and the textual features of the text. The use of forward and backward fusion strategies significantly improves the model's retrieval performance. Forward fusion at the feature level fuses modalities to obtain multi-modal fusion features, while backward fusion integrates the optimal prediction results from each branch. Furthermore, a matching branch is designed for each modal information to mutually retrieve the textual information of the text to be matched, reducing the risk of modal confusion. Hierarchical similarity matching of multiple modal information with the textual information of the text to be matched further reduces the risk of modal confusion and improves the model's detection performance.

[0351] The video-text matching method in this application also explores modality alignment, which can achieve the effect of guiding video features even when modalities are missing, thus increasing the robustness of the model.

[0352] For model training, this application proposes two training modes for its video-text matching method: ensemble training mode and E2E training mode. The ensemble training mode utilizes the proposed multimodal post-fusion processing strategy to integrate the prediction results of each branch. The E2E training mode proposes two loss functions for integrating the branches: a multimodal balance loss function and an optimal loss function. The multimodal balance loss function balances the losses of each branch, adjusting the weight of each branch. The optimal loss function integrates the lowest loss from the training samples of each branch as the final loss.

[0353] Compared to traditional methods, the video-text matching method of this application can deeply mine effective information in the video and match it with the text, thereby effectively improving the retrieval performance for both t2v and v2t tasks.

[0354] It is understood that the video text matching method of this application can be applied to multiple scenarios such as video content understanding, video content recommendation, and video text retrieval.

[0355] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0356] Based on the same inventive concept, embodiments of this application also provide a video text matching model training apparatus for implementing the video text matching model training method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method. Embodiments of this application also provide a video text matching apparatus for implementing the video text matching method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method. Therefore, the specific limitations of the one or more video text matching model training apparatus embodiments provided below can be found in the limitations of the video text matching model training method described above, and will not be repeated here.

[0357] In one embodiment, such as Figure 14 As shown, a video text matching model training device is provided, including: a training sample pair set acquisition module 1402, a feature input module 1404, a feature enhancement module 1406, a similarity calculation module 1408, and a model adjustment module 1410, wherein:

[0358] The training sample pair set acquisition module 1402 is used to acquire the training sample pair set; the training sample pairs in the training sample pair set include positive sample pairs and negative sample pairs that match the positive sample pairs, and the training sample pairs include training videos and training texts, and the training videos and training texts in the positive sample pairs match each other.

[0359] The feature input module 1404 is used to input the video features, reference features, and training text features corresponding to the training videos in the training sample pair set into the initial video-text matching model; the reference features include at least one of the audio features and motion features corresponding to the training videos.

[0360] The feature enhancement module 1406 is used to enhance the corresponding video features based on the reference features corresponding to the same training video to obtain the reference enhanced video features corresponding to the training video; the reference enhanced video features include at least one of action enhanced video features and audio enhanced video features.

[0361] The similarity calculation module 1408 is used to calculate the similarity between the training text features corresponding to the training text and the video features and reference enhanced video features corresponding to the training video for the same training sample pair, so as to obtain the similarity set corresponding to each training sample pair.

[0362] The model adjustment module 1410 is used to calculate the training loss based on the similarity set corresponding to the positive sample pairs and the similarity set corresponding to the matching negative sample pairs, and adjust the model parameters of the initial video-text matching model based on the training loss until the convergence condition is met to obtain the target video-text matching model; the target video-text matching model is used to determine the matching result between the video and the text.

[0363] The aforementioned video-to-text matching model training device utilizes video features to provide image information, audio features to provide sound information, and motion features to provide motion information. By training the video-to-text matching model based on the video features corresponding to the training video, reference features, and the training text features corresponding to the training text, the rich modal information within the video can be leveraged to enhance the model's understanding of the video content, thereby improving the model's prediction accuracy. Furthermore, feature enhancement and guidance based on audio or motion features can highlight important information within the video. Similarity calculations are performed between the video features and the enhanced video features (both based on the video features and reference features) and the training text features. Adjusting the model parameters based on the training loss generated from the calculated similarity set enables the model to better establish the relationship between the video and the text, further improving the model's prediction accuracy.

[0364] In one embodiment, the training sample pair set acquisition module is further configured to:

[0365] Obtain multiple positive sample pairs; reorganize the data of each positive sample pair to obtain multiple negative sample pairs; take the negative sample pairs that have overlapping data with the positive sample pairs as the negative sample pairs that are matched with the positive sample pairs; based on each positive sample pair and each matched negative sample pair, obtain the training sample pair set.

[0366] In one embodiment, the feature enhancement module is also used for:

[0367] Intramodal attention processing is performed on the video features and action features corresponding to the current training video to obtain self-attention video features and self-attention action features corresponding to the current training video. Intermodal attention processing is performed on the video features and self-attention action features corresponding to the current training video to obtain cross-attention video features corresponding to the current training video. Intermodal attention processing is also performed on the action features and self-attention video features corresponding to the current training video to obtain cross-attention action features corresponding to the current training video. The cross-attention action features and cross-attention video features corresponding to the current training video are fused to obtain the action-video fusion features corresponding to the current training video. Channel attention processing is performed on the action-video fusion features corresponding to the current training video to obtain the first channel attention weight. Based on the first channel attention weight, feature enhancement is performed on the video features corresponding to the current training video to obtain the action-enhanced video features corresponding to the current training video.

[0368] In one embodiment, the feature enhancement module is also used for:

[0369] The video features corresponding to the current training video are fused intramodally to obtain the first self-attention weight. The video features corresponding to the current training video and the first self-attention weight are fused to obtain the first attention feature. Based on the video features corresponding to the current training video and the first attention feature, the self-attention video feature is obtained. The action features corresponding to the current training video are fused intramodally to obtain the second self-attention weight. The action features corresponding to the current training video and the second self-attention weight are fused to obtain the second attention feature. Based on the action features corresponding to the current training video and the second attention feature, the self-attention action feature is obtained.

[0370] In one embodiment, the feature enhancement module is also used for:

[0371] The video features and self-attention action features corresponding to the current training video are concatenated to obtain the first concatenated feature. The video features and the first concatenated feature are then fused between modalities to obtain the first cross-attention weight. The first concatenated feature and the first cross-attention weight are then fused to obtain the first cross-attention feature. Based on the first concatenated feature and the first cross-attention feature, a cross-attention video feature is obtained. The action features and self-attention video features corresponding to the current training video are concatenated to obtain the second concatenated feature. The action features and the second concatenated feature are then fused between modalities to obtain the second cross-attention weight. The second concatenated feature and the second cross-attention weight are then fused to obtain the second cross-attention feature. Based on the second concatenated feature and the second cross-attention feature, a cross-attention action feature is obtained.

[0372] In one embodiment, the feature enhancement module is also used for:

[0373] The cross-attention action features and cross-attention video features corresponding to the current training video are concatenated to obtain cross-attention concatenated features; the cross-attention action features and cross-attention video features corresponding to the current training video are fused to obtain cross-attention fused features; the cross-attention concatenated features and cross-attention fused features are fused between modalities to obtain third cross-attention weights; the cross-attention concatenated features and third cross-attention weights are fused to obtain third cross-attention features; and the action-video fused features are obtained based on the cross-attention concatenated features and third cross-attention features.

[0374] In one embodiment, the initial video-text matching model includes an action-enhanced video feature and text feature matching network, which includes a first intramodal attention layer, a second intramodal attention layer, a first intermodal attention layer, a second intermodal attention layer, an action-video fusion attention layer, a first channel attention layer, and a first similarity calculation layer.

[0375] The first intramodal attention layer performs intramodal attention processing on video features, and the second intramodal attention layer performs intramodal attention processing on action features. The first intermodal attention layer performs intermodal attention processing on video features and self-attention action features, and the second intermodal attention layer performs intermodal attention processing on action features and self-attention video features. The action-video fusion attention layer fuses cross-attention action features and cross-attention video features corresponding to the same training video. The first channel attention layer performs channel attention processing on the action-video fusion features. The first similarity calculation layer calculates the similarity between action-enhanced video features and training text features.

[0376] In one embodiment, the feature enhancement module is also used for:

[0377] The video and audio features corresponding to the current training video are fused to obtain the initial audio-video fusion features. The initial audio-video fusion features are then subjected to random deactivation and pooling to obtain the intermediate audio-video fusion features. The intermediate audio-video fusion features are then normalized to obtain the target audio-video fusion features. The target audio-video fusion features are then subjected to channel attention processing to obtain the second channel attention weights. Based on the second channel attention weights, the video features corresponding to the current training video are then augmented to obtain the audio-enhanced video features corresponding to the current training video.

[0378] In one embodiment, the initial video-text matching model includes an audio-enhanced video feature and text feature matching network, which includes an audio-video fusion layer, a random deactivation layer, a pooling layer, a normalization layer, a second channel attention layer, and a second similarity calculation layer.

[0379] The audio-video fusion layer is used to fuse video features and audio features; the random deactivation layer is used to randomly deactivate the input data; the pooling layer is used to pool the input data; the normalization layer is used to normalize the input data; the second channel attention layer is used to perform channel attention processing on the input data; and the second similarity calculation layer is used to calculate the similarity between the audio-enhanced video features and the training text features.

[0380] In one embodiment, the video-text matching model training apparatus is also used for:

[0381] When audio is missing in the training video, the preset features or the video features corresponding to the training video are obtained as the audio features corresponding to the training video.

[0382] In one embodiment, the similarity set includes at least two target similarities, the current video feature is either the video feature corresponding to the training video in the current training sample pair or the reference enhanced video feature, the current text feature is the training text feature corresponding to the training text in the current training sample pair, the training video includes multiple video frames, and the training text includes multiple text words.

[0383] The similarity calculation module is also used for:

[0384] Calculate the initial similarity between the current video features and the current text features; the initial similarity includes the sub-similarity between multiple text words in the training text and the same video frame, and the sub-similarity between multiple video frames in the training video and the same text word; obtain text weights based on the current text features, and obtain video weights based on the current video features; for the initial similarity, take the maximum value from multiple sub-similarity values ​​corresponding to the same video frame as the first sub-similarity, and take the maximum value from multiple sub-similarity values ​​corresponding to the same text word as the second sub-similarity, to obtain the first sub-similarity corresponding to each video frame and the second sub-similarity corresponding to each text word; obtain the first similarity based on each first sub-similarity, and obtain the second similarity based on each second sub-similarity; fuse the first similarity and text weights to obtain the first fused data, and fuse the second similarity and video weights to obtain the second fused data; obtain the target similarity between the current video features and the current text features based on the first fused data and the second fused data.

[0385] In one embodiment, the similarity set includes at least two target similarities.

[0386] The target category is determined from each similarity category. Based on the target similarity of positive sample pairs and each negative sample pair containing the same training text as the positive sample pairs in the target category, the first similarity weight corresponding to each training sample pair in the target category is obtained. Based on the target similarity of positive sample pairs and each negative sample pair containing the same training video as the positive sample pairs in the target category, the second similarity weight corresponding to each training sample pair in the target category is obtained. The target similarity and the first similarity weight of the same training sample pair in the target category are fused to obtain the first updated similarity corresponding to each training sample pair. The target similarity and the second similarity weight of the same training sample pair in the target category are fused to obtain the second updated similarity corresponding to each training sample pair. Similarity; based on the first updated similarity corresponding to each positive sample pair and each negative sample pair containing the same training video as the positive sample pair, a first loss is obtained; based on the second updated similarity corresponding to each positive sample pair and each negative sample pair containing the same training text as the positive sample pair, a second loss is obtained; based on the first loss and the second loss, the training sub-loss corresponding to the target category is obtained; the next similarity category is obtained as the target category, and the steps of obtaining the first similarity weight corresponding to each training sample pair on the target category based on the target similarity of each positive sample pair and each negative sample pair containing the same training text as the positive sample pair are executed until the training sub-loss corresponding to each similarity category is determined; the training loss is obtained based on the various training sub-losses.

[0387] In one embodiment, the model adjustment module is further configured to:

[0388] Based on the target similarity of each training sample pair in the target category, a first similarity matrix is ​​obtained. The first dimension of the first similarity matrix represents the target similarity of each training sample pair containing the same training video in the target category. The second dimension of the target similarity matrix represents the target similarity of each training sample pair containing the same training text in the target category. The diagonal of the target similarity matrix represents the target similarity of each positive sample pair in the target category. Based on the current matrix element in the first similarity matrix and the forward matrix element of the current matrix element in the second dimension, the second matrix element corresponding to the current matrix element is generated. Based on the second matrix elements corresponding to each matrix element in the first similarity matrix, a second similarity matrix is ​​generated. The current matrix element and its forward matrix element in the first dimension are used to generate the third matrix element corresponding to the current matrix element. Based on the third matrix elements corresponding to each matrix element in the first similarity matrix, a third similarity matrix is ​​generated. Based on the target similarity of positive sample pairs in the target category, each matrix element in the second similarity matrix is ​​adjusted to obtain the fourth similarity matrix. Based on the target similarity of positive sample pairs in the target category, each matrix element in the third similarity matrix is ​​adjusted to obtain the fifth similarity matrix. The fourth similarity matrix represents the first similarity weight corresponding to each training sample pair in the target category, and the fifth similarity matrix represents the second similarity weight corresponding to each training sample pair in the target category.

[0389] In one embodiment, the model adjustment module is further configured to:

[0390] The first update similarity of each positive sample pair and each negative sample pair containing the same training video as the positive sample pair is fused to obtain the first similarity statistics of each positive sample pair. Based on the first update similarity and the first similarity statistics of the same positive sample pair, the first sub-loss of each positive sample pair is obtained. The first loss is obtained based on each first sub-loss. The second update similarity of each positive sample pair and each negative sample pair containing the same training text as the positive sample pair is fused to obtain the second similarity statistics of each positive sample pair. Based on the second update similarity and the second similarity statistics of the same positive sample pair, the second sub-loss of each positive sample pair is obtained. The second loss is obtained based on each second sub-loss.

[0391] In one embodiment, the initial video-text matching model includes a video feature and text feature matching network and a reference enhanced video feature and text feature matching network. The reference enhanced video feature and text feature matching network includes at least one of an action enhanced video feature and text feature matching network and an audio enhanced video feature and text feature matching network. The training loss includes the training sub-losses corresponding to each network.

[0392] The model adjustment module is also used for:

[0393] Adjust the model parameters of the corresponding networks in the initial video-text matching model based on each training sub-loss until the convergence conditions of each network are met, thus obtaining the target video-text matching model.

[0394] In one embodiment, the training loss includes a first training sub-loss corresponding to each similarity category. The model tuning module is also used for:

[0395] The minimum value among the first training sub-losses is obtained as the first target sub-loss. Based on the difference between the first target sub-loss and each remaining training sub-loss, the loss contribution of each remaining training sub-loss is obtained. Based on the loss contribution of each remaining training sub-loss, the loss weight of each remaining training sub-loss is obtained. Based on the loss weight of each training sub-loss, the training sub-losses are fused to obtain the first target loss. The loss weight of the first target sub-loss is a preset weight. The model parameters of the initial video text matching model are adjusted based on the first target loss until the convergence condition is met, thus obtaining the target video text matching model.

[0396] In one embodiment, the training loss includes a second training sub-loss for each positive sample pair on each similarity category. The model tuning module is also used for:

[0397] For the same positive sample pair, the minimum value is obtained from each second training sub-loss as the second target sub-loss, and the second target sub-loss corresponding to each positive sample pair is obtained. Based on the statistical value of each second target sub-loss, the second target loss is obtained. Based on the second target loss, the model parameters of the initial video text matching model are adjusted until the convergence condition is met, and the target video text matching model is obtained.

[0398] In one embodiment, the video-text matching model training apparatus is also used for:

[0399] The video features, reference features, and test text features corresponding to the test videos and test texts in the test sample pair set are input into the target video-text matching model to obtain the similarity set corresponding to each test sample pair in the test sample pair set. Based on the similarity set corresponding to each test sample pair, a similarity test matrix corresponding to each similarity category is generated. Based on the similarity test matrix corresponding to each similarity category, the predicted matching sub-ranking corresponding to each test sample pair in each similarity category is determined. Based on the predicted matching sub-rankings corresponding to the same test sample pair, the predicted matching ranking is determined to obtain the predicted matching ranking corresponding to each test sample pair. Based on the predicted matching ranking corresponding to the matching sample pairs in each test sample pair, the prediction accuracy corresponding to the target video-text matching model is determined.

[0400] In one embodiment, such as Figure 15 As shown, a video text matching device is provided, including: a feature acquisition module 1502, a feature enhancement module 1504, a similarity calculation module 1506, and a matching result determination module 1508, wherein:

[0401] The feature acquisition module 1502 is used to acquire the video features to be matched and the reference features to be matched corresponding to the video to be matched, and to acquire the text features to be matched corresponding to the text to be matched; the reference features to be matched include at least one of the audio features and motion features corresponding to the video to be matched.

[0402] The feature enhancement module 1504 is used to enhance the features of the video to be matched based on the reference features to be matched, so as to obtain the reference enhanced video features corresponding to the video to be matched; the reference enhanced video features include at least one of motion enhanced video features and audio enhanced video features.

[0403] The similarity calculation module 1506 is used to calculate the similarity between the text features to be matched and the video features to be matched and the reference enhanced video features, respectively, to obtain the similarity set between the video and the text to be matched.

[0404] The matching result determination module 1508 is used to determine the matching result between the video to be matched and the text to be matched based on the similarity set corresponding to the video to be matched and the text to be matched.

[0405] The aforementioned video-text matching device utilizes video features to provide image information, audio features to provide sound information, and motion features to provide motion information. Based on the video features corresponding to the video to be matched, reference features, and the text features corresponding to the text to be matched, the matching result between the video and the text to be matched is determined. This leverages the rich modal information within the video to enhance understanding of the video content, thereby improving matching accuracy. Furthermore, feature enhancement and guidance based on audio or motion features can highlight important information in the video, further improving understanding of the video content. Calculating similarity between the video features and the text features based on the video features and the reference enhanced video features, respectively, and determining the matching result based on the calculated similarity set, further enhances matching accuracy.

[0406] In one embodiment, the video text matching device is also used for:

[0407] The similarity between the video text features corresponding to the video to be matched and the target text features corresponding to the text to be matched is calculated to obtain a reference similarity between the video text features and the target text features. The video text features include at least one of audio text features and image text features. The audio text features are obtained by extracting features from the audio of the video to be matched, and the image text features are obtained by extracting features from the image of the video to be matched. Based on the similarity set and the reference similarity between the video to be matched and the text to be matched, the matching result between the video to be matched and the text to be matched is determined.

[0408] In one embodiment, the video text matching device is also used for:

[0409] Calculate the initial similarity between video text features and target text features to obtain an initial similarity matrix; count the number of matrix elements in the initial similarity matrix whose values ​​are greater than a preset threshold to obtain a first count; fuse the number of text words corresponding to the text to be matched and the video text respectively to obtain a second count; video text refers to the text corresponding to video text features; based on the first and second counts, obtain the reference similarity between video text features and target text features.

[0410] In one embodiment, the current text is any one of audio text, image text, and text to be matched, and the text feature corresponding to the current text is any one of audio text features, image text features, or target text features. The video text matching device is also used for:

[0411] Noun extraction is performed on the current text to obtain text nouns; feature extraction is performed on the text nouns to obtain the text features corresponding to the current text.

[0412] In one embodiment, the video text matching device is also used for:

[0413] The features of the video to be matched, the reference features to be matched, and the features of the text to be matched are input into the target video-text matching model to obtain the similarity set corresponding to the video and the text to be matched; the target video-text matching model is used for feature enhancement and similarity calculation.

[0414] The modules in the aforementioned video-text matching model training device and video-text matching device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0415] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 16 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores training sample pairs, test sample pairs, target video-text matching models, and other data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a video-text matching model training method and a video-text matching method.

[0416] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 17As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a video-text matching model training method and a video-text matching method. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0417] Those skilled in the art will understand that Figure 16 , 17 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0418] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0419] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0420] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0421] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0422] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0423] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0424] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for training a video text matching model, characterized in that, The method includes: Obtain a set of training sample pairs; the training sample pairs in the set of training sample pairs include positive sample pairs and negative sample pairs that match the positive sample pairs, the training sample pairs include training videos and training text, and the training videos and training text in the positive sample pairs match each other; The video features, reference features, and training text features corresponding to the training videos in the training sample pair set are input into the initial video-text matching model; the reference features include at least one of the audio features and motion features corresponding to the training videos. Based on the reference features corresponding to the same training video, feature enhancement is performed on the corresponding video features to obtain reference enhanced video features corresponding to the training video; the reference enhanced video features include at least one of motion enhanced video features and audio enhanced video features. For the same training sample pair, the training text features corresponding to the training text are compared with the video features corresponding to the training video and the reference enhanced video features to calculate the similarity, thus obtaining the similarity set corresponding to each training sample pair; the similarity set includes at least two target similarities. The target category is determined from each similarity category; Based on the target similarity of positive sample pairs and each negative sample pair containing the same training text as the positive sample pairs in the target category, the first similarity weights corresponding to each training sample pair in the target category are obtained. Based on the target similarity of positive sample pairs and each negative sample pair containing the same training video as the positive sample pairs in the target category, the second similarity weights corresponding to each training sample pair in the target category are obtained. By fusing the target similarity and first similarity weight of the same training sample pair on the target category, the first updated similarity corresponding to each training sample pair is obtained. By fusing the target similarity and second similarity weight of the same training sample pair on the target category, the second updated similarity corresponding to each training sample pair is obtained. A first loss is obtained based on the first updated similarity corresponding to each positive sample pair and each negative sample pair containing the same training video as the positive sample pair; a second loss is obtained based on the second updated similarity corresponding to each positive sample pair and each negative sample pair containing the same training text as the positive sample pair. Based on the first loss and the second loss, the training sub-loss corresponding to the target category is obtained; The training loss is obtained based on the training sub-loss corresponding to each similarity category; The model parameters of the initial video-text matching model are adjusted based on the training loss until the convergence condition is met, thereby obtaining the target video-text matching model; the target video-text matching model is used to determine the matching result between the video and the text.

2. The method according to claim 1, characterized in that, The acquisition of the training sample pair set includes: Obtain multiple positive sample pairs; The positive sample pairs are recombined to obtain multiple negative sample pairs; Negative sample pairs that have overlapping data with positive sample pairs are considered as negative sample pairs that are matched with positive sample pairs. The training sample pair set is obtained based on each positive sample pair and each matching negative sample pair.

3. The method according to claim 1, characterized in that, The process of generating motion-enhanced video features includes the following steps: Intramodal attention processing is performed on the video features and action features corresponding to the current training video to obtain the self-attention video features and self-attention action features corresponding to the current training video. Intermodal attention processing is performed on the video features and self-attention action features corresponding to the current training video to obtain cross-attention video features corresponding to the current training video; intermodal attention processing is performed on the action features and self-attention video features corresponding to the current training video to obtain cross-attention action features corresponding to the current training video. By fusing the cross-attention action features and cross-attention video features corresponding to the current training video, the action-video fusion features corresponding to the current training video are obtained; Channel attention processing is performed on the motion video fusion features corresponding to the current training video to obtain the first channel attention weight. Based on the first channel attention weight, feature enhancement is performed on the video features corresponding to the current training video to obtain the motion-enhanced video features corresponding to the current training video.

4. The method according to claim 3, characterized in that, The step of performing intra-modal attention processing on the video features and action features corresponding to the current training video to obtain the self-attention video features and self-attention action features corresponding to the current training video includes: The video features corresponding to the current training video are fused intramodally to obtain a first self-attention weight. The video features corresponding to the current training video and the first self-attention weight are fused to obtain a first attention feature. Based on the video features corresponding to the current training video and the first attention feature, the self-attention video feature is obtained. The action features corresponding to the current training video are fused intramodally to obtain the second self-attention weight. The action features corresponding to the current training video and the second self-attention weight are fused to obtain the second attention feature. Based on the action features corresponding to the current training video and the second attention feature, the self-attention action feature is obtained.

5. The method according to claim 3, characterized in that, The step of performing intermodal attention processing on the video features and self-attention action features corresponding to the current training video to obtain cross-attention video features corresponding to the current training video, and performing intermodal attention processing on the action features and self-attention video features corresponding to the current training video to obtain cross-attention action features corresponding to the current training video, includes: The video features and self-attention action features corresponding to the current training video are concatenated to obtain a first concatenated feature. The video features and the first concatenated feature are fused between modalities to obtain a first cross-attention weight. The first concatenated feature and the first cross-attention weight are fused to obtain a first cross-attention feature. Based on the first concatenated feature and the first cross-attention feature, the cross-attention video feature is obtained. The action features and self-attention video features corresponding to the current training video are concatenated to obtain a second concatenated feature. The action features and the second concatenated feature are then fused between modalities to obtain a second cross-attention weight. The second concatenated feature and the second cross-attention weight are then fused to obtain a second cross-attention feature. Based on the second concatenated feature and the second cross-attention feature, the cross-attention action feature is obtained.

6. The method according to claim 3, characterized in that, The process of fusing the cross-attention action features and cross-attention video features corresponding to the current training video to obtain the action-video fusion features corresponding to the current training video includes: The cross-attention action features and cross-attention video features corresponding to the current training video are concatenated to obtain the cross-attention concatenated features; The cross-attention action features and cross-attention video features corresponding to the current training video are fused to obtain the cross-attention fusion features; The cross-attention splicing feature and the cross-attention fusion feature are fused intermodally to obtain the third cross-attention weight. The cross-attention splicing feature and the third cross-attention weight are then fused to obtain the third cross-attention feature. Based on the cross-attention splicing feature and the third cross-attention feature, the motion video fusion feature is obtained.

7. The method according to claim 3, characterized in that, The initial video-text matching model includes an action-enhanced video feature and text feature matching network, which includes a first intramodal attention layer, a second intramodal attention layer, a first intermodal attention layer, a second intermodal attention layer, an action-video fusion attention layer, a first channel attention layer, and a first similarity calculation layer. The first intramodal attention layer is used to perform intramodal attention processing on video features, and the second intramodal attention layer is used to perform intramodal attention processing on action features; The first intermodal attention layer is used to perform intermodal attention processing on video features and self-attention action features, and the second intermodal attention layer is used to perform intermodal attention processing on action features and self-attention video features; The motion video fusion attention layer is used to fuse cross-attention motion features and cross-attention video features corresponding to the same training video; The first channel attention layer is used to perform channel attention processing on the motion video fusion features; The first similarity calculation layer is used to calculate the similarity between action-enhanced video features and training text features.

8. The method according to claim 1, characterized in that, The process of generating the audio-enhanced video features includes the following steps: The video and audio features corresponding to the current training video are fused to obtain the initial audio-video fusion features; The initial audio-video fusion features are subjected to random deactivation and pooling to obtain intermediate audio-video fusion features; The intermediate audio-video fusion features are normalized to obtain the target audio-video fusion features; Channel attention processing is performed on the target audio-video fusion features to obtain the second channel attention weights. Based on the second channel attention weights, feature enhancement is performed on the video features corresponding to the current training video to obtain the audio-enhanced video features corresponding to the current training video.

9. The method according to claim 8, characterized in that, The initial video-text matching model includes an audio-enhanced video feature and text feature matching network, which includes an audio-video fusion layer, a random deactivation layer, a pooling layer, a normalization layer, a second channel attention layer, and a second similarity calculation layer. The audio-video fusion layer is used to fuse video features and audio features; The random deactivation layer is used to randomly deactivate the input data; The pooling layer is used to pool the input data; The normalization layer is used to normalize the input data; The second channel attention layer is used to perform channel attention processing on the input data; The second similarity calculation layer is used to calculate the similarity between audio-enhanced video features and training text features.

10. The method according to claim 1, characterized in that, The method further includes: When audio is missing in the training video, the preset features or the video features corresponding to the training video are obtained as the audio features corresponding to the training video.

11. The method according to claim 1, characterized in that, The current video feature is either the video feature corresponding to the training video in the current training sample pair or the reference enhanced video feature. The current text feature is the training text feature corresponding to the training text in the current training sample pair. The training video includes multiple video frames and the training text includes multiple text words. The process of calculating the target similarity between current video features and current text features includes the following steps; Calculate the initial similarity between the current video features and the current text features; the initial similarity includes the sub-similarity between multiple text words in the training text and the same video frame, and the sub-similarity between multiple video frames in the training video and the same text word; Text weights are obtained based on current text features, and video weights are obtained based on current video features. For the initial similarity, the maximum value among multiple sub-similarity values ​​corresponding to the same video frame is taken as the first sub-similarity, and the maximum value among multiple sub-similarity values ​​corresponding to the same text word is taken as the second sub-similarity, so as to obtain the first sub-similarity value corresponding to each video frame and the second sub-similarity value corresponding to each text word. The first similarity is obtained based on each first sub-similarity, and the second similarity is obtained based on each second sub-similarity; The first similarity and the text weight are fused to obtain the first fused data, and the second similarity and the video weight are fused to obtain the second fused data; Based on the first fused data and the second fused data, the target similarity between the current video features and the current text features is obtained.

12. The method according to claim 1, characterized in that, The method involves obtaining a first similarity weight for each training sample pair in the target category based on the target similarity between the positive sample pair and each negative sample pair containing the same training text as the positive sample pair, and obtaining a second similarity weight for each training sample pair in the target category based on the target similarity between the positive sample pair and each negative sample pair containing the same training video as the positive sample pair, including: Based on the target similarity of each training sample pair in the target category, a first similarity matrix is ​​obtained; the first dimension of the first similarity matrix represents the target similarity of each training sample pair containing the same training video in the target category, the second dimension of the first similarity matrix represents the target similarity of each training sample pair containing the same training text in the target category, and the diagonal of the first similarity matrix represents the target similarity of each positive sample pair in the target category. Based on the current matrix element in the first similarity matrix and the forward matrix element of the current matrix element in the second dimension, generate the second matrix element corresponding to the current matrix element. Based on the second matrix elements corresponding to each matrix element in the first similarity matrix, generate the second similarity matrix. Based on the current matrix element in the first similarity matrix and the forward matrix element of the current matrix element in the first dimension, the third matrix element corresponding to the current matrix element is generated. Based on the third matrix element corresponding to each matrix element in the first similarity matrix, the third similarity matrix is ​​generated. Based on the target similarity of positive sample pairs in the target category, the elements of the second similarity matrix are adjusted to obtain the fourth similarity matrix. Based on the target similarity of positive sample pairs in the target category, the elements of the third similarity matrix are adjusted to obtain the fifth similarity matrix. The fourth similarity matrix represents the first similarity weight of each training sample pair in the target category, and the fifth similarity matrix represents the second similarity weight of each training sample pair in the target category.

13. The method according to claim 1, characterized in that, The first loss is obtained based on the first updated similarity corresponding to each positive sample pair and each negative sample pair containing the same training video as the positive sample pair, and the second loss is obtained based on the second updated similarity corresponding to each positive sample pair and each negative sample pair containing the same training text as the positive sample pair, including: The first updated similarity corresponding to each positive sample pair and each negative sample pair containing the same training video as the positive sample pair is fused to obtain the first similarity statistics corresponding to each positive sample pair. Based on the first updated similarity and the first similarity statistics corresponding to the same positive sample pair, the first sub-loss corresponding to each positive sample pair is obtained. The first loss is obtained based on each first sub-loss. By fusing the second updated similarity of each positive sample pair and each negative sample pair containing the same training text as the positive sample pair, the second similarity statistics corresponding to each positive sample pair are obtained. Based on the second updated similarity and the second similarity statistics corresponding to the same positive sample pair, the second sub-loss corresponding to each positive sample pair is obtained. Based on each second sub-loss, the second loss is obtained.

14. The method according to any one of claims 1 to 13, characterized in that, The initial video-text matching model includes a video feature and text feature matching network and a reference enhanced video feature and text feature matching network. The reference enhanced video feature and text feature matching network includes at least one of an action enhanced video feature and text feature matching network and an audio enhanced video feature and text feature matching network. The training loss includes training sub-losses corresponding to each network. The step of adjusting the model parameters of the initial video-text matching model based on the training loss until the convergence condition is met to obtain the target video-text matching model includes: The model parameters of the corresponding networks in the initial video-text matching model are adjusted based on each training sub-loss until the convergence conditions of each network are met, thus obtaining the target video-text matching model.

15. The method according to any one of claims 1 to 13, characterized in that, The training loss includes the first training sub-loss corresponding to each similarity category; The step of adjusting the model parameters of the initial video-text matching model based on the training loss until the convergence condition is met to obtain the target video-text matching model includes: The minimum value among the various first training sub-losses is taken as the first target sub-loss; Based on the differences between the first target sub-loss and each remaining training sub-loss, the loss contribution of each remaining training sub-loss is obtained. Based on the loss contribution of each remaining training sub-loss, the loss weights corresponding to each remaining training sub-loss are obtained. Based on the loss weights corresponding to each training sub-loss, the losses of each training sub-loss are fused to obtain the first target loss; the loss weights corresponding to the first target sub-loss are preset weights. The model parameters of the initial video-text matching model are adjusted based on the first target loss until the convergence condition is met, thus obtaining the target video-text matching model.

16. The method according to any one of claims 1 to 13, characterized in that, The training loss includes the second training sub-loss corresponding to each positive sample pair in each similarity category; The step of adjusting the model parameters of the initial video-text matching model based on the training loss until the convergence condition is met to obtain the target video-text matching model includes: For the same positive sample pair, the minimum value is obtained from each second training sub-loss as the second target sub-loss, thus obtaining the second target sub-loss corresponding to each positive sample pair; The second objective loss is obtained based on the statistical values ​​of the losses of each second objective sub-objective. The model parameters of the initial video-text matching model are adjusted based on the second target loss until the convergence condition is met, thus obtaining the target video-text matching model.

17. The method according to any one of claims 1 to 13, characterized in that, The method further includes: The video features, reference features, and test text features corresponding to the test video in the test sample pair set are input into the target video-text matching model to obtain the similarity set corresponding to each test sample pair in the test sample pair set. Based on the similarity sets corresponding to each test sample pair, a similarity test matrix corresponding to each similarity category is generated. Based on the similarity test matrix corresponding to each similarity category, the predicted matching sub-ranking of each test sample pair in each similarity category is determined. The predicted matching ranking is determined based on the predicted matching sub-rankings corresponding to the same test sample pair, and the predicted matching rankings corresponding to each test sample pair are obtained respectively. Based on the predicted matching ranking of the matching sample pairs in each test sample pair, the prediction accuracy of the target video text matching model is determined.

18. A video text matching method, characterized in that, The method further includes: Obtain the video features and reference features corresponding to the video to be matched, and obtain the text features corresponding to the text to be matched; the reference features to be matched include at least one of the audio features and motion features corresponding to the video to be matched. By using a target video text matching model, based on the reference features to be matched, feature enhancement is performed on the features of the video to be matched to obtain reference enhanced video features corresponding to the video to be matched; the reference enhanced video features include at least one of motion enhanced video features and audio enhanced video features; wherein, the target video text matching model is trained by the video text matching model training method according to any one of claims 1-17; Using the target video text matching model, the similarity between the text features to be matched and the video features to be matched and the reference enhanced video features is calculated to obtain the similarity set between the video to be matched and the text to be matched. Based on the similarity set corresponding to the video to be matched and the text to be matched, the matching result between the video to be matched and the text to be matched is determined.

19. The method according to claim 18, characterized in that, The method further includes: The similarity between the video text features corresponding to the video to be matched and the target text features corresponding to the text to be matched is calculated to obtain a reference similarity between the video text features and the target text features; the video text features include at least one of audio text features and image text features, wherein the audio text features are obtained by extracting features from the audio of the video to be matched, and the image text features are obtained by extracting features from the image of the video to be matched; Based on the similarity set corresponding to the video to be matched and the text to be matched, and the reference similarity, the matching result between the video to be matched and the text to be matched is determined.

20. A video text matching model training device, characterized in that, The device includes: The training sample pair set acquisition module is used to acquire a training sample pair set; the training sample pairs in the training sample pair set include positive sample pairs and negative sample pairs that match the positive sample pairs, the training sample pairs include training videos and training text, and the training videos and training text in the positive sample pairs match each other; The feature input module is used to input the video features, reference features, and training text features corresponding to the training videos in the training sample pair set into the initial video-text matching model; the reference features include at least one of the audio features and motion features corresponding to the training videos. The feature enhancement module is used to enhance the corresponding video features based on the reference features corresponding to the same training video to obtain the reference enhanced video features corresponding to the training video; the reference enhanced video features include at least one of motion enhanced video features and audio enhanced video features. The similarity calculation module is used to calculate the similarity between the training text features corresponding to the training text and the video features and reference enhanced video features corresponding to the training video for the same training sample pair, so as to obtain the similarity set corresponding to each training sample pair; the similarity set includes at least two target similarities. The model adjustment module is used to determine the target category from each similarity category; based on the target similarity of positive sample pairs and each negative sample pair containing the same training text as the positive sample pairs in the target category, it obtains the first similarity weight corresponding to each training sample pair in the target category; based on the target similarity of positive sample pairs and each negative sample pair containing the same training video as the positive sample pairs in the target category, it obtains the second similarity weight corresponding to each training sample pair in the target category; it fuses the target similarity and the first similarity weight of the same training sample pair in the target category to obtain the first updated similarity corresponding to each training sample pair; and it fuses the target similarity and the second similarity weight of the same training sample pair in the target category to obtain the first updated similarity corresponding to each training sample pair. The training sample pairs are respectively assigned a second updated similarity; a first loss is obtained based on the first updated similarity of each positive sample pair and each negative sample pair containing the same training video as the positive sample pair; a second loss is obtained based on the second updated similarity of each positive sample pair and each negative sample pair containing the same training text as the positive sample pair; a training sub-loss is obtained based on the first loss and the second loss; a training loss is obtained based on the training sub-loss of each similarity category; the model parameters of the initial video-text matching model are adjusted based on the training loss until the convergence condition is met, thus obtaining the target video-text matching model; the target video-text matching model is used to determine the matching result between the video and the text.

21. The video text matching model training device according to claim 20, characterized in that, The training sample pair set acquisition module is further configured to: acquire multiple positive sample pairs; reorganize the data of each positive sample pair to obtain multiple negative sample pairs; take the negative sample pairs that have overlapping data with the positive sample pairs as the negative sample pairs that match the positive sample pairs; and obtain the training sample pair set based on each positive sample pair and each matching negative sample pair.

22. The video text matching model training device according to claim 20, characterized in that, The feature enhancement module is further configured to: perform intramodal attention processing on the video features and action features corresponding to the current training video respectively, to obtain the self-attention video features and self-attention action features corresponding to the current training video; Intermodal attention processing is performed on the video features and self-attention action features corresponding to the current training video to obtain cross-attention video features corresponding to the current training video; intermodal attention processing is performed on the action features and self-attention video features corresponding to the current training video to obtain cross-attention action features corresponding to the current training video. By fusing the cross-attention action features and cross-attention video features corresponding to the current training video, the action-video fusion features corresponding to the current training video are obtained; Channel attention processing is performed on the motion video fusion features corresponding to the current training video to obtain the first channel attention weight. Based on the first channel attention weight, feature enhancement is performed on the video features corresponding to the current training video to obtain the motion-enhanced video features corresponding to the current training video.

23. The video text matching model training device according to claim 22, characterized in that, The feature enhancement module is further configured to: perform intramodal fusion of the video features corresponding to the current training video to obtain a first self-attention weight; fuse the video features corresponding to the current training video and the first self-attention weight to obtain a first attention feature; and obtain the self-attention video feature based on the video features corresponding to the current training video and the first attention feature. The action features corresponding to the current training video are fused intramodally to obtain the second self-attention weight. The action features corresponding to the current training video and the second self-attention weight are fused to obtain the second attention feature. Based on the action features corresponding to the current training video and the second attention feature, the self-attention action feature is obtained.

24. The video text matching model training device according to claim 22, characterized in that, The feature enhancement module is further configured to: concatenate the video features and self-attention action features corresponding to the current training video to obtain a first concatenated feature; perform intermodal fusion of the video features and the first concatenated feature corresponding to the current training video to obtain a first cross-attention weight; fuse the first concatenated feature and the first cross-attention weight to obtain a first cross-attention feature; and obtain the cross-attention video feature based on the first concatenated feature and the first cross-attention feature. The action features and self-attention video features corresponding to the current training video are concatenated to obtain a second concatenated feature. The action features and the second concatenated feature are then fused between modalities to obtain a second cross-attention weight. The second concatenated feature and the second cross-attention weight are then fused to obtain a second cross-attention feature. Based on the second concatenated feature and the second cross-attention feature, the cross-attention action feature is obtained.

25. The video text matching model training device according to claim 22, characterized in that, The feature enhancement module is further configured to: concatenate the cross-attention action features and cross-attention video features corresponding to the current training video to obtain cross-attention concatenated features; fuse the cross-attention action features and cross-attention video features corresponding to the current training video to obtain cross-attention fused features; perform intermodal fusion of the cross-attention concatenated features and the cross-attention fused features to obtain a third cross-attention weight; fuse the cross-attention concatenated features and the third cross-attention weight to obtain a third cross-attention feature; and obtain the action video fused features based on the cross-attention concatenated features and the third cross-attention features.

26. The video text matching model training device according to claim 22, characterized in that, The initial video-text matching model includes an action-enhanced video feature and text feature matching network, which includes a first intramodal attention layer, a second intramodal attention layer, a first intermodal attention layer, a second intermodal attention layer, an action-video fusion attention layer, a first channel attention layer, and a first similarity calculation layer. The first intramodal attention layer is used to perform intramodal attention processing on video features, and the second intramodal attention layer is used to perform intramodal attention processing on action features; The first intermodal attention layer is used to perform intermodal attention processing on video features and self-attention action features, and the second intermodal attention layer is used to perform intermodal attention processing on action features and self-attention video features; The motion video fusion attention layer is used to fuse cross-attention motion features and cross-attention video features corresponding to the same training video; The first channel attention layer is used to perform channel attention processing on the motion video fusion features; The first similarity calculation layer is used to calculate the similarity between action-enhanced video features and training text features.

27. The video text matching model training device according to claim 20, characterized in that, The feature enhancement module is further configured to: fuse the video features and audio features corresponding to the current training video to obtain initial audio-video fusion features; perform random deactivation and pooling processing on the initial audio-video fusion features to obtain intermediate audio-video fusion features; perform normalization processing on the intermediate audio-video fusion features to obtain target audio-video fusion features; perform channel attention processing on the target audio-video fusion features to obtain second channel attention weights; and perform feature enhancement on the video features corresponding to the current training video based on the second channel attention weights to obtain audio-enhanced video features corresponding to the current training video.

28. The video text matching model training device according to claim 27, characterized in that, The initial video-text matching model includes an audio-enhanced video feature and text feature matching network, which includes an audio-video fusion layer, a random deactivation layer, a pooling layer, a normalization layer, a second channel attention layer, and a second similarity calculation layer. The audio-video fusion layer is used to fuse video features and audio features; The random deactivation layer is used to randomly deactivate the input data; The pooling layer is used to pool the input data; The normalization layer is used to normalize the input data; The second channel attention layer is used to perform channel attention processing on the input data; The second similarity calculation layer is used to calculate the similarity between audio-enhanced video features and training text features.

29. The video text matching model training device according to claim 20, characterized in that, The device is further configured to: when audio is missing in the training video, obtain a preset feature or the video feature corresponding to the training video as the audio feature corresponding to the training video.

30. The video text matching model training device according to claim 20, characterized in that, The current video feature is either the video feature corresponding to the training video in the current training sample pair or the reference enhanced video feature. The current text feature is the training text feature corresponding to the training text in the current training sample pair. The training video includes multiple video frames and the training text includes multiple text words. The similarity calculation module is also used to: calculate the initial similarity between the current video features and the current text features; the initial similarity includes the sub-similarity between multiple text words in the training text and the same video frame, and the sub-similarity between multiple video frames in the training video and the same text word; obtain text weights based on the current text features, and obtain video weights based on the current video features; For the initial similarity, the maximum value among multiple sub-similarity values ​​corresponding to the same video frame is obtained as the first sub-similarity, and the maximum value among multiple sub-similarity values ​​corresponding to the same text word is obtained as the second sub-similarity, thus obtaining the first sub-similarity corresponding to each video frame and the second sub-similarity corresponding to each text word; a first similarity is obtained based on each first sub-similarity, and a second similarity is obtained based on each second sub-similarity; the first similarity and the text weight are fused to obtain the first fused data, and the second similarity and the video weight are fused to obtain the second fused data; Based on the first fused data and the second fused data, the target similarity between the current video features and the current text features is obtained.

31. The video text matching model training device according to claim 20, characterized in that, The model adjustment module is further configured to: obtain a first similarity matrix based on the target similarity of each training sample pair in the target category; the first dimension of the first similarity matrix represents the target similarity of each training sample pair containing the same training video in the target category, the second dimension of the first similarity matrix represents the target similarity of each training sample pair containing the same training text in the target category, and the diagonal of the first similarity matrix represents the target similarity of each positive sample pair in the target category. Based on the current matrix element in the first similarity matrix and the forward matrix element of the current matrix element in the second dimension, generate the second matrix element corresponding to the current matrix element. Based on the second matrix elements corresponding to each matrix element in the first similarity matrix, generate the second similarity matrix. Based on the current matrix element in the first similarity matrix and the forward matrix element of the current matrix element in the first dimension, the third matrix element corresponding to the current matrix element is generated. Based on the third matrix element corresponding to each matrix element in the first similarity matrix, the third similarity matrix is ​​generated. Based on the target similarity of positive sample pairs in the target category, the elements of each matrix in the second similarity matrix are adjusted to obtain the fourth similarity matrix. Based on the target similarity of positive sample pairs in the target category, the elements of each matrix in the third similarity matrix are adjusted to obtain the fifth similarity matrix. The fourth similarity matrix represents the first similarity weight of each training sample pair in the target category, and the fifth similarity matrix represents the second similarity weight of each training sample pair in the target category.

32. The video text matching model training device according to claim 20, characterized in that, The model adjustment module is further configured to: fuse the first updated similarity corresponding to each positive sample pair and each negative sample pair containing the same training video as the positive sample pair, to obtain the first similarity statistical value corresponding to each positive sample pair; based on the first updated similarity and the first similarity statistical value corresponding to the same positive sample pair, to obtain the first sub-loss corresponding to each positive sample pair; and based on each first sub-loss, to obtain the first loss; fuse the second updated similarity corresponding to each positive sample pair and each negative sample pair containing the same training text as the positive sample pair, to obtain the second similarity statistical value corresponding to each positive sample pair; based on the second updated similarity and the second similarity statistical value corresponding to the same positive sample pair, to obtain the second sub-loss corresponding to each positive sample pair; and based on each second sub-loss, to obtain the second loss.

33. The video text matching model training apparatus according to any one of claims 20 to 32, characterized in that, The initial video-text matching model includes a video feature and text feature matching network and a reference enhanced video feature and text feature matching network. The reference enhanced video feature and text feature matching network includes at least one of an action enhanced video feature and text feature matching network and an audio enhanced video feature and text feature matching network. The training loss includes training sub-losses corresponding to each network. The model adjustment module is also used for: The model parameters of the corresponding networks in the initial video-text matching model are adjusted based on each training sub-loss until the convergence conditions of each network are met, thus obtaining the target video-text matching model.

34. The video text matching model training apparatus according to any one of claims 20 to 32, characterized in that, The training loss includes the first training sub-loss corresponding to each similarity category; The model adjustment module is further configured to: obtain the minimum value from each first training sub-loss as the first target sub-loss; obtain the loss contribution of each remaining training sub-loss based on the difference between the first target sub-loss and each remaining training sub-loss; obtain the loss weight of each remaining training sub-loss based on the loss contribution of each remaining training sub-loss; fuse each training sub-loss based on the loss weight of each training sub-loss to obtain the first target loss; the loss weight of the first target sub-loss is a preset weight; adjust the model parameters of the initial video text matching model based on the first target loss until the convergence condition is met to obtain the target video text matching model.

35. The video text matching model training apparatus according to any one of claims 20 to 32, characterized in that, The training loss includes the second training sub-loss corresponding to each positive sample pair in each similarity category; The model adjustment module is further configured to: for the same positive sample pair, obtain the minimum value from each second training sub-loss as the second target sub-loss, and obtain the second target sub-loss corresponding to each positive sample pair; obtain the second target loss based on the statistical value of each second target sub-loss; and adjust the model parameters of the initial video text matching model based on the second target loss until the convergence condition is met, thereby obtaining the target video text matching model.

36. The video text matching model training apparatus according to any one of claims 20 to 32, characterized in that, The apparatus is further configured to: input the video features, reference features, and test text features corresponding to the test video in the test sample pair set into the target video-text matching model to obtain a similarity set corresponding to each test sample pair in the test sample pair set; generate a similarity test matrix corresponding to each similarity category based on the similarity set corresponding to each test sample pair; and determine the predicted matching sub-ranking of each test sample pair in each similarity category based on the similarity test matrix corresponding to each similarity category. The predicted matching ranking is determined based on the predicted matching sub-rankings corresponding to the same test sample pair, and the predicted matching rankings corresponding to each test sample pair are obtained respectively. Based on the predicted matching ranking of the matching sample pairs in each test sample pair, the prediction accuracy of the target video text matching model is determined.

37. A video text matching device, characterized in that, The device includes: The feature acquisition module is used to acquire the video features to be matched and the reference features to be matched corresponding to the video to be matched, and to acquire the text features to be matched corresponding to the text to be matched; the reference features to be matched include at least one of the audio features and motion features corresponding to the video to be matched; A feature enhancement module is used to enhance the features of the video to be matched based on the reference features to be matched using a target video text matching model, thereby obtaining reference enhanced video features corresponding to the video to be matched; the reference enhanced video features include at least one of motion enhanced video features and audio enhanced video features; wherein, the target video text matching model is trained by the video text matching model training method according to any one of claims 1-17; The similarity calculation module is used to calculate the similarity between the text features to be matched and the video features to be matched and the reference enhanced video features through the target video text matching model, so as to obtain the similarity set corresponding to the video to be matched and the text to be matched; The matching result determination module is used to determine the matching result between the video to be matched and the text to be matched based on the similarity set corresponding to the video to be matched and the text to be matched.

38. The video text matching device according to claim 37, characterized in that, The apparatus is further configured to: calculate the similarity between the video text features corresponding to the video to be matched and the target text features corresponding to the text to be matched, and obtain a reference similarity between the video text features and the target text features; the video text features include at least one of audio text features and image text features, wherein the audio text features are obtained by extracting features from the audio of the video to be matched, and the image text features are obtained by extracting features from the image of the video to be matched; and determine the matching result between the video to be matched and the text to be matched based on the similarity set corresponding to the video to be matched and the text to be matched and the reference similarity.

39. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 19.

40. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 19.

41. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 19.

Images