Cross-modal model training method, device, equipment and storage medium

By learning the feature representation of semantic anchors in explicit vocabulary space, combining video encoder, text encoder and perceptron, the modal divide problem in the dual encoder model is solved, and the robustness of cross-modal alignment is achieved and generalization.

CN118535765BActive Publication Date: 2024-12-06INST OF AUTOMATION CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411003383.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-12-06
Estimated Expiration
2044-07-25

AI Technical Summary

Technical Problem

The dual encoder model has a modal divide problem in video-text retrieval, resulting in the separation of video and text features in hidden space, affecting the semantic alignment of cross-modal representations and the robustness and migration capabilities of downstream tasks.

Method used

Learn semantic anchor feature representation in an explicit vocabulary space, build a cross-modal pre-trained model through a combination of video encoder, text encoder, and perceptron, and map video and text features to the same vocabulary space using mapping heads. And optimized through global and local loss functions to achieve cross-modal alignment.

Benefits of technology

It alleviates the modal divide problem, improves the semantic anchoring and generalization ability of feature representation, and enhances the robustness and migration ability of cross-modal alignment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118535765B_ABST
    Figure CN118535765B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of cross-modal technology, and provides a training method, device, equipment and storage medium of a cross-modal model, including extracting a first video feature by a video encoder, extracting a first text feature and a text mask feature by a text encoder, a second video feature and a second text feature in the same vocabulary space, and a third video feature and a third text feature in the same vocabulary space; obtaining a first global loss function based on the second video feature and the second text feature, obtaining a second global loss function based on the second video feature and the text mask feature; obtaining a first local loss function based on the first video feature, the first text feature and a perceptron; obtaining a second local loss function based on the first video feature, the text mask feature and the perceptron; and training based on the first global loss function, the second global loss function, the first local loss function, the second local loss function and the mask loss function. The problem of alleviating the modality gap is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of cross-modal technology, and in particular to a training method, apparatus, device and storage medium for a cross-modal model. Background Art

[0002] Video-text retrieval aims to retrieve corresponding videos based on query text, and has a wide range of applications in real-world scenarios, such as video surveillance, search engines, etc. To achieve fast retrieval, most current methods use a dual encoder architecture. Video and text are converted into a shared latent space through their respective encoders, and contrastive learning methods are used to train in this space to achieve cross-modal representation alignment.

[0003] Although previous methods have made significant progress by adopting pre-training techniques and contrastive learning strategies, dual-encoder models still face a long-standing challenge: the modality gap problem.

[0004] The modality gap problem manifests itself as the features of video and text being clustered in separate clusters in the latent space, with a clear boundary between the two. This prevents the feature representations of semantically related video-text pairs from being grouped together in this latent space. At the same time, related studies have also shown that the dual encoder pre-training method based on latent space is significantly restricted by the modality gap problem and cannot fully complete the semantic alignment task between cross-modal representations, affecting the robustness and transferability of the learned feature representations in downstream tasks.

[0005] However, how to effectively alleviate the modal gap problem caused by latent space has not yet been deeply studied and solved. Summary of the invention

[0006] The present invention provides a cross-modal model training method, apparatus, device and storage medium, which are used to solve the defect of the modality gap existing in the dual encoder pre-training method in the prior art.

[0007] The present invention provides a cross-modal model training method, the method comprising:

[0008] Constructing a cross-modal pre-training model, wherein the cross-modal pre-training model includes a video encoder, a text encoder, and a perceptron;

[0009] Extracting a first video feature of the video data by the video encoder, and extracting a first text feature and a text mask feature of the text data matched by the video data by the text encoder;

[0010] Mapping the first video feature and the first text feature to the same vocabulary space to obtain corresponding second video features and second text features, and mapping the first video feature and the text mask feature to the same vocabulary space to obtain corresponding third video features and third text features;

[0011] Based on the second video feature and the second text feature, obtaining a first global loss function, and based on the second video feature and the text mask feature, obtaining a second global loss function;

[0012] Based on the first video feature, the first text feature and the perceptron, obtaining a first local loss function; and based on the first video feature, the text mask feature and the perceptron, obtaining a second local loss function;

[0013] The cross-modal pre-training model is trained based on the first global loss function, the second global loss function, the first local loss function, the second local loss function, and the mask loss function corresponding to the text mask feature to obtain a trained cross-modal pre-training model.

[0014] According to a cross-modal model training method provided by the present invention, mapping the first video feature and the first text feature to the same vocabulary space to obtain corresponding second video features and second text features includes:

[0015] A mapping head that builds a vocabulary space based on the pre-trained masked language head;

[0016] Performing feature mapping on the first video feature based on the mapping head, and performing dimension suppression processing on the feature-mapped video feature to obtain a second video feature corresponding to the first video feature;

[0017] Feature mapping is performed on the first text feature based on the mapping head, and dimension suppression processing is performed on the text feature after feature mapping to obtain a second text feature corresponding to the first text feature.

[0018] A cross-modal model training method provided by the present invention further includes:

[0019] The dimension of the video features after feature mapping is suppressed based on the Relu activation function and the logarithmic transformation function;

[0020] The dimension of the text features after feature mapping is suppressed based on the Relu activation function and the logarithmic transformation function.

[0021] According to a cross-modal model training method provided by the present invention, obtaining a first global loss function based on the second video feature and the second text feature includes:

[0022] Performing a maximum pooling-based aggregation operation on the second video feature and the second text feature respectively to obtain a video global feature corresponding to the second video feature and a text global feature corresponding to the second text feature;

[0023] Based on the video global feature and the text global feature, obtaining a first contrastive learning loss function and a first FLOPs loss function;

[0024] The first contrastive learning loss function and the first FLOPs loss function are used as a first global loss function.

[0025] According to a cross-modal model training method provided by the present invention, the first local loss function is obtained based on the first video feature, the first text feature and the perceptron, including:

[0026] Inputting the first video feature and the learnable query vector into the perceptron to obtain a fused query vector output by the perceptron;

[0027] Filtering out a first target text feature from the first text feature, wherein the first target text feature includes a word-element feature of a target part of speech;

[0028] Determining an optimal match between the fused query vector and the first target text feature;

[0029] Based on the optimal match, obtaining a first matching loss between the fused query vector and the first target text feature;

[0030] The first matching loss is used as a first local loss function.

[0031] According to a cross-modal model training method provided by the present invention, the determining the optimal match between the fused query vector and the first target text feature includes:

[0032] Determining a cosine similarity between the fused query vector and the first target text feature;

[0033] Taking the cosine similarity as the matching cost, constructing an optimal matching cost mapping function between the fused query vector and the first target text feature;

[0034] Based on the optimal matching cost mapping function, an optimal match between the fused query vector and the first target text feature is determined.

[0035] According to a cross-modal model training method provided by the present invention, the cross-modal pre-training model is trained based on the first global loss function, the second global loss function, the first local loss function, the second local loss function, and the mask loss function corresponding to the text mask feature to obtain a trained cross-modal pre-training model, including:

[0036] In a first training phase, the video encoder and the perceptron are optimized based on the first global loss function and the first local loss function;

[0037] In a second training phase, the video encoder, the perceptron, and the text encoder are optimized based on the second global loss function, the second local loss function, and a mask loss function corresponding to the text mask feature;

[0038] Based on the first training stage and the second training stage, a trained cross-modal pre-training model is obtained.

[0039] The present invention also provides a cross-modal model training device, the device comprising:

[0040] A first cross-modal model training module, used to construct a cross-modal pre-training model, wherein the cross-modal pre-training model includes a video encoder, a text encoder, and a perceptron;

[0041] A training module of a second cross-modal model, configured to extract a first video feature of the video data through the video encoder, and to extract a first text feature and a text mask feature of the text data matched with the video data through the text encoder;

[0042] a training module of a third cross-modal model, configured to map the first video feature and the first text feature to the same vocabulary space to obtain corresponding second video features and second text features, and to map the first video feature and the text mask feature to the same vocabulary space to obtain corresponding third video features and third text features;

[0043] a training module of a fourth cross-modal model, configured to obtain a first global loss function based on the second video feature and the second text feature, and to obtain a second global loss function based on the second video feature and the second text feature;

[0044] a training module of a fifth cross-modal model, configured to obtain a first local loss function based on the first video feature, the first text feature, and the perceptron; and, to obtain a second local loss function based on the first video feature, the text mask feature, and the perceptron;

[0045] The sixth cross-modal model training module is used to train the cross-modal pre-trained model based on the first global loss function, the second global loss function, the first local loss function, the second local loss function and the mask loss function corresponding to the text mask feature to obtain a trained cross-modal pre-trained model.

[0046] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a training method for a cross-modal model as described above is implemented.

[0047] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for training a cross-modal model as described in any one of the above.

[0048] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described cross-modal model training methods.

[0049] The cross-modal model training method, apparatus, device and storage medium provided by the present invention are used to alleviate the problem of the modality gap by learning semantically anchored feature representations in an explicit vocabulary space. In addition, in order to realize the semantically anchored characteristics of feature representations in the vocabulary space, the present invention also proposes a training strategy for multi-granularity video semantics and text semantics based on this, which realizes cross-modal alignment in an explicit vocabulary space during the training phase, and has generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 It is a schematic diagram of the flow chart of the training method of the cross-modal model provided by the present invention;

[0052] Figure 2 This is one of the scenario schematic diagrams of the cross-modal model training method provided by the present invention;

[0053] Figure 3 This is the second scenario schematic diagram of the cross-modal model training method provided by the present invention;

[0054] Figure 4 It is a structural schematic diagram of a training device for a cross-modal model provided by the present invention;

[0055] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] In the related art, the model of video text retrieval based on the dual encoder architecture has made significant progress. Its basic structure includes two encoder networks, which are used to encode the spatiotemporal representation of the video and the contextual representation of the text. Then, the cosine similarity between the cross-modal positive and negative sample pairs is calculated respectively, and the cross-modal alignment ability of the model is trained through contrastive learning.

[0058] Most current dual encoder architecture methods are dedicated to modeling detailed visual-text data alignment. Although these methods have made significant progress, related studies have shown that there is a modality gap problem in current dual encoder methods. The modality gap problem directly indicates that the cross-modal representation alignment obtained by these methods is insufficient, which further limits the robustness and transferability of the representations learned in the model pre-training stage in downstream tasks.

[0059] There are some methods dedicated to alleviating the problem of the modal gap. For example, the expectation maximization algorithm is used to find a set of compact bases of the latent space, and the cross-modal features are represented as linear combinations of these bases. For example, a cyclic translation mechanism between the source modal space and the target modal space is proposed. Another example is a geodesic multimodal mixing method to generate hard negative samples. The model is then fine-tuned based on the hard negative samples and the initial positive and negative samples. However, these methods only try to solve the modal gap problem in the fine-tuning stage, and the learned features lack generalization.

[0060] Therefore, in response to the above problems, the present invention provides a training method for a cross-modal model. The present invention proposes learning semantically anchored feature representations in an explicit vocabulary space to alleviate the problem of the modality gap. In addition, in order to realize the semantic anchoring characteristics of feature representations in the vocabulary space, the present invention also proposes a training strategy for multi-granular video semantics and text semantics based on this, which realizes cross-modal alignment in an explicit vocabulary space during the training phase, and has generalization.

[0061] Figure 1is a flow chart of the training method of the cross-modal model provided by the present invention, such as Figure 1 As shown, the method includes:

[0062] Step 110, constructing a cross-modal pre-training model, wherein the cross-modal pre-training model includes a video encoder, a text encoder, and a perceptron;

[0063] Here, reference Figure 2 As shown, the cross-modal pre-training model in this embodiment includes three main components: a video encoder, a text encoder, and a perceptron.

[0064] The video encoder is responsible for converting video data into a representation that the model can understand. It can use existing video processing models such as Convolutional Neural Network (CNN).

[0065] The text encoder is responsible for converting text data into a representation that the model can understand. It can use existing natural language processing models such as BERT (Bidirectional Encoder Representations from Transformers, based on Transformer's bidirectional encoder representation).

[0066] The perceptron is responsible for capturing visual concepts in video data and consists of several self-attention and cross-attention layers.

[0067] In this embodiment, a cross-modal pre-training model is formed by combining a video encoder, a text encoder and a perceptron. This model can process video data and text data at the same time and establish a connection between them. By pre-training this model with a large amount of training data, it can learn how to effectively extract and integrate information from cross-modal data.

[0068] Step 120, extracting a first video feature of the video data by the video encoder, and extracting a first text feature and a text mask feature of the text data matched by the video data by the text encoder;

[0069] Here, the training data of the cross-modal pre-training model consists of two parts: one is video data, and the other is text data corresponding to the video data. These text data are usually descriptions or labels of the video data.

[0070] The video encoder is responsible for processing the video data and extracting the useful first video features therefrom. The text encoder is responsible for processing the text data and extracting the useful first text features therefrom.

[0071] The specific reference is as follows, using the video encoder and text encoder Extracting video data The first video feature and text data The first text feature :

[0072] ;

[0073] ;

[0074] in, is the number of picture blocks of video data, is the number of tokens in the text data, Represents the dimension of the original data.

[0075] In addition, in order to avoid weakening the feature representation ability of the semantic anchoring of the text encoder in the subsequent training process, the corresponding text mask features of the text data are used to maintain the text semantics. The text mask feature refers to the feature representation obtained by processing the text encoder after masking some words in the text data. These features are used to help the text encoder learn how to predict the masked words based on the context during the training process.

[0076] Specifically, the text semantics is maintained through the MLM (Masked Language Modeling) task. Here, the MLM task is a self-supervised learning task that inputs text data with masked tokens into the text encoder through random masking (i.e., replacement or deletion), and then requires the text encoder to output the corresponding text mask features.

[0077] Step 130, mapping the first video feature and the first text feature to the same vocabulary space to obtain corresponding second video features and second text features, and mapping the first video feature and the text mask feature to the same vocabulary space to obtain corresponding third video features and third text features;

[0078] refer to Figure 3 As shown in Figure 1, each dimension of the vocabulary space is mapped to a specific word in the vocabulary. In the feature representation of a video or text, the dimension related to its semantics has a higher activation value, while the dimension irrelevant to it is suppressed to zero. This allows it to reflect a distinct semantic anchoring property.

[0079] In this embodiment, this step is to transform the original features into a unified vocabulary space so that these features have clear semantic interpretations. After mapping, each dimension in the feature representation can be directly associated with a word in the vocabulary.

[0080] Specifically, based on the principle of semantic anchoring, after mapping, the first video feature and the first text feature are converted into the second video feature and the second text feature in the vocabulary space, respectively. Each feature dimension of the vocabulary space explicitly corresponds to a semantic concept related to a word in the vocabulary space.

[0081] Similarly, based on the principle of semantic anchoring, after mapping, the first video feature and the text mask feature are respectively converted into the third video feature and the third text feature in the vocabulary space.

[0082] In this embodiment, based on the principle of semantic anchoring, video features and text features are mapped to a preset vocabulary space. Specifically, the feature representation of video features and text features in the vocabulary space is expected to have the following characteristics: semantically related feature dimensions have higher activation values, and semantically irrelevant feature dimensions are suppressed to 0.

[0083] Step 140, obtaining a first global loss function based on the second video feature and the second text feature, and obtaining a second global loss function based on the third video feature and the third text feature;

[0084] In this step, the first global loss function calculated based on the second video feature and the second text feature is used to measure a certain global relationship between the second video feature and the second text feature. Similarly, the second global loss function calculated based on the third video feature and the third text feature is used to measure a certain global relationship between the third video feature and the third text feature.

[0085] Step 150, obtaining a first local loss function based on the first video feature, the first text feature, and the perceptron; and obtaining a second local loss function based on the first video feature, the text mask feature, and the perceptron;

[0086] In this embodiment, a first local loss function is calculated based on the first video feature, the first text feature, and the perceptron. The first local loss function is used to measure the relationship between the two features at the local level. Similarly, a second local loss function is calculated based on the first video feature, the text mask feature, and the perceptron.

[0087] Step 160: Train the cross-modal pre-training model based on the first global loss function, the second global loss function, the first local loss function, the second local loss function, and the mask loss function corresponding to the text mask feature to obtain a trained cross-modal pre-training model.

[0088] In this embodiment, during the joint training process, the values ​​of these loss functions will be considered simultaneously, and their weighted sum will be used as the total loss for cross-modal pre-training model training. The cross-modal pre-training model will try to minimize these loss functions simultaneously, thereby optimizing performance in multiple aspects.

[0089] In this embodiment, semantically anchored feature representations are learned in an explicit vocabulary space to alleviate the problem of the modal gap. In addition, in order to realize the semantically anchored characteristics of feature representations in the vocabulary space, the present invention also proposes a training strategy for multi-granularity video semantics and text semantics, that is, model training is performed by using a global loss function for cross-modal alignment under global features and a global loss function for cross-modal alignment under local features, thereby achieving cross-modal alignment in an explicit vocabulary space during the training phase, which is generalizable.

[0090] It should be noted that each implementation method of the present application can be freely combined, the order can be changed, or it can be executed separately, and does not need to rely on or depend on a fixed execution order.

[0091] In addition, it should be noted that all steps involved in the first video feature and the first text feature in this embodiment are the same as all steps involved in the first video feature and the text mask feature.

[0092] In other words, the step of mapping the first video feature and the first text feature to the same vocabulary space to obtain the corresponding second video feature and second text feature in this embodiment is the same as the step of mapping the first video feature and the text mask feature to the same vocabulary space to obtain the corresponding third video feature and third text feature, the step of obtaining the first global loss function based on the second video feature and the second text feature is the same as the step of obtaining the second global loss function based on the third video feature and the third text feature; the step of obtaining the first local loss function based on the first video feature, the first text feature and the perceptron is the same as the step of obtaining the second local loss function based on the first video feature, the text mask feature and the perceptron.

[0093] In this embodiment, all steps involved in the first video feature and the first text feature are taken as representatives for explanation, and all steps involved in the first video feature and the text mask feature are not further described in detail.

[0094] In some embodiments, mapping the first video feature and the first text feature to the same vocabulary space to obtain corresponding second video features and second text features includes:

[0095] A mapping head that builds a vocabulary space based on the pre-trained masked language head;

[0096] Performing feature mapping on the first video feature based on the mapping head, and performing dimension suppression processing on the feature-mapped video feature to obtain a second video feature corresponding to the first video feature;

[0097] Feature mapping is performed on the first text feature based on the mapping head, and dimension suppression processing is performed on the text feature after feature mapping to obtain a second text feature corresponding to the first text feature.

[0098] Here, the Masked Language Modeling (MLM) head in the pre-trained language model (such as BERT, etc.) is used as the mapping head, that is, the MLM head in the pre-trained model is used as a mapping task-specific layer.

[0099] In this embodiment, the original features (ie, the first video features and the first text features) are converted into a new feature space using the constructed mapping head, and then a dimension suppression process is performed to suppress the semantically irrelevant feature dimensions to zero.

[0100] In some embodiments, the method further comprises:

[0101] The dimension of the video features after feature mapping is suppressed based on the Relu activation function and the logarithmic transformation function;

[0102] The dimension of the text features after feature mapping is suppressed based on the Relu activation function and the logarithmic transformation function.

[0103] Specifically, the mapping head H maps the first video feature and the first text feature to the vocabulary space to obtain the video feature after feature mapping: Text features after feature mapping :

[0104] ;

[0105] ;

[0106] in, is the size of the vocabulary space.

[0107] Then perform the following operations on the features to obtain the corresponding second video features: With the second text feature :

[0108] ;

[0109] ;

[0110] Among them, the activation function Set negative values ​​to 0, logarithmic transformation function It is used to smooth the entire feature data distribution and avoid some dimensions in the feature representation being too large.

[0111] In this embodiment, the above method is used to learn semantically anchored feature representation in an explicit vocabulary space to alleviate the problem of the modality gap.

[0112] In some embodiments, obtaining a first global loss function based on the second video feature and the second text feature includes:

[0113] Performing a maximum pooling-based aggregation operation on the second video feature and the second text feature respectively to obtain a video global feature corresponding to the second video feature and a text global feature corresponding to the second text feature;

[0114] Based on the video global feature and the text global feature, obtaining a first contrastive learning loss function and a first FLOPs loss function;

[0115] The first contrastive learning loss function and the first FLOPs loss function are used as a first global loss function.

[0116] In this embodiment, the first global loss function is divided into two parts, one part is the first contrastive learning loss function, and the other part is the first FLOPs loss function.

[0117] Specifically, for the second video feature With the second text feature Perform aggregation operations based on maximum pooling to obtain the corresponding video global features and global features of the text Here, maximum pooling is a downsampling method used to reduce the spatial size of features while retaining the most important features. In this embodiment, the second video feature is With the second text feature Perform aggregation operations separately.

[0118] Next, we optimize the Video-Text Contrastive (VTC) objective with Noise Contrastive Estimation (NCE) to learn global video features in the vocabulary space. and global features of the text Alignment, calculate the first contrastive learning loss function of this process:

[0119] ;

[0120] in, and Respectively represent i Video global features and text global features under contrastive learning of batches; is the batch size.

[0121] Here, ;

[0122] in, is the temperature hyperparameter, same The calculation formula is the same as that of , and will not be described in detail here.

[0123] In addition, since the vocabulary space is high-dimensional, in order to prevent a large number of non-zero values ​​from appearing in semantically irrelevant feature dimensions and covering semantically relevant feature dimensions, this embodiment also introduces the first FLOPs loss function to encourage the sparsity of vocabulary representation:

[0124] ;

[0125] in, and are the weight coefficients, and Respectively represent i The video global features and text global features of the kth feature dimension under contrastive learning of batches.

[0126] The first contrastive learning loss function and the first FLOPs loss function are combined to form a first global loss function. During the training process, this first global loss function is used to guide the optimization direction of the model.

[0127] In some embodiments, obtaining a first local loss function based on the first video feature, the first text feature, and the perceptron includes:

[0128] Inputting the first video feature and the learnable query vector into the perceptron to obtain a fused query vector output by the perceptron;

[0129] Filtering out a first target text feature from the first text feature, wherein the first target text feature includes a word-element feature of a target part of speech;

[0130] Determining an optimal match between the fused query vector and the first target text feature;

[0131] Based on the optimal match, obtaining a first matching loss between the fused query vector and the first target text feature;

[0132] The first matching loss is used as a first local loss function.

[0133] Here, a set of learnable query vectors and the first video feature are fed into the perceptron to obtain a fused query vector output by the perceptron. , which is used to capture visual concepts present in video data, such as objects, attributes, and actions.

[0134] Here, the target part of speech is the part of speech of a meaningful word in the text data obtained by using a text parsing tool, such as a verb or a noun. In this embodiment, the target part of speech in the text data is obtained by using a text parsing tool, and then the first target text feature corresponding to the target part of speech is obtained. .

[0135] Here, the best match refers to the first target text feature that best matches a certain fused query vector among multiple first target text features, or the fused query vector that best matches a certain first target text feature among multiple fused query vectors. In this embodiment, after obtaining the fused query vector and the filtered first target text feature, the similarity between the two is calculated, and one or more best matches are selected based on the similarity.

[0136] Finally, the best match is used as supervision information, and the matching loss between the fused query vector and the first target text feature is optimized through NCE to calculate the first matching loss function of this process:

[0137] ;

[0138] in, is the temperature hyperparameter, is the number of valid tokens in the entire batch, Refers to the first target text feature The best match in the fused query vector obtained using bipartite matching The similarity between .

[0139] In some embodiments, determining the optimal match between the fused query vector and the first target text feature includes:

[0140] Determining a cosine similarity between the fused query vector and the first target text feature;

[0141] Taking the cosine similarity as the matching cost, constructing an optimal matching cost mapping function between the fused query vector and the first target text feature;

[0142] Based on the optimal matching cost mapping function, an optimal match between the fused query vector and the first target text feature is determined.

[0143] In this embodiment, the first target text feature obtained by using bipartite matching is calculated The fused query vector With the first target text feature The cosine similarity between , construct the optimal matching cost mapping function of the fused query vector and the first target text feature:

[0144] ;

[0145] Here, the optimal matching cost mapping function can be quickly calculated using the Hungarian algorithm to obtain the optimal match.

[0146] In some embodiments, the cross-modal pre-training model is trained based on the first global loss function, the second global loss function, the first local loss function, the second local loss function, and the mask loss function corresponding to the text mask feature to obtain a trained cross-modal pre-training model, including:

[0147] In a first training phase, the video encoder and the perceptron are optimized based on the first global loss function and the first local loss function;

[0148] In a second training phase, the video encoder, the perceptron, and the text encoder are optimized based on the second global loss function, the second local loss function, and a mask loss function corresponding to the text mask feature;

[0149] Based on the first training stage and the second training stage, a trained cross-modal pre-training model is obtained.

[0150] In this embodiment, the entire model training is divided into two training stages.

[0151] In the first training stage, the parameters of the text encoder are frozen, that is, the parameters of the text encoder are kept unchanged, and the video encoder and the perceptron are optimized only through the first global loss function and the first local loss function.

[0152] In the second training stage, the video encoder, perceptron, and text encoder are jointly optimized. It should be noted that in the second training stage, since unfreezing the parameters of the text encoder will weaken the semantic anchoring ability of the text encoder, a mask loss function corresponding to the text mask feature is also used in the second training stage to maintain the semantic anchoring ability of the text encoder.

[0153] That is, in the second training stage, the text encoder is optimized once using the mask loss function corresponding to the text mask feature, and then the video encoder, perceptron and text encoder are jointly optimized using the second global loss function and the second local loss function.

[0154] It should be noted here that, since in this embodiment, text data with masked word units is input to the text encoder through the MLM task, and then the text encoder is required to output the corresponding text mask features, the mask loss function corresponding to the text mask features in this embodiment can be expressed as follows:

[0155] ;

[0156] in, represents the true vector of the masked word, represents the predicted vector of the masked word, Represents text data with masked tokens.

[0157] In this embodiment, the model is trained by using a global loss function for cross-modal alignment under global features and a global loss function for cross-modal alignment under local features, thereby achieving cross-modal alignment in an explicit vocabulary space during the training phase, which is generalizable.

[0158] Based on any of the above embodiments, the present invention also provides a cross-modal model training device, Figure 4 is a schematic diagram of the structure of the training device of the cross-modal model provided by the present invention, such as Figure 4 As shown, the device comprises:

[0159] A first cross-modal model training module 410, used to construct a cross-modal pre-training model, wherein the cross-modal pre-training model includes a video encoder, a text encoder, and a perceptron;

[0160] A training module 420 of a second cross-modal model, configured to extract a first video feature of the video data through the video encoder, and to extract a first text feature and a text mask feature of the text data matched by the video data through the text encoder;

[0161] A training module 430 of a third cross-modal model is used to map the first video feature and the first text feature to the same vocabulary space to obtain a corresponding second video feature and a second text feature, and to map the first video feature and the text mask feature to the same vocabulary space to obtain a corresponding third video feature and a third text feature;

[0162] A training module 440 of a fourth cross-modal model, configured to obtain a first global loss function based on the second video feature and the second text feature, and to obtain a second global loss function based on the second video feature and the second text feature;

[0163] A training module 450 of a fifth cross-modal model is configured to obtain a first local loss function based on the first video feature, the first text feature, and the perceptron; and obtain a second local loss function based on the first video feature, the text mask feature, and the perceptron;

[0164] The training module 460 of the sixth cross-modal model is used to train the cross-modal pre-trained model based on the first global loss function, the second global loss function, the first local loss function, the second local loss function and the mask loss function corresponding to the text mask feature to obtain a trained cross-modal pre-trained model.

[0165] The device provided by the embodiment of the present invention alleviates the problem of the modal gap by learning semantically anchored feature representations in an explicit vocabulary space. In addition, in order to realize the semantically anchored characteristics of feature representations in the vocabulary space, the present invention also proposes a training strategy for multi-granularity video semantics and text semantics based on this, that is, model training is performed by using a global loss function for cross-modal alignment under global features and a global loss function for cross-modal alignment under local features, thereby achieving cross-modal alignment in an explicit vocabulary space during the training phase, which is generalizable.

[0166] The cross-modal model training device described in this embodiment and the cross-modal model training method provided by the present invention described above can be referenced to each other and will not be described in detail here.

[0167] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the cross-modal model training method, which includes:

[0168] Constructing a cross-modal pre-training model, wherein the cross-modal pre-training model includes a video encoder, a text encoder, and a perceptron;

[0169] Extracting a first video feature of the video data by the video encoder, and extracting a first text feature and a text mask feature of the text data matched by the video data by the text encoder;

[0170] Mapping the first video feature and the first text feature to the same vocabulary space to obtain corresponding second video features and second text features, and mapping the first video feature and the text mask feature to the same vocabulary space to obtain corresponding third video features and third text features;

[0171] Based on the second video feature and the second text feature, obtaining a first global loss function, and based on the second video feature and the text mask feature, obtaining a second global loss function;

[0172] Based on the first video feature, the first text feature and the perceptron, obtaining a first local loss function; and based on the first video feature, the text mask feature and the perceptron, obtaining a second local loss function;

[0173] The cross-modal pre-training model is trained based on the first global loss function, the second global loss function, the first local loss function, the second local loss function, and the mask loss function corresponding to the text mask feature to obtain a trained cross-modal pre-training model.

[0174] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0175] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer can execute the cross-modal model training method provided by the above methods, the method comprising:

[0176] Constructing a cross-modal pre-training model, wherein the cross-modal pre-training model includes a video encoder, a text encoder, and a perceptron;

[0177] Extracting a first video feature of the video data by the video encoder, and extracting a first text feature and a text mask feature of the text data matched by the video data by the text encoder;

[0178] Mapping the first video feature and the first text feature to the same vocabulary space to obtain corresponding second video features and second text features, and mapping the first video feature and the text mask feature to the same vocabulary space to obtain corresponding third video features and third text features;

[0179] Based on the second video feature and the second text feature, obtaining a first global loss function, and based on the second video feature and the text mask feature, obtaining a second global loss function;

[0180] Based on the first video feature, the first text feature and the perceptron, obtaining a first local loss function; and based on the first video feature, the text mask feature and the perceptron, obtaining a second local loss function;

[0181] The cross-modal pre-training model is trained based on the first global loss function, the second global loss function, the first local loss function, the second local loss function, and the mask loss function corresponding to the text mask feature to obtain a trained cross-modal pre-training model.

[0182] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the training method of the cross-modal model provided by the above methods, the method comprising:

[0183] Constructing a cross-modal pre-training model, wherein the cross-modal pre-training model includes a video encoder, a text encoder, and a perceptron;

[0184] Extracting a first video feature of the video data by the video encoder, and extracting a first text feature and a text mask feature of the text data matched by the video data by the text encoder;

[0185] Mapping the first video feature and the first text feature to the same vocabulary space to obtain corresponding second video features and second text features, and mapping the first video feature and the text mask feature to the same vocabulary space to obtain corresponding third video features and third text features;

[0186] Based on the second video feature and the second text feature, obtaining a first global loss function, and based on the second video feature and the text mask feature, obtaining a second global loss function;

[0187] Based on the first video feature, the first text feature and the perceptron, obtaining a first local loss function; and based on the first video feature, the text mask feature and the perceptron, obtaining a second local loss function;

[0188] The cross-modal pre-training model is trained based on the first global loss function, the second global loss function, the first local loss function, the second local loss function, and the mask loss function corresponding to the text mask feature to obtain a trained cross-modal pre-training model.

[0189] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0190] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0191] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-modal model training method, It is characterized in that The method comprises: Constructing a cross-modal pre-training model, wherein the cross-modal pre-training model includes a video encoder, a text encoder, and a perceptron; Extracting a first video feature of the video data by the video encoder, and extracting a first text feature and a text mask feature of the text data matched by the video data by the text encoder; Mapping the first video feature and the first text feature to the same vocabulary space to obtain a corresponding second video feature and a second text feature, and mapping the first video feature and the text mask feature to the same vocabulary space to obtain a corresponding third video feature and a third text feature; Based on the second video feature and the second text feature, obtaining a first global loss function, and based on the third video feature and the third text feature, obtaining a second global loss function; Based on the first video feature, the first text feature and the perceptron, obtaining a first local loss function; and based on the first video feature, the text mask feature and the perceptron, obtaining a second local loss function; Training the cross-modal pre-training model based on the first global loss function, the second global loss function, the first local loss function, the second local loss function, and the mask loss function corresponding to the text mask feature to obtain a trained cross-modal pre-training model; The obtaining a first global loss function based on the second video feature and the second text feature includes: Performing a maximum pooling-based aggregation operation on the second video feature and the second text feature respectively to obtain a video global feature corresponding to the second video feature and a text global feature corresponding to the second text feature; Based on the video global feature and the text global feature, obtaining a first contrastive learning loss function and a first FLOPs loss function; Using the first contrastive learning loss function and the first FLOPs loss function as a first global loss function; The cross-modal pre-training model is trained based on the first global loss function, the second global loss function, the first local loss function, the second local loss function, and the mask loss function corresponding to the text mask feature to obtain a trained cross-modal pre-training model, including: In a first training phase, the video encoder and the perceptron are optimized based on the first global loss function and the first local loss function; In a second training phase, the video encoder, the perceptron, and the text encoder are optimized based on the second global loss function, the second local loss function, and a mask loss function corresponding to the text mask feature; Based on the first training stage and the second training stage, a trained cross-modal pre-training model is obtained.

2. The cross-modal model training method according to claim 1, It is characterized in that The mapping the first video feature and the first text feature to the same vocabulary space to obtain corresponding second video features and second text features includes: A mapping head that builds a vocabulary space based on the pre-trained masked language head; Performing feature mapping on the first video feature based on the mapping head, and performing dimension suppression processing on the feature-mapped video feature to obtain a second video feature corresponding to the first video feature; Feature mapping is performed on the first text feature based on the mapping head, and dimension suppression processing is performed on the text feature after feature mapping to obtain a second text feature corresponding to the first text feature.

3. The cross-modal model training method according to claim 2, It is characterized in that The method further comprises: The dimension of the video features after feature mapping is suppressed based on the Relu activation function and the logarithmic transformation function; The dimension of the text features after feature mapping is suppressed based on the Relu activation function and the logarithmic transformation function.

4. The cross-modal model training method according to claim 1, It is characterized in that The obtaining a first local loss function based on the first video feature, the first text feature, and the perceptron includes: Inputting the first video feature and the learnable query vector into the perceptron to obtain a fused query vector output by the perceptron; Filtering out a first target text feature from the first text feature, wherein the first target text feature includes a word-element feature of a target part of speech; Determining an optimal match between the fused query vector and the first target text feature; Based on the optimal match, obtaining a first matching loss between the fused query vector and the first target text feature; The first matching loss is used as a first local loss function.

5. The cross-modal model training method according to claim 4, It is characterized in that The determining the optimal match between the fused query vector and the first target text feature includes: Determining a cosine similarity between the fused query vector and the first target text feature; Taking the cosine similarity as the matching cost, constructing an optimal matching cost mapping function between the fused query vector and the first target text feature; Based on the optimal matching cost mapping function, an optimal match between the fused query vector and the first target text feature is determined.

6. A training device for a cross-modal model, It is characterized in that The device comprises: A first cross-modal model training module, used to construct a cross-modal pre-training model, wherein the cross-modal pre-training model includes a video encoder, a text encoder, and a perceptron; A training module of a second cross-modal model, configured to extract a first video feature of the video data through the video encoder, and to extract a first text feature and a text mask feature of the text data matched with the video data through the text encoder; a training module of a third cross-modal model, configured to map the first video feature and the first text feature to the same vocabulary space to obtain corresponding second video features and second text features, and to map the first video feature and the text mask feature to the same vocabulary space to obtain corresponding third video features and third text features; a training module of a fourth cross-modal model, configured to obtain a first global loss function based on the second video feature and the second text feature, and to obtain a second global loss function based on the third video feature and the third text feature; a training module of a fifth cross-modal model, configured to obtain a first local loss function based on the first video feature, the first text feature, and the perceptron; and, to obtain a second local loss function based on the first video feature, the text mask feature, and the perceptron; a sixth cross-modal model training module, configured to train the cross-modal pre-trained model based on the first global loss function, the second global loss function, the first local loss function, the second local loss function, and the mask loss function corresponding to the text mask feature to obtain a trained cross-modal pre-trained model; The obtaining a first global loss function based on the second video feature and the second text feature includes: Performing a maximum pooling-based aggregation operation on the second video feature and the second text feature respectively to obtain a video global feature corresponding to the second video feature and a text global feature corresponding to the second text feature; Based on the video global feature and the text global feature, obtaining a first contrastive learning loss function and a first FLOPs loss function; Using the first contrastive learning loss function and the first FLOPs loss function as a first global loss function; The cross-modal pre-training model is trained based on the first global loss function, the second global loss function, the first local loss function, the second local loss function, and the mask loss function corresponding to the text mask feature to obtain a trained cross-modal pre-training model, including: In a first training phase, the video encoder and the perceptron are optimized based on the first global loss function and the first local loss function; In a second training phase, the video encoder, the perceptron, and the text encoder are optimized based on the second global loss function, the second local loss function, and a mask loss function corresponding to the text mask feature; Based on the first training stage and the second training stage, a trained cross-modal pre-training model is obtained.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the training method of the cross-modal model as described in any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the training method of the cross-modal model as described in any one of claims 1 to 5 is implemented.