Multimodal pre-training methods and apparatuses

Abstract

The disclosure provides a multimodal pre-training method and device. The multimodal pre-training method comprises: sampling a video in a video-text pair to obtain a first video frame sequence; performing word segmentation processing on the text in the video-text pair to obtain a first word segmentation sequence; performing mask processing on the first video frame sequence to obtain a second video frame sequence; performing mask processing on the first word segmentation sequence to obtain a second word segmentation sequence; encoding the first video frame sequence to obtain a first video feature, and encoding the first word segmentation sequence to obtain a first word segmentation feature; encoding the second video frame sequence to obtain a second video feature, and encoding the second word segmentation sequence to obtain a second word segmentation feature; determining a pre-training objective function using the first video feature, the first word segmentation feature, the second video feature, and the second word segmentation feature; and performing multimodal pre-training using the objective function.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of information processing, and in particular, to a multimodal pre-training method and device. BACKGROUND

[0002] The visual language multimodal pre-training technology is one of the emerging topics in the field of multimodal in recent years. The purpose is to enable the model to be pre-trained on a large scale of weakly annotated visual (such as image, video) and text data pairs to obtain a better multimodal feature representation, thereby improving the performance of various multimodal downstream task models.

[0003] The related technologies of visual language multimodal pre-training are basically borrowed from the method of BERT (Bidirectional Encoder Representations From Transformer, bidirectional encoder representations from transformer) pre-training model in the field of natural language processing. SUMMARY

[0004] The inventors have noticed that in the related art, the video text multimodal pre-training technology only utilizes the input video text with mask to learn the global feature representation correlation during pre-training in order to explore the correlation between the two modalities. This learning method does not fully explore the overall video-text relationship between the input video frames and the word sequence, thereby leading to a decline in the quality of multimodal features.

[0005] Accordingly, the present disclosure provides a multimodal pre-training scheme that can enhance the correlation between cross-modal data and effectively improve the understanding ability of the multimodal pre-training model for the content of the multimodal data.

[0006] According to a first aspect of an embodiment of the present disclosure, a multimodal pre-training method is provided, comprising: sampling a video in a video-text pair to obtain a first video frame sequence; performing word segmentation processing on a text in the video-text pair to obtain a first word segmentation sequence; performing mask processing on the first video frame sequence to obtain a second video frame sequence; performing mask processing on the first word segmentation sequence to obtain a second word segmentation sequence; encoding the first video frame sequence to obtain a first video feature, and encoding the first word segmentation sequence to obtain a first word segmentation feature; encoding the second video frame sequence to obtain a second video feature, and encoding the second word segmentation sequence to obtain a second word segmentation feature; determining a pre-training objective function using the first video feature, the first word segmentation feature, the second video feature, and the second word segmentation feature; and performing multimodal pre-training using the pre-training objective function.

[0007] In some embodiments, determining the pre-trained objective function comprises: determining a first contrastive loss value by using the first segmented feature, the second video feature, and a preset first negative sample feature; determining a second contrastive loss value by using the first video feature, the second segmented feature, and a preset second negative sample feature; determining a first target according to the first contrastive loss value and the second contrastive loss value; determining a third contrastive loss value by using the first video feature, the second video feature, and the second negative sample feature; determining a fourth contrastive loss value by using the first segmented feature, the second segmented feature, and the first negative sample feature; determining a second target according to the third contrastive loss value and the fourth contrastive loss value; and determining the objective function according to the first target and the second target.

[0008] In some embodiments, determining the first contrastive loss value comprises: converting the first segmented feature into a global first positive sample feature; converting the second video feature into a global video query feature; and determining the first contrastive loss value by using the video query feature, the first positive sample feature, and the first negative sample feature.

[0009] In some embodiments, determining the second contrastive loss value comprises: converting the first video feature into a global second positive sample feature; converting the second segmented feature into a global text query feature; and determining the second contrastive loss value by using the text query feature, the second positive sample feature, and the second negative sample feature.

[0010] In some embodiments, determining the third contrastive loss value comprises: determining the third contrastive loss value by using the video query feature, the second positive sample feature, and the second negative sample feature.

[0011] In some embodiments, determining the fourth contrastive loss value comprises: determining the fourth contrastive loss value by using the text query feature, the first positive sample feature, and the first negative sample feature.

[0012] In some embodiments, the first target is a sum of the first contrastive loss value and the second contrastive loss value; and the second target is a sum of the third contrastive loss value and the fourth contrastive loss value.

[0013] In some embodiments, the objective function is a sum of the first target and the second target.

[0014] In some embodiments, the second video feature and the second tokenization feature are fused to obtain a fusion feature; the fusion feature is input into a masked language modeling (MLM) model to obtain a third target, and the fusion feature is input into a masked sequence generation (MSG) model to obtain a fourth target; and the target function is determined according to the first target, the second target, the third target, and the fourth target.

[0015] In some embodiments, the target function is a sum of the first target, the second target, the third target, and the fourth target.

[0016] According to a second aspect of the embodiments of the present disclosure, a multi-modal pre-training apparatus is provided, including: a first processing module configured to sample a video in a video-text pair to obtain a first video frame sequence, and configured to perform tokenization processing on a text in the video-text pair to obtain a first tokenization sequence; a second processing module configured to perform mask processing on the first video frame sequence to obtain a second video frame sequence, and configured to perform mask processing on the first tokenization sequence to obtain a second tokenization sequence; a third processing module configured to encode the first video frame sequence to obtain a first video feature, and configured to encode the first tokenization sequence to obtain a first tokenization feature; a fourth processing module configured to encode the second video frame sequence to obtain a second video feature, and configured to encode the second tokenization sequence to obtain a second tokenization feature; a fifth processing module configured to determine a pre-training target function by using the first video feature, the first tokenization feature, the second video feature, and the second tokenization feature; and a sixth processing module configured to perform multi-modal pre-training by using the pre-training target function.

[0017] According to a third aspect of the embodiments of the present disclosure, a multi-modal pre-training apparatus is provided, including: a memory configured to store instructions; and a processor coupled to the memory, the processor being configured to execute the instructions stored in the memory to implement the method according to any one of the above embodiments.

[0018] According to a fourth aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, where the computer readable storage medium stores computer instructions, and the instructions are executed by a processor to implement the method according to any one of the above embodiments.

[0019] Other features and advantages of the present disclosure will be apparent from the following detailed description of exemplary embodiments thereof, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only represent some of the embodiments of the present disclosure, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0021] Figure 1 Flowchart of a multimodal pre-training method according to an embodiment of the present disclosure;

[0022] Figure 2 Flowchart of a multimodal pre-training method according to another embodiment of the present disclosure;

[0023] Figure 3 Structure diagram of a multimodal pre-training device according to an embodiment of the present disclosure;

[0024] Figure 4 Structure diagram of a multimodal pre-training device according to another embodiment of the present disclosure;

[0025] Figure 5 Schematic diagram of a multimodal pre-training model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only some of the embodiments of the present disclosure, but not all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative, but not as any limitation on the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present disclosure.

[0027] Unless otherwise specified, the relative arrangement, numerical expressions and values of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0028] At the same time, it should be understood that, for the convenience of description, the sizes of the various parts shown in the drawings are not drawn in accordance with the actual proportional relationship.

[0029] The technology, methods and devices known to those skilled in the relevant art can not be discussed in detail, but in appropriate cases, the technology, methods and devices should be considered as part of the authorized description.

[0030] In all examples shown and discussed here, any specific value should be interpreted as merely exemplary, and not as a limitation. Therefore, other examples of exemplary embodiments can have different values.

[0031] It should be noted that like reference numerals and characters refer to like elements throughout the following figures and the detailed description, not upon the drawings alone such that, when certain terms are used in the following detailed description and / or in the claims (such as those terms set forth above), that meaning is maintained even though some of these terms are not explicitly described as maintained anywhere else in the following description.

[0032] Figure 1 A flowchart of a multi-modal pre-training method for an embodiment of the present disclosure. In some embodiments, the following multi-modal pre-training method is performed by a multi-modal pre-training device.

[0033] At step 101, the video in the video-text pair is sampled to obtain a first video frame sequence, and the text in the video-text pair is tokenized to obtain a first token sequence.

[0034] In some embodiments, the video is sampled in an equidistant manner to obtain the first video frame sequence.

[0035] In some embodiments, a mark [CLS] and a mark [SEP] are respectively arranged at the beginning and the end of the first token sequence for subsequent processing convenience.

[0036] At step 102, the first video frame sequence is masked to obtain a second video frame sequence, and the first token sequence is masked to obtain a second token sequence.

[0037] In some embodiments, the video frames in the first video frame sequence are replaced by masks with a random probability to obtain the second video frame sequence.

[0038] In some embodiments, the tokens in the first token sequence are replaced by masks with a random probability to obtain the second token sequence.

[0039] At step 103, the first video frame sequence is encoded to obtain first video features, and the first token sequence is encoded to obtain first token features.

[0040] In some embodiments, the first video frame sequence is encoded using a video key encoder to obtain the first video features, and the first token sequence is encoded using a text key encoder to obtain the first token features.

[0041] The first video features output by the video key encoder reflect the contextual characteristics of the unmasked video frames. The first token features output by the text key reflect the contextual characteristics of the unmasked token sequence.

[0042] Since the video key and the text key are not the points of the present disclosure, they will not be described here.

[0043] At step 104, the second video frame sequence is encoded to obtain second video features, and the second token sequence is encoded to obtain second token features.

[0044] In some embodiments, the second video frame sequence is encoded using a video query encoder to obtain the second video features, and the second token sequence is encoded using a text query encoder to obtain the second token features.

[0045] The second video features output by the video query encoder reflect the relevance between frames in the video modality, and the second token features output by the text query encoder reflect the relevance between tokens in the text modality.

[0046] Since the video query encoder and the text query encoder are not the invention points of the present disclosure, they will not be described here.

[0047] At step 105, a pre-trained objective function is determined using the first video features, the first token features, the second video features, and the second token features.

[0048] In some embodiments, the pre-trained objective function is determined as shown in Figure 2 .

[0049] At step 201, a first contrastive loss value is determined using the first token features, the second video features, and a preset first negative sample feature.

[0050] In some embodiments, an MLP (Multi-layer Perceptron) model is used to convert the first token features into a global first positive sample feature An MLP model is used to convert the second video features into a global video query feature The video query feature The first positive sample feature And the first negative sample feature The first contrastive loss value is determined.

[0051] It should be noted that the first negative sample feature is:

[0052]

[0053] Where K represents the size of the negative sample queue included in the first negative sample feature, represents the i-th negative sample in the negative sample queue.

[0054] In some embodiments, the first contrast loss value is calculated using formula (2)

[0055]

[0056] where t is a hyper-parameter for controlling scaling. The operator <A, B> represents the cosine similarity of vectors A and B.

[0057] At step 202, a second contrast loss value is determined using the first video feature, the second tokenized feature, and a preset second negative sample feature.

[0058] In some embodiments, the first video feature is converted into a global second positive sample feature using an MLP model The second tokenized feature is converted into a global text query feature using an MLP model The text query feature is used to determine the second contrast loss value. The second positive sample feature and the second negative sample feature determine the second contrast loss value.

[0059] It should be noted that the second negative sample feature is:

[0060]

[0061] where K represents the size of the negative sample queue included in the second negative sample feature, represents the i-th negative sample in the negative sample queue.

[0062] In some embodiments, the second contrast loss value is calculated using formula (4)

[0063]

[0064] where t is a hyper-parameter for controlling scaling. The operator <A, B> represents the cosine similarity of vectors A and B.

[0065] At step 203, a first target is determined according to the first contrast loss value and the second contrast loss value.

[0066] In some embodiments, the first target is the sum of the first contrast loss value and the second contrast loss value. For example, the first target is calculated using formula (5). The first target is used to represent the combination of the video-to-text and text-to-video video matching loss.

[0067]

[0068] At step 204, a third contrast loss value is determined using the first video feature, the second video feature, and the second negative sample feature.

[0069] In some embodiments, the video query feature is utilized the second positive sample feature and the second negative sample feature determining a third contrastive loss value.

[0070] In some embodiments, the third contrastive loss value is calculated using formula (6)

[0071]

[0072] where t is a hyper-parameter for controlling scaling. The operator <A, B> represents the cosine similarity of vectors A and B.

[0073] In step 205, a fourth contrastive loss value is determined using the first tokenized feature, the second tokenized feature, and the first negative sample feature.

[0074] In some embodiments, the text query feature is utilized the first positive sample feature and the first negative sample feature determining a fourth contrastive loss value.

[0075] In some embodiments, the fourth contrastive loss value is calculated using formula (7)

[0076]

[0077] where t is a hyper-parameter for controlling scaling. The operator <A, B> represents the cosine similarity of vectors A and B.

[0078] In step 206, a second target is determined according to the third contrastive loss value and the fourth contrastive loss value.

[0079] In some embodiments, the second target is the sum of the third contrastive loss value and the fourth contrastive loss value. For example, the second target is calculated using formula (8). The second target is used to represent the denoising loss within the video modality and within the text modality.

[0080]

[0081] In step 207, a target function is determined according to the first target and the second target.

[0082] In some embodiments, the target function is the sum of the first target and the second target. For example, the target function L is calculated using formula (9).

[0083] L = L Co-IM + L Co-5D (9)

[0084] Return Figure 1 In step 106, the multi-modal pre-training is performed by using the pre-trained objective function.

[0085] In the multi-modal pre-training method provided by the above-mentioned embodiments of the present disclosure, the pre-trained objective function is determined based on the cross-modal matching loss and the intra-modal denoising loss, which can enhance the correlation between cross-modal data and effectively improve the understanding ability of the multi-modal pre-training model for the content of multi-modal data.

[0086] In some embodiments, the second video feature and the second tokenized feature are fused to obtain a fused feature. The fused feature is input into an MLM (Masked Language Modelling) model to obtain a third target L MLM The fused feature is input into an MSG (Masked Language Generation) model to obtain a fourth target L MSG .

[0087] In some embodiments, the second video feature and the second tokenized feature are fused using a cross-modal decoder to obtain a fused feature. The cross-modal decoder is used to output the fused feature of the video and text multi-modal information, and provide feature input for subsequent tasks.

[0088] Since the cross-modal decoder is not the point of the present disclosure, it will not be described here.

[0089] In some embodiments, the objective function L is determined according to the first target L Co-5M , the second target L Co-5D , the third target L MLM and the fourth target L MSG .

[0090] In some embodiments, the objective function L is the sum of the first target L Co-5M , the second target L Co-5D , the third target L MLM and the fourth target L MSG .

[0091] For example, the objective function L is calculated by using the following formula (10).

[0092] L = L Co-5M + L Co-5D + L MLM + L MSG (10)

[0093] Figure 3Structural schematic diagram of a multi-modal pre-training device for an embodiment of the present disclosure. As shown in Figure 3 The multi-modal pre-training device includes a first processing module 31, a second processing module 32, a third processing module 33, a fourth processing module 34, a fifth processing module 35, and a sixth processing module 36.

[0094] The first processing module 31 is configured to sample the video in the video-text pair to obtain a first video frame sequence, and is also configured to perform word segmentation processing on the text in the video-text pair to obtain a first word sequence.

[0095] In some embodiments, the video is sampled in an equidistant sampling manner to obtain the first video frame sequence.

[0096] In some embodiments, a mark [CLS] and a mark [SEP] are respectively arranged at the beginning and the end of the first word sequence, so as to facilitate subsequent processing.

[0097] The second processing module 32 is configured to perform mask processing on the first video frame sequence to obtain a second video frame sequence, and is also configured to perform mask processing on the first word sequence to obtain a second word sequence.

[0098] In some embodiments, the video frames in the first video frame sequence are replaced by masks with a random probability to obtain the second video frame sequence.

[0099] In some embodiments, the word sequences in the first word sequence are replaced by masks with a random probability to obtain the second word sequence.

[0100] The third processing module 33 is configured to encode the first video frame sequence to obtain first video features, and is also configured to encode the first word sequence to obtain first word features.

[0101] In some embodiments, the first video frame sequence is encoded using a video key-value encoder to obtain the first video features, and the first word sequence is encoded using a text key-value encoder to obtain the first word features.

[0102] The first video features output by the video key-value encoder reflect the contextual characteristics of the video frames without masks. The first word features output by the text key-value encoder reflect the contextual characteristics of the word sequence without masks.

[0103] The fourth processing module 34 is configured to encode the second video frame sequence to obtain second video features, and is also configured to encode the second word sequence to obtain second word features.

[0104] In some embodiments, the second video sequence is encoded using the video query encoder to obtain second video features, and the second token sequence is encoded using the text query encoder to obtain second token features.

[0105] The second video features output by the video query encoder reflect the relevance between frames in the video modality, and the second token features output by the text query encoder reflect the relevance between words in the text modality.

[0106] The fifth processing module 35 is configured to determine a pre-trained objective function using the first video features, the first token features, the second video features, and the second token features. In some embodiments, the fifth processing module 35 determines a first contrastive loss value using the first token features, the second video features, and a preset first negative sample feature.

[0107] For example, the first token features are converted into global first positive sample features using an MLP model The second video features are converted into global video query features using an MLP model The video query features are used to determine the first contrastive loss value. The first positive sample features and the first negative sample features to determine the first contrastive loss value.

[0108] In some embodiments, the first contrastive loss value is calculated using the above formula (2)

[0109] The fifth processing module 35 determines a second contrastive loss value using the first video features, the second token features, and a preset second negative sample feature. For example, the first video features are converted into global second positive sample features using an MLP model The second token features are converted into global text query features using an MLP model The text query features are used to determine the second contrastive loss value. The second positive sample features and the second negative sample features to determine the second contrastive loss value.

[0110] In some embodiments, the second contrastive loss value is calculated using the above formula (4)

[0111] The fifth processing module 35 determines a first objective according to the first contrastive loss value and the second contrastive loss value. In some embodiments, the first objective is the sum of the first contrastive loss value and the second contrastive loss value. For example, the first objective is calculated using the above formula (5). The first objective is used to represent the combination of the video-to-text and text-to-video video matching loss.

[0112] The fifth processing module 35 determines a third contrast loss value by using the first video feature, the second video feature and the second negative sample feature. In some embodiments, the video query feature The second positive sample feature and the second negative sample feature determines the third contrast loss value. For example, the third contrast loss value is calculated by using the above formula (6)

[0113] The fifth processing module 35 determines a fourth contrast loss value by using the first word segmentation feature, the second word segmentation feature and the first negative sample feature. In some embodiments, the text query feature The first positive sample feature and the first negative sample feature determines the fourth contrast loss value.

[0114] In some embodiments, the fourth contrast loss value is calculated by using the above formula (7)

[0115] The fifth processing module 35 determines a second target according to the third contrast loss value and the fourth contrast loss value. In some embodiments, the second target is the sum of the third contrast loss value and the fourth contrast loss value. For example, the second target is calculated by using the above formula (8). The second target is used to represent the denoising loss within the video modality and within the text modality.

[0116] The fifth processing module 35 determines a target function according to the first target and the second target. In some embodiments, the target function is the sum of the first target and the second target. For example, the target function L is calculated by using the above formula (9).

[0117] In some embodiments, the fifth processing module 35 performs fusion processing on the second video feature and the second word segmentation feature to obtain a fusion feature. The fusion feature is input into the MLM model to obtain a third target L MLM , and the fusion feature is input into the MSG model to obtain a fourth target L MSG .

[0118] In some embodiments, the second video feature and the second word segmentation feature are fused by using a cross-modal decoder to obtain a fusion feature. The cross-modal decoder is used to output the fusion feature of the video and text multi-modal information, and provide feature input for subsequent tasks.

[0119] In some embodiments, the target function L is determined according to the first target L Co-5M , the second target L Co-5D , the third target L MLM and the fourth target L MSG . In some embodiments, the target function L is the sum of the first target L Co-5MSecond objective L Co-5D Third goal L MLM and the fourth target L MSG The sum of these. For example, the objective function L can be calculated using the formula (10) above.

[0120] The sixth processing module 36 is configured to perform multimodal pre-training using a pre-trained objective function.

[0121] Figure 4 This is a schematic diagram of the structure of a multimodal pre-training device according to another embodiment of this disclosure. Figure 4 As shown, the multimodal pre-training device includes a memory 41 and a processor 42.

[0122] Memory 41 is used to store instructions, and processor 42 is coupled to memory 41. Processor 42 is configured to execute instructions based on the memory storage, as shown in the example. Figure 1 or Figure 2 The method involved in any of the embodiments.

[0123] like Figure 4 As shown, the multimodal pre-training device also includes a communication interface 43 for exchanging information with other devices. Additionally, the multimodal pre-training device includes a bus 44, through which the processor 42, communication interface 43, and memory 41 communicate with each other.

[0124] The memory 41 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device. The memory 41 may also be a memory array. The memory 41 may also be divided into blocks, and the blocks may be combined into virtual volumes according to certain rules.

[0125] Furthermore, processor 42 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present disclosure.

[0126] This disclosure also relates to a computer-readable storage medium storing computer instructions that, when executed by a processor, implement... Figure 1 or Figure 2 The method involved in any of the embodiments.

[0127] Figure 5 This is a schematic diagram of a multimodal pre-trained model according to an embodiment of the present disclosure.

[0128] like Figure 5As shown, the video in the video-text pair is sampled to obtain a first video frame sequence, and the text in the video-text pair is segmented to obtain a first segmentation sequence. A mask is used to replace video frames in the first video frame sequence with a random probability to obtain a second video frame sequence. A mask is used to replace segmentation in the first segmentation sequence with a random probability to obtain a second segmentation sequence.

[0129] The first video frame sequence is encoded using a video key-value encoder to obtain first video features, and the first segmentation sequence is encoded using a text key-value encoder to obtain first segmentation features.

[0130] The second video frame sequence is encoded using a video query encoder to obtain second video features, and the second segmentation sequence is encoded using a text query encoder to obtain second segmentation features.

[0131] The first segmentation features are converted into global first positive sample features using an MLP model The first video features are converted into global second positive sample features using an MLP model The second video features are converted into global video query features using an MLP model The second segmentation features are converted into global text query features using an MLP model

[0132] In the Co-IM (Contrastive Inter-modal Matching) module, according to the above formula (2), the video query features The first positive sample features and the first negative sample features determine the first contrastive loss value

[0133] According to the above formula (4), the text query features The second positive sample features and the second negative sample features determine the second contrastive loss value

[0134] Next, the first target L is calculated using the above formula (5) C4-IM .

[0135] In the Co-ID (Contrastive Intra-modal Denoising) module, according to the above formula (6), the video query features The second positive sample features and the second negative sample features determine the third contrastive loss value

[0136] According to the above formula (7), the text query feature The first positive sample feature And the first negative sample feature Determine the fourth contrast loss value

[0137] Next, according to the above formula (8), the second target L is determined according to the third contrast loss value and the fourth contrast loss value C4-ID .

[0138] In addition, the second video feature and the second tokenization feature are fused using the cross-modal decoder to obtain a fusion feature. The fusion feature is input into the MLM model to obtain the third target L MLM , and the fusion feature is input into the MSG model to obtain the fourth target L MSG .

[0139] Next, according to the above formula (10), the sum of the first target L Co-IM , the second target L C4-ID , the third target L MLM and the fourth target L MSG is taken as the target function L.

[0140] In some embodiments, the functional unit modules described above can be implemented as a general-purpose processor, a programmable logic controller (PLC), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or any appropriate combination thereof, for executing the functions described in the present disclosure.

[0141] A person of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by a program instructing related hardware, and the program can be stored in a computer-readable storage medium, and the storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0142] The description of the present disclosure is given for illustrative and descriptive purposes, and is not exhaustive or limited to the forms disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are chosen and described in order to best explain the principles of the present disclosure and its practical application, and to thereby enable others skilled in the art to best utilize the present disclosure for various embodiments with various modifications as are suited to the particular use contemplated.

Claims

1. A multimodal pre-training method, comprising: The video in the video-text pair is sampled to obtain the first video frame sequence; The text in the video-text pair is segmented to obtain the first segmentation sequence; The first video frame sequence is subjected to random masking to obtain the second video frame sequence; The first word segmentation sequence is subjected to random masking to obtain the second word segmentation sequence; The first video frame sequence is encoded to obtain the first video feature, and the first word segmentation sequence is encoded to obtain the first word segmentation feature; The second video frame sequence is encoded to obtain the second video features, and the second word segmentation sequence is encoded to obtain the second word segmentation features; Using the first video features, the first word segmentation features, the second video features, and the second word segmentation features, a pre-trained objective function is determined based on cross-modal matching loss and intra-modal denoising loss; Multimodal pre-training is performed using the pre-trained objective function; The objective function for pre-training includes: A first contrast loss value is determined using the first word segmentation feature, the second video feature, and a preset first negative sample feature; The second contrast loss value is determined using the first video features, the second word segmentation features, and the preset second negative sample features; The first target is determined based on the first comparison loss value and the second comparison loss value; A third contrast loss value is determined using the first video feature, the second video feature, and the second negative sample feature; The fourth contrast loss value is determined using the first word segmentation feature, the second word segmentation feature, and the first negative sample feature; The second target is determined based on the third and fourth contrastive loss values; The objective function is determined based on the first objective and the second objective.

2. The method according to claim 1, wherein, Determining the first contrast loss value includes: Convert the first word segmentation feature into the global first positive sample feature; Convert the second video feature into a global video query feature; The first contrast loss value is determined using the video query features, the first positive sample features, and the first negative sample features.

3. The method according to claim 2, wherein, Determining the second contrast loss value includes: The first video feature is converted into a global second positive sample feature; The second word segmentation feature is converted into a global text query feature; The second contrast loss value is determined using the text query features, the second positive sample features, and the second negative sample features.

4. The method according to claim 3, wherein, Determining the third contrast loss value includes: The third contrast loss value is determined using the video query features, the second positive sample features, and the second negative sample features.

5. The method according to claim 4, wherein, Determining the fourth contrast loss value includes: The fourth contrast loss value is determined using the text query features, the first positive sample features, and the first negative sample features.

6. The method according to claim 1, wherein, The first objective is the sum of the first contrastive loss value and the second contrastive loss value; The second objective is the sum of the third contrast loss value and the fourth contrast loss value.

7. The method according to any one of claims 1-6, wherein, The objective function is the sum of the first objective and the second objective.

8. The method according to any one of claims 1-6, further comprising: The second video features and the second word segmentation features are fused together to obtain fused features; The fused features are input into a masked text modeling MLM model to obtain the third objective, and the fused features are input into a masked text generation MSG model to obtain the fourth objective; Determining the objective function based on the first objective and the second objective includes: The objective function is determined based on the first objective, the second objective, the third objective, and the fourth objective.

9. The method according to claim 8, wherein, The objective function is the sum of the first objective, the second objective, the third objective, and the fourth objective.

10. A multimodal pre-training device, comprising: The first processing module is configured to sample the video in the video-text pair to obtain a first video frame sequence, and is also configured to perform word segmentation on the text in the video-text pair to obtain a first word segmentation sequence. The second processing module is configured to perform random masking on the first video frame sequence to obtain a second video frame sequence, and is also configured to perform random masking on the first word segmentation sequence to obtain a second word segmentation sequence. The third processing module is configured to encode the first video frame sequence to obtain the first video feature, and is also configured to encode the first word segmentation sequence to obtain the first word segmentation feature; The fourth processing module is configured to encode the second video frame sequence to obtain second video features, and is also configured to encode the second word segmentation sequence to obtain second word segmentation features; The fifth processing module is configured to determine a pre-trained target function based on cross-modal matching loss and intra-modal denoising loss using the first video features, the first word segmentation features, the second video features, and the second word segmentation features. Specifically, it determines a first contrastive loss value using the first word segmentation features, the second video features, and a preset first negative sample feature; determines a second contrastive loss value using the first video features, the second word segmentation features, and a preset second negative sample feature; determines a first target based on the first contrastive loss value and the second contrastive loss value; determines a third contrastive loss value using the first video features, the second video features, and the second negative sample feature; determines a fourth contrastive loss value using the first word segmentation features, the second word segmentation features, and the first negative sample feature; determines a second target based on the third contrastive loss value and the fourth contrastive loss value; and determines the target function based on the first target and the second target. The sixth processing module is configured to perform multimodal pre-training using the pre-trained objective function.

11. A multimodal pre-training device, comprising: The memory is configured to store instructions; A processor, coupled to a memory, configured to implement the method as described in any one of claims 1-9 based on memory-stored instruction execution.

12. A non-transient computer-readable storage medium, wherein, A computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-9.

Images