A multimodal pre-training method, system, terminal and storage medium based on gradient self-calibration

Abstract

The present invention discloses a multimodal pre-training method, system, terminal and storage medium based on gradient self-calibration. The present invention utilizes a hybrid modality interaction network to learn the gradient of a unified modality interaction module using a task-independent loss function for a specific modality, and determines which modalities have smaller differences according to the differences between the gradient directions of different modalities; uses the combined gradient of these modal data with smaller differences on a shared encoder as a constructed positive gradient direction, and uses the gradient direction to correct the directions of the remaining modal gradients; according to the corrected modality gradient direction, the parameters of the corresponding modality on the shared encoder are updated in advance before the direction is propagated and updated, so as to achieve the purpose of balanced modality training, use a hybrid transformer network structure to extract fusion features of different modality data, and use a modality gradient self-calibration module to correct the gradient descent directions of different modalities for a joint encoder, so as to improve the effect of fusion representation of different modalities.

Description

Technical Field

[0001] The present invention relates to the field of deep learning and computer vision technology, and in particular to a multimodal pre-training method, system, terminal and storage medium based on gradient self-calibration. Background Art

[0002] Multimodal data such as images, text, video, audio, etc. can be directly received by humans and converted into the same semantics. Multimodal pre-training simulates human understanding and imagination of daily modal data by inputting massive amounts of multimodal data to obtain a large model with a huge number of parameters. By training a large multimodal model, downstream tasks of different modalities can be fine-tuned to avoid the need to train models of different modal data from scratch for specific tasks. Multimodal pre-training models have continued to emerge in recent years, have made great progress, and have refreshed the performance of all downstream visual tasks.

[0003] For example, in 2018, the BERT model was born, which greatly improved the upper limit of understanding tasks in the field of natural language processing. Subsequently, the text-image multimodal model VL-BERT based on the Transformer architecture built a text-image pre-training model based on regional image features based on this structure. However, such models need to extract regional features of image modalities in advance, which is extremely time-consuming. Subsequently, the text-image multimodal pre-training model based on Vision Transformer omitted the step of extracting image regional features and directly used image blocks to achieve better image feature extraction. Subsequently, in order to integrate more modal features, trimodal fusion such as image, text, and sound was used to further improve the overall performance of downstream tasks.

[0004] However, these models are affected by the greed of the network during the training process, that is, the network only trains one or several dominant modal data during the learning process, while ignoring the semantic complementarity brought by other modalities, so that the performance of the dominant modality can be greatly improved, but the performance of the non-dominant modality decreases. Therefore, for training tasks with more modalities, it is necessary to consider the semantic complementarity between different modalities. In the process of multimodal fusion, it is very necessary to adjust the semantic complementary relationship between different modalities. The main work is to adopt a model based on multimodal classification. For the classification model, the category information can accurately evaluate whether the semantics learned by different modalities are consistent. However, in the multimodal pre-training model, the category information is scarce or even non-existent. Therefore, how to only use the modal pair information to achieve more effective modal information fusion is the main bottleneck in improving the effect of downstream tasks in the current multimodal pre-training model. Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide a multimodal pre-training method, system, terminal and storage medium based on gradient self-calibration, aiming to solve the problem in the prior art that in the multimodal pre-training process, the network only focuses on several specific modalities and ignores the information complementarity of the remaining modal data.

[0006] To achieve the above object, the present invention provides a multimodal pre-training method based on gradient self-calibration, and the multimodal pre-training method based on gradient self-calibration comprises the following steps:

[0007] Construct feature input modules of multiple modalities, encode different modal data, and obtain feature representations of different modal data;

[0008] Construct a modality joint encoding module and construct a contrastive learning loss between different modality data;

[0009] The obtained joint modal representation is divided to obtain the modal representation after the fusion of different modalities, and tasks unrelated to downstream tasks using different modal data are used to constrain each modality;

[0010] Based on the downstream task loss function of each modality data, perform gradient propagation on the last layer of all modality joint encoders to obtain the gradient value corresponding to the last layer parameter of each downstream task loss function;

[0011] According to the gradient value, the correlation between different modal data at the gradient level is constructed, and the directional similarity of different modalities relative to other modalities is obtained based on the correlation;

[0012] The positive direction gradient and the negative direction gradient are divided according to the direction similarity, the combined gradient is constructed according to the direction of the positive direction gradient, and the modal gradient in the negative gradient direction is adjusted according to the combined gradient;

[0013] According to the adjusted modal gradient in the negative gradient direction, the modal gradients of different modalities on the last layer of the joint encoder are fine-tuned and updated.

[0014] Optionally, the multimodal pre-training method based on gradient self-calibration, wherein the step of constructing feature input modules of multiple modalities, encoding different modal data, and obtaining feature representations of different modal data, specifically includes:

[0015] For text modal data, the Bert network is used to extract the relational features between different tokens. For text sequences, an increasing natural number sequence is used to represent the position information of the text sequence, which is passed into the linear fully connected layer to obtain the position encoding. 1 is used as the segmentation information and passed into the linear fully connected layer to obtain the segmentation encoding. The text is passed into the linear fully connected layer to obtain the feature encoding of the text. The position encoding, segmentation encoding and feature encoding are added together to obtain the embedded representation of the text modal data.

[0016] For image modality data, the Vison Transformer pre-trained network is used to extract the features of the main target frame and coordinate position. Each image modality data is evenly divided into n blocks, and each block is straightened into a vector. The position encoding corresponding to the straightened vector is merged and input into the Vison Transformer.

[0017] After concatenating each row of the table data, the same processing method as the text modality data is used. For the table sequence, the table data of the same row is stacked to share the same encoder as the text sequence, and the position information of the table sequence is represented by an increasing natural number sequence, which is passed into the linear fully connected layer to obtain the position encoding; 1 is used as the segmentation information and passed into the linear fully connected layer to obtain the segmentation encoding; the table data is passed into the linear fully connected layer to obtain the feature encoding of the table data; the position encoding, segmentation encoding and feature encoding are added together to obtain the embedded representation of the table data;

[0018] For video data, the S3D network is used to extract video embedding features with spatiotemporal characteristics. According to the video embedding features, a natural number sequence is used to represent the position information of the video modality. The sequential relationship of different frames is passed in, and the video data is applied to the linear fully connected layer to obtain the position encoding; 1 is used as the segmentation information and passed into the linear fully connected layer to obtain the segmentation encoding; the video feature sequence is passed into the linear fully connected layer to obtain the feature encoding of the video data; the position encoding, segmentation encoding and feature encoding are added together to obtain the embedded representation of the video data.

[0019] Optionally, the multimodal pre-training method based on gradient self-calibration, wherein the step of constructing a modality joint encoding module and constructing a contrastive learning loss between different modality data, specifically includes:

[0020] For each modality data, a comparative learning module between different modality data is constructed to learn the semantic alignment between different modality data, and the output features of the modality data after passing through different modality encoders are put into a public multi-head self-attention network, wherein the input length of the public multi-head self-attention network is the feature length of each modality data stack;

[0021] The public multi-head self-attention network extracts the fully integrated features among the five modal data, splices the text, image, table, video and audio features, uses Q and K to calculate the weight of each vector paying attention to all features, and then multiplies it by V to obtain the common feature representation of the five modal data, wherein Q, K, and V are obtained by the features after splicing the five modalities; for each modal data, the multi-head attention mechanism is used to calculate the attention weights for all features of the five modal data, and obtain the features of each modal data after comprehensive fusion;

[0022] Among them, Q represents the query vector, K represents the vector of the correlation between the queried information and other information, and V represents the vector of the queried information.

[0023] Optionally, the multimodal pre-training method based on gradient self-calibration, wherein the obtained joint modal representation is divided to obtain a modal representation after fusion of different modalities, and tasks unrelated to downstream tasks using different modal data are used to constrain each modality, specifically including:

[0024] Construct the Mask Language Modeling downstream task by masking the words in the title text and inputting the text sequence with masked words into the multimodal pre-trained model. The multimodal pre-trained model learns to restore the masked words during the training process to extract a feature representation with text information.

[0025] Construct the downstream task of Image-Text Matching for images. Use the image-text pairs as positive samples and the remaining sample pairs as negative samples to perform image-text matching tasks and construct downstream tasks for image modality.

[0026] Constructing the Mask Entity Modeling task of the table, by masking the words with specific meanings in the table data, inputting the entity sequence with the masked words into the multimodal pre-trained model. The multimodal pre-trained model learns to restore the masked entity words during the training process to extract a feature representation with the table data information;

[0027] Construct the video masking task Mask Frame Modeling, which masks some frames in the video data, inputs the video frame sequence with masked frames into the multimodal pre-trained model, and uses the remaining frames to predict and restore the masked video tokens to construct downstream tasks for the video modality;

[0028] The fused features generated in the joint encoder are divided according to the location of the modality.

[0029] Optionally, the multimodal pre-training method based on gradient self-calibration, wherein the downstream task loss function based on each modality data performs gradient propagation on the last layer of all modality joint encoders to obtain the gradient value corresponding to the last layer parameter of each downstream task loss function, specifically includes:

[0030] According to the downstream task loss functions of different modalities, the gradients of the parameters of the last layer of the joint encoder modality sharing network are calculated separately:

[0031] ;

[0032] in, represents the gradient of the parameters of the last layer of the joint encoder modality sharing network, Indicates Downstream task loss function, and represents the temporary parameters in the network, Indicates modal data.

[0033] Optionally, the multimodal pre-training method based on gradient self-calibration, wherein the step of constructing the correlation between different modal data at the gradient level according to the gradient value, and obtaining the directional similarity of different modalities relative to other modalities based on the correlation, specifically includes:

[0034] According to the calculated gradient values ​​of different modes, the correlation between different modal data is calculated respectively:

[0035] ;

[0036] in, is the distance correlation between different gradient vectors;

[0037] According to the correlation between different modal data, learn the directional consistency of a single modality relative to the other modalities:

[0038] ;

[0039] Different gradient correlations are converted into discrete vectors through binarization.

[0040] Optionally, the multimodal pre-training method based on gradient self-calibration, wherein the positive direction gradient and the negative direction gradient are divided according to the direction similarity, a combined gradient is constructed according to the direction of the positive direction gradient, and the modal gradient in the negative gradient direction is adjusted according to the combined gradient, specifically includes:

[0041] According to the directional consistency of the learned single mode relative to the other modes, the modal directional consistency is sorted to obtain the gradient value corresponding to the modal data with the greatest correlation, using a hyperparameter Construct a combined gradient consisting of data from other modalities:

[0042] ;

[0043] in, Represented as the last top gradient, Represents the combined gradient of the last top gradients;

[0044] According to the obtained combined gradient, the modal gradient in the negative gradient direction is adjusted. The negative gradient direction is determined by the modal correlation. If the sum of the modal correlation is negative, it is defined as the negative modal gradient direction:

[0045] ;

[0046] in, Represents the directional angle between different gradients, represents a hyperparameter.

[0047] Optionally, the multimodal pre-training method based on gradient self-calibration, wherein the step of fine-tuning and updating the modal gradients of different modalities on the last layer of the joint encoder according to the modal gradient in the adjusted negative gradient direction, specifically includes:

[0048] For the shared parameters in the joint encoder, for the mode of negative gradient, the gradient direction after the mode adjustment of negative gradient is used to adjust the shared parameters in the shared encoder to achieve balanced training of mode gradient:

[0049] ;

[0050] in, Indicates the parameters of the shared network, represents the adjusted shared parameters, represents the adjusted gradient, and represents a hyperparameter.

[0051] In addition, to achieve the above-mentioned purpose, the present invention also provides a multimodal pre-training system based on gradient self-calibration, wherein the multimodal pre-training system based on gradient self-calibration includes:

[0052] The modal data encoding module is used to construct a feature input module of multiple modalities, encode different modal data, and obtain feature representations of different modal data;

[0053] The modular loss building module is used to build the modality joint encoding module and construct the contrastive learning loss between different modality data;

[0054] The modality partitioning constraint module is used to partition the obtained joint modality representation to obtain the modality representation after the fusion of different modalities, and to constrain each modality using tasks that are unrelated to downstream tasks and different modality data;

[0055] The gradient propagation calculation module is used to perform gradient propagation on the last layer of all modal joint encoders based on the downstream task loss function of each modal data, and obtain the gradient value corresponding to the last layer parameter of each downstream task loss function;

[0056] The correlation similarity processing module is used to construct the correlation between different modal data at the gradient level according to the gradient value, and obtain the directional similarity of different modalities relative to other modalities based on the correlation;

[0057] The modal gradient adjustment module is used to divide the positive direction gradient and the negative direction gradient according to the direction similarity, construct the combined gradient according to the positive direction gradient, and adjust the modal gradient in the negative gradient direction according to the combined gradient;

[0058] The modality gradient update module is used to fine-tune and update the modality gradients of different modalities on the last layer of the joint encoder according to the modality gradients in the adjusted negative gradient direction.

[0059] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a multimodal pre-training program based on gradient self-calibration stored in the memory and executable on the processor, wherein the multimodal pre-training program based on gradient self-calibration implements the steps of the multimodal pre-training method based on gradient self-calibration as described above when executed by the processor.

[0060] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multimodal pre-training program based on gradient self-calibration, and when the multimodal pre-training program based on gradient self-calibration is executed by a processor, the steps of the multimodal pre-training method based on gradient self-calibration as described above are implemented.

[0061] In the present invention, feature input modules of multiple modalities are constructed, different modal data are encoded, and feature representations of different modal data are obtained; a modal joint encoding module is constructed, and a comparative learning loss between different modal data is constructed; the obtained joint modal representation is divided to obtain a modal representation after fusion of different modalities, and tasks unrelated to downstream tasks of different modal data are used to constrain each modality; based on the downstream task loss function of each modal data, gradient propagation is performed on the last layer of all modal joint encoders to obtain the gradient value corresponding to the last layer parameter of each downstream task loss function; the correlation between different modal data at the gradient level is constructed according to the gradient value, and the directional similarity of different modalities relative to other modalities is obtained based on the correlation; the positive direction gradient and the negative direction gradient are divided according to the directional similarity, a combined gradient is constructed according to the direction of the positive direction gradient, and the modal gradient in the negative gradient direction is adjusted according to the combined gradient; according to the adjusted modal gradient in the negative gradient direction, the modal gradient of different modalities on the last layer of the joint encoder is fine-tuned and updated. The present invention uses a hybrid transformer network structure to extract the fusion features of different modal data, and uses a modal gradient self-calibration module to correct the gradient descent direction of different modalities for the joint encoder, thereby improving the effect of different modal fusion representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a flow chart of a preferred embodiment of the multimodal pre-training method based on gradient self-calibration of the present invention;

[0063] Figure 2 It is a schematic diagram of an extended modality example of M5Product in a preferred embodiment of the multimodal pre-training method based on gradient self-calibration of the present invention;

[0064] Figure 3 is a schematic diagram of a gradient self-calibration module in a preferred embodiment of the multi-modal pre-training method based on gradient self-calibration of the present invention;

[0065] Figure 4 Schematic diagram of the overall framework of multimodal pre-training in a preferred embodiment of the multimodal pre-training method based on gradient self-calibration of the present invention;

[0066] Figure 5 It is a schematic diagram of the principle of a preferred embodiment of a multi-modal pre-training system based on gradient self-calibration of the present invention;

[0067] Figure 6 Schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0069] The present invention relates to multi-modal fusion representation by a computer, and can achieve good results by inputting multiple modal data so that each different modality can be used alone or in combination, and belongs to the field of deep learning and computer vision technology.

[0070] The multimodal pre-training method based on gradient self-calibration described in the preferred embodiment of the present invention is as follows: Figure 1 As shown, the multimodal pre-training method based on gradient self-calibration includes the following steps:

[0071] Step S10: construct feature input modules of multiple modalities, encode different modal data, and obtain feature representations of different modal data.

[0072] Specifically, for text modal data, the Bert network is used to extract the relational features between different tokens. Specifically, for text sequences, an increasing natural number sequence is used to represent the position information of the text sequence, which is passed into the linear fully connected layer to obtain the position encoding; 1 is used as the segmentation information and passed into the linear fully connected layer to obtain the segmentation encoding; the text is passed into the linear fully connected layer to obtain the feature encoding of the text; finally, the position encoding, segmentation encoding and feature encoding are added together to obtain the embedded representation of the text modal data.

[0073] For image modality data, the Vison Transformer pre-trained network is used to extract the features of the main target frame and coordinate position. Each image modality data is evenly divided into n blocks, and each block is straightened into a vector. The position encoding corresponding to the straightened vector is merged and input into the Vison Transformer.

[0074] After concatenating each row of the tabular data together, the same processing method as the text modal data is used. For the table sequence, the table data in the same row are stacked to share the same encoder as the text sequence, and an increasing natural number sequence is used to represent the position information of the table sequence, which is passed into the linear fully connected layer to obtain the position encoding; 1 is used as the segmentation information and passed into the linear fully connected layer to obtain the segmentation encoding; the table data is passed into the linear fully connected layer to obtain the feature encoding of the table data; finally, the position encoding, segmentation encoding and feature encoding are added together to obtain the embedded representation of the table data.

[0075] For video data, the S3D network is first used to extract video embedding features with spatiotemporal characteristics. According to the video embedding features, a natural number sequence is used to represent the position information of the video modality. The sequential relationship of different frames is passed in, and the video data is applied to the linear fully connected layer to obtain the position code; 1 is used as the segmentation information and passed into the linear fully connected layer to obtain the segmentation code; the video feature sequence is passed into the linear fully connected layer to obtain the feature code of the video data; finally, the position code, segmentation code and feature code are added together to obtain the embedded representation of the video data.

[0076] like Figure 2 As shown, it is an extended modality example of M5Product, which mainly includes five types of modal data, such as image data, text data, table data, video data and audio data.

[0077] The present invention mainly uses the M5Product dataset as the dataset for modal expansion. The dataset mainly consists of more than 5 million images, texts, tables, videos and audio pairs, each of which has 3 levels of annotations, namely industrial categories, first-level best-selling subcategories, and second-level best-selling subcategories. For this dataset, images, texts, tables and video data will be used to conduct relevant experiments. At the same time, the CC3M image-text dataset is also used for training. The dataset is an English image-text pair dataset with more than 2.6 million image-text pairs. A large amount of image-text data in the dataset has relatively accurate image-text semantics, and most of the data is single-target data. Therefore, these two datasets are selected as pre-training datasets.

[0078] Step S20: construct a modality joint encoding module and construct a contrastive learning loss between different modality data.

[0079] Specifically, a modal joint encoding module is constructed to enable token-level semantic fusion between different modal data, thereby obtaining a more accurate modal representation, and at the same time construct a comparative learning loss between different modal data.

[0080] Firstly, for each modal data, a comparative learning module between different modal data is constructed to learn the semantic alignment between different modal data. The output features of the modal data after passing through different modal encoders are put into a common multi-head self-attention network, where the input length of the common multi-head self-attention network is the feature length of the stack of each modal data.

[0081] The public multi-head self-attention network extracts the features of comprehensive fusion among the five modal data, splices the text, image, table, video and audio features, uses Q and K to calculate the weight of each vector paying attention to all features, and then multiplies it by V to obtain the common feature representation of the five modal data, wherein Q, K, and V are obtained by the features after splicing the five modalities; for each modal data, the multi-head attention mechanism is used to calculate the attention weights for all features of the five modal data, and obtain the features of each modal data after comprehensive fusion; wherein Q, K, and V are part of a transformer module, Q represents the query vector, K represents the vector of the correlation between the queried information and other information, and V represents the vector of the queried information.

[0082] Step S30: divide the obtained joint modal representation to obtain a modal representation after fusion of different modalities, and use tasks unrelated to downstream tasks with different modal data to constrain each modality.

[0083] Specifically, a text downstream task called Mask Language Modeling is constructed. By masking the words in the title text, the text sequence with masked words is input into a multimodal pre-trained model. During the training process, the multimodal pre-trained model learns to restore the masked words to extract a feature representation with text information.

[0084] Construct the downstream task Image-Text Matching for images, take the image-text pairs as positive samples and the remaining sample pairs as negative samples, perform the image-text matching task, and construct the downstream task for the image modality.

[0085] Construct a table masking task, Mask Entity Modeling, by masking words with specific meanings in the table data and inputting the entity sequence with masked words into the multimodal pre-trained model. During the training process, the multimodal pre-trained model learns to restore the masked entity words to extract a feature representation with table data information.

[0086] Construct a video masking task, Mask Frame Modeling, by masking some frames in the video data, inputting the video frame sequence with masked frames into the multimodal pre-trained model, and using the remaining frames to predict and restore the masked video tokens to build downstream tasks for the video modality.

[0087] The fused features generated in the joint encoder are divided according to the location of the modality, that is, the modal data is segmented according to the splicing order of different modalities.

[0088] Step S40: Based on the downstream task loss function of each modal data, gradient propagation is performed on the last layer of all modal joint encoders to obtain the gradient value corresponding to the last layer parameter of each downstream task loss function.

[0089] Specifically, the present invention constructs a hybrid structure transformer network, which has multiple different single-modal encoders, followed by multiple single-modal encoders, and a joint modal encoder will be used for feature fusion of different modal data. For each modal data, a separate task-independent downstream task is used for constraint. According to the downstream task loss function of different modalities, the gradient of the parameters of the last layer of the joint encoder modality sharing network is calculated separately:

[0090] ;

[0091] in, represents the gradient of the parameters of the last layer of the joint encoder modality sharing network, Indicates Downstream task loss function, and represents the temporary parameters in the network, Indicates modal data.

[0092] Step S50: construct the correlation between different modal data at the gradient level according to the gradient value, and obtain the directional similarity of different modalities relative to other modalities based on the correlation.

[0093] Specifically, according to the calculated gradient values ​​of different modes, the correlation between different modal data is calculated respectively:

[0094] ;

[0095] in, is the distance correlation between different gradient vectors, expressed as cosine vector distance;

[0096] According to the correlation between different modal data, learn the directional consistency of a single modality relative to the other modalities:

[0097] ;

[0098] Different gradient correlations are converted into discrete vectors through binarization.

[0099] Step S60: divide the positive direction gradient and the negative direction gradient according to the direction similarity, construct a combined gradient according to the direction of the positive direction gradient, and adjust the modal gradient in the negative gradient direction according to the combined gradient.

[0100] Specifically, according to the learned directional consistency of a single modality relative to the remaining modalities, the modal directional consistency is Sort and obtain the gradient value corresponding to the modal data with the greatest correlation, using a hyperparameter Construct a combined gradient consisting of data from other modalities:

[0101] ;

[0102] in, Represented as the last top gradient, Represents the combined gradient of the last top gradients; selects the first top gradient as the basis to construct the combined gradient;

[0103] According to the obtained combined gradient , adjust the modal gradient in the negative gradient direction, where the negative gradient direction is determined by the modal correlation. If the sum of the modal correlation is negative, it is defined as the negative modal gradient direction:

[0104] ;

[0105] in, Represents the directional angle between different gradients. Based on this directional angle, orthogonal projection can be performed to obtain , Represents a hyperparameter.

[0106] like Figure 3 As shown, it represents the gradient self-calibration module, which is the main body of the training framework and is used to realize the self-calibration of the gradient.

[0107] Step S70: fine-tune and update the modal gradients of different modalities on the last layer of the joint encoder according to the adjusted modal gradients in the negative gradient direction.

[0108] Specifically, for the shared parameters in the joint encoder, for the mode of the negative gradient, the gradient direction after the mode adjustment of the negative gradient is used to adjust the shared parameters in the shared encoder to achieve balanced training of the mode gradient:

[0109] ;

[0110] in, Indicates the parameters of the shared network, represents the adjusted shared parameters, represents the adjusted gradient, and Represents the hyperparameter, where the adjusted gradient Appended to the optimization process for each shared parameter.

[0111] like Figure 4 As shown, the overall framework of multimodal pre-training is represented, which shows the pre-training process of data with different modalities.

[0112] A set of modal data is selected from the training set, and different encoding forms are used for different modal data. The present invention builds a network model through the pytorch framework. The experiment is iterated 5 times. The initial learning rate is set to 0.0001, the batch size is set to 32, and 32 groups of modal data are randomly selected for training each time. The Adam optimizer is used for training. During the training process, the image and text modal encoders are initialized using the Vision Transformer and Bert pre-training models, and the model is trained with a separate downstream task for each modality and comparative learning between different modal data.

[0113] Extract all modal features of the corresponding target and obtain the fused representation of each modality for target retrieval to obtain the final retrieval sequence.

[0114] The present invention adopts the above technical means, and thus has the following beneficial effects:

[0115] (1) Expand the types of modalities. Expand the image-text modal data, add tables and video data sets, and increase the complementary information of the modalities by expanding the types of modalities. In addition, the different modalities in the collected data sets are highly semantically aligned, that is, different modalities are more accurate in representing the same target. Therefore, the information interaction capabilities between different modalities can be effectively improved.

[0116] (2) Using gradients to evaluate the semantic alignment of modal data provides stronger semantic information and more accurate modal representation than directly using data.

[0117] (3) Using combined modal gradients to fine-tune the gradient of non-dominant modal data has stronger robustness. It can improve the modal fusion capability while maintaining good single-modal encoding capabilities, and the training convergence speed is faster.

[0118] Furthermore, if Figure 5 As shown, based on the above-mentioned multimodal pre-training method based on gradient self-calibration, the present invention also provides a multimodal pre-training system based on gradient self-calibration, wherein the multimodal pre-training system based on gradient self-calibration includes:

[0119] A modality data encoding module 51 is used to construct a feature input module of multiple modalities, encode different modality data, and obtain feature representations of different modality data;

[0120] A module loss construction module 52 is used to construct a modality joint encoding module and construct a contrastive learning loss between different modality data;

[0121] A modality division constraint module 53 is used to divide the obtained joint modality representation to obtain a modality representation after fusion of different modalities, and to constrain each modality using tasks unrelated to downstream tasks using different modality data;

[0122] A gradient propagation calculation module 54 is used to perform gradient propagation on the last layer of all modal joint encoders based on the downstream task loss function of each modal data, and obtain the gradient value corresponding to the last layer parameter of each downstream task loss function;

[0123] The correlation similarity processing module 55 is used to construct the correlation between different modal data at the gradient level according to the gradient value, and obtain the directional similarity of different modalities relative to other modalities based on the correlation;

[0124] The modal gradient adjustment module 56 is used to divide the positive direction gradient and the negative direction gradient according to the direction similarity, construct a combined gradient according to the direction of the positive direction gradient, and adjust the modal gradient in the negative gradient direction according to the combined gradient;

[0125] The modality gradient updating module 57 is used to fine-tune and update the modality gradients of different modalities on the last layer of the joint encoder according to the modality gradients in the adjusted negative gradient direction.

[0126] Furthermore, if Figure 6 As shown, based on the above-mentioned multimodal pre-training method and system based on gradient self-calibration, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Figure 6 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0127] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 can also be used to temporarily store data that has been output or is to be output. In one embodiment, a multimodal pre-training program 40 based on gradient self-calibration is stored on the memory 20, and the multimodal pre-training program 40 based on gradient self-calibration can be executed by the processor 10, thereby realizing the multimodal pre-training method based on gradient self-calibration in the present application.

[0128] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the multimodal pre-training method based on gradient self-calibration.

[0129] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0130] In one embodiment, when the processor 10 executes the multimodal pre-training program 40 based on gradient self-calibration in the memory 20, the steps of the multimodal pre-training method based on gradient self-calibration are implemented.

[0131] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a multimodal pre-training program based on gradient self-calibration, and when the multimodal pre-training program based on gradient self-calibration is executed by a processor, the steps of the multimodal pre-training method based on gradient self-calibration as described above are implemented.

[0132] In summary, the present invention provides a multimodal pre-training method, system, terminal and storage medium based on gradient self-calibration, the method comprising: constructing feature input modules of multiple modalities, encoding different modal data, and obtaining feature representations of different modal data; constructing a modal joint encoding module, and constructing a comparative learning loss between different modal data; dividing the obtained joint modal representation to obtain a modal representation after fusion of different modalities, and using tasks unrelated to downstream tasks of different modal data to constrain each modality; based on the downstream task loss function of each modal data, performing gradient propagation on the last layer of all modal joint encoders, and obtaining the gradient value corresponding to the last layer parameter of each downstream task loss function; constructing the correlation between different modal data at the gradient level according to the gradient value, and obtaining the directional similarity of different modalities relative to other modalities based on the correlation; dividing the positive direction gradient and the negative direction gradient according to the directional similarity, constructing a combined gradient according to the positive direction gradient direction, and adjusting the modal gradient in the negative gradient direction according to the combined gradient; and fine-tuning and updating the modal gradients of different modalities on the last layer of the joint encoder according to the adjusted modal gradient in the negative gradient direction. The present invention uses a hybrid transformer network structure to extract the fusion features of different modal data, and uses a modal gradient self-calibration module to correct the gradient descent direction of different modalities for the joint encoder, thereby improving the effect of different modal fusion representation.

[0133] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.

[0134] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0135] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A multimodal pre-training method based on gradient self-calibration, characterized in that: The multimodal pre-training method based on gradient self-calibration includes: Construct feature input modules of multiple modalities, encode different modal data, and obtain feature representations of different modal data; Construct a modality joint encoding module and construct a contrastive learning loss between different modality data; The obtained joint modal representation is divided to obtain the modal representation after the fusion of different modalities, and tasks unrelated to downstream tasks using different modal data are used to constrain each modality; Based on the downstream task loss function of each modality data, perform gradient propagation on the last layer of all modality joint encoders to obtain the gradient value corresponding to the last layer parameter of each downstream task loss function; According to the gradient value, the correlation between different modal data at the gradient level is constructed, and the directional similarity of different modalities relative to other modalities is obtained based on the correlation; The positive direction gradient and the negative direction gradient are divided according to the direction similarity, the combined gradient is constructed according to the direction of the positive direction gradient, and the modal gradient in the negative gradient direction is adjusted according to the combined gradient; According to the adjusted modal gradient in the negative gradient direction, the modal gradients of different modalities on the last layer of the joint encoder are fine-tuned and updated.

2. The multimodal pre-training method based on gradient self-calibration according to claim 1, characterized in that: The step of constructing a feature input module of multiple modalities, encoding different modal data, and obtaining feature representations of different modal data specifically includes: For text modal data, the Bert network is used to extract the relational features between different tokens. For text sequences, an increasing natural number sequence is used to represent the position information of the text sequence, which is passed into the linear fully connected layer to obtain the position encoding. 1 is used as the segmentation information and passed into the linear fully connected layer to obtain the segmentation encoding. The text is passed into the linear fully connected layer to obtain the feature encoding of the text. The position encoding, segmentation encoding and feature encoding are added together to obtain the embedded representation of the text modal data. For image modality data, the Vison Transformer pre-trained network is used to extract the features of the main target frame and coordinate position. Each image modality data is evenly divided into n blocks, and each block is straightened into a vector. The position encoding corresponding to the straightened vector is merged and input into the Vison Transformer. After concatenating each row of the table data, the same processing method as the text modality data is used. For the table sequence, the table data of the same row is stacked to share the same encoder as the text sequence, and the position information of the table sequence is represented by an increasing natural number sequence, which is passed into the linear fully connected layer to obtain the position encoding; 1 is used as the segmentation information and passed into the linear fully connected layer to obtain the segmentation encoding; the table data is passed into the linear fully connected layer to obtain the feature encoding of the table data; the position encoding, segmentation encoding and feature encoding are added together to obtain the embedded representation of the table data; For video data, the S3D network is used to extract video embedding features with spatiotemporal characteristics. According to the video embedding features, a natural number sequence is used to represent the position information of the video modality. The sequential relationship of different frames is passed in, and the video data is applied to the linear fully connected layer to obtain the position encoding; 1 is used as the segmentation information and passed into the linear fully connected layer to obtain the segmentation encoding; the video feature sequence is passed into the linear fully connected layer to obtain the feature encoding of the video data; the position encoding, segmentation encoding and feature encoding are added together to obtain the embedded representation of the video data.

3. The multimodal pre-training method based on gradient self-calibration according to claim 1, characterized in that: The construction of the modality joint encoding module and the construction of the contrastive learning loss between different modality data specifically include: For each modality data, a comparative learning module between different modality data is constructed to learn the semantic alignment between different modality data, and the output features of the modality data after passing through different modality encoders are put into a public multi-head self-attention network, wherein the input length of the public multi-head self-attention network is the feature length of each modality data stack; The public multi-head self-attention network extracts the fully integrated features among the five modal data, splices the text, image, table, video and audio features, uses Q and K to calculate the weight of each vector paying attention to all features, and then multiplies it by V to obtain the common feature representation of the five modal data, wherein Q, K, and V are obtained by the features after splicing the five modalities; for each modal data, the multi-head attention mechanism is used to calculate the attention weights for all features of the five modal data, and obtain the features of each modal data after comprehensive fusion; Among them, Q represents the query vector, K represents the vector of the correlation between the queried information and other information, and V represents the vector of the queried information.

4. The multimodal pre-training method based on gradient self-calibration according to claim 1, characterized in that: The obtained joint modal representation is divided to obtain modal representations after fusion of different modalities, and tasks irrelevant to downstream tasks using different modal data are used to constrain each modality, specifically including: Construct the Mask Language Modeling downstream task by masking the words in the title text and inputting the text sequence with masked words into the multimodal pre-trained model. The multimodal pre-trained model learns to restore the masked words during the training process to extract a feature representation with text information. Construct the downstream task of Image-Text Matching for images. Use the image-text pairs as positive samples and the remaining sample pairs as negative samples to perform image-text matching tasks and construct downstream tasks for image modality. Constructing the Mask Entity Modeling task of the table, by masking the words with specific meanings in the table data, inputting the entity sequence with the masked words into the multimodal pre-trained model. The multimodal pre-trained model learns to restore the masked entity words during the training process to extract a feature representation with the table data information; Construct the video masking task Mask Frame Modeling, which masks some frames in the video data, inputs the video frame sequence with masked frames into the multimodal pre-trained model, and uses the remaining frames to predict and restore the masked video tokens to construct downstream tasks for the video modality; The fused features generated in the joint encoder are divided according to the location of the modality.

5. The multimodal pre-training method based on gradient self-calibration according to claim 1, characterized in that: The downstream task loss function based on each modality data performs gradient propagation on the last layer of all modality joint encoders to obtain the gradient value corresponding to the last layer parameter of each downstream task loss function, specifically including: According to the downstream task loss functions of different modalities, the gradients of the parameters of the last layer of the joint encoder modality sharing network are calculated separately: ； in, represents the gradient of the parameters of the last layer of the joint encoder modality sharing network, Indicates Downstream task loss function, and represents the temporary parameters in the network, Indicates modal data.

6. The multimodal pre-training method based on gradient self-calibration according to claim 5, characterized in that: The method of constructing the correlation between different modal data at the gradient level according to the gradient value and obtaining the directional similarity of different modalities relative to other modalities based on the correlation specifically includes: According to the calculated gradient values ​​of different modes, the correlation between different modal data is calculated respectively: ； in, is the distance correlation between different gradient vectors; According to the correlation between different modal data, learn the directional consistency of a single modality relative to the other modalities: ； Different gradient correlations are converted into discrete vectors through binarization.

7. The multimodal pre-training method based on gradient self-calibration according to claim 6, characterized in that: The method of dividing the positive direction gradient and the negative direction gradient according to the direction similarity, constructing a combined gradient according to the direction of the positive direction gradient, and adjusting the modal gradient in the negative gradient direction according to the combined gradient specifically includes: According to the directional consistency of the learned single mode relative to the other modes, the modal directional consistency is sorted to obtain the gradient value corresponding to the modal data with the greatest correlation, using a hyperparameter Construct a combined gradient consisting of data from other modalities: ； in, Represented as the last top gradient, Represents the combined gradient of the last top gradients; According to the obtained combined gradient, the modal gradient in the negative gradient direction is adjusted. The negative gradient direction is determined by the modal correlation. If the sum of the modal correlation is negative, it is defined as the negative modal gradient direction: ； in, Represents the directional angle between different gradients, Represents a hyperparameter.

8. The multimodal pre-training method based on gradient self-calibration according to claim 7, characterized in that: The method of fine-tuning and updating the modal gradients of different modalities on the last layer of the joint encoder according to the modal gradient in the adjusted negative gradient direction specifically includes: For the shared parameters in the joint encoder, for the mode of negative gradient, the gradient direction after the mode adjustment of negative gradient is used to adjust the shared parameters in the shared encoder to achieve balanced training of mode gradient: ； in, Indicates the parameters of the shared network, represents the adjusted shared parameters, represents the adjusted gradient, and Represents a hyperparameter.

9. A multimodal pre-training system based on gradient self-calibration, characterized in that: The multimodal pre-training system based on gradient self-calibration includes: The modal data encoding module is used to construct a feature input module of multiple modalities, encode different modal data, and obtain feature representations of different modal data; The modular loss building module is used to build the modality joint encoding module and construct the contrastive learning loss between different modality data; The modality partitioning constraint module is used to partition the obtained joint modality representation to obtain the modality representation after the fusion of different modalities, and to constrain each modality using tasks that are unrelated to downstream tasks and different modality data; The gradient propagation calculation module is used to perform gradient propagation on the last layer of all modal joint encoders based on the downstream task loss function of each modal data, and obtain the gradient value corresponding to the last layer parameter of each downstream task loss function; The correlation similarity processing module is used to construct the correlation between different modal data at the gradient level according to the gradient value, and obtain the directional similarity of different modalities relative to other modalities based on the correlation; The modal gradient adjustment module is used to divide the positive direction gradient and the negative direction gradient according to the direction similarity, construct the combined gradient according to the positive direction gradient, and adjust the modal gradient in the negative gradient direction according to the combined gradient; The modality gradient update module is used to fine-tune and update the modality gradients of different modalities on the last layer of the joint encoder according to the modality gradients in the adjusted negative gradient direction.

10. A terminal, characterized in that: The terminal includes: a memory, a processor, and a multimodal pre-training program based on gradient self-calibration stored in the memory and executable on the processor. When the multimodal pre-training program based on gradient self-calibration is executed by the processor, the steps of the multimodal pre-training method based on gradient self-calibration as described in any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a multimodal pre-training program based on gradient self-calibration, and when the multimodal pre-training program based on gradient self-calibration is executed by a processor, the steps of the multimodal pre-training method based on gradient self-calibration as described in any one of claims 1-8 are implemented.

Images