Training method, device and electronic device for multimodal data labeling model

Abstract

The present application proposes a training method, device and electronic device for a multimodal data label model, including: constructing a candidate multimodal data label model to be trained, and obtaining a multimodal training sample; extracting and resampling the multimodal training sample to obtain the multimodal splicing feature of the multimodal training sample; obtaining the candidate multimodal fusion prompt vector and the candidate label prompt vector corresponding to the multimodal training sample; based on the candidate multimodal fusion prompt vector, the candidate label prompt vector and the multimodal splicing feature, obtaining the training loss set of the candidate multimodal data label model to train the candidate multimodal data label model until the training is completed, and obtaining the trained target multimodal data label model. The stability of the candidate multimodal data label model is improved, the noise filtering effect and learning effect of the model are optimized, and the alignment degree of multimodal information is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a training method, device and electronic device for a multimodal data labeling model. Background Art

[0002] In real life, people interact with the real world through multiple senses such as vision, hearing and touch, and integrate different sensory stimulations into a unified multi-sensory experience. This multi-sensory experience is the multimodality of the machine. The machine extracts information from the multimodal data to obtain an accurate description of the object of the multimodal data.

[0003] Optionally, in the process of machine model learning and training based on multimodal data, different unimodal information needs to be aligned. In related technologies, the information of a single modality or two modalities can be aligned by comparing the learning loss function. In scenarios where multimodal information needs to be aligned, there may be a situation where the unimodal information in the multimodal information is not fully aligned.

[0004] Optionally, during the learning and training of the machine model based on multimodal data, the sequence dimension of each unimodal feature vector extracted by the machine model may exceed the upper limit of the model input dimension, resulting in the inability of the machine model to accurately extract key information, thereby affecting the learning effect of the machine model to a certain extent. Summary of the invention

[0005] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0006] To this end, the first aspect of the present application proposes a training method for a multimodal data labeling model.

[0007] A second aspect of the present application proposes a training device for a multimodal data labeling model.

[0008] A third aspect of the present application provides an electronic device.

[0009] A fourth aspect of the present application provides a computer-readable storage medium.

[0010] A fifth aspect of the present application provides a computer program product.

[0011] In a first aspect, the present application proposes a training method for a multimodal data label model, the method comprising constructing a candidate multimodal data label model to be trained, and obtaining a multimodal training sample; performing feature extraction and resampling on the multimodal training sample to obtain a multimodal splicing feature of the multimodal training sample; obtaining a candidate multimodal fusion prompt vector and a candidate label prompt vector corresponding to the multimodal training sample; based on the candidate multimodal fusion prompt vector, the candidate label prompt vector and the multimodal splicing feature, obtaining a training loss set of the candidate multimodal data label model to perform model training on the candidate multimodal data label model until the training is completed, thereby obtaining a trained target multimodal data label model.

[0012] In addition, the training method of the multimodal data labeling model proposed in the first aspect of the present application may also have the following additional technical features:

[0013] According to one embodiment of the present application, the feature extraction and resampling of the multimodal training samples to obtain the multimodal splicing features of the multimodal training samples includes: extracting the unimodal features of each unimodal sample in the multimodal training samples based on the multimodal feature extraction encoder of the candidate multimodal data label model; resampling the unimodal features of each unimodal sample based on the multimodal feature resampling layer of the candidate multimodal data label model to obtain the unimodal resampled features of each unimodal sample; and splicing the unimodal resampled features of each unimodal sample to obtain the multimodal splicing features extracted from the multimodal training samples.

[0014] According to one embodiment of the present application, the step of obtaining the candidate multimodal fusion hint vector and the candidate label hint vector corresponding to the multimodal training sample includes: obtaining a downstream task set of the candidate multimodal data label model, and a query hint vector combination of each downstream task in the downstream task set, wherein, for any downstream task, the query hint vector combination of the downstream task includes an initial multimodal fusion hint vector and an initial label hint vector of the downstream task; obtaining a target downstream task associated with the multimodal training sample in the downstream task set, and determining the initial multimodal fusion hint vector in the query hint vector combination of the target downstream task as the candidate multimodal fusion hint vector, and determining the initial label hint vector as the candidate label hint vector.

[0015] According to one embodiment of the present application, the downstream task set of the candidate multimodal data label model is obtained, and the query prompt vector combination of each downstream task in the downstream task set, wherein the query prompt vector combination includes a multimodal fusion query prompt vector and a label query prompt vector, comprises: obtaining the feature extraction space dimension of the candidate multimodal data label model; randomly constructing an initial multimodal query prompt vector and an initial label query prompt vector for each downstream task in the downstream task set based on the feature extraction space dimension, to obtain a query prompt vector combination for each downstream task.

[0016] According to one embodiment of the present application, the training loss set of the candidate multimodal data label model is obtained based on the candidate multimodal fusion prompt vector, the candidate label prompt vector and the multimodal splicing feature, including: splicing the candidate multimodal fusion prompt vector, the candidate label prompt vector and the sample label of the multimodal training sample to obtain a first splicing vector; obtaining a mask strategy of the first self-attention layer of the multimodal feature fusion encoder and the second self-attention layer of the label encoder of the candidate multimodal data label model; based on the mask strategy, obtaining the training loss set through the first splicing vector and / or the multimodal splicing feature.

[0017] According to an embodiment of the present application, the training loss set is obtained through the first splicing vector and / or the multimodal splicing feature based on the mask strategy, including: obtaining the first fusion feature output by the multimodal feature fusion encoder through the first splicing vector and / or the multimodal splicing feature, and obtaining the second fusion feature output by the label encoder; in response to the mask strategy being the first mask strategy, obtaining the multimodal label contrast learning training loss MLC according to the first fusion feature and the second fusion feature; in response to the mask strategy being the second mask strategy, obtaining the multimodal label matching training loss MLM according to the first fusion feature and the binary classification label of the multimodal training sample; in response to the mask strategy being the third mask strategy, obtaining the multimodal label generation training loss MLG according to the output label in the second fusion feature and the sample label of the multimodal training sample; obtaining the training loss set according to the multimodal label contrast learning training loss, the multimodal label matching training loss and the multimodal label generation training loss.

[0018] According to an embodiment of the present application, the obtaining of the first fusion feature output by the multimodal feature fusion encoder through the first splicing vector and / or the multimodal splicing feature, and the obtaining of the second fusion feature output by the label encoder include: obtaining M first blocks in the multimodal feature fusion encoder and N second blocks in the label encoder, wherein each first block in the M first blocks is connected in series, each second block in the N second blocks is connected in series, and M and N have the same value; for the i-th first block and the i-th second block, in response to i being equal to M, obtaining the first candidate fusion feature output by the i-1th first block and the second candidate fusion feature output by the i-1th second block through the first splicing vector and / or the multimodal splicing feature; obtaining the first candidate fusion feature, the second candidate fusion feature and the second splicing vector of the sample label to obtain the first fusion feature output by the i-th first block, and the second fusion feature output by the i-th second block.

[0019] According to an embodiment of the present application, for the i-th first block and the i-th second block, in response to i being equal to M, obtaining the first candidate fusion feature output by the i-1th first block and the second candidate fusion feature output by the i-1th second block through the first splicing vector and / or the multimodal splicing feature, including: in response to i-1 being equal to 1, obtaining the third splicing vector outputted by the first self-attention layer of the i-1th first block and the second self-attention layer of the i-1th second block according to the first splicing vector, and obtaining the third splicing vector outputted by the first self-attention layer of the i-1th first block and the second self-attention layer of the i-1th second block according to the candidate multimodal fusion prompt vector, the candidate label The first candidate fusion feature and the second candidate fusion feature are obtained by signing the prompt vector and the third splicing vector; in response to i-1 being greater than 1, according to the third candidate fusion feature output by the i-2th first block and the fourth candidate fusion feature output by the i-2th second block, the third splicing vector output by the first self-attention layer of the i-1th first block and the second self-attention layer of the i-1th second block are obtained, and according to the third candidate fusion feature, the third splicing vector, the multimodal splicing feature and the sample label, the first candidate fusion feature and the second candidate fusion feature are obtained.

[0020] According to an embodiment of the present application, the obtaining of the first candidate fusion feature, the second candidate fusion feature, and the second splicing vector of the sample label to obtain the first fusion feature output by the i-th first block and the second fusion feature output by the i-th second block includes: in response to i being equal to M, according to the second splicing vector, obtaining a fourth splicing vector output by each of the first self-attention layer of the i-th first block and the second self-attention layer of the i-th second block; and obtaining the first fusion feature and the second fusion feature according to the first candidate fusion feature, the fourth splicing vector, the multimodal splicing feature, the second candidate fusion feature and the sample label.

[0021] According to one embodiment of the present application, the training loss set of the candidate multimodal data label model is obtained based on the candidate multimodal fusion prompt vector, the candidate label prompt vector and the multimodal splicing feature to perform model training on the candidate multimodal data label model until the training is completed to obtain a trained target multimodal data label model, including: obtaining a target training loss based on the training loss set; adjusting the iteration parameters of the candidate multimodal data label model based on the target training loss, wherein the iteration parameters at least include the iteration vector parameters of the candidate multimodal fusion prompt vector and the candidate label prompt vector respectively, and the iteration model parameters of the multimodal data label model, and the iteration model parameters do not include the encoder parameters of the image unimodal feature extraction encoder in the multimodal feature extraction encoder; returning to obtain the next multimodal training sample to continue training the candidate multimodal data label model after the iteration parameters are adjusted until the training is completed to obtain the trained target multimodal data label model.

[0022] According to one embodiment of the present application, the method also includes: obtaining a target multimodal fusion prompt vector and a target label prompt vector used in a target training round of the candidate multimodal data label model, wherein the target training round is a training round for obtaining the trained target multimodal data label model; based on the target multimodal fusion prompt vector and the target label prompt vector, obtaining a target query prompt vector combination of the target downstream task to which the multimodal training sample belongs.

[0023] The second aspect of the present application proposes a training device for a multimodal data label model, which includes: a construction module, used to construct a candidate multimodal data label model to be trained, and to obtain multimodal training samples; a resampling module, used to extract features and resample the multimodal training samples to obtain multimodal splicing features of the multimodal training samples; an acquisition module, used to obtain candidate multimodal fusion prompt vectors and candidate label prompt vectors corresponding to the multimodal training samples; a training module, used to obtain a training loss set of the candidate multimodal data label model based on the candidate multimodal fusion prompt vector, the candidate label prompt vector and the multimodal splicing feature, so as to perform model training on the candidate multimodal data label model until the training is completed, and a trained target multimodal data label model is obtained.

[0024] In addition, the training device for the multimodal data labeling model proposed in the second aspect of the present application may also have the following additional technical features:

[0025] According to one embodiment of the present application, the resampling module is also used to: extract the unimodal features of each unimodal sample in the multimodal training samples based on the multimodal feature extraction encoder of the candidate multimodal data label model; resample the unimodal features of each unimodal sample based on the multimodal feature resampling layer of the candidate multimodal data label model to obtain the unimodal resampled features of each unimodal sample; and splice the unimodal resampling features of each unimodal sample to obtain the multimodal splicing features extracted from the multimodal training samples.

[0026] According to one embodiment of the present application, the acquisition module is also used to: acquire a downstream task set of the candidate multimodal data label model, and a query prompt vector combination of each downstream task in the downstream task set, wherein, for any downstream task, the query prompt vector combination of the downstream task includes an initial multimodal fusion prompt vector and an initial label prompt vector of the downstream task; acquire a target downstream task associated with the multimodal training sample in the downstream task set, and determine the initial multimodal fusion prompt vector in the query prompt vector combination of the target downstream task as the candidate multimodal fusion prompt vector, and determine the initial label prompt vector as the candidate label prompt vector.

[0027] According to one embodiment of the present application, the acquisition module is also used to: obtain the feature extraction space dimension of the candidate multimodal data label model; based on the feature extraction space dimension, randomly construct an initial multimodal query prompt vector and an initial label query prompt vector for each downstream task in the downstream task set to obtain a query prompt vector combination for each downstream task.

[0028] According to one embodiment of the present application, the training module is also used to: splice the candidate multimodal fusion prompt vector, the candidate label prompt vector and the sample label of the multimodal training sample to obtain a first splicing vector; obtain the masking strategy of the first self-attention layer of the multimodal feature fusion encoder and the second self-attention layer of the label encoder of the candidate multimodal data label model; based on the masking strategy, obtain the training loss set through the first splicing vector and / or the multimodal splicing feature.

[0029] According to one embodiment of the present application, the training module is also used to: obtain the first fusion feature output by the multimodal feature fusion encoder through the first splicing vector and / or the multimodal splicing feature, and obtain the second fusion feature output by the label encoder; in response to the masking strategy being the first masking strategy, obtain the multimodal label contrast learning training loss MLC according to the first fusion feature and the second fusion feature; in response to the masking strategy being the second masking strategy, obtain the multimodal label matching training loss MLM according to the first fusion feature and the binary classification label of the multimodal training sample; in response to the masking strategy being the third masking strategy, obtain the multimodal label generation training loss MLG according to the output label in the second fusion feature and the sample label of the multimodal training sample; obtain the training loss set according to the multimodal label contrast learning training loss, the multimodal label matching training loss and the multimodal label generation training loss.

[0030] According to one embodiment of the present application, the training module is also used to: obtain M first blocks in the multimodal feature fusion encoder and N second blocks in the label encoder, wherein each first block in the M first blocks is connected in series, each second block in the N second blocks is connected in series, and M and N have the same value; for the i-th first block and the i-th second block, in response to i being equal to M, obtain the first candidate fusion feature output by the i-1th first block and the second candidate fusion feature output by the i-1th second block through the first splicing vector and / or the multimodal splicing feature; obtain the first candidate fusion feature, the second candidate fusion feature and the second splicing vector of the sample label to obtain the first fusion feature output by the i-th first block, and the second fusion feature output by the i-th second block.

[0031] According to one embodiment of the present application, the training module is also used to: in response to i-1 being equal to 1, obtain, according to the first splicing vector, a third splicing vector outputted by the first self-attention layer of the i-1th first block and the second self-attention layer of the i-1th second block, respectively, and obtain the first candidate fusion feature and the second candidate fusion feature according to the candidate multimodal fusion prompt vector, the candidate label prompt vector and the third splicing vector; in response to i-1 being greater than 1, obtain, according to the third candidate fusion feature outputted by the i-2th first block and the fourth candidate fusion feature outputted by the i-2th second block, a third splicing vector outputted by the first self-attention layer of the i-1th first block and the second self-attention layer of the i-1th second block, respectively, and obtain the first candidate fusion feature and the second candidate fusion feature according to the third candidate fusion feature, the third splicing vector, the multimodal splicing feature and the sample label.

[0032] According to one embodiment of the present application, the training module is also used to: in response to i being equal to M, obtain, according to the second splicing vector, a fourth splicing vector output by the first self-attention layer of the i-th first block and the second self-attention layer of the i-th second block; obtain the first fusion feature and the second fusion feature according to the first candidate fusion feature, the fourth splicing vector, the multimodal splicing feature, the second candidate fusion feature and the sample label.

[0033] According to one embodiment of the present application, the training module is also used to: obtain a target training loss based on the training loss set; adjust the iteration parameters of the candidate multimodal data label model based on the target training loss, wherein the iteration parameters include at least the iteration vector parameters of the candidate multimodal fusion prompt vector and the candidate label prompt vector, and the iteration model parameters of the multimodal data label model, and the iteration model parameters do not include the encoder parameters of the image unimodal feature extraction encoder in the multimodal feature extraction encoder; return to obtain the next multimodal training sample to continue training the candidate multimodal data label model after the iteration parameters are adjusted until the training is completed, and obtain the trained target multimodal data label model.

[0034] According to one embodiment of the present application, the training module is also used to: obtain the target multimodal fusion prompt vector and the target label prompt vector used in the target training round of the candidate multimodal data label model, wherein the target training round is the training round for obtaining the trained target multimodal data label model; based on the target multimodal fusion prompt vector and the target label prompt vector, obtain the target query prompt vector combination of the target downstream task to which the multimodal training sample belongs.

[0035] The third aspect of the present application proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the training method of the multimodal data label model proposed in the first aspect above.

[0036] The fourth aspect of the present application proposes a computer-readable storage medium, wherein the computer instructions are used to enable the computer to execute the training method of the multimodal data labeling model proposed in the first aspect.

[0037] The fifth aspect of the present application proposes a computer program product, including a computer program, which, when executed by a processor, implements the training method of the multimodal data labeling model proposed in the first aspect above.

[0038] The training method and device of the multimodal data label model proposed in the present application construct a candidate multimodal data label model to be trained, and obtain multimodal training samples of the candidate multimodal data label model, perform feature extraction and resample on the multimodal training samples, obtain multimodal splicing features of the multimodal training samples, and obtain the training loss set of the candidate multimodal data label model based on the multimodal splicing features and the candidate multimodal fusion prompt vectors and candidate label prompt vectors corresponding to the obtained multimodal training samples, and then train the candidate multimodal data label model according to the training loss set until the training is completed, so as to obtain a trained target multimodal data label model. In the present application, the features extracted from each unimodal sample of the multimodal training samples are resampled to avoid the abnormal situation that the sequence dimension of the features extracted from each unimodal sample exceeds the upper limit of the input dimension of the candidate multimodal data label model, thereby improving the stability of the candidate multimodal data label model, optimizing the noise filtering effect and learning effect of the candidate multimodal data label model, and improving the alignment degree of multimodal information of the candidate multimodal data label model based on multiple training losses included in the training loss set. The candidate multimodal data label model is trained based on the multimodal training samples, thereby improving the accuracy of the trained target multimodal data label acquisition model in acquiring labels for multimodal data.

[0039] It should be understood that the contents described in this application are not intended to identify the key or important features of the embodiments of this application, nor are they intended to limit the scope of this application. Other features of this application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0041] Figure 1 A schematic diagram of a flow chart of a method for training a multimodal data labeling model according to an embodiment of the present application;

[0042] Figure 2 A schematic diagram of a flow chart of a method for training a multimodal data labeling model according to another embodiment of the present application;

[0043] Figure 3 A schematic diagram of a flow chart of a method for training a multimodal data labeling model according to another embodiment of the present application;

[0044] Figure 4 A schematic diagram of a flow chart of a method for training a multimodal data labeling model according to another embodiment of the present application;

[0045] Figure 5 A schematic diagram of a flow chart of a method for training a multimodal data labeling model according to another embodiment of the present application;

[0046] Figure 6 A schematic diagram of a mask strategy according to an embodiment of the present application;

[0047] Figure 7 A schematic diagram of the structure of a training device for a multimodal data label model according to an embodiment of the present application;

[0048] Figure 8 It is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0049] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0050] The following describes the training method, device and electronic device of the multimodal data labeling model of the embodiment of the present application with reference to the accompanying drawings.

[0051] Figure 1 A flowchart of a method for training a multimodal data label model according to an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes:

[0052] S101, constructing a candidate multimodal data label model to be trained, and obtaining multimodal training samples.

[0053] In an embodiment of the present application, a corresponding model can be constructed for obtaining labels for multimodal data, and the constructed model can be trained to obtain a model for obtaining labels for multimodal data.

[0054] Among them, the constructed model for obtaining labels for multimodal data can be determined as a candidate multimodal data label model to be trained, and the samples used when training the model can be determined as multimodal training samples.

[0055] Optionally, a model may be constructed based on a model construction method in related technologies to obtain a candidate multimodal data label model to be trained.

[0056] In this scenario, multimodal data with confirmed accurate labels can be obtained, and the multimodal data and its corresponding accurate labels can be determined as multimodal training samples of the candidate multimodal data label model to be trained.

[0057] S102, performing feature extraction and resampling on the multimodal training samples to obtain multimodal splicing features of the multimodal training samples.

[0058] In an embodiment of the present application, when extracting features from each unimodal sample in a multimodal training sample, there may be differences in the parameters of the features extracted, such as the sequence length parameters and information density parameters of the features extracted from each unimodal sample.

[0059] In this scenario, the features extracted from each unimodal training sample in the multimodal training sample can be resampled to obtain unimodal features obtained after feature extraction and resampling of each unimodal training sample.

[0060] Optionally, the unimodal features obtained after feature extraction and resampling of each unimodal training sample may be further spliced ​​together, and the spliced ​​features may be determined as multimodal splicing features of the multimodal training sample.

[0061] As an example, the candidate multimodal data label model can be set with Figure 2 The multi-modal feature extraction encoder is shown, and the resampling layer is set after each single-modal feature extraction encoder.

[0062] like Figure 2 As shown, the multimodal training samples may include samples of image modality, samples of text modality, samples of audio modality, and samples of other modalities. The multimodal training samples may be based on Figure 2 The candidate multimodal data label model is input into the image modality input, text modality input, audio modality input and other modality input methods as shown.

[0063] Optionally, feature extraction is performed on the image modal input through the image modality encoder of the image modality, feature extraction is performed on the text modal input through the text modality encoder of the text modality, feature extraction is performed on the audio modal input through the audio modality encoder of the audio modality, and feature extraction is performed on other modal inputs through other modality encoders of other modalities, and the features extracted by each unimodal encoder are resampled based on the resampling layer set after the feature extraction encoder of each unimodal modality, so as to obtain the unimodal features of the samples of each unimodality included in the multimodal training samples after feature extraction and resampling.

[0064] like Figure 2 As shown, it can be Figure 2 The feature concatenation layer shown concatenates the unimodal features obtained after feature extraction and resampling of each unimodal sample in the multimodal training sample, thereby obtaining the multimodal concatenation features of the multimodal training sample.

[0065] S103, obtaining candidate multimodal fusion prompt vectors and candidate label prompt vectors corresponding to the multimodal training samples.

[0066] In an embodiment of the present application, the candidate multimodal data label model is provided with a candidate multimodal fusion prompt vector and a candidate label prompt vector.

[0067] Optionally, the candidate multimodal fusion hint vector and the candidate label hint vector correspond to the multimodal training samples, wherein the downstream task to which the multimodal training samples belong can be obtained, and the multimodal fusion query hint vector of the downstream task to which the multimodal training samples belong is used as the candidate multimodal fusion query hint vector used in the training process of the candidate multimodal data label model, and the label query hint vector of the downstream task to which the multimodal training samples belong is used as the candidate label hint vector used in the training process of the candidate multimodal data label model.

[0068] like Figure 2 As shown, the candidate multimodal data label model is set up with Figure 2 The multimodal feature fusion encoder group and the label encoder group shown in FIG. 1 are used as the candidate multimodal fusion prompt vector Q1 during the operation of the candidate multimodal data label model. Figure 2 As shown in the input of the multimodal data fusion encoder group, the candidate label prompt vector Q2 can be used as Figure 2 The input of the label encoder group is shown to implement the operation of the candidate multimodal data label model.

[0069] S104, based on the candidate multimodal fusion prompt vector, the candidate label prompt vector and the multimodal splicing feature, obtain the training loss set of the candidate multimodal data label model to perform model training on the candidate multimodal data label model until the training is completed, and obtain the trained target multimodal data label model.

[0070] In an embodiment of the present application, the candidate multimodal data label model can be trained using multimodal training samples to obtain multiple training losses of the candidate multimodal data label model, and the target training loss of the candidate multimodal data label model is obtained based on the multiple training losses, thereby achieving alignment of each single modal information in the multimodal information.

[0071] Optionally, for any training round of the candidate multimodal data label model, the candidate multimodal data label model may have multiple training losses during the training process of this round, and a set consisting of the multiple training losses obtained can be determined as the training loss set of the candidate multimodal data label model obtained in this training round.

[0072] Among them, the training loss set may include multimodal label contrast learning training loss, multimodal label matching training loss, multimodal label generation training loss and other training losses, which are not specifically limited here.

[0073] Optionally, the inputs of the multimodal feature fusion encoder group and the label encoder group of the candidate multimodal data label model can be obtained respectively through the candidate multimodal fusion prompt vector, the candidate label prompt vector and the multimodal splicing features, and the training loss set of the candidate multimodal data label model can be obtained based on the respective output results of the multimodal feature fusion encoder group and the label encoder group.

[0074] Among them, for any training loss, the algorithm factor required by the loss value algorithm corresponding to the training loss can be obtained, and the specific value of the algorithm factor required by the loss value algorithm can be obtained from the output result of the multimodal feature fusion encoder group and / or the output result of the label encoder group, and then the specific value of the algorithm factor is algorithmically processed based on the loss value algorithm, and then the corresponding loss value is obtained as the training loss based on the result of the algorithm processing.

[0075] It should be noted that the specific values ​​of the algorithm factors required for the operation of the loss value algorithm may need to be obtained only from the output results of the multimodal feature fusion encoder group, may need to be obtained only from the output results of the label encoder group, or may need to be obtained from the output results of the multimodal feature fusion encoder group and the output results of the label encoder group.

[0076] Optionally, in a scenario where the specific values ​​of the algorithm factors required for the operation of the loss value algorithm only need to be obtained from the output results of the multimodal feature fusion encoder group, the label encoder group may not be run during the operation of the candidate multimodal data label model.

[0077] Optionally, in a scenario where the specific values ​​of the algorithm factors required for the operation of the loss value algorithm only need to be obtained from the output results of the label encoder group, the multimodal feature fusion encoder group may not be run during the operation of the candidate multimodal data label model.

[0078] Optionally, in a scenario where the specific values ​​of the algorithm factors required for the operation of the loss value algorithm need to be obtained from the output results of the multimodal feature fusion encoder group and the output results of the label encoder group, the multimodal feature fusion encoder group and the label encoder group can be run simultaneously during the operation of the candidate multimodal data label model.

[0079] Optionally, the candidate multimodal data label model is adjusted based on the training loss set, and the adjusted candidate multimodal data label model is subjected to the next round of model training based on the next multimodal training sample until the training end condition is met to obtain a trained target multimodal data label model.

[0080] Optionally, a training end condition can be set based on the training rounds. If the cumulative training rounds of the candidate multimodal data label model in the current round meet the training end condition, the model training of the candidate multimodal data label model can be ended, and the model obtained by the current round of training can be determined as the trained target multimodal data label model.

[0081] Optionally, a training end condition can be set based on the loss value of the training loss. If the target training loss value obtained by the candidate multimodal data label model in the current training round meets the training end condition, the model training of the candidate multimodal data label model can be ended, and the model obtained in the current round of training can be determined as the trained target multimodal data label model.

[0082] The training method of the multimodal data label model proposed in the present application constructs a candidate multimodal data label model to be trained, obtains multimodal training samples of the candidate multimodal data label model, performs feature extraction and resampling on the multimodal training samples, obtains multimodal splicing features of the multimodal training samples, and obtains the training loss set of the candidate multimodal data label model based on the multimodal splicing features and the candidate multimodal fusion prompt vectors and candidate label prompt vectors corresponding to the obtained multimodal training samples, and then trains the candidate multimodal data label model according to the training loss set until the training is completed, thereby obtaining a trained target multimodal data label model. In the present application, the features extracted from each unimodal sample of the multimodal training samples are resampled to avoid the abnormal situation that the sequence dimension of the features extracted from each unimodal sample exceeds the upper limit of the input dimension of the candidate multimodal data label model, thereby improving the stability of the candidate multimodal data label model, optimizing the noise filtering effect and learning effect of the candidate multimodal data label model, and improving the alignment degree of multimodal information of the candidate multimodal data label model based on multiple training losses included in the training loss set. The candidate multimodal data label model is trained based on the multimodal training samples, thereby improving the accuracy of the trained target multimodal data label acquisition model in acquiring labels for multimodal data.

[0083] In the above embodiment, the acquisition of multimodal splicing features can be combined with Figure 3 Further understanding, Figure 3 FIG. 1 is a flow chart of a method for training a multimodal data label model according to another embodiment of the present application. Figure 3 As shown, the method includes:

[0084] S301, extracting unimodal features of each unimodal sample in the multimodal training sample based on a multimodal feature extraction encoder of a candidate multimodal data label model.

[0085] In an embodiment of the present application, the candidate multimodal data label model is provided with a multimodal feature extraction encoder, wherein the multimodal feature extraction encoder may include multiple single-modal feature extraction encoders.

[0086] like Figure 2 As shown, Figure 2 The image modality encoder shown is a unimodal feature extraction encoder corresponding to the image modality, the text modality encoder is a unimodal feature extraction encoder corresponding to the text modality, the audio modality encoder is a unimodal feature extraction encoder corresponding to the audio modality, and the other modality encoders are unimodal feature extraction encoders corresponding to other modalities.

[0087] Optionally, feature extraction may be performed on the multimodal training samples by using a plurality of unimodal feature extraction encoders included in the multimodal feature extraction encoder, thereby obtaining unimodal features of each unimodal sample.

[0088] Among them, for any unimodal sample in the multimodal training samples, a unimodal feature extraction encoder of the same modality can be obtained in the multimodal feature extraction encoding, and the unimodal sample can be input into the unimodal feature extraction encoder to obtain the unimodal features of the unimodal sample output by the unimodal feature extraction encoding.

[0089] like Figure 2 As shown, for the image modality samples in the multimodal samples, the image modality encoder of the image modality can be obtained in the multimodal feature extraction encoder, and the image modality samples can be extracted by Figure 2 The image modality input method shown is input into the image modality encoder to obtain the unimodal features of the image modality output by the image modality encoder.

[0090] It should be noted that the process of acquiring the unimodal features of the text modal samples, audio modal samples and other modal samples in the multimodal samples can be understood in conjunction with the details of the process of acquiring the unimodal features of the image modality of the above-mentioned image modal samples, and will not be repeated here.

[0091] S302, based on the multimodal feature resampling layer of the candidate multimodal data label model, resample the unimodal features of each unimodal sample to obtain the unimodal resampled features of each unimodal sample.

[0092] In an embodiment of the present application, the candidate multimodal data label model is provided with a post-multimodal feature resampling layer after the multimodal feature extraction encoder.

[0093] Optionally, the multimodal feature resampling layer may be constructed based on an attention method or other methods, which are not specifically limited here.

[0094] Among them, the unimodal features output by feature extraction coding of the same modality can be resampled through each unimodal resampling layer included in the multimodal resampling layer, so as to obtain the unimodal resampling features of each unimodal sample in the multimodal training sample.

[0095] like Figure 2 As shown, the multimodal feature resampling layer may include an image modality resampling layer, a text modality resampling layer, an audio modality resampling layer, and other modality resampling layers.

[0096] In this scenario, the unimodal features of the image modality output by the image modality encoder can be resampled through the image modality resampling layer to obtain the unimodal resampled features of the unimodal samples of the image modality in the multimodal training samples.

[0097] Correspondingly, the unimodal features of the text modality output by the text modality encoder can be resampled through the text modality resampling layer to obtain the unimodal resampled features of the unimodal samples of the text modality in the multimodal training samples.

[0098] It should be noted that the process of acquiring the unimodal resampling features of unimodal samples corresponding to the audio modality in the multimodal training samples, and the unimodal resampling features of unimodal samples corresponding to other modalities, can be understood in conjunction with the details of the process of acquiring the unimodal resampling features of unimodal samples in the above-mentioned image modality and the unimodal resampling features of unimodal samples in the text modality, and will not be repeated here.

[0099] S303, splicing the unimodal resampling features of each unimodal sample to obtain multimodal splicing features extracted from the multimodal training sample.

[0100] In an embodiment of the present application, the candidate multimodal data label model is provided with a feature splicing layer, through which the unimodal resampling features of each unimodal sample included in the multimodal training sample can be spliced ​​to obtain the multimodal splicing features of the multimodal training sample.

[0101] like Figure 2 As shown, the unimodal resampling features of the unimodal samples of the image modality output by the image modality resampling layer, the unimodal resampling features of the unimodal samples of the text modality output by the text modality resampling layer, the unimodal resampling features of the unimodal samples of the audio modality output by the audio modality resampling layer, and the unimodal resampling features of the unimodal samples of other modalities output by the other modality resampling layers can be obtained respectively, and Figure 2 The feature splicing layer shown splices the unimodal resampling features of unimodal samples in image modality, the unimodal resampling features of unimodal samples in text modality, the unimodal resampling features of unimodal samples in audio modality, and the unimodal resampling features of unimodal samples in other modalities, thereby obtaining multimodal splicing features extracted from multimodal training samples.

[0102] The training method of the multimodal data label model proposed in the present application extracts features from each unimodal sample in the multimodal training sample through a multimodal feature extraction encoder to obtain the unimodal features of each unimodal sample, and resamples the unimodal features of each unimodal sample based on the multimodal feature resampling layer to obtain the unimodal resampled features of each unimodal sample. Furthermore, the unimodal resampled features of each unimodal sample are spliced ​​to obtain the multimodal splicing features extracted from the multimodal training sample. In the present application, the features extracted from each unimodal sample of the multimodal training sample are resampled to avoid the abnormal situation that the sequence dimension of the features extracted from each unimodal sample exceeds the upper limit of the input dimension of the candidate multimodal data label model, thereby improving the stability of the candidate multimodal data label model and optimizing the noise filtering effect and learning effect of the candidate multimodal data label model.

[0103] In the above embodiment, the acquisition of the candidate multimodal fusion hint vector and the candidate label hint vector can be combined with Figure 4 Further understanding, Figure 4 FIG. 1 is a flow chart of a method for training a multimodal data label model according to another embodiment of the present application. Figure 4 As shown, the method includes:

[0104] S401, obtaining a downstream task set of a candidate multimodal data labeling model, and a query prompt vector combination of each downstream task in the downstream task set, wherein, for any downstream task, the query prompt vector combination of the downstream task includes an initial multimodal fusion prompt vector and an initial label prompt vector of the downstream task.

[0105] In an embodiment of the present application, after obtaining labels for multimodal data through a candidate multimodal data label model, downstream tasks may be performed based on the multimodal data with the labels obtained.

[0106] In this scenario, the candidate multimodal data labeling model may include a downstream task set consisting of multiple downstream tasks.

[0107] In implementation, for different downstream tasks, there may be differences between the focus of multimodal data fusion representation and the focus of label text.

[0108] In this scenario, for any downstream task in the downstream task set, a multimodal task query hint vector and a label hint vector can be constructed for it respectively, and marked as an initial multimodal fusion hint vector and an initial label hint vector respectively.

[0109] Optionally, the feature extraction space dimension of the candidate multimodal data label model can be obtained, and based on the feature extraction space dimension, an initial multimodal query hint vector and an initial label query hint vector are randomly constructed for each downstream task in the downstream task set to obtain a query hint vector combination for each downstream task.

[0110] In an embodiment of the present application, for any downstream task, the preset length of the multimodal fusion prompt vector can be obtained, and based on the feature extraction space dimension and the preset length, a query prompt vector with a shape of (preset length, feature extraction space dimension) is randomly constructed as the initial multimodal fusion prompt vector for the downstream task.

[0111] And, obtaining a preset length of the multi-label hint vector, and randomly constructing a query hint vector of shape (preset length, feature extraction space dimension) based on the feature extraction space dimension and the preset length as the initial label hint vector of the downstream task.

[0112] As an example, the feature extraction space dimension of the candidate multimodal data label model is set to h, the preset length of the multimodal fusion prompt vector is s1, and the preset length of the label prompt vector is s2.

[0113] For any downstream task, a query hint vector with a shape of (s1, h) can be randomly constructed as the initial multimodal fusion hint vector of the downstream task, and a query hint vector with a shape of (s2, h) can be randomly constructed as the initial multimodal fusion hint vector of the downstream task.

[0114] Furthermore, the combination of the initial multimodal fusion hint vector of each downstream task and the initial multimodal fusion hint vector is marked as the query hint vector combination of each downstream task.

[0115] S402, obtaining a target downstream task associated with the multimodal training sample in the downstream task set, and determining an initial multimodal fusion hint vector in the query hint vector combination of the target downstream task as a candidate multimodal fusion hint vector, and determining an initial label hint vector as a candidate label hint vector.

[0116] In an embodiment of the present application, there is a downstream task set for the candidate multimodal data label model, wherein there are differences between the multimodal training samples of each downstream task. In this scenario, the multimodal training samples used for training the current candidate multimodal data label model can be obtained, and the downstream task associated with the multimodal training sample can be obtained from each downstream task in the downstream task set as the target downstream task of the multimodal training sample.

[0117] Optionally, an initial multimodal fusion hint vector and an initial label hint vector included in the query hint vector combination of the target downstream task may be obtained, and the initial multimodal fusion hint vector may be determined as a candidate multimodal fusion hint vector adopted by the candidate multimodal data label model in the current training process.

[0118] And, the initial label prompt vector is determined as the candidate label prompt vector adopted by the candidate multimodal data label model in the current training process.

[0119] The training method of the multimodal data label model proposed in the present application obtains the downstream task set of the candidate multimodal data label model and the query prompt vector combination including the initial multimodal fusion prompt vector and the initial label prompt vector of each downstream task in the downstream task set, and determines the target downstream task associated with the multimodal training sample in the downstream task set. Further, the initial multimodal fusion prompt vector included in the query prompt vector combination of the target downstream task is determined as the candidate multimodal fusion prompt vector, and the initial label prompt vector is determined as the candidate label prompt vector. In the present application, the initial multimodal fusion prompt vector and the initial label prompt vector of the downstream task associated with the multimodal training sample are obtained as the candidate multimodal fusion prompt vector and the candidate label prompt vector used by the candidate multimodal data label model when training based on the multimodal training sample, so that the candidate multimodal data label model can output the output result suitable for the corresponding downstream task, improve the degree of adaptation between the output result of the trained target multimodal data label model and the corresponding downstream task, enhance the practicality and applicability of the target multimodal data label model, and optimize the performance of the target multimodal data label model.

[0120] In the above embodiment, the training of the candidate multimodal data label model can be combined with Figure 5 Further understanding, Figure 5 FIG. 1 is a flow chart of a method for training a multimodal data label model according to another embodiment of the present application. Figure 5 As shown, the method includes:

[0121] S501 , concatenating a candidate multimodal fusion hint vector, a candidate label hint vector, and a sample label of a multimodal training sample to obtain a first concatenated vector.

[0122] In an embodiment of the present application, the candidate multimodal data label model is provided with a multimodal feature fusion encoder group and a label encoder group, which can realize the operation of the candidate multimodal data label model based on the candidate multimodal fusion prompt vector, the candidate label prompt vector and the multimodal training samples corresponding to the multimodal training samples, thereby obtaining the training loss set of the candidate multimodal data label model.

[0123] Optionally, sample labels of multimodal training samples may be obtained, and the candidate multimodal fusion prompt vector, the candidate label prompt vector and the sample label may be concatenated, and the concatenated vector may be determined as a first concatenated vector.

[0124] The first concatenated vector may be used as an input vector of the first encoder of each of the multimodal feature fusion encoder group and the label encoder group.

[0125] As an example, Figure 2 As shown, set Figure 2 The multimodal feature fusion encoder shown is the first encoder in the group of multimodal feature fusion encoders of the candidate multimodal data label model, and, Figure 2 The label encoder shown is the first encoder in the set of label encoders for the candidate multimodal data label model.

[0126] In this example, the first concatenation vectors can be used as Figure 2 The input vectors of the first encoder in the multimodal feature fusion encoder group and the first encoder in the label encoder group are shown.

[0127] S502, obtaining a mask strategy of a first self-attention layer of a multimodal feature fusion encoder and a second self-attention layer of a label encoder of a candidate multimodal data label model.

[0128] In an embodiment of the present application, a mask strategy is set for the self-attention of the multimodal feature fusion encoder and the label encoder of the candidate multimodal data label model, wherein the self-attention layer of the multimodal feature fusion encoder can be marked as the first self-attention layer, and the self-attention layer of the label encoder can be marked as the second self-attention layer, and the first self-attention layer and the second self-attention layer are set to share parameters.

[0129] As an example, the masking strategy of the first self-attention layer of the multimodal feature fusion encoder and the second self-attention layer of the label encoder can be as follows Figure 6 As shown, Figure 6 Three masking strategies are shown, where the first position represents the character (token) position of the candidate multimodal fusion hint vector, and the second position represents the character (token) position of the candidate label hint vector.

[0130] Among them, the mask strategy in the first and second self-attention layers is set to Figure 6 In the scenario of the first mask strategy shown, the first self-attention layer can learn the carried information in the candidate multimodal fusion prompt vector, but cannot learn the carried information in the candidate label prompt vector and the sample label.

[0131] Correspondingly, the second self-attention layer can learn the carrying information in the candidate label hint vector and the sample label, but cannot learn the carrying information in the candidate multimodal fusion hint vector.

[0132] The mask strategy in the first and second self-attention layers is set to Figure 6 In the scenario of the second mask strategy shown, both the first self-attention layer and the second self-attention layer can learn the carried information in the candidate multimodal fusion prompt vector and the carried information in the candidate label prompt vector and the sample label.

[0133] The mask strategy in the first and second self-attention layers is set to Figure 6 In the scenario of the third mask strategy shown, the second self-attention layer can learn the carrying information in the candidate multimodal fusion prompt vector and the carrying information in the candidate label prompt vector, but cannot learn the carrying information in the sample label.

[0134] S503: Based on the mask strategy recognition result, a training loss set is obtained through the first splicing vector and / or the multimodal splicing feature.

[0135] In an embodiment of the present application, in a scenario where the first self-attention layer and the second self-attention layer adopt different masking strategies, the candidate multimodal data label model can obtain different training losses, thereby obtaining a training loss set of the candidate multimodal data label model.

[0136] Optionally, a first fusion feature output by the multimodal feature fusion encoder is obtained through the first splicing vector and / or the multimodal splicing feature, and a second fusion feature output by the label encoder is obtained.

[0137] In an embodiment of the present application, a fusion feature output by the multimodal feature fusion encoder based on the first splicing vector and / or the multimodal splicing feature may be obtained, and the fusion feature may be marked as the first fusion feature output by the multimodal feature fusion encoder.

[0138] Accordingly, a fusion feature of the output obtained by the label encoder based on the first splicing vector and / or the multimodal splicing feature can be obtained, and the fusion feature can be marked as the second fusion feature output by the label encoder.

[0139] Optionally, M first blocks in the multimodal feature fusion encoder and N second blocks in the label encoder are obtained, wherein each first block in the M first blocks is connected in series, each second block in the N second blocks is connected in series, and M and N have the same value.

[0140] In an embodiment of the present application, M encoders are provided in the multimodal feature fusion encoder, wherein the M encoders provided in the multimodal feature fusion encoder can be marked as M first processing units (blocks), and correspondingly, N encoders are provided in the label encoder, wherein the N encoders provided in the label encoder can be marked as N second blocks.

[0141] It should be noted that the values ​​of M and N are the same.

[0142] like Figure 2 As shown, Figure 2 shows one of the first blocks set in the multimodal feature fusion encoder, through Figure 2 From the ×M shown, we can see that Figure 2 The multimodal feature fusion encoder shown is provided with M first blocks.

[0143] Accordingly, Figure 2 shows one of the second blocks set in the tag encoder, through Figure 2 From the ×N shown, we can see that Figure 2 The tag encoder shown is provided with N second blocks.

[0144] Optionally, for the i-th first block and the i-th second block, in response to i being equal to M, the first candidate fusion feature output by the i-1-th first block and the second candidate fusion feature output by the i-1-th second block are obtained through the first splicing vector and / or the multimodal splicing feature.

[0145] In the embodiment of the present disclosure, when i is equal to M, the i-th first block is the first block ranked at the Mth position among the M first blocks in series. In this scenario, the output result of the first block can be obtained and determined as the first fusion feature output by the multimodal feature fusion encoder.

[0146] Correspondingly, since M and N have the same value, it can be seen that when i is equal to M, the i-th second block is the second block ranked at the Nth position in the N second blocks in series. In this scenario, the output result of the second block can be obtained and determined as the second fusion feature output by the label fusion encoder.

[0147] In this scenario, the first candidate fusion feature output by the i-1th first block and the second candidate fusion feature output by the i-1th second block can be used to obtain the inputs of the Mth first block and the Nth second block respectively.

[0148] Optionally, a second concatenated vector of the first candidate fusion feature, the second candidate fusion feature, and the sample label is obtained to obtain a first fusion feature output by the i-th first block and a second fusion feature output by the i-th second block.

[0149] As a possible implementation method, in response to i-1 being equal to 1, a third splicing vector output by the first self-attention layer of the i-1th first block and the second self-attention layer of the i-1th second block is obtained according to the first splicing vector, and a first candidate fusion feature and a second candidate fusion feature are obtained according to the candidate multimodal fusion prompt vector, the candidate label prompt vector and the third splicing vector.

[0150] In the embodiment of the present application, when i-1 is equal to 1, it can be determined that the i-1th first block is the first block in the multimodal feature fusion encoder, and the i-1th second block is the first block in the label encoder.

[0151] As an example, Figure 2 As shown, the first splicing vector obtained by splicing the candidate multimodal fusion prompt vector Q1, the candidate label prompt vector Q2 and the sample label can be respectively input into the first self-attention layer in the i-1th block and the second self-attention layer of the i-1th second block, and the output vectors of the first self-attention layer of the i-1th first block and the second self-attention layer of the i-1th second block based on the first splicing vector are respectively marked as the third splicing vectors output by the first self-attention layer of the i-1th first block and the second self-attention layer of the i-1th second block.

[0152] Optionally, a first partial vector of the candidate multimodal fusion hint vector in the third splicing vector may be obtained, the first partial vector may be determined as a first Q value, and the first partial vector and the multimodal splicing feature may be spliced ​​to obtain a first K value and a first V value.

[0153] Based on the above example, Figure 2 As shown, the splicing position of Q1 in the first splicing vector can be obtained, and the vector at the same position in the third splicing vector is determined as the first partial vector in the third splicing vector, and the first splicing vector is determined as the first Q value of the cross-attention layer of the input i-1th first block.

[0154] Furthermore, if Figure 2 As shown, it can be Figure 2The feature splicing module shown splices the first part of the vector with the multimodal splicing feature again, and then determines the spliced ​​vector as the first K value and the first V value of the cross attention layer of the i-1th first block.

[0155] Optionally, the first K value, the first V value, and the first Q value are input into the cross-attention layer of the i-1th first block to obtain a first initial fused feature output by the cross-attention layer of the i-1th first block, and the first initial fused feature is linearly transformed based on the first linear layer of the i-1th first block to obtain a first candidate fused feature output by the i-1th first block.

[0156] like Figure 2 As shown, the first K value, the first V value and the first Q value can be input into the cross-attention layer of the i-1th first block, and the output vector obtained by the cross-attention layer of the i-1th first block based on the first K value, the first V value and the first Q value can be determined as the first initial fusion feature output by the cross-attention layer of the i-1th first block.

[0157] Furthermore, the first initial fusion feature is input into Figure 2 The first linear layer of the i-1th first block shown is used to obtain the output of the linear transformation of the first initial fusion feature by the first linear layer of the i-1th first block, and the output is used as the first candidate fusion feature output by the i-1th first block.

[0158] Optionally, a second partial vector of the candidate label prompt vector and the sample label in the second concatenated vector is obtained, and a linear transformation is performed on the second partial vector based on the second linear layer of the i-1th second block to obtain a second candidate fusion feature output by the i-1th second block.

[0159] like Figure 2 As shown, the positions of the candidate label prompt vector Q2 and the sample label in the first splicing vector can be obtained, and the vectors at the same positions in the third splicing vector are determined as the second part vectors in the third splicing vector.

[0160] Further, the second partial vector is used as an input of the second linear layer of the i-1th second block, the second partial vector is linearly transformed through the second linear layer, and the output of the second linear layer is used as the second candidate fusion feature output by the i-1th second block.

[0161] As another possible implementation, in response to i-1 being greater than 1, based on the third candidate fusion feature output by the i-2th first block and the fourth candidate fusion feature output by the i-2th second block, the third splicing vector output by the first self-attention layer of the i-1th first block and the second self-attention layer of the i-1th second block are obtained, and based on the third candidate fusion feature, the third splicing vector, the multimodal splicing feature and the sample label, the first candidate fusion feature and the second candidate fusion feature are obtained.

[0162] In this embodiment of the present application, when i-1 is greater than 1, it can be determined that the i-1th first block is a non-first encoder in the multimodal feature fusion encoder, and the i-1th second block is a non-first encoder in the label encoder.

[0163] In this scenario, the inputs of the i-1th first block and the i-1th second block can be obtained according to the outputs of the i-2th first block and the i-2th second block.

[0164] Among them, the output of the i-2th first block can be marked as the third candidate fusion feature, and the output of the i-2th second block can be marked as the fourth candidate fusion feature.

[0165] Optionally, a concatenated vector of the third candidate fusion feature, the fourth candidate fusion feature, and the sample label is obtained as the second input vector of each of the i-1th first block and the i-1th second block.

[0166] In an embodiment of the present application, the third candidate fusion feature, the fourth candidate fusion feature and the sample label can be spliced ​​based on the feature splicing algorithm in the related technology, and the spliced ​​vector is determined as the input of the i-1th first block and the i-1th second block respectively, and marked as the second input vector.

[0167] Optionally, the second input vector is respectively input into the first self-attention layer of the i-1th first block and the second self-attention layer of the i-1th second block to obtain a third splicing vector output by the first self-attention layer of the i-1th first block and the second self-attention layer of the i-1th second block.

[0168] In an embodiment of the present application, the second input vector can be input into the i-1th first block and the i-1th second block respectively, wherein it can be input into the first self-attention layer of the i-1th first block and the second self-attention layer of the i-1th second block, and the output vectors obtained by the first self-attention layer of the i-1th first block and the second self-attention layer of the i-1th second block based on the second input vector are determined as the third splicing vectors output by the first self-attention layer of the i-1th first block and the second self-attention layer of the i-1th second block.

[0169] Optionally, a third partial vector of the third candidate fusion feature in the third splicing vector is obtained, the third partial vector is determined as the second Q value, and the third partial vector and the multimodal splicing feature are spliced ​​to obtain a second K value and a second V value.

[0170] Among them, the splicing position of the third candidate fusion feature in the second input vector can be obtained, and the vector at the same position in the third splicing vector is determined as the third partial vector in the third splicing vector, so as to obtain the Q value of the cross-attention layer of the input i-1th first block, and mark it as the second Q value.

[0171] Accordingly, based on Figure 2 The feature splicing module shown splices the third part vector with the multimodal splicing feature, and determines the spliced ​​vector as the K value and V value of the cross attention layer of the input i-1th first block, and marks them as the second K value and the second V value respectively.

[0172] Optionally, the second K value, the second V value, and the second Q value are input into the cross-attention layer of the i-1th first block to obtain a second initial fused feature output by the cross-attention layer of the i-1th first block, and the second initial fused feature is linearly transformed based on the first linear layer of the i-1th first block to obtain a first candidate fused feature output by the i-1th first block.

[0173] In an embodiment of the present application, the second K value, the second V value and the second Q value can be input into the cross-attention layer of the i-1th first block, and the output vector obtained by the cross-attention layer of the i-1th first block based on the second K value, the second V value and the second Q value can be determined as the second initial fusion feature output by the cross-attention layer of the i-1th first block.

[0174] Furthermore, the second initial fused feature is input into the first linear layer of the i-1th first block, and the output of the linear transformation of the second initial fused feature by the first linear layer of the i-1th first block is obtained, and the output is used as the first candidate fused feature output by the i-1th first block.

[0175] Optionally, a fourth partial vector of the fourth candidate fusion feature and the sample label in the third concatenated vector is obtained, and a linear transformation is performed on the fourth partial vector based on the second linear layer of the i-1th second block to obtain a second candidate fusion feature output by the i-1th second block.

[0176] In the embodiment of the present application, the splicing position of the fourth candidate fusion feature and the sample label in the second input vector can be obtained, and the vector at the same position in the third splicing vector is determined as the fourth partial vector.

[0177] Further, the fourth partial vector is input into the second linear layer of the i-1th second block, and the fourth partial vector is linearly transformed based on the second linear layer of the i-1th second block, so as to obtain the output of the second linear layer of the i-1th second block based on the fourth partial vector, and the output is used as the second candidate fusion feature of the i-1th second block output.

[0178] Optionally, a second concatenated vector of the first candidate fusion feature, the second candidate fusion feature, and the sample label is obtained to obtain a first fusion feature output by the i-th first block and a second fusion feature output by the i-th second block.

[0179] In which, in response to i being equal to M, according to the second splicing vector, the fourth splicing vector output by the first self-attention layer of the i-th first block and the second self-attention layer of the i-th second block are obtained, and the first fusion feature and the second fusion feature are obtained according to the first candidate fusion feature, the fourth splicing vector, the multimodal splicing feature, the second candidate fusion feature and the sample label.

[0180] In an embodiment of the present application, for the M first blocks in the multimodal feature fusion encoder, when any first block is not the first block among the M first blocks, and for the N second blocks in the label encoder, when any second block is not the first block among the N second blocks, the inputs of the first block and the second block can be obtained from the output of the first block at the previous level of the first block and the output of the second block at the previous level of the second block.

[0181] In this scenario, the inputs of the i-th first block and the i-th second block can be obtained based on the first candidate fusion features output by the i-1-th first block and the second candidate fusion features output by the i-1-th second block.

[0182] Among them, the first candidate fusion feature, the second candidate fusion feature and the sample label can be spliced, and the spliced ​​vector is determined as the input vector of the i-th first block and the i-th second block respectively, and marked as the first input vector.

[0183] Optionally, based on the first input vector, a first fused feature output by the i-th first block and a second fused feature output by the i-th second block are obtained.

[0184] As a possible implementation, when i is not equal to M, it can be determined that the i-th first block is not the last block among the M first blocks of the multimodal feature fusion encoder, and the i-th second block is not the last block among the N second blocks of the label encoder.

[0185] In this scenario, the output vectors and sample labels of the i-th first block and the i-th second block may be concatenated to obtain the input vectors of the i+1-th first block and the i+1-th second block.

[0186] As another possible implementation, when i is equal to M, in response to i being equal to M, the first input vector is respectively input into the first self-attention layer of the i-th first block and the second self-attention layer of the i-th second block, and a fourth splicing vector outputted by the first self-attention layer of the i-th first block and the second self-attention layer of the i-th second block is obtained.

[0187] It can be understood that when i is equal to M, it can be determined that the i-th first block is the last block of the M first blocks of the multimodal feature fusion encoder, and the i-th second block is the last block of the N second blocks of the label encoder.

[0188] In this scenario, the output result of the i-th first block can be used as the output result of the multimodal feature fusion encoder, and the output result of the i-th second block can be used as the output result of the label encoder.

[0189] Among them, the first input vector can be respectively input into the first self-attention layer of the i-th first block and the second self-attention layer of the i-th second block to obtain the splicing vectors output by the first self-attention layer of the i-th first block and the second self-attention layer of the i-th second block, and marked as the fourth splicing vector.

[0190] Optionally, the fifth partial vector of the first candidate fusion feature in the fourth splicing vector is obtained, the fifth partial vector is determined as the third Q value, and the fifth partial vector and the multimodal splicing feature are spliced ​​to obtain a third K value and a third V value.

[0191] The splicing position of the first candidate fusion feature in the first input vector may be obtained, and the vector at the same position in the fourth splicing vector may be determined as the fifth partial vector.

[0192] Further, the fifth partial vector is determined as the third Q value of the cross attention layer input into the i-th first block.

[0193] Accordingly, the fifth part of the vector is concatenated with the multimodal concatenation feature again, and the concatenated vector is determined as the third K value and the third V value of the cross attention layer of the i-th first block.

[0194] Optionally, the third K value, the third V value, and the third Q value are input into the cross-attention layer of the i-th first block to obtain a third initial fused feature output by the cross-attention layer of the i-th first block, and the third initial fused feature is linearly transformed based on the first linear layer of the i-th first block to obtain a first fused feature output by the i-th first block.

[0195] Among them, the third K value, the third V value and the third Q value can be input into the cross attention layer of the i-th first block, so as to obtain the fusion features output by the cross attention layer of the i-th first block as the three initial fusion features.

[0196] Furthermore, the third initial fused feature is input into the first linear layer of the i-th first block, and the third initial fused feature is linearly transformed by the first linear layer of the i-th first block to obtain the fused feature output by the first linear layer of the i-th first block as the first fused feature output by the multimodal feature fusion encoder.

[0197] Optionally, a sixth partial vector of the second candidate fusion feature and the sample label in the fourth concatenated vector is obtained, and a linear transformation is performed on the sixth partial vector based on the second linear layer of the i-th second block to obtain a second fusion feature output by the i-th second block.

[0198] The splicing position of the second candidate fusion feature and the sample label in the first input vector may be obtained, and the vector at the same position in the fourth splicing vector may be determined as the sixth partial vector.

[0199] Furthermore, the sixth partial vector can be input into the second linear layer of the i-th second block, and the sixth partial vector can be linearly processed by the second linear layer of the i-th second block to obtain the fused feature output by the second linear layer of the i-th second block, and determine it as the second fused feature output by the label encoder.

[0200] In the embodiment of the present application, the training loss set of the candidate multimodal data label model may include a multimodal label contrastive learning training loss (MLC). For the acquisition of the multimodal label contrastive learning training loss MLC, please refer to the following content for understanding:

[0201] Optionally, in response to the mask strategy being the first mask strategy, a multimodal label contrast learning training loss MLC is obtained according to the first fusion feature and the second fusion feature.

[0202] In an embodiment of the present application, in a scenario where the first self-attention layer and the second self-attention layer adopt a first mask strategy, the first splicing vector or the multimodal splicing feature can be input into the multimodal feature fusion encoder and the label encoder, and the output result of the multimodal fusion feature encoder can be marked as the first fusion feature, and the output result of the label encoder can be marked as the second fusion feature.

[0203] Optionally, based on the contrastive learning training loss algorithm in the related technology, the first fusion feature and the second fusion feature are algorithmically processed, and then the corresponding loss value is obtained according to the algorithm processing result, and the loss value is determined as the multimodal label contrastive learning training loss of the candidate multimodal data label model.

[0204] Optionally, based on a preset multimodal label contrast learning training loss MLC algorithm, the first fusion feature and the second fusion feature can be algorithmically processed, and a loss value is obtained according to the result of the algorithm processing to be determined as the multimodal label contrast learning training loss MLC of the candidate multimodal data label model.

[0205] It should be noted that the multimodal label contrastive learning training loss MLC can shorten the distance between the positive examples of the multimodal data and the labeled data and increase the distance between the negative examples. In-batch negative sampling can be used to construct negative examples. Momentum queues can also be used to sample negative examples so that the anchor data can see more negative examples, thereby achieving the purpose of aligning the feature space representation of the multimodal data and the feature space representation of the labeled data.

[0206] Among them, the mask strategies of the first self-attention layer and the second self-attention layer can be set to Figure 6 The first mask strategy shown corresponds to the form, so that the multimodal part and the label part can only see their own information.

[0207] Optionally, a first loss value of the first fusion feature based on the second fusion feature and a second loss value of the second fusion feature based on the first fusion feature can be obtained, and a loss mean of the first loss value and the second loss value can be obtained as the multimodal label contrast learning training loss MLC.

[0208] As an example, the algorithm formula of multimodal label contrastive learning training loss MLC can be shown as follows:

[0209]

[0210]

[0211]

[0212] In the above formula, is the first loss value of the first fusion feature based on the second fusion feature, is the second loss value of the second fusion feature based on the first fusion feature. Further, the loss mean of the first loss value and the second loss value is obtained. , That is the multimodal label contrast learning training loss MLC, is the average pooling vector of the output of the multimodal feature fusion encoder based on the kth training sample, is the output average pooling vector of the label encoder based on the sample label corresponding to the kth training sample, is the temperature coefficient, B is the number of training samples included in the training sample set to which the training sample belongs, j represents the traversal subscript, and T represents the transpose of the vector.

[0213] Among them, the training loss set of the candidate multimodal data label model may also include a multimodal label matching training loss (multimodal-label matching, MLM). For the acquisition of the multimodal label matching training loss MLM, please refer to the following content for understanding:

[0214] Optionally, in response to the mask strategy being the second mask strategy, a multimodal label matching training loss MLM is obtained according to the first fusion feature and the binary classification label of the multimodal training sample.

[0215] In an embodiment of the present application, in a scenario where the first self-attention layer and the second self-attention layer adopt a second mask strategy, the first self-attention layer can learn all the information of the candidate label prompt vector and the sample label. In this scenario, the output result of the multimodal feature fusion encoder obtained after the first splicing vector is input into the multimodal feature fusion encoder carries the feature information of the candidate label prompt vector and the sample label.

[0216] It should be noted that the multimodal label matching training loss MLM is obtained based on the label information in the output result of the multimodal feature fusion encoder and the sample labels in the training samples. It can be seen that in the process of obtaining the multimodal label matching training loss MLM, only the multimodal feature fusion encoder can be run without running the label encoder.

[0217] Optionally, M first blocks in a multimodal feature fusion encoder can be obtained, wherein each of the M first blocks is connected in series, and, for the i-th first block, in response to i being equal to M, the fifth candidate fusion feature output by the i-1-th first block is obtained through the first splicing vector and / or the multimodal splicing feature.

[0218] In the disclosed embodiment, M first blocks are provided in the multimodal feature fusion encoder, wherein each first block is connected in series, and when i is equal to M, the first block is the first block ranked at the Mth position among the M first blocks in series. In this scenario, the first block ranked at the Mth position can be obtained, and the output result of the first block is determined as the first fusion feature output by the multimodal feature fusion encoder.

[0219] In this scenario, the fifth candidate fusion feature output by the i-1th first block can be obtained, thereby obtaining the input of the Mth first block.

[0220] As a possible implementation manner, in response to i-1 being equal to 1, the first splicing vectors are respectively input into the first self-attention layer of the i-1th first block to obtain the fifth splicing vector output by the first self-attention layer of the i-1th first block.

[0221] In an embodiment of the present application, when i-1 is equal to 1, it can be determined that the i-1th first block is the first block in the multimodal feature fusion encoder. In this scenario, the first spliced ​​vector obtained by splicing the candidate multimodal feature fusion task query prompt vector, the candidate label prompt vector and the sample label can be used as the input vector of the i-1th first block.

[0222] As an example, Figure 2 As shown, set Figure 2 The first block shown is the first block in the multimodal feature fusion encoder. The first concatenated vector obtained by concatenating the candidate multimodal feature fusion task query prompt vector Q1, the candidate label prompt vector Q2 and the sample label can be input Figure 2 The first self-attention layer is shown, and the splicing vector output by the first attention layer is marked as the fifth splicing vector.

[0223] Optionally, obtain the seventh partial vector of the candidate multimodal fusion hint vector in the fifth splicing vector, determine the seventh partial vector as the fourth Q value, and splice the seventh partial vector and the multimodal splicing feature to obtain a fourth K value and a fourth V value, and input the fourth K value, the fourth V value and the fourth Q value into the cross-attention layer of the i-1th first block to obtain the fourth initial fusion feature output by the cross-attention layer of the i-1th first block, and perform a linear transformation on the fourth initial fusion feature based on the first linear layer of the i-1th first block to obtain the fifth candidate fusion feature output by the i-1th first block.

[0224] In an embodiment of the present application, the position of the candidate multimodal fusion hint vector in the first splicing vector can be obtained, and the vector at the same position in the fifth splicing vector is determined as a partial vector of the candidate multimodal fusion hint vector in the fifth splicing vector, marked as the seventh partial vector.

[0225] like Figure 2 As shown, the position of the candidate multimodal fusion hint vector Q1 in the first splicing vector can be obtained, and the vector at the same position in the fifth splicing vector is determined as the seventh partial vector Q1' of Q1 in the first splicing vector.

[0226] In this scenario, the seventh partial vector can be used as the Q value input to the cross attention layer of the i-1th first block and marked as the fourth Q value.

[0227] Correspondingly, the seventh part vector is concatenated with the multimodal concatenation feature to obtain the K value and V value of the cross-attention layer input, and are marked as the fourth V value and the fourth K value respectively.

[0228] like Figure 2 As shown, the fourth K value, the fourth V value and the fourth Q value can be input to Figure 2 The cross attention layer shown in FIG. 4 obtains the fourth initial fusion feature of the cross attention layer output. Further, through Figure 2 The first linear layer of the i-1th first block shown performs a linear transformation on the fourth initial fused feature, and determines the output of the first linear layer after the linear transformation on the fourth initial fused feature as the fifth candidate fused feature output by the i-1th first block.

[0229] As another possible implementation, in response to i-1 being greater than 1, the sixth candidate fusion feature output by the i-2th first block is obtained, and the concatenated vector of the sixth candidate fusion feature and the sample label is obtained as the fourth input vector of the i-1th first block.

[0230] In an embodiment of the present application, when i-1 is greater than 1, it can be determined that the i-2th first block is a block located at an intermediate position between the first block and the last block in the multimodal feature fusion encoder. In this scenario, the output of the i-2th first block can be obtained and marked as the sixth candidate fusion feature, and the input of the i-1th first block can be obtained based on the sixth candidate fusion feature.

[0231] Optionally, the sixth candidate fusion feature may be concatenated with the sample label, and a vector obtained after the concatenation is determined as an input vector of the i-1th first block, and is marked as a fourth input vector.

[0232] Among them, the fourth input vector can be input into the first self-attention layer of the i-1th first block to obtain the sixth splicing vector output by the first self-attention layer of the i-1th first block.

[0233] In an embodiment of the present application, the fourth input vector can be input into the first self-attention layer of the i-1th first block to obtain the output of the first attention layer of the i-1th first block based on the fourth input vector and marked as the sixth splicing vector.

[0234] Optionally, obtain the eighth partial vector of the sixth candidate fusion feature in the sixth splicing vector, determine the eighth partial vector as the fifth Q value, and splice the eighth partial vector and the multimodal splicing feature to obtain a fifth K value and a fifth V value, and input the fifth K value, the fifth V value and the fifth Q value into the cross-attention layer of the i-1th first block to obtain the fifth initial fusion feature output by the cross-attention layer of the i-1th first block, and perform a linear transformation on the fifth initial fusion feature based on the first linear layer of the i-1th first block to obtain the fifth candidate fusion feature output by the i-1th first block.

[0235] In this embodiment of the present application, the position of the sixth candidate fusion feature in the fourth input vector can be obtained, and the vector at the same position in the sixth splicing vector can be determined as the eighth partial vector of the sixth candidate fusion feature in the sixth splicing vector.

[0236] Further, the eighth partial vector is determined as the Q value of the cross-attention layer of the output i-1th first block, marked as the fifth Q value, and the eighth partial vector is spliced ​​with the multimodal splicing feature to obtain the K value and V value of the cross-attention layer of the input i-1th first block, and are marked as the fifth K value and the fifth V value, respectively.

[0237] Optionally, the fifth Q value, the fifth K value, and the fifth V value may be input into the cross-attention layer of the i-1th first block, and the fused features output by the i-1th cross-attention layer may be marked as the fifth initial fused features.

[0238] Furthermore, the fifth initial fused feature is linearly transformed by the first linear layer in the i-1th first block, and the fused feature output by the first linear layer through the linear transformation of the fifth initial fused feature is determined as the fifth candidate fused feature output by the i-1th first block.

[0239] Optionally, a concatenated vector of the fifth candidate fusion feature and the sample label is obtained as a third input vector of the i-th first block, and a third fusion feature output by the i-th first block is obtained based on the third input vector and the multimodal concatenated feature.

[0240] In the embodiment of the present application, the fifth candidate fusion feature may be concatenated with the sample label, and the concatenated vector may be determined as the third input vector of the i-th first block.

[0241] Furthermore, based on the third input vector and the multimodal concatenation feature, a third fusion feature output by the i-th first block is obtained.

[0242] As a possible implementation, in response to i being equal to M, the third input vector is input into the first self-attention layer of the i-th first block to obtain the seventh splicing vector output by the first self-attention layer of the i-th first block.

[0243] Among them, when i is equal to M, it can be determined that the i-th first block is the last block in the multimodal feature fusion encoder. In this scenario, the output result of the i-th first block is the output result of the multimodal feature fusion encoder.

[0244] Among them, the third input vector can be input into the first self-attention layer of the i-th first block, and the splicing vector output by the first self-attention layer is determined as the seventh splicing vector.

[0245] Optionally, obtain the ninth partial vector of the fifth candidate fusion feature in the seventh splicing vector, determine the ninth partial vector as the sixth Q value, and splice the ninth partial vector and the multimodal splicing feature to obtain a sixth K value and a sixth V value, and input the sixth K value, the sixth V value and the sixth Q value into the cross-attention layer of the i-th first block to obtain the sixth initial fusion feature output by the cross-attention layer of the i-th first block, and perform a linear transformation on the sixth initial fusion feature based on the first linear layer of the i-th first block to obtain a third fusion feature output by the i-th first block.

[0246] The position of the fifth candidate fusion feature in the third input vector may be obtained, and the vector at the same position in the seventh splicing vector may be determined as the ninth partial vector of the fifth candidate fusion feature in the seventh splicing vector.

[0247] Optionally, the ninth partial vector can be determined as the Q value of the cross-attention layer of the input i-th first block and marked as the sixth Q value, and the concatenated vector of the ninth partial vector and the multimodal fusion feature is obtained to obtain the K value and V value of the cross-attention layer of the input i-th first block, and marked as the sixth K value and sixth V value, respectively.

[0248] Further, the sixth Q value, the sixth K value and the sixth V value are input into the cross attention layer of the i-th first block to obtain the fused features output by the cross attention layer as the sixth initial fused features, and the sixth initial fused features are linearly transformed according to the first linear layer of the i-th first block, and the fused features output by the first linear layer are determined as the third fused features output by the i-th first block.

[0249] Optionally, the label information in the third fusion feature may be obtained based on a loss value of a sample label corresponding to a multimodal training sample, and the loss value may be determined as a multimodal label matching training loss MLM algorithm.

[0250] Among them, based on the preset multimodal label matching training loss MLM algorithm, the third fusion feature and the multimodal training sample can be algorithmically processed, and the loss value is obtained according to the result of the algorithm processing, which is determined as the multimodal label matching training loss MLM of the candidate multimodal data label model.

[0251] It should be noted that the multimodal label matching training loss MLM learns a finer-grained alignment of the two feature space representations by constructing data pairs of multimodal data and label data for matching binary classification tasks. Through a bidirectional self-attention mask strategy as the second mask strategy, the multimodal part and the label part can see each other's information, so that the third fusion feature generated and output can contain the information of both the multimodal part and the label part, and then pass it through a binary classification linear layer to obtain the matching score.

[0252] In this optimization goal, the hard example sampling strategy can be used to construct two hard example data for each anchor data. One is to sample a label outside the anchor point as a negative sample based on the similarity distribution within the batch from multimodal to label, and the other is to sample a multimodal data outside the anchor point as a negative sample based on the similarity distribution from label to multimodal.

[0253] Among them, the mask strategy of the first self-attention layer can be set to Figure 6 The second mask strategy shown corresponds to the form, so that the multimodal part and the label part can see each other's information.

[0254] Optionally, a binary label matching result of the third fusion feature and the multimodal training sample is obtained, and based on the binary label matching result, a third loss value of the third fusion feature based on the multimodal training sample is obtained as the multimodal label matching training loss MLM.

[0255] Among them, the binary classification label matching result of the label in the third fusion feature and the sample label of the multimodal training sample can be obtained, and according to the binary classification label matching result, the third loss value of the third fusion feature based on the multimodal training sample can be obtained as the multimodal label matching training loss MLM.

[0256] Optionally, the binary matching result may include a match and a mismatch, where a match may be represented by 1 and a mismatch may be represented by 0.

[0257] As an example, the algorithm formula of the multimodal label matching training loss MLM can be shown as follows:

[0258]

[0259] In the above formula, is the multimodal label matching training loss, is the binary classification matching result between the sample label corresponding to the kth training sample and the label in the third fusion feature output by the multimodal feature fusion encoder, It is the predicted probability value corresponding to the positive sample of the vector at the corresponding position of the candidate multimodal fusion hint vector in the third fusion feature after being processed by the binary classification head, average pooling and normalization.

[0260] Among them, the training loss set of the candidate multimodal data label model may include multimodal grounded label generation (MLG) training loss. For the acquisition of multimodal grounded label generation training loss MLG, please refer to the following content for understanding:

[0261] Optionally, in response to the mask strategy being the third mask strategy, a multimodal label generation training loss MLG is obtained according to the output label in the second fusion feature and the sample label of the multimodal training sample.

[0262] When the first self-attention layer and the second self-attention layer adopt the third mask strategy, the first self-attention layer and the second self-attention layer can learn the relevant information of the candidate multimodal fusion prompt vector and the candidate label prompt vector. In this scenario, the first concatenated vector can be input into the label encoder, and the multimodal label generation training loss of the candidate multimodal data label model can be obtained through the output result of the label encoder and the sample labels in the training samples.

[0263] It should be noted that the multimodal label generation training loss MLG is obtained based on the predicted label output by the label encoder and the sample label of the training sample. It can be seen that in the process of obtaining the multimodal label generation training loss, only the label encoder can be run without running the multimodal feature fusion encoder.

[0264] Optionally, N second blocks in the label encoder can be obtained, wherein each of the N second blocks is connected in series, and, for the i-th second block, in response to i being equal to N, the seventh candidate fusion feature output by the i-1-th second block is obtained through the first splicing vector and / or the multimodal splicing feature.

[0265] In an embodiment of the present application, N second blocks connected in series are provided in the tag encoder, wherein, for the i-th second block, when i is equal to N, the i-th second block is the last block in the tag encoder.

[0266] In this scenario, the fused features output by the i-th second block can be obtained and determined as the output result of the label encoder.

[0267] Among them, the fusion feature output by the i-1th second block can be obtained and marked as the seventh candidate fusion feature, and then the input vector of the i-th second block is obtained according to the seventh candidate fusion feature.

[0268] As a possible implementation, in response to i-1 being equal to 1, the first splicing vector is input into the second self-attention layer of the i-1th second block to obtain the eighth splicing vector output by the second self-attention layer of the i-1th second block.

[0269] In the embodiment of the present application, when i-1 is equal to 1, it can be determined that the i-1th second block is the first block in the label encoder. In this scenario, the first concatenated vector can be determined as the input vector of the i-1th second block.

[0270] like Figure 2 As shown, the first splicing vector can be determined as the K value, V value and Q value of the input i-1 second block, and the splicing vector output by the i-1 second block is obtained and marked as the eighth splicing vector.

[0271] Optionally, a tenth partial vector of the candidate label prompt vector and the sample label in the eighth concatenated vector is obtained, and a linear transformation is performed on the tenth partial vector based on the second linear layer of the i-1th second block to obtain the seventh candidate fusion feature output by the i-1th second block.

[0272] The positions of the candidate label prompt vector and the sample label in the first splicing vector may be obtained, and the vector at the same position in the eighth splicing vector may be determined as the tenth partial vector in the eighth rating vector.

[0273] like Figure 2 As shown, the tenth part of the vector is linearly transformed by the second linear layer in the i-1th second block, and the fused feature output by the second linear layer is determined as the seventh candidate fused feature output by the i-1th second block.

[0274] As another possible implementation, in response to i-1 being greater than 1, the eighth candidate fusion feature output by the i-2th second block is obtained.

[0275] In an embodiment of the present application, when i-1 is greater than 1, it can be determined that the i-1th second block is a block in the middle position between the first block and the last block in the label encoder. In this scenario, the input of the i-1th second block can be obtained based on the output of the i-2th second block.

[0276] Among them, the fused feature output by the i-2th second block can be marked as the eighth candidate fused feature.

[0277] Optionally, obtain the concatenated vector of the eighth candidate fusion feature and the sample label as the sixth input vector of the i-1th second block, and input the sixth input vector into the second self-attention layer of the i-1th second block to obtain the ninth concatenated vector output by the second self-attention layer of the i-1th second block.

[0278] The eighth candidate fusion feature may be concatenated with the sample label, and the concatenated vector may be determined as the input vector of the i-1th second block and marked as the sixth input vector.

[0279] Furthermore, the sixth input vector is input to the second self-attention layer of the i-1th second block, and the ninth splicing vector is obtained through the second self-attention layer of the i-1th second block.

[0280] Optionally, a linear transformation is performed on the ninth concatenated vector based on the second linear layer of the i-1th second block to obtain a seventh candidate fusion feature output by the i-1th second block.

[0281] The ninth splicing vector can be input into the second linear layer of the i-1th second block, the ninth splicing vector can be linearly transformed by the second linear layer, and the fused feature input by the second linear layer can be determined as the seventh candidate fused feature output by the i-1th second block.

[0282] Optionally, a concatenated vector of the seventh candidate fused feature and the sample label is obtained as the fifth input vector of the i-th second block, and based on the fifth input vector, a fourth fused feature output by the i-th second block is obtained.

[0283] In the embodiment of the present application, the seventh candidate fusion feature can be spliced ​​with the sample label, and the vector obtained after the splicing is determined as the input vector of the i-th second block, marked as the fifth input vector.

[0284] Furthermore, the fifth input vector is input into the i-th second block, and the i-th second block performs the second self-attention layer processing and the second linear layer linear transformation on the fifth input vector, so as to obtain the fused feature output by the i-th second block, which is marked as the fourth fused feature.

[0285] Optionally, a fourth loss value of a sample label of a multimodal training sample based on the output label in the fourth fusion feature may be obtained as the multimodal label generation training loss MLG.

[0286] It can be understood that the loss value of the sample label of the predicted label output by the label encoder based on the training sample can be obtained, marked as the fourth loss value, and the fourth loss value is determined as the multimodal label generation training loss MLG of the candidate multimodal data label model.

[0287] Among them, the algorithm for obtaining the training loss MLG based on the preset modal label generation can be used to perform algorithmic processing on the fourth fusion feature and the multimodal training samples, and according to the result of the algorithmic processing, the fourth loss value of the sample label of the predicted label output by the label encoder based on the training sample is obtained, and then the multimodal label generation training loss MLG of the candidate multimodal data label model is obtained.

[0288] As an example, the algorithm formula of the multimodal label generation training loss MLG can be shown as follows:

[0289]

[0290] In the above formula, is the output vector matrix of the candidate multimodal fusion prompt vector Q1 after the k-th multimodal training sample is encoded by the multimodal feature fusion encoder, Q2 is the candidate label prompt vector of the label encoder, is the sample label corresponding to the k-th multimodal training sample, It means that when the first t-1 time steps of sequence S and the information are known, the next time step is The conditional probability of Represents a subsequence consisting of the first t-1 elements of the ith sequence in a batch.

[0291] Optionally, a training loss set is obtained according to the multimodal label contrast learning training loss, the multimodal label matching training loss and the multimodal label generation training loss.

[0292] In an embodiment of the present application, a set consisting of a multimodal label contrast learning training loss, a multimodal label matching training loss, and the multimodal label generation training loss can be marked as a training loss set of a candidate multimodal data label model.

[0293] S504: Obtain a target training loss based on the training loss set.

[0294] Optionally, the weighting coefficients of the multimodal label contrast learning training loss, the multimodal label matching training loss and the multimodal label generation training loss in the training loss set can be obtained, and the multimodal label contrast learning training loss, the multimodal label matching training loss and the multimodal label generation training loss are weightedly summed based on the weighting coefficients to obtain the target training loss of the candidate multimodal data label acquisition model.

[0295] In a scenario where the training loss set of the candidate multimodal data labeling model includes multimodal label contrastive learning training loss, multimodal label matching training loss, and multimodal label generation training loss, the multimodal label contrastive learning training loss, the multimodal label matching training loss, and the multimodal label generation training loss have respective weighting coefficients.

[0296] In this scenario, the weighted coefficients of the multimodal label contrastive learning training loss, the multimodal label matching training loss, and the multimodal label generation training loss can be obtained respectively, and the multimodal label contrastive learning training loss, the multimodal label matching training loss, and the multimodal label generation training loss are weighted and summed according to the weighted coefficients to obtain the target training loss of the candidate multimodal data label model.

[0297] As an example, the target training loss of the candidate multimodal data label model is calculated by weighted summing the multimodal label contrastive learning training loss, the multimodal label matching training loss, and the multimodal label generation training loss, which can be shown as follows:

[0298]

[0299] In the above formula, L represents the target training loss of the candidate multimodal data label model. represents the multimodal label contrastive learning training loss, represents the weighted coefficient of the multimodal label contrastive learning training loss, represents the multimodal label matching training loss, represents the weighting coefficient of the multimodal label matching training loss, represents the multimodal label generation training loss, Represents the weighting coefficient of the multimodal label generation training loss.

[0300] S505, based on the target training loss, adjust the iteration parameters of the candidate multimodal data label model, wherein the iteration parameters include at least the iteration vector parameters of the candidate multimodal fusion prompt vector and the candidate label prompt vector respectively, and the iteration model parameters of the multimodal data label model, and the iteration model parameters do not include the encoder parameters of the image unimodal feature extraction encoder in the multimodal feature extraction encoder.

[0301] In an embodiment of the present application, among the model parameters of the candidate multimodal data labeling model, there are at least some parameters that need to be iteratively optimized during the model training process, and these model parameters can be determined as the iterative model parameters of the candidate multimodal data labeling model.

[0302] It should be noted that the iterative model parameters of the multimodal data labeling model do not include the encoder parameters of the image unimodal feature extraction encoder in the multimodal feature extraction encoder of the candidate multimodal data acquisition model.

[0303] It can be understood that, during the training iteration process of the candidate multimodal data label model, the encoder parameters of the image unimodal feature extraction encoder in the multimodal feature extraction encoder used to extract image model features do not participate in the iterative optimization of the candidate multimodal data label model.

[0304] Optionally, during the iterative training of the candidate multimodal data label model, the candidate multimodal fusion prompt vector corresponding to the multimodal feature fusion encoder and the candidate label prompt vector corresponding to the label encoder are also parameters in the iterative optimization process of the model.

[0305] It can be understood that while the candidate multimodal data label model is iteratively optimized based on the target training loss, the candidate multimodal fusion prompt vector and the candidate label prompt vector also need to be iteratively optimized, wherein the parameters of the candidate multimodal fusion prompt vector and the candidate label prompt vector's respective vector parameters that participate in iterative optimization during the model training process can be determined as the iterative vector parameters of the candidate multimodal fusion prompt vector and the candidate label prompt vector's respective iterative vector parameters.

[0306] Furthermore, the iteration model parameters and the iteration vector parameters are determined as iteration parameters that need to be adjusted during the iterative optimization process of the candidate multimodal data label model.

[0307] S506, returning to obtain the next multimodal training sample to continue training the candidate multimodal data label model after the iterative parameter adjustment, until the training is completed, to obtain a trained target multimodal data label model.

[0308] In an embodiment of the present application, after obtaining the target training loss of the candidate multimodal data label model, the iteration parameters including the iteration model parameters and the iteration vector parameters can be adjusted and optimized based on the target training loss, thereby obtaining the adjusted candidate multimodal data label model, the adjusted candidate multimodal fusion prompt vector and the adjusted candidate label prompt vector.

[0309] Furthermore, the next multimodal training sample is returned to continue training the candidate multimodal data label model after the iterative parameter adjustment until the training is completed, thereby obtaining a trained target multimodal data label model.

[0310] In an embodiment of the present application, after obtaining the adjusted candidate multimodal data label model, the adjusted candidate multimodal fusion prompt vector, and the adjusted candidate label prompt vector, it is possible to return to obtain the next multimodal training sample, and continue to train the adjusted candidate multimodal data label model based on the next multimodal training sample, the adjusted candidate multimodal fusion prompt vector, and the adjusted candidate label prompt vector until the training is completed, thereby obtaining a trained target multimodal data label model.

[0311] The conditions for ending model training may be set based on training rounds or based on the model output results during the training process, which are not specifically limited here.

[0312] Furthermore, when the model training meets the set training end conditions, the model training can be ended, and the model obtained from the training round that meets the training end conditions is determined as the trained target multimodal data label model.

[0313] It should be noted that in an embodiment of the present application, the candidate multimodal data label model is set with three training losses. In this scenario, it is necessary to run the candidate multimodal data label model three times based on the same multimodal training sample, and obtain one of the training losses in each running round. After completing three runs and obtaining three training losses, the target training loss of the candidate multimodal data label model in the current round of model training is calculated.

[0314] Furthermore, after adjusting the candidate multimodal data label model according to the target training loss, in the process of returning to obtain the next multimodal training sample to continue the next round of model training for the adjusted candidate multimodal data label model, it is also necessary to run the adjusted candidate multimodal data label model three times based on the next multimodal training sample, and obtain one of the training losses in each running round. After completing three runs and obtaining three training losses, the target training loss of the adjusted candidate multimodal data label model in the next round of model training is calculated.

[0315] Optionally, a target multimodal fusion prompt vector and a target label prompt vector used in a target training round of the candidate multimodal data label model are obtained, wherein the target training round is a training round for obtaining a trained target multimodal data label model, and based on the target multimodal fusion prompt vector and the target label prompt vector, a target query prompt vector combination of the target downstream task to which the multimodal training sample belongs is obtained.

[0316] In an embodiment of the present application, a target multimodal data label model has a corresponding downstream task, wherein the downstream task corresponding to the trained target multimodal data label model can be determined based on the multimodal training samples used in the training process before obtaining the target multimodal data label model, and marked as the target downstream task.

[0317] It should be noted that after obtaining the trained target multimodal data label model, the iteration vector parameters of the candidate multimodal fusion prompt vector and the candidate label prompt vector used in the model training process are iteratively optimized.

[0318] Among them, the training round of the trained target multimodal data label model can be marked as the target training round, and the candidate multimodal fusion prompt vector and candidate label prompt vector used in the target training round can be determined as the target multimodal fusion prompt vector and target label prompt vector that have completed iterative optimization.

[0319] In an embodiment of the present application, there is a correlation between the target multimodal fusion prompt vector and the target label prompt vector and the target downstream task of the target multimodal data label model. In this scenario, the vector combination composed of the target multimodal fusion prompt vector and the target label prompt vector can be determined as the target query prompt vector combination of the target downstream task.

[0320] It can be understood that, for any target downstream task, during the application of the model, when the target multimodal data label model needs to provide label information for the target downstream task, the multimodal feature fusion encoder in the target multimodal data label model needs to obtain the output result of the multimodal feature fusion encoder based on the target multimodal fusion prompt vector in the target query prompt vector combination, and the label encoder in the target multimodal data label model needs to obtain the output result of the label encoder based on the target label prompt vector in the target query prompt vector combination, thereby obtaining the output result of the target multimodal data label model.

[0321] The training method of the multimodal data label model proposed in the present application obtains the sample labels of the multimodal training samples, and splices the candidate multimodal fusion prompt vector, the candidate label prompt vector and the sample label to obtain the first splicing vector of the multimodal feature fusion encoder and the label encoder input into the candidate multimodal data label model, and obtains the training loss set composed of multiple training losses of the candidate multimodal data label model according to the setting of different mask strategies of the first self-attention layer and the second self-attention layer, and then obtains the target training loss according to the training loss set, and adjusts the iteration parameters including the iteration model parameters of the candidate multimodal data label model and the iteration vector parameters of the candidate multimodal fusion prompt vector and the candidate label prompt vector according to the target training loss, and returns the adjusted candidate multimodal data label model to continue training until the training is completed, so as to obtain the trained target multimodal data label model. In the present application, by setting different mask strategies, the accuracy of the calculation of each training loss in the training loss set is improved, thereby improving the model adjustment accuracy of the candidate multimodal data label model, optimizing the iteration effect of the model, and obtaining the target training loss of the candidate multimodal data label model through the multiple training losses included in the training loss set. The alignment degree of the multimodal information of the candidate multimodal data label model is improved, and the candidate multimodal data label model is trained based on multimodal training samples, thereby improving the accuracy of the trained target multimodal data label acquisition model in acquiring labels for multimodal data.

[0322] Corresponding to the training methods of the multimodal data label models proposed in the above-mentioned embodiments, an embodiment of the present application further proposes a training device for a multimodal data label model. Since the training device for the multimodal data label model proposed in the embodiment of the present application corresponds to the training methods for the multimodal data label models proposed in the above-mentioned embodiments, the implementation methods of the above-mentioned training methods for the multimodal data label models are also applicable to the training device for the multimodal data label model proposed in the embodiment of the present application, and will not be described in detail in the following embodiments.

[0323] Figure 7 Schematic diagram of the structure of a training device for a multimodal data label model according to an embodiment of the present application, such as Figure 7 As shown, the training device 700 of the multimodal data label model includes a construction module 71, a resampling module 72, an acquisition module 73 and a training module 74, wherein:

[0324] A construction module 71 is used to construct a candidate multimodal data label model to be trained and obtain multimodal training samples;

[0325] A resampling module 72, used for extracting and resampling features of the multimodal training samples to obtain multimodal splicing features of the multimodal training samples;

[0326] An acquisition module 73 is used to acquire a candidate multimodal fusion prompt vector and a candidate label prompt vector corresponding to the multimodal training sample;

[0327] The training module 74 is used to obtain a training loss set of the candidate multimodal data label model based on the candidate multimodal fusion prompt vector, the candidate label prompt vector and the multimodal splicing feature, so as to perform model training on the candidate multimodal data label model until the training is completed, thereby obtaining a trained target multimodal data label model.

[0328] In the embodiment of the present application, the resampling module 72 is also used to: extract the unimodal features of each unimodal sample in the multimodal training sample based on the multimodal feature extraction encoder of the candidate multimodal data label model; resample the unimodal features of each unimodal sample based on the multimodal feature resampling layer of the candidate multimodal data label model to obtain the unimodal resampled features of each unimodal sample; and splice the unimodal resampled features of each unimodal sample to obtain the multimodal splicing features extracted from the multimodal training sample.

[0329] In an embodiment of the present application, the acquisition module 73 is also used to: obtain a downstream task set of candidate multimodal data label models, and a query prompt vector combination of each downstream task in the downstream task set, wherein, for any downstream task, the query prompt vector combination of the downstream task includes an initial multimodal fusion prompt vector and an initial label prompt vector of the downstream task; obtain a target downstream task associated with the multimodal training sample in the downstream task set, and determine the initial multimodal fusion prompt vector in the query prompt vector combination of the target downstream task as a candidate multimodal fusion prompt vector, and determine the initial label prompt vector as a candidate label prompt vector.

[0330] In an embodiment of the present application, the acquisition module 73 is also used to: obtain the feature extraction space dimension of the candidate multimodal data label model; randomly construct an initial multimodal query prompt vector and an initial label query prompt vector for each downstream task in the downstream task set based on the feature extraction space dimension, and obtain a query prompt vector combination for each downstream task.

[0331] In an embodiment of the present application, the training module 74 is also used to: splice the candidate multimodal fusion prompt vector, the candidate label prompt vector and the sample label of the multimodal training sample to obtain a first splicing vector; obtain the mask strategy of the first self-attention layer of the multimodal feature fusion encoder and the second self-attention layer of the label encoder of the candidate multimodal data label model; based on the mask strategy, obtain the training loss set through the first splicing vector and / or multimodal splicing features.

[0332] In an embodiment of the present application, the training module 74 is also used to: obtain the first fusion feature output by the multimodal feature fusion encoder through the first splicing vector and / or the multimodal splicing feature, and obtain the second fusion feature output by the label encoder; in response to the masking strategy being the first masking strategy, obtain the multimodal label comparison learning training loss MLC according to the first fusion feature and the second fusion feature; in response to the masking strategy being the second masking strategy, obtain the multimodal label matching training loss MLM according to the first fusion feature and the binary classification label of the multimodal training sample; in response to the masking strategy being the third masking strategy, obtain the multimodal label generation training loss MLG according to the output label in the second fusion feature and the sample label of the multimodal training sample; obtain a training loss set according to the multimodal label comparison learning training loss, the multimodal label matching training loss and the multimodal label generation training loss.

[0333] In the embodiment of the present application, the training module 74 is also used to: obtain M first blocks in the multimodal feature fusion encoder and N second blocks in the label encoder, wherein each first block in the M first blocks is connected in series, and each second block in the N second blocks is connected in series, and M and N have the same value; for the i-th first block and the i-th second block, in response to i being equal to M, obtain the first candidate fusion feature output by the i-1th first block and the second candidate fusion feature output by the i-1th second block through the first splicing vector and / or the multimodal splicing feature; obtain the second splicing vector of the first candidate fusion feature, the second candidate fusion feature and the sample label to obtain the first fusion feature output by the i-th first block, and the second fusion feature output by the i-th second block.

[0334] In the embodiment of the present application, the training module 74 is also used to: in response to i-1 being equal to 1, obtain the third splicing vector outputted by the first self-attention layer of the i-1th first block and the second self-attention layer of the i-1th second block according to the first splicing vector, and obtain the first candidate fusion feature and the second candidate fusion feature according to the candidate multimodal fusion prompt vector, the candidate label prompt vector and the third splicing vector. In response to i-1 being greater than 1, obtain the third splicing vector outputted by the first self-attention layer of the i-1th first block and the second self-attention layer of the i-1th second block according to the third candidate fusion feature outputted by the i-2th first block and the fourth candidate fusion feature outputted by the i-2th second block, and obtain the first candidate fusion feature and the second candidate fusion feature according to the third candidate fusion feature, the third splicing vector, the multimodal splicing feature and the sample label.

[0335] In the embodiment of the present application, the training module 74 is also used to: in response to i being equal to M, obtain, according to the second splicing vector, a fourth splicing vector output by the first self-attention layer of the i-th first block and the second self-attention layer of the i-th second block; obtain the first fusion feature and the second fusion feature according to the first candidate fusion feature, the fourth splicing vector, the multimodal splicing feature, the second candidate fusion feature and the sample label.

[0336] In the embodiment of the present application, the training module 74 is also used to: obtain the target training loss based on the training loss set. Based on the target training loss, adjust the iteration parameters of the candidate multimodal data label model, wherein the iteration parameters at least include the iteration vector parameters of the candidate multimodal fusion prompt vector and the candidate label prompt vector, and the iteration model parameters of the multimodal data label model, and the iteration model parameters do not include the encoder parameters of the image unimodal feature extraction encoder in the multimodal feature extraction encoder; return to obtain the next multimodal training sample to continue training the candidate multimodal data label model after the iteration parameters are adjusted until the training is completed, and obtain the trained target multimodal data label model.

[0337] In an embodiment of the present application, the training module 74 is also used to: obtain the target multimodal fusion prompt vector and the target label prompt vector used in the target training round of the candidate multimodal data label model, wherein the target training round is the training round for obtaining the trained target multimodal data label model; based on the target multimodal fusion prompt vector and the target label prompt vector, obtain the target query prompt vector combination of the target downstream task to which the multimodal training sample belongs.

[0338] The training device of the multimodal data label model proposed in the present application constructs a candidate multimodal data label model to be trained, obtains multimodal training samples of the candidate multimodal data label model, performs feature extraction and resampling on the multimodal training samples, obtains multimodal splicing features of the multimodal training samples, and obtains the training loss set of the candidate multimodal data label model based on the multimodal splicing features and the candidate multimodal fusion prompt vectors and candidate label prompt vectors corresponding to the obtained multimodal training samples, and then trains the candidate multimodal data label model according to the training loss set until the training is completed, thereby obtaining a trained target multimodal data label model. In the present application, the features extracted from each unimodal sample of the multimodal training samples are resampled to avoid the abnormal situation that the sequence dimension of the features extracted from each unimodal sample exceeds the upper limit of the input dimension of the candidate multimodal data label model, thereby improving the stability of the candidate multimodal data label model, optimizing the noise filtering effect and learning effect of the candidate multimodal data label model, and improving the alignment degree of multimodal information of the candidate multimodal data label model based on multiple training losses included in the training loss set. The candidate multimodal data label model is trained based on the multimodal training samples, thereby improving the accuracy of the trained target multimodal data label acquisition model in acquiring labels for multimodal data.

[0339] To achieve the above embodiments, the present application also proposes an electronic device, a computer-readable storage medium and a computer program product.

[0340] Figure 8 FIG. 1 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 8 As shown, the device 800 includes a memory 81, a processor 82, and a computer program stored in the memory 81 and executable on the processor 82. When the processor 82 executes the program instructions, the training method of the multimodal data label model proposed in the above embodiment is implemented.

[0341] The electronic device proposed in the present application constructs a candidate multimodal data label model to be trained, obtains multimodal training samples of the candidate multimodal data label model, performs feature extraction and resampling on the multimodal training samples, obtains multimodal splicing features of the multimodal training samples, and obtains a training loss set of the candidate multimodal data label model based on the multimodal splicing features and the candidate multimodal fusion prompt vectors and candidate label prompt vectors corresponding to the obtained multimodal training samples, and then trains the candidate multimodal data label model according to the training loss set until the training is completed, thereby obtaining a trained target multimodal data label model. In the present application, the features extracted from each unimodal sample of the multimodal training samples are resampled to avoid the abnormal situation that the sequence dimension of the features extracted from each unimodal sample exceeds the upper limit of the input dimension of the candidate multimodal data label model, thereby improving the stability of the candidate multimodal data label model, optimizing the noise filtering effect and learning effect of the candidate multimodal data label model, and improving the alignment degree of multimodal information of the candidate multimodal data label model based on multiple training losses included in the training loss set. The candidate multimodal data label model is trained based on the multimodal training samples, thereby improving the accuracy of the trained target multimodal data label acquisition model in acquiring labels for multimodal data.

[0342] An embodiment of the present application proposes a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the training method of the multimodal data label model proposed in the above embodiment is implemented.

[0343] The computer-readable storage medium of the embodiment of the present application constructs a candidate multimodal data label model to be trained, obtains multimodal training samples of the candidate multimodal data label model, performs feature extraction and resampling on the multimodal training samples, obtains multimodal splicing features of the multimodal training samples, and obtains a training loss set of the candidate multimodal data label model based on the multimodal splicing features and the candidate multimodal fusion prompt vectors and candidate label prompt vectors corresponding to the obtained multimodal training samples, and then trains the candidate multimodal data label model according to the training loss set until the training is completed, thereby obtaining a trained target multimodal data label model. In the present application, the features extracted from each unimodal sample of the multimodal training samples are resampled to avoid the abnormal situation that the sequence dimension of the features extracted from each unimodal sample exceeds the upper limit of the input dimension of the candidate multimodal data label model, thereby improving the stability of the candidate multimodal data label model, optimizing the noise filtering effect and learning effect of the candidate multimodal data label model, and improving the alignment degree of multimodal information of the candidate multimodal data label model based on multiple training losses included in the training loss set. The candidate multimodal data label model is trained based on the multimodal training samples, thereby improving the accuracy of the trained target multimodal data label acquisition model in acquiring labels for multimodal data.

[0344] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0345] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0346] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0347] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0348] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0349] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0350] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0351] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

[0352] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0353] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for training a multimodal data label model, characterized in that: The method comprises: Constructing a candidate multimodal data label model to be trained, and obtaining multimodal training samples, wherein the multimodal training samples include samples of image modality, samples of text modality, and samples of audio modality; Performing feature extraction and resampling on the multimodal training samples to obtain multimodal splicing features of the multimodal training samples; Obtaining a candidate multimodal fusion prompt vector and a candidate label prompt vector corresponding to the multimodal training sample; splicing the candidate multimodal fusion prompt vector, the candidate label prompt vector and the sample label of the multimodal training sample to obtain a first splicing vector; Obtaining a mask strategy of a first self-attention layer of a multimodal feature fusion encoder and a second self-attention layer of a label encoder of the candidate multimodal data label model; Obtaining, through the first splicing vector and the multimodal splicing feature, a first fusion feature output by the multimodal feature fusion encoder and a second fusion feature output by the label encoder; In response to the mask strategy being the first mask strategy, obtaining a multimodal label contrast learning training loss MLC according to the first fusion feature and the second fusion feature; In response to the mask strategy being the second mask strategy, obtaining a multimodal label matching training loss MLM according to the first fusion feature and the binary classification label of the multimodal training sample; In response to the mask strategy being a third mask strategy, obtaining a multimodal label generation training loss MLG according to the output label in the second fusion feature and the sample label of the multimodal training sample; The training loss set is obtained according to the multimodal label contrast learning training loss, the multimodal label matching training loss and the multimodal label generation training loss to perform model training on the candidate multimodal data label model until the training is completed, thereby obtaining a trained target multimodal data label model.

2. The method according to claim 1, characterized in that: The extracting and resampling the multimodal training samples to obtain multimodal splicing features of the multimodal training samples includes: A multimodal feature extraction encoder based on the candidate multimodal data label model extracts unimodal features of each unimodal sample in the multimodal training sample; Based on the multimodal feature resampling layer of the candidate multimodal data label model, resampling the unimodal features of each unimodal sample to obtain the unimodal resampled features of each unimodal sample; The unimodal resampling features of each unimodal sample are spliced ​​to obtain the multimodal splicing features extracted from the multimodal training sample.

3. The method according to claim 1, characterized in that The obtaining of the candidate multimodal fusion prompt vector and the candidate label prompt vector corresponding to the multimodal training sample includes: Acquire a downstream task set of the candidate multimodal data label model, and a query prompt vector combination of each downstream task in the downstream task set, wherein, for any downstream task, the query prompt vector combination of the downstream task includes an initial multimodal fusion prompt vector and an initial label prompt vector of the downstream task; A target downstream task associated with the multimodal training sample in the downstream task set is obtained, and an initial multimodal fusion prompt vector in the query prompt vector combination of the target downstream task is determined as the candidate multimodal fusion prompt vector, and an initial label prompt vector is determined as the candidate label prompt vector.

4. The method according to claim 3, characterized in that The downstream task set of the candidate multimodal data label model is obtained, and the query prompt vector combination of each downstream task in the downstream task set, wherein, for any downstream task, the query prompt vector combination of the downstream task includes an initial multimodal fusion prompt vector and an initial label prompt vector of the downstream task, including: Obtaining a feature extraction space dimension of the candidate multimodal data label model; An initial multimodal query prompt vector and an initial label query prompt vector are randomly constructed for each downstream task in the downstream task set based on the feature extraction space dimension to obtain a query prompt vector combination for each downstream task.

5. The method according to claim 1, characterized in that The obtaining, through the first splicing vector and the multimodal splicing feature, a first fusion feature output by the multimodal feature fusion encoder and a second fusion feature output by the label encoder, comprises: Obtaining M first blocks in the multimodal feature fusion encoder and N second blocks in the label encoder, wherein each first block in the M first blocks is connected in series, each second block in the N second blocks is connected in series, and M and N have the same value; For the i-th first block and the i-th second block, in response to i being equal to M, obtaining, by using the first splicing vector and the multimodal splicing feature, a first candidate fusion feature output by the i-1-th first block and a second candidate fusion feature output by the i-1-th second block; The first candidate fusion feature, the second candidate fusion feature, and a second concatenated vector of the sample label are obtained to obtain the first fusion feature output by the i-th first block and the second fusion feature output by the i-th second block.

6. The method according to claim 5, characterized in that The step of acquiring, for the i-th first block and the i-th second block, in response to i being equal to M, the first candidate fusion feature output by the i-1-th first block and the second candidate fusion feature output by the i-1-th second block through the first splicing vector and the multimodal splicing feature, includes: In response to i-1 being equal to 1, obtaining, according to the first splicing vector, a third splicing vector outputted by the first self-attention layer of the i-1th first block and the second self-attention layer of the i-1th second block, respectively, and obtaining the first candidate fusion feature and the second candidate fusion feature according to the candidate multimodal fusion prompt vector, the candidate label prompt vector and the third splicing vector; In response to i-1 being greater than 1, according to the third candidate fusion feature output by the i-2th first block and the fourth candidate fusion feature output by the i-2th second block, the third splicing vector output by the first self-attention layer of the i-1th first block and the second self-attention layer of the i-1th second block are obtained, and according to the third candidate fusion feature, the third splicing vector, the multimodal splicing feature and the sample label, the first candidate fusion feature and the second candidate fusion feature are obtained.

7. The method according to claim 5, characterized in that The acquiring the first candidate fusion feature, the second candidate fusion feature, and the second concatenated vector of the sample label to obtain the first fusion feature output by the i-th first block and the second fusion feature output by the i-th second block includes: In response to i being equal to M, obtaining, according to the second splicing vector, a fourth splicing vector outputted by the first self-attention layer of the i-th first block and the second self-attention layer of the i-th second block respectively; The first fusion feature and the second fusion feature are obtained according to the first candidate fusion feature, the fourth splicing vector, the multimodal splicing feature, the second candidate fusion feature and the sample label.

8. The method according to claim 1, characterized in that The step of obtaining the training loss set according to the multimodal label contrast learning training loss, the multimodal label matching training loss and the multimodal label generation training loss, so as to perform model training on the candidate multimodal data label model until the training is completed, and obtaining a trained target multimodal data label model, includes: Based on the training loss set, obtaining a target training loss; Based on the target training loss, adjusting the iteration parameters of the candidate multimodal data label model, wherein the iteration parameters include at least iteration vector parameters of each of the candidate multimodal fusion prompt vector and the candidate label prompt vector, and iteration model parameters of the multimodal data label model, and the iteration model parameters do not include encoder parameters of the image unimodal feature extraction encoder in the multimodal feature extraction encoder; Return to obtain the next multimodal training sample to continue training the candidate multimodal data label model after the iterative parameter adjustment until the training is completed, and obtain the trained target multimodal data label model.

9. The method according to claim 8, characterized in that The method further comprises: Obtaining a target multimodal fusion prompt vector and a target label prompt vector used in a target training round of the candidate multimodal data label model, wherein the target training round is a training round for obtaining the trained target multimodal data label model; Based on the target multimodal fusion prompt vector and the target label prompt vector, a target query prompt vector combination of the target downstream task to which the multimodal training sample belongs is obtained.

10. A training device for a multimodal data label model, characterized in that: The device comprises: A construction module is used to construct a candidate multimodal data label model to be trained and obtain multimodal training samples, wherein the multimodal training samples include samples of image modality, samples of text modality, and samples of audio modality; A resampling module, used for performing feature extraction and resampling on the multimodal training samples to obtain multimodal splicing features of the multimodal training samples; An acquisition module, used to acquire a candidate multimodal fusion prompt vector and a candidate label prompt vector corresponding to the multimodal training sample; A training module, configured to concatenate the candidate multimodal fusion prompt vector, the candidate label prompt vector and the sample label of the multimodal training sample to obtain a first concatenated vector; Obtaining a mask strategy of a first self-attention layer of a multimodal feature fusion encoder and a second self-attention layer of a label encoder of the candidate multimodal data label model; Obtaining, through the first splicing vector and the multimodal splicing feature, a first fusion feature output by the multimodal feature fusion encoder and a second fusion feature output by the label encoder; In response to the mask strategy being the first mask strategy, obtaining a multimodal label contrast learning training loss MLC according to the first fusion feature and the second fusion feature; In response to the mask strategy being the second mask strategy, obtaining a multimodal label matching training loss MLM according to the first fusion feature and the binary classification label of the multimodal training sample; In response to the mask strategy being a third mask strategy, obtaining a multimodal label generation training loss MLG according to the output label in the second fusion feature and the sample label of the multimodal training sample; The training loss set is obtained according to the multimodal label contrast learning training loss, the multimodal label matching training loss and the multimodal label generation training loss to perform model training on the candidate multimodal data label model until the training is completed, thereby obtaining a trained target multimodal data label model.

11. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions to be executed by the at least one processor, and the instructions are executed by the at least one processor so as to enable the at least one processor to perform the method according to any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.

13. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images