Modal prompt generation method, and multi-modal data prediction method and device

Abstract

The invention discloses a modal prompt generation method and a multi-modal data prediction method and device, and is applied to the technical field of multi-modal learning, and the generation method comprises the steps: obtaining a comprehensive prompt matrix, a plurality of data samples, and a prompt weight matrix of a plurality of preset modals; training a preset multi-modal large model by using the data, and optimizing the comprehensive prompt matrix and the prompt weight matrix of each preset modal during training to obtain a target comprehensive prompt matrix and a target prompt weight matrix of each preset modal as modal prompts; the prediction method comprises the steps of obtaining to-be-detected multi-modal data, wherein the multi-modal data lacks at least one preset modal; and obtaining a prediction result associated with the multi-modal data according to the target comprehensive prompt matrix, the multi-modal data and the target prompt weight matrix of the missing modal of the multi-modal data in combination with a preset multi-modal large model. According to the method, the design complexity of modal prompt is effectively reduced, and the prediction precision of the multi-modal large model is improved.

Description

Technical Field

[0001] The present application relates to the field of multimodal learning technology, and in particular to a modal prompt generation method, a multimodal data prediction method and a device. Background Art

[0002] Modality loss refers to the lack or non-existence of data in some modalities in multimodal data. In multimodal learning, modality loss is a common phenomenon that may occur during the training or prediction process of a large multimodal model.

[0003] In the related art, the phenomenon of modal missing is dealt with by designing corresponding modal prompts for each modality or each missing situation. However, the method of designing corresponding modal prompts for each missing situation in the related art easily leads to an increase in the design complexity of the modal prompts, and there is a serious problem of parameter redundancy. In addition, the method of designing corresponding modal prompts for each modality in the related art is insufficient for prompting certain missing modalities, and there is uncertainty related to the missing modality. This uncertainty will increase the risk of misleading the prediction of the multimodal large model, resulting in a decrease in the prediction accuracy of the multimodal large model. Summary of the invention

[0004] The embodiments of the present application provide a modal prompt generation method, a multimodal data prediction method and a device for reducing the design complexity of modal prompts and improving the prediction accuracy of multimodal large models.

[0005] On the one hand, an embodiment of the present application provides a method for generating a modal prompt, comprising the following steps: Obtaining a comprehensive prompt matrix, a plurality of data samples, and a prompt weight matrix of a plurality of preset modalities; wherein the data sample includes label information and a plurality of the preset modalities, and a preset virtual input is used as the preset modality missing from the data sample; the prompt weight matrix of the preset modality is used to characterize the impact of the missing of the preset modality on the data sample; the comprehensive prompt matrix is ​​used to characterize the modal prompts of all the preset modalities; The preset multimodal large model is trained using the comprehensive prompt matrix, each of the data samples and the prompt weight matrix of the preset modality that is missing from each of the data samples. During the training of the multimodal large model, the comprehensive prompt matrix and the prompt weight matrix of each of the preset modalities are optimized to obtain a target comprehensive prompt matrix and a target prompt weight matrix of each of the preset modalities as modal prompts.

[0006] In addition, the modal prompt generating method according to the above embodiment of the present invention may also have the following additional technical features: Further, in one embodiment of the present invention, the use of the comprehensive prompt matrix, each of the data samples and the prompt weight matrix of the preset modality missing from each of the data samples to train the preset multimodal large model includes: For each of the data samples, adding the prompt weight matrices of the preset modalities missing from the data sample to obtain a first prompt sub-matrix, and performing block Kronecker similarity multiplication processing on the first prompt sub-matrix and the comprehensive prompt matrix to obtain a first prompt matrix of the data sample; Performing input embedding processing on each of the data samples to obtain an embedding vector for each of the data samples; Concatenating the embedding vector of each of the data samples and the first prompt matrix to obtain input information of each of the data samples; The input information of each of the data samples is used as the input of the multimodal large model to train the multimodal large model.

[0007] Furthermore, in one embodiment of the present invention, during the training of the multimodal large model, the comprehensive prompt matrix and the prompt weight matrix of each of the preset modalities are optimized to obtain a target comprehensive prompt matrix and a target prompt weight matrix of each of the preset modalities, including: Based on the network parameters and the data samples of the current training batch, the multimodal large model is trained for the current training batch; wherein the network parameters include the comprehensive prompt matrix, the model parameters of the multimodal large model and the prompt weight matrices of the plurality of preset modes; When the training of the current training batch is completed, obtaining the loss function of the multimodal large model according to the predicted label information and label information of each of the data samples of the current training batch; According to the loss function of the multimodal large model, the optimized network parameters and the optimization objective function of the multimodal large model are obtained; When the optimization objective function is greater than or equal to a preset optimization threshold, the current training batch is increased by one, the optimized network parameters are used as new network parameters, and the process returns to the step of training the current training batch of the multimodal large model based on the network parameters and the data samples of the current training batch until the optimization objective function is less than the optimization threshold, and the optimized comprehensive prompt matrix in the optimized network parameters is used as the target comprehensive prompt matrix, and the optimized prompt weight matrix of each of the preset modes in the optimized network parameters is used as the target prompt weight matrix of each of the preset modes.

[0008] Furthermore, in one embodiment of the present invention, obtaining the loss function of the multimodal large model according to the predicted label information and label information of each of the data samples in the current training batch includes: According to the predicted label information and label information of each of the data samples in the current training batch, an evidence theory loss function is obtained as the loss function of the multimodal large model.

[0009] Further, in one embodiment of the present invention, obtaining the optimized network parameters and the optimization objective function of the multimodal large model according to the loss function of the multimodal large model includes: The network parameters are optimized by utilizing the loss function of the multimodal large model in combination with a back propagation algorithm to obtain the optimized network parameters.

[0010] Further, in one embodiment of the present invention, obtaining the optimized network parameters and the optimization objective function of the multimodal large model according to the loss function of the multimodal large model includes: Obtain the product of a preset penalty item parameter and a preset regularization item parameter as an auxiliary item parameter; Obtaining a difference between a preset value and the penalty item parameter as a penalty item difference; Obtaining the product of the penalty term difference and the loss function as a loss function term parameter; The sum of the auxiliary item parameters and the loss function item parameters is obtained as the optimization objective function.

[0011] On the other hand, an embodiment of the present application provides a multimodal data prediction method, comprising the following steps: Acquire multimodal data to be tested, wherein the multimodal data lacks at least one of the preset modalities; Obtaining a prediction result associated with the multimodal data according to a target comprehensive prompt matrix, the multimodal data, and a target prompt weight matrix of a preset modality missing from the multimodal data, in combination with a preset multimodal macro model; Among them, the target comprehensive prompt matrix and the target prompt weight matrix of the preset modality missing from the multimodal data are both obtained through one of the above-mentioned modality prompt generation methods.

[0012] In addition, the multimodal data prediction method according to the above embodiment of the present invention may also have the following additional technical features: Further, in one embodiment of the present invention, the target comprehensive prompt matrix, the multimodal data and the target prompt weight matrix of the preset modality missing from the multimodal data, combined with a preset multimodal large model, obtains a prediction result associated with the multimodal data, including: Adding target prompt weight matrices of preset modalities that are missing from the multimodal data to obtain a second prompt sub-matrix; Performing block Kronecker similarity multiplication processing on the target comprehensive cue matrix and the second cue sub-matrix to obtain a second cue matrix of the multimodal data; Performing embedding processing on the multimodal data to obtain an embedding vector of the multimodal data; Concatenating the embedding vector of the multimodal data and the second prompt matrix to obtain input information of the multimodal data as input of the multimodal large model; Prediction processing is performed using the multimodal large model to obtain a prediction result associated with the multimodal data.

[0013] On the other hand, an embodiment of the present application provides a modal prompt generating device, comprising: The first acquisition module is used to acquire a comprehensive prompt matrix, a plurality of data samples and a prompt weight matrix of a plurality of preset modalities; wherein the data sample includes label information and a plurality of the preset modalities, and a preset virtual input is used as the preset modality missing from the data sample; the prompt weight matrix of the preset modality is used to characterize the impact of the missing of the preset modality on the data sample; the comprehensive prompt matrix is ​​used to characterize the modal prompts of all the preset modalities; The first processing module is used to train a preset multimodal large model using the comprehensive prompt matrix, each of the data samples and the prompt weight matrix of the preset modality that is missing from each of the data samples, and during the training of the multimodal large model, the comprehensive prompt matrix and the prompt weight matrix of each of the preset modalities are optimized to obtain a target comprehensive prompt matrix and a target prompt weight matrix of each of the preset modalities as modal prompts.

[0014] In another aspect, an embodiment of the present application provides a multimodal data prediction device, comprising: A second acquisition module is used to acquire multimodal data to be tested, wherein the multimodal data lacks at least one of the preset modes; A second processing module, configured to obtain a prediction result associated with the multimodal data according to a target comprehensive prompt matrix, the multimodal data and a target prompt weight matrix of a preset modality missing from the multimodal data, in combination with a preset multimodal large model; Among them, the target comprehensive prompt matrix and the target prompt weight matrix of the preset modality missing from the multimodal data are both obtained by the above-mentioned modality prompt generating device.

[0015] The beneficial effects of the present application are as follows: a modal prompt generation method, a multimodal data prediction method and a device are provided. In terms of modal prompt generation, a comprehensive prompt matrix, multiple data samples and prompt weight matrices of multiple preset modalities are first obtained, and then the preset multimodal large model is trained using the above data. During the training period, the comprehensive prompt matrix and the prompt weight matrix of each preset modality are optimized to obtain a target comprehensive prompt matrix and a target prompt weight matrix of each preset modality as the modal prompt. In this way, the key information of each preset modality can be extracted without increasing the design complexity of the modal prompt, which can not only effectively improve the accuracy of the modal prompt, but also reduce the error. The parameter redundancy of low modal prompts is improved, and the generation efficiency of modal prompts is improved. In terms of multimodal data prediction, the multimodal data to be tested is first obtained, and the multimodal data is missing at least one preset modality. Then, according to the target comprehensive prompt matrix, the multimodal data and the target prompt weight matrix of the missing modality of the multimodal data, combined with the preset multimodal large model, the prediction results associated with the multimodal data are obtained. In this way, only a target comprehensive prompt matrix and the target prompt weight matrix of the missing preset modality can be used to make up for the missing preset modality information of the multimodal data and realize multimodal data prediction for modality missing, which can effectively improve the prediction accuracy of the multimodal large model.

[0016] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of a modal prompt generating method provided in an embodiment of the present application; Figure 2 This is a schematic diagram of the principle of generating a modal prompt provided in an embodiment of the present application; Figure 3 is a flowchart of a multimodal data prediction method provided in an embodiment of the present application; Figure 4 is a comparison diagram of the modal prompt generation method provided in the embodiment of the present application; Figure 5 is a structural diagram of a modal prompt generating device provided in an embodiment of the present application; Figure 6 It is a structural diagram of a multimodal data prediction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0019] The present application is further described below in conjunction with the accompanying drawings and specific embodiments. The described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application.

[0020] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0022] Modality loss refers to the lack or non-existence of data in certain modalities in multimodal data. For example, in social media data, data in modalities such as video descriptions, subtitles, and audio may be missing for various reasons. In multimodal learning, modality loss is a common phenomenon that may occur during the training or prediction process of a large multimodal model. In related technologies, the phenomenon of modality loss is addressed by designing corresponding modality prompts for each modality or each missing situation.

[0023] For example, MAP (Missing-Aware Prompting) requires designing multiple prompts for all possible missing situations, with a total of Tips, including It represents the number of modes, which leads to the increase of design complexity of modal prompts and the emergence of problems such as serious parameter redundancy. Especially as the number of modes increases, the problem of serious parameter redundancy becomes more prominent.

[0024] For another example, MSP (Modality Specific Prompting) simplifies the design of modality prompts by assigning a prompt to each modality. However, this method does not provide sufficient prompts for some missing modalities and there are uncertainties associated with the missing modalities. This uncertainty will increase the risk of misleading predictions for large multimodal models, resulting in reduced prediction accuracy of large multimodal models.

[0025] In view of this, the embodiments of the present application provide a modal prompt generation method, a multimodal data prediction method and a device, aiming to reduce the design complexity of modal prompts, alleviate the serious problem of parameter redundancy, and at the same time enhance the ability of large multimodal models to deal with the uncertainty of missing modes, thereby improving the prediction accuracy of large multimodal models.

[0026] First, the implementation steps of the modal prompt generating method provided in the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0027] The modal prompt generation method provided in the embodiment of the present application can be applied to a terminal, a server, or software running in a terminal or a server. The terminal can be a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to this. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. In addition, the server can also be a node server in a blockchain network, but is not limited to this. Among them, blockchain is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm.

[0028] Reference Figure 1 , Figure 1 It is a flowchart of a modal prompt generating method provided in an embodiment of the present application. The modal prompt generating method may include the following steps S101-S102.

[0029] S101, obtaining a comprehensive prompt matrix, a plurality of data samples, and a plurality of prompt weight matrices of preset modes.

[0030] It should be noted that the data sample may include but is not limited to label information and multiple preset modalities, and the preset virtual input is used as the preset modality that is missing from the data sample. In addition, the prompt weight matrix of the preset modality is used to characterize the impact of the missing preset modality on the data sample, and the comprehensive prompt matrix is ​​used to characterize the modal prompts of all preset modalities.

[0031] In this step, multiple data samples are obtained, each of which includes label information and multiple preset modes. Among them, label information refers to the actual predicted label corresponding to the data sample, and the label information can be pre-calibrated according to the actual situation, and the preset mode refers to the mode of the data sample. Each data sample can have multiple preset modes and lack at least one preset mode. For the preset mode missing in the data sample, the missing preset mode usually has no value that can be input. At this time, a virtual input is used as the preset mode missing from the data sample, that is, a virtual input is used to replace the blank without input. For ease of understanding, the first A data sample can be represented as , Indicates the total number of preset modes, Indicates The data sample Preset modes, Indicates The label information of the data samples.

[0032] At the same time, obtain the prompt weight matrices of multiple preset modes, that is, for the Mode , defining a unique cue weight matrix , the prompt weight matrix of the preset modality is mainly used to characterize the impact of the absence of the preset modality on the data sample. Specifically, the size of the prompt weight matrix of the preset modality is , and each row and column of the preset modality prompt weight matrix has only one non-zero element, and the value of the non-zero element is 1. The sum of the prompt weight matrices of all preset modalities can form a matrix with all elements being non-zero. A matrix, that is, a matrix with all 1s matrix.

[0033] At the same time, the initial comprehensive prompt is obtained, which can be expressed as , is the number of rows of the comprehensive prompt matrix, is the number of columns of the comprehensive prompt matrix. The initial comprehensive prompt is a preset comprehensive matrix, which is mainly used to characterize the modal prompts of all preset modalities. By optimizing the initial comprehensive prompt, the initial comprehensive prompt is decomposed into a low-rank subspace to obtain a comprehensive prompt matrix. This decomposition can greatly reduce the amount of trainable parameters of the comprehensive prompt matrix. Among them, the comprehensive prompt matrix can be expressed as , which can include For ease of understanding, The cue sub-matrix can be expressed as , , , represents the minimum matrix decomposition dimension of the prompt submatrix, and These are randomly initialized prompt decomposition matrices, which can be changed and adjusted during the training process.

[0034] The above-mentioned preset mode can be set according to actual conditions, and the embodiments of the present application do not specifically limit this. For example, the preset mode can be a text mode, an image mode, an audio mode, etc., but is not limited thereto.

[0035] The virtual input can be set according to actual conditions, and the embodiment of the present application does not specifically limit this. For example, the virtual input can be a blank text string or a blank image pixel, but is not limited thereto.

[0036] The minimum matrix decomposition dimension of the above-mentioned prompt submatrix refers to the minimum matrix decomposition dimension that can make the prompt submatrix maintain the model performance and modality prompt representation ability. For example, for a size of If we use a matrix of size The submatrix and size are The matrix is ​​replaced by multiplying the submatrix of , and the replaced matrix can maintain the original modal cue representation ability and the original model performance to 90%. is the minimum matrix decomposition dimension.

[0037] The above randomly initialized matrix can be changed and adjusted through a training process, and the embodiments of the present application do not specifically limit this.

[0038] S102, training the preset multimodal large model using the comprehensive prompt matrix, each data sample and the prompt weight matrix of the preset modality that is missing from each data sample, and during the training of the multimodal large model, optimizing the comprehensive prompt matrix and the prompt weight matrix of each preset modality to obtain the target comprehensive prompt matrix and the target prompt weight matrix of each preset modality as the modal prompt.

[0039] In this step, after obtaining each data sample, the preset modality missing from each data sample is first determined, and then the preset multimodal large model is trained using the comprehensive prompt matrix, each data sample, and the prompt weight matrix of the preset modality missing from each data sample. During the training period, the comprehensive prompt matrix and the prompt weight matrix of multiple preset modalities are used as part of the network parameters of the multimodal large model, and the comprehensive prompt matrix and the prompt weight matrix of each preset modality are optimized to determine the target comprehensive prompt matrix and the target prompt weight matrix of each preset modality. The target comprehensive prompt matrix and the target prompt weight matrix of each preset modality are modal prompts. In this way, using the obtained two-class prompt matrix, the two-class matrix can be used for input according to the above method to conduct experiments and other downstream tasks, and the robustness of the output of the multimodal large model can be guaranteed by assisting the input with the two-class prompt matrix.

[0040] The preset mode missing from the above data samples can be set according to actual conditions, and the embodiments of the present application do not specifically limit this. For example, the preset mode missing from a part of the data samples can be a text mode, and the preset mode missing from another part of the data samples can be an audio mode, but it is not limited thereto.

[0041] The type of the multimodal large model can be set according to actual conditions, and the embodiment of the present application does not specifically limit this. For example, the multimodal large model can be a multimodal transformer, EfficientNLP or EfficientMTG, but is not limited thereto.

[0042] The above target comprehensive prompt matrix refers to the optimal comprehensive prompt matrix.

[0043] The target prompt weight matrix of each preset mode mentioned above refers to the optimal prompt weight matrix of each preset mode.

[0044] In summary, the embodiment of the present application realizes the generation of modal prompts by introducing a comprehensive prompt matrix and a prompt weight matrix of each preset modality, wherein the comprehensive prompt matrix is ​​used to characterize the modal prompts of all preset modalities, and the prompt weight matrix of each preset modality is used to characterize the impact of the absence of the preset modality on the data sample. Specifically, the embodiment of the present application trains the preset multimodal large model through a comprehensive prompt matrix, prompt weight matrices of multiple preset modalities, and multiple data samples, and optimizes the comprehensive prompt matrix and the prompt weight matrix of each preset modality during training to obtain the target comprehensive prompt matrix and the target prompt weight matrix of each preset modality. In this way, the key information of each preset modality can be extracted without increasing the design complexity of the modal prompt, which can not only effectively improve the accuracy of the modal prompt, but also reduce the parameter redundancy of the modal prompt, and improve the generation efficiency of the modal prompt.

[0045] The above steps will be further described below with reference to the accompanying drawings. Figure 2 , Figure 2 is a principle example diagram of the modal prompt generation method provided in an embodiment of the present application, Figure 2 In it, “Prompt Weight Matrix” represents the prompt weight matrix, “Comprehensive Prompt Matrix” represents the comprehensive prompt matrix, “BK-M” represents block Kronecker similarity multiplication, “Prompts” represents prompt words, and in the embodiment of the present application, “Prompts” refers to the first prompt matrix, “Case 1: Modality Complete” represents the case of complete modality, that is, the data sample has no missing modality, “Case 2: Missing Text” represents the case where the data sample is missing text modality, “Case 3: Missing Image” represents the case where the data sample is missing image modality, “Learnable” means learnable, “Frozen” means frozen, “Input Data” refers to input data samples, “Text Embedding” refers to text vectorization, “Patch Embedding” refers to dividing the image into blocks and vectorizing, “Task Token” refers to task segmentation, “Multimodal Transformer” refers to a multimodal large model, and the multimodal transformer is used in the embodiment of the present application, and “Pooler&FC “Layer” refers to the pooling layer and the fully connected layer, which are part of the multimodal large model. Since the pooling layer and the fully connected layer are conventional designs, they are not described in detail in the embodiments of the present application. “Evidence-based Loss” refers to the evidence theory loss function, and “Prediction” refers to the prediction result of the multimodal large model.

[0046] In some embodiments, in the above step S102, the implementation process of training the preset multimodal large model using the comprehensive prompt matrix, each data sample and the prompt weight matrix of the preset modality missing from each data sample may include the following steps A01-A04.

[0047] A01, for each data sample, add the prompt weight matrices of the preset modalities that are missing in the data sample to obtain a first prompt sub-matrix, and perform block Kronecker similarity multiplication on the first prompt sub-matrix and the comprehensive prompt matrix to obtain a first prompt matrix for the data sample.

[0048] In this step, for each data sample, for all missing preset modalities in the data sample, the prompt weight matrices of each missing preset modality are added together to obtain a first prompt sub-matrix corresponding to the data sample. The first prompt sub-matrix is ​​mainly used to learn context information to make up for the information of the preset modality missing from the data sample, so as to subsequently generate targeted modal prompts. After obtaining the first prompt sub-matrix corresponding to the data sample, the similarity multiplication result of the first prompt sub-matrix and the comprehensive prompt matrix is ​​calculated by block Kronecker similarity multiplication (BK-M), that is, , Indicates The first hint submatrix of data samples, Represents the comprehensive prompt matrix, and the similarity multiplication result is the first prompt matrix of the data sample. In this way, information interaction and data fusion are realized between the comprehensive prompt matrix and the prompt weight matrix of the preset modality that is missing in the data sample, which can more comprehensively perceive the key information of the preset modality that is missing in the data sample, thereby improving the accuracy of the modal prompt and compressing the training parameters of the multimodal large model.

[0049] A02, perform input embedding processing on each data sample to obtain an embedding vector for each data sample.

[0050] In this step, an embedding operation is performed on each data sample. In the embedding operation, since the label information of the data sample and the preset mode of the data sample both have values ​​that can be input, the label information of the data sample and the preset mode of the data sample can be converted into a vector form that can be understood by the multimodal large model through the embedding operation. Since the preset mode missing from the data sample has no value that can be input, a virtual input is used. The preset mode missing from the data sample cannot be converted into a vector form through the embedding operation. Therefore, the all-zero embedding information corresponding to the preset mode missing from the data sample is used as the embedding vector of the preset mode missing from the data sample, so that the missing preset mode can also be converted into a vector form that can be understood by the multimodal large model, thereby ensuring the normal training of the multimodal large model.

[0051] The above-mentioned all-zero embedding information corresponding to the preset modality in which the data sample is missing can be set according to the actual situation, and this embodiment does not specifically limit this. For example, for the missing text modality, an all-zero vector is used as the embedding vector of this text modality, and the all-zero vector is the all-zero embedding information; for another example, for the missing image modality, an image with all zero pixels is used as the embedding vector of this image modality, and the image with all zero pixels is the all-zero embedding information.

[0052] A03, concatenating the embedding vector of each data sample and the first prompt matrix to obtain input information of each data sample.

[0053] In this step, the embedding vector and the first prompt matrix of each data sample are concatenated (Concatenate) to obtain the input information of each data sample and input it into the multimodal large model. Specifically, for each data sample, the embedding vector and the first prompt matrix of the data sample are processed by six layers of sequentially connected Transformers to obtain the input information of the data sample. In the six layers of sequentially connected Transformers, the first prompt matrix of the data sample is used as the prompt word (Prompts), and the embedding vector and the prompt word of the data sample are concatenated and input into the first layer Transformer, and the output of the first layer Transformer and the prompt word are concatenated and input into the second layer Transformer, and the output of the second layer Transformer and the prompt word are concatenated and input into the third layer Transformer, and so on. The output of the last layer Transformer is the input information of the data sample. For ease of understanding, the output of each layer Transformer can be expressed as the following formula (1): (1); In formula (1), ; Embedding vector representing the data sample; A first prompt matrix representing a data sample; Indicates Layer Transformer; Indicates The hidden layer of the Transformer is The output of the layer Transformer; Indicates The hidden layer of the layer, that is, The output of the layer Transformer. A04, using the input information of each data sample as the input of the multimodal large model to train the multimodal large model.

[0054] In this step, the input information of each data sample is input into the multimodal large model to train the multimodal large model.

[0055] In some embodiments, in the above step S102, during the training of the multimodal large model, the comprehensive prompt matrix and the prompt weight matrix of each preset modality are optimized, and the implementation process of obtaining the target comprehensive prompt matrix and the target prompt weight matrix of each preset modality may include the following steps B01-B06.

[0056] B01, based on the network parameters and the data samples of the current training batch, train the multimodal large model for the current training batch.

[0057] It should be noted that the network parameters refer to the parameters of the multimodal large model, which may include but are not limited to the comprehensive prompt matrix, the model parameters of the multimodal large model and the prompt weight matrices of multiple preset modalities.

[0058] In this step, each training batch has a corresponding data sample. training batches, according to the network parameters and The multimodal large model is trained with data samples from training batches, where the network parameters include the comprehensive prompt matrix , Model parameters of multimodal large models and the prompt weight matrix for multiple preset modalities , comprehensive prompt matrix It can be visualized as the prompt decomposition matrix in each prompt sub-matrix and prompt decomposition matrix .

[0059] The model parameters of the above-mentioned multimodal large model refer to the parameters used to define and adjust the behavior of the multimodal large model. These parameters will be continuously optimized during the training process so that the multimodal large model can more accurately predict or classify the input data.

[0060] The model parameters of the multimodal large model can be set according to actual conditions, and this embodiment does not specifically limit this. For example, the model parameters of the multimodal large model may include fully connected layer parameters, pooling layer parameters, etc., but are not limited thereto.

[0061] B02, when the training of the current training batch is completed, the loss function of the multimodal large model is obtained according to the predicted label information and label information of each data sample of the current training batch.

[0062] In this step, for training batches, when When the training of the training batch is completed, the prediction results output by the multimodal large model are obtained, and the prediction results include The predicted label information of each data sample in the training batch is then used The predicted label information and label information of each data sample in a training batch are used to calculate the loss function of the multimodal large model. The loss function can measure the degree of difference between the predicted label information and the label information.

[0063] The predicted label information of the above data samples refers to the value of the output layer of the multimodal large model before the activation function is applied.

[0064] The label information of the above data samples refers to the label information mentioned in the above step S101, that is, the real label information.

[0065] The above loss function is used to measure the difference between the label information predicted by the multimodal large model and the actual label information.

[0066] The above loss function can be set according to actual conditions, and this embodiment does not specifically limit this. For example, the above loss function can be a cross entropy loss function, but is not limited thereto.

[0067] B03, according to the loss function of the multimodal large model, obtain the optimized network parameters and the optimization objective function of the multimodal large model.

[0068] In this step, after obtaining the loss function of the multimodal large model, the network parameters are optimized according to the loss function of the multimodal large model to obtain the optimized network parameters, and the optimization objective function of the multimodal large model is calculated by the loss function of the multimodal large model. The optimization objective function is used to characterize the degree of difference between the current value and the optimal value of the network parameters.

[0069] The above optimization objective function is used to measure the difference between the current value and the optimal value of the network parameter. The smaller the optimization objective function is, the closer the network parameter is to the optimal value; the larger the optimization objective function is, the further away the network parameter is from the optimal value.

[0070] B04, determine whether the optimization objective function is greater than or equal to a preset optimization threshold; if so, proceed to step B05; if not, proceed to step B06; B05, increment the current training batch by one, use the optimized network parameters as new network parameters, and return to the step of training the current training batch for the multimodal large model based on the network parameters and the data samples of the current training batch; B06, using the optimized comprehensive prompt matrix in the optimized network parameters as the target comprehensive prompt matrix, and using the optimized prompt weight matrix of each preset mode in the optimized network parameters as the target prompt weight matrix of each preset mode.

[0071] In the above steps, by optimizing the network parameters of the multimodal large model, including the prompt decomposition matrix , prompt decomposition matrix , Model parameters of multimodal large models and the prompt weight matrix for multiple preset modalities , and finally realize the effective processing of modal prompts. The optimization process of the multimodal large model must meet the optimization objective function. When the optimization objective function is minimized, it means that the network parameters of the multimodal large model have reached the optimal value. At this time, the training is terminated and the network parameters of the multimodal large model are output. Specifically, for the first training batches. If the optimization objective function is greater than or equal to the optimization threshold, it means that the network parameters have not reached the optimal value. , the optimized network parameters are used as new network parameters, and return to the above step B01; if the optimization objective function is less than the optimization threshold, it means that the network parameters have reached the optimal value. At this time, the optimized comprehensive prompt matrix in the optimized network parameters is used as the target comprehensive prompt matrix, and the optimized prompt weight matrix of each preset mode in the optimized network parameters is used as the target prompt weight matrix of each preset mode, so as to obtain the modal prompt and realize the effective processing of the modal prompt. In this way, using the obtained two-class prompt matrix, the two-class matrix can be used for input according to the above method to conduct experiments and other downstream tasks, and the robustness of the output of the multi-modal large model can be guaranteed by assisting the input of the two-class prompt matrix.

[0072] The above optimization threshold can be set according to actual conditions, and this embodiment does not specifically limit this. In some embodiments, in the above step B02, the implementation process of obtaining the loss function of the multimodal large model according to the predicted label information and label information of each data sample in the current training batch may include the following steps: According to the predicted label information and label information of each data sample in the current training batch, the evidence theory loss function is obtained as the loss function of the multimodal large model.

[0073] In this embodiment, in response to the possible uncertainty in reasoning caused by the missing preset modality, the evidence theory (Evidence-Theory) loss function is calculated through the predicted label information and label information of each data sample in the current training batch, and the evidence theory loss function is used as the loss function of the multimodal large model. This can not only reduce the uncertainty of the missing preset modality in the multimodal data and reduce the risk of misleading the prediction of the multimodal large model by uncertainty information, but also help the multimodal large model to better integrate uncertainty information when facing modality missing, and enhance the ability of the multimodal large model to deal with uncertainty information. For example, in film analysis, text analysis may mistakenly classify a thriller as a comedy, but a comprehensive consideration of multimodal information such as text and images can make a more accurate judgment, thereby improving the decision-making ability of the multimodal large model in prediction tasks and enhancing the robustness of the multimodal large model.

[0074] The above evidence theory loss function uses the predicted label information (logits) of the multimodal large model as evidence. The predicted label information refers to the value of the output layer of the multimodal large model before the activation function is applied. It provides the confidence or score information of the multimodal large model for different categories. For ease of understanding, the above evidence theory loss function satisfies the following formula (2): (2); In formula (2), represents the evidence-theoretic loss function; Represents the total number of classification categories in the classification task; Indicates Activation information for each category, , Indicates The predicted label information of each category after being processed by the activation function; is the gamma function; Indicates The label information of each category; is the sum of activation information of all categories, that is .

[0075] In some implementations, in the above step B03, the implementation process of obtaining the optimized network parameters and the optimization objective function of the multimodal large model according to the loss function of the multimodal large model may include: The loss function of the multimodal large model is used in combination with the back propagation algorithm to optimize the network parameters and obtain the optimized network parameters.

[0076] In this implementation, the back propagation algorithm (BP) is used in combination with the evidence theory loss function obtained in the above steps to optimize the network parameters of the multimodal large model, thereby obtaining optimized network parameters.

[0077] In some implementations, in the above step B03, the process of obtaining the optimized network parameters and the optimization objective function of the multimodal large model according to the loss function of the multimodal large model may include the following steps: Obtain the product of a preset penalty item parameter and a preset regularization item parameter as an auxiliary item parameter; Obtaining the difference between the preset value and the penalty item parameter as the penalty item difference; Get the product of the penalty term difference and the loss function as the loss function term parameter; The sum of the auxiliary item parameters and the loss function item parameters is obtained as the optimization objective function.

[0078] In this embodiment, by optimizing the network parameters of the multimodal large model, including the prompt decomposition matrix , prompt decomposition matrix , Model parameters of multimodal large models and the prompt weight matrix for multiple preset modalities , and finally realize the effective processing of modal prompts, wherein the optimization process of the multimodal large model needs to meet the optimization objective function. When the optimization objective function is minimized, it means that the network parameters of the multimodal large model have reached the optimal value. At this time, the training is terminated and the network parameters of the multimodal large model are output. For the optimization objective function, first, the product of the preset penalty term parameter and the preset regularization term parameter is obtained as the auxiliary term parameter, wherein the penalty term parameter is used to improve the generalization ability of the multimodal large model, and the regularization term parameter is used to alleviate the adverse effects caused by misleading evidence; then, the difference between the preset value and the penalty term parameter is obtained as the penalty term difference, wherein the preset value can be flexibly set according to the actual situation; then, the product of the penalty term difference and the loss function obtained in the above steps is obtained as the loss function term parameter; finally, the sum of the auxiliary term parameter and the loss function term parameter is obtained as the optimization objective function.

[0079] The above penalty parameters are used to improve the generalization ability of large multimodal models.

[0080] The above regularization term parameters are used to mitigate the adverse effects caused by misleading evidence. Misleading evidence refers to misleading prediction label information, which can be understood as data that will cause misleading predictions in the training of multimodal large models.

[0081] The penalty term parameters and the regularization term parameters can be set according to actual conditions, and this embodiment does not specifically limit this.

[0082] The above-mentioned preset value can be set according to actual conditions, and this embodiment does not specifically limit this. For example, the above-mentioned preset value can be one, but is not limited thereto.

[0083] The above optimization objective function satisfies the following formula (3): (3); In formula (3), It means that when the optimization objective function is minimized, the network parameters of the multimodal large model reach the optimal value. represents the prompt weight matrix, and Both represent the prompt decomposition matrix in the comprehensive prompt matrix. Represents the model parameters of the multimodal large model; represents the preset regularization term parameter, which is used to alleviate the adverse effects caused by misleading evidence; Represents the preset penalty parameter; Indicates the preset value, the preferred value is 1; represents the evidence-theoretic loss function.

[0084] Secondly, the implementation steps of the multimodal data prediction method provided by the embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0085] The multimodal data prediction method provided in the embodiment of the present application can be applied to a terminal, a server, or software running in a terminal or a server. The terminal can be a tablet computer, a laptop computer, a desktop computer, etc., but is not limited thereto. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. In addition, the server can also be a node server in a blockchain network, but is not limited thereto. Among them, blockchain is a new application model of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, and encryption algorithm.

[0086] Reference Figure 3 , Figure 3 is a flowchart of a multimodal data prediction method provided in an embodiment of the present application. The multimodal data prediction method may include the following steps S201-S202.

[0087] S201, obtaining multimodal data to be tested, where the multimodal data lacks at least one preset modality.

[0088] In this step, the multimodal data to be tested is obtained as the prediction object, and the multimodal data lacks at least one preset mode, wherein the preset mode missing from the multimodal data can be set according to the actual situation, and the embodiment of the present application does not specifically limit this. For example, the preset mode missing from the multimodal data can be a video description mode, but is not limited to this. It can be understood that if the multimodal data does not lack any preset mode, the multimodal data can be directly input into the multimodal large model to achieve prediction processing.

[0089] S202, based on the target comprehensive prompt matrix, the multimodal data and the target prompt weight matrix of the preset modality missing from the multimodal data, combined with the multimodal large model, obtain the prediction results associated with the multimodal data; wherein, the target comprehensive prompt matrix and the target prompt weight matrix of the preset modality missing from the multimodal data are both obtained by the above-mentioned modal prompt generation method.

[0090] In this step, a modal prompt can be obtained through the above-mentioned modal prompt generation method. The modal prompt includes a target comprehensive prompt matrix and a target prompt weight matrix of each preset modality. Based on this, first, the preset modality missing from the multimodal data is determined; then, from the target prompt weight matrix of each preset modality, the target prompt weight matrix of the preset modality missing from the multimodal data is found; thereafter, according to the multimodal data, the target comprehensive prompt matrix in the modal prompt and the target prompt weight matrix of the preset modality missing from the multimodal data, combined with the multimodal large model, a prediction result associated with the multimodal data is obtained, thereby realizing multimodal data prediction for modality missing.

[0091] From the above, it can be seen that the embodiment of the present application can make up for the missing information of the preset modality in the multimodal data and realize multimodal data prediction for modality missing only through a target comprehensive prompt matrix and a target prompt weight matrix of the missing preset modality, which can effectively improve the prediction accuracy of the multimodal large model.

[0092] In some embodiments, in the above step S202, the implementation process of obtaining the prediction result associated with the multimodal data based on the target comprehensive prompt matrix, the multimodal data and the target prompt weight matrix of the preset modality missing from the multimodal data, combined with the multimodal large model, may include the following steps C01-C05.

[0093] C01, obtaining the sum of target prompt weight matrices of preset modalities that are missing from the multimodal data as the second prompt sub-matrix.

[0094] In this step, for all missing preset modalities in the multimodal data, the prompt weight matrices of each missing preset modality are added together to obtain a second prompt sub-matrix corresponding to the multimodal data. The second prompt sub-matrix is ​​mainly used to learn contextual information to make up for the missing preset modality information in the multimodal data.

[0095] C02, perform block Kronecker similarity multiplication on the target comprehensive cue matrix and the second cue sub-matrix to obtain the second cue matrix of multimodal data.

[0096] In this step, after obtaining the second prompt sub-matrix corresponding to the multimodal data, the similarity multiplication result of the target comprehensive prompt matrix and the second prompt sub-matrix is ​​calculated by block Kronecker similarity multiplication (BK-M). The obtained similarity multiplication result is the second prompt matrix of the multimodal data. In this way, information interaction and data fusion are achieved between the comprehensive prompt matrix and the prompt weight matrix of the preset modality missing from the multimodal data, which can more comprehensively perceive the key information of the preset modality missing from the multimodal data and improve the accuracy of the modal prompt.

[0097] C03, embedding the multimodal data to obtain an embedding vector of the multimodal data.

[0098] In this step, an embedding operation is performed on the multimodal data. In the embedding operation, since the preset modes of the multimodal data all have values ​​that can be input, the preset modes of the multimodal data can be converted into a vector form that can be understood by the multimodal large model through the embedding operation. Since the preset modes missing from the multimodal data have no values ​​that can be input, virtual input is used. The preset modes missing from the multimodal data cannot be converted into a vector form through the embedding operation. Therefore, the all-zero embedding information corresponding to the preset modes missing from the multimodal data is used as the embedding vector of the preset modes missing from the multimodal data, so that the missing preset modes can also be converted into a vector form that can be understood by the multimodal large model, so as to ensure that the prediction of the multimodal large model is carried out normally.

[0099] The above-mentioned all-zero embedding information corresponding to the preset modality missing from the multimodal data can be set according to the actual situation, and this embodiment does not specifically limit this. For example, for the missing text modality, an all-zero vector is used as the embedding vector of this text modality, and the all-zero vector is the all-zero embedding information; for another example, for the missing image modality, an image with all zero pixels is used as the embedding vector of this image modality, and the image with all zero pixels is the all-zero embedding information.

[0100] C04, concatenate the embedding vector of the multimodal data and the second prompt matrix to obtain the input information of the multimodal data as the input of the multimodal large model.

[0101] In this step, the embedding vector of the multimodal data and the second prompt matrix are concatenated to obtain the input information of the multimodal data and input it into the multimodal large model. Specifically, the embedding vector of the multimodal data and the second prompt matrix are processed by a six-layer sequentially connected Transformer to obtain the input information of the multimodal data. The implementation is the same as the above-mentioned implementation method, which will not be described in detail.

[0102] C05, performs prediction processing through a multimodal large model to obtain prediction results associated with multimodal data.

[0103] In this step, the input information of the multimodal data is input into the multimodal large model, and the multimodal large model performs prediction processing and outputs a prediction result associated with the multimodal data. It should be noted that the prediction processing of the above-mentioned multimodal large model includes prediction or classification, which means that the multimodal large model can be used to process classification tasks or prediction tasks, and the specific tasks can be flexibly designed according to actual conditions.

[0104] The contents of the above-mentioned modal prompt generation method embodiment are all applicable to the present multimodal data prediction method embodiment. The functions specifically implemented by the present multimodal data prediction method embodiment are the same as those of the above-mentioned modal prompt generation method embodiment, and the beneficial effects achieved are also the same as those achieved by the above-mentioned modal prompt generation method embodiment.

[0105] To facilitate the understanding of the above-mentioned modal prompt generation method and the above-mentioned multimodal data prediction method in the embodiments of the present application, an application scenario of the embodiments of the present application is used as an example for illustration.

[0106] Reference Figure 2 The modal prompt in the embodiment of the present application includes a target comprehensive prompt matrix and The target prompt weight matrix of a preset modality, and the generation process of the modality prompt is specifically as follows steps S301-S304.

[0107] S301, obtain multiple data samples, A data sample can be represented as , Indicates the total number of preset modes, Indicates The data sample Preset modes, Indicates The label information of the data samples.

[0108] At the same time, obtain The prompt weight matrix of the preset mode, that is, for the Mode , defining a unique cue weight matrix , the size of the prompt weight matrix of the preset modality is , and each row and column of the preset modality prompt weight matrix has only one non-zero element, and the value of the non-zero element is 1. The sum of all preset modality prompt weight matrices can form a full-1 matrix.

[0109] In one example, the preset modality includes text modality and image modality, that is, the text modality and the image modality constitute a complete modality, which means that when a data sample has text modality and image modality, the data sample does not lack any modality. Among them, the prompt weight matrix of the text modality can be expressed as , the cue weight matrix of the image modality can be expressed as The hint weight matrix of the text modality and the image modality can be added together to form the hint weight matrix of the complete modality , that is, a full 1 matrix.

[0110] At the same time, the initial comprehensive prompt is obtained, which can be expressed as , is the number of rows of the comprehensive prompt matrix, is the number of columns of the comprehensive prompt matrix. By optimizing the initial comprehensive prompt, the initial comprehensive prompt is decomposed into a low-rank subspace to obtain a comprehensive prompt matrix. This decomposition can greatly reduce the amount of trainable parameters of the comprehensive prompt matrix. Among them, the comprehensive prompt matrix can be expressed as , which can include The prompt sub-matrix, The cue sub-matrix can be expressed as , , , represents the minimum matrix decomposition dimension of the prompt submatrix, and These are randomly initialized prompt decomposition matrices, which can be changed and adjusted during the training process.

[0111] In one example, the preset modes include text mode and image mode. , the initial comprehensive prompt Decompose into low-rank subspace, and get Comprehensive Tips Matrix , , , , , , , , , , , , .

[0112] S302: For each data sample, the following processing steps S3021-S3024 are performed: S3021, for all missing preset modes in the data sample, add the prompt weight matrices of each missing preset mode to obtain a first prompt sub-matrix corresponding to the data sample; S3022, calculating the similarity multiplication result of the first prompt sub-matrix and the comprehensive prompt matrix by block Kronecker similarity multiplication, that is, , Indicates The first hint submatrix of data samples, represents the comprehensive prompt matrix, and the similarity multiplication result is the first prompt matrix of the data sample; S3023, embedding operation is performed on the data sample. In the embedding operation, since both the label information of the data sample and the preset mode of the data sample have values ​​that can be input, the label information of the data sample and the preset mode of the data sample can be converted into a vector form that can be understood by the multimodal large model through the embedding operation. Since the preset mode missing from the data sample has no value that can be input, a virtual input is used. The preset mode missing from the data sample cannot be converted into a vector form through the embedding operation. Therefore, the all-zero embedding information corresponding to the preset mode missing from the data sample is used as the embedding vector of the preset mode missing from the data sample.

[0113] In one example, the preset modalities include text modality and image modality. If the data sample lacks text modality, an all-zero vector is used as the embedding vector of this text modality. If the data sample lacks image modality, an image with all zero pixels is used as the embedding vector of this image modality.

[0114] S3024, for each data sample, the embedding vector and the first prompt matrix of the data sample are processed by six layers of sequentially connected Transformers to obtain the input information of the data sample. In the six layers of sequentially connected Transformers, the first prompt matrix of the data sample is used as the prompt word, the embedding vector of the data sample and the prompt word are concatenated and input into the first layer Transformer, the output of the first layer Transformer and the prompt word are concatenated and input into the second layer Transformer, the output of the second layer Transformer and the prompt word are concatenated and input into the third layer Transformer, and so on. The output of the last layer Transformer is the input information of the data sample.

[0115] S303, inputting the input information of each data sample into the multimodal large model, and training the multimodal large model, from To begin with, for There are the following steps S3031-S3034 for each training batch: S3031, according to the network parameters and The multimodal large model is trained with data samples from training batches, where the network parameters include the prompt decomposition matrix , prompt decomposition matrix , Model parameters of multimodal large models and the prompt weight matrix for multiple preset modalities ; S3032, when When the training of the training batch is completed, the prediction results output by the multimodal large model are obtained, and the prediction results include The predicted label information of each data sample in the training batch is then used The predicted label information and label information of each data sample in the training batch are used to calculate the evidence theory loss function shown in the above formula (2) as the loss function of the multimodal large model; S3033, using the loss function of the multimodal large model in combination with the back propagation algorithm, optimizing the network parameters to obtain the optimized network parameters, and using the loss function of the multimodal large model in combination with the above formula (3) to calculate the optimization objective function; S3034, determining whether the optimization objective function is greater than or equal to a preset optimization threshold; If so, then , taking the optimized network parameters as new network parameters, and returning to the above step S3031; If not, go to step S304.

[0116] S304, using the optimized comprehensive prompt matrix in the optimized network parameters as the target comprehensive prompt matrix, and using the optimized prompt weight matrix of each preset mode in the optimized network parameters as the target prompt weight matrix of each preset mode, thereby obtaining a target comprehensive prompt matrix and The target cue weight matrix of a preset modality is obtained. In this way, using the obtained two-class cue matrix, the two-class matrix can be used as input according to the above method to conduct experiments and other downstream tasks, and the two-class cue matrix can be used to assist in input to ensure the robustness of the multi-modal large model output.

[0117] The technical effect of the above-mentioned modal prompt generation method in the embodiment of the present application will be verified below.

[0118] Reference Figure 4 , Figure 4 Medium Case 1, Case 2, Case 3,…, Case Indicates the situation of missing modality, Modality 1, Modality 2, Modality 3, ..., Modality Indicates the preset mode, comparing the method of the embodiment of the present application with the existing MAP and MSP: MAP needs to generate Modal prompts , MSP needs to be generated Modal prompts The method of the embodiment of the present application needs to generate a modal prompt, which includes a comprehensive prompt matrix and Cue weight matrix , and due to a comprehensive prompt matrix and All cue weight matrices are in matrix form, so their parameter size is extremely small.

[0119] Refer to Table 1, which is a complexity and parameter table of the method, MAP and MSP of the embodiment of the present application, where "Method" refers to the modal enhancement generation method, "EPE-P (Ours)" represents the method of the embodiment of the present application, "Params" refers to the parameter amount of the modal enhancement generation method, and "Complexity" refers to the design complexity of the modal enhancement generation method.

[0120] Table 1: Comparison of the complexity of modal prompts in the embodiment of the present application, MAP and MSP

[0121] As shown in Table 1, the method of the embodiment of the present application is compared with the existing MAP and MSP: For the parameter amount, the parameter amount of MAP is , the parameters of MSP are , the parameter quantity of the method of the embodiment of the present application is It can be seen that the number of parameters in this application is much smaller than that of MAP and MSP. As for complexity, the complexity of MAP and MSP is , and the complexity of the method of the embodiment of the present application is , it can be seen that the complexity of this application is much smaller than that of MAP and MSP.

[0122] It can be seen that the embodiments of the present application can extract key information of each preset mode without increasing the design complexity of the modal prompt, which can not only effectively improve the accuracy of the modal prompt, but also reduce the parameter redundancy of the modal prompt and improve the generation efficiency of the modal prompt.

[0123] The specific process of implementing multimodal data prediction in the absence of modality in the embodiment of the present application is as follows: steps S401-S406.

[0124] S401, obtaining multimodal data to be tested as a prediction object, wherein the multimodal data lacks at least one preset mode.

[0125] S402, for all missing preset modalities in the multimodal data, the prompt weight matrices of each missing preset modality are added together to obtain a second prompt sub-matrix corresponding to the multimodal data, and the second prompt sub-matrix is ​​mainly used to learn context information to make up for the missing preset modality information in the multimodal data.

[0126] S403, calculating the similarity multiplication result of the target comprehensive prompt matrix and the second prompt sub-matrix by block Kronecker similarity multiplication, and the obtained similarity multiplication result is the second prompt matrix of the multimodal data.

[0127] S404, performing embedding operation on the multimodal data. In the embedding operation, since the preset modes of the multimodal data all have values ​​that can be input, the preset modes of the multimodal data can be converted into a vector form that can be understood by the multimodal large model through the embedding operation. Since the preset modes missing from the multimodal data have no values ​​that can be input, virtual input is used. The preset modes missing from the multimodal data cannot be converted into a vector form through the embedding operation. Therefore, the all-zero embedding information corresponding to the preset modes missing from the multimodal data is used as the embedding vector of the preset modes missing from the multimodal data.

[0128] S405, the embedding vector of the multimodal data and the second prompt matrix are processed by six layers of sequentially connected Transformers to obtain input information of the multimodal data, which is implemented in the same way as the above steps.

[0129] S406, inputting the input information of the multimodal data into the multimodal large model, the multimodal large model performs prediction processing and outputs a prediction result associated with the multimodal data.

[0130] The technical effect of the above-mentioned multimodal data prediction method in the embodiment of the present application will be verified below.

[0131] Refer to Table 2, which is a table of relevant experimental results of the embodiments of the present application, "Datasets" represents data sets, "F1-Macro" represents macro F1 value, and macro F1 value is used as the performance indicator of the multimodal large model in the MM-IMDb data set, "AUROC" is the full name of Area Under the Receiver Operating Characteristic curve, that is, the area under the receiver operating characteristic curve, and the area under the receiver operating characteristic curve is used as the performance indicator of the multimodal large model in the Hateful Memes data set, "Missing-Rate" represents the loss rate of the modality, "Training" represents the training set, "Text" represents the data proportion of the text modality, "Image" represents the data proportion of the image modality, "Testing" represents the test set, "Baseline" represents the baseline indicator value, "EPE-P (Ours)" represents the method of the embodiment of the present application, and "EPE-P (- )" refers to a method that does not use the evidence theory loss function but has other features that are the same as the embodiment of the present application.

[0132] Table 2: Table of relevant experimental results of the embodiments of the present application

[0133] It can be seen from Table 2 that: compared with the existing MAP and benchmark indicators, in the prediction task of the existing MM-IMDb dataset, the method of the embodiment of the present application achieved the highest macro F1 value, and in the prediction task of the existing Hateful Memes dataset, the method of the embodiment of the present application achieved the highest area under the receiver operating characteristic curve. It can be seen that the embodiment of the present application can improve the prediction accuracy of multimodal data, especially for modal missing situations. Compared with the method that does not use the evidence theory loss function but has other features that are the same as the embodiment of the present application, the performance of the embodiment of the present application on the existing MM-IMDb dataset and the Hateful Memes dataset has been improved, that is, the embodiment of the present application uses the evidence theory loss function as the loss function of the multimodal large model, which can effectively reduce the risk of uncertainty information misleading the prediction of the multimodal large model, and improve the ability of the multimodal large model to deal with uncertainty information, thereby improving the decision-making ability of the multimodal large model in the prediction task and enhancing the robustness of the multimodal large model.

[0134] In summary, it can be seen that the embodiments of the present application can not only effectively deal with the problem of missing modalities, but also optimize the use of prompts for complete modal samples during the reasoning process, thereby achieving a comprehensive improvement in effectiveness and efficiency.

[0135] In addition, refer to Figure 5 , the embodiment of the present application further provides a modal prompt generating device, which may include: The first acquisition module 501 is used to obtain a comprehensive prompt matrix, multiple data samples and a prompt weight matrix of multiple preset modalities; wherein the data sample includes label information and multiple preset modalities, and a preset virtual input is used as the preset modality missing from the data sample; the prompt weight matrix of the preset modality is used to characterize the impact of the missing of the preset modality on the data sample; the comprehensive prompt matrix is ​​used to characterize the modal prompts of all preset modalities; The first processing module 502 is used to train the preset multimodal large model using the comprehensive prompt matrix, each data sample and the prompt weight matrix of the preset modality that is missing from each data sample, and during the training of the multimodal large model, optimize the comprehensive prompt matrix and the prompt weight matrix of each preset modality to obtain the target comprehensive prompt matrix and the target prompt weight matrix of each preset modality as the modal prompt.

[0136] The contents of the above-mentioned modal prompt generation method embodiment are all applicable to the present modal prompt generation device embodiment. The functions specifically implemented by the present modal prompt generation device embodiment are the same as those of the above-mentioned modal prompt generation method embodiment, and the beneficial effects achieved are also the same as those achieved by the above-mentioned modal prompt generation method embodiment.

[0137] Finally, refer to Figure 6The embodiment of the present application further provides a multimodal data prediction device, which may include: A second acquisition module 601 is used to acquire multimodal data to be tested, where the multimodal data lacks at least one of the preset modes; The second processing module 602 is used to obtain a prediction result associated with the multimodal data according to the target comprehensive prompt matrix, the multimodal data and the target prompt weight matrix of the preset mode missing from the multimodal data, combined with the multimodal large model; Among them, the target comprehensive prompt matrix and the target prompt weight matrix of the preset mode missing in the multimodal data are both obtained through the above-mentioned modality prompt generation device.

[0138] The contents of the above-mentioned modal prompt generation method embodiment and the above-mentioned multimodal data prediction method embodiment are applicable to the present multimodal data prediction device embodiment. The functions specifically implemented by the present multimodal data prediction device embodiment are the same as those of the above-mentioned modal prompt generation method embodiment and the above-mentioned multimodal data prediction method embodiment, and the beneficial effects achieved are also the same as those achieved by the above-mentioned modal prompt generation method embodiment and the above-mentioned multimodal data prediction method embodiment.

[0139] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the application is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are expected, wherein the order of various operations is changed and the sub-operation described as a part of a larger operation is performed independently.

[0140] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present application. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional techniques of the engineer. Therefore, those skilled in the art can implement the present application set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the attached claims and their equivalents.

[0141] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several programs to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0142] The logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable programs for implementing logical functions, and may be embodied in any computer-readable medium for use by a program execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch and execute a program from a program execution system, device or apparatus), or in conjunction with such program execution system, device or apparatus. For purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by a program execution system, device or apparatus, or in conjunction with such program execution system, device or apparatus.

[0143] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0144] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable program execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0145] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0146] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0147] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the described embodiments. Technical personnel familiar with the field may make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A method for generating a modal prompt, characterized in that: The following steps are involved: Obtaining a comprehensive prompt matrix, a plurality of data samples, and a prompt weight matrix of a plurality of preset modalities; wherein the data sample includes label information and a plurality of the preset modalities, and a preset virtual input is used as the preset modality missing from the data sample; the prompt weight matrix of the preset modality is used to characterize the impact of the missing of the preset modality on the data sample; the comprehensive prompt matrix is ​​used to characterize the modal prompts of all the preset modalities; The preset multimodal large model is trained using the comprehensive prompt matrix, each of the data samples and the prompt weight matrix of the preset modality that is missing from each of the data samples. During the training of the multimodal large model, the comprehensive prompt matrix and the prompt weight matrix of each of the preset modalities are optimized to obtain a target comprehensive prompt matrix and a target prompt weight matrix of each of the preset modalities as modal prompts.

2. The method for generating a modal prompt according to claim 1, characterized in that: The method of training a preset multimodal large model by using the comprehensive prompt matrix, each of the data samples, and a prompt weight matrix of a preset modality missing from each of the data samples includes: For each of the data samples, adding the prompt weight matrices of the preset modalities missing from the data sample to obtain a first prompt sub-matrix, and performing block Kronecker similarity multiplication processing on the first prompt sub-matrix and the comprehensive prompt matrix to obtain a first prompt matrix of the data sample; Performing input embedding processing on each of the data samples to obtain an embedding vector for each of the data samples; Concatenating the embedding vector of each of the data samples and the first prompt matrix to obtain input information of each of the data samples; The input information of each of the data samples is used as the input of the multimodal large model to train the multimodal large model.

3. The method for generating a modal prompt according to claim 1, characterized in that: During the training of the multimodal large model, the comprehensive prompt matrix and the prompt weight matrix of each preset mode are optimized to obtain a target comprehensive prompt matrix and a target prompt weight matrix of each preset mode, including: Based on the network parameters and the data samples of the current training batch, the multimodal large model is trained for the current training batch; wherein the network parameters include the comprehensive prompt matrix, the model parameters of the multimodal large model and the prompt weight matrices of the plurality of preset modalities; When the training of the current training batch is completed, obtaining the loss function of the multimodal large model according to the predicted label information and label information of each of the data samples of the current training batch; According to the loss function of the multimodal large model, the optimized network parameters and the optimization objective function of the multimodal large model are obtained; When the optimization objective function is greater than or equal to a preset optimization threshold, the current training batch is increased by one, the optimized network parameters are used as new network parameters, and the process returns to the step of training the current training batch of the multimodal large model based on the network parameters and the data samples of the current training batch until the optimization objective function is less than the optimization threshold, and the optimized comprehensive prompt matrix in the optimized network parameters is used as the target comprehensive prompt matrix, and the optimized prompt weight matrix of each of the preset modes in the optimized network parameters is used as the target prompt weight matrix of each of the preset modes.

4. The method for generating a modal prompt according to claim 3, characterized in that: The step of obtaining the loss function of the multimodal large model according to the predicted label information and label information of each of the data samples in the current training batch includes: According to the predicted label information and label information of each of the data samples in the current training batch, an evidence theory loss function is obtained as the loss function of the multimodal large model.

5. The method for generating a modal prompt according to claim 3, characterized in that: The step of obtaining optimized network parameters and an optimization objective function of the multimodal large model according to the loss function of the multimodal large model comprises: The network parameters are optimized by utilizing the loss function of the multimodal large model in combination with a back propagation algorithm to obtain the optimized network parameters.

6. The method for generating a modal prompt according to claim 3, characterized in that: The step of obtaining optimized network parameters and an optimization objective function of the multimodal large model according to the loss function of the multimodal large model comprises: Obtain the product of a preset penalty item parameter and a preset regularization item parameter as an auxiliary item parameter; Obtaining a difference between a preset value and the penalty item parameter as a penalty item difference; Obtaining the product of the penalty term difference and the loss function as a loss function term parameter; The sum of the auxiliary item parameters and the loss function item parameters is obtained as the optimization objective function.

7. A multimodal data prediction method, characterized in that: The following steps are involved: Acquire multimodal data to be tested, wherein the multimodal data lacks at least one of the preset modalities; Obtaining a prediction result associated with the multimodal data according to a target comprehensive prompt matrix, the multimodal data, and a target prompt weight matrix of a preset modality missing from the multimodal data, in combination with a preset multimodal macro model; Wherein, the target comprehensive prompt matrix and the target prompt weight matrix of the preset modality missing from the multimodal data are both obtained by a modality prompt generation method as described in any one of claims 1-6.

8. The multimodal data prediction method according to claim 7, characterized in that: The method of obtaining a prediction result associated with the multimodal data based on the target comprehensive prompt matrix, the multimodal data, and the target prompt weight matrix of the preset modality missing from the multimodal data, in combination with a preset multimodal large model, includes: Adding target prompt weight matrices of preset modalities that are missing from the multimodal data to obtain a second prompt sub-matrix; Performing block Kronecker similarity multiplication processing on the target comprehensive cue matrix and the second cue sub-matrix to obtain a second cue matrix of the multimodal data; Performing embedding processing on the multimodal data to obtain an embedding vector of the multimodal data; Concatenating the embedding vector of the multimodal data and the second prompt matrix to obtain input information of the multimodal data as input of the multimodal large model; Prediction processing is performed using the multimodal large model to obtain a prediction result associated with the multimodal data.

9. A modal prompt generating device, characterized in that: include: The first acquisition module is used to acquire a comprehensive prompt matrix, a plurality of data samples and a prompt weight matrix of a plurality of preset modalities; wherein the data sample includes label information and a plurality of the preset modalities, and a preset virtual input is used as the preset modality missing from the data sample; the prompt weight matrix of the preset modality is used to characterize the impact of the missing of the preset modality on the data sample; the comprehensive prompt matrix is ​​used to characterize the modal prompts of all the preset modalities; The first processing module is used to train a preset multimodal large model using the comprehensive prompt matrix, each of the data samples and the prompt weight matrix of the preset modality that is missing from each of the data samples, and during the training of the multimodal large model, the comprehensive prompt matrix and the prompt weight matrix of each of the preset modalities are optimized to obtain a target comprehensive prompt matrix and a target prompt weight matrix of each of the preset modalities as modal prompts.

10. A multimodal data prediction device, characterized in that: include: A second acquisition module is used to acquire multimodal data to be tested, wherein the multimodal data lacks at least one of the preset modes; A second processing module, configured to obtain a prediction result associated with the multimodal data according to a target comprehensive prompt matrix, the multimodal data and a target prompt weight matrix of a preset modality missing from the multimodal data, in combination with a preset multimodal large model; Wherein, the target comprehensive prompt matrix and the target prompt weight matrix of the preset modality missing from the multimodal data are both obtained by a modality prompt generating device as described in claim 9.

Images