Model training method, information prediction method and related products

Abstract

The invention discloses a model training method, an information prediction method and a related product. The training method comprises the steps of obtaining a first attention processing result of the first modal data on the second modal data and a second attention processing result of the second modal data on the first modal data according to a first feature vector of the first modal data and a second feature vector of the second modal data; fusing the first feature vector and the first attention processing result to obtain a first fused feature vector; fusing the second feature vector and the second attention processing result to obtain a second fused feature vector; determining the similarity between the first modal data and the second modal data based on the first fusion feature vector and the second fusion feature vector; obtaining a third fusion feature vector based on the similarity and the first feature vector; performing fusion based on the similarity and the second feature vector to obtain a fourth fusion feature vector; and training the model based on the third fusion feature vector and the fourth fusion feature vector.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a model training method, an information prediction method, and related products. Background Art

[0002] At present, artificial intelligence models are often used to predict information. This information can be user behavior, emotions, information to be recommended, etc. This application does not list them one by one. For example, it is mainly through the artificial intelligence model to perform simple fusion (for example, splicing) processing on the acquired multimodal data and then input it into a general prediction model for prediction. Since the quality of the multimodal data fusion cannot be guaranteed, the accuracy of the model prediction cannot be guaranteed.

[0003] Therefore, how to fuse multimodal features to improve the accuracy of model prediction is an urgent problem to be solved. Summary of the Invention

[0004] This application provides a model training method, information prediction method and related products, which improve the accuracy of model prediction by improving the fusion quality of multimodal data.

[0005] In a first aspect, the present application provides a model training method, the method comprising:

[0006] Calculating, based on a first eigenvector of the first modal data and a second eigenvector of the second modal data, a first attention processing result of the first modal data on the second modal data and a second attention processing result of the second modal data on the first modal data;

[0007] fusing the first feature vector and the first attention processing result to obtain a first fused feature vector corresponding to the first modal data;

[0008] fusing the second feature vector and the second attention processing result to obtain a second fused feature vector corresponding to the second modal data;

[0009] determining a similarity between the first modality data and the second modality data based on the first fused feature vector and the second fused feature vector;

[0010] Performing fusion based on the similarity and the first feature vector to obtain a third fused feature vector corresponding to the first modal data;

[0011] performing fusion based on the similarity and the second feature vector to obtain a fourth fused feature vector corresponding to the second modal data;

[0012] The model is trained based on the third fused feature vector and the fourth fused feature vector.

[0013] In a second aspect, the present application provides a model training method, the method comprising:

[0014] Calculating, based on a first eigenvector of the first modal data and a second eigenvector of the second modal data, a first attention processing result of the first modal data on the second modal data and a second attention processing result of the second modal data on the first modal data;

[0015] Determining a first weight corresponding to the first modal data based on the first attention processing result, and determining a second weight corresponding to the second modal data based on the second attention processing result;

[0016] Based on the first weight and the first attention processing result, a fifth eigenvector corresponding to the first modal data is obtained, and based on the second weight and the second attention processing result, a ninth eigenvector corresponding to the second modal data is obtained;

[0017] Obtaining a first fused feature vector corresponding to the first modality data based on the fifth eigenvector and the first eigenvector of the first modality data, and obtaining a second fused feature vector corresponding to the second modality data based on the ninth eigenvector and the second eigenvector of the second modality data;

[0018] The model is trained based on the first fused feature vector and the second fused feature vector.

[0019] In a third aspect, the present application provides a model training method, the method comprising:

[0020] determining, based on the first eigenvector of the first modal data and the second eigenvector of the second modal data, a third eigenvector corresponding to the first modal data, and determining a tenth eigenvector corresponding to the second modal data;

[0021] Determining a fourth eigenvector corresponding to the first modal data based on the third eigenvector and the similarity between the first modal data and the second modal data, and determining an eleventh eigenvector corresponding to the second modal data based on the ninth eigenvector and the similarity;

[0022] Determining a third fused eigenvector corresponding to the first modality data based on the fourth eigenvector and the first eigenvector, and determining a fourth fused eigenvector corresponding to the second modality data based on the tenth eigenvector and the second eigenvector;

[0023] The model is trained based on the third fused feature vector and the third fused feature vector.

[0024] In a fourth aspect, the present application provides an information prediction method, the method comprising:

[0025] Obtain multimodal data of users;

[0026] Information prediction is performed based on multimodal data through a model, wherein the model is trained through the model training method of the first aspect, the second aspect, or the third aspect.

[0027] In a fifth aspect, the present application provides a model training device, the device comprising: a first acquisition unit and a first processing unit;

[0028] A first acquiring unit, configured to acquire first modal data and second modal data;

[0029] a first processing unit, configured to calculate, based on a first eigenvector of the first modal data and a second eigenvector of the second modal data, a first attention processing result of the first modal data on the second modal data and a second attention processing result of the second modal data on the first modal data;

[0030] The first processing unit is further configured to fuse the first feature vector and the first attention processing result to obtain a first fused feature vector corresponding to the first modal data;

[0031] The first processing unit is further configured to fuse the second feature vector and the second attention processing result to obtain a second fused feature vector corresponding to the second modal data;

[0032] The first processing unit is further configured to determine a similarity between the first modality data and the second modality data based on the first fused feature vector and the second fused feature vector;

[0033] The first processing unit is further configured to fuse the first feature vector based on the similarity to obtain a third fused feature vector corresponding to the first modal data, and to fuse the second feature vector based on the similarity to obtain a fourth fused feature vector corresponding to the second modal data;

[0034] The first processing unit is further configured to train the model based on the third fused feature vector and the fourth fused feature vector.

[0035] or,

[0036] A first acquiring unit, configured to acquire first modal data and second modal data;

[0037] a first processing unit, configured to calculate, based on a first eigenvector of the first modal data and a second eigenvector of the second modal data, a first attention processing result of the first modal data on the second modal data and a second attention processing result of the second modal data on the first modal data;

[0038] The first processing unit is further configured to determine a first weight corresponding to the first modal data based on the first attention processing result, and to determine a second weight corresponding to the second modal data based on the second attention processing result;

[0039] The first processing unit is further configured to obtain a fifth eigenvector corresponding to the first modal data based on the first weight and the first attention processing result, and to obtain an eighth eigenvector corresponding to the second modal data based on the second weight and the second attention processing result;

[0040] The first processing unit is further configured to obtain a first fused feature vector corresponding to the first modal data based on the fifth eigenvector and the first eigenvector of the first modal data, and to obtain a second fused feature vector corresponding to the second modal data based on the eighth eigenvector and the second eigenvector of the second modal data;

[0041] The first processing unit is further configured to train the model based on the first fused feature vector and the second fused feature vector.

[0042] or,

[0043] A first acquiring unit, configured to acquire first modal data and second modal data;

[0044] a first processing unit, configured to determine, based on the first eigenvector of the first modal data and the second eigenvector of the second modal data, a third eigenvector corresponding to the first modal data and a ninth eigenvector corresponding to the second modal data;

[0045] The first processing unit is further configured to determine a fourth eigenvector corresponding to the first modal data based on the third eigenvector and the similarity between the first modal data and the second modal data, and to determine a tenth eigenvector corresponding to the second modal data based on the ninth eigenvector and the similarity;

[0046] The first processing unit is further configured to determine a third fused eigenvector corresponding to the first modality data based on the fourth eigenvector and the first eigenvector, and to determine a fourth fused eigenvector corresponding to the second modality data based on the tenth eigenvector and the second eigenvector;

[0047] The first processing unit is further used to train the model based on the third fused feature vector and the third fused feature vector.

[0048] In a sixth aspect, the present application provides an information prediction device, the device comprising: a second acquisition unit and a second processing unit;

[0049] A second acquiring unit, configured to acquire multimodal data of the user;

[0050] The second processing unit is used to predict information based on multimodal data through a model, wherein the model is trained through the model training method of the first aspect, the second aspect or the third aspect.

[0051] In the seventh aspect, the present application provides an electronic device comprising: a processor and a memory, the processor being connected to the memory, the memory being used to store computer programs, and the processor being used to execute the computer programs stored in the memory, so that the electronic device executes the method of the first aspect, the second aspect, the third aspect or the fourth aspect.

[0052] In a sixth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method of the first aspect, the second aspect, the third aspect or the fourth aspect is executed.

[0053] In a seventh aspect, the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, performs the method of the first aspect, the second aspect, the third aspect or the fourth aspect.

[0054] The implementation of this application has the following beneficial effects:

[0055] In order to improve the accuracy of model prediction, an embodiment of the present application provides a model training method for training a model, by obtaining a training sample including multimodal data, namely, first modal data and second modal data, and then based on the first eigenvector of the first modal data and the second eigenvector of the second modal data in the multimodal data, calculating the first attention processing result of the first modal data on the second modal data and the second attention processing result of the second modal data on the first modal data, which can capture the interaction relationship between cross-modal data; then the first eigenvector and the first attention processing result are fused to obtain a first fused eigenvector corresponding to the first modal data, and the second eigenvector and the second attention processing result are fused to obtain a second fused eigenvector corresponding to the second modal data, so that the first fused eigenvector and the second fused eigenvector capture the interaction characteristics with other modal data; then further, based on the first fused eigenvectors corresponding to the first modal data and the second modal data respectively and the second fusion feature vector to determine the similarity between the first modal data and the second modal data. The more unrelated the two modal data are, the smaller the corresponding similarity is, and the more related the two modal data are, the greater the corresponding similarity is. Then, based on the similarity between the first modal data and the second modal data and the first feature vector of the first modal data, fusion is performed to obtain the third fusion feature vector corresponding to the first modal data. This can suppress the interference and noise brought by the modal data with low correlation with the first modal data, improve the stability of the model, and improve the fusion quality of multimodal data. Similarly, the fourth fusion feature vector corresponding to the second modal data is obtained, and the effect is similar to the third fusion feature vector. Then, based on the third fusion feature vector and the fourth fusion feature vector corresponding to the first modal data and the second modal data respectively, the model is trained to ensure the training accuracy of the model, and then based on the trained model, the multimodal data corresponding to the user is obtained. Information prediction, such as user behavior prediction, can be performed to improve the accuracy of model prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0057] Figure 1 A flowchart of a model training method provided in an embodiment of the present application;

[0058] Figure 2 A schematic diagram of a model training method provided in an embodiment of the present application;

[0059] Figure 3 A schematic diagram of another model training method provided in an embodiment of the present application;

[0060] Figure 4 A flowchart of another model training method provided in an embodiment of the present application;

[0061] Figure 5 A flowchart of another model training method provided in an embodiment of the present application;

[0062] Figure 6 A flowchart of an information prediction method provided in an embodiment of the present application;

[0063] Figure 7 A schematic diagram of an information prediction system provided in an embodiment of the present application;

[0064] Figure 8 A schematic diagram of a model training system provided in an embodiment of the present application;

[0065] Figure 9 A block diagram of the functional units of a model training device provided in an embodiment of the present application;

[0066] Figure 10 A block diagram of the functional units of an information prediction device provided in an embodiment of the present application;

[0067] Figure 11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0069] The terms "first," "second," "third," and "fourth," etc., in the specification, claims, and drawings of this application are used to distinguish between different objects, not to describe a particular order. In addition, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0070] References herein to "embodiments" mean that a particular feature, result, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0071] First, the relevant terms and related technologies involved in the embodiments of this application are explained:

[0072] Model: In the embodiments of the present application, the model can be a prediction model for information prediction, such as user behavior prediction (which can also be called a behavior prediction model), identification and prediction of user emotions (which can also be called an emotion recognition model), prediction of personalized recommendation content (which can also be called a recommendation model), etc. This application does not list them one by one. It should be noted that for the sake of ease of understanding, the embodiments of the present application are uniformly explained as "models" without making specific distinctions.

[0073] Training samples: In the embodiments of the present application, the number of training samples used for the training model is not limited in this application, and the embodiments of the present application are mainly explained by taking one as an example; in addition, each training sample includes multimodal data corresponding to the user. As the name suggests, multimodal data includes data of multiple modalities. This application does not limit the number and type of multiple modalities, and the embodiments throughout the text are mainly explained by taking the first modal data and the second modal data in the multimodal data as an example.

[0074] Multimodal data: Regardless of which model the model in the embodiments of the present application belongs to in the above examples, the data used in training or application can be the user's multimodal data, where the multimodal data can be the user's multimodal behavior data, and the multimodal behavior data can at least include the user's audio data, video data, time series data, text data and other multimodal data.

[0075] User behavior prediction: In the financial field, for example, it can include lending behavior prediction (such as predicting whether there is a demand for lending, whether a loan will be applied for, predicting risk level, etc.), and can also include user consumption behavior prediction (such as predicting consumers' purchasing tendencies, product choices, consumption amounts, etc.), health behavior prediction (such as disease risk prediction, etc.), financial investment behavior prediction, etc. This application does not list them one by one, and the specific usage scenarios can be set according to actual needs.

[0076] The following will illustrate multimodal data and user behavior prediction with reference to specific embodiments, for example:

[0077] When the model is a behavior prediction model, if it is used to predict the lending behavior of users in the financial field, the audio data in the multimodal behavior data at this time can be understood as including the audio generated by the user in the video conversation (used to capture the user's voice emotions, intonation, etc.), the video data can be understood as including the video generated by the user in the video conversation (used to capture the user's expression, line of sight, etc.), the text data can be understood as including the text input content of the user in the loan application (such as questions, answers, etc.), and the time series data can be understood as including the user's historical lending records (such as application time, amount, repayment behavior, etc. time series data). At this time, the text data and time series data include features related to the lending behavior, and contextual information can be captured based on the text data and time series data; further, the behavior prediction model can be trained based on these multimodal data, and then the multimodal data corresponding to the user will be obtained. The behavioral prediction model trained by the user is used to input the behavioral data of the state behavior to predict the user's lending behavior, wherein this lending behavior can be understood as including whether the user will apply for a loan (such as the output result is "the user will apply for a loan" or "the user will not apply for a loan"), whether the user has a loan demand (such as the output result is "the user has a loan demand" or "the user has no loan demand"), whether the user meets the loan qualifications or loan conditions (such as the output result is "qualified" or "not qualified" for the loan, or "low-risk" or "high-risk" borrower), the user's corresponding loan amount or interest rate (such as the output result is "loan amount: XXX yuan"), the user's lending decision (such as the output result is "recommended loan amount: XXX yuan, annual interest rate: X%), the user's corresponding risk level (such as the output result is a numerical value, the higher the numerical value, the greater the risk level), etc. Some or all of these are not listed one by one in this application. Furthermore, based on the predicted user's lending behavior, corresponding language (such as customer service language, etc.), financial products, etc. can also be recommended to the user, which are not limited in this application.

[0078] When the model is an emotion recognition model, it can be used to predict the user's emotional information. For example, in the financial field, the multimodal behavioral data at this time can be understood as including video data, audio data, text data, time series data, etc. generated by the interaction between the user and the agent (such as customer service personnel, virtual agents, etc.); in the field of mental health monitoring, the multimodal behavioral data at this time can be understood as including video data, audio data, text data, time series data, etc. generated based on the interaction with the user during psychological counseling and treatment; and then the emotion recognition model can be trained based on these multimodal data, and then the obtained multimodal behavior data corresponding to the user can be input into the trained emotion recognition model, and the emotional information corresponding to the user can be output, and then in the financial field, based on the predicted user's emotional information, corresponding customer service scripts, financial products, etc. can be recommended to the user, and in the field of mental health monitoring, mental health counseling and treatment assistance support can be provided to mental health therapists, and in the field of medical technology, medical assistance support can be provided to medical staff, etc., which are not listed one by one in this application.

[0079] When the model is a recommendation model, it can be used to recommend personalized content to users. For example, in the financial field, the multimodal behavior data at this time can be understood as including video data, audio data, text data, time series data, etc. generated by the interaction between users and agents; in the field of education and training, the multimodal behavior data at this time can be understood as including video data, audio data, text data, time series data, etc. generated by users such as students and teachers; in the metaverse field such as virtual pet adoption and virtual fitting, the multimodal behavior can be understood as including video data, audio data, text data, time series data, etc. generated by users during the interaction process. , text data, time series data, etc.; and then the recommendation model can be trained based on these multimodal data, and then the obtained multimodal behavior data corresponding to the user can be input into the trained recommendation model. For the financial field, personalized content such as financial products and sales talk that match the user can be output. In the field of education and training, personalized content such as learning resources that match students or teaching content that matches the classroom can be output. In the field of virtual pet adoption, virtual pets that match the user can be output. In the field of virtual fitting, clothing, accessories and other content that match the user can be output, and so on. This application does not list them one by one.

[0080] From the above background technology, it can be seen that currently, single-modal data is mainly used for prediction, or multi-modal data is simply fused for prediction. However, a single modality is difficult to fully reflect the multi-dimensional characteristics of user behavior. For example, when predicting user lending behavior, a single modality is difficult to capture the user's emotions, intentions and other information in different situations, so it is difficult to identify the potential risks corresponding to the user that are more sensitive in the financial field, which can easily lead to low accuracy of the prediction results; and simple multi-modal fusion, such as directly splicing the features of different modalities, is a simple fusion process that is then input into a general prediction model for prediction. Since the features of each modality have large differences in structure and semantics, simple fusion processing is difficult to fully utilize the information of each modality, resulting in information loss, and the fusion quality cannot be guaranteed, and thus the accuracy of subsequent model predictions cannot be guaranteed.

[0081] Therefore, in order to improve the accuracy of model prediction, the embodiment of the present application provides a model training method to train the model. The model training method of the present application is explained below with reference to the accompanying drawings. It should be noted that this embodiment is mainly explained by taking multimodal data including first modal data and second modal data as an example. Specifically:

[0082] See Figure 1 , Figure 1 This is a flow chart of a model training method provided in an embodiment of the present application. The method is applied to a model training device, and the embodiment of the method includes but is not limited to steps S101-S106:

[0083] S101. Based on the first eigenvector of the first modal data and the second eigenvector of the second modal data, calculate a first attention processing result of the first modal data on the second modal data and a second attention processing result of the second modal data on the first modal data.

[0084] In an embodiment of the present application, taking a training sample as an example, the training sample includes multimodal data, and the multimodal data includes first modal data and second modal data, that is, the first modal data is any one of the multimodal data, and the second modal data is another one of the multimodal data, and the first modal data and the second modal data correspond to different modalities; and the multimodal data can be the user's multimodal behavior data, at least including the user's audio data, video data, time series data, and text data, which will not be repeated here. For details, please refer to the corresponding explanations in the above embodiments.

[0085] First, for each modal data in the multimodal data, feature extraction can be performed on each modal data through a feature extraction module to obtain a feature vector of each modal data. For example, feature extraction can be performed on the first modal data through a feature extraction module to obtain a first feature vector, and feature extraction can be performed on the second modal data through a feature extraction module to obtain a second feature vector. In order to ensure the quality of feature extraction of different modal data, an encoder matching each modal data can be used to extract features for each modal data, that is, the feature extraction module includes an encoder matching each modal data. For example, a video encoder such as (Vision Transformer, ViT) is used to extract features from video data, such as extracting dynamic visual features of video frames to obtain feature vectors of video data; an audio encoder such as a convolutional neural network is used to extract audio features of audio data such as speech, emotion and other information to obtain feature vectors of audio data; a text encoder such as a pre-trained language model (Bidirectional Encoder Representations from Transformers, BERT) and (Generative Pre-trained Transformer, GPT) are used to extract features from text data to obtain feature vectors of text data; a time series encoder such as a time series-based Transformer structure is used to extract features of time series data to obtain feature vectors of time series data.

[0086] Then, after obtaining the first eigenvector of the first modal data and the second eigenvector of the second modal data, the first attention processing result of the first modal data on the second modal data can be determined based on the first eigenvector and the second eigenvector, and the second attention processing result of the second modal data on the first modal data can be determined. For example, the first attention processing result can be expressed by formula (1), and the second attention processing result can be expressed by formula (2):

[0087]

[0088] Among them, Q P1 represents the query corresponding to the first modal data (denoted as P1), represents the transposition of the key corresponding to the second modal data (denoted as P2), V P2 Indicates the value corresponding to the second modal data, d K represents the scaling factor, softmax represents the activation function; Q P2 Indicates the query corresponding to the second modal data, Represents the transposition of the key corresponding to the first modal data, V P1Indicates the value corresponding to the first modal data. Similarly, the attention processing result of any modal data in the multimodal data on the other modal data can be determined later, and the principle will not be repeated here.

[0089] It should be explained that when performing attention processing on two modal data, it is necessary to map the feature vectors of each modal data through a linear projection matrix, that is, mapping the query (Query, Q), key (Key, K), and value (Value, V) to the same dimension, so that the dimensions are consistent before performing attention processing. It can be seen that by using independent encoder modules for feature extraction of different modal data, the problem of heterogeneity of multimodal data is overcome, the extraction and processing of multimodal data is improved, and the foundation is laid for achieving high-quality feature fusion.

[0090] S102. Fuse the first eigenvector and the first attention processing result to obtain a first fused eigenvector corresponding to the first modal data.

[0091] Exemplarily, first obtain the fourth preset weight corresponding to the first modal data, and obtain the fifth preset weight between the first modal data and the second modal data; then weighting is performed based on the fourth preset weight corresponding to the first modal data and the first eigenvector to obtain the twelfth eigenvector corresponding to the first modal data, and weighting is performed based on the fifth preset weight and the first attention processing result to obtain the thirteenth eigenvector corresponding to the first modal data; then, based on the twelfth eigenvector and the thirteenth eigenvector corresponding to the first modal data, determine the first fused eigenvector corresponding to the first modal data, such as performing splicing, summing or averaging on the twelfth eigenvector and the thirteenth eigenvector corresponding to the first modal data to obtain the first fused eigenvector. This application does not limit the fusion processing of the twelfth eigenvector and the thirteenth eigenvector.

[0092] In an optional embodiment, when obtaining the first fused feature vector corresponding to the first modal data, the first weight corresponding to the first modal data can also be determined first. For example, the first weight can be preset, and this application does not limit the specific value. Alternatively, the first weight can be determined based on the first attention processing result. Specifically, the norm of the first attention processing result is determined, such as the L1 norm or the L2 norm, etc., which is not limited in this application; then, based on the norm of the first attention processing result, the first weight is determined. For example:

[0093] If the only data in the multimodal data that is different from the first modal data is the second modal data, then the first weight is the ratio of the norm of the first attention processing result to the norm of the first attention processing result, that is, the weight is 1 at this time. If there are multiple data in the multimodal data that are different from the first modal data, then there are multiple first attention processing results corresponding to the first modal data (one-to-one corresponding to the multiple modal data in the multimodal data that are different from the first modal data, the principle will not be repeated here), then the norms of the multiple first attention processing results are summed to obtain the sum of the norms corresponding to the first modal data; then, based on the first attention processing result and the sum of the norms corresponding to the first modal data, the first weight between the first modal data and the second modal data is determined. Similarly, the first weight between the first modal data and each modal data in the multiple modal data that are different from the first modal data can be determined, that is, the number of first weights corresponding to the first modal data is also multiple (one-to-one corresponding to the multiple modal data in the multimodal data that are different from the first modal data, the principle will not be repeated here).

[0094] For example, taking multimodal data including video data (denoted as P1), audio data (denoted as P2), text data (denoted as P3) and time series data (denoted as P4) as an example, assuming that the first modal data is P1, the second modal data is P2, the third modal data is P3, and the fourth modal data is P4, then for P1, we first determine the norm of the first attention processing result of P1 on P2 (denoted as U1), the norm of the first attention processing result of P1 on P3 (denoted as U2), the norm of the first attention processing result of P1 on P4 (denoted as U3); then sum U1, U2, and U3 to obtain the sum of the norms corresponding to P1 (denoted as U4); then the first weight between P1 and P2 can be determined based on the ratio of U1 and U4, and the first weight between P1 and P3 can be determined based on the ratio of U2 and U4, and the first weight between P1 and P4 can be determined based on the ratio of U3 and U4. At this time, there are three first weights corresponding to P1, namely the first weight between P1 and P2, the first weight between P1 and P3, and the first weight between P1 and P4.

[0095] Then, based on the first weight and the first attention processing result corresponding to the first modal data, a weighted processing is performed, for example, to obtain a fifth eigenvector corresponding to the first modal data. Of course, if there are multiple data different from the first modal data in the multimodal data, that is, there are multiple first attention processing results and first weights corresponding to the first modal data, and the multiple first attention processing results correspond to the multiple first weights one-to-one, then based on each first weight corresponding to the first modal data and the first attention processing result corresponding to each first weight, a weighted processing is performed, for example, to obtain multiple fifth eigenvectors corresponding to the first modal data. At this time, the multiple fifth eigenvectors correspond one-to-one to the multiple modal data different from the first modal data in the multimodal data.

[0096] Then, based on the fifth eigenvector corresponding to the first modal data and the first eigenvector of the first modal data, a first fused eigenvector corresponding to the first modal data is obtained. For example, a first preset weight corresponding to the first modal data is obtained; then, based on the first eigenvector of the first modal data and the first preset weight, such as weighted processing, a sixth eigenvector corresponding to the first modal data is obtained; then, based on the fifth eigenvector and the sixth eigenvector corresponding to the first modal data, a fusion processing such as splicing, summing or averaging is performed to obtain the first fused eigenvector corresponding to the first modal data. Of course, if there are multiple data in the multimodal data that are different from the first modal data, that is, there are multiple fifth eigenvectors corresponding to the first modal data, then the first fused eigenvector corresponding to the first modal data is obtained based on the multiple fifth eigenvectors and sixth eigenvectors corresponding to the first modal data, such as splicing, summing or averaging.

[0097] S103: Fuse the second eigenvector and the second attention processing result to obtain a second fused eigenvector corresponding to the second modal data.

[0098] In an embodiment of the present application, the principle of obtaining the second fused feature vector corresponding to the second modality data is similar to the principle of obtaining the first fused feature vector corresponding to the first modality data in the above step S102. The parameters can be adaptively adjusted and replaced, which will not be repeated here.

[0099] It should be noted that in the above embodiment, by performing attention processing on the first modal data and the second modal data to obtain the attention processing result between the two, the interaction characteristics between the cross-modal data can be captured; then the first weight between the cross-modalities is evaluated based on the interaction characteristics between the cross-modalities, that is, the sparsity of the corresponding attention processing results. For example, the norm of the attention results between the cross-modalities is taken. That is, if the more elements that are 0 in the attention processing results between the cross-modalities, the more sparse it is, the lower the importance, and the smaller the corresponding first weight. If the fewer elements that are 0 in the attention processing results between the cross-modalities, the less sparse it is, the higher the importance, and the larger the corresponding first weight. The larger the first weight corresponding to the cross-modal data, the stronger the interactivity between the cross-modal data, and vice versa. Then, based on the norm of the attention processing result, the first weight is determined, the significance of the synergy between the modal data can be effectively captured, and the features between the cross-modal data can be fully fused to ensure the fusion accuracy of the first fusion feature vector and the second fusion feature vector.

[0100] S104 : Determine the similarity between the first modality data and the second modality data based on the first fused feature vector and the second fused feature vector.

[0101] In the embodiments of the present application, the similarity between the first modal data and the second modal data can be determined by cosine similarity principle, Euclidean distance, Pearson correlation coefficient, etc., which is not limited in the present application. For example, the similarity between the first modal data and the second modal data can be obtained by formula (3):

[0102]

[0103] Among them, C ij represents the similarity between the first modality data and the second modality data, represents the first fused feature vector corresponding to the first modality data, The second fused feature vector corresponding to the second modal data is represented by , where the first modal data and the second modal data belong to two modal data in the multimodal data, and i≠j. Similarly, the similarity between any two modal data in the multimodal data can be determined.

[0104] S105 : Fusing the first eigenvector based on the similarity to obtain a third fused eigenvector corresponding to the first modal data.

[0105] In an embodiment of the present application, after determining the similarity between the first modal data and the second modal data, for the first modal data, firstly, based on the first eigenvector of the first modal data and the second eigenvector of the second modal data, a third eigenvector corresponding to the first modal data is determined. For example, the difference between the second eigenvector and the first eigenvector is determined to obtain the third eigenvector corresponding to the first modal data. At this time, the third eigenvector can be understood as the residual of the first eigenvector of the first modal data.

[0106] Then, based on the third eigenvector corresponding to the first modal data and the similarity between the first modal data and the second modal data, the fourth eigenvector corresponding to the first modal data is determined. For example, the fourth eigenvector corresponding to the first modal data is obtained based on the product of the third eigenvector corresponding to the first modal data and the similarity between the first modal data and the second modal data.

[0107] Then, based on the fourth eigenvector corresponding to the first modal data and the first eigenvector of the first modal data, the third fused eigenvector corresponding to the first modal data is determined, such as by summing or concatenating the fourth eigenvector and the first eigenvector, which is not limited in this application. Alternatively, weighting can be performed based on the first eigenvector of the first modal data and the sixth preset weight corresponding to the first modal data (whose value can be the same as or different from the first preset weight) to obtain a weighted eigenvector corresponding to the first modal data; then, based on the weighted eigenvector corresponding to the first modal data and the fourth eigenvector corresponding to the first modal data, a fusion process such as summing or concatenating is performed to obtain the third fused eigenvector corresponding to the first modal data.

[0108] For example, the third fusion feature vector corresponding to the first modal data can be obtained by formula (4):

[0109]

[0110] in, represents the third fused feature vector corresponding to the first modality data, F i Represents the first eigenvector of the first modal data, F j The second eigenvector representing the second modal data, C ij represents the similarity between the first modality data and the second modality data, (F j -F i ) represents the third eigenvector corresponding to the first modal data.

[0111] Of course, the above embodiment is explained using the first modal data and the second modal data as an example, that is, at this time, the only data in the multimodal data that is different from the first modal data is the second modal data; however, if there are multiple data in the multimodal data that are different from the first modal data (in order to simplify the description, they are referred to as multiple fifth modal data, and at this time, the multiple fifth modal data include the above second modal data), then by the same token, based on the first eigenvector and the eigenvectors of the multiple fifth modal data (the determination principle is similar to the determination principle of the above first eigenvector or the second eigenvector, and will not be repeated here), multiple third eigenvectors corresponding to the first modal data can be determined. The principle will not be repeated here. At this time, the multiple third eigenvectors and the multiple fifth modal data correspond one to one. Or optionally, for multiple fifth modal data, the similarity between the first modal data and each fifth modal data can be obtained in the same way, that is, multiple similarities corresponding to the first modal data can be obtained, and at this time, the multiple similarities correspond one-to-one to the multiple fifth modal data; then, based on the similarity between the first modal data and each fifth modal data, the multiple fifth modal data can be screened to obtain multiple sixth modal data, for example, at this time, the similarity corresponding to each sixth modal data is greater than or equal to the first threshold; then, based on the first eigenvector of the first modal data and the eigenvector of each sixth modal data (the determination principle is similar to the determination principle of the above-mentioned first eigenvector or second eigenvector, and will not be repeated here), multiple third eigenvectors corresponding to the first modal data are determined, and the principle will not be repeated here, wherein at this time, the multiple third eigenvectors correspond one-to-one to the multiple sixth modal data.

[0112] Then, by the same token, based on the multiple third eigenvectors and multiple similarities corresponding to the first modal data, multiple fourth eigenvectors corresponding to the first modal data can be obtained. At this time, these multiple fourth eigenvectors correspond one-to-one to the multiple fifth modal data, and the multiple similarities correspond one-to-one to the multiple fifth modal data.

[0113] Then, based on the multiple fourth eigenvectors corresponding to the first modal data and the first eigenvector of the first modal data, the third fused eigenvector corresponding to the first modal data can be determined. For example, the multiple fourth eigenvectors corresponding to the first modal data and the first eigenvector of the first modal data can be fused by summing, splicing, or other fusion processing to obtain the third fused eigenvector corresponding to the first modal data. Alternatively, in a similar manner, the first eigenvector of the first modal data and the sixth preset weight corresponding to the first modal data can be weighted to obtain a weighted eigenvector corresponding to the first modal data; then, based on the weighted eigenvector corresponding to the first modal data and the multiple fourth eigenvectors corresponding to the first modal data, a fusion processing such as summing or splicing can be performed to obtain the third fused eigenvector corresponding to the first modal data. It should be noted that this application does not limit the specific method for determining the third fused eigenvector.

[0114] For example, if the number of multimodal data is N, and N ≥ 2, that is, the multimodal data includes not only the first modal data and the second modal data, but also other modal data. At this time, the number of data different from the first modal data in the multimodal data, that is, multiple fifth modal data (including the second modal data), is N-1. At this time, the third fused feature vector corresponding to the first modal data can be obtained by formula (5):

[0115]

[0116] in, represents the third fused feature vector corresponding to the first modality data; F i Represents the first eigenvector of the first modal data, F y The principle of determining the eigenvector of the yth modal data is similar to that of the first eigenvector or the second eigenvector, and will not be repeated here; iy The principle of the similarity between the first modal data and the yth modal data is similar to that of the similarity between the first modal data and the second modal data, which will not be repeated here; (F y -F i ) represents the third eigenvector corresponding to the first modal data and the yth modal data; the yth modal data is any one of the multiple fifth modal data.

[0117] S106 , fusing the second eigenvector based on the similarity to obtain a fourth fused eigenvector corresponding to the second modal data.

[0118] In an embodiment of the present application, the principle of obtaining the fourth fused feature vector corresponding to the second modal data is similar to the principle of obtaining the third fused feature vector corresponding to the first modal data in the above embodiment. Only adaptive parameter adjustment and replacement are required, which will not be repeated here.

[0119] It should be noted that in the above embodiment, the similarity between cross-modal data is determined by the first fused feature vector corresponding to the first modal data and the second fused feature vector corresponding to the second modal data. Since the first fused feature vector and the second fused feature vector cover the interaction characteristics between the cross-modal data and the significance of the synergy between the cross-modal data, the similarity between the cross-modal data determined based on the first fused feature vector corresponding to the first modal data and the second fused feature vector corresponding to the second modal data is more accurate. Then, based on the first eigenvector of the first modal data and the second eigenvector of the second modal data, the third eigenvector corresponding to the first modal data is determined. This third eigenvector captures the unique features between the first modal data and the second modal data. Then, based on the third eigenvector corresponding to the first modal data and the similarity between the first modal data and the second modal data, a fusion is performed to determine the fourth eigenvector corresponding to the first modal data, thereby obtaining the third fused eigenvector. While achieving deep fusion of the features of multimodal data, it can also suppress the interference and noise brought by the modal data with low correlation with the first modal data, thereby improving the stability of the model. The greater the correlation between the two modal data, the more the features of the highly correlated modal data can be "amplified" by weighting, thereby improving the contribution of the features of the highly correlated modal data in subsequent predictions, thereby improving the accuracy of the model prediction.

[0120] S107: Train the model based on the third fused feature vector and the fourth fused feature vector.

[0121] After obtaining the third fused feature vector corresponding to the first modal data and the fourth fused feature vector corresponding to the second modal data based on the above embodiment, prediction can be performed based on the third fused feature vector and the fourth fused feature vector to obtain a prediction result corresponding to the training sample. For example, the third fused feature vector and the fourth fused feature vector are fused, such as by concatenation or summation, to obtain a fifth fused feature vector; the fifth fused feature vector is then input into a prediction module (such as a large language model GPT, T5, etc., which is not limited in this application), and a prediction result corresponding to the training sample (including the first modal data and the second modal data) is output. The prediction result here includes at least two ways of understanding:

[0122] (1) The prediction result can be a numerical value, indicating that the prediction result of the model is a numerical value. It can also be called the "prediction value" of the model for the training sample. For example, in the scenario explained in the above embodiment for predicting the risk level of the user, the risk level is represented by a numerical value. The larger the numerical value, the greater the risk level. Subsequently, the loss calculation can be directly performed based on the "prediction value" and the "true value" corresponding to the training sample.

[0123] (2) The prediction result can be a probability, whether it is a discriminative classification scenario or a generative scenario, and the final prediction result of the model for the training sample can be determined based on the probability. For example, in a discriminative classification scenario, the probability includes the probability of the training sample under each classification category, and the classification category with the largest probability is selected as the prediction result; in a generative scenario, the probability includes the probability corresponding to each word in the preset vocabulary, and the word with the largest probability can be selected as the element of the output sequence until a complete output sequence is obtained; at this time, the probability can be called "predicted probability", and the loss calculation can be performed based on the "predicted probability" and "true probability".

[0124] Then, based on the prediction result, the seventh eigenvector corresponding to the first modal data and the eighth eigenvector corresponding to the second modal data, the first loss is determined. The seventh eigenvector is determined based on one or more of the first eigenvector, the first fused eigenvector and the third fused eigenvector. For example, the seventh eigenvector can be any one of the first eigenvector, the first fused eigenvector or the third fused eigenvector, or it can be obtained based on the fusion processing of the first eigenvector and the first fused eigenvector (such as splicing, weighting, etc.), or it can be obtained based on the fusion processing of the first eigenvector and the third fused eigenvector (such as splicing, weighting, etc.), or it can be obtained based on the fusion processing of the first eigenvector and the first fused eigenvector and the third fused eigenvector (such as splicing, weighting, etc.), or it can be obtained based on the fusion processing of the first eigenvector and the first fused eigenvector and the third fused eigenvector (such as splicing, weighting, etc.). This application does not limit the specific way of obtaining the seventh eigenvector. Similarly, the eighth eigenvector is determined based on one or more of the second eigenvector, the second fused eigenvector and the fourth fused eigenvector. The principle is similar to the determination principle of the seventh eigenvector and will not be repeated here.

[0125] Exemplarily, the second loss can be determined based on the prediction result. Taking the above two explanations of the prediction result as an example, for the above explanation (1), since the prediction result is a numerical value, representing the prediction result of the model, the second loss can be obtained based on the regression loss function, such as the mean square error loss function or the mean absolute error loss function, etc., which is not limited in this application. For example, the second loss is obtained by formula (6):

[0126]

[0127] Among them, L POL Represents the second loss, E[] represents the expectation, at this time Represents the prediction result, that is, the predicted value, and y represents the true value corresponding to the training sample, that is, the true label corresponding to the training sample.

[0128] For the above explanation (2), the second loss can be obtained by the cross entropy loss function. For example, the second loss is obtained by formula (7):

[0129]

[0130] Among them, L POL Represents the second loss. For the above discriminant or generative scenarios, P k Indicates the prediction probability corresponding to the k-th classification category or the k-th word in the preset vocabulary in the prediction result, It represents the true probability of the k-th classification category corresponding to the training sample or the k-th word in the preset vocabulary, that is, the true label corresponding to the training sample, and V is the number of classification categories or the number of words in the preset vocabulary.

[0131] Then, based on the seventh eigenvector and the eighth eigenvector, the third loss is determined. For example, based on the seventh eigenvector and the eighth eigenvector, the average value of the characteristic distribution of the first modal data and the second modal data is determined. Then, based on the average value of the characteristic distribution of the first modal data and the second modal data, the seventh eigenvector and the eighth eigenvector, the fifth loss corresponding to the first modal data and the second modal data is determined. Then, based on the fifth loss, the third loss can be determined. For example, if the multimodal data only includes the first modal data and the second modal data, the third loss is the fifth loss. If the multimodal data includes not only the first modal data and the second modal data, but also other modal data, the fifth loss corresponding to any two modal data in the multimodal data can be determined based on the above principle. For example, assuming that the first modal data is the i-th modal data and the second modal data is the j-th modal data, the fifth loss corresponding to the first modal data and the second modal data can be obtained by formula (8):

[0132]

[0133] Among them, L JS (F i ′,F j ′) represents the fifth loss between the first modality data and the second modality data, D KL represents (Kullbac k-Leibler Divergence, KL divergence), F i ′ represents the seventh eigenvector corresponding to the first modal data, F j ′ represents the eighth eigenvector corresponding to the second modal data, Represents the average value of the feature distribution of the first modality data and the second modality data.

[0134] Furthermore, the third loss can be obtained based on the fifth loss between any two modal data in the multimodal data, such as summation processing. For example, the third loss is obtained by formula (9):

[0135]

[0136] Among them, the uth modal data and the vth modal data belong to any two modal data in the multimodal data, N is the number of multimodal data, L JS (F u ′,F v ′) represents the fifth loss between the u-th modal data and the v-th modal data.

[0137] Then, the preset weight corresponding to the first modal data can be obtained, and the third preset weight corresponding to the second modal data can be obtained. This application does not limit the value. Then, based on the prediction result, the seventh eigenvector, the eighth eigenvector, the second preset weight and the third preset weight, the fourth loss is determined. For example, the first correlation coefficient between the prediction result and the seventh eigenvector is determined, and the second correlation coefficient between the prediction result and the eighth eigenvector is determined. The correlation coefficient can be determined by the Pearson correlation coefficient (Correlation coefficient, Corr), the canonical correlation analysis coefficient (Canonical Correlation Analysis, CCA), etc., and this application does not limit it.

[0138] Then, based on the first correlation coefficient and the second correlation coefficient, as well as the second preset weight and the third preset weight, the fourth loss is determined. For example, the first correlation coefficient can be negative and then weighted with the second preset weight to obtain the sixth loss corresponding to the first modal data, and the second correlation coefficient can be negative and then weighted with the third preset weight to obtain the seventh loss corresponding to the second modal data; then, based on the sixth loss and the seventh loss, the fourth loss is obtained by, for example, summing, weighting or averaging.

[0139] Of course, if the number of multimodal data is N, and N ≥ 2, the fourth loss can also be obtained by formula (10):

[0140]

[0141] Among them, L MDWL represents the fourth loss, N represents the number of multimodal data, F′ x represents the seventh eigenvector corresponding to any one modal data in the multimodal data, i.e., the u-th modal data (the principle is similar to that of the seventh eigenvector corresponding to the first modal data or the eighth eigenvector corresponding to the second modal data, and will not be repeated here). Represents the prediction results, represents the correlation coefficient between the prediction result and the seventh eigenvector corresponding to the u-th modal data (similar to the principle of the first correlation coefficient or the second correlation coefficient in the above embodiment, which will not be repeated here), w u It represents the second preset weight corresponding to the u-th modal data (the principle is similar to the second preset weight corresponding to the first modal data or the third preset weight corresponding to the second modal data, and will not be repeated here).

[0142] Then, based on the second loss, the third loss and the fourth loss, the first loss is obtained by performing a process such as summing or weighted averaging. Of course, in an optional embodiment, the first loss can also be determined based on part of the second loss, the third loss and the fourth loss. That is, the second loss can be determined based on the prediction result, and then the third loss can be determined based on the seventh eigenvector and the eighth eigenvector. The principle will not be repeated here. Then, the first loss can be obtained by performing a process such as summing or weighted averaging based on the second loss and the third loss. Alternatively, the second loss can be determined based on the prediction result, and then the fourth loss can be determined based on the prediction result, the seventh eigenvector, the eighth eigenvector, the second preset weight and the third preset weight. The principle will not be repeated here. Then, the first loss can be obtained by performing a process such as summing or weighted averaging based on the second loss and the fourth loss. It should be noted that this application does not limit the specific determination principle of the first loss.

[0143] In an optional embodiment, the seventh eigenvector corresponding to the first modal data may include one or more of the first eigenvector, the first fused eigenvector and the third fused eigenvector, and the eighth eigenvector corresponding to the second modal data may include one or more of the second eigenvector, the second fused eigenvector and the fourth fused eigenvector; when the seventh eigenvector and the eighth eigenvector both include corresponding multiple of them, that is, at this time there are multiple seventh eigenvectors and multiple eighth eigenvectors, then based on the prediction results, each seventh eigenvector in the multiple seventh eigenvectors and each eighth eigenvector in the multiple eighth eigenvectors, multiple eighth losses corresponding to the first modal data and multiple ninth losses corresponding to the second modal data are determined, at this time the multiple eighth losses correspond one-to-one to the multiple seventh eigenvectors, and the multiple ninth losses correspond one-to-one to the multiple eighth eigenvectors, and the principle of determining each eighth loss and each ninth loss is similar to the principle of the first loss in the above embodiment, and will not be repeated here); then based on the multiple eighth losses and the multiple ninth losses, such as summing or averaging, the final first loss is obtained.

[0144] Finally, after determining the total loss, i.e., the first loss, the model can be trained based on the first loss, wherein all parameters of the model can be gradient updated based on the first loss, or some parameters of the model can be frozen, and only the unfrozen parameters can be gradient updated, which is not limited in this application. For example, the model of the embodiment of the present application can at least include a feature extraction module, a feature fusion module, and a prediction module, wherein the feature extraction module includes an encoder for matching each modal data, the feature fusion module is used to execute the relevant embodiments corresponding to the above steps S101-S106, which will not be repeated here, and the prediction module is used to predict based on the third fused feature vector and the fourth fused feature vector to obtain a prediction result; of course, optionally, the model can also include an input module, such as can be used for data acquisition, preprocessing, etc., which is not limited in this application; therefore, when updating based on the first loss, at least some or all of the parameters in the input module, feature extraction module, feature fusion module, prediction module, etc. included in the model can be gradient updated, which is not limited in this application.

[0145] It should be noted that in the above embodiment, after obtaining the prediction result, the second loss, the third loss and the fourth loss are calculated, wherein the second loss is used to measure the difference between the prediction result and the true label, that is, to measure the difference between the predicted value and the true value, and the goal of making the prediction result closer to the true label is achieved through gradient updating; the third loss is used to measure the difference between the modal data, and the goal of reducing the difference in features between different modal data is achieved through gradient updating; the fourth loss is used to measure the importance of the modality, by determining the correlation coefficient between the prediction result and the seventh eigenvector and the eighth eigenvector respectively, that is, calculating the correlation or association or causality between the prediction result and the input feature. Because the more accurate the prediction result, the more correlated the prediction result is with the input feature, which means that the prediction result is obtained based on the input feature prediction, and the fourth loss gradient is updated to achieve the goal of obtaining an accurate prediction result based on the input feature; then, based on the multi-objective loss function, that is, the second loss, the third loss, and the fourth loss, the first loss is obtained to perform gradient update training on the model, which can improve the quality of multimodal fusion and enhance the model's adaptability to different modal data and prediction accuracy.

[0146] It should be noted that since each training sample in the embodiment of the present application includes multimodal data, the number of modal data in the multimodal data is N, N≥2, and when N=2, it includes the first modal data and the second modal data; when N>2, the multimodal data includes not only the first modal data and the second modal data, but also other modal data. The embodiment of the present application is mainly illustrated by taking the multimodal data including 2 modal data as an example. If N>2, any one of the N modal data (assuming the xth modal data) needs to be processed with each of the remaining modal data in the N modal data except the xth modal data, and the processing operation between the first modal data and the second modal data in the above embodiment is performed to obtain the final training loss.

[0147] For example, to understand this, see Figure 2 , Figure 2 This is a schematic diagram of a model training method provided in an embodiment of the present application. Figure 2 As shown, a training sample includes multimodal data including video data, audio data, text data, and time series data as an example for explanation, that is, at this time N>2, and accordingly, the feature extraction module includes a video encoder matching the video data, an audio encoder matching the audio data, a text encoder matching the text data, and a time series encoder matching the time series data, which will not be repeated here.

[0148] First, the training samples are input into the input module of the model, such as for preprocessing; then, the video data is subjected to feature extraction by the video encoder in the feature extraction module to obtain the feature vector of the video data, and the audio data is subjected to feature extraction by the audio encoder to obtain the feature vector of the audio data, and the text data is subjected to feature extraction by the text encoder to obtain the feature vector of the text data, and the time series data is subjected to feature extraction by the time series encoder to obtain the feature vector of the time series data.

[0149] Then, for any modal data in this multimodal data, assuming it is video data, the attention processing result of the video data on the audio data is calculated through the feature vector of the video data and the feature vector of the audio data. Similarly, the attention processing result of the video data on the text data and the attention processing result of the video data on the time series data are obtained.

[0150] Then, the feature fusion module is used to fuse the feature vectors of the video data and the attention processing results of the video data on the audio data, the attention processing results on the text data, and the attention processing results on the time series data, respectively, to obtain the feature vectors corresponding to the video data (the number is 3 at this time, namely feature vector 1, feature vector 2, and feature vector 3, wherein the principle is similar to the principle of the fifth feature vector mentioned above, and no further details are given); then, based on the feature vector of the video data, the feature vector 1, feature vector 2, and feature vector 3 corresponding to the video data, the fused feature vector 1 corresponding to the video data can be determined. Similarly, the fused feature vector 2 corresponding to the text data, the fused feature vector 3 corresponding to the audio data, and the fused feature vector 4 corresponding to the time series data can be obtained respectively. The principle is similar to the principle of the first fused feature vector or the second fused feature vector mentioned above, and no further details are given).

[0151] Then, the feature fusion module determines the similarity between the video data and the data different from the video data in the multimodal data (i.e., audio data, text data, and time series data) based on the fused feature vector 1 corresponding to the video data, the fused feature vector 2 corresponding to the text data, the fused feature vector 3 corresponding to the audio data, and the fused feature vector 4 corresponding to the time series data. Assume that they are similarity 1, similarity 2, and similarity 3 respectively. Similarly, the similarity between any modal data and the modal data different from it in the multimodal data can be determined. The principle will not be repeated here.

[0152] Then, the feature extraction module obtains the fused feature vector 5 corresponding to the video data based on the feature vector, similarity 1, similarity 2 and similarity 3 of the video data. Similarly, the fused feature vector 6, fused feature vector 7 and fused feature vector 8 corresponding to the audio data, text data and time series data respectively can be obtained. The principle is similar to that of the third fused feature vector and the fourth fused feature vector mentioned above, and will not be repeated here.

[0153] The feature fusion module can then fuse the fused feature vectors corresponding to each modal data, i.e., fused feature vector 5, fused feature vector 6, fused feature vector 7, and fused feature vector 8, and then input them into the prediction module after processing such as summation and splicing to obtain the prediction result; then based on the prediction result and the feature vector 4 corresponding to each modal data (the principle is similar to the determination principle of the above-mentioned seventh feature vector and eighth feature vector, and will not be repeated here), determine the first loss, and the specific principle will not be repeated here; then based on the first loss, all or part of the parameters of the model are updated, which is not limited in this application.

[0154] It should be noted that the fusion steps performed by the feature fusion module mainly include two stages of feature fusion. Figure 2 Based on the examples, see Figure 3 , Figure 3 A schematic diagram of another model training method provided in an embodiment of the present application.

[0155] like Figure 3 As shown, in the first feature fusion stage, the feature fusion module mainly executes the embodiments corresponding to the above steps S101-S103, that is, after the feature fusion module obtains the feature vector of each modal data from the feature extraction module (this embodiment is illustrated by taking the feature vector of video data, the feature vector of audio data, the feature vector of text data, and the feature vector of time series data as an example), for any modal data in the multimodal data, assuming it is video data, the attention processing result of the video data on the audio data is calculated through the feature vector of the video data and the feature vector of the audio data. Similarly, the attention processing result of the video data on the text data and the attention processing result of the video data on the time series data are obtained.

[0156] Then, the feature fusion module is used to fuse the feature vectors of the video data and the attention processing results of the video data on the audio data, the attention processing results on the text data, and the attention processing results on the time series data, respectively, to obtain the feature vectors corresponding to the video data (the number is 3 at this time, namely feature vector 1, feature vector 2, and feature vector 3, where the principle is similar to the principle of the fifth feature vector mentioned above, and no further details are given). Then, based on the feature vector of the video data, the feature vector 1, feature vector 2, and feature vector 3 corresponding to the video data, the fused feature vector 1 corresponding to the video data can be determined. Similarly, the fused feature vector 2 corresponding to the text data, the fused feature vector 3 corresponding to the audio data, and the fused feature vector 4 corresponding to the time series data can be obtained respectively. The principle is similar to the principle of the first fused feature vector or the second fused feature vector mentioned above, and no further details are given).

[0157] After obtaining the fused feature vectors corresponding to each modal data, namely, fused feature vector 1, fused feature vector 2, fused feature vector 3, and fused feature vector 4, the feature fusion module starts the second stage of feature fusion. In the second feature fusion stage, the feature extraction module mainly executes steps S104-S106, that is, the feature fusion module determines the similarity between the video data and the data different from the video data in the multimodal data (i.e., audio data, text data, and time series data) based on the fused feature vector 1 corresponding to the video data, the fused feature vector 2 corresponding to the text data, the fused feature vector 3 corresponding to the audio data, and the fused feature vector 4 corresponding to the time series data. It is assumed that they are similarity 1, similarity 2, and similarity 3 respectively. Similarly, the similarity between any modal data and the modal data different from the any modal data in the multimodal data can be determined, and the principle will not be repeated here. Then, the feature extraction module obtains the fused feature vector 5 corresponding to the video data based on the feature vector, similarity 1, similarity 2 and similarity 3 of the video data. Similarly, the fused feature vector 6, fused feature vector 7 and fused feature vector 8 corresponding to the audio data, text data and time series data respectively can be obtained. The principle is similar to that of the third fused feature vector and the fourth fused feature vector mentioned above, and will not be repeated here.

[0158] After two stages of feature fusion, the feature fusion module can obtain a fused feature vector based on the fused feature vector corresponding to each modal data, i.e., fused feature vector 5, fused feature vector 6, fused feature vector 7, and fused feature vector 8, such as splicing, summation, and other fusion processing; then the fused feature vector is input into the prediction module to obtain a prediction result, and then based on the prediction result and the feature vector 4 corresponding to each modal data (the principle is similar to the determination principle of the above-mentioned seventh feature vector and eighth feature vector, which will not be repeated here), the first loss is determined, and the specific principle will not be repeated here; then all or part of the parameters of the model are updated based on the first loss, which is not limited in this application.

[0159] In an alternative embodiment, see Figure 4 , Figure 4 A flow chart of another model training method provided in an embodiment of the present application. The method is applied to a model training device, and includes but is not limited to steps S401-S405:

[0160] S401: Calculate, based on a first eigenvector of first modal data and a second eigenvector of second modal data, a first attention processing result of the first modal data on the second modal data and a second attention processing result of the second modal data on the first modal data;

[0161] S402: Determine a first weight corresponding to the first modal data based on a first attention processing result of the first modal data on the second modal data, and determine a second weight corresponding to the second modal data based on a second attention processing result of the second modal data on the second modal data;

[0162] The determination principle of the first weight and the second weight may refer to the corresponding explanation of the above embodiment and will not be repeated here.

[0163] S403: obtaining a fifth eigenvector corresponding to the first modal data based on the first weight and the first attention processing result, and obtaining a ninth eigenvector corresponding to the second modal data based on the second weight and the second attention processing result;

[0164] The determination principle of the ninth eigenvector is similar to that of the fifth eigenvector and will not be repeated here.

[0165] S404: Obtain a first fused feature vector corresponding to the first modal data based on the fifth eigenvector and the first eigenvector of the first modal data, and obtain a second fused feature vector corresponding to the second modal data based on the ninth eigenvector and the second eigenvector of the second modal data;

[0166] S405: Train the model based on the first fused feature vector and the second fused feature vector.

[0167] It should be noted that the principles of steps S401 to S405 can be referred to above. Figure 1 The explanation of the principles of steps S101-S107 in the embodiment can achieve the same technical effect, so they are not repeated here.

[0168] In an alternative embodiment, see Figure 5 , Figure 5 A flow chart of another model training method provided in an embodiment of the present application. The method is applied to a model training device, and the method includes but is not limited to steps S501-S504:

[0169] S501: Based on a first eigenvector of the first modal data and a second eigenvector of the second modal data, determine a third eigenvector corresponding to the first modal data, and determine a tenth eigenvector corresponding to the second modal data;

[0170] The determination principle of the tenth eigenvector is similar to that of the third eigenvector and will not be described in detail.

[0171] S502: Determine a fourth eigenvector corresponding to the first modal data based on the third eigenvector and the similarity between the first modal data and the second modal data, and determine an eleventh eigenvector corresponding to the second modal data based on the ninth eigenvector and the similarity;

[0172] The determination principle of the eleventh eigenvector is similar to that of the fourth eigenvector and will not be described in detail.

[0173] S503: Determine a third fused eigenvector corresponding to the first modal data based on the fourth eigenvector and the first eigenvector, and determine a fourth fused eigenvector corresponding to the second modal data based on the tenth eigenvector and the second eigenvector;

[0174] S504: Train the model based on the third fused feature vector and the fourth fused feature vector.

[0175] It should be noted that the principles of steps S501-S504 can refer to the above Figure 1 The explanation of the principles of steps S101-S105 in the embodiment can achieve the same technical effect, so they are not repeated here.

[0176] Furthermore, after the model is trained based on the model training method of the above embodiment and a trained model is obtained, application reasoning can be performed based on the trained model, for example, Figure 6 , Figure 6 This is a flow chart of an information prediction method provided in an embodiment of the present application. The method is applied to an information prediction device, and the method includes but is not limited to the following steps S601-S602:

[0177] S601: Acquire multimodal data of the user.

[0178] S602: Use the model to predict information based on multimodal data.

[0179] The model in step S602 is obtained by Figure 1-Figure 5 The model training method in the embodiment is obtained by training and will not be described in detail here.

[0180] Exemplarily, after obtaining the multimodal data corresponding to the user, the model fuses the feature vector of the modal data 1 in the multimodal data and the attention processing result of the modal data 1 on the modal data 2 to obtain the sixth fused feature vector corresponding to the modal data 1; then, based on the feature vector of the modal data 2 in the multimodal data and the attention processing result of the second modal data 2 on the modal data 1, the model fuses the feature vector of the modal data 2 in the multimodal data and the attention processing result of the second modal data 2 to obtain the seventh fused feature vector corresponding to the modal data 2; then, based on the sixth fused feature vector and the seventh fused feature vector, the similarity between the modal data 1 and the modal data 2 is determined; then, based on the model The similarity between the modal data 1 and the modal data 2 and the feature vector of the modal data 1 are fused to obtain an eighth fused feature vector corresponding to the modal data 1; and the similarity between the modal data 1 and the modal data 2 and the feature vector of the modal data 2 are fused to obtain a ninth fused feature vector corresponding to the modal data 2; then, based on the eighth fused feature vector and the ninth fused feature vector, a prediction result is obtained; and then, based on the prediction result, predicted information can be obtained, such as predicted user behavior, risk level, etc. In the financial field, this behavior can be a lending behavior, such as it can be predicted whether the user will apply for a loan, etc.

[0181] It should be noted that the information prediction device performs Figure 6 Each step in the embodiment can also be executed accordingly Figure 1-Figure 5 The corresponding steps in the embodiments have similar principles and can achieve the same technical effects, so they will not be described one by one here.

[0182] It can be seen that in the field of information prediction, especially in the field of modern finance, the prediction of user lending behavior plays a vital role in risk management and decision support. However, traditional lending behavior prediction models mainly rely on single-modal data sources, such as users' historical lending records or text information. This single-modal approach is often difficult to fully describe the user's behavioral characteristics, resulting in low prediction accuracy and robustness. However, in the embodiments of the present application, multimodal information such as time series information obtained from video, text, audio and historical behavior is deeply integrated to fully capture the user's behavior, emotions and historical characteristics, so that the model can more accurately predict user behavior such as lending behavior. Specifically:

[0183] By performing attention processing on the first modal data and the second modal data to obtain the attention processing results between the two, the interaction characteristics between the cross-modal data can be captured; then the first cross-modal weight is evaluated based on the cross-modal interaction characteristics, that is, the sparsity of the corresponding attention results. For example, the first norm of the cross-modal attention result is taken. That is, if the more elements that are 0 in the cross-modal attention processing result, the more sparse it is, the lower the importance, and the smaller the corresponding first weight is. If the fewer elements that are 0 in the cross-modal attention processing result, the less sparse it is, the higher the importance, and the larger the corresponding first weight is. The larger the first weight corresponding to the cross-modal data, the stronger the interactivity between the cross-modal data, and vice versa. Then the second norm is determined based on the first norm. This second norm is obtained by summing the first norms corresponding to the first modal data and the second modal data. The first weight determined based on the first norm and the second norm can effectively capture the significance of the synergy between the modal data, and then the features between the cross-modal data can be fully fused to ensure the fusion accuracy of the first fusion feature vector.

[0184] Furthermore, taking the first modal data and the second modal data as an example, it should be noted that in the above embodiment, the similarity between cross-modal data is determined by the first fused feature vector corresponding to the first modal data and the second fused feature vector corresponding to the second modal data. Since the first fused feature vector and the second fused feature vector cover the interaction characteristics between the cross-modal data and the significance of the synergy between the cross-modal data, the similarity between the cross-modal data determined based on the first fused feature vector corresponding to the first modal data and the second fused feature vector corresponding to the second modal data is more accurate. Then, based on the first eigenvector of the first modal data and the second eigenvector of the second modal data, the third eigenvector corresponding to the first modal data is determined. This third eigenvector captures the unique features between the first modal data and the second modal data. Then, based on the third eigenvector corresponding to the first modal data and the similarity between the first modal data and the second modal data, a fusion is performed to determine the fourth eigenvector corresponding to the first modal data, thereby obtaining the third fused eigenvector. While achieving deep fusion of the features of multimodal data, it can also suppress the interference and noise brought by the modal data with low correlation with the first modal data, thereby improving the stability of the model. The greater the correlation between the two modal data, the more the features of the highly correlated modal data can be "amplified" by weighting, thereby improving the contribution of the features of the highly correlated modal data in subsequent predictions, thereby improving the accuracy of the model prediction.

[0185] Finally, by calculating the second loss, the third loss and the fourth loss, the second loss is used to measure the difference between the predicted result and the true label, that is, to measure the difference between the predicted value and the true value, and to achieve the goal of making the predicted result closer to the true label through gradient update; the third loss is used to measure the difference between modal data, and to achieve the goal of reducing the difference in features between different modal data through gradient update; the fourth loss is used to measure the importance of the modality, by determining the correlation coefficient between the predicted result and the seventh eigenvector and the eighth eigenvector respectively, that is, calculating the correlation or association or causality between the predicted result and the input feature. Because the more accurate the prediction result, the more correlated the prediction result is with the input feature, which means that the prediction result is obtained based on the input feature prediction, and the fourth loss gradient update is used to achieve the goal of obtaining an accurate prediction result based on the input feature; then, based on the multi-objective loss function, that is, the second loss, the third loss, and the fourth loss, the first loss is obtained to perform gradient update training on the model, which can improve the quality of multimodal fusion and enhance the model's adaptability and prediction accuracy to different modal data.

[0186] Furthermore, the information prediction device can be a terminal device, such as a smartphone, tablet computer, laptop computer, desktop computer, smart TV, desktop computer, smart watch, smart car, or other intelligent terminal, which is not limited in this application. The trained model in the above embodiment can be deployed on the terminal device, and then, after obtaining multimodal behavior data corresponding to the user, the information prediction device predicts the behavior corresponding to the user based on the multimodal behavior data corresponding to the user by deploying the locally trained model. The principle will not be elaborated here.

[0187] Of course, the above information prediction device can also be a server, that is, the trained model is deployed on the server, in this case, refer to Figure 7 , Figure 7 A schematic diagram of an information prediction system provided in an embodiment of the present application.

[0188] Figure 7 The system shown includes an information prediction device and a client; the information prediction device can be a server, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, as well as basic cloud computing services such as big data and artificial intelligence platforms, which are not specifically limited in this application; the client can be a smart phone, tablet computer, laptop computer, desktop computer, smart TV, desktop computer, smart watch, smart car and other smart terminals, but is not limited to these.

[0189] The user can interact with the client, and then the client obtains the multimodal data corresponding to the user, and then the client sends the multimodal data corresponding to the user to the information prediction device, and then the information prediction device uses the trained model to make information predictions based on the multimodal data. Figure 7 The principles of the steps executed by the information prediction device in the embodiment can refer to the corresponding explanations of the above embodiment, which will not be repeated here, and the steps executed by the information prediction device in the above embodiment can also be executed accordingly, which will not be repeated here.

[0190] Similarly, the model training device in the above embodiment can be a terminal device, such as a smart phone, tablet computer, laptop computer, desktop computer, smart TV, desktop computer, smart watch, smart car and other smart terminals, which are not limited in this application. The model to be trained can be deployed on the terminal device, and then after the model training device obtains the training sample, the model training device calculates the first attention processing result of the first modal data on the second modal data and the second attention processing result of the second modal data on the first modal data based on the first eigenvector of the first modal data and the second eigenvector of the second modal data; then the model training device fuses the first eigenvector of the first modal data and the first attention processing result of the first modal data on the second modal data to obtain the first fused eigenvector corresponding to the first modal data, and the second eigenvector of the second modal data and the second attention processing result of the second modal data on the first modal data. The force processing results are fused to obtain a second fused feature vector corresponding to the second modal data; then the model training device determines the similarity between the first modal data and the second modal data based on the first fused feature vector and the second fused feature vector; then the model training device fuses the first feature vector based on the similarity to obtain a third fused feature vector corresponding to the first modal data, and fuses the second feature vector based on the similarity to obtain a fourth fused feature vector corresponding to the second modal data; then the model training device trains the model based on the third fused feature vector and the fourth fused feature vector to obtain a trained model, and the principle will not be repeated here. Or optionally, the model training device can also be used in the above embodiment. Figure 4 or Figure 5 The model training method in is used to train the model to obtain a trained model. The principle will not be described here.

[0191] Of course, the above model training device can also be a server, that is, the model to be trained is deployed on the server. Figure 8 , Figure 8 A schematic diagram of a model training system provided in an embodiment of the present application.

[0192] Figure 8 The system shown includes a model training device and a client; the model training device can be a server, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, as well as basic cloud computing services such as big data and artificial intelligence platforms. This application does not make specific limitations; the client can be a smart phone, tablet computer, laptop computer, desktop computer, smart TV, desktop computer, smart watch, smart car and other smart terminals, but is not limited to this.

[0193] The user can upload the training sample through the client, and then the client sends the training sample to the model training device, and then the model training device performs the following steps: according to the first eigenvector of the first modal data and the second eigenvector of the second modal data, calculate the first attention processing result of the first modal data on the second modal data and the second attention processing result of the second modal data on the first modal data; then determine the first weight corresponding to the first modal data based on the first attention processing result of the first modal data on the second modal data, and determine the second weight corresponding to the second modal data based on the second attention processing result of the second modal data on the second modal data; based on the first weight and the first attention processing result, obtain the fifth eigenvector corresponding to the first modal data, and based on the second weight and the second attention processing result, obtain the ninth eigenvector corresponding to the second modal data; based on the fifth eigenvector and the first eigenvector of the first modal data, obtain the first fused eigenvector corresponding to the first modal data, and based on the ninth eigenvector and the second eigenvector of the second modal data, obtain the second fused eigenvector corresponding to the second modal data; based on the first fused eigenvector and the second fused eigenvector, train the model to obtain a trained model. The principle will not be repeated here. Alternatively, the model training device can also be implemented through the above embodiment. Figure 4 or Figure 5 The model training method in is used to train the model to obtain a trained model. The principle will not be described here.

[0194] It should be noted that Figure 8 The principles of the steps executed by the model training device in the embodiment can refer to the corresponding explanations of the above embodiment, which will not be repeated here, and the steps executed by the model training device in the above embodiment can also be executed accordingly, which will not be repeated here.

[0195] See Figure 9 , Figure 9This is a block diagram of the functional units of a model training device provided in an embodiment of the present application. The model training device 900 includes: a first acquisition unit 901 and a first processing unit 902;

[0196] A first acquiring unit, configured to acquire first modal data and second modal data;

[0197] a first processing unit, configured to calculate, based on a first eigenvector of the first modal data and a second eigenvector of the second modal data, a first attention processing result of the first modal data on the second modal data and a second attention processing result of the second modal data on the first modal data;

[0198] The first processing unit is further configured to fuse the first feature vector and the first attention processing result to obtain a first fused feature vector corresponding to the first modal data;

[0199] The first processing unit is further configured to fuse the second feature vector and the second attention processing result to obtain a second fused feature vector corresponding to the second modal data;

[0200] The first processing unit is further configured to determine a similarity between the first modality data and the second modality data based on the first fused feature vector and the second fused feature vector;

[0201] The first processing unit is further configured to fuse the first feature vector based on the similarity to obtain a third fused feature vector corresponding to the first modal data, and to fuse the second feature vector based on the similarity to obtain a fourth fused feature vector corresponding to the second modal data;

[0202] The first processing unit is further configured to train the model based on the third fused feature vector and the fourth fused feature vector.

[0203] or,

[0204] A first acquiring unit, configured to acquire first modal data and second modal data;

[0205] a first processing unit, configured to calculate, based on a first eigenvector of the first modal data and a second eigenvector of the second modal data, a first attention processing result of the first modal data on the second modal data and a second attention processing result of the second modal data on the first modal data;

[0206] The first processing unit is further configured to determine a first weight corresponding to the first modal data based on the first attention processing result, and to determine a second weight corresponding to the second modal data based on the second attention processing result;

[0207] The first processing unit is further configured to obtain a fifth eigenvector corresponding to the first modal data based on the first weight and the first attention processing result, and to obtain an eighth eigenvector corresponding to the second modal data based on the second weight and the second attention processing result;

[0208] The first processing unit is further configured to obtain a first fused feature vector corresponding to the first modal data based on the fifth eigenvector and the first eigenvector of the first modal data, and to obtain a second fused feature vector corresponding to the second modal data based on the eighth eigenvector and the second eigenvector of the second modal data;

[0209] The first processing unit is further configured to train the model based on the first fused feature vector and the second fused feature vector.

[0210] or,

[0211] A first acquiring unit, configured to acquire first modal data and second modal data;

[0212] a first processing unit, configured to determine, based on the first eigenvector of the first modal data and the second eigenvector of the second modal data, a third eigenvector corresponding to the first modal data and a ninth eigenvector corresponding to the second modal data;

[0213] The first processing unit is further configured to determine a fourth eigenvector corresponding to the first modal data based on the third eigenvector and the similarity between the first modal data and the second modal data, and to determine a tenth eigenvector corresponding to the second modal data based on the ninth eigenvector and the similarity;

[0214] The first processing unit is further configured to determine a third fused eigenvector corresponding to the first modality data based on the fourth eigenvector and the first eigenvector, and to determine a fourth fused eigenvector corresponding to the second modality data based on the tenth eigenvector and the second eigenvector;

[0215] The first processing unit is further used to train the model based on the third fused feature vector and the third fused feature vector.

[0216] In one embodiment of the present application, in terms of fusing the similarity and the first feature vector to obtain a third fused feature vector corresponding to the first modal data, the first processing unit 902 is specifically configured to:

[0217] Determining a third eigenvector corresponding to the first modal data based on the first eigenvector and the second eigenvector;

[0218] determining, based on the third eigenvector and the similarity, a fourth eigenvector corresponding to the first modal data;

[0219] The third fused feature vector is determined based on the first feature vector and the fourth feature vector.

[0220] In one embodiment of the present application, in fusing the first feature vector and the first attention processing result to obtain a first fused feature vector corresponding to the first modal data, the first processing unit 902 is specifically configured to:

[0221] Determining a first weight corresponding to the first modal data based on the first attention processing result;

[0222] Obtaining a fifth eigenvector corresponding to the first modal data based on a first weight corresponding to the first modal data and the first attention processing result;

[0223] The first fused feature vector is obtained based on the first feature vector and the fifth feature vector.

[0224] In one embodiment of the present application, in determining the first weight corresponding to the first modal data based on the first attention processing result, the first processing unit 902 is specifically configured to:

[0225] Obtaining a norm of the first attention processing result;

[0226] Based on the norm of the first attention processing result, a first weight corresponding to the first modal data is determined.

[0227] In one embodiment of the present application, in obtaining the first fused feature vector based on the first feature vector and the fifth feature vector, the first processing unit 902 is specifically configured to:

[0228] Obtaining a first preset weight corresponding to the first modal data;

[0229] Obtaining a sixth eigenvector corresponding to the first modal data based on the first eigenvector and the first preset weight;

[0230] The first fused feature vector is obtained based on the fifth feature vector and the sixth feature vector.

[0231] In one embodiment of the present application, in terms of training the model based on the third fused feature vector and the fourth fused feature vector, the first processing unit 902 is specifically configured to:

[0232] Perform prediction based on the third fused feature vector and the fourth fused feature vector to obtain a prediction result;

[0233] Determining a first loss based on the prediction result, a seventh eigenvector corresponding to the first modal data, and an eighth eigenvector corresponding to the second modal data, wherein the seventh eigenvector is determined based on one or more of the first eigenvector, the first fused eigenvector, and the third fused eigenvector, and the eighth eigenvector is determined based on one or more of the second eigenvector, the second fused eigenvector, and the fourth fused eigenvector;

[0234] The model is trained based on the first loss.

[0235] In one embodiment of the present application, in determining the first loss based on the prediction result, the seventh eigenvector corresponding to the first modal data, and the eighth eigenvector corresponding to the second modal data, the first processing unit 902 is specifically configured to:

[0236] determining a second loss based on the prediction result;

[0237] determining a third loss based on the seventh eigenvector and the eighth eigenvector;

[0238] Obtaining a second preset weight corresponding to the first modal data, and obtaining a third preset weight corresponding to the second modal data;

[0239] determining a fourth loss based on the prediction result, the seventh eigenvector, the eighth eigenvector, the second preset weight, and the third preset weight;

[0240] The first loss is determined based on the second loss, the third loss, and the fourth loss.

[0241] In one embodiment of the present application, in determining the fourth loss based on the prediction result, the seventh eigenvector, the eighth eigenvector, the second preset weight, and the third preset weight, the first processing unit 902 is specifically configured to:

[0242] determining a first correlation coefficient between the prediction result and the seventh eigenvector, and determining a second correlation coefficient between the prediction result and the eighth eigenvector;

[0243] The fourth loss is determined based on the first correlation coefficient and the second correlation coefficient, and the second preset weight and the third preset weight.

[0244] In a specific implementation, the first acquisition unit 901 and the first processing unit 902 described in the embodiment of the present invention can also execute other implementation methods described in the embodiments of other model training methods provided by the embodiment of the present invention, which will not be repeated here.

[0245] See Figure 10 , Figure 10 This is a block diagram of the functional units of an information prediction device provided in an embodiment of the present application. The information prediction device 1000 includes: a second acquisition unit 1001 and a second processing unit 1002;

[0246] The second acquisition unit 1001 is used to acquire multimodal data of the user;

[0247] The second processing unit 1002 is configured to perform information prediction based on multimodal data using a model, wherein the model is obtained by training using the embodiment of the above-mentioned model training method.

[0248] In a specific implementation, the second acquisition unit 1001 and the second processing unit 1002 described in the embodiment of the present invention may also execute other implementations described in the embodiment of the information prediction method provided by the embodiment of the present invention, which will not be described in detail here.

[0249] See Figure 11 , Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 11 As shown, the electronic device 1100 includes a transceiver 1101, a processor 1102, and a memory 1103. These are connected via a bus 1104. The memory 1103 is used to store computer programs and data, and can transmit the data stored in the memory 1103 to the processor 1102.

[0250] The electronic device 1100 may be the model training device 900 or the information prediction device 1000;

[0251] When the electronic device 1100 is the model training device 900, the processor 1102 is configured to read the computer program in the memory 1103 and perform the following operations:

[0252] Controlling the transceiver 1101 to obtain the first modality data and the second modality data;

[0253] Calculating, based on a first eigenvector of the first modal data and a second eigenvector of the second modal data, a first attention processing result of the first modal data on the second modal data and a second attention processing result of the second modal data on the first modal data;

[0254] fusing the first feature vector and the first attention processing result to obtain a first fused feature vector corresponding to the first modal data;

[0255] fusing the second feature vector and the second attention processing result to obtain a second fused feature vector corresponding to the second modal data;

[0256] determining a similarity between the first modality data and the second modality data based on the first fused feature vector and the second fused feature vector;

[0257] Performing fusion based on the similarity and the first feature vector to obtain a third fused feature vector corresponding to the first modal data;

[0258] performing fusion based on the similarity and the second feature vector to obtain a fourth fused feature vector corresponding to the second modal data;

[0259] The model is trained based on the third fused feature vector and the fourth fused feature vector.

[0260] or,

[0261] Controlling the transceiver 1101 to obtain the first modality data and the second modality data;

[0262] Calculating, based on a first eigenvector of the first modal data and a second eigenvector of the second modal data, a first attention processing result of the first modal data on the second modal data and a second attention processing result of the second modal data on the first modal data;

[0263] Determining a first weight corresponding to the first modal data based on a first attention processing result of the first modal data on the second modal data, and determining a second weight corresponding to the second modal data based on a second attention processing result of the second modal data on the second modal data;

[0264] Based on the first weight and the first attention processing result, a fifth eigenvector corresponding to the first modal data is obtained, and based on the second weight and the second attention processing result, a ninth eigenvector corresponding to the second modal data is obtained;

[0265] Obtaining a first fused feature vector corresponding to the first modality data based on the fifth eigenvector and the first eigenvector of the first modality data, and obtaining a second fused feature vector corresponding to the second modality data based on the ninth eigenvector and the second eigenvector of the second modality data;

[0266] The model is trained based on the first fused feature vector and the second fused feature vector.

[0267] or,

[0268] Controlling the transceiver 1101 to obtain the first modality data and the second modality data;

[0269] determining, based on the first eigenvector of the first modal data and the second eigenvector of the second modal data, a third eigenvector corresponding to the first modal data, and determining a tenth eigenvector corresponding to the second modal data;

[0270] Determining a fourth eigenvector corresponding to the first modal data based on the third eigenvector and the similarity between the first modal data and the second modal data, and determining an eleventh eigenvector corresponding to the second modal data based on the ninth eigenvector and the similarity;

[0271] Determining a third fused eigenvector corresponding to the first modality data based on the fourth eigenvector and the first eigenvector, and determining a fourth fused eigenvector corresponding to the second modality data based on the tenth eigenvector and the second eigenvector;

[0272] The model is trained based on the third fused feature vector and the third fused feature vector.

[0273] In a specific implementation, the transceiver 1101 and processor 1102 described in the embodiment of the present invention may also execute other implementation methods described in the embodiment of the model training method provided in the embodiment of the present invention, which will not be repeated here.

[0274] When the electronic device 1100 is the information prediction device 1000, the processor 1102 is configured to read the computer program in the memory 1103 and perform the following operations:

[0275] Controlling the transceiver 1101 to obtain the user's multimodal data;

[0276] Information prediction is performed based on multimodal data through a model, wherein the model is trained through an embodiment of the above-mentioned model training method.

[0277] In a specific implementation, the transceiver 1101 and the processor 1102 described in the embodiment of the present invention may also execute other implementation methods described in the embodiment of the information prediction method provided by the embodiment of the present invention, which will not be repeated here.

[0278] Specifically, the transceiver 1101 may be Figure 9 The first acquisition unit 901 of the model training device 900 of the embodiment or Figure 10 The second acquisition unit 1001 of the information prediction device 1000 of the embodiment, the processor 1102 may be Figure 9 The first processing unit 902 of the model training device 900 of the embodiment or Figure 10 The second processing unit 1002 of the information prediction device 1000 of the embodiment.

[0279] It should be understood that the electronic device in this application can be a model training device or an information prediction device, and both the model training device and the information prediction device can be terminal devices or servers, wherein the terminal device can be a smart phone, tablet computer, laptop computer, desktop computer, smart TV, desktop computer, smart watch, smart car and other smart terminals, but is not limited to this; the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and basic cloud computing services such as big data and artificial intelligence platforms. This application does not make specific limitations. The above-mentioned electronic devices are only examples, not exhaustive, and include but are not limited to the above-mentioned electronic devices.

[0280] It should be understood that the embodiments of the present application also provide a computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement part or all of the steps of any model training method or information prediction method recorded in the above method embodiments.

[0281] An embodiment of the present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute part or all of the steps of any model training method or information prediction method described in the above method embodiments.

[0282] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all optional embodiments, and the actions and modules involved are not necessarily required by this application.

[0283] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0284] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0285] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0286] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or in the form of software program modules.

[0287] If the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0288] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable memory, and the memory can include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0289] The above is a detailed introduction to the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A model training method, characterized in that: The method comprises: Calculating, based on a first eigenvector of the first modal data and a second eigenvector of the second modal data, a first attention processing result of the first modal data on the second modal data and a second attention processing result of the second modal data on the first modal data; fusing the first feature vector and the first attention processing result to obtain a first fused feature vector corresponding to the first modal data; fusing the second feature vector and the second attention processing result to obtain a second fused feature vector corresponding to the second modal data; Determining a similarity between the first modality data and the second modality data based on the first fused feature vector and the second fused feature vector; Performing fusion based on the similarity and the first feature vector to obtain a third fused feature vector corresponding to the first modal data; performing fusion based on the similarity and the second feature vector to obtain a fourth fused feature vector corresponding to the second modal data; The model is trained based on the third fused feature vector and the fourth fused feature vector.

2. The method according to claim 1, characterized in that The fusing based on the similarity and the first feature vector to obtain a third fused feature vector corresponding to the first modal data includes: Determining a third eigenvector corresponding to the first modal data based on the first eigenvector and the second eigenvector; determining, based on the third eigenvector and the similarity, a fourth eigenvector corresponding to the first modal data; The third fused feature vector is determined based on the first feature vector and the fourth feature vector.

3. The method according to claim 1 or 2, characterized in that The fusing the first feature vector and the first attention processing result to obtain a first fused feature vector corresponding to the first modal data includes: Determining a first weight corresponding to the first modal data based on the first attention processing result; Obtaining a fifth eigenvector corresponding to the first modal data based on a first weight corresponding to the first modal data and the first attention processing result; The first fused feature vector is obtained based on the first feature vector and the fifth feature vector.

4. The method according to claim 3, characterized in that The determining, based on the first attention processing result, a first weight corresponding to the first modal data includes: Obtaining a norm of the first attention processing result; Based on the norm of the first attention processing result, a first weight corresponding to the first modal data is determined.

5. The method according to claim 3 or 4, characterized in that The obtaining the first fused feature vector based on the first feature vector and the fifth feature vector includes: Obtaining a first preset weight corresponding to the first modal data; Obtaining a sixth eigenvector corresponding to the first modal data based on the first eigenvector and the first preset weight; The first fused feature vector is obtained based on the fifth feature vector and the sixth feature vector.

6. The method according to any one of claims 1 to 5, characterized in that The training of the model based on the third fused feature vector and the fourth fused feature vector includes: Perform prediction based on the third fused feature vector and the fourth fused feature vector to obtain a prediction result; Determining a first loss based on the prediction result, a seventh eigenvector corresponding to the first modal data, and an eighth eigenvector corresponding to the second modal data, wherein the seventh eigenvector is determined based on one or more of the first eigenvector, the first fused eigenvector, and the third fused eigenvector, and the eighth eigenvector is determined based on one or more of the second eigenvector, the second fused eigenvector, and the fourth fused eigenvector; The model is trained based on the first loss.

7. The method according to claim 6, characterized in that The determining a first loss based on the prediction result, the seventh eigenvector corresponding to the first modal data, and the eighth eigenvector corresponding to the second modal data includes: determining a second loss based on the prediction result; determining a third loss based on the seventh eigenvector and the eighth eigenvector; Obtaining a second preset weight corresponding to the first modal data, and obtaining a third preset weight corresponding to the second modal data; determining a fourth loss based on the prediction result, the seventh eigenvector, the eighth eigenvector, the second preset weight, and the third preset weight; The first loss is determined based on the second loss, the third loss, and the fourth loss.

8. The method according to claim 7, characterized in that The determining the fourth loss based on the prediction result, the seventh eigenvector, the eighth eigenvector, the second preset weight, and the third preset weight includes: determining a first correlation coefficient between the prediction result and the seventh eigenvector, and determining a second correlation coefficient between the prediction result and the eighth eigenvector; The fourth loss is determined based on the first correlation coefficient and the second correlation coefficient, and the second preset weight and the third preset weight.

9. An information prediction method, characterized in that: The method comprises: Obtain multimodal data of users; Information prediction is performed based on the multimodal data using a model, wherein the model is trained using the training method described in any one of claims 1 to 8.

10. A model training device, characterized in that: The model training device includes: a first acquisition unit and a first processing unit; The first acquisition unit is configured to acquire first modal data and second modal data; The first processing unit is configured to calculate, based on the first eigenvector of the first modal data and the second eigenvector of the second modal data, a first attention processing result of the first modal data on the second modal data and a second attention processing result of the second modal data on the first modal data; The first processing unit is further configured to fuse the first feature vector and the first attention processing result to obtain a first fused feature vector corresponding to the first modal data; The first processing unit is further configured to fuse the second feature vector and the second attention processing result to obtain a second fused feature vector corresponding to the second modal data; The first processing unit is further configured to determine a similarity between the first modality data and the second modality data based on the first fused feature vector and the second fused feature vector; The first processing unit is further configured to fuse the first feature vector based on the similarity to obtain a third fused feature vector corresponding to the first modal data, and to fuse the second feature vector based on the similarity to obtain a fourth fused feature vector corresponding to the second modal data; The first processing unit is further used to train the model based on the third fused feature vector and the fourth fused feature vector.

11. An information prediction device, characterized in that: The information prediction device includes: a second acquisition unit and a second processing unit; The second acquiring unit is configured to acquire multimodal data of the user; The second processing unit is used to perform information prediction based on the multimodal data through a model, wherein the model is trained by the training method according to any one of claims 1 to 8.

12. An electronic device, characterized in that: include: A processor and a memory, the processor is connected to the memory, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 9.

Images