Multi-modal intelligent decision-making method and device, electronic equipment and storage medium

Abstract

The invention relates to a multi-modal intelligent decision-making method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining multi-modal sample data, and extracting a local feature vector and a position coding vector of each modal sample data through employing a local attention layer and an adaptive position coding technology; the global attention layer is combined with the local features and the position codes, and the global feature vector of each modal sample data is extracted. Different modal features are learned through a cross-modal attention layer, and a fusion feature vector is generated; and obtaining a prediction result of each mode according to the fusion feature vector through a decision module. And completing model training based on the prediction result and the real result to obtain a target model for processing the to-be-processed multi-modal data and outputting a target result. According to the method, through an adaptive hierarchical attention mechanism, the defects of feature extraction and fusion design in the prior art are overcome, and the performance and robustness of the model are remarkably improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and machine learning technology, and more specifically, to a multimodal intelligent decision-making method, apparatus, electronic device, and storage medium. Background Technology

[0002] In the field of multimodal intelligent decision-making, existing technologies typically employ the following methods to process multimodal data and make decisions: 1. Feature extraction and fusion: Independent Feature Extraction: Existing techniques typically extract features separately for each modality of data. For example, convolutional neural networks (CNNs) are used to extract image features, while recurrent neural networks (RNNs) or Transformers are used to extract text features. While these feature extraction methods can effectively extract features from a single modality, they face difficulties in fusing features from different modalities.

[0003] Simple fusion methods: Common fusion methods include early fusion (directly concatenating or adding features from different modalities) and late fusion (processing features from each modality separately before fusion). While these methods are simple, they often fail to fully utilize the complementary information between different modalities, resulting in poor fusion performance.

[0004] 2. Attention mechanisms: Traditional attention mechanisms: In existing technologies, attention mechanisms are typically used for feature extraction from unimodal data. For example, self-attention is used in text processing to capture long-range dependencies in text. However, these attention mechanisms often fail to effectively handle hierarchical feature extraction between different modalities when processing multimodal data.

[0005] Existing technologies have several shortcomings in the field of multimodal intelligent decision-making. In feature extraction and fusion, traditional attention mechanisms are ineffective at extracting features from multimodal data and fail to consider the interrelationships between different modalities. Furthermore, existing fusion methods, such as early fusion and late fusion, often fail to fully utilize the complementary information between different modalities, resulting in poor fusion performance. These problems limit the application effectiveness of existing technologies in multimodal intelligent decision-making tasks, necessitating new methods and technologies to address these issues. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a multimodal intelligent decision-making method, device, electronic device and storage medium, which aims to solve at least one of the above-mentioned technical problems.

[0007] In a first aspect, the technical solution of the present invention to solve the above-mentioned technical problems is as follows: a multimodal intelligent decision-making method, the method comprising: Acquire multimodal sample data, with each modality's sample data corresponding to a real result; For sample data of each modality, local feature vectors of the sample data of the modality are extracted through a local attention layer, and the position encoding vector corresponding to the position of each element in the sample data is extracted through an adaptive position encoding technique. For each modality's sample data, based on the local feature vector of the modality's sample data and the encoding vectors of all positions corresponding to the modality's sample data, the global feature vector corresponding to each position is extracted through the global attention layer to obtain the global feature vector corresponding to the modality's sample data; Based on the global feature vectors corresponding to the sample data of all modalities, the features of different modalities are learned through cross-modal attention layers to obtain the fused feature vectors corresponding to the sample data of each modality. Based on the fusion feature vectors corresponding to the sample data of all modalities, the prediction results corresponding to the sample data of each modality are obtained through the decision module; Based on the predicted and actual results corresponding to the sample data of all modalities, the model is trained to obtain the target model. The multimodal data to be processed is then processed according to the target model to obtain the target result.

[0008] The beneficial effects of this invention are: by using an adaptive hierarchical attention mechanism, hierarchical feature extraction of multimodal data can be achieved, enabling the model to capture the information interaction between different modalities more fully. This allows the target model trained based on the fused feature vectors corresponding to sample data from all modalities to accurately make decisions on the multimodal data to be processed. In addition, this solution also has the advantages of high computational efficiency and strong interpretability, supports multimodal decision-making tasks in different fields, and has broad application prospects.

[0009] Based on the above technical solution, the present invention can be further improved as follows.

[0010] Furthermore, in the cross-modal attention layer, the above method also includes: For the sample data of each modality, extract the statistical features corresponding to the sample data of that modality; For each modality's sample data, the first weight corresponding to the modality's sample data is dynamically determined using a modality-aware attention mechanism based on the statistical characteristics of the modality's sample data. Based on the global feature vectors corresponding to the sample data of all modalities, features of different modalities are learned through a cross-modal attention layer to obtain the fused feature vectors corresponding to the sample data of each modality, including: Based on the global feature vectors corresponding to the sample data of all modalities and the first weights corresponding to the sample number of each modality, the features of different modalities are learned through the cross-modal attention layer to obtain the fused feature vectors corresponding to the sample data of each modality.

[0011] Furthermore, the above methods also include: For each modality's sample data, the importance feature corresponding to the modality's sample data is extracted. The importance feature characterizes the importance of the modality's sample data in the corresponding prediction result. For each modality's sample data, the second weight corresponding to the modality's sample data is determined by the dynamic modality fusion module based on the importance characteristics of the modality's sample data. For each modality's sample data, the comprehensive feature vector corresponding to the modality's sample data is determined based on the fusion feature vector corresponding to the modality's sample data and the second weight corresponding to the modality's sample data. Based on the fused feature vectors corresponding to the sample data of all modalities, the prediction results for the sample data of each modality are obtained through the decision module, including: Based on the comprehensive feature vectors corresponding to the sample data of all modalities, the prediction results corresponding to the sample data of each modality are obtained through the decision module.

[0012] Furthermore, the above methods also include: Based on the fusion feature vectors corresponding to the sample data of all modalities, the confidence level of the prediction result corresponding to the sample data of each modality is obtained through the decision module. The magnitude of the confidence level represents the probability that the corresponding prediction result is the true result. Based on the predicted and actual results corresponding to the sample data of all modalities, the model is trained to obtain the target model, including: The basic cross-entropy loss is determined based on the prediction and actual results corresponding to the sample data of all modalities. Based on the confidence level of the prediction results corresponding to the sample data of the modality, the confidence regularization loss is determined. The confidence regularization loss characterizes the difference between the confidence level of each prediction result and the ideal confidence level. The model is trained using the basic cross-entropy loss and confidence regularization loss to obtain the target model.

[0013] Furthermore, the above methods also include: Based on the fusion feature vectors corresponding to the sample data of all modalities, determine the mean of the fusion feature vectors corresponding to the sample data of all modalities; Based on the fusion feature vectors and the mean of the fusion feature vectors corresponding to the sample data of all modalities, the modality consistency loss is determined. The modality consistency loss characterizes the difference between the fusion feature vector and the mean of the fusion feature vector for each modality. Based on the basic cross-entropy loss and confidence regularization loss, the model is trained to obtain the target model, including: The model is trained using the basic cross-entropy loss, confidence regularization loss, and modality consistency loss to obtain the target model.

[0014] Furthermore, the above methods also include: Based on the fusion feature vectors corresponding to the sample data of all modalities, calculate the similarity between the fusion feature vectors corresponding to each two different modalities; Modality diversity loss is determined based on the similarity between the fused feature vectors corresponding to each two different modalities. Modality diversity loss characterizes the diversity between the fused feature vectors of different modalities. Based on the basic cross-entropy loss, confidence regularization loss, and modality consistency loss, the model is trained to obtain the target model, including: The model is trained using the basic cross-entropy loss, confidence regularization loss, modality consistency loss, and modality diversity loss to obtain the target model.

[0015] Furthermore, the aforementioned multimodal sample data includes text data, voice data, and image data.

[0016] Secondly, to solve the above-mentioned technical problems, the present invention also provides a multimodal intelligent decision-making device, the device comprising: The acquisition module is used to acquire multimodal sample data, with each modality's sample data corresponding to a real result; The local feature extraction module is used to extract the local feature vector of the sample data of each modality through the local attention layer, and to extract the position encoding vector corresponding to the position of each element in the sample data through the adaptive position encoding technique. The global feature extraction module is used to extract the global feature vector corresponding to each position of the sample data for each modality, based on the local feature vector of the sample data of the modality and the encoding vector of all positions corresponding to the sample data of the modality, through the global attention layer, so as to obtain the global feature vector corresponding to the sample data of the modality. The fusion feature extraction module is used to learn the features of different modalities through a cross-modal attention layer based on the global feature vectors corresponding to the sample data of all modalities, and obtain the fusion feature vector corresponding to the sample data of each modality. The prediction module is used to obtain the prediction result for the sample data of each modality through the decision module based on the fused feature vectors corresponding to the sample data of all modalities. The training module is used to train the model based on the prediction results and actual results corresponding to the sample data of all modalities, to obtain the target model, and then process the multimodal data to be processed according to the target model to obtain the target result.

[0017] Thirdly, in order to solve the above-mentioned technical problems, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the multimodal intelligent decision-making method of the present application.

[0018] Fourthly, in order to solve the above-mentioned technical problems, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the multimodal intelligent decision-making method of the present application.

[0019] Additional aspects and advantages of this application will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of this application. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below.

[0021] Figure 1 This is a flowchart illustrating a multimodal intelligent decision-making method according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of a multimodal intelligent decision-making device according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present invention. Detailed Implementation

[0022] The principles and features of the present invention are described below. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0023] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0024] The solutions provided in these embodiments of the invention are applicable to any application scenario requiring optimization of the target model corresponding to a decision-making domain. The solutions provided in these embodiments of the invention can be executed by any electronic device, such as a user's terminal device, including at least one of the following: smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, smart TV, or smart in-vehicle device.

[0025] This invention provides a possible implementation, such as... Figure 1 The diagram shows a flowchart of a multimodal intelligent decision-making method. This method can be executed by any electronic device, such as a terminal device, or jointly executed by a terminal device and a server. For ease of description, the method provided in this embodiment will be described below using a terminal device as the execution subject as an example. Figure 1 The flowchart shown indicates that the method may include the following steps: S10: Obtain multimodal sample data, with each modality's sample data corresponding to a real result; S20, for sample data of each modality, the local feature vector of the sample data of the modality is extracted through the local attention layer, and the position encoding vector corresponding to the position of each element in the sample data is extracted through the adaptive position encoding technique. S30, For sample data of each modality, based on the local feature vector of the sample data of the modality and the encoding vector of all positions corresponding to the sample data of the modality, the global feature vector corresponding to each position is extracted through the global attention layer to obtain the global feature vector corresponding to the sample data of the modality. S40: Based on the global feature vectors corresponding to the sample data of all modalities, the features of different modalities are learned through the cross-modal attention layer to obtain the fused feature vectors corresponding to the sample data of each modality. S50: Based on the fusion feature vectors corresponding to the sample data of all modalities, the prediction results corresponding to the sample data of each modality are obtained through the decision module. S60: Based on the prediction results and actual results corresponding to the sample data of all modalities, complete the training of the model to obtain the target model, and then process the multimodal data to be processed according to the target model to obtain the target result.

[0026] The method of this invention can achieve hierarchical feature extraction of multimodal data through an adaptive hierarchical attention mechanism, enabling the model to more fully capture the information interaction between different modalities. As a result, the target model trained based on the fused feature vectors corresponding to the sample data of all modalities can accurately make decisions on the multimodal data to be processed. In addition, this scheme also has the advantages of high computational efficiency and strong interpretability, supports multimodal decision-making tasks in different fields, and has broad application prospects.

[0027] The following specific embodiments further illustrate the solution of the present invention. In this embodiment, a multimodal intelligent decision-making method may include the following steps: S10: Obtain multimodal sample data, with each modality's sample data corresponding to a real result; Optionally, the aforementioned multimodal sample data includes text modal data, audio modal data, image modal data, and numerical modal data.

[0028] Multimodal sample data refers to a collection of samples containing multiple different modalities (i.e., different types or sources of data). In multimodal intelligent decision-making systems, this sample data typically comes from multiple different data sources or data types, with each modality providing different aspects of the same object or event.

[0029] As an example, suppose we build an intelligent recommendation system to recommend products. The sample data for each product might contain the following modalities: 1. Text-based data, including product descriptions: a text describing the product's features, uses, etc. User reviews: user ratings and feedback on the product.

[0030] 2. Image modality data, including product images: images showing the appearance of the product. User-uploaded images: photos taken by users after using the product.

[0031] 3. Numerical modal data, including product price: a numerical value representing the selling price of a product. Sales data: numerical data such as product sales volume and inventory level.

[0032] 4. Audio modal data, including the audio portion of the product introduction video: the voice portion of the product introduction video.

[0033] Optionally, sample data from different modalities can be grouped into a single sample, and each sample can be represented as a structure containing data from multiple modalities, for example: Specifically, a sample of a product might be represented as: Sample={Text:"This phone features a high-performance processor and a long-lasting battery.", Image:[Image data], Audio:[Audio data], Numerical:{Price:2999, Sales:1000}}.

[0034] After obtaining the multimodal sample data, it can be preprocessed to obtain multimodal feature vectors. The preprocessing process is as follows: 1. Text modality data: Improved BPE segmentation is used to process the text modality data to obtain multiple segments. Each segment is processed by adaptive positional encoding to obtain the positional feature vector of each segment in the text modality data. 2. Image modality data: ResNet feature extraction + spatial attention enhancement. Specifically, ResNet is used to extract feature vectors from image modality data, and then the spatial attention mechanism is used to enhance the model's attention to important regions in the image, resulting in enhanced feature vectors. 3. Numerical modal data: Standardization + feature engineering enhancement. Specifically, the numerical modal data is first standardized, and then the standardized data is transformed and combined to generate more useful feature vectors. 4. Audio modal data: Converted to text modal data and processed using the methods for processing text modal data.

[0035] After preprocessing, the input to the local attention layer in S20 below is the preprocessed data or features.

[0036] In this application, an Adaptive Hierarchical Attention Module (AHAM module) is adopted. This module adopts a three-layer hierarchical architecture, including a local attention layer, a global attention layer, and a cross-modal attention layer, which are described in detail below.

[0037] Before S20, the sample data for each modality can be converted into a multimodal feature vector, which can be represented as: X m Then, in the subsequent processing of S20, the multimodal feature vector is used as input.

[0038] S20, for sample data of each modality, the local feature vector of the sample data of the modality is extracted through the Local Attention Layer, and the position encoding vector corresponding to the position of each element in the sample data is extracted through the adaptive position encoding technique. The input to the local attention layer can be the multimodal feature vector corresponding to the sample data of the modality, which can be represented as: X m ∈R^(n×d m ), where m represents the modal type, X m The feature matrix representing the m-th mode has dimensions n×d. m , n represents the number of samples, d m The matrix X represents the feature dimension of the m-th mode. m There is d m Columns, matrix X m Each row in the diagram represents a feature vector of a sample, and each column represents a feature dimension.

[0039] In S20, the specific implementation process of extracting the local feature vector of the sample data of each modality through the Local Attention Layer is as follows: Attention_local = softmax(Q_local K_local^T / sqrt(d_k)) V_local Where W_q^(local), W_k^(local), W_v^(local)∈R^(dm×d_k) represent the learnable query, key, and value weight matrices, respectively, b_q^(local), b_k^(local), b_v^(local)∈R^d_k represent the bias vectors, and d_k represents the attention dimension.

[0040] The multimodal feature vector X is processed through a local attention layer. m The local feature vector is transformed into a matrix consisting of Q_local, K_local, and V_local (query, key, and value vectors, respectively). Each row in the matrix represents a local feature at a specific location in the sequence corresponding to the sample data for each modality. These local features are computed using an attention mechanism, which captures the dependencies between each location and its related locations.

[0041] Here, an element refers to a part of the sample data. For example, in the sample data of a text modality, an element can refer to a word in the sample data of the text modality.

[0042] In S20, for each modality of sample data, the specific implementation process of extracting the position encoding vector corresponding to the position of each element in the sample data through Adaptive Positional Encoding (APE) is as follows: Among them, α(pos) = sigmoid(W_α [pos; seq_len] + b_α); β(pos) = tanh(W_β [pos; seq_len] + b_β).

[0043] Where pos represents the position index in the sequence (the sequence corresponding to the sample data of each modality), indicating the position of the current element in the sequence, usually counted starting from 0. i represents the dimension index, indicating the dimension index of the currently calculated position encoding vector in the multimodal feature vector. Let represent the first adaptive weight matrix, used to calculate α(pos). The first adaptive weight matrix has d rows. model / 2, the number of columns is 2. Let d represent the second adaptive weight matrix, used to calculate β(pos). The second adaptive weight matrix has d rows. model / 2, the number of columns is 2. This represents the first bias vector, used to calculate α(pos), and its length is d. model / 2, This represents the second bias vector used to calculate β(pos), and the length of the second bias vector is d. model / 2, σ represents the Sigmoid activation function, which maps the input value to the (0, 1) interval to generate α(pos), controlling the weights of the sine component. tanh represents the hyperbolic tangent activation function, which maps the input value to the (-1, 1) interval to generate β(pos), controlling the weights of the cosine component. seq_len represents the length of the sequence.

[0044] By dynamically adjusting the weights of the sine and cosine components using α(pos) and β(pos), the positional encoding can adapt to sequence data of different lengths.

[0045] S30, For sample data of each modality, based on the local feature vector of the sample data of the modality and the encoding vector of all positions corresponding to the sample data of the modality, the global feature vector corresponding to each position is extracted through the global attention layer to obtain the global feature vector corresponding to the sample data of the modality. Global feature vectors capture global dependencies throughout a sequence or dataset. They not only contain information from local features but also integrate these local features through a global attention mechanism, generating a comprehensive feature representation. This type of global feature vector allows for a better understanding of the overall structure and context of the sequence, improving model performance and generalization ability.

[0046] The specific implementation process of S30 includes: S301, add the positional encoding vectors (PE_adaptive) corresponding to all positions to the local feature vector (Attention_local) to obtain the enhanced feature matrix X_global, which can be represented as: X_global = Attention_local + PE_adaptive S302, based on the enhanced feature matrix X_global, the global feature vector corresponding to each position is extracted through the global attention layer, which can be expressed as: The global feature vector consists of Q_global, K_global, and V_global.

[0047] Where W_q^(global), W_k^(global), W_v^(global)∈R^(d m ×d_k) represent the learnable query, key, and value weight matrices, respectively, b_q^(global), b_k^(global), b_v^(global)∈R^d_k represents the bias vector, and d_k represents the attention dimension.

[0048] S303, fuse the global feature vectors corresponding to all positions to obtain the global feature vectors corresponding to the sample data of the modality. The specific process is as follows: Attention_global = MultiHead_Attention(Q_global, K_global, V_global) Here, `Attention_global` represents the global feature vector corresponding to the sample data of the modality, and `MultiHead_Attention` is a powerful attention mechanism that splits the input data into multiple heads, calculates attention scores for each head, concatenates the outputs of these heads, and finally obtains the final output through a linear transformation. This mechanism allows the model to learn information from different representation subspaces, thereby more comprehensively capturing the features and dependencies in the data and improving the model's performance and expressive power.

[0049] S40: Based on the global feature vectors corresponding to the sample data of all modalities, the features of different modalities are learned through the cross-modal attention layer to obtain the fused feature vectors corresponding to the sample data of each modality. The specific implementation process of the above S40 includes: S401, for each position i of each mode m, use the query vector Q of mode m. m (i) The key vector K with all other modes m′ Sum vector V m′ The interaction generates multiple cross-modal attention scores, which reflect the importance of position i in different modalities, enabling the model to dynamically focus on key information in different modalities.

[0050] S402, for each position i, based on the calculated cross-modal attention scores, the value vector V for all modalities... m′ By performing a weighted summation, we obtain the fused feature vector corresponding to position i. This fused feature vector represents the fusion of information from different modalities, preserving the important features of each position in different modalities.

[0051] S403, concatenate the fused feature vectors at all positions i to generate the fused feature vectors corresponding to the sample data of each modality.

[0052] The above splicing process is as follows: Initialize the fusion feature matrix: For each modality m, initialize an empty fusion feature matrix Cross_Attention_Output(m) with dimensions n×d. v Where n is the sequence length, d v It is the dimension of the fused feature vector.

[0053] Fill the fused feature vector by position: Iterate through each position i (from 1 to n) in the sequence, and put the fused feature vector Cross_Attention_Output(i) of each position i into the i-th row of the fused feature matrix Cross_Attention_Output(m). In this way, each row of the fused feature matrix corresponds to the fused feature vector of a position, and each column corresponds to a feature dimension.

[0054] Generate the final fusion feature matrix: After the above steps, the fusion feature matrix Cross_Attention_Output(m) is filled in, where each row is a fusion feature vector of a location and each column is a feature dimension.

[0055] The filled matrix is ​​the fusion feature matrix corresponding to the sample data of mode m.

[0056] Optionally, in the cross-modal attention layer, the method further includes: For the sample data of each modality, extract the statistical features corresponding to the sample data of that modality; For each modality's sample data, a modality-aware attention mechanism is used to dynamically determine the first weight corresponding to the modality's sample data based on the statistical characteristics of the modality's sample data. Through dynamic weights, the model can better understand the relationship between different modalities and improve the effect of cross-modal information interaction.

[0057] The specific implementation process of S40 above includes: Based on the global feature vectors corresponding to the sample data of all modalities and the first weights corresponding to the sample number of each modality, the features of different modalities are learned through the cross-modal attention layer to obtain the fused feature vectors corresponding to the sample data of each modality.

[0058] Among them, the above statistical features include those based on multimodal feature vector X m The mean, standard deviation, and maximum value are determined (or can be represented as X_m).

[0059] For each modality's sample data, the specific implementation process of dynamically determining the first weight corresponding to the modality's sample data using a modality-aware attention mechanism based on the statistical characteristics of the modality's sample data can be expressed as follows: Modal_Weight_m = softmax(W_modal [mean(X_m); std(X_m); max(X_m)] +b_modal); Where mean(X_m) represents the mean, std(X_m) represents the standard deviation, max(X_m) represents the maximum value, and W_modal represents the modality-aware weight matrix. Assuming the input feature vector has dimension d, then... b_modal represents the modality-aware bias vector. Assuming the output weight vector has dimension d, then... ; softmax represents the softmax function, used to normalize the input vector into a probability distribution such that the sum of all weights is 1. This ensures that the weights of different modalities are comparable in subsequent calculations. Modal_Weight_m represents the dynamic weight of modality m, i.e., the first weight.

[0060] The process described above, which learns the features of different modalities through a cross-modal attention layer based on the global feature vectors corresponding to the sample data of all modalities and the first weight corresponding to the number of samples of each modality, to obtain the fused feature vectors corresponding to the sample data of each modality, can be represented as follows: For each position i of modality m, calculate the Cross_Attention_score (including attention scores for different positions) for modality m with respect to other modalities. This score reflects the importance of the information that modality m obtains from other modalities at position i, where: Cross_Attention_score =Σ_m (Modal_Weight_m Attention(Q_m, K_other,V_other)); Where Q_m represents the query vector of modality m, K_other represents the key vector of other modalities, V_other represents the value vector of other modalities, Attention(Q_m, K_other, V_other) represents the attention score between modality m and other modalities. The similarity between modality m and other modalities is calculated by taking the dot product of the query vector Q_m and the key vector K_other, and then weighted and summed using the value vector V_other to obtain the information obtained by modality m from other modalities. Cross_Attention_score represents the cross-modal attention score, which is used to measure the weighted sum of the information obtained by modality m from other modalities.

[0061] For each position i in modality m, the cross-modal attention scores Cross_Attentions_core(i) of all modalities are weighted and summed to generate the fused feature vector at position i.

[0062] For modality m, the fusion feature vectors Cross_Attention_Output(i) at all positions i are arranged in order to generate the fusion feature vector (also called the fusion feature matrix) corresponding to the sample data of each modality.

[0063] S50: Based on the fusion feature vectors corresponding to the sample data of all modalities, the prediction results corresponding to the sample data of each modality are obtained through the decision module. Optionally, the method further includes: For each modality's sample data, the importance feature corresponding to the modality's sample data is extracted. The importance feature characterizes the importance of the modality's sample data in the corresponding prediction result. For each modality's sample data, a second weight is determined by the dynamic modality fusion module based on the importance characteristics of the sample data for that modality. This second weight enables the model to better fuse features from different modalities, generating a comprehensive feature representation and improving the accuracy and robustness of decision-making. For each modality's sample data, the comprehensive feature vector corresponding to the modality's sample data is determined based on the fusion feature vector corresponding to the modality's sample data and the second weight corresponding to the modality's sample data. The above-mentioned prediction results for each modality's sample data are obtained through the decision module based on the fused feature vectors corresponding to the sample data of all modalities, including: Based on the comprehensive feature vectors corresponding to the sample data of all modalities, the prediction results corresponding to the sample data of each modality are obtained through the decision module.

[0064] The process of determining the second weight corresponding to the sample data of each modality can be expressed as follows: Importance_m = MLP([Feature_variance_m; Feature_entropy_m; Cross_correlation_m]) Dynamic_Weight_m = softmax(Importance_m / τ) The importance features include feature variance (Feature_variance_m), feature entropy (Feature_entropy_m), and cross-correlation (Cross_correlation_m) between modality m and other modalities. Feature variance measures the degree of variation of modality m's features across different samples or locations. Higher variance indicates greater diversity in the modality's features, potentially containing more information or noise. Feature entropy measures the uncertainty or information content of modality m's features. Higher entropy indicates a more uniform distribution of the modality's features in the data, containing more uncertainty. Entropy can help the model assess whether the modality's features have sufficient information content. Cross-correlation between modality m and other modalities measures the correlation between modality m's features and features from other modalities. Higher cross-correlation indicates strong information sharing between the modality and other modalities, potentially playing a crucial role in multimodal fusion.

[0065] MLP stands for Multi-Layer Perceptron, a neural network architecture used to learn complex nonlinear mappings from input features. Here, MLP combines feature variance, feature entropy, and cross-correlation into a comprehensive importance score. Importance_m represents the importance score of modality m, and the comprehensive importance score calculated by MLP is used in the subsequent calculation of the second weight.

[0066] τ represents the temperature parameter, used to control the smoothness of the softmax function. A higher temperature parameter makes the softmax output smoother, while a lower temperature parameter makes the softmax output sharper, meaning the weight distribution is more concentrated. Dynamic_Weight_m represents the second weight, used to measure the importance of mode m in the current task. The higher the weight, the more important the mode is in the decision-making process.

[0067] After determining the second weight, the comprehensive feature vector corresponding to the sample data of each modality, based on the fused feature vector and the second weight, can be expressed as follows: Wherein, Fused_Feature represents the comprehensive feature vector corresponding to the sample data of a modality, Feature_m represents the fused feature vector corresponding to the sample data of a modality, and m represents the type of modality.

[0068] S60: Based on the prediction results and actual results corresponding to the sample data of all modalities, complete the training of the model to obtain the target model, and then process the multimodal data to be processed according to the target model to obtain the target result.

[0069] Optionally, the above method further includes: Based on the fusion feature vectors corresponding to the sample data of all modalities, the confidence level of the prediction result corresponding to the sample data of each modality is obtained through the decision module. The magnitude of the confidence level represents the probability that the corresponding prediction result is the true result. Specifically, based on the fused feature vectors corresponding to the sample data of all modalities, the prediction result (Decision_logits) corresponding to the sample data of each modality can be determined through the decision multi-level perception mechanism (MLP_decision) in the decision module. The prediction result can be a prediction score, which, in the classification domain, represents the prediction score for a certain category. The specific process can be represented as follows: Decision_logits = MLP_decision(Fused_Feature); Then, the prediction results (Decision_logits) corresponding to the sample data for each modality are normalized into a probability distribution to obtain the prediction probability (Decision_prob) corresponding to the sample data for each modality, which represents the probability of the sample data for each modality belonging to a certain category. Specifically, it can be expressed as: Decision_prob = softmax(Decision_logits); Next, based on the predicted probability (Decision_prob) and the total number of classes (num_classes) corresponding to the sample data for each modality, the confidence score (Confidence_score) of the prediction result corresponding to the sample data for each modality is determined, which can be specifically expressed as: Confidence_score = 1 - entropy(Decision_prob) / log(num_classes); Here, entropy represents the entropy function, used to measure the uncertainty of a probability distribution. The higher the entropy, the more uniform the probability distribution, and the greater the uncertainty. log(num_classes) represents the natural logarithm of the total number of classes, used to normalize the entropy value so that the confidence score is in the range [0, 1].

[0070] Based on the above, the final output of the model can include the prediction result corresponding to the sample data of each modality, the confidence level of the prediction result corresponding to the sample data of each modality, and the prediction probability corresponding to the sample data of each modality, which can be specifically expressed as: Final_output={"decision":argmax(Decision_prob),"probability":Decision_prob,"confidence": Confidence_score}; Here, Final_output represents the final output, argmax(Decision_prob) represents the index of the maximum value in the probability distribution, which is the class label predicted by the model (the true result), "decision" represents the class label predicted by the model (the prediction result), "probability" represents the model's predicted probability for each class, and "confidence" represents the model's confidence in the prediction result.

[0071] Based on the above, in step S60, the model is trained using the prediction and actual results corresponding to the sample data of all modalities, resulting in the target model, including: S601, Determine the basic cross-entropy loss based on the prediction and actual results corresponding to the sample data of all modalities; S602, Based on the confidence level of the prediction results corresponding to the sample data of the modality, determine the confidence level regularization loss. The confidence level regularization loss characterizes the difference between the confidence level of each prediction result and the ideal confidence level. S603, based on the basic cross-entropy loss and confidence regularization loss, completes the training of the model and obtains the target model.

[0072] The aforementioned basic cross-loss characterizes the difference between the predicted result and the actual result, and can be specifically expressed as: Where L_base represents the basic crossover loss, The sample data representing the i-th modality in the distribution of true labels (true results) represents the true result of the i-th category. In one-hot encoding, the value of the true result is 1, and the value of other categories is 0. This represents the model's predicted probability for the i-th category. Represents the probability of prediction Take the natural logarithm.

[0073] The confidence regularization loss mentioned above can be specifically expressed as: in, This represents the confidence level of the model in predicting the results for the sample data of the i-th modality. Indicates confidence level Take the natural logarithm, and let L_confidence represent the confidence regularization loss.

[0074] Furthermore, based on the above scheme, the method also includes: Based on the fusion feature vectors corresponding to the sample data of all modalities, determine the mean of the fusion feature vectors corresponding to the sample data of all modalities; Based on the fusion feature vectors and the mean of the fusion feature vectors corresponding to the sample data of all modalities, the modality consistency loss is determined. The modality consistency loss characterizes the difference between the fusion feature vector and the mean of the fusion feature vector for each modality. Specifically, the modal consistency loss mentioned above can be expressed as: Among them, L consistency (Also represented as L_consistency) represents modal consistency loss, Feature m Let represent the fused feature vector corresponding to the sample data of the m-th modality, and Mean_Feature represent the mean of the fused feature vector corresponding to the sample data of the m-th modality. The square of the L2 norm is used to measure the square of the Euclidean distance between two vectors, and in this scheme, it is used to calculate the Feature. m The difference between Mean_Feature and Mean_Feature.

[0075] One implementation of S603 above is as follows: S6031, based on the basic cross-entropy loss, confidence regularization loss and modality consistency loss, complete the training of the model to obtain the target model.

[0076] Furthermore, based on the above scheme, the method also includes: Based on the fusion feature vectors corresponding to the sample data of all modalities, calculate the similarity between the fusion feature vectors corresponding to each two different modalities; Modality diversity loss is determined based on the similarity between the fused feature vectors corresponding to each two different modalities. Modality diversity loss characterizes the diversity between the fused feature vectors of different modalities. The modal diversity loss mentioned above can be expressed as: Among them, L diversity (Also represented as L_diversity) represents modal diversity loss, Feature m The feature vector represents the fused feature vector corresponding to the sample data of the m-th modality. n cos_similarity(Feature) represents the fused feature vector corresponding to the sample data of the nth modality. m Feature n ) indicates Feature m and Feature n Cosine similarity between them.

[0077] One implementation of S6031 above is as follows: The model is trained using the basic cross-entropy loss, confidence regularization loss, modality consistency loss, and modality diversity loss to obtain the target model.

[0078] Specifically, the total loss L_ACAL can be determined first based on the basic cross-entropy loss, confidence regularization loss, modality consistency loss, and modality diversity loss, and is specifically expressed as follows: in, , , Let represent the weight hyperparameters, where ; ; ; Where epoch_ratio = current_epoch / total_epochs; epoch_ratio represents the proportion of the current training progress, current_epoch is the current training epoch number, total_epochs is the total number of training epochs, representing the proportion of the current training progress in the total training process, ranging from 0 to 1; α is the hyperparameter controlling the rate of change of λ1, β is the hyperparameter controlling the rate of change of λ2, and γ is the hyperparameter controlling the rate of change of λ3.

[0079] In this application, a multi-stage training strategy can be adopted: Phase 1: Single-modal pre-training; Phase 2: Multimodal joint training; Phase 3: End-to-end fine-tuning; The learning rate scheduling in this application is as follows: lr(t) = lr_base cos(π t / (2 T)) (1 +α sin(2π t / period)) Where lr(t) represents the learning rate at time t, lr_base represents the base learning rate, i.e., the initial value for learning rate scheduling, representing the learning rate in the early stages of training, t represents the current time step or training epoch, T represents the total training time or total number of training epochs, period represents the period length of the periodic change of the learning rate, i.e., the number of time steps in each period, and cos(π) t / (2 T) represents the cosine annealing part. Through the cosine function, the learning rate gradually decreases from lr_base to 0, simulating the annealing process. This helps the model converge more stably in the later stages of training. sin(2π) The periodic adjustment part (t / period) uses a sine function to periodically adjust the learning rate in each period, which helps the model avoid getting stuck in local minima during training and improves the dynamics of training.

[0080] In this application, the model parameters are set as follows: Hidden dimension: d_model = 512; Number of attention heads: num_heads = 8; Number of layers: num_layers = 6; Learning rate: lr_base = 1e-4; Batch size: batch_size = 32; The application scenarios applicable to this application include: 1. Financial risk assessment: integrating financial statements, news articles, and market data; 2. Intelligent Recommendation System: Combining user behavior, product images, and review text; 3. Supply chain optimization: Integrating logistics data, weather information, and demand forecasting; 4. Intelligent Customer Service: Integrating text-based dialogue, voice emotion, and user profiling; 5. Content review: Combining text semantics, image content, and user reputation.

[0081] As an example, in the field of financial risk assessment, the multimodal data to be processed includes: financial statements: containing a company's financial data, such as balance sheets, income statements, and cash flow statements; news texts: news reports related to the company, including both positive and negative news; and market data: stock prices, market indices, industry trends, etc.

[0082] The corresponding objective can be risk assessment: predicting the likelihood of a company defaulting or going bankrupt, and outputting a risk score or risk level.

[0083] In the field of intelligent recommendation systems, the multimodal data to be processed can include user behavior: browsing history, purchasing behavior, click-through rates, etc.; product images: image data of products used to showcase their appearance and features; and review text: user comments and evaluations of products.

[0084] The corresponding target result can be a recommendation list: recommending products that the user is most likely to be interested in, and outputting a list of recommended products.

[0085] In the field of supply chain optimization, the multimodal data to be processed can include logistics data: transportation time, transportation costs, transportation routes, etc.; weather information: weather conditions, such as temperature, precipitation, wind speed, etc.; and demand forecasting: market demand forecasts for goods.

[0086] The corresponding target outcome can be an optimization solution: providing the optimal logistics and inventory management solution and outputting an optimization suggestion report.

[0087] In the field of intelligent customer service, the multimodal data to be processed can include: text dialogues: records of text exchanges between users and customer service representatives; voice sentiment analysis: emotional analysis of user voice, such as anger, satisfaction, confusion, etc.; and user profiles: personal information and preferences of users, such as age, gender, and consumption habits.

[0088] The corresponding target result can be customer service response: generate the most suitable customer service response and output a customer service reply text.

[0089] In the field of content moderation, the multimodal data to be processed can include textual semantics: user-posted text content, including comments, posts, etc.; image content: user-uploaded images, which may contain inappropriate content; and user reputation: users' historical behavior records and reputation scores.

[0090] The corresponding target result is the review result: determine whether the content is compliant and output a review result label (such as "passed" or "failed").

[0091] The following section further explains the proposed solution in the context of financial risk assessment: 1. Data processing: Financial statements: Extract financial features through a numerical modal data processing workflow (standardization, feature engineering enhancement).

[0092] News text: Text features are extracted through a data processing workflow of text modality (improved BPE segmentation and adaptive positional encoding).

[0093] Market data: Market characteristics are extracted through numerical modal data processing.

[0094] 2. Feature Fusion Financial, textual, and market features are fused using the Adaptive Hierarchical Attention Mechanism (AHAM) and Dynamic Modality Fusion Module (DMFM) in the target model. Risk assessment results are generated through the Confidence-Aware Decision Module (CADM).

[0095] The proposed solution has the following advantages: 1. Performance Improvement: Decision accuracy is improved by 15-25% compared to existing methods; 2. Enhanced robustness: Maintains stable performance in noisy environments; 3. Interpretability: Provides visualization of decision confidence and attention weights; 4. High adaptability: Supports multimodal decision-making tasks in different fields; 5. Computational efficiency: Reduce computational complexity through hierarchical design.

[0096] Based on and Figure 1 Based on the same principle as the method shown, this embodiment of the invention also provides a multimodal intelligent decision-making device 20, such as... Figure 2 As shown, the multimodal intelligent decision-making device 20 may include an acquisition module 210, a local feature extraction module 220, a global feature extraction module 230, a fusion feature extraction module 240, a prediction module 250, and a training module 260, wherein: The acquisition module 210 is used to acquire multimodal sample data, with each modality's sample data corresponding to a real result; The local feature extraction module 220 is used to extract the local feature vector of the sample data of each modality through the local attention layer, and to extract the position encoding vector corresponding to the position of each element in the sample data through the adaptive position encoding technique. The global feature extraction module 230 is used to extract the global feature vector corresponding to each position through the global attention layer for each modality sample data, based on the local feature vector of the modality sample data and the encoding vector of all positions corresponding to the modality sample data. The fusion feature extraction module 240 is used to learn the features of different modalities through a cross-modal attention layer based on the global feature vectors corresponding to the sample data of all modalities, and obtain the fusion feature vectors corresponding to the sample data of each modality. The prediction module 250 is used to obtain the prediction result for the sample data of each modality through the decision module based on the fused feature vectors corresponding to the sample data of all modalities. The training module 260 is used to train the model based on the prediction results and actual results corresponding to the sample data of all modalities, to obtain the target model, and then process the multimodal data to be processed according to the target model to obtain the target result.

[0097] Optionally, in the cross-modal attention layer, the device further includes: The first weight determination module is used to extract the statistical features corresponding to the sample data of each modality; and to dynamically determine the first weight corresponding to the sample data of the modality based on the statistical features corresponding to the sample data of the modality using a modality-aware attention mechanism. The aforementioned fusion feature extraction module 240, when learning features of different modalities through a cross-modal attention layer based on the global feature vectors corresponding to the sample data of all modalities, specifically obtains the fusion feature vector corresponding to the sample data of each modality, is used for: Based on the global feature vectors corresponding to the sample data of all modalities and the first weights corresponding to the sample number of each modality, the features of different modalities are learned through the cross-modal attention layer to obtain the fused feature vectors corresponding to the sample data of each modality.

[0098] Optionally, the device further includes: The second weight determination module is used to extract the importance features corresponding to the sample data of each modality for each modality sample data. The importance features characterize the importance of the sample data of the modality in the corresponding prediction result. Based on the importance features corresponding to the sample data of the modality, the second weight corresponding to the sample data of the modality is determined by the dynamic modality fusion module. The comprehensive feature vector determination module is used to determine the comprehensive feature vector corresponding to the sample data of each modality based on the fused feature vector corresponding to the sample data of the modality and the second weight corresponding to the sample data of the modality. When the prediction module 250 obtains the prediction result for the sample data of each modality through the decision module based on the fused feature vectors corresponding to the sample data of all modalities, it is specifically used for: Based on the comprehensive feature vectors corresponding to the sample data of all modalities, the prediction results corresponding to the sample data of each modality are obtained through the decision module.

[0099] Optionally, the device further includes: The confidence determination module is used to obtain the confidence of the prediction result corresponding to the sample data of each modality through the decision module based on the fusion feature vector corresponding to the sample data of all modalities. The magnitude of the confidence represents the probability that the corresponding prediction result is the true result. When the training module 260 above completes the model training based on the prediction results and actual results corresponding to the sample data of all modalities to obtain the target model, it is specifically used for: The basic cross-entropy loss is determined based on the prediction and actual results corresponding to the sample data of all modalities. Based on the confidence level of the prediction results corresponding to the sample data of the modality, the confidence regularization loss is determined. The confidence regularization loss characterizes the difference between the confidence level of each prediction result and the ideal confidence level. The model is trained using the basic cross-entropy loss and confidence regularization loss to obtain the target model.

[0100] Optionally, the device further includes: The modality consistency loss determination module is used to determine the mean of the fusion feature vectors corresponding to the sample data of all modalities based on the fusion feature vectors corresponding to the sample data of all modalities; and to determine the modality consistency loss based on the fusion feature vectors corresponding to the sample data of all modalities and the mean of the fusion feature vectors. The modality consistency loss characterizes the difference between the fusion feature vector corresponding to each modality and the mean of the fusion feature vectors. The aforementioned training module 260, when training the model based on the basic cross-entropy loss and confidence regularization loss to obtain the target model, is specifically used for: The model is trained using the basic cross-entropy loss, confidence regularization loss, and modality consistency loss to obtain the target model.

[0101] Optionally, the device further includes: The modality diversity loss determination module is used to calculate the similarity between the fusion feature vectors corresponding to each two different modalities based on the fusion feature vectors corresponding to the sample data of all modalities; and to determine the modality diversity loss based on the similarity between the fusion feature vectors corresponding to each two different modalities. The modality diversity loss characterizes the diversity between the fusion feature vectors of different modalities. The aforementioned training module 260, when training the model based on the basic cross-entropy loss, confidence regularization loss, and modality consistency loss to obtain the target model, is specifically used for: The model is trained using the basic cross-entropy loss, confidence regularization loss, modality consistency loss, and modality diversity loss to obtain the target model.

[0102] Optionally, the aforementioned multimodal sample data includes text data, voice data, and image data.

[0103] The multimodal intelligent decision-making device of this invention can execute the multimodal intelligent decision-making method provided in this invention. The implementation principle is similar. The actions performed by each module and unit in the multimodal intelligent decision-making device of each embodiment of this invention correspond to the steps in the multimodal intelligent decision-making method of each embodiment of this invention. For detailed functional descriptions of each module of the multimodal intelligent decision-making device, please refer to the descriptions in the corresponding multimodal intelligent decision-making methods shown above. They will not be repeated here.

[0104] The aforementioned multimodal intelligent decision-making device can be a computer program (including program code) running on a computer device, such as an application software; the device can be used to execute the corresponding steps in the method provided in the embodiments of the present invention.

[0105] In some embodiments, the multimodal intelligent decision-making device provided in this invention can be implemented using a combination of hardware and software. As an example, the multimodal intelligent decision-making device provided in this invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the multimodal intelligent decision-making method provided in this invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0106] In other embodiments, the multimodal intelligent decision-making device provided in this invention can be implemented in software. Figure 2 A multimodal intelligent decision-making device stored in a memory is shown. It can be software in the form of programs and plug-ins, and includes a series of modules, including an acquisition module 210, a local feature extraction module 220, a global feature extraction module 230, a fusion feature extraction module 240, a prediction module 250, and a training module 260, for implementing the multimodal intelligent decision-making method provided in the embodiments of the present invention.

[0107] The modules described in the embodiments of the present invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0108] Based on the same principles as the methods shown in the embodiments of the present invention, the embodiments of the present invention also provide an electronic device, which may include, but is not limited to: a processor and a memory; the memory for storing computer programs; and the processor for executing the methods shown in any embodiment of the present invention by invoking the computer programs.

[0109] In one alternative embodiment, an electronic device is provided, such as Figure 3 As shown, Figure 3 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0110] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0111] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0112] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0113] The memory 4003 stores application code (computer program) for executing the present invention, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the content shown in the foregoing method embodiments.

[0114] Among these, electronic devices can also be terminal devices. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0115] This invention provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments.

[0116] According to another aspect of the present invention, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various embodiments described above.

[0117] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0118] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0119] The computer-readable storage medium provided in this invention can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0120] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0121] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A multi-modal intelligent decision making method, characterized in that, The method comprises: acquiring multi-modal sample data, each modality of sample data corresponding to a true result; for each modality of sample data, extracting a local feature vector of the modality of sample data through a local attention layer, and extracting a position encoding vector corresponding to the position of each element in the sample data through an adaptive position encoding technology; for each modality of sample data, extracting a global feature vector corresponding to each position through a global attention layer according to the local feature vector of the modality of sample data and all position encoding vectors corresponding to the modality of sample data, to obtain a global feature vector corresponding to the modality of sample data; learning features of different modalities through a cross-modality attention layer according to the global feature vectors corresponding to all modalities of sample data, to obtain a fusion feature vector corresponding to each modality of sample data; obtaining a prediction result corresponding to each modality of sample data through a decision module according to the fusion feature vectors corresponding to all modalities of sample data; completing the training of the model according to the prediction results and the true results corresponding to all modalities of sample data, to obtain a target model, so as to process multi-modal data to be processed according to the target model and obtain a target result.

2. The method of claim 1, wherein, In the cross-modality attention layer, the method further comprises: for each modality of sample data, extracting a statistical feature corresponding to the modality of sample data; for each modality of sample data, dynamically determining a first weight corresponding to the modality of sample data by using a modality-aware attention mechanism according to the statistical feature corresponding to the modality of sample data; the step of learning features of different modalities through a cross-modality attention layer according to the global feature vectors corresponding to all modalities of sample data, to obtain a fusion feature vector corresponding to each modality of sample data, comprises: learning features of different modalities through a cross-modality attention layer according to the global feature vectors corresponding to all modalities of sample data and the first weight corresponding to each modality of sample data, to obtain a fusion feature vector corresponding to each modality of sample data.

3. The method of claim 1, wherein, The method further comprises: for each modality of sample data, extracting an importance feature corresponding to the modality of sample data, the importance feature representing the importance of the modality of sample data in the corresponding prediction result; for each modality of sample data, determining a second weight corresponding to the modality of sample data through a dynamic modality fusion module according to the importance feature corresponding to the modality of sample data; for each modality of sample data, determining a comprehensive feature vector corresponding to the modality of sample data according to the fusion feature vector corresponding to the modality of sample data and the second weight corresponding to the modality of sample data; the step of obtaining a prediction result corresponding to each modality of sample data through a decision module according to the fusion feature vectors corresponding to all modalities of sample data, comprises: obtaining a prediction result corresponding to each modality of sample data through a decision module according to the comprehensive feature vectors corresponding to all modalities of sample data.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: According to the fusion feature vectors corresponding to the sample data of all modalities, a confidence degree of a prediction result corresponding to the sample data of each modality is obtained through a decision module, and the confidence degree represents a possibility that the corresponding prediction result is a true result; The training of the model is completed according to the prediction result and the true result corresponding to the sample data of all modalities to obtain a target model, including: A basic cross-entropy loss is determined according to the prediction result and the true result corresponding to the sample data of all modalities; A confidence degree regularization loss is determined according to the confidence degree of the prediction result corresponding to the sample data of the modality, and the confidence degree regularization loss represents a difference between the confidence degree of each prediction result and an ideal confidence degree; The training of the model is completed according to the basic cross-entropy loss and the confidence degree regularization loss to obtain the target model.

5. The method of claim 4, wherein, The method further includes: A fusion feature vector mean value corresponding to the sample data of all modalities is determined according to the fusion feature vectors corresponding to the sample data of all modalities; A modality consistency loss is determined according to the fusion feature vectors corresponding to the sample data of all modalities and the fusion feature vector mean value, and the modality consistency loss represents a difference between the fusion feature vector corresponding to each modality and the fusion feature vector mean value; The training of the model is completed according to the basic cross-entropy loss, the confidence degree regularization loss and the modality consistency loss to obtain the target model. The method further includes:

6. The method of claim 5, wherein, Similarities between the fusion feature vectors corresponding to each two different modalities are calculated according to the fusion feature vectors corresponding to the sample data of all modalities; A modality diversity loss is determined according to the similarities between the fusion feature vectors corresponding to each two different modalities, and the modality diversity loss represents a diversity between the fusion feature vectors of different modalities corresponding to different modality features; The training of the model is completed according to the basic cross-entropy loss, the confidence degree regularization loss, the modality consistency loss and the modality diversity loss to obtain the target model. The multi-modal sample data includes data of a text modality, data of an audio modality, data of an image modality and data of a numerical modality. includes:

7. The method according to any one of claims 1 to 3, characterized in that, An acquisition module is configured to acquire multi-modal sample data, and each modality of the sample data corresponds to a true result; 8. A multi-modal intelligent decision making apparatus, characterized by, A local feature extraction module is configured to, for sample data of each modality, extract a local feature vector of the sample data of the modality through a local attention layer, and extract a position encoding vector corresponding to a position of each element in the sample data through an adaptive position encoding technology; ​ ​ The global feature extraction module is configured to, for sample data of each modality, extract a global feature vector corresponding to each position by a global attention layer according to a local feature vector of the sample data of the modality and all position encoding vectors corresponding to the sample data of the modality, to obtain a global feature vector corresponding to the sample data of the modality; The fusion feature extraction module is configured to learn features of different modalities by a cross-modality attention layer according to the global feature vectors corresponding to the sample data of all modalities, to obtain a fusion feature vector corresponding to the sample data of each modality; The prediction module is configured to obtain a prediction result corresponding to the sample data of each modality by a decision module according to the fusion feature vectors corresponding to the sample data of all modalities. The training module is configured to complete training of the model according to the prediction results and real results corresponding to the sample data of all modalities, to obtain a target model, and to process the multi-modal data to be processed according to the target model to obtain a target result.

9. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method in any one of claims 1-7.

Images