Multi-modal data prediction method and device based on hybrid expert attention network

Abstract

The invention relates to a multi-modal data prediction method and device based on a hybrid expert attention network, and the method comprises the steps: carrying out the deep interaction and fusion of initial features obtained by carrying out the feature extraction of multi-modal sample data through employing a cross attention mechanism, and obtaining the interaction features corresponding to different modals, according to the interaction characteristics corresponding to all the modalities, prediction is carried out through expert networks corresponding to different modalities to obtain a prediction result corresponding to the sample data of each modal, and a first loss function value is determined according to the prediction results corresponding to the sample data of all the modalities and a real label; a second loss function value determined by initial features of different modals corresponding to the same sample data and a third loss function value determined by different expert networks for prediction results corresponding to the sample data of the same modal are trained to obtain a prediction model, and the to-be-processed multi-modal data is predicted based on the prediction model. Through the method of the invention, the multi-modal fusion precision and robustness are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a method and apparatus for multimodal data prediction based on a hybrid expert attention network. Background Technology

[0002] In recent years, artificial intelligence technology has made significant progress in the field of multimodal data processing. However, existing technologies still face the following challenges when processing multi-source heterogeneous data: (1) insufficient information fusion between modalities; (2) difficulty in accurately quantifying the contribution of different modalities; (3) difficulty in effectively capturing long-sequence dependencies; and (4) insufficient generalization ability for diverse scenarios.

[0003] While traditional methods such as LSTM and Transformer have achieved certain results in specific fields, there is still room for improvement in their robustness and accuracy in complex scenarios. Therefore, there is an urgent need for a novel prediction method that can adaptively fuse multimodal information and dynamically adjust attention allocation. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a multimodal data prediction method and apparatus based on a hybrid expert attention network, which aims to solve at least one of the above-mentioned technical problems.

[0005] Firstly, the technical solution of the present invention to solve the above-mentioned technical problems is as follows: a multimodal data prediction method based on a hybrid expert attention network, the method comprising: Acquire training data, which includes multimodal sample data and a true label for each sample data. A true label represents the true prediction result of a sample data. Feature extraction is performed on multimodal sample data to obtain an initial feature set, which includes initial features of the same modality corresponding to different sample data and initial features of different modalities corresponding to the same sample data. A cross-attention mechanism is used to perform deep interaction and fusion on all initial features in the initial feature set to obtain the interaction features between initial features corresponding to different modalities. Based on the interaction features corresponding to all modalities, predictions are made through expert networks corresponding to different modalities to obtain the prediction results corresponding to the sample data of each modality. The value of the first loss function is determined based on the prediction results and true labels corresponding to the sample data of all modalities. The value of the second loss function is determined based on the initial features of different modalities corresponding to the same sample data; The value of the third loss function is determined based on the prediction results of different expert networks for sample data of the same modality. The total loss function value is determined based on the first loss function value, the second loss function value, and the third loss function value. The initial model is trained based on the total loss function value to obtain the prediction model. The prediction model is then used to predict the multimodal data to be processed, and the target prediction result is obtained.

[0006] The beneficial effects of this invention are as follows: This invention employs a complete technical chain of "feature extraction - cross-attention fusion - hybrid expert routing - multi-objective joint optimization." First, it performs unified feature extraction on multimodal training data to obtain an initial feature set. Then, it utilizes a cross-attention mechanism to achieve deep cross-modal interaction and fusion, generating interactive features rich in intermodal correlations. Subsequently, it uses an expert network specifically designed for different modalities to perform dynamic route prediction, obtaining prediction results for each modality. Finally, it uses the first loss function value, the second loss function value, and the third loss function value to jointly construct the total loss function value to train the model. This scheme significantly improves the accuracy and robustness of multimodal fusion, effectively solving the problems of modal information fragmentation, expert homogenization, and insufficient long sequence dependency capture.

[0007] Based on the above technical solution, the present invention can be further improved as follows.

[0008] Furthermore, the above methods also include: Based on all initial features in the initial feature set, a sequence memory network is used to learn the context features corresponding to each initial feature; Based on the interaction features corresponding to all modalities, predictions are made through expert networks corresponding to different modalities to obtain prediction results for the sample data of each modality, including: Based on the interaction features corresponding to all modalities and the context features corresponding to each initial feature, predictions are made through expert networks corresponding to different modalities to obtain the prediction results corresponding to the sample data of each modality.

[0009] Furthermore, based on the context features corresponding to all interaction features and all initial features, predictions are made through different expert networks to obtain prediction results for the sample data of each modality, including: For each expert network, the weights of the expert network are determined by a router based on all interaction features and context features corresponding to all initial features. For each expert network, the prediction result corresponding to the sample data of the corresponding modality is determined based on the interaction features of the modality corresponding to the expert network, the context features corresponding to the initial features, and the weights of the expert network.

[0010] Furthermore, the aforementioned expert network can be any of the following: Residual Attention Expert Network; Deep separable convolutional expert network; Graph Neural Network Expert Network; Diffusion of attention experts network.

[0011] Furthermore, the aforementioned cross-attention mechanism performs deep interaction and fusion on all initial features in the initial feature set to obtain interaction features between initial features corresponding to different modalities, including: Based on the initial features corresponding to different modes in the initial feature set, determine the weights corresponding to the initial features of different modes; Based on the weights of the initial features corresponding to different modalities, a cross-attention mechanism is used to perform deep interaction and fusion of the initial features corresponding to different modalities, thereby obtaining the interaction features between the initial features corresponding to different modalities.

[0012] Furthermore, the above-mentioned feature extraction of multimodal sample data yields an initial feature set, including: Based on the feature extraction networks corresponding to different modalities, features are extracted from the sample data of the corresponding modalities to obtain an initial feature set.

[0013] Furthermore, the above methods also include: Get the regularization term; The above determination of the total loss function value based on the first loss function value, the second loss function value, and the third loss function value includes: The total loss function value is determined based on the first loss function value, the second loss function value, the third loss function value, and the regularization term.

[0014] Secondly, to solve the above-mentioned technical problems, the present invention also provides a multimodal data prediction device based on a hybrid expert attention network, the device comprising: The acquisition module is used to acquire training data, which includes multimodal sample data and a true label for each sample data. A true label represents the true prediction result of a sample data. The first processing module is used to extract features from multimodal sample data to obtain an initial feature set. The initial feature set includes initial features of the same modality corresponding to different sample data and initial features of different modalities corresponding to the same sample data. The second processing module is used to perform deep interaction and fusion of all initial features in the initial feature set using a cross attention mechanism to obtain the interaction features between the initial features corresponding to different modalities. The first prediction module is used to make predictions based on the interaction features corresponding to all modalities, through expert networks corresponding to different modalities, to obtain the prediction results corresponding to the sample data of each modality. The first loss determination module is used to determine the value of the first loss function based on the prediction results and true labels corresponding to the sample data of all modalities. The second loss determination module is used to determine the value of the second loss function based on the initial features of different modalities corresponding to the same sample data. The third loss determination module is used to determine the value of the third loss function based on the prediction results of different expert networks for sample data of the same modality. The total loss determination module is used to determine the total loss function value based on the first loss function value, the second loss function value, and the third loss function value. The second prediction module is used to train the initial model based on the total loss function value to obtain a prediction model, and then use the prediction model to predict the multimodal data to be processed to obtain the target prediction result.

[0015] Thirdly, in order to solve the above-mentioned technical problems, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the multimodal data prediction method based on a hybrid expert attention network of the present application.

[0016] Fourthly, in order to solve the above-mentioned technical problems, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the multimodal data prediction method based on a hybrid expert attention network of the present application.

[0017] Additional aspects and advantages of this application will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of this application. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below.

[0019] Figure 1 A flowchart illustrating a multimodal data prediction method based on a hybrid expert attention network, provided as an embodiment of the present invention; Figure 2 A schematic diagram of a multimodal data prediction device based on a hybrid expert attention network is provided in one embodiment of the present invention. Figure 3 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present invention. Detailed Implementation

[0020] The principles and features of the present invention are described below. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0021] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0022] The solution provided in this invention can be applied to any application scenario that requires prediction of multimodal data. The solution provided in this invention can be executed by any electronic device, such as a user's terminal device, including at least one of the following: smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart TV, or smart in-vehicle device.

[0023] This invention provides a possible implementation, such as... Figure 1 The diagram shows a flowchart of a multimodal data prediction method based on a hybrid expert attention network. This method can be executed by any electronic device, such as a terminal device, or jointly executed by a terminal device and a server. For ease of description, the method provided in this embodiment will be described below using a terminal device as the execution subject as an example. Figure 1 The flowchart shown indicates that the method may include the following steps: S10, Obtain training data. The training data includes multimodal sample data and a true label corresponding to each sample data. A true label represents the true prediction result of a sample data. S20, extract features from multimodal sample data to obtain an initial feature set, which includes initial features of the same modality corresponding to different sample data and initial features of different modalities corresponding to the same sample data. S30 employs a cross-attention mechanism to perform deep interaction and fusion on all initial features in the initial feature set, thereby obtaining the interaction features between initial features corresponding to different modalities; S40, based on the interaction features corresponding to all modalities, predictions are made through expert networks corresponding to different modalities to obtain the prediction results corresponding to the sample data of each modality; S50, determine the value of the first loss function based on the prediction results and true labels corresponding to the sample data of all modalities; S60, determine the value of the second loss function based on the initial features of different modalities corresponding to the same sample data; S70, determine the value of the third loss function based on the prediction results of different expert networks for sample data of the same modality; S80, determine the total loss function value based on the first loss function value, the second loss function value, and the third loss function value; S90, the initial model is trained based on the total loss function value to obtain the prediction model, and the multimodal data to be processed is predicted based on the prediction model to obtain the target prediction result.

[0024] The beneficial effects of this invention are as follows: This invention employs a complete technical chain of "feature extraction - cross-attention fusion - hybrid expert routing - multi-objective joint optimization." First, it performs unified feature extraction on multimodal training data to obtain an initial feature set. Then, it utilizes a cross-attention mechanism to achieve deep cross-modal interaction and fusion, generating interactive features rich in intermodal correlations. Subsequently, it uses an expert network specifically designed for different modalities to perform dynamic route prediction, obtaining prediction results for each modality. Finally, it uses the first loss function value, the second loss function value, and the third loss function value to jointly construct the total loss function value to train the model. This scheme significantly improves the accuracy and robustness of multimodal fusion, effectively solving the problems of modal information fragmentation, expert homogenization, and insufficient long sequence dependency capture.

[0025] The following specific embodiments further illustrate the solution of the present invention. In this embodiment, the present invention proposes a multimodal sequence prediction method based on Dynamic Adaptive Mixture-of-Experts Attention Network (DAMEAN). This method achieves efficient fusion and accurate prediction of multimodal data through innovative model architecture, loss function design, and expert routing mechanism.

[0026] Specifically, the multimodal data prediction method based on a hybrid expert attention network provided in this embodiment may include the following steps: S10, Obtain training data. The training data includes multimodal sample data and a true label corresponding to each sample data. A true label represents the true prediction result of a sample data. Multimodal refers to different types, such as text, images, and time-series signals.

[0027] S20, extract features from multimodal sample data to obtain an initial feature set, which includes initial features of the same modality corresponding to different sample data and initial features of different modalities corresponding to the same sample data. The initial feature set includes initial features corresponding to different modalities, specifically initial features of the same modality corresponding to different sample data, and initial features of different modalities corresponding to the same sample data. An initial feature can be a high-dimensional feature.

[0028] Optionally, the above-mentioned feature extraction of multimodal sample data yields an initial feature set, including: Based on the feature extraction networks corresponding to different modalities, features are extracted from the sample data of the corresponding modalities to obtain an initial feature set.

[0029] The above step S20 can be implemented through a multimodal feature extraction module, which is responsible for extracting high-quality feature representations from different modal data. For the input multimodal sample data X = {X^1, X^2, ..., X^M}, where M is the number of modalities, the sample data X^m of any modality can be processed by the feature extractor F_m (also called the feature extraction network) corresponding to that modality. H^m = F_m(X^m) Where H^m∈R T×dm (T is the sequence length, and dm is the feature dimension of modality m). F_m can be an encoder (i.e., a feature extractor) designed for a specific modality. For example, an improved Transformer encoder can be used for the modality corresponding to text, a ViT variant can be used for the modality corresponding to images, and a temporal convolutional network can be used for the modality corresponding to time series data.

[0030] Optionally, for the modality corresponding to the text, this scheme designs an enhanced Transformer encoder with Rotation Position Encoding (RoPE): Attention(Q, K, V) = Softmax((QK^T) / sqrt(d_k) + M)V Where Q = W_QX, K = W_KX, and V = W_VX are the query, key, and value matrices, respectively, M is the learnable attention bias matrix, and d_k is the dimension of the key vector.

[0031] The position encoding uses the Rotation Position Encoding (RoPE) method: Where m is the position index, theta is the frequency parameter, and x_i and x_{i+1} are elements of adjacent dimensions.

[0032] S30 employs a cross-attention mechanism to perform deep interaction and fusion on all initial features ({H^1, H^2, ...,H^M}) in the initial feature set, thereby obtaining the interaction features between the initial features corresponding to different modalities; The purpose of using the cross-attention mechanism is to learn the features between the initial features, that is, to consider the correlation between different initial features.

[0033] Alternatively, one implementation of S30 above is as follows: S301, Based on the initial features corresponding to different modes in the initial feature set, determine the weights corresponding to the initial features corresponding to different modes; S302, based on the weights of the initial features corresponding to different modalities, a cross-attention mechanism is used to perform deep interaction and fusion of the initial features corresponding to different modalities to obtain the interaction features between the initial features corresponding to different modalities.

[0034] The above step S30 can be implemented through a dynamic cross-attention fusion module. This module designs a novel dynamic cross-attention mechanism to achieve deep interaction and fusion between initial features corresponding to different modalities. The core formula is as follows: Where H_i^m represents the i-th time step feature of the m-th modality, and the time step feature represents the initial feature corresponding to the i-th part of the sample data in the m-th modality. For example, if the m-th modality is the text-related modality, then the i-th time step feature of the m-th modality can be represented as the initial feature corresponding to the i-th word in the text. H_j^n represents the j-th time step feature of the n-th modality; This represents the interaction feature between the i-th time step feature of the m-th mode and the j-th time step feature of the n-th mode.

[0035] Where alpha_{m, n} represents the dynamic attention weights corresponding to a certain modality, which can be calculated in the following way: Wherein, the function g(H^m, H^n) is the modal correlation evaluation function: Where v, W_1, W_2, and b are learnable parameters. H^m represents the initial features corresponding to the m-th mode, and H^n represents the initial features corresponding to the n-th mode.

[0036] In this scheme, different mode pairs are dynamically adjusted using learnable modal weights alpha_{m,n}. The contribution (e.g., the modality corresponding to the image may be) The dominant mode is in the middle, while the corresponding modality of the text is in (More important in the middle).

[0037] S40, based on the interaction features corresponding to all modalities, predictions are made through expert networks corresponding to different modalities to obtain the prediction results corresponding to the sample data of each modality; One modality can correspond to one expert network.

[0038] Alternatively, the aforementioned expert network may be any of the following: Residual attention expert network, combining residual connectivity with multi-head attention mechanism; The Deep Separable Convolutional Expert Network uses depthwise separable convolutions for feature extraction. Graph Neural Networks (GNNs) are expert networks that can process data with graph structures. Diffusion Attention Expert Network: A novel attention mechanism focusing on long-distance dependencies.

[0039] The above S40 step can be implemented by an adaptive hybrid expert routing module, which includes E expert networks and a dynamic router, where E is the number of modes.

[0040] Alternatively, one possible implementation of the above S40 is as follows: S401, For each expert network, the weights of the expert network are determined by a router (also known as a dynamic router) based on the context features corresponding to all interaction features and all initial features. The router can dynamically configure the corresponding weights based on the data input to the adaptive hybrid expert routing module.

[0041] Where y represents the weights corresponding to the i-th expert network E_i, and G(x, theta_g) is the routing function, which outputs an E-dimensional weight vector. Let E_i represent the i-th expert network, x represent the context features input to the router, theta_g represent the learnable parameters of the routing function G, i.e. the weights and biases of the multilayer perceptron f, and theta_i represent all the learnable parameters of the i-th expert network E_i.

[0042] Alternatively, the routing function may be designed as follows: G(x, theta_g) = Sparsemax(f(x, theta_g)) Here, f(x, theta_g) is a multilayer perceptron, and the Sparsemax function is a sparse variant of Softmax, which can produce a more concentrated probability distribution. in, It is an E-1 dimensional simplex.

[0043] S402, For each expert network, based on the interaction features corresponding to the modality of the expert network, the context features corresponding to the initial features, and the weights of the expert network, determine the prediction result corresponding to the sample data of the modality of the expert network.

[0044] Optionally, the method further includes: S41, Based on all the initial features in the initial feature set, a sequence memory network is used to learn the context features corresponding to each initial feature; In S40 above, based on the interaction features corresponding to all modalities, predictions are made through expert networks corresponding to different modalities to obtain prediction results for the sample data of each modality, including: Based on the interaction features corresponding to all modalities and the context features corresponding to each initial feature, predictions are made through expert networks corresponding to different modalities to obtain the prediction results corresponding to the sample data of each modality.

[0045] Optionally, step S41 can be implemented using a sequence reinforcement learning module, which can learn the contextual features of each initial feature. Specifically, in this application, a sequence memory network can be used to learn the contextual features, wherein the sequence memory network can be represented as: m_t = beta⊙m_{t-1} + (1-beta)⊙Update(z_t, m_{t-1}) Where m_t is the memory state at time step t, i.e., the context feature corresponding to the initial feature at time step t, m_{t-1} is the memory state at time step t-1, z_t represents the interaction feature at time step t, beta is the forget gate parameter, ⊙ is element-wise multiplication, and Update is the update function. Where sigma is the activation function, and W_z, W_m and b are learnable parameters.

[0046] S50, determine the value of the first loss function based on the prediction results and true labels corresponding to the sample data of all modalities; The value of the first loss function can be determined based on either the regression loss function or the classification loss function.

[0047] S60, determine the value of the second loss function based on the initial features of different modalities corresponding to the same sample data; One implementation of S60 is as follows: Based on the initial features of different modalities corresponding to the same sample data, the value of the second loss function is determined by comparing the learning loss function, whereby the comparing learning loss function is: in, is the value of the second loss function, z_i and z_i^+ are the initial features of different modalities of the same sample, sim is the similarity function, and tau is the temperature parameter.

[0048] By comparing and learning loss functions, the consistency of representations between modalities can be enhanced.

[0049] S70, determine the value of the third loss function based on the prediction results of different expert networks for sample data of the same modality; One implementation of S70 is as follows: Based on the prediction results of different expert networks for sample data of the same modality, the value of the third loss function is determined through the expert diversity loss function, whereby the expert diversity loss function is: Where L_contrastive is the value of the third loss function, and d(E_i, E_j) is a function that measures the difference between expert networks E_i and E_j, defined as: d(E_i, E_j) = 1 - |A_i∩A_j| / |A_i∪A_j| Here, A_i and A_j represent the sample sets that expert networks E_i and E_j are responsible for processing, respectively.

[0050] The expert diversity loss function can promote the diversity of different experts.

[0051] S80, determine the total loss function value based on the first loss function value, the second loss function value, and the third loss function value; Optionally, the method further includes: Get the regularization term; One implementation of the above S80 is as follows: The total loss function value is determined based on the first loss function value, the second loss function value, the third loss function value, and the regularization term.

[0052] Specifically, the total loss function value can be expressed as: Where L represents the total loss function value, L_main represents the first loss function value, L_diversity represents the second loss function value, L_contrastive represents the third loss function value, L_reg represents the regularization term, and lambda_1, lambda_2, lambda_3, and lambda_4 are weight hyperparameters.

[0053] The complexity of the model can be controlled by using regularization terms.

[0054] S90, the initial model is trained based on the total loss function value to obtain the prediction model, and the multimodal data to be processed is predicted based on the prediction model to obtain the target prediction result.

[0055] The initial model can be the DAMEAN model.

[0056] The solution of the present invention has the following beneficial effects: 1. Significantly improves the efficiency of multimodal data fusion, with an accuracy improvement of 15-25% compared to traditional methods; 2. The adaptive expert routing mechanism enables the model to automatically select the optimal processing path for different data characteristics; 3. The innovative loss function design effectively solves the modal imbalance problem; 4. It has strong interpretability and can clearly demonstrate the contribution of different modalities; 5. Under conditions of limited computing resources, achieve efficient inference through sparse activation mechanism.

[0057] Based on and Figure 1 Using the same principle as the method shown, this embodiment of the invention also provides a multimodal data prediction device 20 based on a hybrid expert attention network, such as... Figure 2 As shown, the multimodal data prediction device 20 based on a hybrid expert attention network may include an acquisition module 210, a first processing module 220, a second processing module 230, a first prediction module 240, a first loss determination module 250, a second loss determination module 260, a third loss determination module 270, a total loss determination module 280, and a second prediction module 290, wherein: The acquisition module 210 is used to acquire training data, which includes multimodal sample data and a true label corresponding to each sample data. A true label represents the true prediction result of a sample data. The first processing module 220 is used to extract features from multimodal sample data to obtain an initial feature set. The initial feature set includes initial features of the same modality corresponding to different sample data and initial features of different modalities corresponding to the same sample data. The second processing module 230 is used to perform deep interaction and fusion on all initial features in the initial feature set using a cross attention mechanism to obtain the interaction features between initial features corresponding to different modalities. The first prediction module 240 is used to make predictions based on the interaction features corresponding to all modalities through expert networks corresponding to different modalities, and to obtain the prediction results corresponding to the sample data of each modality. The first loss determination module 250 is used to determine the value of the first loss function based on the prediction results and true labels corresponding to the sample data of all modalities. The second loss determination module 260 is used to determine the value of the second loss function based on the initial features of different modalities corresponding to the same sample data; The third loss determination module 270 is used to determine the value of the third loss function based on the prediction results of different expert networks for sample data of the same modality. The total loss determination module 280 is used to determine the total loss function value based on the first loss function value, the second loss function value, and the third loss function value; The second prediction module 290 is used to train the initial model based on the total loss function value to obtain a prediction model, and then use the prediction model to predict the multimodal data to be processed to obtain the target prediction result.

[0058] Optionally, the device further includes: The context feature determination module is used to learn the context features corresponding to each initial feature based on all initial features in the initial feature set using a sequence memory network. Optionally, when the first prediction module 240 predicts based on the interaction features corresponding to all modalities and through expert networks corresponding to different modalities to obtain the prediction results corresponding to the sample data of each modality, it is specifically used for: Based on the interaction features corresponding to all modalities and the context features corresponding to each initial feature, predictions are made through expert networks corresponding to different modalities to obtain the prediction results corresponding to the sample data of each modality.

[0059] Optionally, when the first prediction module 240 makes predictions based on the context features corresponding to all interaction features and all initial features, and obtains the prediction results corresponding to the sample data of each modality through different expert networks, it is specifically used for: For each expert network, the weights of the expert network are determined by a router based on all interaction features and context features corresponding to all initial features. For each expert network, the prediction result corresponding to the sample data of the corresponding modality is determined based on the interaction features of the modality corresponding to the expert network, the context features corresponding to the initial features, and the weights of the expert network.

[0060] Alternatively, the aforementioned expert network may be any of the following: Residual Attention Expert Network; Deep separable convolutional expert network; Graph Neural Network Expert Network; Diffusion of attention experts network.

[0061] Optionally, when the second processing module 230 uses a cross-attention mechanism to perform deep interaction and fusion on all initial features in the initial feature set to obtain interaction features between initial features corresponding to different modalities, it is specifically used for: Based on the initial features corresponding to different modes in the initial feature set, determine the weights corresponding to the initial features of different modes; Based on the weights of the initial features corresponding to different modalities, a cross-attention mechanism is used to perform deep interaction and fusion of the initial features corresponding to different modalities, thereby obtaining the interaction features between the initial features corresponding to different modalities.

[0062] Optionally, when the first processing module 220 extracts features from the multimodal sample data to obtain an initial feature set, it is specifically used for: Based on the feature extraction networks corresponding to different modalities, features are extracted from the sample data of the corresponding modalities to obtain an initial feature set.

[0063] Optionally, the device further includes: The regularization term determination module is used to obtain the regularization term; When determining the total loss function value based on the first loss function value, the second loss function value, and the third loss function value, the aforementioned total loss determination module 280 is specifically used for: The total loss function value is determined based on the first loss function value, the second loss function value, the third loss function value, and the regularization term.

[0064] The multimodal data prediction device based on a hybrid expert attention network in this embodiment of the invention can execute the multimodal data prediction method based on a hybrid expert attention network provided in this embodiment of the invention. The implementation principle is similar. The actions performed by each module and unit in the multimodal data prediction device based on a hybrid expert attention network in each embodiment of the invention correspond to the steps in the multimodal data prediction method based on a hybrid expert attention network in each embodiment of the invention. For detailed functional descriptions of each module of the multimodal data prediction device based on a hybrid expert attention network, please refer to the descriptions in the corresponding multimodal data prediction methods based on a hybrid expert attention network shown above, which will not be repeated here.

[0065] The aforementioned multimodal data prediction device based on a hybrid expert attention network can be a computer program (including program code) running on a computer device. For example, the multimodal data prediction device based on a hybrid expert attention network is an application software. The device can be used to execute the corresponding steps in the method provided in the embodiments of the present invention.

[0066] In some embodiments, the multimodal data prediction device based on a hybrid expert attention network provided in this invention can be implemented using a combination of hardware and software. As an example, the multimodal data prediction device based on a hybrid expert attention network provided in this invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the multimodal data prediction method based on a hybrid expert attention network provided in this invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0067] In other embodiments, the multimodal data prediction device based on a hybrid expert attention network provided in this invention can be implemented in software. Figure 2 A multimodal data prediction device based on a hybrid expert attention network, stored in a memory, is shown. It can be software in the form of programs and plug-ins, and includes a series of modules, including an acquisition module 210, a first processing module 220, a second processing module 230, a first prediction module 240, a first loss determination module 250, a second loss determination module 260, a third loss determination module 270, a total loss determination module 280, and a second prediction module 290, for implementing the multimodal data prediction method based on a hybrid expert attention network provided in the embodiments of the present invention.

[0068] The modules described in the embodiments of the present invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0069] Based on the same principles as the methods shown in the embodiments of the present invention, the embodiments of the present invention also provide an electronic device, which may include, but is not limited to: a processor and a memory; the memory for storing computer programs; and the processor for executing the methods shown in any embodiment of the present invention by invoking the computer programs.

[0070] In one alternative embodiment, an electronic device is provided, such as Figure 3 As shown, Figure 3 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0071] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0072] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0073] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0074] The memory 4003 stores application code (computer program) for executing the present invention, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the content shown in the foregoing method embodiments.

[0075] Among these, electronic devices can also be terminal devices. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0076] This invention provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the corresponding content in the aforementioned method embodiments.

[0077] According to another aspect of the present invention, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various embodiments described above.

[0078] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0079] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0080] The computer-readable storage medium provided in this invention can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0081] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0082] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A multimodal data prediction method based on a hybrid expert attention network, characterized in that, include: Acquire training data, which includes multimodal sample data and a true label for each sample data. A true label represents the true prediction result of a sample data. Feature extraction is performed on the multimodal sample data to obtain an initial feature set, which includes initial features of the same modality corresponding to different sample data and initial features of different modalities corresponding to the same sample data. A cross-attention mechanism is used to perform deep interaction and fusion on all initial features in the initial feature set to obtain the interaction features between initial features corresponding to different modalities. Based on the interaction features corresponding to all modalities, predictions are made through expert networks corresponding to different modalities to obtain the prediction results corresponding to the sample data of each modality. The value of the first loss function is determined based on the prediction results and true labels corresponding to the sample data of all modalities. The value of the second loss function is determined based on the initial features of different modalities corresponding to the same sample data; The value of the third loss function is determined based on the prediction results of different expert networks for sample data of the same modality. The total loss function value is determined based on the first loss function value, the second loss function value, and the third loss function value. The initial model is trained based on the total loss function value to obtain a prediction model. The prediction model is then used to predict the multimodal data to be processed, and the target prediction result is obtained.

2. The method according to claim 1, characterized in that, The method further includes: Based on all initial features in the initial feature set, a sequence memory network is used to learn the context features corresponding to each initial feature; The step of predicting based on the interaction features corresponding to all modalities using expert networks corresponding to different modalities to obtain prediction results for sample data of each modality includes: Based on the interaction features corresponding to all modalities and the context features corresponding to each initial feature, predictions are made through expert networks corresponding to different modalities to obtain the prediction results corresponding to the sample data of each modality.

3. The method according to claim 2, characterized in that, The step of predicting based on the context features corresponding to all interaction features and all initial features, and using different expert networks to obtain the prediction results for the sample data of each modality, includes: For each expert network, the weights of the expert network are determined by a router based on the context features corresponding to all interaction features and all initial features. For each expert network, the prediction result corresponding to the sample data of the modality corresponding to the expert network is determined based on the interaction features corresponding to the modality corresponding to the expert network, the context features corresponding to the initial features, and the weights corresponding to the expert network.

4. The method according to any one of claims 1 to 3, characterized in that, The expert network is any of the following: Residual Attention Expert Network; Deeply Separating Convolutional Expert Network; Graph Neural Network Expert Network; Diffusion of attention experts network.

5. The method according to any one of claims 1 to 3, characterized in that, The method employs a cross-attention mechanism to perform deep interaction and fusion on all initial features in the initial feature set, obtaining interaction features between initial features corresponding to different modalities, including: Based on the initial features corresponding to different modes in the initial feature set, determine the weights corresponding to the initial features of different modes; Based on the weights of the initial features corresponding to different modalities, a cross-attention mechanism is used to perform deep interaction and fusion of the initial features corresponding to different modalities, thereby obtaining the interaction features between the initial features corresponding to different modalities.

6. The method according to any one of claims 1 to 3, characterized in that, The step of extracting features from the multimodal sample data to obtain an initial feature set includes: Based on the feature extraction network corresponding to different modalities, feature extraction is performed on the sample data of the corresponding modality to obtain the initial feature set.

7. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Get the regularization term; The step of determining the total loss function value based on the first loss function value, the second loss function value, and the third loss function value includes: The total loss function value is determined based on the first loss function value, the second loss function value, the third loss function value, and the regularization term.

8. A multimodal data prediction device based on a hybrid expert attention network, characterized in that, include: The acquisition module is used to acquire training data, which includes multimodal sample data and a real label corresponding to each sample data. A real label represents the real prediction result of a sample data. The first processing module is used to extract features from the multimodal sample data to obtain an initial feature set, which includes initial features of the same modality corresponding to different sample data and initial features of different modalities corresponding to the same sample data. The second processing module is used to perform deep interaction and fusion on all initial features in the initial feature set using a cross attention mechanism to obtain the interaction features between initial features corresponding to different modalities. The first prediction module is used to make predictions based on the interaction features corresponding to all modalities, through expert networks corresponding to different modalities, to obtain the prediction results corresponding to the sample data of each modality. The first loss determination module is used to determine the value of the first loss function based on the prediction results and true labels corresponding to the sample data of all modalities. The second loss determination module is used to determine the value of the second loss function based on the initial features of different modalities corresponding to the same sample data. The third loss determination module is used to determine the value of the third loss function based on the prediction results of different expert networks for sample data of the same modality. The total loss determination module is used to determine the total loss function value based on the first loss function value, the second loss function value, and the third loss function value; The second prediction module is used to train the initial model based on the total loss function value to obtain a prediction model, and then use the prediction model to predict the multimodal data to be processed to obtain the target prediction result.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-7.

Images