Multi-modal data fusion method and system, readable storage medium and computer equipment

Abstract

The invention provides a multi-modal data fusion method and system, a readable storage medium and computer equipment, and the method comprises the steps: obtaining modal data corresponding to each equipment node, the modal data comprising text data, image data, video data and audio data, and each equipment node corresponding to one modal data; performing gradient learning on the initial local model on each equipment node according to the modal data to obtain a final local model on each equipment node; and fusing the final local models on the equipment nodes by adopting a weighted average algorithm to obtain an initial fusion model, and carrying out iterative convergence on the initial fusion model to obtain a final fusion model. According to the method, multi-modal and cross-modal federated learning can be realized, so that correlation information among different modals can be better captured, the performance and generalization ability of the model are improved, and more comprehensive data understanding and application are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of modal data fusion, and in particular to a multimodal data fusion method, system, readable storage medium and computer device. Background Art

[0002] The rapid development of the Internet of Things has greatly promoted the development of data-centric artificial intelligence. In the past decade, the explosive growth of the number of connected devices, such as wearable devices, sensors, drones, and autonomous vehicles, has made the collected data increasingly decentralized. Federated learning is a distributed learning method that allows multiple participants to jointly train models while protecting the privacy of their respective data. Due to its advantages in performance, data security, and privacy, federated learning is widely used in healthcare, financial services, smart devices and the Internet of Things, retail, and e-commerce. At present, most of the work on federated learning focuses on single-modal data and mainly considers the impact of data heterogeneity on model training.

[0003] However, the data collected by current IoT devices is no longer single-modal, but shows a multimodal trend. Given that it combines the advantages of federated learning and multimodal data processing, multimodal federated learning has received increasing attention. However, the current multimodal federated learning only supports two modalities, which only supports the processing of text and image data, resulting in very limited use. Based on this, how to solve the high-precision fusion problem of multimodal models has attracted much attention. Summary of the invention

[0004] The purpose of the present invention is to provide a multimodal data fusion method, system, readable storage medium and computer device, aiming to realize multimodal and cross-modal federated learning, so as to better capture the correlation information between different modalities, improve the performance and generalization ability of the model, and realize more comprehensive data understanding and application.

[0005] In a first aspect, the present invention provides a multimodal data fusion method, the method comprising:

[0006] Acquire modal data corresponding to each device node, wherein the modal data includes text data, image data, video data, and audio data, and each device node corresponds to one modal data;

[0007] Performing gradient learning on an initial local model on each device node according to the modal data to obtain a final local model on each device node;

[0008] A weighted average algorithm is used to fuse the final local models on each device node to obtain an initial fusion model, and the initial fusion model is iteratively converged to obtain a final fusion model.

[0009] Furthermore, the step of performing gradient learning on the initial local model on each device node according to the modal data to obtain the final local model on each device node includes:

[0010] The initial local model on each device node i performs τ times of gradient descent process and uses γ as a fixed learning rate to gradually learn the gradient model The details are as follows:

[0011]

[0012] Where x represents the modal data of device node i, y is the label corresponding to the modal data, and t represents the update round of the device node. Represents the model after the initial local model performs τ-1 gradient descent;

[0013] Each device node will learn the gradient model Share to the cloud server.

[0014] Furthermore, the step of using a weighted average algorithm to fuse the final local models on each device node to obtain an initial fusion model includes:

[0015] The initial fusion model is obtained according to the following formula:

[0016]

[0017] Among them, θ (t+1) represents the initial fusion model, α i represents the weight of the labeled modal data on device node i, S t Represents the set of node models received by the cloud server;

[0018] The step of iteratively converging the initial fusion model to obtain a final fusion model comprises:

[0019] Iteration convergence is performed according to the following formula:

[0020]

[0021] Among them, n i represents the amount of data in device node i, n represents the total amount of data in all device nodes, L i (θ i ) represents the loss function of device node i, and I represents the set of all device nodes participating in the calculation.

[0022] Furthermore, the weight of the modal data on device node i is calculated according to the following formula:

[0023]

[0024] Among them, e i Represents the attention vector of the i-th device node;

[0025] The attention vector of the i-th device node is calculated according to the following formula:

[0026] e i =q i ·k i

[0027] Among them, q i represents the query vector of the i-th device node, k i A key vector representing the i-th device node.

[0028] Furthermore, the method further comprises:

[0029] The loss function of device node i is calculated according to the following formula:

[0030]

[0031] Among them, β j represents the weight of the unlabeled modal data on device node j, m represents the total amount of labeled modal data, k represents the total amount of unlabeled modal data, and λ i and λ j Both represent the weight of the modality, CE(·) represents the cross entropy loss function, MCE(·) represents the mean square error loss function, and y i represents the true label of modality i, y p represents the predicted label of modality i.

[0032] Furthermore, the step of iteratively converging the initial fusion model to obtain a final fusion model further includes:

[0033] Use the calculated attention weights to perform weighted summation on the model vector to obtain a weighted representation:

[0034]

[0035] c represents the weighted model, v = {v1, v2, v3, …v n},v i Represents the model vector on device node i.

[0036] In a second aspect, the present invention provides a multimodal data fusion system, the system comprising:

[0037] A modality data acquisition module, used to acquire modality data corresponding to each device node, wherein the modality data includes text data, image data, video data, and audio data, and each device node corresponds to one type of modality data;

[0038] A gradient learning module, used for performing gradient learning on an initial local model on each device node according to the modal data, so as to obtain a final local model on each device node;

[0039] The weighted fusion module is used to fuse the final local models on each device node using a weighted average algorithm to obtain an initial fusion model, and iterate the initial fusion model to converge to obtain a final fusion model.

[0040] In a third aspect, the present invention provides a readable storage medium, wherein the readable storage medium stores one or more programs, which, when executed by a processor, implement the above-mentioned multimodal data fusion method.

[0041] In a fourth aspect, the present invention provides a computer device, the computer device comprising a memory and a processor, wherein:

[0042] The memory is used to store computer programs;

[0043] When the processor is used to execute the computer program stored in the memory, the above-mentioned multimodal data fusion method is implemented.

[0044] Compared with the prior art, the present invention has the following advantages:

[0045] 1. The present invention extracts effective features of each modality data through a deep neural network, and adopts an attention mechanism to assign different weights to data of different modalities, thereby integrating data of each modality in a targeted manner, solving the problem that traditional methods can only integrate data of a single or at most two modalities and have limitations; in addition, the fusion method proposed in the present invention has the advantages of low computational complexity and high computational accuracy.

[0046] 2. In order to improve the robustness of the multi-model information fusion mechanism, the present invention provides a loss function suitable for regression tasks and classification tasks, which provides an important basis for the subsequent optimization of attention mechanism parameters and neural network parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A flowchart of a multimodal data fusion method proposed in one embodiment of the present invention;

[0048] Figure 2 It is a schematic diagram of multimodal data;

[0049] Figure 3A schematic diagram of the structure of a multimodal data fusion system proposed in one embodiment of the present invention.

[0050] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be understood by people with general skills in the field to which the present invention belongs. "Including" and similar words used in this article mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0052] See also Figure 1 , which is a flow chart of a multimodal data fusion method provided by an embodiment of the present invention, the method comprises steps S101 to S103, wherein:

[0053] Step S101: Acquire modal data corresponding to each device node, wherein the modal data includes text data, image data, video data, and audio data, and each device node corresponds to one type of modal data.

[0054] See also Figure 2 Voice, text, video, and image data are the mainstream modalities of current IoT data. Since the existing technology can only process data of a single modality or at most two modalities, it is necessary to collect data of multiple modalities first. In this embodiment, three or even more modal data are generally obtained.

[0055] Step S102: performing gradient learning on the initial local model on each device node according to the modal data to obtain the final local model on each device node.

[0056] Step S103: using a weighted average algorithm to fuse the final local models on each device node to obtain an initial fusion model, and iterating and converging the initial fusion model to obtain a final fusion model.

[0057] It should be pointed out that in order to build the final fusion model, it is necessary to first establish a federated learning framework. Each device node communicates directly with a third-party server with powerful computing power, and the federated learning framework is specifically built based on a centralized architecture. The centralized architecture is chosen mainly because the fusion multimodal neural network model has high computational complexity and requires powerful computing power, and third-party cloud servers can perform such operations. In addition, the centralized structure can better reflect the accuracy of the model, thereby better solving the problem of multimodal data fusion.

[0058] Specifically, the federated learning framework is a distributed machine learning framework that aims to allow multiple parties to jointly train a global model without centralizing the original data set in one place. In federated learning, data is scattered across different devices or locations, such as mobile devices, sensors, edge devices, data centers, etc. The federated learning framework allows these devices to train models locally and aggregate the updated model parameters into a global model through a communication protocol. In some embodiments, federated learning includes three steps:

[0059] (1) Local model training: The device nodes participating in the training calculate a neural network model that can represent its characteristics based on the characteristics of its local data. The local model on each device node i performs a gradient descent process τ times and uses γ as a fixed learning rate to gradually learn the gradient model Where x is the modal data of device node i, y is the label corresponding to the modal data, and t represents the update round of the device node. The τth local model update of the node can be expressed as:

[0060]

[0061] Represents the model after the initial local model performs τ-1 gradient descents.

[0062] (2) Model sharing: Each device node shares the learned model Share to the cloud server.

[0063] (3) Model fusion: After receiving the models of all nodes, the cloud server fuses the models using a weighted average algorithm, where a i is the weight of the weighted average algorithm, which can be calculated through the subsequent attention mechanism to obtain the initial fusion model:

[0064]

[0065] Among them, θ (t+1) represents the initial fusion model, α i represents the weight of the labeled modal data on device node i, S tRepresents the collection of node models received by the cloud server.

[0066] (4) Model convergence: When the convergence conditions are met, such as minimizing the loss function, the update iteration stops. The iterative convergence is performed according to the following formula:

[0067]

[0068] Among them, n i represents the amount of data in device node i, n represents the total amount of data in all device nodes, L i (θ i ) represents the loss function of device node i, and I represents the set of all device nodes participating in the calculation.

[0069] As an example but not a limitation, suppose a local data set of a mobile phone user contains the following pictures: Picture 1: cat, labelled [1,0,0], Picture 2: dog, labelled [0,1,0], Picture 3: bird, labelled [0,0,1]. After local model training, the model may give the following prediction probabilities: Picture 1 model prediction is [0.7,0.2,0.1], Picture 2 model prediction is [0.1,0.8,0.1], Picture 3 model prediction is [0.2,0.2,0.6], and then calculate their cross entropy loss. Cross entropy loss calculation formula Where C is the number of categories (here 3), y i Indicates the label, Represents the predicted probability obtained using the softmax function.

[0070] For image 1 (cat), the cross entropy loss is:

[0071] Loss1=-[1×log(0.7)+0×log(0.2)+0×log(0.1)]=-log(0.7)

[0072] For image 2 (dog), the cross entropy loss is:

[0073] Loss2=-[0×l og(0.1)+1×l og(0.8)+0×l og(0.1)]=-l og(0.8);

[0074] For image 3 (bird), the cross entropy loss is:

[0075] Loss3=-[0×l og(0.2)+0×l og(0.2)+1×l og(0.6)]=-l og(0.6);

[0076] The final loss function is the average of these individual losses:

[0077] Total Loss=1 / 3(Loss1+Loss2+Loss3).

[0078] In addition, it should be noted that in the process of constructing the final fusion model, it is considered that the neural network based on the attention mechanism is widely used in multimodal fusion. The attention mechanism is introduced to perform weighted fusion of features of different modalities, so that the network can dynamically pay attention to the information of different modalities, thereby improving the model's ability to capture important information. However, how to determine the weights of different modalities and then fuse them is a major challenge. One of the most common methods is to normalize the attention weights using the Softmax function. Specifically, the attention score is calculated through one or more fully connected layers to quantify the degree of association between different features. Usually, the attention score is generated by one or more neural network layers, such as fully connected layers, convolutional layers, etc., and then an activation function is applied to ensure the non-negativity of the attention score. The attention score is normalized using the Softmax function to ensure that each attention weight ranges between 0 and 1, and the sum of all weights is 1. The original feature is multiplied by the attention weight to weightedly fuse different features. In this way, the model will pay more attention to features with higher attention weights.

[0079] Common attention mechanisms can be divided into self-attention mechanisms and multi-head attention mechanisms. In the self-attention mechanism, each input is compared with itself to calculate the self-attention weight. These weights are usually obtained by calculating the similarity between the inputs and then normalized to ensure that the sum of the weights is 1. The self-attention mechanism can effectively capture global semantic information by learning the correlation between different positions in the input sequence, so that the model performs better when dealing with long-distance dependency tasks. In addition, the self-attention mechanism has strong flexibility in processing sequences of different lengths. It can dynamically calculate the weight between each input position and all other positions, so that the model can adjust the allocation of attention according to different parts of the input. However, the computational complexity of the self-attention mechanism is very high. In order to capture global semantic information, it is necessary to calculate the correlation between each input position and all other positions, resulting in an increase in computational complexity as the length of the sequence increases, limiting its application on large-scale data.

[0080] In the multi-head attention mechanism, different attention weights are calculated in parallel by multiple attention heads, and then these weights are linearly transformed and concatenated, and finally further linearly transformed to generate the final output. The multi-head attention mechanism uses multiple attention heads to calculate different attention weights in parallel, so that the model can pay attention to the input from different angles, thereby enhancing the expressiveness of the model. In addition, the multi-head attention mechanism reduces the risk of overfitting by averaging the outputs of multiple attention heads, thereby improving the generalization ability of the model. Finally, since each attention head can be calculated independently, the multi-head attention mechanism can effectively take advantage of parallel computing and speed up the training of the model. However, the use of the multi-head attention mechanism will increase the number of parameters of the model, especially when each head has independent parameters, which may lead to a more complex model and increased demand for computing resources. In view of this, in practical applications, it is necessary to select a suitable attention mechanism based on task requirements and computing resource constraints.

[0081] Based on this, in some optional embodiments, the final fusion model constructed by the present invention also introduces an improved attention mechanism, as follows:

[0082] (1) Calculate Attention Scores: First, calculate the attention score for each position based on the input data and the current state of the model. This step usually involves a learnable attention weight matrix (usually called a mapping matrix of queries, keys, and values), and a dot product operation of the query vector and the key vector. Specifically, for a given query vector q and key vector k, its attention score e is:

[0083] e i =q i ·k i

[0084] Among them, q i represents the query vector of the i-th device node, k i A key vector representing the i-th device node.

[0085] In addition, it should be pointed out that the query vector q represents the information point or context that the current model needs to pay attention to, while the key vector k contains the feature representation of each element in the input sequence, which can be regarded as the "address" or "identifier" of each element in the sequence. The attention score e determines the contribution of each element in the input sequence to the current query when generating the output. The higher the score, the greater the relevance of the element to the query, and therefore the greater the weight it occupies in the final context vector.

[0086] (2) Calculate the attention weight: Then, the obtained attention score is normalized by the softmax function to obtain the attention weight. The softmax function can convert the score into a probability distribution so that the sum of all weights is 1. Specifically, for a given attention vector e, its attention weight α can be calculated by the following formula:

[0087]

[0088] Among them, e i Represents the attention vector of the i-th device node;

[0089] (3) Weighted summation: Finally, the model vectors are weighted summed using the calculated attention weights to obtain a weighted representation. Specifically, for a given model vector v = {v1, v2, v3, … v n}, can be calculated by the following formula:

[0090]

[0091] Where c represents the weighted model, v = {v1,v2,v3,…v n},v i Represents the model quantity on device node i.

[0092] (4) Output generation: The weighted model c, i.e., the model aggregated on the server side, is used as the input for the next round of server-node updates, and the final output result is generated iteratively.

[0093] In some optional embodiments, for the loss function Considering that the currently commonly used loss function cannot be applied to the multimodal fusion mechanism of mixed labeled data and unlabeled data, the present invention attempts to fuse the cross entropy loss function and the multimodal mean square loss error to obtain an improved loss function, as follows:

[0094] First, for each modality, the cross entropy loss between its prediction result and the true label is calculated, and then the loss of each modality is weighted summed or averaged. The loss function can be expressed as:

[0095]

[0096] Where n is the number of modes, λ i is the weight of each mode, y i is the true label of modality i, y pis the predicted label of modality i, and CE(·) represents the cross entropy loss function. Multimodal cross entropy loss is usually used for classification tasks and is suitable for scenarios where the model needs to output category probability distribution. Moreover, since the cross entropy loss is based on probability distribution, its results are highly interpretable and can clearly reflect the classification ability of the model. However, cross entropy loss is usually used for classification tasks and is not suitable for regression tasks.

[0097] For multimodal mean square error loss. This loss function is used for regression tasks. It calculates the mean square error between the prediction results of each modality and the true label, and then performs weighted summation or average of the losses of each modality. Its loss function can be expressed as:

[0098]

[0099] Wherein, MSE(·) represents mean square error. Mean square error loss is simple to calculate, easy to understand and implement. It is usually used for regression tasks and is suitable for scenarios where the model needs to output continuous values. However, in some cases, the predicted values ​​of different modalities may have different ranges or importances, which may lead to an imbalance in weights between modalities and require additional processing to solve. In view of this, the present invention proposes a loss function that comprehensively considers regression tasks and classification tasks to adapt to a multimodal fusion mechanism with mixed labeled data and unlabeled data:

[0100]

[0101] Among them, β j represents the weight of the unlabeled modal data on device node j, m represents the total amount of labeled modal data, k represents the total amount of unlabeled modal data, and λ i and λ j Both represent the weight of the modality, CE(·) represents the cross entropy loss function, MCE(·) represents the mean square error loss function, and y i represents the true label of modality i, y p represents the predicted label of modality i.

[0102] It should also be noted that the data model trained on the labeled data set is more reliable, so the corresponding weight can be assigned according to the amount of labeled data. Specifically, the weights of m labeled data are standardized:

[0103]

[0104] In summary, the multimodal data fusion method proposed in the present invention can better capture the correlation information between different modalities, improve the performance and generalization ability of the model, and achieve more comprehensive data understanding and application. The principle is to design a loss function that is suitable for labeled data and unlabeled data, and use a random sampling method to approximate the calculation of weights in the attention mechanism, thereby reducing computational overhead and model complexity.

[0105] See also Figure 3 , which shows a multimodal data fusion system in one embodiment of the present invention, the system comprises:

[0106] A modality data acquisition module 10 is used to acquire modality data corresponding to each device node, wherein the modality data includes text data, image data, video data, and audio data, and each device node corresponds to one type of modality data;

[0107] A gradient learning module 20, configured to perform gradient learning on an initial local model on each device node according to the modal data, so as to obtain a final local model on each device node;

[0108] The weighted fusion module 30 is used to fuse the final local models on each device node using a weighted average algorithm to obtain an initial fusion model, and iterate and converge the initial fusion model to obtain a final fusion model.

[0109] On the other hand, the present invention further provides a readable storage medium on which one or more programs are stored, and when the program is executed by a processor, the multimodal data fusion method described above is implemented.

[0110] On the other hand, the present invention further proposes a computer device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the above-mentioned multimodal data fusion method.

[0111] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0112] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0113] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0114] Although the embodiments of the present invention are described in detail above, it is obvious to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations are within the scope and spirit of the present invention as described in the claims. Moreover, the present invention described herein may have other embodiments and may be implemented or realized in a variety of ways.

Claims

1. A multimodal data fusion method, It is characterized in that The method comprises: Acquire modal data corresponding to each device node, wherein the modal data includes text data, image data, video data, and audio data, and each device node corresponds to one modal data; Performing gradient learning on an initial local model on each device node according to the modal data to obtain a final local model on each device node; A weighted average algorithm is used to fuse the final local models on each device node to obtain an initial fusion model, and the initial fusion model is iteratively converged to obtain a final fusion model.

2. The multimodal data fusion method according to claim 1, It is characterized in that The step of performing gradient learning on the initial local model on each device node according to the modal data to obtain the final local model on each device node includes: The initial local model on each device node i performs τ times of gradient descent process and uses γ as a fixed learning rate to gradually learn the gradient model The details are as follows: Where x represents the modal data of device node i, y is the label corresponding to the modal data, and t represents the update round of the device node. Represents the model after the initial local model performs τ-1 gradient descent; Each device node will learn the gradient model Share to the cloud server.

3. The multimodal data fusion method according to claim 2, It is characterized in that The step of using a weighted average algorithm to fuse the final local models on each device node to obtain an initial fusion model includes: The initial fusion model is obtained according to the following formula: Among them, θ (t+1) represents the initial fusion model, α i represents the weight of the labeled modal data on device node i, S t Represents the set of node models received by the cloud server; The step of iteratively converging the initial fusion model to obtain a final fusion model comprises: Iteration convergence is performed according to the following formula: Among them, n i represents the amount of data in device node i, n represents the total amount of data in all device nodes, L i (θ i ) represents the loss function of device node i, and I represents the set of all device nodes participating in the calculation.

4. The multimodal data fusion method according to claim 3, It is characterized in that The weight of the modal data on device node i is calculated according to the following formula: Among them, e i Represents the attention vector of the i-th device node; The attention vector of the i-th device node is calculated according to the following formula: e i =q i ·k i Among them, q i represents the query vector of the i-th device node, k i A key vector representing the i-th device node.

5. The multimodal data fusion method according to claim 4, It is characterized in that The method further comprises: The loss function of device node i is calculated according to the following formula: Among them, β j represents the weight of the unlabeled modal data on device node j, m represents the total amount of labeled modal data, k represents the total amount of unlabeled modal data, and λ i and λ j Both represent the weight of the modality, CE(·) represents the cross entropy loss function, MCE(·) represents the mean square error loss function, and y i represents the true label of modality i, y p represents the predicted label of modality i.

6. The multimodal data fusion method according to claim 5, It is characterized in that The step of iteratively converging the initial fusion model to obtain a final fusion model further comprises: Use the calculated attention weights to perform weighted summation on the model vector to obtain a weighted representation: c represents the weighted model, v = {v 1 ,v 2 ,v 3 ,…v n },v i Represents the model vector on device node i.

7. A multimodal data fusion system, It is characterized in that The system comprises: A modality data acquisition module, used to acquire modality data corresponding to each device node, wherein the modality data includes text data, image data, video data, and audio data, and each device node corresponds to one type of modality data; A gradient learning module, used for performing gradient learning on an initial local model on each device node according to the modal data, so as to obtain a final local model on each device node; The weighted fusion module is used to fuse the final local models on each device node using a weighted average algorithm to obtain an initial fusion model, and iterate the initial fusion model to converge to obtain a final fusion model.

8. A readable storage medium, It is characterized in that The readable storage medium stores one or more programs, which, when executed by a processor, implement the multimodal data fusion method according to any one of claims 1 to 6.

9. A computer device comprising a memory and a processor, in: The memory is used to store computer programs; When the processor is used to execute the computer program stored in the memory, the multimodal data fusion method according to any one of claims 1 to 6 is implemented.

Images