Multi-modal data fusion method and device based on large model

Abstract

The embodiment of the invention discloses a multi-modal data fusion method and device based on a large model. The method comprises the steps that multi-modal data to be processed are converted into embedded vectors corresponding to all modals; on the basis of a preset CNN model, extracting specific features of the embedded vectors corresponding to the modals, and obtaining the specific features; determining the weight of the specific feature through a preset reinforcement learning algorithm based on the specific feature; and on the basis of the weight of the specific feature, gate control is added, feature extraction is carried out on the specific feature, a shared feature is obtained, and the shared feature is a fusion result of the to-be-processed multi-modal data.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a multimodal data fusion method and device based on a large model. Background Art

[0002] In the field of artificial intelligence, the most common multimodal data include text, images, audio, and video. Each modality has its own unique characteristics and information, and combining these different modal data can provide a more comprehensive understanding and richer information.

[0003] In the prior art, the fusion of multimodal data is generally processed by machine learning methods. This method generally uses a shallow model and relies on a large amount of labeled data for supervised or semi-supervised training. Therefore, it usually requires a lot of preprocessing and manual intervention, and cannot generate missing data.

[0004] Based on this, this specification provides a multimodal data fusion method and device based on a large model. Summary of the invention

[0005] The embodiments of this specification provide a multimodal data fusion method and device based on a large model, which are used to solve the following technical problems: In the prior art, the fusion of multimodal data is generally processed using a machine learning method. This method generally uses a shallow model and relies on a large amount of labeled data for supervised or semi-supervised training. Therefore, it usually requires a lot of preprocessing and manual intervention, and cannot generate missing data.

[0006] To solve the above technical problems, the embodiments of this specification are implemented as follows:

[0007] The embodiment of this specification provides a multimodal data fusion method based on a large model, including:

[0008] Convert the multimodal data to be processed into embedding vectors corresponding to each modality;

[0009] Based on a preset CNN model, extract the specific features of the embedded vectors corresponding to each modality to obtain the specific features;

[0010] Based on the specific feature, determine the weight of the specific feature through a preset reinforcement learning algorithm;

[0011] Based on the weight of the specific feature, a gate control is added to perform feature extraction on the specific feature to obtain a shared feature, where the shared feature is a fusion result of the multimodal data to be processed.

[0012] The embodiment of this specification also provides a multimodal data fusion device based on a large model, including:

[0013] The conversion module converts the multimodal data to be processed into the embedding vector corresponding to each modality;

[0014] A specific feature extraction module, based on a preset CNN model, extracts specific features of the embedding vectors corresponding to each modality to obtain specific features;

[0015] A weight determination module, based on the specific feature, determines the weight of the specific feature through a preset reinforcement learning algorithm;

[0016] The fusion module adds gate control based on the weight of the specific feature, extracts the specific feature, and obtains the shared feature, which is the fusion result of the multimodal data to be processed.

[0017] The large-model-based multimodal data fusion method provided in the embodiments of the present specification converts the multimodal data to be processed into an embedding vector corresponding to each modality; based on a preset CNN model, extracts the specific features of the embedding vector corresponding to each modality to obtain the specific features; based on the specific features, determines the weight of the specific features through a preset reinforcement learning algorithm; based on the weight of the specific features, adds gating control, performs feature extraction on the specific features, and obtains shared features, which are the fusion results of the multimodal data to be processed, so that different types of data can be represented in a unified manner, and can dynamically adjust the information flow, dynamically adjust the fusion of different modal features, realize fast and efficient fusion of multimodal data, and maximize the utilization efficiency of information and the prediction function of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 A flowchart of a multimodal data fusion method based on a large model provided in an embodiment of this specification;

[0020] Figure 2 A framework diagram of a multimodal data fusion method based on a large model provided in an embodiment of this specification;

[0021] Figure 3 A schematic diagram of a multimodal data fusion device based on a large model provided in an embodiment of this specification;

[0022] Figure 4The structural diagram of a computer system suitable for implementing the large model-based multimodal data fusion method and device of the embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0024] Multimodal data refers to data from multiple different sources or types that together describe the same object or scene. Since multimodal data describes the same object through data from different sources or types, fusing multimodal data is beneficial to understanding the information of the research object. However, the existing fusion of multimodal data is generally processed by machine learning methods. This method generally uses shallow models and relies on a large amount of labeled data for supervised or semi-supervised training. Therefore, it usually requires a lot of preprocessing and manual intervention, and cannot generate missing data. The multimodal data fusion effect is not good.

[0025] Based on this, the embodiment of this specification provides a multimodal data fusion method based on a large model.

[0026] Figure 1 The following is a flowchart of a multimodal data fusion method based on a large model provided in the embodiments of this specification. From the program perspective, the execution subject of the process can be a program installed on an application server or an application terminal. It can be understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Figure 1 As shown, the method includes:

[0027] Step S101: convert the multimodal data to be processed into embedding vectors corresponding to each modality.

[0028] In the embodiment of the present specification, the multimodal data to be processed is data based on the same object, and the multimodal data to be processed includes: image data, and / or table data, and / or text data, and / or audio data, and / or video data;

[0029] The step of converting the multimodal data to be processed into an embedding vector corresponding to each modality specifically includes:

[0030] If the multimodal data to be processed is image data, the image data is divided into patches of preset size based on a preset ResNet, the patches of preset size are linearly transformed to generate patch embeddings, and then feature extraction is performed through a Transformer to generate an embedding vector corresponding to the image data;

[0031] If the multimodal data to be processed is tabular data, and the tabular data is numerical data, an embedding vector corresponding to the tabular data is generated by normalization; if the tabular data is categorical data, an embedding vector corresponding to the tabular data is generated by one-hot encoding;

[0032] If the multimodal data to be processed is text data, the text data is processed into tokens using the BPE word segmentation algorithm to generate an embedding vector corresponding to the text data;

[0033] If the multimodal data to be processed is audio data, the audio data is converted into features on a Mel frequency scale using Mel cepstral coefficients to generate an embedding vector corresponding to the audio data;

[0034] If the multimodal data to be processed is video data, the video data is decomposed into audio data and image data, and data conversion is performed on the decomposed audio data and the decomposed image data respectively to generate an embedding vector corresponding to the video data.

[0035] In the embodiment of the present specification, the preset ResNet includes: an input layer, a residual block, downsampling, a global average pooling and a fully connected layer, wherein the input layer includes: a 7*7 convolution layer with a step size of 2 and an output channel of 64, followed by a maximum pooling layer, the maximum pooling layer is 3*3, and the step size is 2; the residual block includes 3 bottleneck residual blocks and 4 basic residual blocks, the number of the bottleneck residual blocks is three convolution layers, wherein the convolution kernel size of the first and third convolution layers is 1*1, which is used to reduce and restore the dimension of the feature map, and the convolution kernel size of the second convolution layer is 3*3; the basic residual block contains two convolution layers, and the convolution kernel size is 3*3.

[0036] In this embodiment, the preset size patch is 7*7. The preset size patch is linearly transformed to generate patch embedding. Specifically, the preset size patch is converted into a vector of fixed dimension through a fully connected layer as patch embedding. In this embodiment, the size of the input image is 224*224. After being processed by the preset ResNet, the size of the obtained feature map is 7*7*2048, the size of each patch is 7*7*2048, and the dimension of each patch embedding is 768.

[0037] In this embodiment, the patch embedding also includes sinusoidal position coding, which is used to retain the spatial information of the patch. The encoding structure of the Transformer includes 5 encoder layers, each of which includes a multi-head self-attention mechanism and a feedforward neural network.

[0038] One-hot encoding is to convert each category value into a binary vector, in which only one element is 1 and the rest are 0, and the position of 1 represents the index of the category. In the embodiment of this specification, the specific method of one-hot encoding adopts the existing technology, which will not be repeated here.

[0039] The BPE (Byte Pair Encoding) word segmentation algorithm is a byte pair encoding word segmentation algorithm. The algorithm segments text data into subword units, selects the subwords with the highest frequency for merging, and forms new subword units, thereby updating the subword representations in the vocabulary and training data, and continuously iterates and merges until the termination condition is reached, that is, the iteration is terminated. In this embodiment, the termination condition includes: a preset number of iterations, or a preset vocabulary size, or no new subwords can be merged. Specifically, the preset number of iterations is 1000 times, and the preset vocabulary size is 100.

[0040] Step S103: Based on a preset CNN model, extract the specific features of the embedded vectors corresponding to each modality to obtain the specific features.

[0041] After converting the multimodal data to be processed into corresponding embedding vectors, it does not mean that they can be directly combined and used to fuse the multimodal data. In order to ensure that the embedding vectors corresponding to each modality of the multimodal data to be processed can be fused, it is further necessary to extract cross-modal features from the embedding vectors corresponding to each modality to obtain the unique features corresponding to each modality data, so as to obtain the unique features.

[0042] In the embodiments of this specification, the cross-modal feature extraction of the embedding vectors corresponding to each modality is implemented based on the encoder-decoder architecture of the large model. In a specific embodiment, the specific type of the large model does not constitute a specific limitation on this application. The large model is a large model with an encoder-decoder structure and can realize the fusion of multimodal data.

[0043] In the embodiments of this specification, the specific features include:

[0044] When the multimodal data to be processed is image data, the specific features are the edge, texture, color and target object in the image;

[0045] When the multimodal data to be processed is tabular data, the specific features are the mean value and standard deviation of the numerical features corresponding to the tabular data, and the mode and missing value of the categorical features;

[0046] When the multimodal data to be processed is text data, the specific features include the grammatical structure, text style, and emotional tendency corresponding to the text data;

[0047] When the multimodal data to be processed is audio data, the specific features include: pitch, loudness, and rhythm of the audio.

[0048] In the embodiments of this specification, the extraction of specific features is achieved by adding a preset CNN model to the encoder based on the large model.

[0049] In the embodiment of this specification, the specific features of the embedded vectors corresponding to each modality are extracted based on the preset CNN model to obtain the specific features, which specifically includes:

[0050] Using the embedding vectors corresponding to the modalities as the input of the preset CNN model, performing local feature extraction through the convolution layer of the preset CNN model, and obtaining a local feature extraction result;

[0051] Pooling the local feature extraction result based on the pooling layer of the preset CNN model, reducing the dimension of the local feature extraction result, and obtaining a pooling result;

[0052] The features of the intermediate layer in the pooling result are transferred to the final feature fusion layer through skip links to perform feature fusion and obtain the local feature extraction result, which is used as the specific feature.

[0053] In the embodiments of the present specification, the specific feature is a detail feature in the multimodal data to be processed, which is used to effectively distinguish features of different categories, and the local feature extraction result is used as the specific feature.

[0054] In an embodiment of the present specification, the preset CNN model is ResNet-50, and the initial convolution layer of the CNN model includes: a 7*7 convolution kernel with a step size of 2, a batch normalization layer, a ReLU activation layer and a maximum pooling layer, and the maximum pooling layer is a 3*3 pooling kernel with a step size of 2; the CNN model includes 16 residual blocks, and the residual blocks include: 1*1, 3*3 and 1*1 convolution kernels, a batch normalization layer and a ReLU function and a jump connection. The 16 residual blocks constitute a first residual layer, a second residual layer, a third residual layer and a fourth residual layer. The first residual layer includes 3 residual blocks, the second residual layer includes 4 residual blocks, the third residual layer includes 6 residual blocks, and the fourth residual layer includes 3 residual blocks; the CNN model also includes a global average pooling layer and a fully connected layer.

[0055] Step S105: Based on the specific feature, determine the weight of the specific feature through a preset reinforcement learning algorithm.

[0056] In the embodiment of the present specification, the preset reinforcement learning algorithm is a policy gradient, and the state of the policy gradient is: the currently input multimodal data features to be processed, the internal state of the current large model, and the historical actions; the action of the policy gradient is to adjust the weight of the specific features of the multimodal data to be processed, and to retain or suppress the features of a certain modality; the reward of the policy gradient is the accuracy of the classification task and the mean square error of the regression task.

[0057] In this embodiment, the preset reinforcement learning algorithm is based on adding a preset CNN model in the encoder of the aforementioned large model, and further adding a preset reinforcement learning algorithm in the encoder of the large model.

[0058] Reinforcement Learning (RL) is a machine learning method that learns the optimal behavior strategy by interacting with the environment. In reinforcement learning, the agent interacts with the environment by performing actions and adjusts its behavior based on the feedback from the environment to maximize the cumulative reward.

[0059] In reinforcement learning algorithms, agents are the subjects that learn and make decisions. The environment is the world that the agent interacts with, which can be the physical world, a virtual game, an economic system, etc. The state is the situation of the agent in the environment, which can be represented by a vector or a set of features. Actions are behaviors that the agent can take, which can be discrete or continuous. Rewards are the immediate feedback that the agent gets from the environment to evaluate the quality of its actions. Strategies are the rules for the agent to choose actions, which can be deterministic or random. The value function evaluates the long-term reward expectations after a given state or taking an action. The return is the cumulative sum of all rewards from the current time step to the future, usually with a discount factor γ to reduce the impact of future rewards.

[0060] In the embodiment of this specification, the reward of the policy gradient is a cumulative reward, and the policy gradient calculates the cumulative reward by using the Monte Carlo method.

[0061] Policy gradient is used to directly optimize policy parameters. Policy gradient optimizes policy parameters to maximize expected return.

[0062] In the embodiment of this specification, in the weights of the specific features, when the multimodal data to be processed is image data, the weights of the specific features are as follows from large to small: the weight of the target object in the image is the largest, followed by the edge of the image, then the texture of the image, and finally the color of the image.

[0063] When the multimodal data to be processed is tabular data, the weights of the specific features are as follows from large to small: the mode of the categorical feature is the largest, followed by the standard deviation of the numerical feature corresponding to the tabular data, then the mean value of the numerical feature corresponding to the tabular data, and finally the missing value of the categorical feature.

[0064] When the multimodal data to be processed is text data, the weights of the specific features are from large to small in the following order: emotional tendency, followed by grammatical structure, and finally text style.

[0065] When the multimodal data to be processed is audio data, the weights of the specific features are, from large to small, the rhythm of the audio, followed by the loudness of the audio, and finally the pitch of the audio.

[0066] Step S107: Based on the weight of the specific feature, gate control is added to extract the specific feature to obtain a shared feature, where the shared feature is a fusion result of the multimodal data to be processed.

[0067] In the embodiment of this specification, the gate control is a gated cycle unit, and the gated cycle unit includes a reset gate and an update gate;

[0068] The calculation formula of the reset gate is:

[0069] r t =σ(W r ·[h t-1 ,x t ]+b r )

[0070] in,

[0071] r t represents the output of the reset gate;

[0072] W r A weight matrix representing the reset gate;

[0073] b r represents a bias term of the reset gate;

[0074] σ represents the sigmoid function;

[0075] x t Represents the input at the current moment;

[0076] [h t-1 ,x t ] means to change the hidden state h of the previous moment t-1 and the current input x t To splice;

[0077] The calculation formula of the update gate is:

[0078] z t =σ(W z ·[h t-1 ,x t ]+b z )

[0079] in,

[0080] z t represents the output of the update gate;

[0081] W z A weight matrix representing the update gate;

[0082] b rz represents a bias term of the update gate;

[0083] σ represents the sigmoid function;

[0084] x t Represents the input at the current moment;

[0085] [h t-1 ,x t ] means to change the hidden state h of the previous moment t-1 and the current input x tMake stitching.

[0086] In the embodiments of this specification, the gate control is further integrated with the gate control based on adding a preset CNN model and a preset reinforcement learning algorithm to the large model.

[0087] Based on the weights of the aforementioned specific features, in the process of extracting shared features, the weights of the specific features are given priority, and the shared features are extracted in sequence according to the weights of the specific features. If there are inconsistent shared features, further judgment is made based on the weights of the specific features, and features with larger weights of the specific features are preferred.

[0088] In order to make the acquired shared features more accurate, HCML optimization is further performed after the output of gate control. Specifically, based on the hierarchical structure, the output of gate control at different abstract levels is learned to capture the commonalities and differences between tasks; further parameter adjustment is performed to achieve the best fusion of features.

[0089] In the embodiments of this specification, the shared features include: conceptual features, event features and attribute features;

[0090] in,

[0091] The conceptual features are features describing the same concept that exist in all the multimodal data to be processed;

[0092] The event feature is a feature describing the same event in the multimodal data to be processed, and the event feature includes: time series information, event location information and event participant information;

[0093] The attribute features are features that describe the attributes of objects in the multimodal data to be processed.

[0094] In the embodiments of this specification, it is not necessary to fuse according to the size of the feature weights, but to achieve it through weight sharing. In one embodiment, the output controlled by the gate passes through two shared fully connected layers to achieve weight sharing to ensure that different modal data have the same weights. In a specific embodiment, weight sharing can be achieved by parameter binding, that is, to ensure that the features of different modalities pay attention to each other through the attention mechanism and use the same weights, thereby ensuring the sharing of cross-modal data.

[0095] In order to further understand the multimodal data fusion method based on a large model provided by the present illustrative embodiment, the framework of the multimodal data fusion method based on a large model will be described below.

[0096] Figure 2 The framework diagram of a multimodal data fusion method based on a large model provided in the embodiments of this specification. Figure 2 As shown, the large model accepts multimodal data, specifically, multimodal data includes images, tables, text and audio, and further obtains the embedding (i.e., embedding vector) corresponding to the multimodal data, and then extracts specific features. After the specific features are initialized, the parameters are adjusted by the preset reinforcement learning algorithm to determine the weight of the specific features, and then output through gated control. After the gated control output is optimized by HCML optimization features, shared features are obtained to complete the fusion of multimodal data.

[0097] The large-model-based multimodal data fusion method provided in the embodiments of the present specification converts the multimodal data to be processed into an embedding vector corresponding to each modality; based on a preset CNN model, extracts the specific features of the embedding vector corresponding to each modality to obtain the specific features; based on the specific features, determines the weight of the specific features through a preset reinforcement learning algorithm; based on the weight of the specific features, adds gating control, performs feature extraction on the specific features, and obtains shared features, which are the fusion results of the multimodal data to be processed, so that different types of data can be represented in a unified manner, and can dynamically adjust the information flow, dynamically adjust the fusion of different modal features, realize fast and efficient fusion of multimodal data, and maximize the utilization efficiency of information and the prediction function of the model.

[0098] The above content describes in detail a multimodal data fusion method based on a large model. Correspondingly, this specification also provides a multimodal data fusion device based on a large model, such as Figure 3 shown. Figure 3 A schematic diagram of a multimodal data fusion device based on a large model provided in an embodiment of this specification, the device comprising:

[0099] A conversion module 301 converts the multimodal data to be processed into an embedding vector corresponding to each modality;

[0100] A unique feature extraction module 303 extracts unique features of the embedding vectors corresponding to each modality based on a preset CNN model to obtain unique features;

[0101] A weight determination module 305, based on the specific feature, determines the weight of the specific feature through a preset reinforcement learning algorithm;

[0102] The fusion module 307 adds gate control based on the weight of the specific feature, performs feature extraction on the specific feature, and obtains a shared feature, where the shared feature is a fusion result of the multimodal data to be processed.

[0103] Figure 4The structural diagram of a computer system suitable for implementing the large model-based multimodal data fusion method and device of the embodiment of the present disclosure is schematically shown.

[0104] like Figure 4 As shown, the computer system 400 includes a processing unit 401, which can perform various processes in the above-mentioned embodiments according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage part 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the system 400 are also stored. The processing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0105] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. The drive 810 is also connected to the I / O interface 405 as needed. Removable media 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive 410 as needed, so that the computer program read therefrom is installed into the storage section 408 as needed. Among them, the processing unit 401 can be implemented as a processing unit such as a CPU, a GPU, a TPU, an FPGA, an NPU, etc.

[0106] In particular, according to an embodiment of the present disclosure, the method described above can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes a program code for executing the above method. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 409, and / or installed from a removable medium 411.

[0107] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0108] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, and non-volatile computer storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0109] The apparatus, electronic device, non-volatile computer storage medium and method provided in the embodiments of this specification correspond to each other, and therefore, the apparatus, electronic device and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, electronic device and non-volatile computer storage medium will not be repeated here.

[0110] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0111] The specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media including storage devices.

[0112] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0113] The above is only an embodiment of this specification and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A multimodal data fusion method based on a large model, characterized in that: The multimodal data fusion method comprises: Convert the multimodal data to be processed into embedding vectors corresponding to each modality; Based on a preset CNN model, extract the specific features of the embedded vectors corresponding to each modality to obtain the specific features; Based on the specific feature, determine the weight of the specific feature through a preset reinforcement learning algorithm; Based on the weight of the specific feature, a gate control is added to perform feature extraction on the specific feature to obtain a shared feature, where the shared feature is a fusion result of the multimodal data to be processed.

2. The multimodal data fusion method according to claim 1, characterized in that: The multimodal data to be processed are data based on the same object, and the multimodal data to be processed include: image data, and / or table data, and / or text data, and / or audio data, and / or video data; The step of converting the multimodal data to be processed into an embedding vector corresponding to each modality specifically includes: If the multimodal data to be processed is image data, the image data is divided into patches of preset size based on a preset ResNet, the patches of preset size are linearly transformed to generate patch embeddings, and then feature extraction is performed through a Transformer to generate an embedding vector corresponding to the image data; If the multimodal data to be processed is tabular data, and the tabular data is numerical data, an embedding vector corresponding to the tabular data is generated by normalization; if the tabular data is categorical data, an embedding vector corresponding to the tabular data is generated by one-hot encoding; If the multimodal data to be processed is text data, the text data is processed into tokens using the BPE word segmentation algorithm to generate an embedding vector corresponding to the text data; If the multimodal data to be processed is audio data, the audio data is converted into features on a Mel frequency scale using Mel cepstral coefficients to generate an embedding vector corresponding to the audio data; If the multimodal data to be processed is video data, the video data is decomposed into audio data and image data, and data conversion is performed on the decomposed audio data and the decomposed image data respectively to generate an embedding vector corresponding to the video data.

3. The multimodal data fusion method according to claim 1, characterized in that: The specific features include: When the multimodal data to be processed is image data, the specific features are the edge, texture, color and target object in the image; When the multimodal data to be processed is tabular data, the specific features are the mean value and standard deviation of the numerical features corresponding to the tabular data, and the mode and missing value of the categorical features; When the multimodal data to be processed is text data, the specific features include the grammatical structure, text style, and emotional tendency corresponding to the text data; When the multimodal data to be processed is audio data, the specific features include: pitch, loudness, and rhythm of the audio.

4. The multimodal data fusion method according to claim 1, characterized in that: The method of extracting the specific features of the embedding vectors corresponding to each modality based on the preset CNN model to obtain the specific features specifically includes: Using the embedding vectors corresponding to the modalities as the input of the preset CNN model, performing local feature extraction through the convolution layer of the preset CNN model, and obtaining a local feature extraction result; Pooling the local feature extraction result based on the pooling layer of the preset CNN model, reducing the dimension of the local feature extraction result, and obtaining a pooling result; The features of the intermediate layer in the pooling result are transferred to the final feature fusion layer through skip links to perform feature fusion and obtain the local feature extraction result, which is used as the specific feature.

5. The multimodal data fusion method according to claim 1, characterized in that: The preset CNN model is ResNet-50, and the initial convolution layer of the CNN model includes: a 7*7 convolution kernel with a step size of 2, a batch normalization layer, a ReLU activation layer and a maximum pooling layer, and the maximum pooling layer is a 3*3 pooling kernel with a step size of 2; the CNN model includes 16 residual blocks, and the residual blocks include: 1*1, 3*3 and 1*1 convolution kernels, a batch normalization layer and a ReLU function and a jump connection. The 16 residual blocks constitute the first residual layer, the second residual layer, the third residual layer and the fourth residual layer. The first residual layer includes 3 residual blocks, the second residual layer includes 4 residual blocks, the third residual layer includes 6 residual blocks, and the fourth residual layer includes 3 residual blocks; the CNN model also includes a global average pooling layer and a fully connected layer.

6. The multimodal data fusion method according to claim 1, characterized in that: The preset reinforcement learning algorithm is a policy gradient, and the state of the policy gradient is: the currently input multimodal data features to be processed, the internal state of the current large model, and the historical actions; the action of the policy gradient is to adjust the weights of the specific features of the multimodal data to be processed, and to retain or suppress the features of a certain modality; the reward of the policy gradient is the accuracy of the classification task and the mean square error of the regression task.

7. The multimodal data fusion method according to claim 6, characterized in that: The reward of the policy gradient is a cumulative reward, and the policy gradient calculates the cumulative reward by using a Monte Carlo method.

8. The multimodal data fusion method according to claim 1, characterized in that: The gate control is a gated cycle unit, and the gated cycle unit includes a reset gate and an update gate; The calculation formula of the reset gate is: r t =σ(W r ·[h t-1 ,x t ]+b r ) in, r t represents the output of the reset gate; W r A weight matrix representing the reset gate; b r represents a bias term of the reset gate; σ represents the sigmoid function; x t Represents the input at the current moment; [h t-1 ,x t ] means to change the hidden state h of the previous moment t-1 and the current input x t To splice; The calculation formula of the update gate is: z t =σ(W z ·[h t-1 ,x t ]+b z ) in, z t represents the output of the update gate; W z A weight matrix representing the update gate; b rz represents a bias term of the update gate; σ represents the sigmoid function; x t Represents the input at the current moment; [h t-1 ,x t ] means to change the hidden state h of the previous moment t-1 and the current input x t Make stitching.

9. The multimodal data fusion method according to claim 1, characterized in that: The shared features include: conceptual features, event features and attribute features; in, The conceptual features are features describing the same concept that exist in all the multimodal data to be processed; The event feature is a feature describing the same event in the multimodal data to be processed, and the event feature includes: time series information, event location information and event participant information; The attribute features are features that describe the attributes of objects in the multimodal data to be processed.

10. A multimodal data fusion device based on a large model, characterized in that: The multimodal data fusion device comprises: The conversion module converts the multimodal data to be processed into the embedding vector corresponding to each modality; A specific feature extraction module, based on a preset CNN model, extracts specific features of the embedding vectors corresponding to each modality to obtain specific features; A weight determination module, based on the specific feature, determines the weight of the specific feature through a preset reinforcement learning algorithm; The fusion module adds gate control based on the weight of the specific feature, extracts the specific feature, and obtains the shared feature, which is the fusion result of the multimodal data to be processed.

Images