A method and apparatus for multimodal data fusion based on a large model

Abstract

This specification discloses a method and apparatus for multimodal data fusion based on a large model, comprising: converting multimodal data to be processed into embedding vectors corresponding to each modality; extracting specific features from the embedding vectors corresponding to each modality based on a preset CNN model to obtain specific features; determining the weights of the specific features based on the specific features using a preset reinforcement learning algorithm; and adding gating control based on the weights of the specific features to extract features from the specific features to obtain shared features, wherein the shared features are the fusion result of the multimodal data to be processed.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method and apparatus for multimodal data fusion based on a large model. Background Technology

[0002] In the field of artificial intelligence, the most common multimodal data includes text, images, audio, and video. Each modality has its unique characteristics and information, and combining these different modalities can provide a more comprehensive understanding and richer information.

[0003] In existing technologies, the fusion of multimodal data is generally processed using machine learning methods. These methods typically use shallow models and rely on a large amount of labeled data for supervised or semi-supervised training. Therefore, they usually require a lot of preprocessing and manual intervention and cannot generate missing data.

[0004] Based on this, this specification provides a method and apparatus for multimodal data fusion based on a large model. Summary of the Invention

[0005] This specification provides a method and apparatus for multimodal data fusion based on a large model to solve the following technical problems: In the prior art, the fusion of multimodal data is generally processed using machine learning methods. These methods generally use shallow models and rely on a large amount of labeled data for supervised or semi-supervised training. Therefore, they usually require a lot of preprocessing and manual intervention, and cannot generate missing data.

[0006] To solve the above-mentioned technical problems, the embodiments in this specification are implemented as follows:

[0007] This specification provides an embodiment of a multimodal data fusion method based on a large model, including:

[0008] The multimodal data to be processed is converted into embedding vectors corresponding to each modality;

[0009] Based on a preset CNN model, specific features of the embedding vectors corresponding to each modality are extracted to obtain specific features;

[0010] Based on the specific features, the weights of the specific features are determined through a preset reinforcement learning algorithm;

[0011] Based on the weights of the specific features, gating control is added to extract features from the specific features to obtain shared features, which are the fusion results of the multimodal data to be processed.

[0012] This specification also provides an embodiment of a multimodal data fusion device based on a large model, including:

[0013] The conversion module converts the multimodal data to be processed into embedding vectors corresponding to each modality.

[0014] The specific feature extraction module, based on a preset CNN model, extracts specific features from the embedding vectors corresponding to each modality to obtain specific features;

[0015] The weight determination module determines the weight of the specific feature based on the specific feature using a preset reinforcement learning algorithm.

[0016] The fusion module adds gating control based on the weights of the specific features, extracts features from the specific features, and obtains shared features, which are the fusion results of the multimodal data to be processed.

[0017] The multimodal data fusion method based on a large model provided in this specification converts the multimodal data to be processed into embedding vectors corresponding to each modality; based on a preset CNN model, it extracts specific features from the embedding vectors corresponding to each modality to obtain specific features; based on the specific features, it determines the weights of the specific features through a preset reinforcement learning algorithm; based on the weights of the specific features, it adds gating control to extract features from the specific features to obtain shared features, which are the fusion results of the multimodal data to be processed. This allows different types of data to be represented uniformly and enables dynamic adjustment of information flow and the fusion of different modal features, achieving fast and efficient fusion of multimodal data and maximizing information utilization efficiency and the predictive function of the model. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a multimodal data fusion method based on a large model, provided as an embodiment of this specification;

[0020] Figure 2 A framework diagram of a multimodal data fusion method based on a large model provided in the embodiments of this specification;

[0021] Figure 3 A schematic diagram of a multimodal data fusion device based on a large model provided in the embodiments of this specification;

[0022] Figure 4The diagram illustrates the structure of a computer system suitable for implementing the large-model-based multimodal data fusion method and apparatus of the present disclosure. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0024] Multimodal data refers to data from multiple different sources or types that collectively describe the same object or scenario. Because multimodal data describes the same object through data from different sources or types, fusing multimodal data is beneficial for understanding the research subject. However, current multimodal data fusion methods generally employ machine learning approaches. These methods typically use shallow models, rely on large amounts of labeled data for supervised or semi-supervised training, and therefore usually require extensive preprocessing and manual intervention. Furthermore, they cannot generate missing data, resulting in poor multimodal data fusion performance.

[0025] Based on this, the embodiments of this specification provide a multimodal data fusion method based on a large model.

[0026] Figure 1 This is a flowchart illustrating a multimodal data fusion method based on a large model, provided in an embodiment of this specification. From a programming perspective, the execution entity of the process can be a program hosted on an application server or application terminal. It is understood that this method can be executed by any device, equipment, platform, or cluster of devices with computing and processing capabilities. Figure 1 As shown, the method includes:

[0027] Step S101: Convert the multimodal data to be processed into embedding vectors corresponding to each modality.

[0028] In the embodiments of this specification, the multimodal data to be processed is data based on the same object, and the multimodal data to be processed includes: image data, and / or tabular data, and / or text data, and / or audio data, and / or video data;

[0029] The process of converting the multimodal data to be processed into embedding vectors corresponding to each modality specifically includes:

[0030] If the multimodal data to be processed is image data, the image data is segmented into patches of a preset size based on a preset ResNet. The preset size patches are transformed linearly to generate patch embeddings, which are then extracted by a Transformer to generate the embedding vectors corresponding to the image data.

[0031] If the multimodal data to be processed is tabular data and the tabular data is numerical data, then the embedding vector corresponding to the tabular data is generated by normalization; if the tabular data is categorical data, then the embedding vector corresponding to the tabular data is generated by one-hot encoding.

[0032] If the multimodal data to be processed is text data, then the BPE word segmentation algorithm is used to process the text data into tokens and generate the embedding vector corresponding to the text data.

[0033] If the multimodal data to be processed is audio data, then the audio data is transformed into features on the Mel frequency scale using Mel cepstral coefficients to generate the embedding vector corresponding to the audio data.

[0034] If the multimodal data to be processed is video data, then the video data is decomposed into audio data and image data, and the audio data and image data generated by the decomposition are converted respectively to generate the embedding vector corresponding to the video data.

[0035] In the embodiments of this specification, the preset ResNet includes: an input layer, residual blocks, downsampling, global average pooling, and fully connected layers. The input layer includes: a 7*7 convolutional layer with a stride of 2 and an output channel of 64, followed by a 3*3 max pooling layer with a stride of 2. The residual blocks include 3 bottleneck residual blocks and 4 basic residual blocks. The number of bottleneck residual blocks is three convolutional layers, where the kernel size of the first and third convolutional layers is 1*1, used to reduce and restore the dimension of the feature map, and the kernel size of the second convolutional layer is 3*3. The basic residual blocks contain two convolutional layers, each with a kernel size of 3*3.

[0036] In this embodiment, the preset patch size is 7*7. The preset-size patch undergoes a linear transformation to generate a patchbedding. Specifically, the preset-size patch is converted into a fixed-dimensional vector through a fully connected layer, which serves as the patchbedding. In this embodiment, the input image size is 224*224. After processing with the preset ResNet, the resulting feature map size is 7*7*2048. Each patch is 7*7*2048 in size, and the dimension of each patch embedding is 768.

[0037] In this embodiment, the patch embedding also includes sinusoidal positional encoding, which is used to preserve the spatial information of the patch. The Transformer's encoding structure includes five encoder layers, each of which includes a multi-head self-attention mechanism and a feedforward neural network.

[0038] One-hot encoding converts each category value into a binary vector, where only one element is 1 and the rest are 0. The position of the 1 represents the index of that category. In the embodiments of this specification, the specific method of one-hot encoding adopts existing technology, which will not be described in detail here.

[0039] The BPE (Byte Pair Encoding) word segmentation algorithm divides text data into sub-word units. It selects the most frequent sub-words and merges them to form new sub-word units, thereby updating the vocabulary and sub-word representations in the training data. This iterative merging process continues until a termination condition is met. In this embodiment, the termination condition includes: a preset number of iterations, a preset vocabulary size, or no new sub-words can be merged. Specifically, the preset number of iterations is 1000, and the preset vocabulary size is 100.

[0040] Step S103: Based on the preset CNN model, extract the specific features of the embedding vectors corresponding to each modality to obtain the specific features.

[0041] Converting the multimodal data to be processed into corresponding embedding vectors does not mean that they can be directly combined for multimodal data fusion. In order to ensure that the embedding vectors corresponding to each modality of the multimodal data to be processed can be fused, it is necessary to further extract cross-modal features from the obtained embedding vectors corresponding to each modality to obtain the unique features corresponding to each modality data, thereby obtaining specific features.

[0042] In the embodiments of this specification, cross-modal feature extraction of the embedding vectors corresponding to each modality is implemented based on a large-scale encoder-decoder architecture. In specific embodiments, the specific type of the large-scale model does not constitute a specific limitation on this application; the large-scale model is a large model with an encoder-decoder structure that can achieve the fusion of multimodal data.

[0043] In the embodiments described in this specification, the specific features include:

[0044] When the multimodal data to be processed is image data, the specific features are the edges, textures, colors, and target objects in the image;

[0045] When the multimodal data to be processed is tabular data, the specific features are the mean and standard deviation of the numerical features corresponding to the tabular data, and the mode and missing values ​​of the categorical features.

[0046] When the multimodal data to be processed is text data, the specific features include the grammatical structure, text style, and sentiment tendency corresponding to the text data;

[0047] When the multimodal data to be processed is audio data, the specific features include: audio pitch, loudness, and rhythm.

[0048] In the embodiments described in this specification, the extraction of specific features is achieved by adding a preset CNN model to the encoder of a large model.

[0049] In the embodiments of this specification, the step of extracting specific features from the embedding vectors corresponding to each modality based on a preset CNN model to obtain specific features specifically includes:

[0050] The embedding vectors corresponding to each modality are used as input to the preset CNN model, and local feature extraction is performed through the convolutional layer of the preset CNN model to obtain the local feature extraction result.

[0051] The local feature extraction results are pooled based on the pooling layer of the preset CNN model to reduce the dimensionality of the local feature extraction results and obtain the pooling result.

[0052] By using skip links, the features of the intermediate layer in the pooling result are passed to the final feature fusion layer for feature fusion, and the local feature extraction result is obtained. The local feature extraction result is used as the specific feature.

[0053] In the embodiments of this specification, the specific feature is the detailed feature in the multimodal data to be processed, which is used to effectively distinguish features of different categories, and the local feature extraction result is used as the specific feature.

[0054] In the embodiments of this specification, the preset CNN model is ResNet-50. The initial convolutional layer of the CNN model includes: a 7*7 convolutional kernel with a stride of 2, a batch normalization layer, a ReLU activation layer, and a max pooling layer. The max pooling layer has a 3*3 pooling kernel with a stride of 2. The CNN model includes 16 residual blocks. The residual blocks include: 1*1, 3*3, and 1*1 convolutional kernels, a batch normalization layer, a ReLU function, and skip connections. The 16 residual blocks form a first residual layer, a second residual layer, a third residual layer, and a fourth residual layer. The first residual layer includes 3 residual blocks, the second residual layer includes 4 residual blocks, the third residual layer includes 6 residual blocks, and the fourth residual layer includes 3 residual blocks. The CNN model also includes a global average pooling layer and a fully connected layer.

[0055] Step S105: Based on the specific features, determine the weights of the specific features using a preset reinforcement learning algorithm.

[0056] In the embodiments of this specification, the preset reinforcement learning algorithm is a policy gradient, and the state of the policy gradient is: the currently input features of the multimodal data to be processed, the internal state of the current large model, and historical actions; the action of the policy gradient is to adjust the weights of the specific features of the multimodal data to be processed, and to retain or suppress the features of a certain modality; the reward of the policy gradient is the accuracy of the classification task and the mean squared error of the regression task.

[0057] In this embodiment, the preset reinforcement learning algorithm is based on adding a preset CNN model to the encoder of the aforementioned large model, and further adding a preset reinforcement learning algorithm to the encoder of the large model.

[0058] Reinforcement learning (RL) is a machine learning method that learns optimal behavioral policies by interacting with the environment. In reinforcement learning, an agent interacts with the environment by performing actions and adjusts its behavior based on feedback from the environment to maximize cumulative rewards.

[0059] In reinforcement learning algorithms, the agent is the entity that learns and makes decisions. The environment is the world the agent interacts with; it can be the physical world, a virtual game, an economic system, etc. The state is the agent's current situation in the environment, which can be represented by a vector or a set of features. An action is the behavior the agent can take; it can be discrete or continuous. The reward is the immediate feedback the agent receives from the environment, used to evaluate the merit of its action. The policy is the rule by which the agent chooses actions; it can be deterministic or stochastic. The value function evaluates the expected long-term reward after taking a given state or action. The reward is the cumulative sum of all rewards from the current time step to the future, usually discounted by a factor γ to reduce the influence of future rewards.

[0060] In the embodiments described in this specification, the reward of the policy gradient is a cumulative reward, which is calculated using the Monte Carlo method.

[0061] Policy gradient is used to directly optimize policy parameters, thereby maximizing the expected return.

[0062] In the embodiments of this specification, when the multimodal data to be processed is image data, the weights of the specific features, from largest to smallest, are as follows: the target object in the image has the largest weight, followed by the image edge, then the image texture, and finally the image color.

[0063] When the multimodal data to be processed is tabular data, the weights of the specific features from largest to smallest are as follows: the mode of the categorical features is the largest, followed by the standard deviation of the corresponding numerical features in the tabular data, then the mean of the corresponding numerical features in the tabular data, and finally the missing values ​​of the categorical features.

[0064] When the multimodal data to be processed is text data, the weights of the specific features from largest to smallest are: sentiment tendency, followed by grammatical structure, and finally text style.

[0065] When the multimodal data to be processed is audio data, the weights of the specific features from largest to smallest are: audio rhythm, followed by audio loudness, and finally audio pitch.

[0066] Step S107: Based on the weight of the specific feature, add gating control, extract features from the specific feature, and obtain shared features, which are the fusion results of the multimodal data to be processed.

[0067] In the embodiments described in this specification, the gate control is a gate control loop unit, which includes a reset gate and an update gate;

[0068] The calculation formula for the reset door is:

[0069] r t =σ(W r ·[h t-1 ,x t ]+b r )

[0070] in,

[0071] r t This indicates the output of the reset gate;

[0072] W r This represents the weight matrix of the reset gate;

[0073] b r This represents the bias term of the reset gate;

[0074] σ represents the sigmoid function;

[0075] x t This represents the input at the current moment;

[0076] [h t-1 ,x t ] indicates that the hidden state h from the previous moment is being displayed. t-1 and the input x at the current time t To splice;

[0077] The calculation formula for the update gate is as follows:

[0078] z t =σ(W z ·[h t-1 ,x t ]+b z )

[0079] in,

[0080] z t This represents the output of the update gate;

[0081] W z This represents the weight matrix of the update gate;

[0082] b rz This represents the bias term of the update gate;

[0083] σ represents the sigmoid function;

[0084] x t This represents the input at the current moment;

[0085] [h t-1 ,x t ] indicates that the hidden state h from the previous moment is being displayed. t-1 and the input x at the current time tThen, the parts are assembled.

[0086] In the embodiments described in this specification, gating control is based on the addition of a preset CNN model and a preset reinforcement learning algorithm to a large model, and further integrates gating control.

[0087] Based on the aforementioned weights of specific features, during the extraction of shared features, the weights of specific features are given priority. Shared features are extracted sequentially according to the weights of specific features. If there are inconsistent shared features, further judgment is made based on the weights of specific features, and features with higher specific feature weights are selected.

[0088] To make the acquired shared features more accurate, HCML optimization is further performed on the gating output. Specifically, based on the hierarchical structure, different levels of abstraction of the gating output are learned to capture the commonalities and differences between tasks; further parameter adjustments are made to achieve optimal feature fusion.

[0089] In the embodiments of this specification, the shared features include: conceptual features, event features, and attribute features;

[0090] in,

[0091] The conceptual features are those that describe the same concept and are present in all the multimodal data to be processed.

[0092] The event-specific features are the features in the multimodal data to be processed that describe the same event. The event-specific features include: time series information, event location information, and event participant information.

[0093] The attribute features are the features that describe the attributes of things in the multimodal data to be processed.

[0094] In the embodiments described in this specification, fusion is not performed based on the weights of the features, but rather through weight sharing. In one embodiment, the gated output passes through two shared fully connected layers to achieve weight sharing, ensuring that data from different modalities have the same weights. In a specific embodiment, weight sharing can be achieved through parameter binding, ensuring that features from different modalities attend to each other through an attention mechanism and use the same weights, thereby guaranteeing the sharing of cross-modal data.

[0095] To further understand the multimodal data fusion method based on a large model provided in this embodiment, the following explanation will be based on the framework of the multimodal data fusion method based on a large model.

[0096] Figure 2 This is a framework diagram of a multimodal data fusion method based on a large model, provided for embodiments of this specification. Figure 2 As shown, the large model accepts multimodal data, specifically including images, tables, text, and audio. It further obtains the embeddings (i.e., embedding vectors) corresponding to the multimodal data, and then extracts specific features. After initialization, the obtained specific features are adjusted using a pre-defined reinforcement learning algorithm to determine their weights. The output is then gated and optimized using HCML to obtain shared features, thus completing the fusion of multimodal data.

[0097] The multimodal data fusion method based on a large model provided in this specification converts the multimodal data to be processed into embedding vectors corresponding to each modality; based on a preset CNN model, it extracts specific features from the embedding vectors corresponding to each modality to obtain specific features; based on the specific features, it determines the weights of the specific features through a preset reinforcement learning algorithm; based on the weights of the specific features, it adds gating control to extract features from the specific features to obtain shared features, which are the fusion results of the multimodal data to be processed. This allows different types of data to be represented uniformly and enables dynamic adjustment of information flow and the fusion of different modal features, achieving fast and efficient fusion of multimodal data and maximizing information utilization efficiency and the predictive function of the model.

[0098] The above describes in detail a multimodal data fusion method based on a large model. Correspondingly, this specification also provides a multimodal data fusion device based on a large model, such as... Figure 3 As shown. Figure 3 This specification provides a schematic diagram of a multimodal data fusion device based on a large model, which includes:

[0099] The conversion module 301 converts the multimodal data to be processed into embedding vectors corresponding to each modality.

[0100] The specific feature extraction module 303 extracts specific features from the embedding vectors corresponding to each modality based on a preset CNN model to obtain specific features;

[0101] The weight determination module 305 determines the weight of the specific feature based on the specific feature using a preset reinforcement learning algorithm.

[0102] The fusion module 307 adds gating control based on the weight of the specific features, extracts features from the specific features, and obtains shared features, which are the fusion results of the multimodal data to be processed.

[0103] Figure 4The diagram illustrates the structure of a computer system suitable for implementing the large-model-based multimodal data fusion method and apparatus of the present disclosure.

[0104] like Figure 4 As shown, the computer system 400 includes a processing unit 401, which can execute various processes described above based on a program stored in a read-only memory (ROM) 402 or a program loaded from a storage section 408 into a random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the system 400. The processing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0105] The following components are connected to I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 410 as needed so that computer programs read from it can be installed into storage section 408 as needed. The processing unit 401 can be implemented as a CPU, GPU, TPU, FPGA, NPU, etc.

[0106] In particular, according to embodiments of this disclosure, the methods described above can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for performing the methods described above. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411.

[0107] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0108] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for apparatus, electronic devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0109] The apparatus, electronic device, and non-volatile computer storage medium and method provided in the embodiments of this specification are corresponding. Therefore, the apparatus, electronic device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, electronic device, and non-volatile computer storage medium will not be repeated here.

[0110] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0111] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside on local and remote computer storage media, including storage devices.

[0112] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0113] The above description is merely an embodiment of this specification and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A multimodal data fusion method based on a large model, characterized in that, The multimodal data fusion method includes: The multimodal data to be processed is converted into embedding vectors corresponding to each modality; Based on a pre-defined CNN model, specific features are extracted from the embedding vectors corresponding to each modality to obtain specific features. These specific features include: when the multimodal data to be processed is image data, the specific features are the image edges, textures, colors, and target objects in the image; when the multimodal data to be processed is tabular data, the specific features are the mean and standard deviation of the numerical features corresponding to the tabular data, and the mode and missing values ​​of the categorical features; when the multimodal data to be processed is text data, the specific features include the grammatical structure, text style, and sentiment of the text data; when the multimodal data to be processed is audio data, the specific features include the pitch, loudness, and rhythm of the audio. Based on the specific features, the weights of the specific features are determined through a preset reinforcement learning algorithm, which is a policy gradient. The state of the policy gradient consists of: the currently input multimodal data features to be processed, the internal state of the current large model, and historical actions. The action of the policy gradient is to adjust the weights of the specific features of the multimodal data to be processed, and to retain or suppress features of a certain modality. The reward of the policy gradient is the accuracy of the classification task and the mean squared error of the regression task. Based on the weights of the specific features, gating control is added to extract features from the specific features to obtain shared features. The shared features are the fusion results of the multimodal data to be processed, and include: conceptual features, event features, and attribute features. Specifically, the conceptual features are features describing the same concept that exist in all of the multimodal data to be processed; the event features are features describing the same event in the multimodal data to be processed, and include: time series information, event location information, and event participant information; the attribute features are features describing the attributes of things in the multimodal data to be processed.

2. The multimodal data fusion method as described in claim 1, characterized in that, The multimodal data to be processed is data based on the same object, and the multimodal data to be processed includes: image data, and / or tabular data, and / or text data, and / or audio data, and / or video data; The process of converting the multimodal data to be processed into embedding vectors corresponding to each modality specifically includes: If the multimodal data to be processed is image data, the image data is segmented into patches of a preset size based on a preset ResNet. The preset size patches are transformed linearly to generate patch embeddings, which are then extracted by a Transformer to generate the embedding vectors corresponding to the image data. If the multimodal data to be processed is tabular data and the tabular data is numerical data, then the embedding vector corresponding to the tabular data is generated by normalization; if the tabular data is categorical data, then the embedding vector corresponding to the tabular data is generated by one-hot encoding. If the multimodal data to be processed is text data, then the BPE word segmentation algorithm is used to process the text data into tokens and generate the embedding vector corresponding to the text data. If the multimodal data to be processed is audio data, then the audio data is transformed into features on the Mel frequency scale using Mel cepstral coefficients to generate the embedding vector corresponding to the audio data. If the multimodal data to be processed is video data, then the video data is decomposed into audio data and image data, and the audio data and image data generated by the decomposition are converted respectively to generate the embedding vector corresponding to the video data.

3. The multimodal data fusion method as described in claim 1, characterized in that, The method based on a preset CNN model extracts specific features from the embedding vectors corresponding to each modality to obtain specific features, specifically including: The embedding vectors corresponding to each modality are used as input to the preset CNN model, and local feature extraction is performed through the convolutional layer of the preset CNN model to obtain the local feature extraction result. The local feature extraction results are pooled based on the pooling layer of the preset CNN model to reduce the dimensionality of the local feature extraction results and obtain the pooling result. By using skip links, the features of the intermediate layer in the pooling result are passed to the final feature fusion layer for feature fusion, and the local feature extraction result is obtained. The local feature extraction result is used as the specific feature.

4. The multimodal data fusion method as described in claim 1, characterized in that, The preset CNN model is ResNet-50. The initial convolutional layers of the CNN model include: 7*7 convolutional kernels with a stride of 2, batch normalization layers, ReLU activation layers, and max pooling layers. The max pooling layers have 3*3 pooling kernels with a stride of 2. The CNN model includes 16 residual blocks. Each residual block includes 1*1, 3*3, and 1*1 convolutional kernels, batch normalization layers, ReLU functions, and skip connections. These 16 residual blocks form a first residual layer, a second residual layer, a third residual layer, and a fourth residual layer. The first residual layer includes 3 residual blocks, the second residual layer includes 4 residual blocks, the third residual layer includes 6 residual blocks, and the fourth residual layer includes 3 residual blocks. The CNN model also includes a global average pooling layer and fully connected layers.

5. The multimodal data fusion method as described in claim 1, characterized in that, The reward of the policy gradient is a cumulative reward, which is calculated using the Monte Carlo method.

6. The multimodal data fusion method as described in claim 1, characterized in that, The gate control is a gate control loop unit, which includes a reset gate and an update gate; The calculation formula for the reset door is: r t =σ(W r ·[h t-1 ,x t ]+b r ) in, r t This indicates the output of the reset gate; W r This represents the weight matrix of the reset gate; b r This represents the bias term of the reset gate; σ represents the sigmoid function; x t This represents the input at the current moment; [h t-1 ,x t ] indicates that the hidden state h from the previous moment is being displayed. t-1 and the input x at the current time t To splice; The calculation formula for the update gate is as follows: z t =σ(W z ·[h t-1 ,x t ]+b z ) in, z t This represents the output of the update gate; W z This represents the weight matrix of the update gate; b z This represents the bias term of the update gate; σ represents the sigmoid function; x t This represents the input at the current moment; [h t-1 ,x t ] indicates that the hidden state h from the previous moment is being displayed. t-1 and the input x at the current time t Then, the parts are assembled.

7. A multimodal data fusion device based on a large model, characterized in that, The multimodal data fusion device includes: The conversion module converts the multimodal data to be processed into embedding vectors corresponding to each modality. The specific feature extraction module, based on a preset CNN model, extracts specific features from the embedding vectors corresponding to each modality to obtain specific features. These specific features include: when the multimodal data to be processed is image data, the specific features are the image's edges, texture, color, and target objects within the image; when the multimodal data to be processed is tabular data, the specific features are the mean and standard deviation of the numerical features corresponding to the tabular data, and the mode and missing values ​​of the categorical features; when the multimodal data to be processed is text data, the specific features include the grammatical structure, text style, and sentiment of the text data; when the multimodal data to be processed is audio data, the specific features include the audio's pitch, loudness, and rhythm. The weight determination module, based on the specific features, determines the weights of the specific features using a preset reinforcement learning algorithm, namely, policy gradient. The state of the policy gradient consists of: the currently input multimodal data features to be processed, the internal state of the current large model, and historical actions. The action of the policy gradient is to adjust the weights of the specific features of the multimodal data to be processed, and to retain or suppress features of a certain modality. The reward of the policy gradient is the accuracy of the classification task and the mean squared error of the regression task. The fusion module, based on the weights of the specific features, adds gating control to extract features from the specific features to obtain shared features. These shared features are the fusion result of the multimodal data to be processed. The shared features include: conceptual features, event features, and attribute features. Specifically, the conceptual features are features describing the same concept that exist in all of the multimodal data to be processed; the event features are features describing the same event in the multimodal data to be processed, including: time series information, event location information, and event participant information; and the attribute features are features describing the attributes of things in the multimodal data to be processed.

Images