Intelligent synaptic fusion system and fusion method

Abstract

Multi-modal data fusion faces challenges in the aspects of cross-modal semantic understanding and scene self-adaption, and particularly has defects in the aspects of real-time processing, dynamic feedback and global control. Although multiple fusion methods exist, limitation still exists in cross-modal alignment, semantic consistency, dynamic adaptability and the like. Therefore, the invention provides a multi-mode fusion system based on flexible control, namely an intelligent synaptic fusion system. The model is composed of a data processing layer, a flexible control layer and a scene production layer, wherein the data processing layer enhances scene adaptability of data by optimizing coding and priority; the flexible control layer simulates synaptic regulation based on a dynamic adaptive neural network to realize information transmission and global control; the scene production layer improves feature extraction and fusion capabilities and learning and memory effects through fine-grained feature extraction, deep fusion and feedback modules. Experiments show that the research is superior to a baseline model in multiple indexes, and Acc, F1, Recall, AUC and IoU are respectively improved by 6.8%, 2.2%, 4.6%, 3.7% and 0.3%.

Description

Technical Field

[0001] The embodiments of the present invention relate to a method for intelligently fusing multimodal data through an intelligent synaptic fusion system to improve data feature quality, and more particularly to an intelligent intelligent synaptic fusion system and a fusion method. Background Art

[0002] Driven by the era of big data, data not only shows large-scale growth, but also shows diverse modalities, which brings new challenges and opportunities to data processing and analysis. With the development of large-scale artificial intelligence models, the field of multimodal data processing has also developed rapidly. Multimodal data fusion technology has therefore become a key research area. However, existing multimodal fusion models based on large models face many problems and challenges in practical applications, such as complex model management, weak adaptive ability, delayed information feedback and processing, and insufficient generalization ability.

[0003] Among these challenges, existing multimodal fusion models are generally difficult to achieve effective flat management, mainly because the complex structure of these models makes the management process complex and difficult to optimize. In addition, these models have poor adaptability, especially when facing different data environments, and often require substantial adjustments to the internal structure of the model. In terms of information feedback and processing, existing models are also unable to conduct effective real-time monitoring, resulting in delayed response. In addition, the performance of the model on different data sets is inconsistent and the generalization ability is low. In terms of feature extraction, existing models are generally unable to fully extract the underlying features, which results in the inability to completely fuse the features in the fusion stage. Multimodal fusion models also lack memory of data and environment, which requires frequent parameter adjustments when facing different environments, increasing the complexity of operations.

[0004] In view of the above problems, the present invention proposes a new multimodal data fusion system and fusion method, the Intelligent Synaptic Fusion System (ISFS), which aims to solve these problems through global dynamic real-time control. The Intelligent Synaptic Fusion System uses the Dynamic Adaptive Neural Network (DANN) proposed in this study as an intelligent control agent to achieve adaptive regulation, dynamic adjustment and real-time supervision, thereby effectively improving the model's adaptability, generalization and information processing speed. In addition, the fine-grained feature extraction module of the Intelligent Synaptic Fusion System combines CNN and Laplace technology, and integrates the adaptive feature screening strategy to enhance the deep extraction capability of features, so that the Intelligent Synaptic Fusion System can more effectively capture the underlying subtle features and improve the thoroughness of feature fusion. The Intelligent Synaptic Fusion System also significantly improves the fusion effect between features by amplifying the correlation of modal features twice. Finally, based on the Feedback Adaptive Memory Network (FAMN) proposed in this paper, by adding a historical feature playback module, the Intelligent Synaptic Fusion System enhances the memory ability of data and environment, so that the model can better adapt to different operating environments and data. These innovations give the intelligent synaptic fusion system significant advantages in accuracy, efficiency and robustness of multimodal data fusion.

[0005] Technical issues: With the widespread application of multimodal data in various fields, how to effectively fuse data of different modalities to achieve efficient information extraction and decision-making has become a hot and difficult issue in current research. Traditional multimodal data fusion methods often have the following technical problems when dealing with complex and changing environments.

[0006] a. It is difficult to achieve effective flat management of multimodal fusion methods: because current multimodal methods are built based on the principles of large models, these methods are internally complex and inefficient to manage.

[0007] b. Lack of real-time adaptability: In a dynamically changing environment, existing multimodal data fusion methods are difficult to adjust in real time according to environmental changes, resulting in unstable performance.

[0008] c. Insufficient feature extraction accuracy: When faced with data of different modalities, traditional methods are often unable to fully extract the underlying more delicate features, especially when processing data of different modalities in the same data source, which can easily cause information loss or ambiguity.

[0009] d. Difficulty in deep fusion: The feature correlations of multimodal data at different levels are complex. Traditional feature fusion methods are difficult to effectively fuse features at a deep level, resulting in insufficient accuracy of the fusion results.

[0010] e. Insufficient model memory: Current multimodal data fusion methods have limited memory capacity when processing long-term continuous data and cannot effectively use historical information to enhance current decision-making capabilities. In particular, the method's learning ability is weak when processing edge and detail information.

[0011] f. Difficulty in intelligent implementation: In any process of traditional multimodal data fusion methods, human intervention is relatively frequent, which leads to a lot of uncertainty. Human energy is limited, which leads to reduced efficiency. Summary of the invention

[0012] Technical solution: In view of the above technical problems, the present invention proposes an Intelligent Synaptic Fusion System (ISFS), which is an intelligent multimodal data fusion method that can be flattened. The Dynamic Adaptive Neural Network (DANN) proposed in this study is used as an intelligent agent to realize global dynamic real-time supervision, global dynamic adjustment and global adaptation in the flexible control layer, which reflects a great degree of intelligence, thereby improving the efficiency, accuracy and adaptability of multimodal data fusion. The model includes three main planes: data processing layer, flexible control layer and scene production layer.

[0013] 1. Data processing layer: The data processing layer is responsible for receiving and preliminarily processing multimodal data from various sensors or data sources. The flexible control layer defines the priority of the input data to ensure the integrity and transmission speed of the data. After that, the data is transmitted to the scene production layer under the real-time supervision and control of the flexible control layer. The data processing layer serves as the entrance of the information flow and provides data support for the flexible control layer and the scene production layer.

[0014] 2. Flexible control layer: The flexible control layer is the core of the present invention. The Dynamic Adaptive Neural Network (DANN) proposed in this study is used as an intelligent agent to achieve global dynamic control. The flexible control layer is responsible for defining the priority of data and dynamically adjusting the data processing strategy according to changes in the environment and data characteristics to ensure the optimal processing of data in the scene production layer. The specific functions are as follows:

[0015] a. Global dynamic real-time supervision: Monitor the data flow of the data processing layer in real time, and dynamically adjust the data priority order according to the importance and urgency of the data to ensure that important data is processed first. In addition, the entire method is dynamically monitored in real time to ensure that the method can respond correctly when dealing with different environments and different data sets.

[0016] b. Global dynamic adjustment: Based on the feedback from the data processing layer and the scene production layer, dynamically adjust processing parameters such as data priority, feature extraction method, and feature fusion strategy to adapt to changes in the environment and data characteristics.

[0017] c. Global Adaptation: The flexible control layer can automatically adjust method parameters according to changes in data characteristics and environmental requirements to ensure the best performance of the method in complex environments.

[0018] 3. Scenario production layer: The scenario production layer is the execution layer for data feature extraction, fusion, and learning memory, and consists of the following three main modules:

[0019] a. Fine-grained feature extraction module (FGFEM): The fine-grained feature extraction module is responsible for extracting fine-grained underlying and high-level features from multimodal data. In particular, it can extract delicate features that are deeply hidden in the underlying layer. FGFEM combines CNN and Laplace technology, and incorporates adaptive feature screening strategies to enhance the deep feature extraction capability. The fine-grained feature extraction module can select the optimal feature extraction strategy according to different multimodal data types. For different modal presentations of the same data, the fine-grained feature extraction module uses convolutional neural networks (CNN) to extract clear low-level features, uses Laplace operators to extract high-level fuzzy features, and uses unsupervised generative adversarial networks (GAN) for defuzzification; for multimodal data with zero correlation and scattered attribute distribution, the fine-grained feature extraction module uses the self-attention mechanism in Transformer for classification, and combines label filtering methods to extract data of specific categories; for multimodal data with strong correlation, the fine-grained feature extraction module uses a joint embedding autoencoder for feature extraction.

[0020] b. Deep Feature Fusion Module (DFFM): The deep feature fusion module is responsible for deep fusion of multimodal data features at different levels. First, the deep feature fusion module uses Kernel Canonical Correlation Analysis (KCCA) to perform correlation analysis on features with strong correlation, and combines it with the attention weighted strategy for fusion; for features with weak surface correlation, the deep feature fusion module uses Deep Canonical Correlation Analysis (DCCA) for correlation analysis, and then performs attention weighted fusion. Through layer-by-layer fusion, the deep feature fusion module can extract clear, accurate, and high-quality fusion features.

[0021] c. Comprehensive Information Feedback Module (GIFM): The main function of the comprehensive information feedback module is to enhance the model's memory learning ability and improve the model's ability to learn edge and detail information. Based on the Feedback Adaptive Memory Network (FAMN) proposed in this paper as a feedback neural network, the comprehensive information feedback module can feed back the learned historical information and edge detail information to the flexible control layer, realize real-time dynamic adjustment and optimization, and enhance the adaptability and robustness of the method.

[0022] Through the above technical solutions, the intelligent synaptic fusion system of the present invention realizes efficient fusion of multimodal data, especially showing excellent real-time adaptive capabilities and high-precision feature extraction and fusion effects in complex and changeable environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is the overall framework of the multimodal data fusion method of the present invention - Intelligent Synaptic Fusion System (ISFS).

[0024] Figure 2 It is the intelligent body framework in the flexible control layer of the multimodal data fusion method of the present invention - Intelligent Synaptic Fusion System (ISFS).

[0025] Figure 3 It is the framework of the fine-grained feature extraction module (FGFEM) in the scene production layer of the multimodal data fusion method of the present invention - Intelligent Synaptic Fusion System (ISFS).

[0026] Figure 4The present invention presents an embodiment feature extraction work for different modalities in the same data.

[0027] Figure 5 The present invention is faced with the feature extraction work of a multimodal data embodiment with zero correlation and scattered attribute distribution.

[0028] Figure 6 The present invention is faced with multimodal data, which is a multimodal data embodiment feature extraction work in which each modal data has a strong correlation.

[0029] Figure 7 It is the framework of the deep feature fusion module (DFFM) in the scene production layer of the multimodal data fusion method of the present invention - Intelligent Synaptic Fusion System (ISFS).

[0030] Figure 8 In order to solve the strong correlation problem after extracting features, the present invention uses kernel canonical correlation analysis (KCCA) to perform correlation analysis, and then combines it with attention weighting strategy to perform fusion work.

[0031] Fig. 9 In the present invention, after the feature embodiment with weak surface correlation after feature extraction, Deep Canonical Correlation Analysis (DCCA) performs correlation analysis, and then performs attention weighted fusion work.

[0032] Fig.10 It is the framework of the comprehensive information feedback module (GIFM) in the scene production layer of the multimodal data fusion method of the present invention - the intelligent synaptic fusion system (ISFS).

[0033] Fig.11 and Fig.12 This is a comparative experiment of the present invention's implementation methods facing different data sets.

[0034] Fig.13 This is a summary diagram of the multimodal data fusion method of the present invention - Intelligent Synaptic Fusion System (ISFS). DETAILED DESCRIPTION

[0035] The technical solution of the embodiment of the present invention is further described in detail below through the drawings and examples.

[0036] The embodiment of the present invention provides an intelligent multi-modal data fusion method for global dynamic real-time control. The performance of the model is verified based on five different data sets. The overall structure of the method is as follows: Figure 1 shown.

[0037] The significance of multimodal data fusion: Multimodal data fusion is not only an effective means to improve data analysis and processing capabilities, but also a key technology to promote the development of intelligent methods to a higher level. Through multimodal data fusion, the comprehensiveness of information, the accuracy of decision-making, and the robustness and adaptability of methods can be achieved, providing strong technical support and innovation power for various complex application scenarios.

[0038] Step 1: Data format consistency

[0039] For multimodal datasets, it is necessary to convert the multimodal datasets into a unified data representation. This paper converts the input multimodal datasets into vectors. The input multimodal datasets in this paper are images, text, audio and video. This paper adopts the following methods:

[0040] V modality =f embed (x modality )

[0041] Among them, x modality represents the original multimodal data, f embed represents the transfer function, V modality Represents the transformed vector representation.

[0042] Step 2: Define data priorities

[0043] The data processing layer is responsible for the preliminary processing and transmission of data, and the flexible control layer prioritizes data according to user needs to ensure that key information can be processed first. The present invention adopts the following data priority adjustment, and the formula is as follows:

[0044]

[0045] Among them, X i is the key vector data d i Score. X j represents each score when summing all modal data, where j is the index, traversing all data vectors. X i It is usually determined based on the characteristics of a specific data set. The flexible control layer determines the characteristics and then controls X i Indexation This makes the high-scoring data have a higher relative probability, so it will be processed first. λ is a parameter used to adjust X iThe scaling of λ is adjusted according to the degree of priority difference required. A larger λ enhances the priority of tasks with high scores, while a smaller λ reduces the priority difference, which is suitable for situations where you want to treat each task more equally. Attention is added through the attention weight learned by the training model, which can adaptively adjust the priority when processing multimodal data in vector form.

[0046] Step 3: The flexible control layer realizes global real-time monitoring, global dynamic adjustment and global adaptation.

[0047] Global real-time monitoring requires a dynamic adaptive neural network (DANN) that can process and monitor data from different modalities in real time. This paper implements real-time supervision by extending the state update formula of the dynamic adaptive neural network. The formula is as follows:

[0048]

[0049] Among them, S new (t) represents the updated network status at time t. old (t) represents the old network state before time t. α represents the decay coefficient, which controls the decay rate of the network state. S(τ) represents the state of the network at time τ. ω i Represents the input signal X i The weight of (τ). b 0 Represents the initial bias. β represents the attenuation coefficient of the bias. S(t) represents the network state at time t. γ represents the coefficient that controls the strength of the nonlinear term. δ i Represents the input signal X i (t) is the nonlinear weight. In the intelligent synaptic fusion system, ω of this formula i ·X i (τ) will change according to the input data format. The attenuation coefficient α determines the attenuation speed of the state. If α is too large, the state decays too quickly and it is difficult to maintain an effective response to the input signal. If it is too small, the state update is too slow and the response is delayed; ω i Determines the proportion of influence of different modes in state update, and these weights need to be adjusted according to specific tasks to ensure that important modes are given proper attention. Through the integral operation, this formula allows the state S(t) of the dynamically adaptive neural network to be updated according to the input signal within the time window, thereby achieving real-time supervision of the input data.

[0050] In order to achieve global dynamic adjustment, the dynamic adaptive neural network needs to adjust its parameters according to the supervisory feedback. The parameters can be updated by the following method. The formula is as follows:

[0051]

[0052] Among them, Θ new Represents the updated network parameters. old Indicates the old network parameters before the update. η 0 Represents the initial learning rate, which is used to control the step size of parameter update. β represents the learning rate decay coefficient, which controls the decay of the learning rate over time. t represents the current time or the number of iterations. γ represents the coefficient of the nonlinear term, which controls the strength of the nonlinear adjustment. S(t) represents the state of the network at time t, which represents the dynamic characteristics of the network. ω i Represents the weight of the i-th input signal, which controls the influence of the signal on parameter update. i (S i ,Y i ) represents the loss function. The innovation lies in realizing global dynamic optimization of model parameters through state perception, historical dependence and multimodal weight allocation, so that it can adapt to changes in different scenarios and improve overall performance.

[0053] Global adaptation achieves adaptive adjustment of parameters through adaptive learning rules within the time window. The following is the method:

[0054]

[0055] Among them, Θ new Represents the updated network parameters. old Represents the old network parameters before the update. β represents the adaptive learning rate attenuation coefficient, which controls the attenuation of the learning rate over time. Represents the dynamic adjustment factor, which depends on the time τ and the network state S(τ), and is used to adjust the step size of the parameter update. ω i (τ) represents the weight of the i-th input signal at time τ, which is used to adjust the step size of parameter update. The state-dependent weight function helps the model adjust parameter updates according to its own state changes, so that the model can adapt to the changing environment more quickly and enhance the response speed and adaptability to external changes. This formula uses integral operations within the time window to comprehensively consider the impact of historical data on parameter adjustment, so that the dynamic adaptive neural network can make adaptive adjustments based on data feedback over a long period of time, thereby improving the ability to process future data.

[0056] The present invention uses the dynamic adaptive neural network proposed in this study as an intelligent agent in the flexible control layer, which not only enhances the dynamic adjustment and adaptive capabilities of the network model, but also enhances the model's ability to handle unknown environments and unforeseen events. The intelligent agent framework is as follows Figure 2 shown.

[0057] Step 4: Fine-grained feature extraction module extracts features

[0058] In an embodiment of the present invention, a fine-grained feature extraction module (FGFEM) of a scene production layer in a multi-modal data fusion model of an intelligent global dynamic real-time control is provided. The structure of the fine-grained feature extraction module (FGFEM) is as follows: Figure 3 shown.

[0059] Step 401: When faced with multimodal data that displays different modes in the same data, such as a part of the image being clear and a part being blurred, a part of the audio being clear and a part being blurred, ambiguous expressions in the text, and partial distortion in the video, the following method will be used:

[0060] (1) In this invention, the clear and shallow feature information on the surface is regarded as low-level features, and the more blurred the information is, the higher the level. A convolutional neural network (CNN) is used to extract the low-level features in this case:

[0061]

[0062] Among them, V X is the input data in vector form, which can have any number of dimensions, such as H×W×C (image), L×D (text), or T×H×W×C (video). K is the convolution kernel, whose dimension matches the local structure of the data to be processed (e.g. 1D for text, 2D for images and audio, and 3D for video). Y is the feature output after convolution, whose dimension depends on the dimension of the input data and the shift of the convolution kernel. 1 ,d 2 ,…,d n and i 1 ,i 2 ,…,i m Represent the dimensionality indicators of data and convolution kernel respectively. The shape and size of the convolution kernel are adjusted according to specific modalities and task requirements to effectively extract features from different types of data.

[0063] (2) Next, the Laplace operator will be used to extract features from various modes. Since the fuzzy information of different modes is presented in different forms, the Laplace operator will also extract them differently. The changes here are all made by the flexible control layer according to the dynamic adjustment of the data mode.

[0064]

[0065] Where M represents the vector set of the input modality. For the image modality (I), M is a set of feature vectors of each pixel. For the text modality (T), M is the vector of each word or character in the text encoded by embedding (such as BERT). For the audio modality (A), M is the set of feature vectors converted into frequency domain after segmenting the audio signal. For the video modality (V), M is the feature vector set of each frame. i Represents the vector value of the modal data at the i-th position. Represents the Laplace operator operation on the i-th vector value. This formula can be used to extract edge and blur information from all modal data (image, text, audio, video).

[0066] (3) Both low-level and high-level features are extracted, and the next step is to deblur the high-level features, that is, feature enhancement. For the feature enhancement of the fuzzy features, that is, deblurring, the present invention adopts an unsupervised generative adversarial network (GAN) method to process it.

[0067] a. Adversarial and reconstruction loss formula

[0068] For each mode m, the optimization objective is defined as follows:

[0069]

[0070] in, Represents vectorized multimodal input data, such as vector representations of images, text, audio, or video. represents the noise vector. m The generator of modality m is responsible for generating Generate the clearest possible data. m The discriminator representing the modality m is responsible for distinguishing whether the data it receives is from the real fuzzy data distribution. Or the generator G m This is the core formula of the Generative Adversarial Network (GAN), which describes the generator G m and the discriminator D m Overall, this optimization goal sets a dynamic balance point, so that the generator tries its best to generate data that is sufficient to deceive the discriminator, while the discriminator tries its best to distinguish the generated data, thereby pushing the generator to further improve the quality of the generated images, and finally achieving high-quality deblurring feature enhancement effect.

[0071] b. Cycle consistency loss formula

[0072]

[0073] in, Represents the cycle consistency loss, which is used to measure how similar the input data is to the original data after special enhancement. Represents the vectorized multi-modal input data. m Represents a generator. The inverse operation of the generator. This formula ensures that when the blurred image x m Through the generator G m When converted into clear data, it goes through a reverse process When converting back to a blurred image, the result should be close to the original blurred data

[0074] Through adversarial and reconstruction losses, the generator is trained to generate clear data that looks real, while through cycle consistency loss, it is ensured that these generated data are consistent with the original blurry data in terms of information and structure. This approach is particularly suitable for relying on the model's internal learning to remove blur when there are no real clear samples. Through these mechanisms, the model not only learns how to deal with blur, but also ensures that the authenticity and integrity of the image are preserved during the sharpening process.

[0075] The above method can solve the problem of displaying the same data in different modes. The present invention verifies the above method using image data, such as Figure 4 shown.

[0076] Step 402: When the multimodal data has zero correlation between the modal data and the attribute distribution of the data of each modality is very scattered, the following method is used:

[0077] (1) Classification is first performed to classify each modality. The classifier used in the present invention is the self-attention mechanism in the transformer mechanism. The self-attention mechanism enables the model to capture long-distance dependencies in the input data and is applicable to various types of data, including text, images, and sounds.

[0078] a. Input representation

[0079] First, for the input sequence X = [x 1 ,x 2 ,…,x n ], first map it to a high-dimensional space and get the corresponding query Q, key K and value V representation:

[0080] Q=X X W Q ,K=X X W K ,V=v X W V

[0081] Among them, v XRepresents the feature vector of each modal data input. Q represents the query vector (Query), which represents the query representation of the input after linear transformation. K represents the key vector (Key), which represents the key representation of the input after linear transformation. V represents the value vector (Value), which represents the value representation of the input after linear transformation. W Q , W K , W V is the weight matrix of the science department. It is used to transform the input feature v X Mapping to vector space.

[0082] b. Self-Attention Layer

[0083]

[0084] Where Q is the query vector used to find relevance or importance. K is the key vector used to calculate the similarity of each query with other elements of the input sequence. V is the value vector, which is the value in the input sequence and will eventually be weighted and summed to form the output. T Represents the transposed matrix of the K vector. k It represents the dimension of the key vector, which is used to scale the multiplication result to prevent the internal multiplication result from being too large and causing the gradient vanishing problem. The softmax function is used for normalization to ensure that the sum of all attention weights is 1.

[0085] c. Feedforward Network

[0086] Each self-attention layer is followed by a feed-forward network (FFN), which usually consists of two linear transformations and an activation function (such as ReLU):

[0087] FFN(v X )=max(0,v X W 1 +b 1 )W 2 +b 2

[0088] Among them, W 1 and W 2 is the weight matrix, b 1 and b 2 is the bias term, max(0,v X W 1 +b 1 ) is the ReLU activation function, which is used to introduce nonlinearity. In this way, FFN further enhances the expressiveness of features based on the global information extracted by the self-attention layer, which is crucial to the performance of the entire Transformer architecture.

[0089] d. Output layer

[0090] The output of the feedforward network is converted into a classification prediction through an output layer, usually using a softmax function to output the probability of each category:

[0091] p i =Output=softmax(v last W c +b c )

[0092] Among them, v last Represents the feature representation of the input data in the last layer of the model after vectorization. ,W c and b c are the weights and biases of the output layer. After classifying each modality independently, the model outputs the probability p of each sample belonging to its respective category. i Based on these probabilities, the category with the highest probability can be selected as the predicted category for each sample. The final classification label is C = {c 1 ,c 2 ,…,c n}.

[0093] e. Data and label reorganization

[0094] For each sample x i The corresponding classification label c i Combine to form a new data structure S′:

[0095] D′={(x 1 ,c 1 ),(x 2 ,c 2 ),…,(x n ,c n )}

[0096] Among them, (x i ,c i ) represents a pair of original data samples and classification labels. In this way, multimodal data can be classified according to the classification labels.

[0097] (2) After using the classifier transformer to classify the multimodal data, if you want to extract data of a specific category from the harmoniously classified multimodal data set, you can use the label filtering method.

[0098] a. Determine the category label

[0099] According to the probability, the predicted class label of each sample Given by:

[0100]

[0101] in, Indicates the predicted category label of the i-th sample, indicating the category predicted by the model. argmax means returning the variable index that maximizes the value of a function. i represents the predicted probability of the i-th category.

[0102] b. Apply threshold

[0103] If necessary, threshold filtering can also be applied to increase the strictness of the selection. For example, a sample is selected only if the highest probability exceeds a certain threshold θ:

[0104]

[0105] c. Data extraction

[0106] For each category c that you want to extract, filter out samples that are predicted to be c and meet the threshold condition:

[0107]

[0108] Among them, D c represents the extracted dataset of a specific category c, x i By using this method, we can effectively and accurately extract the category data of interest from large-scale multi-category datasets to support further analysis or application.

[0109] The above method can solve the problem of feature extraction when the correlation of each modal data is zero and the attribute distribution of each modal data is very scattered. The present invention uses four modal data with zero correlation of each modal data to verify the model, such as Figure 5 shown.

[0110] Step 403: When faced with multimodal data in which each modality is highly correlated, the present invention uses a joint embedding autoencoder, which focuses on extracting interrelated and highly correlated features from different modalities.

[0111] (1) Joint Embedding Autoencoder Components

[0112] a. Multimodal encoder:

[0113] Merge information from different modalities into a shared latent space. This can be achieved through weighted combination or nonlinear combination.

[0114] b. Shared latent space:

[0115] Providing a unified representation that contains relevant features of all modalities.

[0116] In this space, features from different modalities are integrated together through correlation.

[0117] c. Modality specific decoder:

[0118] Reconstruct the data of each modality separately. Mainly used for self-supervision during training to ensure that the learned features can effectively represent the original data.

[0119] (2) Methods

[0120] a. Joint encoding process:

[0121] z=f(W x v x +W y v y +b)

[0122] Among them, v x and v y Represents two different modes of vector input, W x and W y represents the weight of the corresponding modality, b represents the bias term, f represents the activation function, and z represents the joint potential representation.

[0123] b. Correlation calculation:

[0124] The present invention uses cross-modal matching loss to maximize the correlation between different modes:

[0126]

[0127] Where L represents the cross-modal matching loss, which is used to maximize the correlation between different modalities. N represents the number of samples. i and k represent index variables, from 1 to N. i represents the modal representation vector of the i-th sample. j represents the modality representation vector that matches the i-th sample. The sum in the denominator includes all possible negative sample pairs, i.e., unmatched modality representations.

[0128] c. Reconstruction process:

[0129]

[0130] Among them, V x ,V y is the weight of the decoder, c x ,c y is the bias term, g x ,g y is the decoding activation function, is the reconstructed output. The shared latent vector after z encoding represents the representation of the input data in the latent space. In this way, you can effectively exploit the correlation between different modalities without modality conversion. You can integrate the strongly correlated ones together.

[0131] The above method can solve the problem of feature extraction of highly correlated multimodal data. The present invention uses four types of correlated modal data to verify the model, such as Figure 6 shown.

[0132] Step 5: Deep Feature Fusion Module (DFFM) feature fusion.

[0133] In an embodiment of the present invention, a deep feature fusion module (DFFM) of a scene production layer in a multimodal data fusion model of an intelligent global dynamic real-time control is provided. The structure of the deep feature fusion module (DFFM) is as follows: Figure 7 shown.

[0134] (1) After the feature data set is extracted by the fine-grained feature extraction module (FGFEM), it is necessary to encode the extracted feature data set before feature fusion. Here, the convolution method is used for encoding. Different modalities require different appropriate convolution kernel sizes. Images require Conv1 x1, text requires Conv1 x1, audio requires Conv3x3 for encoding, and video requires Conv3 x3 for encoding. Conversions are all tools, and the usage methods can be searched online. The specific encoding methods are as follows:

[0135] F modality =σ(W*v modality +b)

[0136] Among them, F modality represents the encoded features, σ is a nonlinear activation function such as ReLU, W and b represent the weight and bias of the convolution kernel respectively, * represents the convolution operation, v modality Represents the vector input of the mode. In this way, the data of each mode is converted into a modal feature, which contains both modality-specific information and is suitable for further processing.

[0137] (2) First, perform attention weighted fusion on the features with strong correlation. The feature set used here is extracted by the fine-grained feature extraction module (FGFEM). The specific method is as follows:

[0138] a. Correlation analysis

[0139] Here we use Kernel Canonical Correlation Analysis (KCCA) to analyze the correlation:

[0140]

[0141] Among them, α and β represent vectors, representing the two kernel matrices K X and K Y The relevant weight coefficient. This formula can be used to obtain the maximum correlation between the two feature sets f(X) and g(Y) in the kernelized feature space.

[0142] b. Attention Weighted Fusion

[0143] ① Calculate attention weight:

[0144] γ=σ(a(f(v x );θ f )),δ=σ(a(g(v y );θ g ))

[0145] Among them, γ represents a weight parameter in the model, which is the result calculated by the normalization function. δ represents another weight parameter, which is also the result calculated by the normalization function. σ is usually represented by a normalization function (such as the softmax function) to ensure that the sum of the weights is 1. f(v x ) represents the input feature v through the function f x The result after processing. g(v y ) represents the input feature v through the function g y The result after processing. a(·) represents a parameterized function, θ f and θ g represents the learned parameters.

[0146] ② Fusion:

[0147] Z H =γ·f(X)+δ·g(Y)

[0148] Among them, γ represents a weight parameter in the model, which represents the result calculated by the normalization function. δ represents another weight parameter, which also represents the result calculated by the normalization function. The multiplication here is element multiplication, which means that the features of each dimension are weighted by their corresponding weights. The final fused feature Z with strong correlation H Obtained by weighted merging of two eigenvectors.

[0149] (3) The next step is to fuse the features with weak surface correlation. The method used here is Deep Canonical Correlation Analysis (DCCA). The specific method is as follows:

[0150] a. Correlation analysis

[0151] In DCCA, the objective function can be:

[0152]

[0153] Among them, θ f Represents the parameters of the feature transformation function f. g Represents the parameters of the feature transformation function g. corr represents the covariance function. X and Y represent the input vector sets. var represents the variance function. This formula can measure the correlation between two sets of data after being processed by the deep network, and can maximize the features with weak surface correlation.

[0154] b. Attention Weighted Fusion

[0155] ① Calculate attention weight: The calculation process here is the same as (1).

[0156] ② Fusion:

[0157] Z l =γ·f(v x )+δ·g(v y )

[0158] Among them, Z l It represents the weakly correlated features of the final fusion, which is obtained by weighted merging of two feature vectors. γ represents the weighting coefficient, which is used to control the feature f(v x ) in the fusion process. δ represents the weighting coefficient, which is used to control the feature g(v y ) is the weight in the fusion process. x and v y They all represent feature vectors of input data, coming from different modalities.

[0159] (4) Achieve the final fusion, and combine the final fused features with strong correlation Z H The weakly correlated feature Z is finally fused l Perform the final fusion to obtain the final feature vector Z F :

[0160] Z F =Z H +Z l

[0161] The present invention uses the extracted features of different modal data in the same data to perform model verification, such as Figure 8 The present invention uses the extracted and correlated features of multiple modal data to perform model verification, such as Fig. 9 shown.

[0162] Step 6: Comprehensive Information Feedback Module (GIFM) provides overall information feedback.

[0163] In an embodiment of the present invention, a comprehensive information feedback module (GIFM) of a scene production layer in a multimodal data fusion model of an intelligent global dynamic real-time control is provided. The structure of the comprehensive information feedback module (GIFM) is as follows: Fig.10 As shown. The feedback adaptive memory network unit designed by the present invention mainly consists of a forget gate, an input gate, a unit state update, an output gate and a feedback layer. The specific method is as follows:

[0164] (1) Forget Gate

[0165] f t =σ(W f ·[h t-1 ,x t ]+b f )

[0166] Among them, f t Represents the output of the forget gate, which is used to decide which information will be forgotten. f Represents the weight of the forget gate. b f represents the bias term. σ represents the sigmoid activation function, which is used to compress the value between 0 and 1. t-1 represents the hidden state of the previous time step. t Represents the input at the current time step.

[0167] (2) Input gate and candidate memory:

[0168] i t =σ(W i ·[h t-1 ,x t ]+b i )

[0169]

[0170] Among them, i t Indicates the importance of controlling new input information. i and b i Represents the weight and bias of the input gate. W represents candidate memory cells, indicating potential new memories. C and b C Represents the weight and bias of candidate memory cells.

[0171] (3) Unit status update:

[0172]

[0173] Among them, C tRepresents the updated cell state, combining the old memory and the new candidate memory. represents candidate memory cells, indicating potential new memories. t Represents the output of the forget gate. t Indicates the importance of controlling new input information.

[0174] (4) Output gate:

[0175] o t =σ(W o ·[h t-1 ,x t ]+b o )

[0176] h t =o t *tanh(C t )

[0177] Among them, t Represents the output of the output gate, which is used to control the slave unit state C t To the hidden state h t Information flow. o and b o Represents the weight and bias of the output gate.

[0178] (5) Feedback layer:

[0179] z t =W z ·h t +b z

[0180] Among them, z t is the feedback signal received by the flexible control layer. z and b z are the weights and biases of the fully connected layer. t Represents the hidden state. A fully connected layer is used to convert the output of the output gate into a format acceptable to the flexible control layer.

[0181] Step 7: Data reconstruction

[0182] After feature extraction and feature fusion, a high-quality feature or feature set is obtained. The next step is to reconstruct the data. The present invention uses an automatic regression model to reconstruct the data. The specific method is as follows:

[0183]

[0184] in, Represents the reconstructed data, Z F represents a high-level feature set, It is an autoregressive model, which is used in the process of generating data. represents the reconstructed data of the i-th mode, z 1:i represents the features of the ith modality, t represents the time step or spatial position (pixel position for images, time step for audio and video, word or character position for text), T is the length or size of the data (number of pixels for images, number of samples for audio, length of text, or number of frames for video), It represents the probability distribution of the t-th reconstruction value given the features and the first t-1 reconstruction values. Describes the overall process from high-level features to reconstructed data, Describes in detail how data is generated step by step in this process, emphasizing the conditional probability distribution

[0185] Step 8: Comparative experiment

[0186] This paper uses five major data sets, namely BraTS2020 data set, Ch-sims data set, MOSI data set, NYU Depth v2 and Full Modal (ours), to compare the performance indicators of 18 models in terms of accuracy, F1 score, recall, area under the ROC curve (AUC), intersection over union (IoU) and multi-modal specific loss function. Fig.11 and Fig.12 shown.

[0187] An intelligent multimodal data fusion method provided in this embodiment provides a set of processes for intelligent feature extraction, intelligent fusion and intelligent learning of multimodal data to better realize cross-modal data fusion.

[0188] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention rather than to limit them. Although the embodiments of the present invention have been described in detail with reference to specific embodiments, those skilled in the art should understand that they can still modify or replace the technical solutions of the embodiments of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent synaptic fusion system for multimodal data fusion, characterized in that: include: a. The data processing layer receives input multimodal data with priority under the control of the flexible control layer; b. Flexible control layer: The dynamic adaptive neural network proposed in this study is used as an intelligent agent to perform global dynamic real-time monitoring, global real-time adjustment and global adaptation; c.Scene production layer, including: (1) A fine-grained feature extraction module to extract both prominent features and underlying subtle features; (1) A deep feature fusion module for fusing multi-level features; (2) a comprehensive information feedback module to enhance the learning of edge and detail information and feed the information back to the flexible control layer; d. Reconstruction of fused data.

2. The fusion method based on the intelligent synaptic fusion system of claim 1, characterized in that: The method for processing the initially input multimodal data includes: The flexible control layer prioritizes the initial input data and monitors it in real time.

3. The fusion method according to claim 2, characterized in that: The method for executing global dynamic real-time monitoring by the intelligent agent in the flexible control layer includes expanding the state update formula of the dynamic adaptive neural network proposed in this study. Through the integral operation, the state of the dynamic adaptive neural network can be updated according to the input signal within the time window, thereby realizing global dynamic real-time monitoring of the output model.

4. The fusion method according to claim 2, characterized in that: The global real-time adjustment method performed by the intelligent agent in the flexible control layer includes the dynamic adaptive neural network proposed in this study, which needs to adjust its parameters according to the supervisory feedback, and the parameters can be updated by the gradient descent method.

5. The fusion method according to claim 2, characterized in that: The agent in the flexible control layer performs a global adaptive method, including optimizing long-term performance according to a time-related weight function through an adaptive learning rule within a time window, so that the dynamic adaptive neural network can perform adaptive adjustments based on data feedback over a long period of time.

6. The fusion method according to claim 2, characterized in that: The feature extraction method of the fine-grained feature extraction module in the scene production layer includes: Convolutional neural networks are used to extract low-level features, and the Laplacian operator is used to extract high-level features of various modalities. An unsupervised generative adversarial network method is used for feature enhancement. The classifier used is the self-attention mechanism in the transformer mechanism. After using the classifier transformer to classify multimodal data, if you want to extract specific categories of data from the harmoniously classified multimodal data set, you can use the label filtering method. A joint embedding autoencoder is used. This type of autoencoder focuses on extracting interrelated, highly correlated, and underlying subtle features from different modalities.

7. The fusion method according to claim 2, characterized in that: The feature fusion method of the deep feature fusion module in the scene production layer includes: After the feature data set is extracted by the fine-grained feature extraction module, it is necessary to encode these extracted feature data sets before feature fusion. The convolution method is used here for encoding. First, the features with strong correlation are fused with attention weighted method, and the kernelized canonical correlation points are used for correlation analysis, and then attention weighted fusion is performed. Next, the features with weak surface correlation are fused, and the method used is deep canonical correlation analysis for correlation analysis, and then attention weighted fusion is performed.

8. The fusion method according to claim 2, characterized in that: The learning and memory method of the comprehensive information feedback module in the scene production layer includes the feedback adaptive memory network proposed in this study as a feedback neural network to enhance the model's memory learning ability and the learning ability of edge detail information.

9. The fusion method according to claim 2, characterized in that: The method for reconstructing the fused data includes using an automatic regression model to reconstruct the data.

Images