A multi-modal data fusion intelligent decision system

Abstract

The application discloses a kind of multimodal data fusion intelligent decision system, it is related to multimodal fusion technical field, decision system includes: data acquisition module, collects multimodal data from different sources;Data preprocessing module uses adaptive preprocessing algorithm, according to the characteristics of different data modalities dynamically adjusts preprocessing strategy, based on preprocessing strategy, data is preprocessed;Feature extraction module is based on feature extraction model, and data is extracted to feature;Data fusion module carries out data fusion based on graph neural network;Decision model module introduces reinforcement learning mechanism, and updates decision model;Visualization and interactive module provides friendly user interface.The application introduces reinforcement learning mechanism, so that the system can continuously learn and optimize decision strategy, according to the latest fusion features for intelligent decision, the introduction of this mechanism, significantly improve the flexibility and accuracy of decision, strong adaptability, can handle complex decision scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal fusion, and particularly relates to an intelligent decision-making system for multi-modal data fusion. BACKGROUND

[0002] In the era of rapid development of information technology, the sources and types of data are becoming increasingly diverse, especially the generation of multi-modal data. Multi-modal data refers to information from different modalities, such as text, images, audio, and sensor data. The fusion of these data is crucial for the construction of intelligent decision-making systems. Traditional data processing methods usually analyze single-modal data and cannot fully exploit the interrelationships and potential information between modalities. Multi-modal data is diverse and complementary, providing more comprehensive information. For example, in the medical field, doctors not only need patients' text records but also medical images and laboratory test results. This diversity of data provides rich context for decision-making but also poses challenges for data processing and analysis.

[0003] Existing intelligent decision-making systems for multi-modal data fusion have low decision-making intelligence and low efficiency, and cannot provide personalized services for users. SUMMARY

[0004] To solve the above technical problems, the present application provides an intelligent decision-making system for multi-modal data fusion, which solves the above problems.

[0005] To achieve the above purposes, the technical scheme adopted by the present application is as follows:

[0006] An intelligent decision-making system for multi-modal data fusion, the decision-making system comprising:

[0007] A data acquisition module: collects multi-modal data from different sources and performs preliminary processing at the data source based on edge computing technology;

[0008] A data preprocessing module: the data preprocessing module is electrically connected with the data acquisition module, and the data preprocessing module is used to use an adaptive preprocessing algorithm to dynamically adjust the preprocessing strategy according to the characteristics of different data modalities, and to preprocess the data based on the preprocessing strategy;

[0009] A feature extraction module: the feature extraction module is electrically connected with the data preprocessing module, and the feature extraction module is used to extract features from the data based on a feature extraction model;

[0010] A data fusion module: the data fusion module is electrically connected with the feature extraction module, and the data fusion module is used to fuse data based on a graph neural network, capture complex relationships and graph structure information between different modalities, and fuse data;

[0011] Decision model module: the decision model module is electrically connected with the data fusion module, and the decision model module is used for introducing a reinforcement learning mechanism, updating a decision model, and formulating an intelligent decision based on the decision model and fused features;

[0012] Visualization and interaction module: the visualization and interaction module is electrically connected with the decision model module and the data fusion module, and the visualization and interaction module is used for providing a friendly user interface and displaying decision results and related data analysis.

[0013] Preferably, the data acquisition module specifically comprises:

[0014] Data source identification unit: identifying different types of data sources and defining accessible data sources;

[0015] Data acquisition unit: acquiring data from the identified data sources, connecting with the connection interface of each data source, and setting the acquisition frequency;

[0016] Edge computing processing unit: cleaning and formatting the collected raw data, and discarding redundant irrelevant data;

[0017] Data labeling and context information adding unit: adding context information to the collected data, adding time stamp and basic information such as source identification;

[0018] Data transmission unit: transmitting the data after preliminary processing to the data preprocessing module.

[0019] Preferably, the data preprocessing module specifically comprises:

[0020] Data type identification unit: identifying the modality of input data, including image, text and time sequence, and identifying using metadata and basic features;

[0021] Feature analysis unit: analyzing the features of different data modalities, wherein the data modality feature analysis formula is:

[0022]

[0023] In the formula, v is a measure of the variation degree of the data set, n is the total number of data points in the data set, x i is the value of the i-th data point in the data set, and m is the mean value of the data set;

[0024] Adaptive preprocessing strategy generation unit: dynamically generating a preprocessing strategy based on the data modality feature analysis result;

[0025] Data cleaning unit: removing noise and missing values in the raw data;

[0026] Data conversion unit: standardize and independently encode data;

[0027] Data enhancement unit: data augmentation of image and text data modalities based on image transformation techniques.

[0028] Preferably, the feature extraction module specifically includes:

[0029] Feature extraction model establishment unit: based on the pre-processed data, calculate the edge intensity of the image and the local binary of the image, and based on the calculation of the edge intensity of the image and the local binary of the image, establish a data feature extraction model;

[0030] Feature output unit: based on the output of the data feature extraction model, extract the edge features and texture features of the image, and output the features to the data fusion module.

[0031] Preferably, the feature extraction model establishment unit specifically includes:

[0032] The coordinate pixel value of the image is taken as the input, and the edge intensity of the image is taken as the output;

[0033] The coordinate pixel value of the image is taken as the input, and the edge intensity of the image is taken as the output;

[0034] Wherein, the formula of the data feature extraction model is:

[0035]

[0036] In the formula, E(x, y) is the edge intensity at the coordinate (x, y) in the image, I is the pixel value of the image, is the gradient of the image in the x direction, is the gradient of the image in the y direction, L(x, y) is the local binary pattern value calculated at the coordinate (x, y), I j is the pixel value of the j neighborhood, I c is the pixel value of the c neighborhood, p is the total number of pixels in the neighborhood, s(I j -I c ) is the sign function.

[0037] Preferably, the data fusion module specifically includes:

[0038] Input unit: receiving edge features and texture features from the feature extraction unit, and organizing these features into a graph structure;

[0039] Graph construction unit: taking the feature vector of each modality as a node in the graph, constructing edges according to the relationship between different modalities, and assigning weights;

[0040] A graph neural network model unit: based on a graph convolution network model, defining the rules of message passing, including information aggregation of neighbor nodes, each node updates its features after receiving the message of the neighbor, based on the pooling operation, extracting global features from all nodes through iteration;

[0041] A fusion feature generation unit: based on the weighted average method, fusing the global features with the features of other modalities to generate the final fusion features.

[0042] Preferably, the fusion feature generation unit specifically includes:

[0043] Global features obtained from the graph neural network;

[0044] Based on the global features obtained from the graph neural network and the feature vectors from different modalities, the dimensions are unified;

[0045] Define the weight for each modal feature, and use the weighted average formula to fuse the features.

[0046] Preferably, the definition of the weight for each modal feature and the use of the weighted average formula to fuse the features specifically includes:

[0047] Wherein, the weighted average formula is:

[0048]

[0049] In the formula, h 总 is the fusion feature vector, w is the weight of the global feature f, f is the global feature vector extracted from the graph neural network, N represents the number of different modalities, w i is the weight of the i-th modal feature h, h i is the i-th modal feature vector.

[0050] Preferably, the decision model module specifically includes:

[0051] A definition of environment and state space unit: establishing the environment of the decision model, defining the state of the decision problem, including the current system state and the fusion feature vector;

[0052] A reward mechanism design unit: design a reward mechanism to evaluate the goodness of each action in a given state;

[0053] An initialization decision model unit: select a reinforcement learning algorithm, and initialize the parameters of the decision model, generate a fusion feature vector based on weighted average as state information input into the decision model;

[0054] An update decision model unit: based on the new state, action and reward information, use the reinforcement learning algorithm to update the parameters of the decision model.

[0055] Preferably, the visualization and interaction module specifically includes:

[0056] Data preparation and processing unit: obtain the required data from the decision model and data source, process missing values and outliers;

[0057] Data visualization unit: use visualization tools to draw charts according to prepared data, show decision results and related data analysis;

[0058] Interactive design unit: add interactive functions to the visualization chart, including hover tips, filtering, zooming and exporting functions;

[0059] Show decision result unit: design area in user interface to show overall results of decision, including key indicators, scores and trends, provide detailed information and analysis of decision process, including decision basis and influencing factors;

[0060] Data analysis function unit: filter and view data according to different conditions,

[0061] Provide data export options to allow users to export visualization results and data analysis to Excel, CSV or PDF format.

[0062] Compared with the prior art, the beneficial effects of the present application are:

[0063] The present application proposes to collect multi-modal information from different data sources in real time, and to realize preliminary processing of data by using edge computing technology, to ensure the timeliness and accuracy of data. This capability enables the system to respond quickly in a dynamic environment and provide up-to-date information support. The decision model module introduces a reinforcement learning mechanism, which enables the system to continuously learn and optimize decision strategies, and make intelligent decisions based on the latest fusion features. The introduction of this mechanism significantly improves the flexibility and accuracy of decision-making, and has strong adaptability, which can handle complex decision-making scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 The system internal framework of the present application is shown in the figure. DETAILED DESCRIPTION

[0065] The following description is used to disclose the present application so that those skilled in the art can implement it. The preferred embodiments in the following description are only as examples, and other obvious modifications can be made by those skilled in the art.

[0066] Referring to Figure 1 The multi-modal data fusion intelligent decision system shown in the figure, the decision system includes:

[0067] Data acquisition module: Collect multi-modal data from different sources, and perform preliminary processing at the data source based on edge computing technology.

[0068] Data preprocessing module: The data preprocessing module is electrically connected with the data acquisition module, and is used for dynamically adjusting a preprocessing strategy according to characteristics of different data modalities by using an adaptive preprocessing algorithm, and preprocessing data based on the preprocessing strategy.

[0069] Feature extraction module: The feature extraction module is electrically connected with the data preprocessing module, and is used for performing feature extraction on the data based on a feature extraction model.

[0070] Data fusion module: The data fusion module is electrically connected with the feature extraction module, and is used for performing data fusion based on a graph neural network, capturing complex relationships and graph structure information between different modalities, and performing data fusion.

[0071] Decision model module: The decision model module is electrically connected with the data fusion module, and is used for introducing a reinforcement learning mechanism, updating a decision model, and formulating an intelligent decision based on the decision model and the fused features.

[0072] Visualization and interaction module: The visualization and interaction module is electrically connected with the decision model module and the data fusion module, and is used for providing a friendly user interface to display decision results and related data analysis.

[0073] Further, the data acquisition module specifically includes:

[0074] Data source identification unit: Identifies different types of data sources and defines accessible data sources.

[0075] Data acquisition unit: Acquires data from the identified data sources, connects with the connection interfaces of the data sources, and sets the acquisition frequency.

[0076] Edge computing processing unit: Cleans and formats the acquired raw data, and discards redundant and irrelevant data.

[0077] Data labeling and context information adding unit: Adds context information to the acquired data, adds time stamp and basic information such as source identification.

[0078] Data transmission unit: Transmits the preliminarily processed data to the data preprocessing module.

[0079] Further, the data preprocessing module specifically includes:

[0080] Data type identification unit: identify the modality of input data, including image, text and time series, use metadata and basic features to identify;

[0081] Feature analysis unit: analyze the characteristics of different data modalities, where the data modality characteristic analysis formula is:

[0082]

[0083] In the formula, v is a measure of the degree of variation of the data set, n is the total number of data points in the data set, x i is the value of the i-th data point in the data set, and m is the mean of the data set;

[0084] Adaptive preprocessing strategy generation unit: dynamically generate preprocessing strategies based on data modality characteristic analysis results;

[0085] Data cleaning unit: remove noise and missing values in raw data;

[0086] Data conversion unit: standardize and independently encode the data;

[0087] Data augmentation unit: based on image transformation technology, data augmentation of image and text data modalities.

[0088] Further, the feature extraction module specifically includes:

[0089] Feature extraction model establishment unit: based on preprocessed data, calculate the edge intensity of the image and the local binary of the image, based on the calculation of the edge intensity of the image and the local binary of the image, establish the data feature extraction model;

[0090] Feature output unit: based on the output of the data feature extraction model, extract the edge features and texture features of the image, and output the features to the data fusion module.

[0091] Further, the feature extraction model establishment unit specifically includes:

[0092] The coordinate pixel value of the image is taken as the input, and the edge intensity of the image is taken as the output;

[0093] The coordinate pixel value of the image is taken as the input, and the edge intensity of the image is taken as the output;

[0094] Wherein, the formula of the data feature extraction model is:

[0095]

[0096] In the formula, E(x,y) is the edge intensity at coordinate (x,y) in the image, I is the pixel value of the image, is the gradient of the image in the x direction, is the gradient of the image in the y direction, L(x, y) is a local binary pattern value calculated at coordinates (x, y), I j is the pixel value of the j-neighborhood, I c is the pixel value of the c-neighborhood, p is the total number of pixels in the neighborhood, s(I j -I c is a sign function.

[0097] Further, the data fusion module specifically comprises:

[0098] An input unit: receiving edge features and texture features from the feature extraction unit, and organizing these features into a graph structure;

[0099] A graph construction unit: taking the feature vector of each modality as a node in the graph, constructing edges according to the relationship between different modalities, and assigning weights;

[0100] A graph neural network model unit: based on a graph convolution network model, defining the rules of message passing, including information aggregation of neighbor nodes, each node updates its features after receiving the message of neighbors, based on pooling operation, global features are extracted from all nodes through iteration;

[0101] A fusion feature generation unit: based on the weighted average method, fusing the global features with the features of other modalities to generate the final fusion features.

[0102] Further, the fusion feature generation unit specifically comprises:

[0103] Global features obtained from the graph neural network;

[0104] Based on the global features obtained from the graph neural network, the dimension of the feature vectors from different modalities is unified;

[0105] Defining weights for each modality feature, and using a weighted average formula to fuse the features.

[0106] Further, the defining weights for each modality feature and using a weighted average formula to fuse the features specifically comprises:

[0107] Wherein, the weighted average formula is:

[0108]

[0109] In the formula, h 总 is the fusion feature vector, w is the weight of the global feature f, f is the global feature vector extracted from the graph neural network, N represents the number of different modalities, w i is the weight of the i-th modality feature h, h iis the i-th modal feature vector.

[0110] Further, the decision model module specifically includes:

[0111] Define the environment and state space unit: establish the environment of the decision model, define the state of the decision problem, including the current system state and the fused feature vector;

[0112] Reward mechanism design unit: design a reward mechanism to evaluate the goodness of each action in a given state;

[0113] Initialize the decision model unit: select a reinforcement learning algorithm and initialize the parameters of the decision model, generate a fused feature vector based on weighted average as state information input into the decision model;

[0114] Update the decision model unit: update the parameters of the decision model using the reinforcement learning algorithm according to the new state, action and reward information.

[0115] Further, the visualization and interaction module specifically includes:

[0116] Data preparation and processing unit: obtain the required data from the decision model and data sources, and process missing values and outliers;

[0117] Data visualization unit: use visualization tools to draw charts based on prepared data, and show decision results and related data analysis;

[0118] Interactive design unit: add interactive functions to the visualization chart, including hover tips, filtering, zooming and exporting functions;

[0119] Show the decision result unit: design an area in the user interface to display the overall result of the decision, including key indicators, scores and trends, provide detailed information and analysis of the decision process, including decision basis and influencing factors;

[0120] Data analysis function unit: filter and view data according to different conditions,

[0121] Provide data export options to allow users to export visualization results and data analysis in Excel, CSV or PDF format.

[0122] In summary, the advantages of the present application are:

[0123] Through the data acquisition module, the system can collect multi-modal information from different data sources in real time, and use edge computing technology to realize preliminary processing of data, ensuring the timeliness and accuracy of data. This ability enables the system to quickly respond in a dynamic environment and provide the latest information support;

[0124] The data preprocessing module dynamically adjusts the preprocessing strategy according to the characteristics of different data modalities through an adaptive preprocessing algorithm, which can effectively handle noise and missing values in different types of data, improve data quality, and ensure the reliability of subsequent analysis and decision-making;

[0125] The feature extraction module uses advanced feature extraction models to extract key information from the preprocessed data, especially the edge features and texture features of images. This process can deeply mine the internal laws and patterns of data and provide high-quality feature input for data fusion;

[0126] The data fusion module is designed based on graph neural networks, which can effectively capture the complex relationships and graph structure information between different modalities. Through this technology, the system can realize deep-level data fusion and promote comprehensive integration and comprehensive analysis of information;

[0127] The decision model module introduces a reinforcement learning mechanism, which enables the system to continuously learn and optimize decision strategies based on the latest fusion features for intelligent decision-making. The introduction of this mechanism significantly improves the flexibility and accuracy of decision-making, and has strong adaptability, which can handle complex decision-making scenarios;

[0128] The visualization and interaction module provides a friendly interface and interactive experience for users. Through visualization tools, it displays decision results and related data analysis, allowing users to easily understand the basis and influencing factors of decisions. It also supports data filtering, filtering, and export, improving user decision-making efficiency.

[0129] The above shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above examples, and the above examples and descriptions in the specification are only the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements fall within the scope of the claimed present application. The scope of protection claimed by the present application is defined by the appended claims and their equivalents.

Claims

1. An intelligent decision-making system based on multimodal data fusion, characterized in that, The decision-making system includes: Data acquisition module: used to collect multimodal data from different sources, and performs preliminary processing at the source of data generation based on edge computing technology; Data preprocessing module: The data preprocessing module is electrically connected to the data acquisition module. The data preprocessing module is used to use an adaptive preprocessing algorithm to dynamically adjust the preprocessing strategy according to the characteristics of different data modalities, and preprocess the data based on the preprocessing strategy. Feature extraction module: The feature extraction module is electrically connected to the data preprocessing module. The feature extraction module is used to extract features from the data based on the feature extraction model. Data fusion module: The data fusion module is electrically connected to the feature extraction module. The data fusion module is used to perform data fusion based on graph neural networks, capture the complex relationships and graph structure information between different modalities, and perform data fusion. Decision Model Module: The decision model module is electrically connected to the data fusion module. The decision model module is used to introduce a reinforcement learning mechanism, update the decision model, and make intelligent decisions based on the decision model and the fused features. Visualization and Interaction Module: The visualization and interaction module is electrically connected to the decision model module and the data fusion module. The visualization and interaction module is used to provide a user-friendly interface to display decision results and related data analysis. The data preprocessing module specifically includes: Data type recognition unit: Recognizes the modality of input data, including images, text, and time series, using metadata and basic features for recognition; Feature Analysis Unit: Analyzes the features of different data modalities. The formula for data modality feature analysis is as follows: In the formula, v is a measure of the variability of the dataset, n is the total number of data points in the dataset, and x... i Let m be the value of the i-th data point in the dataset, and m be the mean of the dataset. Adaptive preprocessing strategy generation unit: dynamically generates preprocessing strategies based on data modality feature analysis results; Data cleaning unit: Removes noise and missing values ​​from the original data; Data transformation unit: Standardizes and independently encodes the data; Data augmentation unit: Performs data augmentation on image and text data modalities based on image transformation technology; The data fusion module specifically includes: Input unit: Receives edge and texture features from the feature extraction unit and organizes these features into a graph structure; Graph construction unit: The feature vector of each modality is used as a node in the graph, edges are constructed according to the relationships between different modalities, and weights are assigned. Graph Neural Network Model Unit: Based on the graph convolutional network model, it defines the rules for message passing, including the aggregation of information from neighboring nodes. Each node updates its features after receiving a message from a neighbor. Based on pooling operations, it iterates to extract global features from all nodes. Fusion Feature Generation Unit: Based on a weighted average method, global features are fused with features from other modalities to generate the final fused features; The fusion feature generation unit specifically includes: Global features obtained from graph neural networks; The dimensions are unified based on the global features obtained from the graph neural network and the feature vectors from different modalities; Define weights for each modality feature and fuse the features using a weighted average formula; The step of defining weights for each modality feature and using a weighted average formula to fuse features specifically includes: The weighted average formula is as follows: In the formula, h 总 To fuse the feature vectors, w represents the weight of the global feature f, f is the global feature vector extracted from the graph neural network, and N represents the number of different modalities. i Let h be the weight of the i-th modal feature h. i Let be the eigenvector of the i-th modality.

2. The intelligent decision-making system for multimodal data fusion according to claim 1, characterized in that, The data acquisition module specifically includes: Data source identification unit: Identifies different types of data sources and defines the data sources that can be accessed; Data acquisition unit: Collects data from the identified data sources, connects to the connection interfaces of each data source, and sets the acquisition frequency; Edge computing processing unit: Cleans and formats the collected raw data, discarding redundant and irrelevant data; Data annotation and context information appending unit: Applies context information to the collected data, adding basic information such as timestamps and source identifiers; Data transmission unit: transmits the pre-processed data to the data preprocessing module.

3. The intelligent decision-making system for multimodal data fusion according to claim 1, characterized in that, The feature extraction module specifically includes: Feature extraction model building unit: Based on the preprocessed data, calculate the edge intensity and local binary values ​​of the image, and build a data feature extraction model based on the calculation of the edge intensity and local binary values ​​of the image. Feature Output Unit: Based on the output of the data feature extraction model, extracts the edge features and texture features of the image and outputs the features to the data fusion module.

4. The intelligent decision-making system for multimodal data fusion according to claim 3, characterized in that, The feature extraction model building unit specifically includes: The image's coordinate pixel values ​​are used as input, and the image's edge intensity is used as output. The input is the pixel value of the coordinate neighborhood of the image, and the output is the local binary pattern value of the image. The formula for the data feature extraction model is as follows: In the formula, E(x,y) is the edge intensity at coordinates (x,y) in the image, and I is the pixel value of the image. Let be the gradient of the image in the x-direction. Let L(x,y) be the gradient of the image in the y-direction, and let L(x,y) be the local binary pattern value calculated at coordinates (x,y). j Let I be the pixel value of the j-neighborhood. c Let be the pixel value of the c-neighborhood, p be the total number of pixels in the neighborhood, and s(I) be the pixel value of the c-neighborhood. j -I c ) is a symbolic function.

5. The intelligent decision-making system for multimodal data fusion according to claim 1, characterized in that, The decision model module specifically includes: Define the environment and state space unit: establish the environment of the decision model, define the state of the decision problem, including the current state of the system and the fused feature vector; Reward Mechanism Design Unit: Design reward mechanisms to evaluate the merits of each action in a given state; Initialize the decision model unit: Select a reinforcement learning algorithm and initialize the parameters of the decision model. Generate a fusion feature vector based on weighted average as the state information input into the decision model. Update decision model unit: Update the parameters of the decision model using reinforcement learning algorithms based on the new state, action and reward information.

6. The intelligent decision-making system for multimodal data fusion according to claim 1, characterized in that, The visualization and interaction module specifically includes: Data preparation and processing unit: Obtains the required data from the decision model and data source, and processes missing and outlier values; Data Visualization Unit: Using visualization tools, create charts based on prepared data to display decision results and related data analysis; Interaction Design Unit: Add interactive features to visual charts, including hover tools, filtering, zooming, and exporting functions; Display Decision Results Unit: Design an area in the user interface to display the overall results of the decision, including key indicators, scores and trends, and provide detailed information and analysis of the decision-making process, including the basis for the decision and influencing factors; Data analysis function unit: Allows users to filter and view data based on different criteria, and provides data export options, allowing users to export visualization results and data analysis to Excel, CSV or PDF formats.

Images