Intelligent decision-making system for multi-modal data fusion

Abstract

The invention discloses an intelligent decision-making system for multi-modal data fusion, and relates to the technical field of multi-modal fusion, and the decision-making system comprises a data collection module which collects multi-modal data from different sources; the data preprocessing module is used for dynamically adjusting a preprocessing strategy according to the characteristics of different data modes by using a self-adaptive preprocessing algorithm, and preprocessing the data based on the preprocessing strategy; the feature extraction module is used for performing feature extraction on the data based on a feature extraction model; the data fusion module is used for carrying out data fusion based on the graph neural network; the decision model module introduces a reinforcement learning mechanism and updates a decision model; and the visualization and interaction module provides a friendly user interface. According to the method, a reinforcement learning mechanism is introduced, so that the system can continuously learn and optimize a decision strategy, intelligent decision making is carried out according to latest fusion features, the flexibility and accuracy of decision making are remarkably improved through introduction of the mechanism, adaptability is high, and complex decision making scenes can be processed.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal fusion, and in particular to an intelligent decision-making system for multimodal data fusion. Background Art

[0002] In today's era of rapid development of information technology, the sources and types of data are becoming increasingly rich, especially the generation of multimodal data. Multimodal data refers to information from different modalities, such as text, images, audio, and sensor data. The fusion of these data is crucial to the construction of intelligent decision-making systems. Traditional data processing methods usually analyze data of a single modality and cannot fully explore the relationships and potential information between modalities. Multimodal data is diverse and complementary and can provide more comprehensive information. For example, in the medical field, doctors need not only text records of patients, but also medical images and laboratory test results. The diversity of this data provides a rich context for decision-making, but also brings challenges to data processing and analysis.

[0003] The existing intelligent decision-making system based on multimodal data fusion has a low intelligence level, low decision-making efficiency and cannot provide personalized services to users. Summary of the invention

[0004] In order to solve the above technical problems, an intelligent decision-making system for multimodal data fusion is provided. This technical solution solves the above problems.

[0005] In order to achieve the above purpose, the technical solution adopted by the present invention is:

[0006] An intelligent decision-making system for multimodal data fusion, the decision-making system comprising:

[0007] Data acquisition module: collects multimodal data from different sources and performs preliminary processing at the source of data generation based on edge computing technology;

[0008] Data preprocessing module: The data preprocessing module is electrically connected to the data acquisition module, and the data preprocessing module is used to use an adaptive preprocessing algorithm to dynamically adjust the preprocessing strategy according to the characteristics of different data modes, and preprocess the data based on the preprocessing strategy;

[0009] Feature extraction module: The feature extraction module is electrically connected to the data preprocessing module, and the feature extraction module is used to extract features from the data based on the feature extraction model;

[0010] Data fusion module: The data fusion module is electrically connected to the feature extraction module, and the feature extraction module is used to perform data fusion based on the graph neural network, capture the complex relationship and graph structure information between different modalities, and perform data fusion;

[0011] Decision model module: The decision model module is electrically connected to the data fusion module, and the decision model module is used to introduce a reinforcement learning mechanism, update the decision model, and make intelligent decisions based on the decision model and fused features;

[0012] Visualization and interaction module: The visualization and interaction module is electrically connected to the decision model module and the data fusion module. The visualization and interaction module is used to provide a friendly user interface to display decision results and related data analysis.

[0013] Preferably, the data acquisition module specifically includes:

[0014] Data source identification unit: identifies different types of data sources and defines accessible data sources;

[0015] Data collection unit: collects data from the identified data sources, connects to the connection interface of each data source, and sets the collection frequency;

[0016] Edge computing processing unit: cleans and formats the collected raw data and discards redundant and irrelevant data;

[0017] Data annotation and context information addition unit: adds context information to the collected data, adding basic information such as timestamp and source identification;

[0018] Data transmission unit: transmits the data after preliminary processing to the data preprocessing module.

[0019] Preferably, the data preprocessing module specifically includes:

[0020] Data type identification unit: Identify the modality of input data, including images, text, and time series, using metadata and basic features for identification;

[0021] Feature analysis unit: Analyze the features of different data modes, where the data mode feature analysis formula is:

[0022]

[0023] In the formula, v is a measure of the degree of variation of the data set, n is the total number of data points in the data set, and x i is the value of the i-th data point in the data set, and m is the mean of the data set;

[0024] Adaptive preprocessing strategy generation unit: dynamically generates preprocessing strategies based on data modal feature analysis results;

[0025] Data cleaning unit: remove noise and missing values ​​from the original data;

[0026] Data conversion unit: standardize the data and encode it independently;

[0027] Data enhancement unit: performs data enhancement of image and text data modalities based on image transformation technology.

[0028] Preferably, the feature extraction module specifically includes:

[0029] Feature extraction model building unit: based on the preprocessed data, calculate the edge strength of the image and the local binary value of the image, and build a data feature extraction model based on the calculation of the edge strength of the image and the local binary value of the image;

[0030] Feature output unit: Based on the output of the data feature extraction model, it extracts the edge features and texture features of the image and outputs the features to the data fusion module.

[0031] Preferably, the feature extraction model building unit specifically includes:

[0032] The coordinate pixel value of the image is taken as input and the edge strength of the image is taken as output;

[0033] The pixel value of the coordinate domain of the image is taken as input, and the local binary pattern value of the image is taken as output;

[0034] Among them, the formula of the data feature extraction model is:

[0035]

[0036] Where E(x,y) is the edge strength at the coordinate (x,y) in the image, I is the pixel value of the image, is the gradient of the image in the x direction, is the gradient of the image in the y direction, L(x,y) is the local binary pattern value calculated at the coordinate (x,y), I j is the pixel value of the j neighborhood, I c is the pixel value of the neighborhood of c, p is the total number of pixels in the neighborhood, s(I j -I c ) is a sign function.

[0037] Preferably, the data fusion module specifically includes:

[0038] Input unit: receives edge features and texture features from the feature extraction unit and organizes these features into a graph structure;

[0039] Graph construction unit: The feature vector of each modality is used as a node in the graph, edges are constructed based on the relationship between different modalities, and weights are assigned;

[0040] Graph neural network model unit: Based on the graph convolutional network model, it defines the rules of message transmission, including information aggregation of neighbor nodes. Each node updates its features after receiving messages from neighbors. Based on the pooling operation, it extracts global features from all nodes through iterations.

[0041] Fusion feature generation unit: Based on the weighted average method, the global features are fused with the features of other modalities to generate the final fusion features.

[0042] Preferably, the fusion feature generating unit specifically includes:

[0043] Global features obtained from graph neural networks;

[0044] Unify the dimensions of feature vectors from different modalities based on global features obtained from graph neural networks;

[0045] Define weights for each modality feature and use the weighted average formula to fuse the features.

[0046] Preferably, defining a weight for each modal feature and using a weighted average formula to fuse the features specifically includes:

[0047] The weighted average formula is:

[0048]

[0049] In the formula, w is the fusion feature vector, w is the weight of the global feature f, f is the global feature vector extracted from the graph neural network, N represents the number of different modalities, and w i is the weight of the i-th modal feature h, h i is the ith modal eigenvector.

[0050] Preferably, the decision model module specifically includes:

[0051] Define the environment and state space unit: Establish the environment of the decision model and define the state of the decision problem, including the current system state and the fused feature vector;

[0052] Reward Mechanism Design Unit: Design reward mechanisms to evaluate the goodness of each action in a given state;

[0053] Initialize the decision model unit: select the reinforcement learning algorithm, initialize the parameters of the decision model, generate a fusion feature vector based on weighted average as state information and input it into the decision model;

[0054] Update decision model unit: Use reinforcement learning algorithm to update the parameters of the decision model according to the new state, action and reward information.

[0055] Preferably, the visualization and interaction module specifically includes:

[0056] Data preparation and processing unit: obtain the required data from the decision model and data source, and handle missing values ​​and outliers;

[0057] Data Visualization Unit: Use visualization tools to draw charts based on prepared data to display decision results and related data analysis;

[0058] Interaction Design Unit: Add interactive functions to visualization charts, including floating prompts, filtering, zooming and exporting functions;

[0059] Display decision results unit: Design an area in the user interface to display the overall results of the decision, including key indicators, scores and trends, and provide detailed information and analysis of the decision process, including the basis for the decision and influencing factors;

[0060] Data analysis function unit: performs data screening, allowing users to filter and view data according to different conditions,

[0061] Provides data export options, allowing users to export visualization results and data analysis to Excel, CSV or PDF formats.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] The present invention proposes to collect multimodal information from different data sources in real time and use edge computing technology to realize preliminary data processing to ensure the timeliness and accuracy of the data. This capability enables the system to respond quickly in a dynamic environment and provide the latest information support. The decision model module introduces a reinforcement learning mechanism, which enables the system to continuously learn and optimize decision-making strategies and make intelligent decisions based on the latest fusion features. The introduction of this mechanism significantly improves the flexibility and accuracy of decision-making, has strong adaptability, and can handle complex decision-making scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is the internal framework diagram of the system of the present invention. DETAILED DESCRIPTION

[0065] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art may think of other obvious variations.

[0066] Reference Figure 1 As shown, an intelligent decision-making system for multimodal data fusion, the decision-making system includes:

[0067] Data acquisition module: collects multimodal data from different sources and performs preliminary processing at the source of data generation based on edge computing technology;

[0068] Data preprocessing module: The data preprocessing module is electrically connected to the data acquisition module, and the data preprocessing module is used to use an adaptive preprocessing algorithm to dynamically adjust the preprocessing strategy according to the characteristics of different data modes, and preprocess the data based on the preprocessing strategy;

[0069] Feature extraction module: The feature extraction module is electrically connected to the data preprocessing module, and the feature extraction module is used to extract features from the data based on the feature extraction model;

[0070] Data fusion module: The data fusion module is electrically connected to the feature extraction module, and the feature extraction module is used to perform data fusion based on the graph neural network, capture the complex relationship and graph structure information between different modalities, and perform data fusion;

[0071] Decision model module: The decision model module is electrically connected to the data fusion module, and the decision model module is used to introduce a reinforcement learning mechanism, update the decision model, and make intelligent decisions based on the decision model and fused features;

[0072] Visualization and interaction module: The visualization and interaction module is electrically connected to the decision model module and the data fusion module. The visualization and interaction module is used to provide a friendly user interface to display decision results and related data analysis.

[0073] Furthermore, the data acquisition module specifically includes:

[0074] Data source identification unit: identifies different types of data sources and defines accessible data sources;

[0075] Data collection unit: collects data from the identified data sources, connects to the connection interface of each data source, and sets the collection frequency;

[0076] Edge computing processing unit: cleans and formats the collected raw data and discards redundant and irrelevant data;

[0077] Data annotation and context information addition unit: adds context information to the collected data, adding basic information such as timestamp and source identification;

[0078] Data transmission unit: transmits the data after preliminary processing to the data preprocessing module.

[0079] Furthermore, the data preprocessing module specifically includes:

[0080] Data type identification unit: Identify the modality of input data, including images, text, and time series, using metadata and basic features for identification;

[0081] Feature analysis unit: Analyze the features of different data modes, where the data mode feature analysis formula is:

[0082]

[0083] In the formula, v is a measure of the degree of variation of the data set, n is the total number of data points in the data set, and x i is the value of the i-th data point in the data set, and m is the mean of the data set;

[0084] Adaptive preprocessing strategy generation unit: dynamically generates preprocessing strategies based on data modal feature analysis results;

[0085] Data cleaning unit: remove noise and missing values ​​from the original data;

[0086] Data conversion unit: standardize the data and encode it independently;

[0087] Data enhancement unit: performs data enhancement of image and text data modalities based on image transformation technology.

[0088] Furthermore, the feature extraction module specifically includes:

[0089] Feature extraction model building unit: based on the preprocessed data, calculate the edge strength of the image and the local binary value of the image, and build a data feature extraction model based on the calculation of the edge strength of the image and the local binary value of the image;

[0090] Feature output unit: Based on the output of the data feature extraction model, it extracts the edge features and texture features of the image and outputs the features to the data fusion module.

[0091] Furthermore, the feature extraction model building unit specifically includes:

[0092] The coordinate pixel value of the image is taken as input and the edge strength of the image is taken as output;

[0093] The pixel value of the coordinate domain of the image is taken as input, and the local binary pattern value of the image is taken as output;

[0094] Among them, the formula of the data feature extraction model is:

[0095]

[0096] Where E(x,y) is the edge strength at the coordinate (x,y) in the image, I is the pixel value of the image, is the gradient of the image in the x direction, is the gradient of the image in the y direction, L(x,y) is the local binary pattern value calculated at the coordinate (x,y), I jis the pixel value of the j neighborhood, I c is the pixel value of the neighborhood of c, p is the total number of pixels in the neighborhood, s(I j -I c ) is a sign function.

[0097] Furthermore, the data fusion module specifically includes:

[0098] Input unit: receives edge features and texture features from the feature extraction unit and organizes these features into a graph structure;

[0099] Graph construction unit: The feature vector of each modality is used as a node in the graph, edges are constructed based on the relationship between different modalities, and weights are assigned;

[0100] Graph neural network model unit: Based on the graph convolutional network model, it defines the rules of message transmission, including information aggregation of neighbor nodes. Each node updates its features after receiving messages from neighbors. Based on the pooling operation, it extracts global features from all nodes through iterations.

[0101] Fusion feature generation unit: Based on the weighted average method, the global features are fused with the features of other modalities to generate the final fusion features.

[0102] Furthermore, the fusion feature generating unit specifically includes:

[0103] Global features obtained from graph neural networks;

[0104] Unify the dimensions of feature vectors from different modalities based on global features obtained from graph neural networks;

[0105] Define weights for each modality feature and use the weighted average formula to fuse the features.

[0106] Furthermore, the weights are defined for each modal feature, and the weighted average formula is used to fuse the features, specifically including:

[0107] The weighted average formula is:

[0108]

[0109] In the formula, w is the fusion feature vector, w is the weight of the global feature f, f is the global feature vector extracted from the graph neural network, N represents the number of different modalities, and w i is the weight of the i-th modal feature h, h i is the ith modal eigenvector.

[0110] Furthermore, the decision model module specifically includes:

[0111] Define the environment and state space unit: Establish the environment of the decision model and define the state of the decision problem, including the current system state and the fused feature vector;

[0112] Reward Mechanism Design Unit: Design reward mechanisms to evaluate the goodness of each action in a given state;

[0113] Initialize the decision model unit: select the reinforcement learning algorithm, initialize the parameters of the decision model, generate a fusion feature vector based on weighted average as state information and input it into the decision model;

[0114] Update decision model unit: Use reinforcement learning algorithm to update the parameters of the decision model according to the new state, action and reward information.

[0115] Furthermore, the visualization and interaction module specifically includes:

[0116] Data preparation and processing unit: obtain the required data from the decision model and data source, and handle missing values ​​and outliers;

[0117] Data Visualization Unit: Use visualization tools to draw charts based on prepared data to display decision results and related data analysis;

[0118] Interaction Design Unit: Add interactive functions to visualization charts, including floating prompts, filtering, zooming and exporting functions;

[0119] Display decision results unit: Design an area in the user interface to display the overall results of the decision, including key indicators, scores and trends, and provide detailed information and analysis of the decision process, including the basis for the decision and influencing factors;

[0120] Data analysis function unit: performs data screening, allowing users to filter and view data according to different conditions,

[0121] Provides data export options, allowing users to export visualization results and data analysis to Excel, CSV or PDF formats.

[0122] In summary, the advantages of the present invention are:

[0123] Through the data acquisition module, the system can collect multimodal information from different data sources in real time, and use edge computing technology to perform preliminary data processing to ensure the timeliness and accuracy of the data. This capability enables the system to respond quickly in a dynamic environment and provide the latest information support;

[0124] The data preprocessing module uses an adaptive preprocessing algorithm to dynamically adjust the preprocessing strategy according to the characteristics of different data modalities. It can effectively handle noise and missing values ​​in different types of data, improve data quality, and ensure the reliability of subsequent analysis and decision-making.

[0125] The feature extraction module uses advanced feature extraction models to extract key information from preprocessed data, especially edge features and texture features of images. This process can deeply explore the inherent laws and patterns of data and provide high-quality feature input for data fusion.

[0126] The data fusion module is based on the design of graph neural network, which enables the complex relationship and graph structure information between different modalities to be effectively captured. Through this technology, the system can achieve deep fusion of data and promote comprehensive integration and comprehensive analysis of information.

[0127] The decision model module introduces a reinforcement learning mechanism, which enables the system to continuously learn and optimize decision strategies and make intelligent decisions based on the latest fusion features. The introduction of this mechanism significantly improves the flexibility and accuracy of decision-making, has strong adaptability, and can handle complex decision-making scenarios;

[0128] The visualization and interaction module provides users with a friendly interface and interactive experience. By displaying decision results and related data analysis through visualization tools, users can easily understand the decision basis and influencing factors. It also supports data screening, filtering and exporting, improving users' decision-making efficiency.

[0129] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. An intelligent decision-making system for multimodal data fusion, characterized in that: The decision-making system includes: Data acquisition module: used to collect multimodal data from different sources and perform preliminary processing at the source of data generation based on edge computing technology; Data preprocessing module: The data preprocessing module is electrically connected to the data acquisition module, and the data preprocessing module is used to use an adaptive preprocessing algorithm to dynamically adjust the preprocessing strategy according to the characteristics of different data modes, and preprocess the data based on the preprocessing strategy; Feature extraction module: The feature extraction module is electrically connected to the data preprocessing module, and the feature extraction module is used to extract features from the data based on the feature extraction model; Data fusion module: The data fusion module is electrically connected to the feature extraction module, and the feature extraction module is used to perform data fusion based on the graph neural network, capture the complex relationship and graph structure information between different modalities, and perform data fusion; Decision model module: The decision model module is electrically connected to the data fusion module, and the decision model module is used to introduce a reinforcement learning mechanism, update the decision model, and make intelligent decisions based on the decision model and fused features; Visualization and interaction module: The visualization and interaction module is electrically connected to the decision model module and the data fusion module. The visualization and interaction module is used to provide a friendly user interface to display decision results and related data analysis.

2. The intelligent decision-making system for multimodal data fusion according to claim 1, characterized in that: The data acquisition module specifically includes: Data source identification unit: identifies different types of data sources and defines accessible data sources; Data collection unit: collects data from the identified data sources, connects to the connection interface of each data source, and sets the collection frequency; Edge computing processing unit: cleans and formats the collected raw data and discards redundant and irrelevant data; Data annotation and context information addition unit: adds context information to the collected data, adding basic information such as timestamp and source identification; Data transmission unit: transmits the data after preliminary processing to the data preprocessing module.

3. The intelligent decision-making system for multimodal data fusion according to claim 2, characterized in that: The data preprocessing module specifically includes: Data type identification unit: Identify the modality of input data, including images, text, and time series, using metadata and basic features for identification; Feature analysis unit: Analyze the features of different data modes, where the data mode feature analysis formula is: In the formula, v is a measure of the degree of variation of the data set, n is the total number of data points in the data set, and x i is the value of the i-th data point in the data set, and m is the mean of the data set; Adaptive preprocessing strategy generation unit: dynamically generates preprocessing strategies based on data modal feature analysis results; Data cleaning unit: remove noise and missing values ​​from the original data; Data conversion unit: standardize the data and encode it independently; Data enhancement unit: performs data enhancement of image and text data modalities based on image transformation technology.

4. The intelligent decision-making system for multimodal data fusion according to claim 3, characterized in that: The feature extraction module specifically includes: Feature extraction model building unit: based on the preprocessed data, calculate the edge strength of the image and the local binary value of the image, and build a data feature extraction model based on the calculation of the edge strength of the image and the local binary value of the image; Feature output unit: Based on the output of the data feature extraction model, it extracts the edge features and texture features of the image and outputs the features to the data fusion module.

5. The intelligent decision-making system for multimodal data fusion according to claim 4, characterized in that: The feature extraction model building unit specifically includes: The coordinate pixel value of the image is taken as input and the edge strength of the image is taken as output; The pixel value of the coordinate domain of the image is taken as input, and the local binary pattern value of the image is taken as output; Among them, the formula of the data feature extraction model is: Where E(x,y) is the edge strength at the coordinate (x,y) in the image, I is the pixel value of the image, is the gradient of the image in the x direction, is the gradient of the image in the y direction, L(x,y) is the local binary pattern value calculated at the coordinate (x,y), I j is the pixel value of the j neighborhood, I c is the pixel value of the neighborhood of c, p is the total number of pixels in the neighborhood, s(I j -I c ) is a sign function.

6. The intelligent decision-making system for multimodal data fusion according to claim 5, characterized in that: The data fusion module specifically includes: Input unit: receives edge features and texture features from the feature extraction unit and organizes these features into a graph structure; Graph construction unit: The feature vector of each modality is used as a node in the graph, edges are constructed based on the relationship between different modalities, and weights are assigned; Graph neural network model unit: Based on the graph convolutional network model, it defines the rules of message transmission, including information aggregation of neighbor nodes. Each node updates its features after receiving messages from neighbors. Based on the pooling operation, it extracts global features from all nodes through iterations. Fusion feature generation unit: Based on the weighted average method, the global features are fused with the features of other modalities to generate the final fusion features.

7. The intelligent decision-making system for multimodal data fusion according to claim 6, characterized in that: The fusion feature generation unit specifically includes: Global features obtained from graph neural networks; Unify the dimensions of feature vectors from different modalities based on global features obtained from graph neural networks; Define weights for each modality feature and use the weighted average formula to fuse the features.

8. The intelligent decision-making system for multimodal data fusion according to claim 7, characterized in that: Defining weights for each modal feature and using a weighted average formula to fuse features specifically includes: The weighted average formula is: In the formula, w is the fusion feature vector, w is the weight of the global feature f, f is the global feature vector extracted from the graph neural network, N represents the number of different modalities, and w i is the weight of the i-th modal feature h, h i is the ith modal eigenvector.

9. The intelligent decision-making system for multimodal data fusion according to claim 8, characterized in that: The decision model module specifically includes: Define the environment and state space unit: Establish the environment of the decision model and define the state of the decision problem, including the current system state and the fused feature vector; Reward Mechanism Design Unit: Design reward mechanisms to evaluate the goodness of each action in a given state; Initialize the decision model unit: select the reinforcement learning algorithm, initialize the parameters of the decision model, generate a fusion feature vector based on weighted average as state information and input it into the decision model; Update decision model unit: Use reinforcement learning algorithm to update the parameters of the decision model according to the new state, action and reward information.

10. The intelligent decision-making system for multimodal data fusion according to claim 9, characterized in that: The visualization and interaction module specifically includes: Data preparation and processing unit: obtain the required data from the decision model and data source, and handle missing values ​​and outliers; Data Visualization Unit: Use visualization tools to draw charts based on prepared data to display decision results and related data analysis; Interaction Design Unit: Add interactive functions to visualization charts, including floating prompts, filtering, zooming and exporting functions; Display decision results unit: Design an area in the user interface to display the overall results of the decision, including key indicators, scores and trends, and provide detailed information and analysis of the decision process, including the basis for the decision and influencing factors; Data Analysis Function Unit: Performs data screening, allowing users to filter and view data according to different conditions, and provides data export options, allowing users to export visualization results and data analysis into Excel, CSV or PDF formats.

Images