Multi-modal data fusion method

Abstract

The invention discloses a multi-modal data fusion method, and relates to the technical field of data fusion, and the method comprises the steps: deploying a sensor, and collecting the multi-modal data of a family environment in real time; based on the context information and the sensor data, constructing a home environment model, and identifying a mutual relation among different environment factors; dynamically adjusting weights of different data sources according to real-time feedback based on an adaptive algorithm; and fusing the multi-modal data by using a neural network model to generate a comprehensive decision. According to the invention, by fusing various sensor data, the change of the home environment can be more comprehensively captured, the accuracy and reliability of environment monitoring are improved, misjudgment possibly caused by a single sensor is avoided, the weights of different data sources are dynamically adjusted based on real-time feedback, the system can keep efficient decision-making ability when the environment is changed, and the system has high practicability. Due to the flexibility, the system can better adapt to user requirements and changes of the external environment.

Description

Technical Field

[0001] The present invention relates to the technical field of data fusion, and in particular to a multimodal data fusion method. Background Art

[0002] With the rapid development of the Internet of Things and smart home technologies, multimodal data fusion methods have become increasingly important in home environment monitoring and control systems. The multimodal data of the home environment comes from a wide range of sources, including temperature, humidity, light, motion, and gas concentration information. These data can be collected in real time through various sensors. Through effective data fusion technology, the home environment can be fully understood and intelligently managed. Traditional single sensor monitoring methods often cannot accurately reflect the complexity of the home environment because there are many interrelationships between environmental factors. For example, indoor temperature is not only affected by the external climate, but also interacts with humidity, light and other factors. Therefore, fusion technology based on multimodal data can make full use of the information of multiple sensors to more comprehensively and accurately describe and predict the state of the home environment.

[0003] In addition, the dynamic nature of the home environment requires the system to have adaptive capabilities to cope with environmental changes in different scenarios. Therefore, the use of adaptive algorithms to dynamically adjust the weights of different data sources based on real-time feedback can improve the decision-making ability and reliability of the model. The fusion of multimodal data through neural network models can not only improve the accuracy of decision-making, but also provide more personalized and intelligent services for home users. Summary of the invention

[0004] In order to solve the above technical problems, a multimodal data fusion method is provided. This technical solution solves the above problems.

[0005] In order to achieve the above purpose, the technical solution adopted by the present invention is:

[0006] A multimodal data fusion method, the data fusion method comprising:

[0007] Deploy sensors to collect multimodal data of the home environment in real time, and clean and standardize the collected data;

[0008] Based on contextual information and sensor data, a home environment model is constructed to identify the interrelationships between different environmental factors;

[0009] Dynamically adjust the weights of different data sources based on real-time feedback based on adaptive algorithms;

[0010] Preferably, a neural network model is used to fuse multimodal data to generate a comprehensive decision.

[0011] The deployment of sensors to collect multimodal data of the home environment in real time and the cleaning and standardization of the collected data specifically include:

[0012] Deploy sensors, including temperature, humidity, light, motion, and gas concentration, connect the sensors to a central data collection system, and configure the sensor sampling frequency and data transmission method;

[0013] Start real-time data collection and transmit the data to the central server through the network;

[0014] Check whether there are missing values ​​in each sensor data, use interpolation to fill the missing values, and use moving average method to smooth the data;

[0015] The moving average expression is:

[0016]

[0017] In the formula, MA t is the moving average at time point t, n is the window size, i is the index variable, X t-i is the observed value at time point ti, t is the current time point;

[0018] Standardize the data from all sensors to ensure that all data are in the same range, check the distribution of the standardized data, and generate descriptive statistics of the data.

[0019] Preferably, the constructing of the home environment model based on the context information and the sensor data and identifying the relationship between different environmental factors specifically includes:

[0020] Use Pearson's correlation coefficient to calculate the correlation between features and identify important features;

[0021] Use the decision tree method to perform feature selection and filter out features that are closely related to the target variable;

[0022] Select a regression model based on data characteristics and goals to predict numerical variables;

[0023] Divide the data set into a training set and a test set, and use the training set to train the model;

[0024] The interrelationships among environmental factors were analyzed based on the model outputs, and the key factors affecting the family environment were identified.

[0025] Preferably, the using of the Pearson correlation coefficient to calculate the correlation between features and identifying important features specifically includes:

[0026] The data of all relevant features are organized into a matrix, and the missing values ​​are processed using interpolation to obtain the mean and standard deviation of each feature;

[0027] Calculate the Pearson correlation coefficient of two features, where the Pearson correlation coefficient calculation formula is:

[0028]

[0029] According to the absolute value of the calculated correlation coefficient, 0.5 is set as the threshold, the features closely related to the target variable are screened out, the value of the correlation coefficient is recorded, and the positive and negative correlations between the features are analyzed.

[0030] Preferably, the method of using the decision tree method to perform feature selection to screen out features that are closely related to the target variable specifically includes:

[0031] Build a decision tree model, use the training set data to build a decision tree, and divide the data by selecting the optimal features to maximize information gain;

[0032] The importance of each feature is evaluated based on Gini impurity, where the Gini impurity calculation formula is:

[0033]

[0034] In the formula, G(S) represents the Gini impurity of set S, S represents a data set, c represents the total number of classifications, and p i Indicates the proportion of samples belonging to category i in the set S;

[0035] After the decision tree model is trained, the importance score of the feature is extracted from the model, a threshold is set according to the importance score of the feature, and features with importance higher than the threshold are screened out;

[0036] Output the selected features and record their corresponding importance scores.

[0037] Preferably, the step of selecting a regression model according to data characteristics and objectives and predicting numerical variables specifically includes:

[0038] Based on the extracted features and target selection, a regression model is established;

[0039] In each neuron, the input features are processed by weighted and activated functions, and the output of the neuron is nonlinearly transformed by the activation function. In the regression problem, the output layer uses a linear activation function, and the weights and biases are updated by the back-propagation algorithm to minimize the loss function.

[0040] Predict numerical variables based on the regression model, where the regression model expression is:

[0041]

[0042] In the formula, represents the predicted probability value, e is the base of the natural logarithm, β0 is the intercept, β1β2β n is the regression coefficient, x1x2x n is the independent variable.

[0043] Preferably, the dynamically adjusting the weights of different data sources based on the adaptive algorithm according to real-time feedback specifically includes:

[0044] Refuse to set initial weights for different data sources based on historical data;

[0045] Design a feedback mechanism to collect the deviation between the system's decision results and the actual situation in real time, and define evaluation indicators to quantify the performance of different data sources;

[0046] Monitor system performance in real time and record the impact of each data source on decision-making results;

[0047] The weight adjustment rules are set based on the weighted adjustment algorithm, and the Kalman filter is used to dynamically adjust the weights according to new data;

[0048] Based on real-time feedback and evaluation results, the weights of each data source are dynamically adjusted to ensure that the weight adjustment is within a reasonable range. Based on the adjusted weights, the neural network model is retrained regularly, using incremental learning methods to update the model while maintaining previous learning results.

[0049] Preferably, the weight adjustment rule is set based on the weighted adjustment algorithm, and the Kalman filter is used to dynamically adjust the weight according to the new data, specifically including:

[0050] Before using the Kalman filter, define the state variables and initial state, and set the initial state and initial covariance;

[0051] According to the weighted adjustment algorithm, set the initial weights;

[0052] Update the state and covariance according to the new observation data;

[0053] In the update phase, the weights are dynamically adjusted by combining the Kalman gain and the variance of the observation noise;

[0054] The weight adjustment rule dynamically adjusts the weight according to the size of the observation noise and the Kalman gain.

[0055] Preferably, the method of fusing multimodal data using a neural network model to generate a comprehensive decision specifically includes:

[0056] Select the appropriate neural network architecture based on the characteristics of multimodal data and the loss function based on the decision-making goal;

[0057] Training data partitioning: Divide the data into training set, validation set and test set for model training and evaluation;

[0058] Dynamically adjust the weights of different data sources based on real-time feedback based on adaptive algorithms

[0059] Design the output layer based on the decision goal, use the trained model to infer new input data, and generate comprehensive decisions;

[0060] Deploy the trained model to the system to achieve real-time data inference, continuously optimize the model based on actual environment feedback, and retrain the model regularly;

[0061] Integrate the model into the home environment control system so that it can receive data and make decisions in real time.

[0062] Preferably, the step of designing an output layer according to a decision target, using a trained model to infer new input data, and generating a comprehensive decision specifically includes:

[0063] Define the target variable, determine the decision threshold, and design the output layer of the model according to the decision goal;

[0064] After designing the output layer, train the model to ensure its ability to infer new data. Collect and process training data, including the preparation of features and target variables. Perform multiple rounds of iterations to optimize model parameters until performance requirements are met.

[0065] Generate final decisions based on model outputs, converting model outputs into executable decisions, including generating reports, triggering system actions, and notifying users.

[0066] Compared with the prior art, the present invention has the following beneficial effects:

[0067] The present invention proposes that by fusing data from multiple sensors, changes in the home environment can be captured more comprehensively, the accuracy and reliability of environmental monitoring can be improved, and misjudgment that may be caused by a single sensor can be avoided. The weights of different data sources can be dynamically adjusted based on real-time feedback to ensure that the system can maintain efficient decision-making capabilities when the environment changes. This flexibility enables the system to better adapt to user needs and changes in the external environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a step flow framework diagram of the present invention. DETAILED DESCRIPTION

[0069] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art may think of other obvious variations.

[0070] Reference Figure 1 As shown, a multimodal data fusion method, the data fusion method includes:

[0071] Deploy sensors to collect multimodal data of the home environment in real time, and clean and standardize the collected data;

[0072] Based on contextual information and sensor data, a home environment model is constructed to identify the interrelationships between different environmental factors;

[0073] Dynamically adjust the weights of different data sources based on real-time feedback based on adaptive algorithms;

[0074] Use neural network models to fuse multimodal data and generate comprehensive decisions.

[0075] The deployment of sensors to collect multimodal data of the home environment in real time and the cleaning and standardization of the collected data specifically include:

[0076] Deploy sensors, including temperature, humidity, light, motion, and gas concentration, connect the sensors to a central data collection system, and configure the sensor sampling frequency and data transmission method;

[0077] Start real-time data collection and transmit the data to the central server through the network;

[0078] Check whether there are missing values ​​in each sensor data, use interpolation to fill the missing values, and use moving average method to smooth the data;

[0079] The moving average expression is:

[0080]

[0081] In the formula, MA t is the moving average at time point t, n is the window size, i is the index variable, X t-i is the observed value at time point ti, t is the current time point;

[0082] Standardize the data from all sensors to ensure that all data are in the same range, check the distribution of the standardized data, and generate descriptive statistics of the data.

[0083] The constructing of a home environment model based on context information and sensor data and identifying the relationships between different environmental factors specifically includes:

[0084] Use Pearson's correlation coefficient to calculate the correlation between features and identify important features;

[0085] Use the decision tree method to perform feature selection and filter out features that are closely related to the target variable;

[0086] Select a regression model based on data characteristics and goals to predict numerical variables;

[0087] Divide the data set into a training set and a test set, and use the training set to train the model;

[0088] The interrelationships among environmental factors were analyzed based on the model outputs, and the key factors affecting the family environment were identified.

[0089] The Pearson correlation coefficient is used to calculate the correlation between features and identify important features, specifically including:

[0090] The data of all relevant features are organized into a matrix, and the missing values ​​are processed using interpolation to obtain the mean and standard deviation of each feature;

[0091] Calculate the Pearson correlation coefficient of two features, where the Pearson correlation coefficient calculation formula is:

[0092]

[0093] According to the absolute value of the calculated correlation coefficient, 0.5 is set as the threshold, the features closely related to the target variable are screened out, the value of the correlation coefficient is recorded, and the positive and negative correlations between the features are analyzed.

[0094] The decision tree method is used to perform feature selection to screen out features that are closely related to the target variable, specifically including:

[0095] Build a decision tree model, use the training set data to build a decision tree, and divide the data by selecting the optimal features to maximize information gain;

[0096] The importance of each feature is evaluated based on Gini impurity, where the Gini impurity calculation formula is:

[0097]

[0098] In the formula, G(S) represents the Gini impurity of set S, S represents a data set, c represents the total number of classifications, and p i Indicates the proportion of samples belonging to category i in the set S;

[0099] After the decision tree model is trained, the importance score of the feature is extracted from the model, a threshold is set according to the importance score of the feature, and features with importance higher than the threshold are screened out;

[0100] Output the selected features and record their corresponding importance scores.

[0101] The said selecting a regression model according to data characteristics and objectives and predicting numerical variables specifically include:

[0102] Based on the extracted features and target selection, a regression model is established;

[0103] In each neuron, the input features are processed by weighted and activated functions, and the output of the neuron is nonlinearly transformed by the activation function. In the regression problem, the output layer uses a linear activation function, and the weights and biases are updated by the back-propagation algorithm to minimize the loss function.

[0104] Predict numerical variables based on the regression model, where the regression model expression is:

[0105]

[0106] In the formula, represents the predicted probability value, e is the base of the natural logarithm, β0 is the intercept, β1β2β n is the regression coefficient, x1x2x n is the independent variable.

[0107] The method of dynamically adjusting the weights of different data sources based on the adaptive algorithm according to real-time feedback specifically includes:

[0108] Refuse to set initial weights for different data sources based on historical data;

[0109] Design a feedback mechanism to collect the deviation between the system's decision results and the actual situation in real time, and define evaluation indicators to quantify the performance of different data sources;

[0110] Monitor system performance in real time and record the impact of each data source on decision-making results;

[0111] The weight adjustment rules are set based on the weighted adjustment algorithm, and the Kalman filter is used to dynamically adjust the weights according to new data;

[0112] Based on real-time feedback and evaluation results, the weights of each data source are dynamically adjusted to ensure that the weight adjustment is within a reasonable range. Based on the adjusted weights, the neural network model is retrained regularly, using incremental learning methods to update the model while maintaining previous learning results.

[0113] The weight adjustment rules are set based on the weight adjustment algorithm, and the Kalman filter is used to dynamically adjust the weights according to new data, specifically including:

[0114] Before using the Kalman filter, define the state variables and initial state, and set the initial state and initial covariance;

[0115] According to the weighted adjustment algorithm, set the initial weights;

[0116] Update the state and covariance according to the new observation data;

[0117] In the update phase, the weights are dynamically adjusted by combining the Kalman gain and the variance of the observation noise;

[0118] The weight adjustment rule dynamically adjusts the weight according to the size of the observation noise and the Kalman gain.

[0119] The method of using a neural network model to fuse multimodal data and generate a comprehensive decision specifically includes:

[0120] Select the appropriate neural network architecture based on the characteristics of multimodal data and the loss function based on the decision-making goal;

[0121] Training data partitioning: Divide the data into training set, validation set and test set for model training and evaluation;

[0122] Dynamically adjust the weights of different data sources based on real-time feedback based on adaptive algorithms

[0123] Design the output layer based on the decision goal, use the trained model to infer new input data, and generate comprehensive decisions;

[0124] Deploy the trained model to the system to achieve real-time data inference, continuously optimize the model based on actual environment feedback, and retrain the model regularly;

[0125] Integrate the model into the home environment control system so that it can receive data and make decisions in real time.

[0126] The design of the output layer according to the decision-making goal, using the trained model to infer the new input data, and generating a comprehensive decision specifically include:

[0127] Define the target variable, determine the decision threshold, and design the output layer of the model according to the decision goal;

[0128] After designing the output layer, train the model to ensure its ability to infer new data. Collect and process training data, including the preparation of features and target variables. Perform multiple rounds of iterations to optimize model parameters until performance requirements are met.

[0129] Generate final decisions based on model outputs, converting model outputs into executable decisions, including generating reports, triggering system actions, and notifying users.

[0130] In summary, the advantages of the present invention are:

[0131] The present invention proposes that by fusing data from multiple sensors, changes in the home environment can be captured more comprehensively, the accuracy and reliability of environmental monitoring can be improved, and misjudgment that may be caused by a single sensor can be avoided. The weights of different data sources can be dynamically adjusted based on real-time feedback to ensure that the system can maintain efficient decision-making capabilities when the environment changes. This flexibility enables the system to better adapt to changes in user needs and the external environment.

[0132] By cleaning, standardizing and selecting features of sensor data, noise and redundant data can be effectively reduced and data processing efficiency can be improved, which saves computing resources and increases the inference speed of the model.

[0133] Using deep learning technology for data fusion can mine the potential complex relationships and patterns in the data, thereby generating more accurate comprehensive decisions and improving the intelligence level of home environment control systems;

[0134] By deeply analyzing the relationship between home environment and user behavior, intelligent regulation can be performed according to the user's personalized needs to improve the user's comfort and quality of life;

[0135] The deviation between the system's decision-making results and the actual situation is collected in real time to form a closed-loop feedback mechanism, which can continuously optimize the model and ensure that the system gradually improves its performance during use.

[0136] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A multimodal data fusion method, characterized in that: Data fusion methods include: Deploy sensors to collect multimodal data of the home environment in real time, and clean and standardize the collected data; Based on contextual information and sensor data, a home environment model is constructed to identify the interrelationships between different environmental factors; Dynamically adjust the weights of different data sources based on real-time feedback based on adaptive algorithms; Use neural network models to fuse multimodal data and generate comprehensive decisions.

2. A multimodal data fusion method according to claim 1, characterized in that: The deployment of sensors to collect multimodal data of the home environment in real time and the cleaning and standardization of the collected data specifically include: Deploy sensors, including temperature, humidity, light, motion, and gas concentration, connect the sensors to a central data collection system, and configure the sensor sampling frequency and data transmission method; Start real-time data collection and transmit the data to the central server through the network; Check whether there are missing values ​​in each sensor data, use interpolation to fill the missing values, and use moving average method to smooth the data; The moving average expression is: In the formula, MA t is the moving average at time point t, n is the window size, i is the index variable, X t-i is the observed value at time point ti, t is the current time point; Standardize the data from all sensors to ensure that all data are in the same range, check the distribution of the standardized data, and generate descriptive statistics of the data.

3. A multimodal data fusion method according to claim 2, characterized in that: The constructing of a home environment model based on context information and sensor data and identifying the relationships between different environmental factors specifically includes: Use Pearson's correlation coefficient to calculate the correlation between features and identify important features; Use the decision tree method to perform feature selection and filter out features that are closely related to the target variable; Select a regression model based on data characteristics and goals to predict numerical variables; Divide the data set into a training set and a test set, and use the training set to train the model; The interrelationships among environmental factors were analyzed based on the model outputs, and the key factors affecting the family environment were identified.

4. A multimodal data fusion method according to claim 3, characterized in that: The Pearson correlation coefficient is used to calculate the correlation between features and identify important features, specifically including: The data of all relevant features are organized into a matrix, and the missing values ​​are processed using interpolation to obtain the mean and standard deviation of each feature; Calculate the Pearson correlation coefficient of two features, where the Pearson correlation coefficient calculation formula is: According to the absolute value of the calculated correlation coefficient, 0.5 is set as the threshold, the features closely related to the target variable are screened out, the value of the correlation coefficient is recorded, and the positive and negative correlations between the features are analyzed.

5. A multimodal data fusion method according to claim 4, characterized in that: The decision tree method is used to perform feature selection to screen out features that are closely related to the target variable, specifically including: Build a decision tree model, use the training set data to build a decision tree, and divide the data by selecting the optimal features to maximize information gain; The importance of each feature is evaluated based on Gini impurity, where the Gini impurity calculation formula is: In the formula, G(S) represents the Gini impurity of set S, S represents a data set, c represents the total number of classifications, and p i Indicates the proportion of samples belonging to category i in the set S; After the decision tree model is trained, the importance score of the feature is extracted from the model, a threshold is set according to the importance score of the feature, and features with importance higher than the threshold are screened out; Output the selected features and record their corresponding importance scores.

6. A multimodal data fusion method according to claim 5, characterized in that: The said selecting a regression model according to data characteristics and objectives and predicting numerical variables specifically include: Based on the extracted features and target selection, a regression model is established; In each neuron, the input features are processed by weighted and activated functions, and the output of the neuron is transformed nonlinearly by the activation function. In the regression problem, the output layer uses a linear activation function, and the weights and biases are updated by the back-propagation algorithm to minimize the loss function. Predict numerical variables based on the regression model, where the regression model expression is: In the formula, represents the predicted probability value, e is the base of the natural logarithm, β0 is the intercept, β1β2β n is the regression coefficient, x1x2x n is the independent variable.

7. A multimodal data fusion method according to claim 6, characterized in that: The method of dynamically adjusting the weights of different data sources based on the adaptive algorithm according to real-time feedback specifically includes: Refuse to set initial weights for different data sources based on historical data; Design a feedback mechanism to collect the deviation between the system's decision results and the actual situation in real time, and define evaluation indicators to quantify the performance of different data sources; Monitor system performance in real time and record the impact of each data source on decision-making results; The weight adjustment rules are set based on the weighted adjustment algorithm, and the Kalman filter is used to dynamically adjust the weights according to new data; Based on real-time feedback and evaluation results, the weights of each data source are dynamically adjusted to ensure that the weight adjustment is within a reasonable range. Based on the adjusted weights, the neural network model is retrained regularly, using incremental learning methods to update the model while maintaining previous learning results.

8. A multimodal data fusion method according to claim 7, characterized in that: The weight adjustment rules are set based on the weight adjustment algorithm, and the Kalman filter is used to dynamically adjust the weights according to new data, specifically including: Before using the Kalman filter, define the state variables and initial state, and set the initial state and initial covariance; According to the weighted adjustment algorithm, set the initial weights; Update the state and covariance according to the new observation data; In the update phase, the weights are dynamically adjusted by combining the Kalman gain and the variance of the observation noise; The weight adjustment rule dynamically adjusts the weight according to the size of the observation noise and the Kalman gain.

9. A multimodal data fusion method according to claim 8, characterized in that: The method of using a neural network model to fuse multimodal data and generate a comprehensive decision specifically includes: Select the appropriate neural network architecture based on the characteristics of multimodal data and the loss function based on the decision-making goal; Training data partitioning: Divide the data into training set, validation set and test set for model training and evaluation; Dynamically adjust the weights of different data sources based on real-time feedback based on adaptive algorithms Design the output layer based on the decision goal, use the trained model to infer new input data, and generate comprehensive decisions; Deploy the trained model to the system to achieve real-time data inference, continuously optimize the model based on actual environment feedback, and retrain the model regularly; Integrate the model into the home environment control system so that it can receive data and make decisions in real time.

10. A multimodal data fusion method according to claim 9, characterized in that: The design of the output layer according to the decision-making goal, using the trained model to infer the new input data, and generating a comprehensive decision specifically include: Define the target variable, determine the decision threshold, and design the output layer of the model according to the decision goal; After designing the output layer, train the model to ensure its ability to infer new data. Collect and process training data, including the preparation of features and target variables. Perform multiple rounds of iterations to optimize model parameters until performance requirements are met. Generate final decisions based on model outputs, converting model outputs into executable decisions, including generating reports, triggering system actions, and notifying users.

Images