Health monitoring and pain assessment system and method based on multi-modal data fusion

Abstract

The invention discloses a health monitoring and pain assessment system based on multi-modal data fusion, and relates to the technical field of health assessment. The system comprises data acquisition, preprocessing, feature fusion and other modules. Multi-modal data are collected and frequency is adaptively adjusted, various data are processed by using a specific technology during preprocessing, a feature fusion module innovates a fusion mechanism, and an evaluation module uses a reinforcement learning model and comprises a health monitoring module, a user interaction module and a data storage module, so that comprehensive functions are guaranteed. According to the method, the evaluation accuracy is improved through multi-modal collection, data association is uniquely fused and mined, personalized services are realized through self-adaptive adjustment, data security is guaranteed through the block chain technology, and medical health development is powerfully promoted.

Description

Technical Field

[0001] The present invention relates to the technical field of health assessment systems, and in particular to a health monitoring and pain assessment system and method based on multimodal data fusion. Background Art

[0002] In the healthcare field, accurate monitoring of individual health status and effective assessment of pain have always been important research areas. Traditional health monitoring methods often rely on a single type of data, such as measuring only basic physiological indicators such as body temperature and blood pressure to determine health status. This method obtains limited information and fails to fully reflect an individual's health status. Moreover, pain assessment often relies on the patient's subjective description and lacks objective and accurate evaluation methods. As a result, the evaluation results are significantly affected by the patient's ability to express themselves and their subjective feelings, making accuracy difficult to guarantee.

[0003] With the development of wearable devices and sensor technology, multimodal data collection has become possible. However, existing multimodal data fusion methods still face numerous challenges when it comes to health monitoring and pain assessment. The fusion of data from different modalities is inefficient and fails to fully tap into the potential connections between the data, resulting in low accuracy and reliability of assessment results. Furthermore, most existing systems lack precise adaptation to individual differences, making it difficult to provide personalized health monitoring and pain assessment services tailored to the individual user's characteristics.

[0004] Furthermore, privacy protection issues are becoming increasingly prominent during data processing and analysis. Large amounts of personal health data are at risk of being leaked during collection, transmission, and storage. This not only infringes on user privacy but also hinders the widespread application of related technologies. Therefore, developing a system and method that can efficiently integrate multimodal data, accurately assess pain and health status, and ensure data security is of great practical significance. Summary of the Invention

[0005] The present invention proposes a health monitoring and pain assessment system and method based on multimodal data fusion to solve the problems mentioned in the above-mentioned prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a health monitoring and pain assessment system based on multimodal data fusion, comprising:

[0007] Data acquisition module: adopts the sensor array collaborative working mode; equipped with heart rate, skin electricity, acceleration, environmental temperature and humidity sensors, voice acquisition equipment and cameras, and newly added electromyography sensors to collect muscle electrical activity signals EMG. Adopting wearable adaptive fitting technology, the sensor automatically adjusts its fit according to the contours of different parts of the human body. The acquisition frequency is adaptively adjusted according to the mutual information entropy of the dynamic changes of multimodal data. The formula is: Where I(X; Y) is the mutual information entropy between different modal data, f base is the initial acquisition frequency, k is the adjustment coefficient, and Δt is the time interval. In the data acquisition module, all sensors must be regularly calibrated and maintained, especially the newly added electromyographic sensors, which are sensitive to interference and require special protection due to their precision. Focus on adaptive fit technology for smart wearables to prevent friction from clothing, physical activity, and other factors that may affect the fit of the sensor to the body. Manual assistance should be used when necessary to ensure stable data acquisition. The value in the acquisition frequency adjustment formula should be set cautiously based on actual conditions. Ensure accurate calculation of mutual information entropy to prevent errors in acquisition frequency adjustment from affecting data quality.

[0008] Data preprocessing module: Uses a multi-scale convolutional neural network (MS-CNN) to preprocess image data. For speech data, a denoising method based on a generative adversarial network (GAN) is used. At the same time, feature enhancement is performed on various data types, including wavelet transform of physiological data to extract features of different frequency components and expand the dimension of the feature space.

[0009] Feature fusion module: proposes a cross-modal attention fusion mechanism; introduces an attention mechanism to dynamically assign weights to features of different modalities; adopts a view learning method to treat different modal data as different views, and mines cross-modal features through joint learning. The calculation formula of the fused feature vector F is optimized to F = ∑ i α i ×β i ×F i , where α i is the weight based on contribution, β i is the attention weight based on cross-modal dependencies;

[0010] Pain Assessment Model Module: Builds a hybrid deep reinforcement learning model for pain assessment. This model combines long short-term memory (LSTM) networks, convolutional neural networks (CNN) and deep reinforcement learning algorithms. It also introduces meta-learning techniques to learn the pain characteristics of new individuals.

[0011] Health Monitoring Module: This module uses a health monitoring method that integrates fuzzy logic reasoning and machine learning. Fuzzy rules are set to make preliminary judgments on physiological and environmental data, giving a fuzzy description of health status. The fuzzy reasoning results are then combined with machine learning models to conduct health risk assessments.

[0012] Furthermore, it also includes:

[0013] User interaction module: Develop an interactive interface based on virtual reality and augmented reality; users experience health monitoring and pain assessment results through the device;

[0014] Data storage module: The collected multimodal data, pre-processed feature data, and pain assessment results are encrypted and stored on the node. At the same time, contracts are used to enable authorized access and sharing of data to protect user data privacy;

[0015] Risk prediction module: Uses causal inference networks to analyze pain assessment results and health monitoring data to identify causal factors that lead to increased pain or worsening health conditions; combines Bayesian networks and Markov decision processes to predict the probability of different risk events occurring in the future and generate risk response strategies.

[0016] Feedback Adjustment Module: This module introduces a feedback adjustment mechanism that combines transfer learning and federated learning. When users report an incorrect pain assessment result, transfer learning is used to adjust model parameters. Furthermore, federated learning is used to share model updates across user devices, improving the model's overall performance and adaptability while protecting user data privacy.

[0017] Furthermore, the sensor in the data acquisition module adopts energy harvesting technology, and at the same time, the sensor automatically selects the communication protocol according to different environments and data transmission requirements.

[0018] Furthermore, based on the cross-modal attention fusion mechanism, the feature fusion module introduces a graph neural network (GNN) to model the relationship between different modal features. Different modal features are regarded as nodes in the graph, and the association between features is represented by edges. GNN is used to learn the interactive relationship between features to optimize the fusion effect.

[0019] Furthermore, the pain assessment model module adopts a task learning method to simultaneously evaluate pain intensity levels, identify pain types, and predict pain duration; by sharing some network layers, the model's training efficiency and feature utilization capabilities are optimized, while the correlation between tasks is utilized to optimize the accuracy of predictions.

[0020] A method for applying the health monitoring and pain assessment system based on multimodal data fusion, comprising:

[0021] Data collection steps: Utilize the sensor array to collect the user's heart rate (HR), galvanic skin response (GSR), body acceleration (Acc), muscle electrical activity signal (EMG), ambient temperature (T), humidity (H), voice information, and facial expression image data at an adaptive collection frequency based on mutual information entropy; the collection frequency is adjusted according to the formula;

[0022] Data transmission steps: Adopting adaptive data compression algorithm to compress the collected data, dynamically adjusting the compression ratio according to the importance and relevance of the data; transmitting data through wireless communication technology, and performing data encryption and verification during the transmission process;

[0023] Data preprocessing steps: Use the multi-scale convolutional neural network (MS-CNN) to preprocess image data, and use the generative adversarial network (GAN)-based denoising method to process speech data; perform wavelet transform feature enhancement on various data types, and perform outlier detection and correction;

[0024] Feature fusion step: Fuse the preprocessed multimodal data features through the cross-modal attention fusion mechanism and the graph neural network (GNN); calculate the fused feature vector F according to the formula;

[0025] Pain assessment step: The fused feature vector F is input into a hybrid deep reinforcement learning model to simultaneously assess pain intensity, identify pain type, and predict pain duration, using meta-learning techniques to adapt to the pain characteristics of new individuals.

[0026] Health monitoring steps: Using a fusion of fuzzy logic reasoning and machine learning, combined with a health state transition probability matrix, the user's health status is monitored and risk assessed in real time; based on the changing trends of physiological and environmental data, future changes in health status are predicted;

[0027] Result display and interaction steps: Display health monitoring data, pain assessment results and related suggestions through an interactive interface based on virtual reality and augmented reality; support users to interact with the system through voice.

[0028] Furthermore, it also includes:

[0029] Risk prediction step: Use causal inference networks, Bayesian networks, and Markov decision processes to analyze pain assessment results and health monitoring data, predict the probability of different risk events occurring in the future, and generate risk response strategies.

[0030] Furthermore, after the result display and interaction steps, it also includes:

[0031] Feedback adjustment step: Using a feedback adjustment mechanism that combines transfer learning and federated learning, the collection frequency of the data collection module, the weight distribution of the feature fusion module, and the parameters of the pain assessment model are adjusted based on user feedback on pain assessment results and health recommendations, as well as actual changes in health status.

[0032] Furthermore, in the data collection step, the working status and wearing status of the sensor are monitored in real time; when the sensor fails or is worn abnormally, it automatically switches to the backup sensor or issues a reminder message. At the same time, the collection frequency and data transmission strategy are dynamically adjusted according to the working status of the sensor.

[0033] Compared with the existing technology, the beneficial effects of the present invention are:

[0034] This system offers significant advantages. Through multimodal data collection, it comprehensively captures user information, improving the accuracy of health monitoring and pain assessment. Its unique data fusion technology fully exploits connections between data, enabling more accurate assessments of health and pain conditions. The system also adapts based on user feedback and actual conditions, providing personalized services. Furthermore, data security utilizes technologies such as blockchain to effectively protect user privacy. This overall improvement in the quality of health monitoring and pain assessment provides more reliable technical support for the healthcare sector. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a schematic block diagram of the health monitoring and pain assessment system based on multimodal data fusion proposed by the present invention;

[0036] Figure 2 This is a schematic block diagram of the health monitoring and pain assessment method based on multimodal data fusion proposed in the present invention;

[0037] Figure 3 This is a schematic diagram showing the comparison of the pain assessment accuracy of the health monitoring and pain assessment method based on multimodal data fusion proposed in the present invention;

[0038] Figure 4 This is a schematic diagram of data representation of the beneficial effects of the health monitoring and pain assessment method based on multimodal data fusion proposed in the present invention. DETAILED DESCRIPTION

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0040] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0041] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the said features. In the description of the present invention, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined. In addition, the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. The present invention will be further described in detail below with reference to the accompanying drawings.

[0042] Reference Figures 1-4 :A health monitoring and pain assessment system based on multimodal data fusion, including the following modules:

[0043] Data acquisition module: adopts a multi-sensor array collaborative working mode. In addition to being equipped with heart rate, skin electricity, acceleration, ambient temperature and humidity sensors, as well as voice acquisition equipment and cameras, a new electromyography sensor collects muscle electrical activity signals EMG (unit: microvolts), which can more accurately reflect the relationship between muscle fatigue and pain. The use of smart wearable adaptive fitting technology allows the sensor to automatically adjust the fit according to the contours of different parts of the human body to ensure data collection stability. The acquisition frequency is adaptively adjusted based on the mutual information entropy of the dynamic changes in multimodal data. The formula is Where I(X; Y) is the mutual information entropy between different modal data, f base is the initial acquisition frequency, k is the adjustment coefficient, and Δt is the time interval.

[0044] Data Preprocessing Module: Image data is preprocessed using a multi-scale convolutional neural network (MS-CNN). This network uses convolution kernels of varying sizes to extract multi-scale features of facial expressions, enhancing their expressiveness. For speech data, a denoising method based on a generative adversarial network (GAN) is used to generate realistic, pure speech samples, effectively removing background noise interference. Simultaneously, feature enhancement is performed on various data types, such as performing a wavelet transform on physiological data to extract features of different frequency components and expand the dimensionality of the feature space.

[0045] Feature fusion module: Proposes a cross-modal attention fusion mechanism. The introduction of the attention mechanism dynamically assigns weights to features of different modalities, not only considering the contribution of each modality to pain assessment, but also focusing on the interdependence between different modalities. Using a multi-view learning method, different modal data are treated as different views, and potential cross-modal features are mined through joint learning. The calculation formula of the fused feature vector F is optimized as F = ∑ i α i ×β i ×F i , where α i is the weight based on contribution, β i is the attention weight based on cross-modal dependencies.

[0046] Pain Assessment Model Module: Builds a hybrid deep reinforcement learning model for pain assessment. This model combines a long short-term memory (LSTM) network, a convolutional neural network (CNN), and a deep reinforcement learning algorithm (such as A3C). The LSTM captures time series information, the CNN extracts spatial features, and the deep reinforcement learning algorithm adjusts model parameters in real time based on the assessment results to adapt to individual pain characteristics. Furthermore, meta-learning techniques are introduced to rapidly learn the pain characteristics of new individuals, improving the model's generalization and adaptability.

[0047] Health Monitoring Module: This module utilizes a health monitoring method that integrates fuzzy logic reasoning with machine learning. Fuzzy rules are set to make preliminary judgments about physiological and environmental data. For example, when multiple indicators such as heart rate and blood pressure fall within different fuzzy ranges, a corresponding fuzzy description of health status is given. The fuzzy reasoning results are then combined with machine learning models (such as support vector machines) for more accurate health risk assessments. Furthermore, a health status transition probability matrix is ​​established to predict future health status trends.

[0048] User interaction module: Develop an interactive interface based on virtual reality (VR) and augmented reality (AR). Users can immersively experience health monitoring and pain assessment results through VR devices, such as using virtual scenes to display the health status and pain levels of various parts of the body. The AR function allows users to view relevant health information and suggestions through mobile phones and other devices in a real environment. In addition, natural language processing technology is introduced to enable voice interaction between users and the system, making user operations more convenient. Data storage module: Use blockchain technology to build a distributed data storage system. The collected multimodal data, pre-processed feature data, and pain assessment results are encrypted and stored on the blockchain nodes to ensure that the data cannot be tampered with and is secure. At the same time, smart contracts are used to enable authorized access and sharing of data to protect user data privacy.

[0049] The present invention also includes:

[0050] Risk Prediction Module: This module uses a causal inference network to analyze pain assessment results and health monitoring data to identify causal factors that lead to increased pain or worsening health. Combining Bayesian networks and Markov decision processes, it predicts the probability of different risk events occurring over a period of time and generates personalized risk response strategies.

[0051] Feedback Adjustment Module: This module introduces a feedback adjustment mechanism that combines transfer learning and federated learning. When users report inaccurate pain assessment results, transfer learning is used to quickly adjust model parameters. Furthermore, federated learning allows model updates to be shared across multiple user devices, improving the overall performance and adaptability of the model while protecting user data privacy.

[0052] In this invention, the sensors in the data acquisition module utilize energy harvesting technology, such as converting the mechanical energy of human motion into electrical energy to power the sensors, thereby extending their battery life. Furthermore, the sensors feature adaptive communication protocol switching, automatically selecting the optimal communication protocol (e.g., Bluetooth, ZigBee, etc.) based on different environments and data transmission requirements.

[0053] In this paper, the feature fusion module introduces a graph neural network (GNN) based on the cross-modal attention fusion mechanism to model the relationship between features of different modalities. Different modal features are treated as nodes in a graph, and the associations between features are represented by edges. GNN is used to learn the complex interactions between features, further optimizing the fusion effect.

[0054] In this paper, the pain assessment model module uses a multi-task learning approach to simultaneously assess pain intensity, identify pain types, and predict pain duration. By sharing some network layers, the model's training efficiency and feature utilization capabilities are improved, while the correlation between tasks is exploited to enhance prediction accuracy.

[0055] The present invention also discloses a health monitoring and pain assessment method based on multimodal data fusion, comprising the following steps:

[0056] Data collection steps: Use a multi-sensor array to collect the user's heart rate HR, skin galvanic response GSR, body activity acceleration Acc, muscle electrical activity signal EMG, ambient temperature T, humidity H, voice information and facial expression image data according to the adaptive collection frequency based on mutual information entropy. The collection frequency is based on the formula Adjustments should be made. Multi-sensor arrays must be regularly calibrated to ensure data accuracy. Pay attention to the parameter settings in the formulas, and adjust the values ​​based on the actual scenario. During the data collection process, pay attention to device operation to avoid data loss due to malfunctions. Also, respect user privacy and collect data in compliance with regulations.

[0057] Data transmission steps: Collected data is compressed using an adaptive data compression algorithm, dynamically adjusting the compression ratio based on the data's importance and relevance. Data is transmitted via wireless communication technologies (such as Bluetooth, Wi-Fi, and ZigBee), with blockchain technology used for data encryption and verification during transmission to ensure data integrity and security. The adaptive compression algorithm balances compression ratio with data integrity. The selection of wireless communication technology takes into account transmission distance, speed, and power consumption. Blockchain encryption and verification ensure data security while focusing on computational efficiency to prevent transmission delays.

[0058] Data preprocessing steps: Image data is preprocessed using a multi-scale convolutional neural network (MS-CNN), and speech data is processed using a denoising method based on a generative adversarial network (GAN). Feature enhancement operations such as wavelet transforms are performed on all types of data, while outlier detection and correction are also performed. The MS-CNN and GAN models require regular training and optimization. Wavelet transform and other operational parameters are adjusted based on data characteristics. Outlier detection and correction must be appropriate to avoid misclassification of normal data.

[0059] Feature fusion step: The pre-processed multimodal data features are fused through the cross-modal attention fusion mechanism and graph neural network (GNN). According to the formula F = ∑ i α i ×β i ×F i Calculate the fused feature vector F, where α i is the weight based on contribution, β i The attention weight is based on the cross-modal dependency. The cross-modal attention fusion mechanism and GNN parameters need to adapt to the data. i , β i It is necessary to comprehensively consider multiple factors and make dynamic adjustments as data changes to ensure the fusion effect.

[0060] Pain Assessment Step: The fused feature vector F is input into a hybrid deep reinforcement learning model (combining LSTM, CNN, and A3C algorithms) to simultaneously assess pain intensity, identify pain types, and predict pain duration. Meta-learning techniques are used to rapidly adapt to the pain characteristics of new individuals. The training data for the hybrid deep reinforcement learning model should be rich and diverse. The use of meta-learning ensures rapid adaptation to new individuals while preventing overfitting and ensuring accurate assessment.

[0061] Health Monitoring Steps: Fuzzy logic and machine learning are combined with a health state transition probability matrix to conduct real-time health monitoring and risk assessment. Future health status changes are predicted based on trends in physiological and environmental data. Fuzzy logic and machine learning fusion parameters require calibration. The health state transition probability matrix is ​​updated based on population characteristics and medical knowledge to ensure reasonable risk assessments.

[0062] Results display and interaction steps: Health monitoring data, pain assessment results, and related recommendations are presented through interactive interfaces based on virtual reality (VR) and augmented reality (AR). Voice interaction with the system is supported, allowing users to input self-assessment information, which is then combined with sensor data for comprehensive analysis. The VR / AR interactive interface must ensure stability and smoothness. The voice interaction recognition system needs to be continuously optimized to rationally analyze user input and enhance the interactive experience.

[0063] In the present invention, after the pain assessment step, the method further includes:

[0064] Risk prediction step: Use causal inference networks, Bayesian networks, and Markov decision processes to analyze pain assessment results and health monitoring data, predict the probability of different risk events occurring in the future, and generate personalized risk response strategies. In the risk prediction step, when constructing causal inference networks, Bayesian networks, and Markov decision process models, it is necessary to ensure that the input pain assessment results and health monitoring data are accurate, complete, and in the correct format, otherwise it will lead to model prediction deviations. Model parameters need to be calibrated and optimized based on a large amount of historical data and actual scenarios. There are differences in data from different individuals, and over-reliance on universal parameters should be avoided. In addition, the generated personalized risk response strategy must fully consider factors such as the individual's physical condition and living habits. It must be operational and targeted, and the strategy must be updated regularly based on new data to ensure its effectiveness.

[0065] In the present invention, after the result display and interaction steps, the following steps are also included:

[0066] Feedback Adjustment Step: Utilizing a feedback adjustment mechanism that combines transfer learning and federated learning, the data collection module's collection frequency, the feature fusion module's weight distribution, and the pain assessment model's parameters are adjusted based on user feedback on pain assessment results and health recommendations, as well as actual changes in health status. Regarding the feedback adjustment step, when combining transfer learning and federated learning, attention must be paid to privacy protection and data security across different data sources, and relevant regulations must be strictly adhered to. User feedback must be collected comprehensively and accurately to prevent misaligned adjustments due to missing or inaccurate information. When adjusting the data collection module frequency, feature fusion module weights, and pain assessment model parameters, thorough testing and verification are required, with gradual fine-tuning to avoid large adjustments at once that could impact the stability and accuracy of the entire system. Adjustment effectiveness should also be regularly evaluated, and the adjustment strategy further optimized based on the evaluation results.

[0067] In the present invention, in the data collection step, the working status and wearing status of the sensor are monitored in real time. When a sensor fails or is worn abnormally, it automatically switches to a backup sensor or issues a reminder message to ensure the continuity and accuracy of data collection. At the same time, the collection frequency and data transmission strategy are dynamically adjusted according to the working status of the sensor. In the data collection step, the monitoring of the working status and wearing status of the sensor must be real-time and accurate to detect faults and anomalies in a timely manner. The backup sensor needs to be calibrated and tested in advance to ensure that it is available at any time and the data is accurate. The setting of the reminder information should be reasonable to avoid disturbing the user too frequently or being too hidden to cause the user to ignore it. When dynamically adjusting the collection frequency and data transmission strategy, factors such as sensor performance, data volume and network conditions should be comprehensively considered to prevent data loss or transmission congestion due to improper adjustment, and ensure efficient and stable data collection and transmission.

[0068] The above are only preferred specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solutions and inventive concepts of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A health monitoring and pain assessment system based on multimodal data fusion, characterized in that: include: Data acquisition module: adopts the sensor array collaborative working mode; equipped with heart rate, skin electricity, acceleration, environmental temperature and humidity sensors, voice acquisition equipment and cameras, and newly added electromyography sensors to collect muscle electrical activity signals EMG. Adopting wearable adaptive fitting technology, the sensor automatically adjusts its fit according to the contours of different parts of the human body. The acquisition frequency is adaptively adjusted according to the mutual information entropy of the dynamic changes of multimodal data. The formula is: Where I(X; Y) is the mutual information entropy between different modal data, f base is the initial acquisition frequency, k is the adjustment coefficient, and Δt is the time interval; Data preprocessing module: Uses a multi-scale convolutional neural network (MS-CNN) to preprocess image data. For speech data, a denoising method based on a generative adversarial network (GAN) is used. At the same time, feature enhancement is performed on various data types, including wavelet transform of physiological data to extract features of different frequency components and expand the dimension of the feature space. Feature fusion module: proposes a cross-modal attention fusion mechanism; introduces an attention mechanism to dynamically assign weights to features of different modalities; adopts a view learning method to treat different modal data as different views, and mines cross-modal features through joint learning. The calculation formula of the fused feature vector F is optimized to F = ∑ i α i ×β i ×F i , where α i is the weight based on contribution, β i is the attention weight based on cross-modal dependencies; Pain Assessment Model Module: Builds a hybrid deep reinforcement learning model for pain assessment. This model combines long short-term memory (LSTM) networks, convolutional neural networks (CNN) and deep reinforcement learning algorithms. It also introduces meta-learning techniques to learn the pain characteristics of new individuals. Health monitoring module: adopts a health monitoring method that integrates fuzzy logic reasoning and machine learning; sets fuzzy rules to make preliminary judgments on physiological data and environmental data, and gives a fuzzy description of health status; then combines the fuzzy reasoning results with the machine learning model to conduct health risk assessment.

2. The health monitoring and pain assessment system based on multimodal data fusion according to claim 1 is characterized in that: Also includes: User interaction module: Develop interactive interfaces based on virtual reality and augmented reality; Users experience health monitoring and pain assessment results through the device; Data storage module: The collected multimodal data, pre-processed feature data, and pain assessment results are encrypted and stored on the node. At the same time, contracts are used to enable authorized access and sharing of data to protect user data privacy; Risk prediction module: Uses causal inference networks to analyze pain assessment results and health monitoring data to identify causal factors that lead to increased pain or worsening health conditions; combines Bayesian networks and Markov decision processes to predict the probability of different risk events occurring in the future and generate risk response strategies.

3. The health monitoring and pain assessment system based on multimodal data fusion according to claim 1, characterized in that: Also includes: Feedback Adjustment Module: This module introduces a feedback adjustment mechanism that combines transfer learning and federated learning. When users report an incorrect pain assessment result, transfer learning is used to adjust model parameters. Furthermore, federated learning is used to share model updates across user devices, optimizing the model's overall performance and adaptability while protecting user data privacy.

4. The health monitoring and pain assessment system based on multimodal data fusion according to claim 1, characterized in that: The sensor in the data acquisition module adopts energy collection technology. At the same time, the sensor automatically selects the communication protocol according to different environments and data transmission requirements.

5. The health monitoring and pain assessment system based on multimodal data fusion according to claim 1, characterized in that: Based on the cross-modal attention fusion mechanism, the feature fusion module introduces a graph neural network (GNN) to model the relationship between different modal features. Different modal features are regarded as nodes in the graph, and the association between features is represented by edges. GNN is used to learn the interactive relationship between features to optimize the fusion effect.

6. The health monitoring and pain assessment system based on multimodal data fusion according to claim 1, characterized in that: The pain assessment model module adopts a task learning method to simultaneously assess pain intensity levels, identify pain types, and predict pain duration; by sharing some network layers, the model's training efficiency and feature utilization capabilities are optimized, while the correlation between tasks is used to optimize prediction accuracy.

7. A method for applying the health monitoring and pain assessment system based on multimodal data fusion according to any one of claims 1 to 6, characterized in that: include: Data collection steps: Utilize the sensor array to collect the user's heart rate (HR), galvanic skin response (GSR), body acceleration (Acc), muscle electrical activity signal (EMG), ambient temperature (T), humidity (H), voice information, and facial expression image data at an adaptive collection frequency based on mutual information entropy; the collection frequency is adjusted according to the formula; Data transmission steps: Adopting adaptive data compression algorithm to compress the collected data, dynamically adjusting the compression ratio according to the importance and relevance of the data; transmitting data through wireless communication technology, and performing data encryption and verification during the transmission process; Data preprocessing steps: Use the multi-scale convolutional neural network (MS-CNN) to preprocess image data, and use the generative adversarial network (GAN)-based denoising method to process speech data; perform wavelet transform feature enhancement on various data types, and perform outlier detection and correction; Feature fusion step: The pre-processed multimodal data features are fused through the cross-modal attention fusion mechanism and graph neural network (GNN); Calculate the fused feature vector F according to the formula; Pain assessment step: The fused feature vector F is input into a hybrid deep reinforcement learning model to simultaneously assess pain intensity, identify pain type, and predict pain duration, using meta-learning techniques to adapt to the pain characteristics of new individuals. Health monitoring steps: Using a fusion of fuzzy logic reasoning and machine learning, combined with a health state transition probability matrix, to conduct real-time monitoring and risk assessment of the user's health status; Predict future changes in health status based on changing trends in physiological and environmental data; Results display and interaction steps: Display health monitoring data, pain assessment results, and related suggestions through interactive interfaces based on virtual reality and augmented reality; Supports voice interaction between users and the system.

8. The health monitoring and pain assessment method based on multimodal data fusion according to claim 7, characterized in that: Following the pain assessment step, also include: Risk prediction step: Use causal inference networks, Bayesian networks, and Markov decision processes to analyze pain assessment results and health monitoring data, predict the probability of different risk events occurring in the future, and generate risk response strategies.

9. The health monitoring and pain assessment method based on multimodal data fusion according to claim 7, characterized in that: After the result display and interaction steps, it also includes: Feedback adjustment step: Using a feedback adjustment mechanism that combines transfer learning and federated learning, the collection frequency of the data collection module, the weight distribution of the feature fusion module, and the parameters of the pain assessment model are adjusted based on user feedback on pain assessment results and health recommendations, as well as actual changes in health status.

10. The health monitoring and pain assessment method based on multimodal data fusion according to claim 7, characterized in that: During the data collection step, the working status and wearing status of the sensor are monitored in real time; when a sensor fails or is worn abnormally, it automatically switches to a backup sensor or issues a reminder message. At the same time, the collection frequency and data transmission strategy are dynamically adjusted according to the working status of the sensor.

Images