A multi-modal health monitoring data processing method, system and electronic device

Abstract

The application provides a multi-modal health monitoring data processing method, system and electronic equipment, which comprises the following steps: acquiring original health monitoring data, recursively estimating clock bias and drift of a collection node, dynamically adjusting a node synchronization period according to a user context state identifier and a detection result of an abnormal event detection function; mapping a local sampling time to a global reference time based on a clock state parameter, generating a global timing sample with a time uncertainty, and preprocessing an observation stream; using a differentiated strategy to map each modality data in the observation stream to a unified time grid, obtaining a multi-modal feature sequence, and performing timing and cross-modal correlation consistency verification and hierarchical missing compensation; calculating a comprehensive quality score of each time window, and weighting and fusing the multi-modal feature sequence to generate a synchronous fusion matrix. The application improves the timing consistency, fusion reliability and scene adaptability of multi-modal monitoring data.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a multimodal health monitoring data processing method, system, and electronic device. Background Technology

[0002] With the development of home health monitoring, wearable devices, smart home sensors, and home network technologies, the demand for continuous health data collection and analysis in home scenarios is constantly increasing. Home health monitoring typically requires simultaneously acquiring physiological data such as heart rate, blood oxygen saturation, respiration, body movement, and skin temperature, as well as environmental data such as indoor temperature, humidity, PM2.5, CO2, light intensity, and noise. This data, combined with device online status, communication status, and event status, is used to comprehensively assess the user's health condition. Since physiological and environmental data usually come from different types of sensor nodes or terminal devices, there are significant differences between these nodes in terms of sampling frequency, sampling time, device clock, communication links, and data reporting mechanisms. Especially in home scenarios, wearable devices often operate in low-power modes, environmental nodes are mostly accessed via wireless links, and local networks may experience jitter, congestion, disconnections, and retransmissions. This can lead to problems such as time sequence offsets, data out-of-order issues, time rollback, partial missing data, and unstable cross-modal correspondences between different data streams.

[0003] Existing data processing methods often focus on time synchronization of data from single devices, data interpolation at fixed frequencies, or time synchronization processing for general IoT scenarios, typically lacking comprehensive consideration for home health monitoring scenarios. Especially in situations such as nighttime sleep monitoring, abnormal event triggering, frequent changes in user activity, and partial node disconnection, existing technologies struggle to simultaneously ensure time alignment accuracy, fusion continuity, adaptability to abnormal scenarios, and output reliability. This can easily lead to distortion in the correspondence between physiological and environmental data, thereby affecting the accuracy of subsequent health assessments, risk identification, and early warning results.

[0004] Therefore, how to process and fuse multi-source heterogeneous physiological data and environmental data under a unified time reference in home health monitoring scenarios, so as to improve the temporal consistency, fusion reliability and scenario adaptability of multimodal monitoring data, has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, it is necessary to provide a multimodal health monitoring data processing method, system and electronic device to solve the technical problem of how to process and fuse multi-source heterogeneous physiological data and environmental data under a unified time reference in home health monitoring scenarios, thereby improving the temporal consistency, fusion reliability and scenario adaptability of multimodal monitoring data.

[0006] To address the aforementioned technical problems, in a first aspect, the present invention provides a multimodal health monitoring data processing method, comprising: Acquire raw health monitoring data including physiological data, environmental data, and event status data; and generate structured observation samples with local timestamps based on the raw health monitoring data, user context status identifiers, and abnormal event detection functions. Using the local computing node as the master clock node, the clock offset and drift parameters of each acquisition node are recursively estimated, and the node synchronization period is dynamically adjusted based on the user context status identifier and the detection results of the abnormal event detection function. Based on the clock offset and drift parameters and the synchronization period, the local sampling time of each structured observation sample is mapped to the global reference time, and the time uncertainty of each structured observation sample is generated by combining the activity disturbance factor and the network fluctuation factor to obtain the global time series sample. The global time series samples are subjected to out-of-order rearrangement and time rollback correction. The abnormal event sampling window is marked according to the detection result of the abnormal event detection function to obtain a time-ordered observation stream. A unified time grid is constructed, and a differentiated strategy is used to map physiological data, environmental data, and event state data in the observation stream to the unified time grid to obtain a multimodal feature sequence. The temporal and cross-modal correlation consistency verification is performed on the multimodal feature sequence, the missing compensation is performed in stages according to the duration of missing data, and the key modality availability status of each time window is identified to obtain the verified and completed multimodal feature sequence, the temporal and cross-modal correlation consistency verification results, and the key modality availability. Calculate the comprehensive quality score for each time window, and generate a synchronization fusion matrix by weighted fusion of the verified and completed multimodal feature sequences. Based on the comprehensive quality score, the consistency verification results of time series and cross-modal correlation, and the availability of key modes, output the synchronization fusion matrix in normal output mode, local degradation output mode, and conservative output mode.

[0007] In one possible implementation, the recursive estimation of the clock offset and drift parameters of each acquisition node includes: Establish the relationship between the local clock of each acquisition node and the global reference time. The relationship includes the clock drift coefficient, clock offset, and random clock disturbance term. Establish a clock state vector containing the current offset and current drift coefficient for each acquisition node, and construct the state transition equation and observation equation; The clock state of each acquisition node is updated using a recursive state estimation method to obtain the offset estimate, drift estimate and estimated covariance at the current time.

[0008] In one possible implementation, dynamically adjusting the node synchronization period based on the user context state identifier and the detection result of the abnormal event detection function includes: A basic synchronization period is preset, and a scene adjustment coefficient is determined based on the user context state identifier and the detection result of the abnormal event detection function. The product of the basic synchronization period and the scene adjustment coefficient is used as the actual synchronization period. When the user is in a state of sleep monitoring, rest observation, or abnormal event triggering, the scene adjustment coefficient is less than the preset value, thus shortening the actual synchronization cycle. When a user is in a normal daytime activity state and no abnormal events occur, the scenario adjustment coefficient is greater than or equal to the preset value, thus extending or maintaining the actual synchronization cycle.

[0009] In one possible implementation, a differentiated strategy is employed to map physiological data, environmental data, and event state data in the observation stream to the unified time grid, including: For physiological data falling within each window of a unified time grid, the physiological feature vector of each window is extracted based on the temporal proximity and quality score of the samples as a joint weighting criterion. For environmental data, cubic splines with smoothing regularization terms are used to reconstruct the continuous trend of environmental observation points, obtain environmental feature vectors at a unified time grid, and construct environmental exposure load factors based on the load mapping function and weight coefficients corresponding to each environmental variable. For event state data, different methods are used to assign state values ​​to time grid points based on different time state data, and an event state credibility decay factor is introduced to calculate the event state feature vector.

[0010] In one possible implementation, calculating the overall quality score for each time window includes: Based on the time synchronization accuracy, sample coverage, original signal quality, anomaly handling and compensation, and node online status of each time window, sub-scores of time reliability, sample coverage, signal quality, anomaly handling and compensation, and node health status are generated respectively. The comprehensive quality score is obtained by weighting and summing the time reliability sub-score, sample coverage sub-score, signal quality sub-score, anomaly handling and compensation sub-score, and node health status sub-score.

[0011] In one possible implementation, the multimodal feature sequences after verification completion are weighted and fused to generate a synchronization fusion matrix, including: The physiological feature vector, environmental feature vector, environmental exposure load factor, event state feature vector, and comprehensive quality score are concatenated to form a synchronous fusion vector; Based on the comprehensive quality score, scene adaptive fusion weights are generated by combining user context state and environmental exposure load factors. The scene adaptive fusion weights are then normalized with the preset basic modality fusion weights to obtain the actual fusion weights. Based on the actual fusion weights, the feature vectors of each modality in the multimodal feature sequence after verification and completion are weighted and combined to form a synchronous fusion matrix.

[0012] In one possible implementation, when the time credibility sub-score of multiple consecutive time windows is lower than the credibility threshold or the anomaly handling and compensation sub-score continues to deteriorate, feedback control is triggered. The feedback control includes at least one of increasing the node synchronization frequency, reducing the weight of the corresponding node in the fusion result when performing node recalibration, and switching to a degraded operation mode.

[0013] In one possible implementation, the output mode of the synchronization fusion matrix is ​​determined by an output gating function, which includes a comprehensive quality score, consistency verification results of temporal and cross-modal correlations, and key modal availability.

[0014] On the other hand, the present invention also provides a multimodal health monitoring data processing system, comprising: The data acquisition module is used to acquire raw health monitoring data, including physiological data, environmental data, and event status data, and generate structured observation samples with local timestamps based on the raw health monitoring data, user context status identifiers, and abnormal event detection functions. A data processing module, connected to the data acquisition module, includes: The clock synchronization module is used to recursively estimate the clock offset and drift parameters of each acquisition node with the local computing node as the master clock node, and dynamically adjust the node synchronization cycle according to the user context status identifier and the detection results of the abnormal event detection function. The global time series sample generation module is used to map the local sampling time of each structured observation sample to a global reference time based on the clock offset and drift parameters and the synchronization period, and to generate the time uncertainty of each structured observation sample by combining the activity disturbance factor and the network fluctuation factor, so as to obtain the global time series sample. The observation stream generation module is used to perform out-of-order rearrangement and time rollback correction on the global time series samples, mark the abnormal event sampling window according to the detection result of the abnormal event detection function, and obtain a time-ordered observation stream. The differential mapping module is used to construct a unified time grid and use a differential strategy to map physiological data, environmental data and event state data in the observation stream to the unified time grid to obtain a multimodal feature sequence. The verification and compensation module is used to perform temporal and cross-modal correlation consistency verification on the multimodal feature sequence, perform missing compensation in stages according to the duration of missing data, and identify the key modality availability status of each time window to obtain the verified and completed multimodal feature sequence, temporal and cross-modal correlation consistency verification results, and key modality availability. The output management module, connected to the data processing module, is used to calculate the comprehensive quality score for each time window, and to generate a synchronization fusion matrix by weighted fusion of the verified and completed multimodal feature sequences. Based on the comprehensive quality score, the consistency verification results of time series and cross-modal correlation, and the availability of key modes, the synchronization fusion matrix is ​​output in normal output mode, local degradation output mode, and conservative output mode.

[0015] Secondly, the present invention also provides an electronic device, comprising: Data acquisition equipment is used to collect multi-source heterogeneous observation data, including physiological data, environmental data, and event status data. Memory, used to store programs; The processor, coupled to the data acquisition device and the memory respectively, is used to execute the program stored in the memory to implement the steps in the multimodal health monitoring data processing method described in any of the above implementations.

[0016] The beneficial effects of this invention are as follows: The multimodal health monitoring data processing method provided by this invention improves the cross-modal time alignment accuracy during high-risk periods under conditions of weak home network, low power consumption, and disconnection / reconnection by recursively estimating the clock offset and drift parameters of each node using the local computing node as the master clock, and dynamically adjusting the node synchronization cycle in conjunction with user context state and abnormal events. This ensures the temporal consistency of multi-source heterogeneous data under a unified time reference. Furthermore, by mapping the local sampling time to the global reference time and generating sample-level time uncertainty by combining activity disturbance factors and network fluctuation factors, a quantifiable assessment of time alignment error is achieved. Further, by constructing a unified time grid and employing a differentiated strategy to map physiological, environmental, and event data to the unified time grid, the sampling frequency and signal variation patterns of different modal data are taken into account. This avoids the loss of details in high-frequency physiological signals and the amplification of noise in low-frequency environmental signals caused by a single interpolation method, achieving accurate alignment of multi-source heterogeneous data on a unified time axis and ensuring the accuracy of the cross-modal correspondence between physiological and environmental data. In addition, consistency verification and hierarchical missing compensation reduce the risk of misfusion caused by local missing data, activity artifacts or scene conflicts. The reliability of the fusion results is comprehensively improved by calculating a comprehensive quality score to control the credibility of the fusion results. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic flowchart of an embodiment of the multimodal health monitoring data processing method provided by the present invention; Figure 2 For the present invention Figure 1 A schematic diagram of an embodiment of S102; Figure 3 For the present invention Figure 1 A schematic diagram of an embodiment of S102; Figure 4 For the present invention Figure 1 A schematic diagram of an embodiment of S105; Figure 5 For the present invention Figure 1 A schematic diagram of an embodiment of S107; Figure 6 For the present invention Figure 1 Another embodiment of the process diagram of S107; Figure 7 This is a schematic diagram of an embodiment of the multimodal health monitoring data processing system provided by the present invention; Figure 8 A schematic diagram of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0020] In the description of the embodiments of the present invention, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0021] The terms "first," "second," etc., used in the embodiments of this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] This invention provides a multimodal health monitoring data processing method, system, and electronic device, which are described below.

[0024] Figure 1 This is a schematic flowchart of an embodiment of the multimodal health monitoring data processing method provided by the present invention. The executing entity of the method can be a local computing node deployed in a home environment, including but not limited to a home health gateway, an edge computing node, a supporting processing terminal for smart wearable devices, or other embedded devices with corresponding computing capabilities. It is mainly applied to home health monitoring scenarios. Figure 1 As shown, the multimodal health monitoring data processing method includes: S101. Obtain raw health monitoring data including physiological data, environmental data, and event status data. Based on the raw health monitoring data and user context status identifiers and abnormal event detection functions, generate structured observation samples with local timestamps.

[0025] It should be noted that in the home health monitoring scenario, raw health monitoring data is collected through multiple physiological sensor nodes, environmental sensor nodes, and device status nodes. Among them, physiological sensor nodes are used to collect physiological data such as the user's heart rate, blood oxygen, respiration, body movement, and skin temperature; environmental sensor nodes are used to collect environmental data such as indoor temperature, humidity, PM2.5, CO2, light intensity, and noise; and device status nodes are used to collect the terminal's online status, communication status, synchronization status, battery status, cache status, and local inference load status.

[0026] Considering that users in home scenarios exhibit typical behavioral states such as resting, activity, falling asleep, turning over at night, and getting out of bed to use the toilet, this embodiment of the invention introduces a user context state identifier to characterize the current monitoring period and behavioral background. At the same time, it introduces the detection results of an abnormal event detection function to identify abnormal conditions and key events in the monitoring data in real time. Based on the detection results of the abnormal event detection function, it can be used to characterize whether time t is in an abnormal event triggering state or within the event encryption sampling window.

[0027] Furthermore, the user's actual health status during the monitoring period is considered as a continuously changing potential state process, and the outputs of different sensor nodes are considered as asynchronous observations of this potential state process at different times. Each sensor node at the local sampling time The observation can be expressed as:

[0028] in, Indicates the first Observations from each sensor node Indicates the observation mapping relationship. This indicates the potential health status at the global reference time. This indicates environmental disturbances, behavioral disturbances, or equipment operating condition inputs. Indicates the user's context state. This indicates measurement noise.

[0029] It should be understood that each structured observation sample includes at least a node number, data type identifier, local sampling time, original observation value, sampling sequence number, device status information, and initial quality identifier. The outputs of different sensor nodes are not regarded as independent data streams, but are uniformly modeled as asynchronous projections of the same potential health state process under different home environment conditions, thereby providing a unified observation basis for subsequent cross-modal time-consistent reconstruction.

[0030] S102. Using the local computing node as the master clock node, recursively estimate the clock offset and drift parameters of each acquisition node, and dynamically adjust the node synchronization period according to the user context status identifier and abnormal event detection function.

[0031] S103. Based on the clock offset and drift parameters and the synchronization period, the local sampling time of each structured observation sample is mapped to the global reference time, and the time uncertainty of each structured observation sample is generated by combining the activity disturbance factor and the network fluctuation factor to obtain the global time series sample.

[0032] S104. Perform out-of-order rearrangement and time rollback correction on the global time series samples, mark the abnormal event sampling window according to the detection result of the abnormal event detection function, and obtain a time-ordered observation stream.

[0033] S105. Construct a unified time grid and use a differentiated strategy to map the physiological data, environmental data and event state data in the observation stream to the unified time grid to obtain a multimodal feature sequence.

[0034] S106. Perform time series and cross-modal correlation consistency verification on the multimodal feature sequence, perform missing compensation according to the duration of missing data, identify the key modal availability status of each time window, and obtain the verified and completed multimodal feature sequence, time series and cross-modal correlation consistency verification results, and key modal availability.

[0035] S107. Calculate the comprehensive quality score for each time window, and generate a synchronization fusion matrix by weighted fusion of the multimodal feature sequences after verification and completion; based on the comprehensive quality score, the consistency verification results of time series and cross-modal correlation, and the availability of key modes, output the synchronization fusion matrix in normal output mode, local degradation output mode, and conservative output mode.

[0036] In summary, the multimodal health monitoring data processing method provided in this invention improves cross-modal time alignment accuracy during high-risk periods under conditions of weak home network, low power consumption, and disconnection / reconnection by recursively estimating the clock offset and drift parameters of each node using the local computing node as the master clock, and dynamically adjusting the node synchronization cycle in conjunction with user context state and abnormal events. This ensures the temporal consistency of multi-source heterogeneous data under a unified time reference. Furthermore, by mapping the local sampling time to the global reference time and generating sample-level time uncertainty by combining activity disturbance factors and network fluctuation factors, a quantifiable assessment of time alignment error is achieved. Further, by constructing a unified time grid and employing a differentiated strategy to map physiological, environmental, and event data to the unified time grid, the sampling frequency and signal variation patterns of different modal data are taken into account. This avoids the loss of details in high-frequency physiological signals and the amplification of noise in low-frequency environmental signals caused by a single interpolation method, achieving accurate alignment of heterogeneous data on a unified time axis and ensuring the accuracy of the cross-modal correspondence between physiological and environmental data. In addition, consistency verification and hierarchical missing compensation reduce the risk of misfusion caused by local missing data, activity artifacts or scene conflicts. The reliability of the fusion results is comprehensively improved by calculating a comprehensive quality score to control the credibility of the fusion results.

[0037] Because the local clocks of different sensor nodes exhibit offsets, drifts, and random jitter, and wearable devices in home scenarios often reduce communication frequencies due to low-power strategies, home wireless networks may also experience periodic jitter. In some embodiments of this invention, such as... Figure 2 As shown, the recursive estimation of clock offset and drift parameters for each acquisition node in step S102 includes: S201. Establish the relationship between the local clock of each acquisition node and the global reference time, wherein the relationship includes clock drift coefficient, clock offset and random clock disturbance term; S202. Establish a clock state vector for each acquisition node, including the current offset and the current drift coefficient, and construct the state transition equation and observation equation. S203. The clock state of each acquisition node is updated using the recursive state estimation method to obtain the offset estimate, drift estimate and estimated covariance at the current time.

[0038] The relationship between the local clock of each acquisition node and the global reference time is expressed as follows:

[0039] In the formula, Indicates the local clock. Indicates global reference time. Indicates the clock drift coefficient. Indicates the clock offset. This represents the random clock disturbance term.

[0040] To accommodate crystal drift and link delay variations during long-term operation, a clock state vector containing the current offset and current drift coefficient is established for each node, expressed by the following formula:

[0041] In the formula, Represents the clock state vector. Indicates the current offset. This represents the current drift coefficient.

[0042] The state transition equation and observation equation are expressed as follows:

[0043]

[0044] In the formula, This represents the deviation observation obtained from the master-slave clock message exchange. Indicates clock state noise. Indicates observation noise. This represents the node clock state transition matrix, used to describe the evolution of offset and drift coefficient within adjacent synchronization cycles. This represents the timing observation matrix, used to establish the mapping relationship between message exchange observations and node clock states.

[0045] A recursive state estimation method is used to update the clock state of each acquisition node by combining the state transition equation and the observation equation, thereby obtaining the offset estimate, drift estimate, and estimated covariance of each node at the current time. If the rate of change of deviation exceeds a preset deviation threshold after two consecutive synchronizations, the synchronization frequency of that node is increased; if the link jitter is greater than a preset jitter threshold, the timing result of this synchronization is temporarily suspended, and the previous stable clock state is maintained.

[0046] like Figure 3 As shown, in some embodiments of the present invention, step S102, which dynamically adjusts the node synchronization period based on the user context state identifier and the detection result of the abnormal event detection function, includes: S301. Preset a basic synchronization period, determine the scene adjustment coefficient based on the user context state identifier and the detection result of the abnormal event detection function, and use the product of the basic synchronization period and the scene adjustment coefficient as the actual synchronization period. S302. When the user is in a state of sleep monitoring, rest observation or abnormal event triggering, the scene adjustment coefficient is less than the preset value, thus shortening the actual synchronization cycle. S303. When the user is in a normal daytime activity state and no abnormal event occurs, the scene adjustment coefficient is greater than or equal to the preset value, extending or maintaining the actual synchronization cycle.

[0047] The formula for expressing the actual synchronization period is as follows:

[0048] In the formula, For context state identifier, This represents the detection result of the abnormal event detection function. Based on the synchronization period, This is a scene adjustment factor; it is applied when the user is in sleep monitoring, rest observation, or an abnormal event trigger state. When the user is in a normal daytime activity state and no abnormal events occur, .

[0049] Based on the clock offset and drift parameters, the local sampling time of each structured observation sample is mapped to the global reference time, and the time uncertainty of each sample is generated by combining the activity disturbance factor and the network fluctuation factor, thus obtaining the global time series sample.

[0050] For the Local sampling time of each node Its corresponding global reference time estimate Represented as:

[0051] in, This is an estimate of the node drift coefficient. This is an estimated value for the node offset.

[0052] Simultaneously, based on the current time state covariance, link jitter status, and user activity status, the time uncertainty for each sample is recorded and expressed as time variance. .

[0053] To reflect the combined impact of home activity disturbances and network fluctuations on time reliability, time uncertainty correction terms for activity perception and network perception are defined, resulting in the corrected time uncertainty. The formula is as follows:

[0054] in, This represents the activity disturbance factor obtained from body motion intensity, posture switching frequency, or wearing stability. Indicates network volatility factor. and These are the adjustment coefficients for the activity disturbance factor and the network volatility factor, respectively. Therefore, each observation sample is expanded to be represented as:

[0055] in, Indicates an expanded observation sample. This represents the global reference time estimate. Represents the original observation value. This represents the original signal quality score. Additional markers are added to indicate the device status and synchronization status.

[0056] If a sample satisfies the time uncertainty If the time uncertainty is less than or equal to a preset time threshold, it is marked as a high-confidence time sample; if the time uncertainty is less than or equal to a preset time threshold, it is marked as a high-confidence time sample. If the time uncertainty exceeds the preset time threshold, it is allowed to proceed to subsequent processing, but a weighting process is performed during the inter-frequency mapping stage; if L consecutive samples from the same node satisfy the time uncertainty... If the time exceeds the preset time threshold, the node will be triggered to recalibrate. It should be understood that when the cloud connection is interrupted, the local computing node maintains the most recent stable global time reference and drift estimation parameters, continues to perform time mapping on newly arrived samples in local autonomous mode, and accumulates and records the autonomous runtime. When the autonomous runtime exceeds a preset running threshold, the system automatically increases the time reliability penalty coefficient to avoid the time error being underestimated under long-term offline conditions.

[0057] The global time series samples are subjected to out-of-order rearrangement and time rollback correction. The abnormal event sampling window is marked according to the detection result of the abnormal event detection function to obtain a time-ordered observation stream.

[0058] Specifically, for samples with the same node number, sampling sequence number, and timestamp, the one with the higher quality score is retained. An out-of-order tolerance window is set. For those who are late by no more than For samples exceeding a certain time, re-insert and sort them at the same time; for samples exceeding a certain time, ... The samples marked as late entries will not participate in real-time scoring but can participate in offline resubmission. If a negative jump is detected in the timestamp of a sample uploaded after a node reconnects, relative to the tail time of the current buffer, and the time difference exceeds a preset threshold, it is determined to be a time rollback event. For time rollback events, the system performs continuous time smoothing correction on the reconnected batch of samples based on the node's most recent stable drift rate to eliminate timeline breaks caused by batch cache uploads. If a node fails to generate a valid sample within the required sampling period, it is marked as a short-term missing sample, a medium-term missing sample, or a long-term missing sample based on the duration of the missing data. Samples exhibiting obvious jumps, physical out-of-bounds errors, or inconsistencies with device status are initially marked as anomalies. When a critical home monitoring event is detected, such as a sudden increase in heart rate, a potential apnea, abnormal bed exit, a sudden increase in noise, or a rapid change in temperature and humidity, the system will revolve around the event trigger time. Construct an event encryption sampling window, expressed by the following formula:

[0059] in, Event encryption sampling window, and These represent the pre-event backtracking duration and the post-event tracking duration, respectively. Within this window, relevant nodes increase the data reporting frequency and synchronization frequency to more precisely capture the physiological-environmental coupling changes before and after the occurrence of abnormal home events, and set the detection results of the abnormal event detection function for the corresponding time period to a valid state. .

[0060] After the above preprocessing, a standardized observation stream with time reliability, signal quality, callback flag, event window flag, and anomaly flag is obtained.

[0061] In the current analysis period The local computing node stores the data in memory at preset time steps. Building a unified time grid , represented as: .

[0062] To accommodate the different temporal resolution requirements of daytime activity monitoring and nighttime sleep monitoring in home health surveillance, an adaptive time granularity switching mechanism for sensitive monitoring periods is introduced. The base time step is set as follows: The actual time step Represented as:

[0063] in, Indicates the user context state and the detection results of the abnormal event detection function The time granularity adjustment coefficient is jointly determined. This applies during sleep monitoring periods, resting observation periods, and within the event encryption sampling window. During the ordinary steady-state monitoring phase, . After establishing a unified time grid, to avoid distortion caused by using a single interpolation method for high-frequency physiological data, low-frequency environmental data, and discrete event data, in some embodiments of the present invention, such as... Figure 4 As shown, step S105 employs a differentiation strategy to map physiological data, environmental data, and event state data in the observation stream to the unified time grid, including: S401. For physiological data falling within each window of a unified time grid, extract the physiological feature vector of each window by using the temporal proximity and quality score of the samples as the joint weighting basis.

[0064] Among them, regarding physiological data, for those falling into the first... Time window High-frequency physiological sample set within Statistical features are extracted using a combined weighting method of time proximity and quality score. First, the formula for expressing window-level comprehensive time uncertainty is defined as follows:

[0065] in, Indicates the overall time uncertainty. Indicates time uncertainty, This represents the original signal quality score. The formula for its weighted mean is:

[0066] In the formula, This represents the weighted average. Represents a set of high-frequency physiological samples. Indicates the weight.

[0067] Among them, weight The formula for expressing it is:

[0068] In the formula, This represents the global reference time estimate. Indicates the center reference time of the time grid window. Indicates the overall time uncertainty. This represents the original signal quality score.

[0069] Physiological feature vectors are further extracted within each time window, expressed by the following formula:

[0070] in, For physiological feature vectors, For weighted average, Standard deviation, The maximum value of the window. The minimum value of the window. is the coefficient of variation. Considering the impact of activity artifacts on high-frequency physiological signals in home settings, an activity inhibition weight is added to the time window when the intensity of body movement exceeds a preset threshold, so as to avoid misjudging short-term fluctuations caused by strenuous activity as health abnormalities.

[0071] S402. For environmental data, cubic splines with smoothing regularization terms are used to reconstruct the continuous trend of environmental observation points, obtain environmental feature vectors at a unified time grid, and construct environmental exposure load factors based on the load mapping function and weight coefficients corresponding to each environmental variable.

[0072] Specifically, for low-frequency continuous environmental data such as temperature, humidity, PM2.5, and CO2, cubic splines with smoothing regularization terms are used for reconstruction. Let the environmental observation point be... Then the reconstructed function Determined through the following objectives:

[0073] in, This serves as the global reference sampling time for environmental observation samples. These are the raw measurements of the environmental observation samples. The smoothing regularization parameter is used to balance the fitting accuracy and smoothness of the environmental reconstruction curve, thereby suppressing the influence of high-frequency noise from the environmental sensor on the continuous trend reconstruction. At a unified time grid point The environmental feature vector is obtained from the above. .

[0074] To better reflect the cumulative physiological load of the home environment on the human body, an environmental exposure load factor was constructed based on environmental characteristics. The formula is as follows:

[0075] in, Indicates the first The standardized load mapping function corresponding to each environmental variable is used to map the original environmental observations into comparable load components; These represent the weighting coefficients of the corresponding environmental variables, used to describe the relative magnitude of the contribution of each environmental factor to home health risk. The environmental exposure load factor... It is used to enhance the ability to depict home health risk scenarios such as high temperature and humidity, deteriorating air quality, and stuffy nighttime environments.

[0076] S403. For event state data, different methods are used to assign state values ​​to time grid points according to different time state data, and an event state credibility decay factor is introduced to calculate the event state feature vector.

[0077] For discrete event data such as door and window opening / closing, fan start / stop, light status, and bed exit status, a state duration interval expansion method is used to assign corresponding state values ​​to all time grid points between the event occurrence and the next state change. For instantaneously triggered events, a nearest neighbor zero-order hold or window Boolean logic aggregation strategy is used for mapping. If an event falls within a time window... If an event occurs within the specified timeframe, the corresponding event feature position is set to 1; otherwise, it is set to 0, thus forming an event state feature vector. .

[0078] To prevent a state that has not been updated for a long time from being indefinitely equated to a real state, an event state credibility decay factor is introduced when the state exceeds the maximum credible duration since the last valid update. The formula is as follows:

[0079] in, As an event status credibility decay factor, The attenuation coefficient is... The update time is the most recent valid event.

[0080] Suppose the first result obtained based on window Boolean logic aggregation is... The initial event state value of the class is ,in Then the corresponding event feature elements after decay correction are represented as:

[0081] in, For event feature elements, As an event status credibility decay factor, This is the initial event state value. It is determined by the corresponding event category. Combine to form event state feature vector This allows event states that have not been updated for a long time to automatically reduce their impact during the fusion process.

[0082] The consistency of temporal and cross-modal correlations is verified on the multimodal feature sequence formed by the physiological feature vector, environmental feature vector, and event state feature vector. Missing compensation is performed in stages according to the duration of missing data, and the availability status of key modalities in each time window is identified. The results of verification and completion of the multimodal feature sequence, consistency verification of temporal and cross-modal correlations, and availability of key modalities are obtained.

[0083] Among them, a consistency index for time series and cross-modal correlation is defined. Perform time-series and cross-modal correlation consistency checks on multimodal feature sequences, expressed by the following formula:

[0084] in, This represents a consistency judgment function between temporal and cross-modal correlations, used to evaluate physiological feature vectors. Environmental feature vectors and event state feature vector With user context state identifier The degree of logical matching between them. When When the value is below the preset threshold, conflict marking and local deweighting are performed first, rather than directly performing high-confidence compensation.

[0085] After passing the consistency check of temporal and cross-modal correlation, if there is a missing or local distortion within a certain time window, hierarchical compensation and state smoothing estimation are performed according to the duration of the missing data. When the duration of the missing information is less than or equal to the first preset duration, it is determined to be a short-term missing information; when the duration of the missing information is between the first preset duration and the second preset duration, it is determined to be a medium-term missing information; when the duration of the missing information is greater than the second preset duration, it is determined to be a long-term missing information. For short-term missing data, compensation is achieved using local interpolation or extension from the previous stable state. For medium-term missing data and local disturbances, a local state-space model is established, and the state equation of the local state-space model is expressed as follows:

[0086] In the formula, Indicates the first Potential state prediction values ​​under a time window Indicates the first Potential state vectors under each time window This represents environment-driven input and context-assisted input. Indicates process noise. This represents the state transition matrix, used to describe the evolution of potential healthy states between adjacent time windows. This represents the control input matrix, used to introduce the driving effect of environmental disturbances and contextual inputs on the potential state.

[0087] The observation equation of the local state-space model is expressed as follows:

[0088] In the formula, represents the first... A vector of observations within a time window Indicates the first Potential state vectors under each time window Indicates observation noise. This represents the observation matrix, used to establish the mapping relationship between the potential state and the current window observations. State estimates within the missing intervals are obtained through recursive filtering and smoothing estimation. Then, the compensated observation values ​​are obtained through inversion using the observation equation. .

[0089] For cases of long-term missing modes or simultaneous loss of multiple key modes, no forced compensation is performed. Instead, a missing mode flag is output, and the corresponding mode is placed in a degraded state. If the missing mode has a stable correlation with other modes, only relevant auxiliary flags are output, and pseudo-observations are not directly constructed.

[0090] This invention, through contextual consistency verification of multimodal feature sequences, can exclude observation combinations that clearly do not conform to the logic of a home scenario. For example, if the out-of-bed state remains true while the intensity of body movement in bed continues to increase, it is determined to be a state conflict; if doors and windows are closed for a long time and CO2 continues to rise, it is determined to be an enhanced environmental occlusion state. Furthermore, based on the duration of missing data, hierarchical missing data compensation is performed. In logical conflict situations such as the out-of-bed state and continuously increasing body movement, conflicts are directly marked instead of blindly interpolating. Simultaneously, the availability status of key modalities in each time window is identified, effectively reducing the risk of misfusion caused by local missing data, activity artifacts, or scene conflicts. The reliability of the fusion results is comprehensively improved by using a comprehensive quality score to control the credibility of the fusion results.

[0091] like Figure 5 As shown, in some embodiments of the present invention, step S107, calculating the comprehensive quality score for each time window, includes: S501. Generate time reliability sub-score, sample coverage sub-score, signal quality sub-score, anomaly handling and compensation sub-score, and node health status sub-score based on the time synchronization accuracy, sample coverage, original signal quality, anomaly handling and compensation status, and node online status of each time window. S502. The time reliability sub-score, sample coverage sub-score, signal quality sub-score, anomaly handling and compensation sub-score, and node health status sub-score are weighted and summed to obtain a comprehensive quality score.

[0092] The formula for expressing the comprehensive quality score is as follows:

[0093] In the formula, Indicates the time reliability sub-score. This represents the sample coverage sub-score. Indicates the signal quality sub-score. This indicates the sub-score for exception handling and compensation. This represents a sub-score indicating the node's health status. , , , , These are the corresponding weight coefficients, satisfying... The weights can be dynamically initialized based on prior knowledge such as home network environment, user monitoring mode and device stability, and can be adaptively adjusted during operation by combining historical quality statistics.

[0094] The comprehensive quality score, as credibility context information, participates in the final fusion process. In some embodiments of the present invention, when multiple consecutive time windows of time credibility sub-scores... Below the confidence threshold or anomaly handling and compensation sub-score When the situation continues to deteriorate, feedback control is triggered. The feedback control includes at least one of the following: increasing the node synchronization frequency, reducing the weight of the corresponding node in the fusion result when performing node recalibration, and switching to a degraded operation mode.

[0095] like Figure 6 As shown, in some embodiments of the present invention, step S107, which involves weighted fusion of the verified and completed multimodal feature sequences to generate a synchronization fusion matrix, includes: S601. The physiological feature vector, environmental feature vector, environmental exposure load factor, event state feature vector and comprehensive quality score are concatenated to form a synchronous fusion vector. S602. Based on the comprehensive quality score, and combined with the user context state and environmental exposure load factor, generate scene adaptive fusion weights. Normalize the scene adaptive fusion weights with the preset basic modal fusion weights to obtain the actual fusion weights. S603. Based on the actual fusion weights, the modal feature vectors in the multimodal feature sequence after verification and completion are weighted and combined to form a synchronous fusion matrix.

[0096] The formula for expressing the scene adaptive fusion weights is as follows:

[0097] In the formula, Indicates scene adaptive fusion weights, This represents the detection result from the anomaly event detection function. Environmental exposure load factors and user context state identifier The weighted adjustment function, which is jointly determined, increases the fusion weight of features related to respiration, body movement, and environmental suffocation during the nighttime sleep monitoring phase; and increases the fusion weight of features related to activity inhibition and homeostasis recovery during the daytime activity monitoring phase.

[0098] The formula for expressing the actual fusion weight is:

[0099] In the formula, Indicates the actual fusion weight. Represents the basic modal fusion weights. Indicates scene adaptive fusion weights, This represents the product of corresponding elements. This represents the normalization operator, used to ensure that the adjusted modal weights meet the preset constraints.

[0100] Within each time window of the unified time grid, physiological characteristics, environmental characteristics, event state characteristics, environmental exposure load factors, and comprehensive quality scores are concatenated to form a synchronous fusion vector. The formula is as follows:

[0101] In the formula, Represents physiological feature vectors, Represents the environmental feature vector. Represents the event state feature vector. Indicates environmental exposure load factor, This indicates the overall quality score.

[0102] Based on the actual fusion weights, the weights of each component of the synchronous fusion vector are adjusted to obtain the scene adaptive fusion representation. Its formula is:

[0103] In the formula, Indicates the actual fusion weight. Represents the synchronous fusion vector. This represents the weighting operator for modal grouping, used to group and weight the components in the synchronization fusion vector.

[0104] Finally, the fusion vectors of all time windows are arranged in chronological order to form a multimodal synchronous fusion matrix.

[0105] In some embodiments of the present invention, the output mode of the synchronization fusion matrix is ​​determined by an output gating function, which includes a comprehensive quality score, a context consistency check result, and a critical modality availability.

[0106] Before outputting the fusion matrix, an output permission flag is set to determine the security constraints for the home scenario. If the current window meets the criteria of missing key modalities, severe contextual conflicts, or a quality score continuously below the security threshold, then generating high-confidence risk inputs is prohibited, and only conservative monitoring results are retained. The formula for the output permission flag is:

[0107] In the formula, Indicates the output permission flag. The overall quality score is represented by the expression. Context consistency metrics and key modality availability The output gating function is jointly determined. Indicates the first The set of availability states of key physiological modalities, key environmental modalities, and key event modalities within a time window. Based on the availability and quality score of the key modalities within the current time window, the output state of the synchronization fusion matrix is ​​divided into normal output mode, locally degraded output mode, and conservative output mode. Specifically, when the key modalities are complete, the overall quality score meets the threshold requirement, and a permission flag is output... When output is permitted, a complete fusion matrix is ​​output, and the edge inference engine is allowed to perform high-confidence risk identification or linkage control; this is classified as normal output mode. When some environmental or event modalities are unavailable but key physiological modalities remain complete, a partial fusion matrix is ​​output with a degradation flag attached, while unnecessary linkage actions are restricted; this is classified as partial degradation output mode. When key physiological modalities fail, the overall quality score remains below the safety threshold, or a permission flag is output... When output is prohibited, only the conservative monitoring status is output. At this time, it is classified as conservative output mode. No high-confidence risk input is generated. Only basic monitoring records, conservative prompts or manual review requests are maintained, thereby maintaining continuous, interpretable and self-protective monitoring output in a real home environment.

[0108] To better implement the multimodal health monitoring data processing method in the embodiments of the present invention, based on the multimodal health monitoring data processing method, correspondingly, as follows: Figure 7 As shown in the figure, this embodiment of the invention also provides a multimodal health monitoring data processing system. The multimodal health monitoring data processing system 700 includes: The data acquisition module 701 is used to acquire raw health monitoring data including physiological data, environmental data and event status data, and generate structured observation samples with local timestamps based on the raw health monitoring data, user context status identifiers and abnormal event identifiers. Data processing module 702, connected to data acquisition module 701, includes: The clock synchronization module is used to recursively estimate the clock offset and drift parameters of each acquisition node with the local computing node as the master clock node, and dynamically adjust the node synchronization cycle according to the user context status identifier and the detection results of the abnormal event detection function. The global time series sample generation module is used to map the local sampling time of each structured observation sample to a global reference time based on the clock offset and drift parameters and the synchronization period, and to generate the time uncertainty of each structured observation sample by combining the activity disturbance factor and the network fluctuation factor, so as to obtain the global time series sample. The observation stream generation module is used to perform out-of-order rearrangement and time rollback correction on the global time series samples, mark the abnormal event sampling window according to the detection result of the abnormal event detection function, and obtain a time-ordered observation stream. The differential mapping module is used to construct a unified time grid and use a differential strategy to map physiological data, environmental data and event state data in the observation stream to the unified time grid to obtain a multimodal feature sequence. The verification and compensation module is used to perform context consistency verification on the multimodal feature sequence, perform missing compensation in stages according to the duration of missing data, and identify the availability status of key modalities in each time window, so as to obtain the verified and completed multimodal feature sequence, context consistency verification results and key modal availability. The output management module 703, connected to the data processing module 702, is used to calculate the comprehensive quality score for each time window, and to generate a synchronization fusion matrix by weighted fusion of the multimodal feature sequences after verification and completion. Based on the comprehensive quality score, context consistency verification results, and key modality availability, the synchronization fusion matrix is ​​output in normal output mode, local degradation output mode, and conservative output mode.

[0109] The multimodal health monitoring data processing system 700 provided in the above embodiments can implement the technical solutions described in the above multimodal health monitoring data processing method embodiments. The specific implementation principles of each module or unit can be found in the corresponding content in the above multimodal health monitoring data processing method embodiments, and will not be repeated here.

[0110] like Figure 8 As shown, the present invention also provides an electronic device 800. The electronic device 800 includes: Data acquisition device 801 is used to collect raw health monitoring data, including physiological data, environmental data, and event status data. Memory 802 is used to store programs; The processor 803, coupled to the data acquisition device 801 and the memory 802, is used to execute the program stored in the memory 802 to implement the steps in the multimodal health monitoring data processing method described in any of the above implementations. Figure 8 Only some components of the sub-device 800 are shown; however, it should be understood that implementation of all shown components is not required, and more or fewer components may be implemented instead.

[0111] In some embodiments, processor 803 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in memory 802 or process data, such as the multimodal health monitoring data processing method of the present invention.

[0112] In some embodiments, processor 803 may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, processor 803 may be local or remote. In some embodiments, processor 803 may be implemented on a cloud platform. In one embodiment, the cloud platform may include a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, intranet, multi-cloud, etc., or any combination thereof.

[0113] In some embodiments, memory 802 may be an internal storage unit of electronic device 800, such as a hard disk or memory of electronic device 800. In other embodiments, memory 802 may also be an external storage device of electronic device 800, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 800.

[0114] Furthermore, the memory 802 may include both internal storage units of the electronic device 800 and external storage devices. The memory 802 is used to store application software and various types of data installed on the electronic device 800.

[0115] In one embodiment, when the processor 803 executes the multimodal health monitoring data processing program in the memory 802, the following steps can be implemented: Acquire raw health monitoring data including physiological data, environmental data, and event status data. Based on the raw health monitoring data and the detection results of the user context status identifier and the abnormal event detection function, generate structured observation samples with local timestamps. Using the local computing node as the master clock node, the clock offset and drift parameters of each acquisition node are recursively estimated, and the node synchronization cycle is dynamically adjusted based on the user context status identifier and the detection results of the abnormal event detection function. Based on the clock offset and drift parameters and the synchronization period, the local sampling time of each structured observation sample is mapped to the global reference time, and the time uncertainty of each structured observation sample is generated by combining the activity disturbance factor and the network fluctuation factor to obtain the global time series sample. The global time series samples are subjected to out-of-order rearrangement and time rollback correction. The abnormal event sampling window is marked according to the detection result of the abnormal event detection function to obtain a time-ordered observation stream. A unified time grid is constructed, and a differentiated strategy is used to map physiological data, environmental data, and event state data in the observation stream to the unified time grid to obtain a multimodal feature sequence. Context consistency verification is performed on the multimodal feature sequence, missing compensation is performed in stages according to the duration of missing data, and the availability status of key modalities in each time window is identified to obtain the verified and completed multimodal feature sequence, context consistency verification results, and key modal availability. Calculate the comprehensive quality score for each time window, and generate a synchronization fusion matrix by weighted fusion of the verified and completed multimodal feature sequences. Based on the comprehensive quality score, context consistency verification results, and key modality availability, output the synchronization fusion matrix in normal output mode, local degradation output mode, and conservative output mode.

[0116] It should be understood that when the processor 803 executes the multimodal health monitoring data processing program in the memory 802, in addition to the functions mentioned above, it can also perform other functions, as detailed in the description of the corresponding method embodiments above.

[0117] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0118] The multimodal health monitoring data processing method, system, and electronic device provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for processing multimodal health monitoring data, characterized in that, include: Acquire raw health monitoring data including physiological data, environmental data, and event status data; and generate structured observation samples with local timestamps based on the raw health monitoring data, user context status identifiers, and abnormal event detection functions. Using the local computing node as the master clock node, the clock offset and drift parameters of each acquisition node are recursively estimated, and the node synchronization cycle is dynamically adjusted based on the user context status identifier and the detection results of the abnormal event detection function. Based on the clock offset and drift parameters and the synchronization period, the local sampling time of each structured observation sample is mapped to the global reference time, and the time uncertainty of each structured observation sample is generated by combining the activity disturbance factor and the network fluctuation factor to obtain the global time series sample. The global time series samples are subjected to out-of-order rearrangement and time rollback correction. The abnormal event sampling window is marked according to the detection result of the abnormal event detection function to obtain a time-ordered observation stream. A unified time grid is constructed, and a differentiated strategy is used to map physiological data, environmental data, and event state data in the observation stream to the unified time grid to obtain a multimodal feature sequence. The temporal and cross-modal correlation consistency verification is performed on the multimodal feature sequence, the missing compensation is performed in stages according to the duration of missing data, and the key modality availability status of each time window is identified to obtain the verified and completed multimodal feature sequence, the temporal and cross-modal correlation consistency verification results, and the key modality availability. Calculate the comprehensive quality score for each time window, and generate a synchronization fusion matrix by weighted fusion of the verified and completed multimodal feature sequences. Based on the comprehensive quality score, the consistency verification results of time series and cross-modal correlation, and the availability of key modes, output the synchronization fusion matrix in normal output mode, local degradation output mode, and conservative output mode.

2. The multimodal health monitoring data processing method according to claim 1, characterized in that, The recursive estimation of clock offset and drift parameters for each acquisition node includes: Establish the relationship between the local clock of each acquisition node and the global reference time. The relationship includes the clock drift coefficient, clock offset, and random clock disturbance term. Establish a clock state vector containing the current offset and current drift coefficient for each acquisition node, and construct the state transition equation and observation equation; The clock state of each acquisition node is updated using a recursive state estimation method to obtain the offset estimate, drift estimate and estimated covariance at the current time.

3. The multimodal health monitoring data processing method according to claim 2, characterized in that, The step of dynamically adjusting the node synchronization period based on the user context state identifier and the detection result of the abnormal event detection function includes: A basic synchronization period is preset, and a scene adjustment coefficient is determined based on the user context state identifier and the detection result of the abnormal event detection function. The product of the basic synchronization period and the scene adjustment coefficient is used as the actual synchronization period. When the user is in a state of sleep monitoring, rest observation, or abnormal event triggering, the scene adjustment coefficient is less than the preset value, thus shortening the actual synchronization cycle. When a user is in a normal daytime activity state and no abnormal events occur, the scenario adjustment coefficient is greater than or equal to the preset value, thus extending or maintaining the actual synchronization cycle.

4. The multimodal health monitoring data processing method according to claim 2, characterized in that, A differentiated strategy is employed to map physiological data, environmental data, and event state data from the observation stream to the unified time grid, including: For physiological data falling within each window of a unified time grid, the physiological feature vector of each window is extracted based on the temporal proximity and quality score of the samples as a joint weighting criterion. For environmental data, cubic splines with smoothing regularization terms are used to reconstruct the continuous trend of environmental observation points, obtain environmental feature vectors at a unified time grid, and construct environmental exposure load factors based on the load mapping function and weight coefficients corresponding to each environmental variable. For event state data, different methods are used to assign state values ​​to time grid points based on different time state data, and an event state credibility decay factor is introduced to calculate the event state feature vector.

5. The multimodal health monitoring data processing method according to claim 1, characterized in that, The calculation of the overall quality score for each time window includes: Based on the time synchronization accuracy, sample coverage, original signal quality, anomaly handling and compensation, and node online status of each time window, sub-scores for time reliability, sample coverage, signal quality, anomaly handling and compensation, and node health status are generated respectively. The comprehensive quality score is obtained by weighting and summing the time reliability sub-score, sample coverage sub-score, signal quality sub-score, anomaly handling and compensation sub-score, and node health status sub-score.

6. The multimodal health monitoring data processing method according to claim 4, characterized in that, The step of weighted fusing the verified and completed multimodal feature sequences to generate a synchronization fusion matrix includes: The physiological feature vector, environmental feature vector, environmental exposure load factor, event state feature vector, and comprehensive quality score are concatenated to form a synchronous fusion vector; Based on the comprehensive quality score, scene adaptive fusion weights are generated by combining user context state and environmental exposure load factors. The scene adaptive fusion weights are then normalized with the preset basic modality fusion weights to obtain the actual fusion weights. Based on the actual fusion weights, the feature vectors of each modality in the multimodal feature sequence after verification and completion are weighted and combined to form a synchronous fusion matrix.

7. The multimodal health monitoring data processing method according to claim 5, characterized in that, When the time credibility sub-score for multiple consecutive time windows is lower than the credibility threshold or the anomaly handling and compensation sub-score continues to deteriorate, feedback control is triggered. The feedback control includes at least one of the following: increasing the node synchronization frequency, reducing the weight of the corresponding node in the fusion result when performing node recalibration, and switching to a degraded operation mode.

8. The multimodal health monitoring data processing method according to claim 1, characterized in that, The output mode of the synchronization fusion matrix is ​​determined by an output gating function, which includes a comprehensive quality score, consistency verification results of temporal and cross-modal correlation, and key modal availability.

9. A multimodal health monitoring data processing system, characterized in that, include: The data acquisition module is used to acquire raw health monitoring data, including physiological data, environmental data, and event status data, and generate structured observation samples with local timestamps based on the raw health monitoring data, user context status identifiers, and abnormal event detection functions. A data processing module, connected to the data acquisition module, includes: The clock synchronization module is used to recursively estimate the clock offset and drift parameters of each acquisition node with the local computing node as the master clock node, and dynamically adjust the node synchronization cycle according to the user context status identifier and the detection results of the abnormal event detection function. The global time series sample generation module is used to map the local sampling time of each structured observation sample to a global reference time based on the clock offset and drift parameters and the synchronization period, and to generate the time uncertainty of each structured observation sample by combining the activity disturbance factor and the network fluctuation factor, so as to obtain the global time series sample. The observation stream generation module is used to perform out-of-order rearrangement and time rollback correction on the global time series samples, mark the abnormal event sampling window according to the detection result of the abnormal event detection function, and obtain a time-ordered observation stream. The differential mapping module is used to construct a unified time grid and use a differential strategy to map physiological data, environmental data and event state data in the observation stream to the unified time grid to obtain a multimodal feature sequence. The verification and compensation module is used to perform temporal and cross-modal correlation consistency verification on the multimodal feature sequence, perform missing compensation in stages according to the duration of missing data, and identify the key modality availability status of each time window to obtain the verified and completed multimodal feature sequence, temporal and cross-modal correlation consistency verification results, and key modality availability. The output management module, connected to the data processing module, is used to calculate the comprehensive quality score for each time window, and to generate a synchronization fusion matrix by weighted fusion of the verified and completed multimodal feature sequences. Based on the comprehensive quality score, the consistency verification results of time series and cross-modal correlation, and the availability of key modes, the synchronization fusion matrix is ​​output in normal output mode, local degradation output mode, and conservative output mode.

10. An electronic device, characterized in that, include: Data acquisition equipment is used to collect raw health monitoring data, including physiological data, environmental data, and event status data. Memory, used to store programs; A processor, coupled to the data acquisition device and the memory, is configured to execute the program stored in the memory to implement the steps of the method according to any one of claims 1 to 8.

Images