A multi-modal somatosensory information processing method and system for physiological state assessment

Abstract

The present application relates to the technical field of medical data processing, and particularly relates to a multi-modal somatosensory information processing method and system for physiological state evaluation, which comprises: multi-modal parameter acquisition; consistency feature construction; individual normal behavior band modeling; consistency trajectory compression generation; deviation path feature extraction; physiological trajectory similarity matching; state type and stage determination; behavior band adaptive correction; and health intervention prompt output. By simultaneously introducing multi-source physiological signals, using the time sequence collaborative relationship of different types of parameters, and constructing a cross-modal consistency feature matrix, the evaluation process is no longer dependent on the isolated changes of a single signal, but is based on the comprehensive determination of the linked evolution of multi-dimensional information, effectively solving the problem that the physiological state evaluation is easily affected by individual differences and long-term state changes, resulting in misjudgment and drift due to the dependence on a single physiological signal and static evaluation benchmark.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical data processing, and in particular to a multi-modal somatosensory information processing method and system for physiological state assessment. BACKGROUND

[0002] With the rapid development of wearable devices, intelligent monitoring terminals and remote health services, people's demand for continuous and refined assessment of physiological state is increasing, especially in the scenes of chronic disease management, sub-health monitoring, rehabilitation assessment and health protection of high-pressure workers. How to objectively identify the real physiological state of individuals in a non-invasive and non-laboratory environment has become a common concern in the industry. Existing technologies mostly rely on single physiological indicators or local behavior characteristics for judgment, which are easily affected by individual differences, short-term emotional fluctuations, environmental noise and behavior disguises, resulting in insufficient stability and reliability of the assessment results. At the same time, different individuals have significant differences in long-term life rhythm, body state and behavior habits, making it difficult for unified thresholds and static standards to be applicable for a long time, and thus misjudgment, delayed judgment or state drift distortion may occur in actual application. How to achieve continuous, reliable and evolvable assessment of physiological state under the realistic conditions of strong coupling of multi-source information, complex state evolution, significant individual differences and long-term dynamic changes still remains a key challenge to be broken through in the field of intelligent health monitoring.

[0003] Chinese patent application publication No. CN116919360A discloses a multi-modal integrated closed-loop active health system and method. The system includes a multi-modal physiological signal monitoring module, a health state assessment module, a multi-modal health intervention module and a host computer module. The multi-modal physiological signal monitoring module is connected to the input end of the health state assessment module and is used to monitor physiological signals and send the monitored physiological signals to the health state assessment module. The health state assessment module is connected to the input end of the multi-modal health intervention module and the first input end of the host computer module, and is used to comprehensively assess the health state of a user according to the received physiological signals to obtain a health state assessment result. The multi-modal health intervention module is connected to the input end of the multi-modal physiological signal monitoring module and the first output end of the health state assessment module, and is used to perform multi-modal physical stimulation on corresponding acupoints according to the health state assessment result. The host computer module is also connected to another output end of the multi-modal physiological signal monitoring module, and is used to display physiological signals and store health state assessment results.

[0004] Therefore, the multimodal integrated closed-loop active health system has the following problems: the system's health status assessment is mainly based on physiological signal input, lacking multimodal collaborative analysis, resulting in limited representation dimensions for complex physiological states; the system adopts an immediate assessment method based on current physiological signals, without establishing a dynamic reference benchmark for long-term individual behavioral characteristics, which is prone to assessment bias when there are large individual differences or long-term slow changes in status; the system's health status determination results are directly mapped to intervention strategies, lacking continuous stage analysis of the physiological state evolution process, making it difficult to finely characterize the process of changes in status from mild to severe or from severe to slow. Summary of the Invention

[0005] To address this, the present invention provides a multimodal somatosensory information processing method for physiological state assessment. This method overcomes the problem in existing technologies where physiological state assessment is susceptible to misjudgment and drift due to reliance on a single physiological signal and static assessment benchmark, which is caused by dynamic modeling and trajectory-level analysis of the co-evolutionary characteristics of multi-source somatosensory information.

[0006] To achieve the above objectives, the present invention provides a multimodal somatosensory information processing method for physiological state assessment, comprising:

[0007] Real-time acquisition of multimodal somatosensory parameters of the target object, including facial expression micro-movement parameters, body posture parameters, speech acoustic parameters, and physiological signal parameters, including heart rate variability (HRV), skin conductance (EDA), and respiratory rate.

[0008] Based on the aforementioned multimodal somatosensory parameters, a cross-modal consistency feature matrix is ​​constructed to characterize the collaborative change relationship between visual modality, posture modality, speech modality, and physiological modality on the same time scale;

[0009] Based on the cross-modal consistency characteristics of the target object within a preset historical period, a corresponding individual normal somatosensory behavior band is constructed to characterize the range of multimodal collaborative changes of the target object under non-abnormal conditions.

[0010] The current cross-modal consistency features are subjected to trajectory compression processing to generate a somatosensory consistency trajectory;

[0011] Based on the spatial deviation relationship between the somatosensory consistency trajectory and the individual's normal somatosensory behavior band, the corresponding deviation path features are extracted;

[0012] The deviation path features are input into a preset physiological state trajectory database for similarity matching to obtain several matching results;

[0013] The physiological state type and corresponding evolutionary stage of the target object are determined based on the matching results.

[0014] The individual's normal somatosensory behavior band is modified based on the continuous distribution characteristics of the evolutionary stage within a preset stable duration;

[0015] Output health intervention suggestions based on the physiological state type.

[0016] Furthermore, the facial expression micro-movement parameters include eyebrow displacement amplitude, eyelid closure frequency, and mouth corner stretching rate;

[0017] The body posture parameters include trunk forward tilt angle, limb curling range, and posture stability.

[0018] The speech acoustic parameters include speech fundamental frequency, speech rate, pause ratio, and energy fluctuation rate.

[0019] Furthermore, the cross-modal consistency feature matrix is ​​jointly constructed based on the temporal synchronization degree, amplitude co-variation rate, and directional correlation among the visual modality, the posture modality, the speech modality, and the physiological modality.

[0020] Furthermore, the individual normal somatic behavior band is obtained by aggregate modeling based on the cross-modal consistency features of the target object within the preset historical time window.

[0021] Furthermore, the trajectory compression process includes performing temporal dimensionality reduction or key node extraction on all the cross-modal consistency features within a preset trajectory compression time to generate the somatosensory consistency trajectory used to characterize the current multimodal collaborative change trend.

[0022] Furthermore, the deviation path features are determined based on the comprehensive deviation features of the somatosensory consistency trajectory relative to the individual's normal somatosensory behavior band in terms of direction, amplitude, and duration.

[0023] Furthermore, the physiological state trajectory library includes pain evolution trajectory, fatigue evolution trajectory, and stress evolution trajectory, and the pain evolution trajectory, fatigue evolution trajectory, and stress evolution trajectory are all composed of trajectory samples formed by the evolution of the multimodal cross-consistency features over time.

[0024] Furthermore, the evolutionary stages include an initial stage, a sustained stage, and a mitigation stage, which characterize the degree of evolution of the target object's current physiological state.

[0025] Furthermore, the process of modifying the individual's normal somatosensory behavior band based on the continuous distribution characteristics of the evolutionary stage within a preset stable duration includes:

[0026] Obtain a continuous distribution sequence of deviation path features corresponding to the evolutionary stage within the preset stable duration;

[0027] Based on the boundary offset relationship between the continuous distribution sequence and the individual's normal somatosensory behavior band, the corresponding correction direction and correction magnitude are determined;

[0028] The upper and lower boundaries of the individual's normal somatosensory behavior band are adaptively updated based on the correction direction and correction magnitude to correct the individual's normal somatosensory behavior band.

[0029] On the other hand, the present invention also provides a multimodal somatosensory information processing system for physiological state assessment, comprising:

[0030] The acquisition module is used to acquire multimodal somatosensory parameters of the target object in real time. The multimodal somatosensory parameters include facial expression micro-movement parameters, body posture parameters, speech acoustic parameters, and physiological signal parameters. Among them, the physiological signal parameters include heart rate variability (HRV), skin conductance (EDA), and respiratory rate.

[0031] The matrix construction module is used to construct a cross-modal consistency feature matrix based on the multimodal somatosensory parameters to characterize the collaborative change relationship between visual modality, posture modality, speech modality and physiological modality on the same time scale;

[0032] The behavior band construction module is used to construct a corresponding individual normal somatosensory behavior band based on the cross-modal consistency characteristics of the target object within a preset historical period, which is used to characterize the range of multimodal collaborative changes of the target object in a non-abnormal state.

[0033] The compression module is used to perform trajectory compression processing on the current cross-modal consistency features to generate a somatosensory consistency trajectory.

[0034] The extraction module is used to extract the corresponding deviation path features based on the spatial deviation relationship between the somatosensory consistency trajectory and the individual's normal somatosensory behavior band.

[0035] The matching module is used to input the deviation path features into a preset physiological state trajectory database for similarity matching to obtain several matching results;

[0036] The determination module is used to determine the physiological state type and corresponding evolutionary stage of the target object based on the matching result;

[0037] The correction module is used to correct the individual's normal somatosensory behavior band based on the continuous distribution characteristics of the evolutionary stage within a preset stable duration;

[0038] The output module is used to output health intervention prompts based on the physiological state type.

[0039] Compared with existing technologies, the beneficial effects of this invention are as follows: By simultaneously introducing multi-source physiological signals such as facial micro-movements, body posture, speech acoustic features, heart rate variability, skin conductance, and respiratory rate, and utilizing the synergistic relationships of different types of parameters in terms of temporal variation rhythm, amplitude fluctuation trend, and direction of change, a cross-modal consistency feature matrix is ​​constructed. This allows the evaluation process to no longer rely on isolated changes of a single signal, but rather to make a comprehensive judgment based on the linkage evolution of multi-dimensional information. Furthermore, by establishing individual normal somatic behavior bands, the system can characterize the multi-modal synergistic change range of different individuals in a stable state, effectively avoiding the problem of insufficient adaptability of group models to individual differences. On this basis, by performing trajectory compression and deviation path extraction on the consistency features, the continuous change process is transformed into a trajectory feature with trend directionality, making the physiological state change process more transparent and predictable. The process transforms from discrete judgment to continuous evolutionary analysis; by combining a pre-set physiological state trajectory database to complete matching and evolutionary stage division, it can not only identify the current physiological state type, but also determine whether the state is in the initial, ongoing, or remission stage; at the same time, by dynamically correcting the individual's normal somatic behavior band within a stable period, the assessment baseline is automatically updated with the individual's long-term state changes, thereby effectively avoiding misjudgment problems caused by baseline drift. It significantly improves the comprehensive performance of physiological state assessment in terms of temporal continuity, individual adaptability, state classification accuracy, and intervention trigger reliability. It is particularly suitable for real-time dynamic assessment and precise intervention scenarios of complex states such as pain, fatigue, and stress, and effectively solves the problem that physiological state assessment is easily affected by individual differences and long-term state changes due to reliance on a single physiological signal and static assessment benchmark, resulting in misjudgment and drift.

[0040] Furthermore, by jointly collecting and fusing facial expression micro-movement parameters such as eyebrow displacement amplitude, eyelid closure frequency, and mouth corner stretching rate, body posture parameters such as trunk forward tilt angle, limb curling amplitude, and posture stability, and speech acoustic parameters such as speech fundamental frequency, speech rate, pause ratio, and energy fluctuation rate, it is possible to simultaneously characterize the physiological stress response, pain tolerance, and fatigue accumulation state of the target object from multiple levels, such as facial muscle activity intensity, overall body defensive posture, and vocal modulation changes. This effectively avoids recognition bias caused by scene occlusion, individual differences, or short-term occasional fluctuations of single modal parameters, thereby significantly improving the overall reliability of physiological state assessment results in terms of temporal continuity, state discrimination, and sensitivity to early anomaly recognition.

[0041] Furthermore, by synchronizing visual, postural, vocal, and physiological modalities under a unified time reference, and jointly characterizing the linkage relationships between different modalities from three dimensions—the degree of temporal synchronization, the intensity of amplitude co-variation, and the consistency of change direction—it is possible to form stable and quantifiable linkage feature expressions for changes in facial muscle activity, body posture contraction or instability, fluctuations in vocal modulation, and changes in heart rate variability, skin conductance, and respiratory rhythm on the same time scale. When the target object experiences changes in state such as pain, discomfort, fatigue, or stress, the above multimodal parameters often exhibit a combination of synchronous aggravation, amplitude co-enhancement, and consistent trend changes. This cross-modal consistency feature matrix is ​​a concentrated characterization of this linkage characteristic, thereby enabling physiological state assessment to no longer rely on the absolute numerical judgment of a single signal, but to make a comprehensive judgment based on the co-variation relationship between multi-source somatosensory information, significantly improving the stability, robustness, and sensitivity to early abnormal changes in state recognition.

[0042] Furthermore, by continuously collecting, outlier-eliminating, and statistically modeling the cross-modal consistency characteristics of the target object in a non-abnormal state within a preset historical period, the long-term stable range of synergistic changes among facial micro-expression activity intensity, body posture stability, speech acoustic fluctuation characteristics, heart rate variability, skin conductance, and respiratory rhythm is accurately characterized. This range is then used to form an individual's normal somatic behavior band in the form of a multidimensional feature envelope. This behavior band can truly reflect the upper and lower bounds of changes in different modalities under normal physiological regulation and their linkage strength. When an individual experiences changes in state such as pain, fatigue, or stress, the synergistic relationship between the above modalities often manifests as amplitude amplification, synchronous enhancement, or trend shift, which can be sensitively identified and judged as deviation events by this behavior band. This allows subsequent consistency trajectory deviation analysis to be based on the individual's own long-term stable somatic characteristics, avoiding misjudgments caused by group thresholds or static standards, and improving the individual adaptability, stability, and reliability of long-term continuous monitoring of physiological state assessment.

[0043] Furthermore, by continuously tracing and smoothing cross-modal consistency features within a preset trajectory compression time and combining temporal dimensionality reduction and key node extraction, the originally high-dimensional, dense, and easily affected by transient disturbances multimodal collaborative change process is transformed into a structurally clear, continuous, and interpretable somatosensory consistency trajectory. This trajectory can simultaneously preserve the dominant trend of the time-dependent co-evolution of facial micro-movements, posture stability, speech acoustic fluctuations, and heart rate, skin conductance, and respiratory rhythm, while effectively reducing the interference of short-term anomalies, transient noise, and sampling jitter on the overall judgment. This allows subsequent judgments on the direction, rate, and duration of collaborative change deviations to be based on a stable and reliable continuous trajectory, avoiding misjudgments caused by discrete feature point jumps, and improving the smoothness, continuity, and stability of stage judgment in the physiological state evolution process recognition.

[0044] Furthermore, by extracting deviation path features, the changes in the somatosensory consistency trajectory relative to the individual's normal somatosensory behavior band in three dimensions—direction, amplitude, and duration—are jointly characterized. This can simultaneously reflect the trend direction, deviation intensity, and duration of multimodal physiological and behavioral signals shifting from a stable state to an abnormal state, enabling effective differentiation between transient fluctuations and persistent abnormalities. Among these, the deviation direction characterizes the directional feature of the evolution of multimodal collaborative changes from an equilibrium state to a specific imbalance state; the deviation amplitude reflects the magnitude difference in the intensity of corresponding physiological load or emotional activation; and the deviation duration reflects the dynamic relationship between the body's regulatory capacity and the cumulative effect of abnormal states. The joint encoding of these three aspects enables the deviation path features to not only have instantaneous discrimination capabilities but also continuous representation capabilities of the state evolution process, thus providing a more stable, discriminative, and noise-insensitive temporal feature basis for subsequent physiological state trajectory similarity matching.

[0045] Furthermore, by constructing a standard trajectory library formed by the evolution of multimodal cross-consistency features corresponding to different physiological states such as pain, fatigue, and stress in historical samples over time, and by performing similarity matching on a unified scale between the real-time extracted deviation path features and various evolutionary trajectories in terms of morphological structure, direction of change, magnitude of change, and temporal evolution trend, the current multimodal somatic changes can be identified not only at the level of "whether it is abnormal" but also at the level of "which physiological state the abnormality belongs to" and "how the state develops over time". This trajectory evolution-based matching mechanism fully utilizes the differences in the rhythm of multimodal collaborative changes, intensity accumulation mode, and recovery characteristics of different physiological states, so that the matching results with high similarity correspond to closer physiological regulation processes in a statistical sense, thereby significantly improving the accuracy and stability of physiological state type determination and providing a reliable temporal reference basis for subsequent evolution stage determination.

[0046] Furthermore, by hierarchically comparing the deviation path characteristics with the evolutionary trajectories of the three physiological states of pain, fatigue, and stress at different time segments, we can achieve a high-confidence determination of the current physiological state type by utilizing the differences in intensity accumulation, sustained stability, and decline trend of multimodal synergistic changes. On the other hand, by combining the significant differences in the rate of change, amplitude level, and duration of time of trajectory segments at each stage, we can accurately locate the current state to the initial, ongoing, or remission stage. This elevates the identification of physiological states from a single result judgment to a two-dimensional judgment mode of "type + stage," which can not only reflect the strength of the body's current regulatory load but also reveal whether the physiological state is in a critical stage of intensification, stabilization, or gradual recovery. This provides a more granular and reliable decision-making basis for the dynamic correction of subsequent behavioral bands and the temporal control of intervention strategies.

[0047] Furthermore, by cumulatively analyzing the continuous distribution of deviation path characteristics within a preset stable duration under the current physiological state evolution stage, the long-term high, low, or continuous drift trends in multimodal collaborative changes can be effectively distinguished from instantaneous fluctuations. Combined with the directionality and cumulative amplitude of continuous offset at the boundary of the behavior band, the upper and lower boundaries of the individual's normal somatosensory behavior band are adaptively adjusted. This allows the behavior band to evolve synchronously with the target object's physiological adaptation level, fatigue accumulation, or stress relief process. This avoids overcorrection caused by short-term abnormalities and prevents the failure of the original normal range due to long-term physiological state migration. Thus, while dynamically maintaining the stability of the individual's baseline, it improves the continuity, accuracy, and long-term reliability of subsequent deviation judgment, physiological state matching, and health intervention output.

[0048] Furthermore, by synchronously acquiring and jointly modeling facial micro-movements, body posture, speech acoustics, and physiological signals such as HRV, EDA, and respiratory rate, the system can uniformly characterize changes in an individual's overt behavior and the state of autonomic nervous activity within the body. This allows for the overall perception and quantitative expression of the synergistic changes in multimodal features of states such as emotional fluctuations, fatigue accumulation, and pain responses. Through dynamic comparison of cross-modal consistency feature matrices with an individual's normal somatic behavior bands, the system can not only distinguish between short-term, occasional fluctuations and persistent changes in physiological state, but also accurately reflect the state evolution trend by utilizing the changes in direction, amplitude, and duration of deviation paths. Furthermore, by combining typical evolutionary patterns in the physiological state trajectory database, the system can achieve refined identification of different state types and their stages. Through adaptive correction of behavior bands, the system can continuously track changes in an individual's long-term physical condition and adaptability, thereby maintaining assessment sensitivity while avoiding the accumulation of misjudgments. Ultimately, this ensures that health intervention outputs are both real-time targeted and possess long-term stability and individual adaptability. Attached Figure Description

[0049] Figure 1 This is a flowchart of the multimodal somatosensory information processing method for physiological state assessment in this embodiment;

[0050] Figure 2 As shown, this is a flowchart of constructing a cross-modal consistency feature matrix based on multimodal somatosensory parameters in this embodiment;

[0051] Figure 3 This embodiment constructs a flowchart of the corresponding individual normal somatosensory behavior band;

[0052] Figure 4 This is a schematic diagram of the multimodal somatosensory information processing system used for physiological state assessment in this embodiment. Detailed Implementation

[0053] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0054] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0055] On the one hand, please refer to Figure 1 As shown, this is a flowchart of the multimodal somatosensory information processing method for physiological state assessment in this embodiment. This embodiment provides a multimodal somatosensory information processing method for physiological state assessment, including:

[0056] S1. Real-time acquisition of multimodal somatosensory parameters of the target object. The multimodal somatosensory parameters include facial expression micro-movement parameters, body posture parameters, speech acoustic parameters, and physiological signal parameters. Among them, the physiological signal parameters include heart rate variability (HRV), skin conductance (EDA), and respiratory rate.

[0057] S2. Based on the multimodal somatosensory parameters, construct a cross-modal consistency feature matrix to characterize the collaborative change relationship between visual modality, posture modality, speech modality and physiological modality on the same time scale;

[0058] S3. Based on the cross-modal consistency characteristics of the target object within a preset historical period, construct a corresponding individual normal somatosensory behavior band to characterize the range of multimodal collaborative changes of the target object under non-abnormal conditions;

[0059] S4. Perform trajectory compression processing on the current cross-modal consistency features to generate a somatosensory consistency trajectory;

[0060] S5. Based on the spatial deviation relationship between the somatosensory consistency trajectory and the individual's normal somatosensory behavior band, extract the corresponding deviation path features;

[0061] S6. Input the deviation path features into a preset physiological state trajectory database for similarity matching to obtain several matching results;

[0062] S7. Determine the physiological state type and corresponding evolutionary stage of the target object based on the matching results;

[0063] S8. Based on the continuous distribution characteristics of the evolutionary stage within a preset stable duration, the individual's normal somatic behavior band is corrected;

[0064] S9. Output health intervention suggestions based on the physiological state type.

[0065] In this embodiment, the acquisition of multimodal somatosensory parameters can be performed in application scenarios such as medical monitoring, rehabilitation training, psychological assessment, remote health management, or intelligent interaction. Facial expression micro-motion parameters include eyebrow displacement amplitude, eyelid closure frequency, and mouth corner stretching rate, which are continuously captured by a visible light camera or infrared camera positioned directly in front of the target object and extracted based on optical flow analysis, key point detection, or facial expression action unit encoding algorithms; body posture parameters include trunk forward tilt angle, limb curling amplitude, and posture stability, which are acquired by a depth camera, infrared structured light sensor, or inertial measurement unit (IMU) and analyzed by posture recognition algorithms or multi-joint angle calculation models; speech acoustic parameters include speech fundamental frequency, speech rate, and so on. The pause ratio and energy fluctuation rate are obtained by acquiring raw speech signals through environmental microphones or wearable voice acquisition devices, and then extracted using short-time Fourier transform, cepstral analysis, or speech endpoint detection algorithms. Heart rate variability (HRV) among physiological signal parameters is calculated by acquiring the heart rate interval through ECG patches, photoplethysmography (PPG) sensors, or wearable heart rate belts. Skin conductance (EDA) is obtained by acquiring skin impedance changes through conductivity sensing electrodes placed on the fingertips or wrist. Respiratory frequency is obtained through chest and abdominal breathing belts, millimeter-wave radar sensors, or respiratory demodulation algorithms based on PPG signals. Data collected by various sensors are synchronously transmitted to the data processing terminal via wired or wireless means, and after being aligned with a unified timestamp, are input as multimodal somatosensory parameters into subsequent processing flows.

[0066] The preset historical period is used to characterize the long-term statistical interval of multimodal somatosensory parameters of the target object in a stable state. Its value depends on the stability of the target object's physiological rhythm, work and rest patterns, and the requirements of the application scenario for the accuracy of individual baseline modeling. It is usually set between 24 hours and 30 days. In this embodiment, it is set to 7 days, which can ensure that both intraday and periodic fluctuations of the individual are covered, while avoiding behavioral drift interference introduced by excessively long historical data. The preset stable duration is used to characterize the continuous stable interval of parameter distribution of the target object in a certain physiological state evolution stage. Its value depends on the rate of change of the target object's physiological state, the real-time requirements of the monitoring scenario, and the time resolution requirements of the system for state evolution determination. It is usually set between 30 seconds and 30 minutes. In this embodiment, it is set to 5 minutes, which can ensure the statistical stability of the state distribution characteristics while taking into account the response sensitivity when the physiological state changes rapidly.

[0067] By simultaneously incorporating multi-source physiological signals such as facial micro-movements, body posture, speech acoustic features, heart rate variability, skin conductance, and respiratory rate, and utilizing the synergistic relationships of different types of parameters in terms of temporal rhythm, amplitude fluctuation trends, and direction of change, a cross-modal consistency feature matrix is ​​constructed. This allows the evaluation process to move beyond the isolated changes of a single signal and instead make a comprehensive judgment based on the interconnected evolution of multi-dimensional information. Furthermore, by establishing individual normal somatic behavior bands, the system can characterize the multimodal synergistic change ranges of different individuals under stable conditions, effectively avoiding the problem of insufficient adaptability of group models to individual differences. On this basis, by performing trajectory compression and deviation path extraction on the consistency features, the continuous change process is transformed into trend-oriented trajectory features, changing the physiological state change process from discrete judgment to continuous judgment. Continued evolutionary analysis; combined with a pre-set physiological state trajectory database, it completes matching and evolutionary stage division, which can not only identify the current physiological state type, but also determine whether the state is in the initial, ongoing, or remission stage; at the same time, by dynamically correcting the individual's normal somatic behavior band within a stable period, the assessment baseline is automatically updated with the individual's long-term state changes, thereby effectively avoiding the misjudgment problem caused by baseline drift. It significantly improves the comprehensive performance of physiological state assessment in terms of temporal continuity, individual adaptability, state classification accuracy, and intervention trigger reliability. It is particularly suitable for real-time dynamic assessment and precise intervention scenarios of complex states such as pain, fatigue, and stress, and effectively solves the problem that physiological state assessment is easily affected by individual differences and long-term state changes due to reliance on a single physiological signal and static assessment benchmark, resulting in misjudgment and drift.

[0068] Specifically, the facial expression micro-movement parameters include eyebrow displacement amplitude, eyelid closure frequency, and mouth corner stretching rate;

[0069] The body posture parameters include trunk forward tilt angle, limb curling range, and posture stability.

[0070] The speech acoustic parameters include speech fundamental frequency, speech rate, pause ratio, and energy fluctuation rate.

[0071] By jointly collecting and fusing facial expression micro-movement parameters such as eyebrow displacement amplitude, eyelid closure frequency, and mouth corner stretching rate; body posture parameters such as trunk forward tilt angle, limb curling amplitude, and postural stability; and speech acoustic parameters such as speech fundamental frequency, speech rate, pause ratio, and energy fluctuation rate, it is possible to simultaneously characterize the physiological stress response, pain tolerance, and fatigue accumulation state of the target object from multiple levels, including facial muscle activity intensity, overall body defensive posture, and vocal modulation changes. This effectively avoids recognition bias caused by scene occlusion, individual differences, or short-term occasional fluctuations of single modal parameters, thereby significantly improving the overall reliability of physiological state assessment results in terms of temporal continuity, state discrimination, and sensitivity to early anomaly recognition.

[0072] Specifically, the cross-modal consistency feature matrix is ​​composed of the temporal synchronization degree, amplitude co-variation rate, and correlation of change direction among the visual modality, the posture modality, the speech modality, and the physiological modality.

[0073] Please see Figure 2 The diagram shows a flowchart of constructing a cross-modal consistency feature matrix based on multimodal somatosensory parameters in this embodiment. In this embodiment, in S2, the process of constructing a cross-modal consistency feature matrix based on multimodal somatosensory parameters includes:

[0074] S21. The collected facial expression micro-movement parameters, body posture parameters, speech acoustic parameters and physiological signal parameters are timestamped according to a unified time base, and the data of each modality are resampled to the same time resolution to eliminate the influence of the difference in sampling frequency of different modalities on subsequent analysis.

[0075] S22. Based on the completion of time alignment, the data of each modality are denoised, normalized and amplitude standardized to obtain the standardized feature sequence corresponding to each modality.

[0076] S23. Within the same time window, calculate the temporal synchronization degree between visual modality and posture modality, visual modality and speech modality, visual modality and physiological modality, posture modality and speech modality, posture modality and physiological modality, and speech modality and physiological modality, respectively, to characterize the degree of synchronization of changes in different modal features in the time dimension;

[0077] S24. Calculate the rate of coordinated change of amplitude among the above modal combinations within the time window to characterize the coordinated relationship between the amplitude changes of different modal characteristics.

[0078] Simultaneously, the consistency correlation of the feature change direction among the above modal combinations is calculated to characterize the degree of consistency of the trend of different modal features changing over time.

[0079] S25. The temporal synchronization degree, amplitude co-change rate and change direction correlation are fused and arranged according to the corresponding modal combination to construct a cross-modal consistency feature matrix, which is used to uniformly represent the co-change structure of multimodal somatosensory information on the same time scale.

[0080] By synchronizing visual, postural, vocal, and physiological modalities under a unified time reference, and jointly characterizing the linkage relationships between different modalities from three dimensions—the degree of temporal synchronization, the intensity of amplitude co-variation, and the consistency of change direction—this approach enables facial muscle activity changes, body posture contraction or instability, vocal modulation fluctuations, and changes in heart rate variability, skin conductance, and respiratory rhythm to form stable and quantifiable linkage feature expressions on the same time scale. When the target subject experiences changes in state such as pain, discomfort, fatigue, or stress, the aforementioned multimodal parameters often exhibit a combination of synchronous aggravation, amplitude co-enhancement, and consistent trend changes. This cross-modal consistency feature matrix is ​​a concentrated characterization of this linkage characteristic, thus enabling physiological state assessment to no longer rely on the absolute numerical judgment of a single signal, but rather on a comprehensive judgment based on the co-variation relationship between multiple sources of somatosensory information. This significantly improves the stability, robustness, and sensitivity to early abnormal changes in state recognition.

[0081] Specifically, the individual normal somatic behavior band is obtained by aggregate modeling based on the cross-modal consistency features of the target object within the preset historical time window.

[0082] Please see Figure 3 As shown, this is a flowchart of the process for constructing the corresponding individual normal somatic behavior band in this embodiment. In this embodiment, in S3, the process of constructing the corresponding individual normal somatic behavior band based on the cross-modal consistency characteristics of the target object within a preset historical period includes:

[0083] S31. Obtain the cross-modal consistency feature matrix of the target object within a preset historical period as a historical sample set, wherein the historical sample set is used to characterize the multimodal collaborative change features of the target object under multiple non-abnormal state stages.

[0084] S32. Perform outlier removal on the cross-modal consistency feature matrix of the historical sample set to remove abnormal samples caused by occasional noise, short-term violent actions or collection anomalies, so as to ensure the stability of subsequent modeling data.

[0085] S33. Perform feature distribution statistics on the historical cross-modal consistency features after removing outliers, and calculate the mean distribution range, fluctuation range and cooperative change density distribution of each modal consistency feature in the historical period to characterize the multimodal cooperative change feature boundary of the target object in a non-abnormal state.

[0086] S34. Based on the statistical results of feature distribution, construct a multidimensional feature envelope structure to envelop the normal multimodal collaborative change range of the target object, and define the multidimensional feature envelope structure as the individual's normal somatosensory behavior band.

[0087] S35. The individual's normal somatic behavior band is used as a reference benchmark for subsequent somatic consistency trajectory deviation judgment and deviation path extraction, in order to distinguish whether the current multimodal collaborative change is within the range of the target object's non-abnormal behavior pattern.

[0088] By continuously collecting, outlier-eliminating, and statistically modeling the cross-modal consistency characteristics of target subjects in a non-abnormal state within a preset historical period, the long-term stable range of synergistic changes among facial micro-expression activity intensity, body posture stability, speech acoustic fluctuation characteristics, heart rate variability, skin conductance, and respiratory rhythm is accurately characterized. This is then presented as a multi-dimensional feature envelope to form an individual's normal somatic behavior band. This behavior band can realistically reflect the upper and lower bounds of changes in different modalities under normal physiological regulation and their linkage strength. When an individual experiences changes in state such as pain, fatigue, or stress, the synergistic relationship between the above modalities often manifests as amplitude amplification, synchronous enhancement, or trend shift, which can be sensitively identified and judged as deviation events by this behavior band. This allows subsequent consistency trajectory deviation analysis to be based on the individual's own long-term stable somatic characteristics, avoiding misjudgments caused by group thresholds or static standards, and improving the individual adaptability, stability, and reliability of long-term continuous monitoring of physiological state assessment.

[0089] Specifically, the trajectory compression process includes performing temporal dimensionality reduction or key node extraction on all the cross-modal consistency features within a preset trajectory compression time to generate the somatosensory consistency trajectory that characterizes the current multimodal collaborative change trend.

[0090] In this embodiment, in S4, the process of performing trajectory compression processing on the current cross-modal consistency features to generate a somatosensory consistency trajectory includes:

[0091] S41. At the current moment, trace back to obtain the cross-modal consistency feature matrix corresponding to each time sampling point within the preset trajectory compression time, and form the original multimodal consistency feature time series sequence;

[0092] S42. Perform time alignment and sequence smoothing on the original multimodal consistency feature time series to eliminate the influence of high-frequency noise caused by sampling jitter or short-term abnormal fluctuations, so as to ensure the continuity and stability of the trajectory shape.

[0093] S43. According to the preset trajectory compression strategy, perform temporal dimensionality reduction or key node extraction on the original multimodal consistency feature time series. The temporal dimensionality reduction is used to map the high-dimensional cross-modal consistency feature series to a low-dimensional feature space to retain the main cooperative change trend. The key node extraction is used to select feature nodes with significant cooperative change amplitude or change direction from the time series as trajectory skeleton points.

[0094] S44. The results after time-series dimensionality reduction or key node extraction are reorganized to generate a somatosensory consistency trajectory that can continuously characterize the current multimodal collaborative change and evolution trend. The somatosensory consistency trajectory is used as the basic input data for subsequent deviation path feature extraction and physiological state evolution stage determination.

[0095] The preset trajectory compression duration refers to the length of the time window used for continuous trajectory modeling of the current multimodal cross-consistency features. It depends on the typical response time of the target object's physiological state changes and the dynamic change rate of the collected multimodal signals. It is usually set between 30 seconds and 5 minutes. In this embodiment, it is set to 120 seconds, which can ensure the continuity and smoothness of the trajectory while taking into account the synchronous representation requirements of short-term physiological fluctuations and gradual state changes.

[0096] By continuously tracing and smoothing cross-modal consistency features within a preset trajectory compression time, and combining temporal dimensionality reduction and key node extraction, the originally high-dimensional, dense, and easily affected multimodal collaborative change process is transformed into a structurally clear, continuous, and interpretable somatosensory consistency trajectory. This trajectory can simultaneously preserve the dominant trend of the time-varying collaborative evolution between facial micro-movements, posture stability, speech acoustic fluctuations, and heart rate, skin conductance, and respiratory rhythm, while effectively reducing the interference of short-term anomalies, transient noise, and sampling jitter on the overall judgment. This allows subsequent judgments on the direction, rate, and duration of collaborative change deviations to be based on a stable and reliable continuous trajectory, avoiding misjudgments caused by discrete feature point jumps, and improving the smoothness, continuity, and stability of stage judgment in the recognition of physiological state evolution processes.

[0097] Specifically, the deviation path characteristics are determined based on the comprehensive deviation characteristics of the somatosensory consistency trajectory relative to the individual's normal somatosensory behavior band in terms of direction, amplitude, and duration.

[0098] In this embodiment, in S5, the process of extracting the corresponding deviation path features based on the spatial deviation relationship between the somatosensory consistency trajectory and the individual's normal somatosensory behavior band includes:

[0099] S51. Map the somatosensory consistency trajectory to the feature space corresponding to the individual's normal somatosensory behavior band, and compare the positional relationship of the somatosensory consistency trajectory relative to the boundary of the individual's normal somatosensory behavior band at each time sampling node.

[0100] S52. For trajectory segments in the somatosensory consistency trajectory that exceed the range of an individual's normal somatosensory behavior zone, calculate the corresponding deviation direction, deviation magnitude, and deviation duration. The deviation direction is used to characterize the changing direction of the somatosensory consistency trajectory relative to the center trajectory of the individual's normal somatosensory behavior zone; the deviation magnitude is used to characterize the spatial distance of the somatosensory consistency trajectory from the boundary of the individual's normal somatosensory behavior zone; and the deviation duration is used to characterize the length of time that the somatosensory consistency trajectory is continuously outside the individual's normal somatosensory behavior zone.

[0101] S53. Jointly encode the deviation direction, deviation magnitude and deviation duration to construct the corresponding deviation path description vector;

[0102] S54. Perform path smoothing and abnormal segment merging processing on the deviation path description vector to eliminate false deviations caused by instantaneous noise or short-term errors, and obtain stable deviation path features. Use the stable deviation path features as input features for subsequent physiological state trajectory similarity matching.

[0103] By extracting deviation path features, the changes in the somatosensory consistency trajectory relative to an individual's normal somatosensory behavior band in three dimensions—direction, amplitude, and duration—are jointly characterized. This can simultaneously reflect the trend direction, deviation intensity, and duration of multimodal physiological and behavioral signals shifting from a stable state to an abnormal state, enabling effective differentiation between transient fluctuations and persistent abnormalities. Specifically, the deviation direction characterizes the directional feature of the evolution of multimodal collaborative changes from an equilibrium state to a specific imbalance state; the deviation amplitude reflects the magnitude difference in the intensity of corresponding physiological load or emotional activation; and the deviation duration reflects the dynamic relationship between the body's regulatory capacity and the cumulative effect of abnormal states. The joint encoding of these three factors enables the deviation path features to not only have instantaneous discrimination capabilities but also continuous representation capabilities of the state evolution process. This provides a more stable, discriminative, and noise-insensitive temporal feature basis for subsequent physiological state trajectory similarity matching.

[0104] Specifically, the physiological state trajectory library includes pain evolution trajectory, fatigue evolution trajectory and stress evolution trajectory, and the pain evolution trajectory, fatigue evolution trajectory and stress evolution trajectory are all composed of trajectory samples formed by the evolution of the multimodal cross-consistency features over time.

[0105] In this embodiment, in S6, the process of inputting the deviation path features into a preset physiological state trajectory database for similarity matching to obtain several matching results includes:

[0106] S61. Construct a physiological state trajectory library, which includes pain evolution trajectory, fatigue evolution trajectory and stress evolution trajectory. The pain evolution trajectory, fatigue evolution trajectory and stress evolution trajectory are all composed of standard trajectory samples formed by the evolution of multimodal cross-consistency features of corresponding physiological states in historical samples over time.

[0107] S62. Perform feature standardization on the input deviation path features to make them consistent with the various evolutionary trajectory samples in the physiological state trajectory library in terms of feature dimension, time scale and amplitude range.

[0108] S63. Based on the degree of consistency between deviation path features and pain evolution trajectory, fatigue evolution trajectory and stress evolution trajectory in terms of trajectory shape, direction of change, magnitude of change and temporal evolution trend, calculate the similarity between deviation path features and various types of evolution trajectory samples respectively.

[0109] S64. Sort the matching results according to the size of each similarity result, select several trajectory samples with similarity higher than the preset matching threshold as valid matching results, and output the valid matching results for subsequent determination of the current physiological state type and corresponding evolution stage of the target object.

[0110] The similarity calculation is achieved through a dynamic time warping algorithm, which comprehensively considers the direction cosine similarity and amplitude difference between trajectory points on the aligned path.

[0111] By constructing a standard trajectory library formed by the evolution of multimodal cross-consistency features corresponding to different physiological states such as pain, fatigue, and stress in historical samples over time, and by performing similarity matching on a unified scale between real-time extracted deviation path features and various evolutionary trajectories in terms of morphological structure, direction of change, magnitude of change, and temporal evolution trend, current multimodal somatic changes can be identified not only at the level of "whether it is abnormal" but also at the level of "which physiological state the abnormality belongs to" and "how the state develops over time." This trajectory evolution-based matching mechanism fully utilizes the differences in the rhythm of multimodal coordinated changes, intensity accumulation methods, and recovery characteristics of different physiological states, so that high similarity matching results correspond to closer physiological regulatory processes in a statistical sense, thereby significantly improving the accuracy and stability of physiological state type determination and providing a reliable temporal reference basis for subsequent evolutionary stage determination.

[0112] Specifically, the evolutionary stages include an initial stage, a sustained stage, and a mitigation stage, which characterize the degree of evolution of the target object's current physiological state.

[0113] In this embodiment, in S7, the process of determining the physiological state type and corresponding evolutionary stage of the target object based on the matching result includes:

[0114] S71. Perform clustering statistics on each valid matching result obtained in S7, and respectively count the number of matches and the distribution of matching similarity corresponding to the pain evolution trajectory, fatigue evolution trajectory and stress evolution trajectory;

[0115] S72. Based on the number of matching results and the overall similarity of various physiological state evolution trajectories, determine the physiological state with the highest similarity and stability that meets the preset judgment conditions as the physiological state type of the target object.

[0116] S73. For the determined physiological state type, further compare the matching relationship between the deviation path features and the evolution trajectory of the physiological state type at different time segments, and calculate the matching similarity of the trajectory segments in the initial stage, the continuous stage, and the relief stage respectively.

[0117] S74. Based on the similarity comparison results between the deviation path features and the trajectory segments corresponding to the above three evolutionary stages, the stage corresponding to the trajectory segment with the highest similarity is determined as the current evolutionary stage of the target object.

[0118] S75. Output the current physiological state determination result, including the physiological state type and the evolutionary stage, as the basis for subsequent individual normal somatic behavior correction and health intervention prompt generation.

[0119] In this embodiment, the division of the initial stage, sustained stage, and relief stage of each standard trajectory sample in the physiological state trajectory library is determined by one of the following methods: Based on external synchronous annotation: When collecting the historical samples, the self-reported state intensity labels or key time points (such as the onset point, peak point, and relief point of pain stimulation) of the target object are recorded synchronously, and the trajectory is segmented into stages based on these labels or time points; Based on trajectory morphological feature analysis: Mathematical analysis is performed on the complete standard trajectory, and the significant turning points of the trajectory slope (first-order difference) or the extreme points of curvature (second-order difference) are identified as stage boundary points, and the trajectory is divided into three stages accordingly; Based on model fitting segmentation: The trajectory is fitted using a hidden Markov model or a piecewise linear regression model, and the model state transition points or piecewise connection points are used as stage boundaries.

[0120] By hierarchically comparing the deviation path characteristics with the evolutionary trajectories of three physiological states—pain, fatigue, and stress—at different time segments, this study leverages the differences in intensity accumulation, sustained stability, and decline trends of multimodal synergistic changes to achieve a high-confidence determination of the current physiological state type. Furthermore, by combining the significant differences in the rate of change, amplitude level, and duration of each stage's trajectory segments, the study precisely locates the current state to the initial, ongoing, or remission stage. This elevates the identification of physiological states from a single-result judgment to a two-dimensional "type + stage" determination model. This not only reflects the strength of the body's current regulatory load but also reveals whether the physiological state is in a critical stage of intensification, stabilization, or gradual recovery. Therefore, it provides a more granular and reliable decision-making basis for the dynamic correction of subsequent behavioral patterns and the temporal control of intervention strategies.

[0121] Specifically, the process of correcting the individual's normal somatic behavior band based on the continuous distribution characteristics of the evolutionary stage within a preset stable duration includes: obtaining a continuous distribution sequence of deviation path characteristics corresponding to the evolutionary stage within the preset stable duration; determining the corresponding correction direction and correction magnitude based on the boundary offset relationship between the continuous distribution sequence and the individual's normal somatic behavior band; and adaptively updating the upper and lower boundaries of the individual's normal somatic behavior band based on the correction direction and correction magnitude to correct the individual's normal somatic behavior band.

[0122] In this embodiment, in S8, the process of correcting the individual's normal somatosensory behavior band based on the continuous distribution characteristics of the evolutionary stage within a preset stable duration is as follows:

[0123] S81. After determining the current physiological state type of the target object and its corresponding evolutionary stage, the system continuously acquires the deviation path features corresponding to the evolutionary stage within the preset stable duration, and constructs a continuous distribution sequence of deviation path features in chronological order. The continuous distribution sequence is used to characterize the dynamic process of the target object deviating from the normal behavior zone in the current physiological state evolutionary stage.

[0124] S82. Compare the boundary mapping of the continuous distribution sequence with the individual's normal somatic behavior band, calculate the offset and offset trend of the current continuous distribution sequence at the upper or lower boundary of the behavior band, and determine the correction direction and correction magnitude of the individual's normal somatic behavior band based on the positive or negative direction of the offset and its cumulative change within the preset stable time. The correction direction is used to indicate whether the behavior band expands outward or contracts inward, and the correction magnitude is used to indicate the scale of the boundary adjustment of the behavior band.

[0125] S83. Based on the correction direction and the correction magnitude, the upper and / or lower boundaries of the individual's normal somatosensory behavior band are adaptively updated so that the corrected individual's normal somatosensory behavior band can dynamically fit the multimodal collaborative change characteristics of the target object in the current physiological state evolution stage, thereby avoiding distortion of the original normal behavior band due to long-term physiological state changes of the individual and improving the stability and adaptability of subsequent physiological state assessment.

[0126] The determination of the correction direction and correction magnitude can be achieved through the following exemplary method: calculate the instantaneous offset of each data point in the continuous distribution sequence relative to the corresponding boundary of the behavior band; perform moving average processing on the instantaneous offset sequence to obtain a smoothed moving average offset sequence; perform linear fitting on the moving average offset sequence and use its slope as a measure of the offset trend; if the last value (or mean) of the moving average offset sequence is positive and the offset trend is positive or stable, then the correction direction is determined to be outward expansion, otherwise it is inward contraction; the correction magnitude is set as the product of the last value (or mean) of the moving average offset sequence and a preset learning rate factor.

[0127] The preset learning rate factor is a dimensionless proportional coefficient, which depends on the individual's normal somatosensory behavior band's adaptive sensitivity to long-term physiological state migration and the system's trade-off between stability and convergence speed. It is usually set between 0.01 and 0.2. In this embodiment, it is set to 0.05, which can ensure smooth and continuous adjustment of the behavior band boundary while avoiding behavior band oscillation or excessive drift caused by excessive single offset.

[0128] By cumulatively analyzing the continuous distribution of deviation path characteristics within a preset stable time period under the current physiological state evolution stage, the long-term high, low, or continuous drift trends in multimodal collaborative changes can be effectively distinguished from instantaneous fluctuations. Combined with the directionality and cumulative amplitude of continuous offset at the boundary of the behavior band, the upper and lower boundaries of the individual's normal somatosensory behavior band are adaptively adjusted. This allows the behavior band to evolve synchronously with the target object's physiological adaptation level, fatigue accumulation, or stress relief process. This avoids overcorrection caused by short-term abnormalities and prevents the failure of the original normal range due to long-term physiological state migration. Thus, while dynamically maintaining the stability of the individual's baseline, it improves the continuity, accuracy, and long-term reliability of subsequent deviation judgment, physiological state matching, and health intervention output.

[0129] In this embodiment, in S9, the process of outputting health intervention prompts based on the physiological state type is as follows: when the target object is determined to be in a pain physiological state, intervention information including suggestions for adjusting rest posture, local relaxation training, or prompts for further medical examination is output; when the target object is determined to be in a fatigue physiological state, intervention prompts including suggestions for reducing the intensity of current activity, taking a short rest, replenishing fluids, or performing soothing stretches are output; when the target object is determined to be in a stress physiological state, intervention prompts including suggestions for deep breathing training, slowing down speech and movement rhythm, and reducing the intensity of external stimuli are output. The above health intervention prompts are presented to the target object or management personnel in real time through a display terminal, mobile client, or voice broadcast, thereby realizing immediate reminders and targeted adjustment guidance for the current physiological state. The above health intervention prompts for different physiological state types are set based on the typical bodily response mechanisms and corresponding non-pharmacological physiological and psychological regulation consensus in each state.

[0130] On the other hand, please see Figure 4 As shown, this is a schematic diagram of a multimodal somatosensory information processing system for physiological state assessment in this embodiment. This embodiment also provides a multimodal somatosensory information processing system for physiological state assessment, including:

[0131] The acquisition module is used to acquire multimodal somatosensory parameters of the target object in real time. The multimodal somatosensory parameters include facial expression micro-movement parameters, body posture parameters, speech acoustic parameters, and physiological signal parameters. Among them, the physiological signal parameters include heart rate variability (HRV), skin conductance (EDA), and respiratory rate.

[0132] A matrix construction module, connected to the acquisition module, is used to construct a cross-modal consistency feature matrix based on the multimodal somatosensory parameters to characterize the collaborative change relationship between visual modality, posture modality, speech modality and physiological modality on the same time scale.

[0133] A behavior band construction module, which is connected to the matrix construction module, is used to construct a corresponding individual normal somatosensory behavior band based on the cross-modal consistency characteristics of the target object within a preset historical period, which characterizes the range of multimodal collaborative changes of the target object in a non-abnormal state.

[0134] A compression module, which is connected to the behavior band construction module, is used to perform trajectory compression processing on the current cross-modal consistency features to generate a somatosensory consistency trajectory.

[0135] An extraction module, which is connected to the compression module and the behavior band construction module respectively, is used to extract the corresponding deviation path features based on the spatial deviation relationship between the somatosensory consistency trajectory and the individual's normal somatosensory behavior band.

[0136] A matching module, which is connected to the extraction module, is used to input the deviation path features into a preset physiological state trajectory database for similarity matching to obtain several matching results;

[0137] A determination module, which is connected to the matching module, is used to determine the physiological state type and corresponding evolutionary stage of the target object based on the matching result;

[0138] The correction module is connected to the determination module and the behavior band construction module respectively, and is used to correct the individual's normal somatosensory behavior band based on the continuous distribution characteristics of the evolutionary stage within a preset stable duration.

[0139] An output module, connected to the determining module, is used to output health intervention prompts based on the physiological state type.

[0140] By synchronously acquiring and jointly modeling facial micro-movements, body posture, speech acoustics, and physiological signals such as HRV, EDA, and respiratory rate, the system can uniformly characterize changes in an individual's overt behavior and autonomic nervous system activity. This allows for the overall perception and quantitative expression of the synergistic changes in multimodal features of states such as emotional fluctuations, fatigue accumulation, and pain responses. Through dynamic comparison of cross-modal consistency feature matrices with an individual's normal somatic behavior bands, the system can not only distinguish between short-term, occasional fluctuations and persistent changes in physiological state, but also accurately reflect the state evolution trend by utilizing the changes in direction, amplitude, and duration of deviation paths. Furthermore, by combining typical evolutionary patterns in the physiological state trajectory database, the system can achieve refined identification of different state types and their stages. Through adaptive correction of behavior bands, the system can continuously track changes in an individual's long-term physical condition and adaptability, thereby maintaining assessment sensitivity while avoiding the accumulation of misjudgments. Ultimately, the health intervention output is both real-time targeted and possesses long-term stability and individual adaptability.

[0141] The multimodal somatosensory information processing method and system for physiological state assessment provided in this embodiment has a clear, complete, and sufficient technical solution. Specifically, the specification and embodiments have provided clear technical means, implementation methods, and key parameter examples for the entire process and each step, from real-time acquisition of multimodal signals (S1), construction of cross-modal consistency feature matrix (S2), modeling of individual normal somatosensory behavior bands (S3), generation and deviation analysis of somatosensory consistency trajectories (S4-S5), to physiological state trajectory library matching and "type-stage" dual determination (S6-S7), and even individual behavior band adaptive correction (S8) and final health intervention prompt output (S9).

[0142] The technical means involved in each step, such as signal time-frequency analysis, feature standardization, statistical modeling, time series dimensionality reduction, trajectory similarity matching, and adaptive control strategies, are all well-known and commonly applied technologies familiar to those skilled in the art. For the specific operations of each step, such as time series synchronization calculation, behavior band envelope construction, trajectory compression strategies, stage division criteria, and rules for determining correction magnitudes, the specification provides directly implementable specific guidance or exemplary solutions. Reasonable value ranges and example values ​​are also provided for relevant parameters (such as trajectory compression duration and learning rate factor).

[0143] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal somatosensory information processing method for physiological state assessment, characterized in that, include: Real-time acquisition of multimodal somatosensory parameters of the target object, including facial expression micro-movement parameters, body posture parameters, speech acoustic parameters, and physiological signal parameters, including heart rate variability (HRV), skin conductance (EDA), and respiratory rate. Based on the aforementioned multimodal somatosensory parameters, a cross-modal consistency feature is constructed to characterize the collaborative change relationship between visual modality, posture modality, speech modality, and physiological modality on the same time scale; Based on the cross-modal consistency characteristics of the target object within a preset historical period, a corresponding individual normal somatosensory behavior band is constructed to characterize the range of multimodal collaborative changes of the target object under non-abnormal conditions. The current cross-modal consistency features are subjected to trajectory compression processing to generate a somatosensory consistency trajectory; Based on the spatial deviation relationship between the somatosensory consistency trajectory and the individual's normal somatosensory behavior band, the corresponding deviation path features are extracted; The deviation path features are input into a preset physiological state trajectory database for similarity matching to obtain several matching results; The physiological state type and corresponding evolutionary stage of the target object are determined based on the matching results. The individual's normal somatosensory behavior band is modified based on the continuous distribution characteristics of the evolutionary stage within a preset stable duration; Output health intervention suggestions based on the physiological state type; The cross-modal consistency feature matrix is ​​jointly constructed based on the temporal synchronization degree, amplitude co-variation rate, and correlation of change direction among the visual modality, the posture modality, the speech modality, and the physiological modality; The deviation path features are determined based on the comprehensive deviation features of the somatosensory consistency trajectory from the individual's normal somatosensory behavior band in terms of direction, amplitude, and duration. The process of modifying the individual's normal somatosensory behavior band based on the continuous distribution characteristics of the evolutionary stage within a preset stable duration includes: Obtain a continuous distribution sequence of deviation path features corresponding to the evolutionary stage within the preset stable duration; Based on the boundary offset relationship between the continuous distribution sequence and the individual's normal somatosensory behavior band, the corresponding correction direction and correction magnitude are determined; The upper and lower boundaries of the individual's normal somatosensory behavior band are adaptively updated based on the correction direction and correction magnitude to correct the individual's normal somatosensory behavior band. The process of constructing corresponding individual normal somatic behavior bands based on the cross-modal consistency characteristics of the target object within a preset historical period includes: The cross-modal consistency feature matrix of the target object within a preset historical period is obtained as a historical sample set, where the historical sample set is used to characterize the multimodal collaborative change features of the target object under multiple non-abnormal state stages; Outlier removal is performed on the cross-modal consistency feature matrix of the historical sample set to remove abnormal samples caused by occasional noise, short-term violent actions or collection anomalies, so as to ensure the stability of subsequent modeling data. The feature distribution statistics of historical cross-modal consistency features after outlier removal are performed. The mean distribution range, fluctuation range and cooperative change density distribution of each modal consistency feature in the historical period are calculated respectively to characterize the multimodal cooperative change feature boundary of the target object in non-abnormal state. Based on the statistical results of feature distribution, a multidimensional feature envelope structure is constructed to encapsulate the normal multimodal collaborative variation range of the target object, and this multidimensional feature envelope structure is defined as the individual's normal somatosensory behavior band. The individual's normal somatic behavior band is used as a reference benchmark for subsequent somatic consistency trajectory deviation judgment and deviation path extraction, in order to distinguish whether the current multimodal collaborative change is within the range of the target object's non-abnormal behavior pattern.

2. The multimodal somatosensory information processing method for physiological state assessment according to claim 1, characterized in that, The facial expression micro-movement parameters include eyebrow displacement amplitude, eyelid closure frequency, and mouth corner stretching rate; The body posture parameters include trunk forward tilt angle, limb curling range, and posture stability. The speech acoustic parameters include speech fundamental frequency, speech rate, pause ratio, and energy fluctuation rate.

3. The multimodal somatosensory information processing method for physiological state assessment according to claim 2, characterized in that, The individual normal somatic behavior band is obtained by aggregate modeling based on the cross-modal consistency features of the target object within a preset historical time window.

4. The multimodal somatosensory information processing method for physiological state assessment according to claim 3, characterized in that, The trajectory compression process includes performing temporal dimensionality reduction or key node extraction on all the cross-modal consistency features within a preset trajectory compression time to generate the somatosensory consistency trajectory that characterizes the current multimodal collaborative change trend.

5. The multimodal somatosensory information processing method for physiological state assessment according to claim 4, characterized in that, The physiological state trajectory library includes pain evolution trajectory, fatigue evolution trajectory, and stress evolution trajectory, and the pain evolution trajectory, fatigue evolution trajectory, and stress evolution trajectory are all composed of trajectory samples formed by the evolution of the cross-modal consistency features over time.

6. The multimodal somatosensory information processing method for physiological state assessment according to claim 5, characterized in that, The evolutionary stages include an initial stage, a sustained stage, and a mitigation stage, which characterize the degree of evolution of the target object's current physiological state.

7. A multimodal somatosensory information processing system for physiological state assessment, based on the multimodal somatosensory information processing method for physiological state assessment according to any one of claims 1-6, characterized in that, include: The acquisition module is used to acquire multimodal somatosensory parameters of the target object in real time. The multimodal somatosensory parameters include facial expression micro-movement parameters, body posture parameters, speech acoustic parameters, and physiological signal parameters. Among them, the physiological signal parameters include heart rate variability (HRV), skin conductance (EDA), and respiratory rate. The feature construction module is used to construct cross-modal consistency features based on the multimodal somatosensory parameters to characterize the collaborative change relationship between visual modality, posture modality, speech modality and physiological modality on the same time scale; The behavior band construction module is used to construct a corresponding individual normal somatosensory behavior band based on the cross-modal consistency characteristics of the target object within a preset historical period, which is used to characterize the range of multimodal collaborative changes of the target object in a non-abnormal state. The compression module is used to perform trajectory compression processing on the current cross-modal consistency features to generate a somatosensory consistency trajectory. The extraction module is used to extract the corresponding deviation path features based on the spatial deviation relationship between the somatosensory consistency trajectory and the individual's normal somatosensory behavior band. The matching module is used to input the deviation path features into a preset physiological state trajectory database for similarity matching to obtain several matching results; The determination module is used to determine the physiological state type and corresponding evolutionary stage of the target object based on the matching result; The correction module is used to correct the individual's normal somatosensory behavior band based on the continuous distribution characteristics of the evolutionary stage within a preset stable duration. The output module is used to output health intervention prompts based on the physiological state type.

Images