Psychological state assessment method and system based on multi-source heterogeneous perception

Abstract

The invention provides a psychological state assessment method and system based on multi-source heterogeneous perception, which are applied to the technical field of artificial intelligence, and the method comprises the following steps: obtaining a multi-source heterogeneous perception data stream for a target user; performing hybrid synchronization processing based on the multi-source heterogeneous sensing data stream to obtain a time-aligned multi-modal data sequence; performing modal special feature extraction on the multi-modal data sequence to obtain millimeter wave radar features, RGB camera features and event camera features; based on the millimeter wave radar features, the RGB camera features and the event camera features, inputting the features to a dynamic cross-modal attention fusion network based on guidance of a psychiatry knowledge base for adaptive attention fusion, and obtaining a deep fusion feature vector; and inputting the deep fusion feature vector into a psychological state inference model to obtain a predicted psychological state of the user output by the psychological state inference model. According to the invention, comprehensive, robust and accurate evaluation of the psychological state of the individual is realized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for assessing psychological states based on multi-source heterogeneous perception. Background Technology

[0002] The representation of human psychological states is complex and multidimensional, and a single source of information often cannot fully and accurately capture its subtle changes.

[0003] In existing technologies, while RGB cameras can acquire visual information such as facial expressions and postures, they are easily affected by lighting and occlusion, and cannot detect internal physiological changes. Millimeter-wave radar can monitor physiological parameters such as heart rate and respiration non-contactly, compensating for the shortcomings of visual sensors, but it lacks the ability to acquire key behavioral cues such as facial expressions. Event cameras, as a novel type of sensor, offer the possibility of capturing rapid micro-expressions, eye movements, and other subtle dynamic behaviors related to psychological states due to their high temporal resolution and high dynamic range, but their asynchronous and sparse data characteristics also pose challenges to processing and fusion.

[0004] It is evident that the psychological state assessment methods in related technologies suffer from poor multimodal data coordination. Summary of the Invention

[0005] This invention provides a psychological state assessment method and system based on multi-source heterogeneous sensing, which addresses the shortcomings of poor multimodal data collaboration in existing psychological state assessment methods. It fully leverages the collaborative advantages of millimeter-wave radar, RGB camera, and event camera to achieve a comprehensive, robust, and accurate assessment of individual psychological states.

[0006] This invention provides a method for assessing psychological state based on multi-source heterogeneous perception, comprising the following steps: Acquiring a multi-source heterogeneous perception data stream for a target user, wherein the multi-source heterogeneous perception data stream includes data collected by millimeter-wave radar, RGB cameras, and event cameras pre-deployed in the user's daily environment; performing hybrid synchronous processing on the multi-source heterogeneous perception data stream to obtain a time-aligned multimodal data sequence; extracting modality-specific features from the multimodal data sequence to obtain millimeter-wave radar features, RGB camera features, and event camera features; inputting the millimeter-wave radar features, RGB camera features, and event camera features into a dynamic cross-modal attention fusion network guided by a psychiatric knowledge base for adaptive attention fusion to obtain a deep fusion feature vector output by the dynamic cross-modal attention fusion network; inputting the deep fusion feature vector into a psychological state inference model to obtain the predicted psychological state of the user output by the psychological state inference model.

[0007] According to the present invention, a psychological state assessment method based on multi-source heterogeneous sensing is provided. The method involves performing hybrid synchronization processing on the multi-source heterogeneous sensing data stream to obtain a time-aligned multimodal data sequence, including: calling a central synchronization controller to generate periodic global synchronization trigger pulses; simultaneously distributing the global synchronization trigger pulses to the external trigger input of the millimeter-wave radar, the external frame synchronization input of the RGB camera, and the external timestamp synchronization input of the event camera; and performing time window alignment on the millimeter-wave radar data, RGB camera data, and event camera data based on the timestamps generated by the global synchronization trigger pulses to obtain a time-aligned multimodal data sequence.

[0008] According to the present invention, a method for assessing psychological state based on multi-source heterogeneous sensing is provided. The multimodal data sequence includes: millimeter-wave radar data sequence, RGB camera data sequence, and event camera data sequence. The modality-specific feature extraction of the multimodal data sequence to obtain millimeter-wave radar features, RGB camera features, and event camera features includes: extracting physiological parameters and micro-motion features from the millimeter-wave radar data sequence to obtain millimeter-wave radar features. Specifically, an electrocardiogram (ECG) waveform is extracted from the millimeter-wave radar data sequence using a deep learning-based feature extraction network, and heart rate variability parameters, respiratory pattern features, and micro-motion pattern features related to psychological state are determined based on the ECG waveform. The system extracts visual behavioral features from the RGB camera data sequence to obtain RGB camera features. Specifically, a facial action unit recognition algorithm is used to extract facial action unit intensity sequences from the RGB camera data sequence, and a three-dimensional human pose estimation algorithm is used to extract body pose features and head movement features from the RGB camera data sequence. The system also extracts micro-dynamic behavioral features from the event camera data sequence to obtain event camera features. Specifically, an adaptive spatiotemporal density coding algorithm is used to convert the asynchronous event stream in the event camera data sequence into a synchronized image sequence, and a preset lightweight spatiotemporal graph convolutional network is used to extract micro-expression features and eye-tracking saccade pattern features from the synchronized image sequence.

[0009] According to the present invention, a psychological state assessment method based on multi-source heterogeneous perception is provided. The dynamic cross-modal attention fusion network includes: a modality-specific embedding and preliminary encoding layer, a modality reliability assessment and gating weighting module, a multi-layer dynamic cross-modal attention fusion layer, and a fusion representation output layer. The method involves inputting the millimeter-wave radar features, the RGB camera features, and the event camera features into a dynamic cross-modal attention fusion network guided by a psychiatric knowledge base for adaptive attention fusion, resulting in a deep fusion feature vector output by the dynamic cross-modal attention fusion network. This includes: encoding the millimeter-wave radar features, the RGB camera features, and the event camera features respectively through the modality-specific embedding and preliminary encoding layer, mapping each modality feature to a unified... A high-dimensional feature space is used, and modality type embedding and position encoding are added to obtain a preliminary encoded feature sequence. The modality reliability assessment and gating weighting module queries the prior importance weights of each modality feature based on a preset psychiatric knowledge base, and combines this with real-time signal quality assessment results to determine the reliability weights of each modality feature. The preliminary encoded feature sequence is weighted based on the reliability weights of each modality feature to obtain a weighted feature sequence. The weighted feature sequence is then processed through a multi-layer dynamic cross-modal attention fusion layer, performing parallel cross-modal attention calculations and intra-modal self-attention calculations to obtain an updated feature sequence. Finally, the updated feature sequence is concatenated and fused through a fusion representation output layer to generate a fixed-length deep fusion feature vector.

[0010] According to the present invention, a method for assessing mental state based on multi-source heterogeneous perception is provided. The multi-layer dynamic cross-modal attention fusion layer is composed of multiple identical dynamic cross-modal attention layers stacked together. The method of performing parallel cross-modal attention calculation and intra-modal self-attention calculation on the weighted feature sequence to obtain an updated feature sequence includes: performing the following operations on the weighted feature sequence through each dynamic cross-modal attention layer to obtain an updated feature sequence: performing parallel cross-modal attention calculation based on the weighted feature sequence to obtain a cross-modal attention output, wherein the feature sequence of each modality is used as a query vector, and the feature sequences of other modalities are used as key vectors and value vectors, and the cross-modal co-regulation factor defined by the psychiatric knowledge base is incorporated into the attention weight calculation; performing parallel intra-modal self-attention calculation based on the weighted feature sequence to obtain an intra-modal self-attention output; integrating information through splicing operation and linear transformation based on the cross-modal attention output and the intra-modal self-attention output to obtain an integrated feature sequence; and performing point-by-point feedforward network processing on the integrated feature sequence, combined with residual connection and layer normalization operation, to output an updated feature sequence.

[0011] According to the present invention, a psychological state assessment method based on multi-source heterogeneous perception, before inputting the deep fusion feature vector into the psychological state inference model to obtain the predicted psychological state of the user output by the psychological state inference model, the method further includes: acquiring a multimodal psychological state clinical validation dataset; and fine-tuning a pre-trained large-scale multimodal basic model based on the multimodal psychological state clinical validation dataset using a hierarchical learning rate and a stepwise unfreezing strategy to obtain a psychological state inference model.

[0012] This invention also provides a psychological state assessment system based on multi-source heterogeneous perception, comprising the following modules: an acquisition module for acquiring multi-source heterogeneous perception data streams for a target user, wherein the multi-source heterogeneous perception data streams include data collected by millimeter-wave radar, RGB camera, and event camera pre-deployed in the user's daily environment; an alignment module for performing hybrid synchronization processing based on the multi-source heterogeneous perception data streams to obtain time-aligned multimodal data sequences; an extraction module for performing modality-specific feature extraction on the multimodal data sequences to obtain millimeter-wave radar features, RGB camera features, and event camera features; a fusion module for inputting the millimeter-wave radar features, the RGB camera features, and the event camera features into a dynamic cross-modal attention fusion network guided by a psychiatric knowledge base for adaptive attention fusion to obtain a deep fusion feature vector output by the dynamic cross-modal attention fusion network; and a prediction module for inputting the deep fusion feature vector into a psychological state inference model to obtain the predicted psychological state of the user output by the psychological state inference model.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the psychological state assessment method based on multi-source heterogeneous perception as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the psychological state assessment method based on multi-source heterogeneous perception as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the psychological state assessment method based on multi-source heterogeneous perception as described above.

[0016] This invention provides a method and system for assessing mental states based on multi-source heterogeneous perception. By acquiring multi-source heterogeneous perception data streams collected by millimeter-wave radar, RGB cameras, and event cameras, it can cover users' daily behaviors and environmental information. Next, the multi-source heterogeneous perception data streams undergo hybrid synchronous processing to obtain time-aligned multimodal data sequences, effectively eliminating temporal deviations and ensuring the accuracy and consistency of subsequent analysis. Then, modality-specific features are extracted from the aligned multimodal data sequences to obtain millimeter-wave radar features, RGB camera features, and event camera features, thereby fully capturing the specific information of each modality. Based on these modal features, they are input into a dynamic cross-modal attention fusion network guided by a psychiatric knowledge base for adaptive attention fusion. This dynamically adjusts the contribution weights of different modalities, achieving effective complementarity between modal features and outputting a more discriminative deep fusion feature vector. Finally, this deep fusion feature vector is input into a mental state inference model, accurately predicting mental states and improving the accuracy and reliability of mental state assessment. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the psychological state assessment method based on multi-source heterogeneous perception provided by the present invention.

[0019] Figure 2 This is a schematic diagram of the overall architecture of the multimodal sensing fusion and psychological state assessment system provided by the present invention.

[0020] Figure 3 This is a schematic diagram of the structure of the dynamic cross-modal attention fusion network guided by the psychiatric knowledge base provided by the present invention.

[0021] Figure 4 This is a schematic diagram of the modules of the psychological state assessment system based on multi-source heterogeneous perception provided by the present invention.

[0022] Figure 5 This is a schematic diagram of the physical structure of an electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] Although multimodal fusion has been recognized as an effective way to improve the performance of mental state assessment, existing fusion technologies still face many bottlenecks: Insufficient optimization of sensor combinations and data synchronization: Existing technologies may fail to maximize information complementarity when selecting sensor combinations. While specific combinations of millimeter-wave radar (internal physiology), RGB cameras (macroscopic visual behavior), and event cameras (microscopic high-speed dynamic behavior) hold great potential, the highly heterogeneous nature of their data (radar phase / point clouds, RGB pixel frames, and asynchronous event streams from event cameras) and their high-precision synchronization (especially sub-millisecond alignment between asynchronous event cameras and synchronous sensors) are pressing challenges. Simple software timestamp alignment is insufficient to meet the requirements of subsequent fine-grained feature fusion.

[0025] The robustness and intelligence of fusion algorithms are lacking: traditional fusion strategies such as feature-level concatenation or simple weighted averaging are difficult to effectively handle real-world problems such as dynamic changes in the quality of signals from different modes, information redundancy or missing information, and cannot deeply explore the complex nonlinear dependencies between modes. When a certain mode is severely disturbed, the overall system performance may drop sharply.

[0026] Lack of in-depth mining and integration of psychological state-specific information: Subtle differences in psychological states may be reflected in specific features of a particular modality or in specific combinations of multimodal features. Existing fusion models often lack a mechanism to dynamically and purposefully focus on and integrate this information that is most critical to the current assessment objective (such as a specific type of anxiety or stress response).

[0027] Limited integration of psychiatric expertise: Although some studies used psychological scales for annotation, the expertise and clinical experience of psychiatrists were not systematically and deeply integrated and utilized in terms of the clinical relevance of feature selection, the sensitivity of the model to specific psychopathological patterns, and the clinical interpretability of the assessment results.

[0028] Therefore, there is an urgent need in this field for a technology that can fully leverage the synergistic advantages of millimeter-wave radar, RGB cameras, and event cameras, through innovative data synchronization and fusion mechanisms, and by deeply integrating psychiatric expertise, to achieve a comprehensive, robust, and accurate assessment of an individual's psychological state.

[0029] Figure 1 This is a flowchart illustrating the psychological state assessment method based on multi-source heterogeneous perception provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps.

[0030] Step 101: Obtain multi-source heterogeneous sensing data streams for the target user.

[0031] Among them, the multi-source heterogeneous sensing data stream includes data collected by millimeter-wave radar, RGB cameras and event cameras that are pre-deployed in the user's daily environment; A collaborative multimodal sensing front-end is deployed in the user's daily environment. This front-end includes millimeter-wave radar, RGB cameras, and event cameras. Millimeter-wave radar is used for non-contact monitoring of the user's physiological parameters, such as heart rate and breathing patterns; RGB cameras are used to capture the user's macroscopic visual behavior, such as facial expressions and body posture; and event cameras, with their high temporal resolution, are specifically used to record high-speed micro-dynamic behaviors, such as micro-expressions and eye movement changes.

[0032] The aforementioned sensors are activated simultaneously to collect continuous multi-source heterogeneous sensing data streams. These multi-source heterogeneous sensing data streams are output in their original format, including point cloud or range image sequences from millimeter-wave radar, image frame sequences from RGB cameras, and asynchronous event streams from event cameras.

[0033] Step 102: Perform hybrid synchronization processing based on multi-source heterogeneous sensing data streams to obtain time-aligned multimodal data sequences.

[0034] In this embodiment of the invention, a millimeter-wave radar with optimized parameters, a high-resolution RGB camera, and a high dynamic range event camera are integrated. A hybrid synchronization mechanism combining hardware-based global trigger signals and high-precision PTP (Precision Time Protocol) network time synchronization is adopted to achieve sub-millisecond-level precise synchronization of multi-source heterogeneous data, thereby overcoming the challenges of data synchronization in existing technologies.

[0035] A central synchronization controller generates periodic global synchronization trigger pulses, which are simultaneously distributed to the millimeter-wave radar, RGB camera, and event camera via dedicated hardware lines, ensuring sub-millisecond simultaneity in data acquisition. Each sensor records its internal high-precision counter value upon receiving the trigger pulse. Simultaneously, all output data packets are assigned a uniformly formatted high-precision timestamp.

[0036] At the software level, feature-level alignment of multimodal data is performed based on timestamps, integrating radar data, RGB image frames, and event stream events within the same time window to form a time-synchronized multimodal data sequence. This process eliminates temporal discrepancies and provides a consistent data foundation for subsequent feature extraction.

[0037] Step 103: Extract modality-specific features from the multimodal data sequence to obtain millimeter-wave radar features, RGB camera features, and event camera features.

[0038] In this embodiment of the invention, for millimeter-wave radar data, a deep learning-based method is used to extract high-fidelity physiological features, such as heart rate variability parameters and micromotor pattern sequences. These features are quantified by a pre-trained neural network model and associated with a psychiatric knowledge base.

[0039] For RGB camera data, face detection and pose estimation algorithms are used to extract facial motion unit sequences and 3D body joint coordinates, and then head motion angles and limb motion statistical features are calculated to capture macroscopic behavioral dynamics.

[0040] For event camera data, the asynchronous event stream is first converted into a synchronized event density map sequence using an adaptive spatiotemporal density encoding algorithm. Then, a lightweight spatiotemporal graph convolutional network is used to extract micro-dynamic features, such as micro-expression intensity and eye saccade parameters, from these sequences.

[0041] During implementation, feature extraction for each mode (millimeter-wave radar mode, RGB camera mode, and event camera mode) is performed in parallel to ensure that each feature vector has a consistent time dimension, providing complementary information for the fusion stage.

[0042] Step 104: Based on millimeter-wave radar features, RGB camera features, and event camera features, input them into a dynamic cross-modal attention fusion network guided by a psychiatric knowledge base for adaptive attention fusion, and obtain the deep fusion feature vector output by the dynamic cross-modal attention fusion network.

[0043] In this embodiment of the invention, the psychiatric knowledge base is constructed as a structured database, storing prior patterns of multimodal features under different psychological states. In the fusion network, each modal feature (i.e., millimeter-wave radar features, RGB camera features, and event camera features) is first mapped to a unified high-dimensional space through a modality-specific embedding layer. Then, it enters a modality reliability assessment and gating weighting module guided by the psychiatric knowledge base. This module evaluates the quality of each modality signal in real time and dynamically calculates modality weights based on the prior importance of the psychiatric knowledge base query, thus weighting the features. The weighted features are input into a multi-layered dynamic cross-modal attention layer. This layer incorporates a cross-modal collaborative adjustment factor defined by the psychiatric knowledge base when calculating attention weights, enabling the network to prioritize information-rich modal combinations. Finally, through stacked attention layers and an integration mechanism, a fixed-length deep fusion feature vector is output, which encapsulates key collaborative information of the multimodalities.

[0044] Step 105: Input the deep fusion feature vector into the mental state inference model to obtain the user's predicted mental state output by the mental state inference model.

[0045] In this embodiment of the invention, the deep fusion feature vector is input into the mental state inference model, which is fine-tuned based on a pre-trained large multimodal base model.

[0046] During implementation, the fine-tuning process utilizes a multimodal clinical validation dataset of mental states annotated by psychiatric experts. A hierarchical learning rate and stepwise unfreezing strategy are employed to adapt the model to specific mental state assessment tasks, such as anxiety or depression level classification. The mental state inference model outputs predicted mental states, such as scores or categories, and the results are validated and interpreted using a psychiatric knowledge base, for example, by providing feature importance analysis based on attention weights. The entire implementation process ensures that the assessment results are clinically relevant and provides an intuitive output interface for users or physicians.

[0047] refer to Figure 2 , Figure 2 This is a schematic diagram of the overall architecture of the multimodal sensing fusion and psychological state assessment system provided by the present invention. First, multi-source heterogeneous sensing data streams from millimeter-wave radar, RGB cameras, and event cameras are acquired in parallel. These data undergo time alignment and formatting through a high-precision hybrid synchronization and multimodal data preprocessing stage to form a unified multimodal data sequence. Subsequently, the data from the three modalities are respectively fed into dedicated feature extraction modules for feature extraction (radar feature extraction, RGB visual feature extraction, and event stream micro-dynamic feature extraction), generating their respective modal features. These features, along with a psychiatric knowledge base, are input into a dynamic cross-modal attention fusion network guided by the core psychiatric knowledge base for adaptive feature fusion, generating a deep fusion feature vector. Finally, this vector is fed into a psychological state inference model, ultimately outputting a predicted psychological state for the target user.

[0048] Through this invention, multi-source heterogeneous sensing data streams acquired from millimeter-wave radar, RGB cameras, and event cameras can cover users' daily behaviors and environmental information. Next, hybrid synchronous processing is performed on the multi-source heterogeneous sensing data streams to obtain time-aligned multimodal data sequences, effectively eliminating temporal deviations and ensuring the accuracy and consistency of subsequent analysis. Then, modality-specific feature extraction is performed on the aligned multimodal data sequences to obtain millimeter-wave radar features, RGB camera features, and event camera features, thereby fully capturing the specific information of each modality. Based on each modality feature, it is input into a dynamic cross-modal attention fusion network guided by a psychiatric knowledge base for adaptive attention fusion. This dynamically adjusts the contribution weights of different modalities, achieving effective complementarity between modal features and outputting a more discriminative deep fusion feature vector. Finally, this deep fusion feature vector is input into a mental state inference model, which can accurately output predicted mental states, improving the accuracy and reliability of mental state assessment.

[0049] According to the present invention, a psychological state assessment method based on multi-source heterogeneous perception is provided, which performs hybrid synchronization processing on multi-source heterogeneous perception data streams to obtain time-aligned multimodal data sequences, including: The central synchronization controller is invoked to generate periodic global synchronization trigger pulses; The global synchronization trigger pulse is simultaneously distributed to the external trigger input of the millimeter-wave radar, the external frame synchronization input of the RGB camera, and the external timestamp synchronization input of the event camera; Based on the timestamps generated by the global synchronization trigger pulses, time windows are aligned for millimeter-wave radar data, RGB camera data, and event camera data to obtain time-aligned multimodal data sequences.

[0050] In this embodiment of the invention, a millimeter-wave radar (79GHz FMCW, 4GHz bandwidth, 4Tx / 8Rx MIMO, 500Hz frame rate) is used to provide core internal physiological data (HRV, respiration, micro-movements). An RGB camera (global shutter, ≥1920×1080@60fps, equipped with a high-quality fixed-focus lens) is used to capture facial expressions (AU), body posture, head movements, etc. An event camera (such as Prophesee EVK4-HD or inivation DAVIS346, temporal resolution equivalent >1kHz, dynamic range >120dB) is used to capture rapid micro-expressions, eye saccades, rapid pupil dilation and contraction, and other micro-dynamics.

[0051] To address the difficulty of multimodal data synchronization in existing technologies, this invention constructs a central synchronization controller (CSC, which can be implemented based on FPGA) based on a high-precision timing GPS module (providing PPS second pulses and high-precision timestamps) or an atomic clock. The CSC (Clock Source Controller) serves as the system's master clock.

[0052] In this embodiment of the invention, a central synchronization controller is invoked to periodically generate a global synchronization trigger pulse (e.g., synchronized with the target acquisition frame rate of the RGB camera, such as 60fps every 16.67ms). This pulse is simultaneously distributed via dedicated hardware lines (such as LVDS (Low-Voltage Differential Signaling) or optical fiber) to the external trigger input of the millimeter-wave radar, the external frame synchronization input of the RGB camera, and the external timestamp synchronization input of the event camera (or as a reference base for the event timestamp).

[0053] Each sensor, upon receiving a trigger pulse, records its internal high-precision counter value with extremely low latency (nanosecond level) and strongly correlates it with the data units acquired at that moment (radar chirp sequence, RGB image frame, and the start event of the event packet).

[0054] All data packets output by the sensors (radar point cloud / range image, RGB image metadata, event camera event stream data packets) are required to include a high-precision timestamp in a globally uniform format (e.g., 64-bit, nanosecond resolution, timed from a common start time) assigned by the CSC or calibrated based on the CSC synchronization signal.

[0055] Following feature extraction, feature vectors extracted from different modalities within the same time window are precisely aligned based on these high-precision global timestamps to form a multimodal data sequence. For asynchronous features from event cameras, they are aligned with the time window of the synchronous sensor based on their start and end timestamps.

[0056] Through the embodiments of this invention, the aforementioned hybrid synchronization mechanism ensures the "simultaneity" of the starting point for the acquisition actions of each sensor through global hardware triggering, avoiding the significant jitter and latency uncertainties that may exist in pure software or network time protocols (such as NTP). Combined with a unified high-precision timestamp format, it provides sub-millisecond-level time alignment accuracy for subsequent feature-level fusion, which is crucial for analyzing rapidly changing psychophysiological phenomena (such as the correlation between micro-expressions and instantaneous changes in heart rate) and is a key prerequisite for achieving high-quality multimodal fusion.

[0057] According to the present invention, a psychological state assessment method based on multi-source heterogeneous perception is provided, wherein the multimodal data sequence includes: millimeter-wave radar data sequence, RGB camera data sequence and event camera data sequence; Modality-specific feature extraction was performed on the multimodal data sequence to obtain millimeter-wave radar features, RGB camera features, and event camera features, including: Physiological parameters and micro-motion features were extracted from millimeter-wave radar data sequences to obtain millimeter-wave radar features. Among them, a deep learning-based feature extraction network was used to extract electrocardiogram waveforms from millimeter-wave radar data sequences, and heart rate variability parameters, respiratory pattern features, and micro-motion pattern features related to psychological state were determined based on the electrocardiogram waveforms. Visual behavioral features are extracted from RGB camera data sequences to obtain RGB camera features. Specifically, a facial action unit recognition algorithm is used to extract facial action unit intensity sequences based on RGB camera data sequences, and a three-dimensional human pose estimation algorithm is used to extract body pose features and head motion features based on RGB camera data sequences. Micro-dynamic behavior features are extracted from event camera data sequences to obtain event camera features. Specifically, an adaptive spatiotemporal density coding algorithm is used to convert the asynchronous event stream in the event camera data sequence into a synchronized image sequence, and micro-expression features and eye-tracking saccade pattern features are extracted from the synchronized image sequence based on a preset lightweight spatiotemporal graph convolutional network.

[0058] In this embodiment of the invention, the millimeter-wave radar features include high-fidelity ECG (Electrocardiogram) waveforms, a complete set of time-domain, frequency-domain, and nonlinear HRV (Heart Rate Variability) parameter sequences, key ECG morphological features (such as P / T wave amplitude and width, QTc interval, etc., all defined with the assistance of psychiatrists regarding their correlation with psychological states), and quantified micromotor pattern features related to specific psychological states (such as the frequency and energy of anxiety tremors, the density of restlessness patterns, etc.). These features together constitute a deep representation of an individual's internal physiological state and subtle bodily behaviors.

[0059] The following is an implementation method that uses a facial action unit recognition algorithm to extract facial action unit intensity sequences based on RGB camera data sequences, and a three-dimensional human pose estimation algorithm to extract body pose features and head motion features based on RGB camera data sequences.

[0060] Advanced face detection algorithms and high-precision facial landmark localization algorithms (such as HRNet-based schemes) are used to track faces in real time. Subsequently, a lightweight deep convolutional neural network optimized for facial expression AU (ActionUnit) sequence regression (e.g., based on MobileNetV3 or EuclideanNet-Lite backbones, pre-trained and fine-tuned on large-scale AU-annotated datasets such as DISFA and BP4D+) is employed to output a continuous intensity value sequence of more than 20 core facial AUs (such as AU1, AU2, AU4, AU6, AU12, AU15, AU17, AU23, AU25, etc.) in each video frame. These AU sequences constitute a fine-grained quantization of macroscopic facial expression dynamics.

[0061] A real-time multi-person 3D human pose estimation algorithm (such as a scheme based on the SMPL-X model combined with a lightweight Transformer encoder, like a simplified version of MotionBERT) is employed to output the 3D spatial coordinate sequence of major human joints (such as head, neck, shoulder, elbow, wrist, hip, knee, and ankle). Based on these coordinate sequences, the following features are calculated: 3D head rotation angles (Eulerian angles: Roll, Pitch, Yaw) and their angular velocities and angular acceleration sequences; body pose openness / closedness indices (e.g., calculated based on the relative position and angles of the torso and limbs); statistical characteristics of macroscopic limb movements such as energy, frequency, and amplitude; and identification of specific behavioral patterns (such as irritable scratching, crossed arms, etc., which can be based on rules or simple classifiers).

[0062] For event camera data sequences, this invention proposes an Adaptive Spatio-Temporal Density Encoding (ASTDE) event stream representation transformation algorithm, which transforms the original event stream into a synchronized image sequence (or event density map sequence) with more uniform information density and more suitable for subsequent neural network processing.

[0063] In this embodiment of the invention, a fixed-length time window is no longer used; instead, the integration window is dynamically adjusted based on the local density of the event stream. When the event stream is dense, the window is shortened; when it is sparse, the window is appropriately lengthened to ensure that each generated event density map contains approximately a similar number of events or a similar amount of information. (Integral window) The window length can be adjusted by a small feedback control mechanism based on the number of events in the previous few windows.

[0064] In each points window (dynamic window) Within the framework, positive and negative events are accumulated onto two separate 2D histograms (consistent with the spatial resolution of the event camera). During accumulation, each event can be time-decayed weighted according to the relative newness of its timestamp within the window (e.g., newer events have higher weights).

[0065] The weighted cumulative histograms of the two polarities are normalized and optionally smoothed (e.g., Gaussian filtering) to form a pair (or merged into a multi-channel) event density map (sequence).

[0066] This invention also constructs a lightweight spatio-temporal graph convolutional network (LST-GCN) for micro-dynamic feature extraction. This network is used to extract high-speed micro-dynamic features related to psychological states, such as micro-expressions, eye saccades, and pupil dynamics, from event density map sequences generated by adaptive spatio-temporal density encoding.

[0067] The lightweight spatiotemporal graph convolutional network includes: a graph construction module, which treats pixels (or superpixels) of highly active regions (e.g., obtained through thresholding or saliency detection) in each event density map sequence as graph nodes. Connections between nodes can be based on spatial proximity (e.g., KNN or fixed-radius connections) and temporal continuity (connections between corresponding regions in adjacent frames). Node features can include the event density, average polarity ratio, motion direction estimation (from local event optical flow), etc. The spatiotemporal graph convolutional layers employ a multi-layer graph convolutional network (GCN) to extract the spatial structure features of the event density map for each frame, and intersperse temporal convolutional layers (1DConv) or gated recurrent units (GRUs) between GCN layers to capture the dynamic evolution patterns of these spatial features over time. An attention mechanism, introduced after the graph convolutional layers or temporal processing layers, can be used to learn the importance of different regions or time points for specific micro-dynamic patterns. The final output is a high-level semantic feature vector, which is trained to represent specific micro-dynamic patterns, such as: micro-expression recognition (e.g., rapid, subtle upward / downward slant of the corners of the mouth, rapid furrowing / relaxation of the brows); saccade parameters (e.g., frequency, amplitude, velocity, and direction distribution); and rapid pupil diameter change patterns (if the event camera can indirectly reflect changes in pupil brightness).

[0068] In practical applications, psychiatrists are involved in identifying which microexpressions, eye movement patterns, or pupil dynamics are associated with specific psychological states (such as panic, disgust, hypervigilance, and cognitive effort). These will serve as training targets or output labels for LST-GCN. For example, psychiatrists have noted that in certain anxiety states, individuals may exhibit increased saccade frequency and an overreaction of the pupils to minor stimuli.

[0069] Through the embodiments of this invention, LST-GCN can learn directly on the irregular spatiotemporal structure of event data. Compared to converting event data into traditional images and then processing it with CNN, it can better preserve the fine spatiotemporal dynamic information of events. Combined with ASTDE, it can efficiently and robustly extract micro-behavioral cues that are difficult to capture with traditional vision but are closely related to psychological states.

[0070] In some embodiments, physiological parameters and micromotor features are extracted from millimeter-wave radar data sequences. This process employs a deep learning-based feature extraction network, specifically comprising two core sub-networks: Adv-ECG RecNet and MicroMotionNet. Adv-ECG RecNet is used to reconstruct high-fidelity electrocardiogram (ECG) waveforms from the raw millimeter-wave radar data sequences. Based on the ECG waveforms, comprehensive heart rate variability parameters are further calculated, covering time-domain, frequency-domain, and nonlinear indices. Simultaneously, this network also extracts features related to respiratory patterns, such as respiratory rate and rhythm. MicroMotionNet is specifically used to quantify micromotor pattern features related to psychological states, such as the specific frequency energy distribution of anxiety tremors or the density of restlessness. The definition and selection of all these features are guided by a psychiatric knowledge base to ensure their clinical relevance. The final output is a millimeter-wave radar feature sequence containing high-precision physiological indicators and quantified behavioral patterns.

[0071] Visual behavioral features are extracted from RGB camera data sequences. This process first employs high-precision face detection and facial keypoint localization algorithms to stably track each frame. Then, a lightweight deep convolutional neural network, pre-trained and fine-tuned on a large-scale facial motion unit dataset, is used to regress and calculate the continuous intensity values ​​of over 20 core facial motion units in each video frame, forming a refined facial motion unit intensity sequence. For body behavior analysis, a model-based 3D human pose estimation algorithm is used to output the 3D spatial coordinate sequence of the main human joints. Based on this 3D spatial coordinate sequence, the rotation angle of the head in 3D space and its dynamic changes (angular velocity, angular acceleration) are calculated, and the open / closedness index of body pose, macroscopic energy statistical features of limb movements, and specific behavioral patterns are extracted. The final output is an RGB camera feature sequence that integrates facial expression dynamics and body movement information.

[0072] This study extracts micro-dynamic behavioral features from event camera data sequences. It aims to handle the unique asynchronous and sparse event stream data of event cameras. First, the original event stream is transformed using an adaptive spatiotemporal density encoding algorithm. This algorithm dynamically adjusts the integral time window based on the local density of the event stream, accumulating asynchronous events and converting them into a synchronized image sequence (i.e., an event density map sequence) with more uniform information density. Then, the generated synchronized image sequence is input into a pre-defined lightweight spatiotemporal graph convolutional network. This network constructs a spatiotemporal graph structure (treating highly active regions as nodes and establishing connections based on spatial proximity and temporal continuity), and uses a combination of graph convolution and temporal convolution to learn the spatiotemporal evolution patterns of high-speed and subtle dynamics, such as micro-expressions (e.g., rapid frowning between the eyebrows, subtle twitching of the mouth) and eye-tracking saccade patterns (e.g., scanning rate, amplitude, and speed). The final output is an event camera feature sequence representing these micro-behavioral events.

[0073] Through the embodiments of the present invention, compared with fixed window integration, the present invention can better adapt to dynamic changes in the scene, avoid motion blur during high-speed movement and generate too many empty frames during low-speed movement, thereby improving the utilization rate of event information and the quality of representation.

[0074] According to the present invention, a psychological state assessment method based on multi-source heterogeneous perception is provided. The dynamic cross-modal attention fusion network includes: a modality-specific embedding and preliminary encoding layer, a modality reliability assessment and gating weighting module, a multi-layer dynamic cross-modal attention fusion layer, and a fusion representation output layer. Based on millimeter-wave radar features, RGB camera features, and event camera features, the data is input into a dynamic cross-modal attention fusion network guided by a psychiatric knowledge base for adaptive attention fusion. This yields a deep fusion feature vector output by the dynamic cross-modal attention fusion network, including: By using modality-specific embedding and a preliminary coding layer, millimeter-wave radar features, RGB camera features, and event camera features are encoded separately. Each modality feature is mapped to a unified high-dimensional feature space and modality type embedding and position coding are added to obtain a preliminary encoded feature sequence. The modal reliability assessment and gating weighting module queries the prior importance weights of each modal feature based on a pre-set psychiatric knowledge base, and determines the reliability weights of each modal feature by combining the real-time signal quality assessment results. The pre-encoded feature sequence is then weighted based on the reliability weights of each modal feature to obtain the weighted feature sequence. By using a multi-layered dynamic cross-modal attention fusion layer, parallel cross-modal attention calculation and intra-modal self-attention calculation are performed on the weighted feature sequence to obtain the updated feature sequence; By fusing the output layer of the fusion representation, the updated feature sequences are spliced ​​and fused to generate a fixed-length deep fusion feature vector.

[0075] In some embodiments, the psychiatric knowledge base is a database that combines structured and unstructured knowledge, with content derived from authoritative textbooks and the latest research literature in fields such as DSM-5 / ICD-11, psychopharmacology, behavioral neuroscience, and affective neuroscience, as well as summaries of expert clinical experience.

[0076] For example, structured knowledge, represented in the form of ontology, semantic networks, or production rules (IF-THEN rules), defines the typical manifestation patterns and their probability distributions or parameter ranges of different psychological states (such as generalized anxiety disorder (GAD), social anxiety disorder (SAD), major depressive episode (MDD), and acute stress response (ASR)) and their related physiological indicators (such as specific HRV patterns, such as decreased SDNN and abnormal LF / HF ratio, increased and irregular respiratory rate, flattened T waves or ST segment depression in ECG), macroscopic behaviors (such as social avoidance, restlessness, frequent occurrence of facial AU4+AU7 combinations), and microscopic behaviors (such as increased saccade frequency, shortened pupillary response latency to threatening stimuli, and rapid flashes of specific microexpressions such as fear or disgust). It also defines the expected diagnostic value or relative importance weight of different modalities of information (physiological vs. behavioral, macroscopic vs. microscopic) in a specific psychological state or situation. For example, "In assessing an acute panic attack, dramatic changes in HRV and rapid breathing are more important than static body posture." Define common comorbidity patterns or key points for differential diagnosis among different psychological states.

[0077] Unstructured knowledge includes relevant clinical case descriptions, examples of typical physiological / behavioral signal fragments, and expert interpretations of complex or borderline cases.

[0078] The dynamic cross-modal attention fusion network uses a multi-layer Transformer encoder as its basic framework, but its internal attention and feature integration mechanisms are specially designed.

[0079] The modality-specific embedding and preliminary encoding layer receives feature sequences from millimeter-wave radar, RGB camera, and event camera, where each modality's feature sequence has a length T and a specific dimension Dmodel. This layer processes these three sequences separately through independent modality-specific embedding networks. These embedding networks can employ structures such as 1D convolutional layers combined with residual connections and layer normalization, or a small Transformer encoder. This step aims to achieve three objectives: first, to map the modality feature sequences with different dimensions and characteristics to a unified high-dimensional feature space, for example, setting the unified dimension Dmodel to 512; second, to further extract temporal context information within each modality during the mapping process to enhance the expressive power of the features; and finally, to add learnable modality type embeddings and absolute or relative position encodings to each sequence to explicitly distinguish modality categories and temporal position information. After processing, this layer outputs three preliminary encoded feature sequences: the millimeter-wave radar feature sequence Smmw (dimension RT × Dmodel), the RGB camera feature sequence Srgb, and the event camera feature sequence Sevt.

[0080] The modal reliability assessment and gating weighting module first performs real-time signal quality assessment on the current input data segment of each modality through a lightweight quality assessment sub-network (such as a small CNN or a statistical feature-based classifier), outputting a signal quality score qm∈[0,1], such as the signal quality index based on millimeter-wave radar, the sharpness score of RGB images, or the event density of an event stream; simultaneously, based on the target psychological state inferred by the current system (such as that set by the user or preliminarily judged by the results of the previous round), it queries the prior importance weight p of each modality from the psychiatric knowledge base. m∈[0,1]; then, the modal gating weight gm is dynamically calculated as a function of the signal quality score and the prior importance weight, using the formula gm=softmax(α·log(qm+ε)+β·log(pm+ε)+bm), where α and β are learnable scaling factors, ε is a small positive constant to prevent log(0), bm is a learnable bias, and Softmax ensures that all weights sum to 1 or that each weight is independent; finally, the calculated gm is applied to the corresponding modal feature sequence Sm to obtain the weighted sequence. This allows the fusion process to intelligently trust modes with good signal quality and high diagnostic value, while suppressing modes that are interfered with or have low information content, thus significantly improving robustness and relevance.

[0081] According to the present invention, a psychological state assessment method based on multi-source heterogeneous perception is provided, wherein a multi-layer dynamic cross-modal attention fusion layer is composed of multiple identical dynamic cross-modal attention layers stacked together; Parallel cross-modal attention computation and intra-modal self-attention computation are performed on the weighted feature sequence to obtain the updated feature sequence, including: The updated feature sequence is obtained by performing the following operations on the weighted feature sequence at each dynamic cross-modal attention layer: Based on the weighted feature sequences, parallel cross-modal attention calculation is performed to obtain cross-modal attention output. In this process, the feature sequences of each modality are used as query vectors, and the feature sequences of other modalities are used as key vectors and value vectors. The cross-modal collaborative adjustment factor defined by the psychiatric knowledge base is incorporated into the attention weight calculation. Based on the weighted feature sequence, parallel intra-modal self-attention calculation is performed to obtain the intra-modal self-attention output; Based on cross-modal attention output and intra-modal self-attention output, information is integrated through concatenation operation and linear transformation to obtain the integrated feature sequence; The integrated feature sequence is processed by a point-forward network, and combined with residual connections and layer normalization operations, the updated feature sequence is output.

[0082] In this embodiment of the invention, the multi-layer dynamic cross-modal attention fusion layer is composed of multiple identical dynamic cross-modal attention layers stacked together. Each dynamic cross-modal attention layer performs the following operations on the input weighted feature sequence to generate an updated feature sequence.

[0083] Parallel cross-modal attention computation is performed based on weighted feature sequences (including weighted feature sequences from multiple modalities): for each feature sequence of modality i, it is used as the query vector. The feature sequences of each other mode j (j≠i) are used as key vectors. Sum value vector When calculating attention weights, cross-modal co-regulatory factors obtained from a psychiatric knowledge base are explicitly incorporated. This factor characterizes the synergistic diagnostic value of modalities i and j under the current analytical objective (e.g., if the psychiatric knowledge base indicates that, in recognizing a specific microexpression, the eye micromovements of the event camera are highly synergistic with the RGB facial AU4 (glabellar depression)). It will be assigned a higher value. This represents the prior modulating factor for cross-modal feature collaboration. It is a value retrieved from the Psychiatry Knowledge Base (PKB) and is specifically used to adjust the attention weights between features of the Event Camera (EVT) modality and features of the RGB Camera (RGB) modality in the Dynamic Cross-Modal Attention (DCMA) layer. This is achieved through the attention calculation formula:

[0084] in, The dimension of the key vector.

[0085] This allows the attention mechanism to prioritize cross-modal feature sequences (pairs) that, based on expert knowledge, are more likely to collaboratively indicate the target state.

[0086] Simultaneously, intramodal self-attention computation is performed in parallel. The feature sequence of each modality is input into a standard multi-head self-attention layer to obtain the intramodal self-attention output. Subsequently, the cross-modal attention output and the intramodal self-attention output are integrated, and feature fusion is achieved through a concatenation operation followed by a linear transformation.

[0087] Finally, the fusion representation output layer performs pointwise feedforward network processing on the integrated feature sequence, and combines residual connections and layer normalization operations to output the updated feature sequence of the dynamic cross-modal attention layer. This sequence will serve as the input to the next layer or the basis for the final fusion representation. The output of the last dynamic cross-modal attention layer (e.g., concatenating the updated sequences of all modalities and then passing them through a global pooling layer or a special [CLS] label) yields a fixed-length deep fusion feature vector. Theoretically, this vector encapsulates the most critical and collaborative information related to the mental state of all modalities within the current time window.

[0088] refer to Figure 3 , Figure 3 This is a schematic diagram of the structure of the dynamic cross-modal attention fusion network guided by the psychiatric knowledge base provided by this invention. Millimeter-wave radar features, RGB camera features, and event camera features are input to their corresponding modality-specific embedding and preliminary encoding layers (i.e., radar modal encoder, RGB modal encoder, and EVT modal encoder) for feature mapping and encoding. Subsequently, the encoded features, along with prior knowledge from the psychiatric knowledge base, are input to the modality reliability assessment and gating weighting module for dynamic weighting based on signal quality and expert knowledge. The weighted feature sequence then enters a multi-layered dynamic cross-modal attention fusion layer composed of multiple identical stacked layers for knowledge-guided cross-modal and intra-modal information interaction, resulting in an updated feature sequence. Finally, the updated feature sequence is integrated by the fusion representation output layer into a fixed-length deep fusion feature vector for subsequent mental state inference.

[0089] In some embodiments, the feature dimension Dmodel can be set to 512, the number of dynamic cross-modal attention layers N can be selected between 4 and 6, and the number of attention heads H can be set to, for example, 8. For the cross-modal co-modulation factors in the psychiatric knowledge base, their representation and learning method can be designed as learnable parameters (usually initialized to fixed values ​​based on expert rules and fine-tuned during model training), or directly used as modulation terms for attention calculation.

[0090] Through this invention, the dynamic cross-modal attention fusion network achieves dynamic reliability weighting of the input modality through modal reliability assessment and gating weighting modules, and intelligently guides cross-modal feature interactions through attention modulation factors guided by a psychiatric knowledge base in the multi-layered dynamic cross-modal attention fusion layers. This enables the model not only to adaptively handle data quality issues, but also to integrate the domain knowledge of psychiatric experts into the subtle mechanisms of feature fusion, thereby learning more clinically meaningful and discriminative fusion representations.

[0091] According to the present invention, a psychological state assessment method based on multi-source heterogeneous perception, before inputting a deep fusion feature vector into a psychological state inference model to obtain the user's predicted psychological state output by the psychological state inference model, the method further includes: Obtain a clinical validation dataset of multimodal mental states; Based on a clinical validation dataset of multimodal mental states, a pre-trained large-scale multimodal basic model was fine-tuned using a hierarchical learning rate and a stepwise unfreezing strategy to obtain a mental state inference model.

[0092] In this embodiment of the invention, the input to the mental state inference model is a deep fusion feature vector output by a dynamic cross-modal attention fusion network (e.g., with a dimension of 512 or 768, representing the comprehensive state within a time window).

[0093] The mental state inference model employs a large-scale multimodal foundational model pre-trained on multimodal physiological and behavioral data. For example, it could be a Transformer-based encoder-decoder architecture (such as variants of BART or T5, with a parameter range of 50M-200M). This model has been self-supervised or pre-trained on a mixed dataset containing large-scale anonymized clinical text (such as summaries of psychiatric records), EEG sequences, wearable device data (such as activity levels and heart rate), and partial visual / auditory behavioral fragments. Pre-training tasks can include masked signal reconstruction, cross-modal translation, and next time step prediction, enabling it to learn the inherent patterns of physiological signals, the temporal dynamics of behavioral patterns, and the potential correlations between different modalities.

[0094] The following describes an example of fine-tuning a pre-trained large-scale multimodal base model to obtain a mental state inference model.

[0095] The dataset utilizes a high-quality, annotated, multimodal clinical validation dataset of mental states. This dataset contains data from at least 500 participants (including healthy controls and subgroups of specific mental disorders diagnosed according to DSM-5 criteria, such as GAD, SAD, MDD, PTSD, etc., as well as individuals at different levels of stress or cognitive load) simultaneously acquired using millimeter-wave radar, RGB cameras, and event cameras in standardized experimental paradigms (such as emotion-inducing images / videos, Stroop tests, public speaking simulations, relaxation guidance, etc.) and some natural interaction scenarios. All participants underwent rigorous clinical interviews (e.g., MINIPlus 7.0.2, SCID-5-CV) and a set of standardized psychological assessment scales (e.g., HAM-A, HAM-D, GAD-7, PHQ-9, PSS, BFI-2, PANAS-X) for multidimensional and refined mental state labeling. The labeling process employed a dual-person independent labeling mechanism plus senior expert review to ensure the accuracy and consistency of the labels.

[0096] During fine-tuning, the deep fusion feature vector Ffused, extracted by the dynamic cross-modal attention fusion network, is used as input to replace the original input embedding layer of the pre-trained model (or an adaptation layer is designed). Output heads specific to the mental state assessment task of this invention are added to the top layer of the pre-trained model (e.g., a multi-label classification head for anxiety / depression levels, two independent linear heads for emotion valence-arousal regression, and a Softmax head for cognitive load level classification). Fine-tuning employs a strategy of differentiating learning rates and gradual unfreezing: first, only the newly added output head and adaptation layer are trained; then, several layers near the output in the pre-trained model are gradually unfrozen; finally, the entire model (or most layers) is fine-tuned end-to-end with a small learning rate. This helps retain the general knowledge learned by the pre-trained model while effectively adapting it to the specific task and data distribution of this invention.

[0097] The loss function of a pre-trained large-scale multimodal base model is selected based on the specific task; for example, cross-entropy loss is used for classification tasks, and mean squared error loss is used for regression tasks. For multi-label or multi-task outputs, a weighted combined loss is used.

[0098] By fine-tuning a powerful pre-trained base model on high-quality, domain-specific datasets annotated by experts, the mental state inference model is expected to achieve high-precision identification and quantitative evaluation of a variety of complex mental states (especially their early, subtle, or atypical manifestations). Its performance (such as in AUC, F1-score, RMSE, etc.) will be significantly better than custom small models trained from scratch or traditional machine learning methods.

[0099] In some embodiments, the preliminary psychological state assessment results output by the model can be compared with the typical physiological-behavioral patterns, clinical diagnostic criteria, and exclusion criteria for that state stored in the psychiatric knowledge base. If significant inconsistencies are found (e.g., the model identifies severe anxiety, but the HRV index and micromotor patterns differ significantly from the typical manifestations of severe anxiety in the psychiatric knowledge base), the system may indicate that the results are of low reliability or suggest manual review.

[0100] By combining the model's own interpretive tools (such as attention maps and feature importance ranking) with clinical knowledge from the psychiatric knowledge base, assessment reports that are easier for clinicians to understand are generated. For example, the report not only gives "Anxiety score: 7 / 10", but also includes explanations such as "Main evidence: The HRV low-frequency / high-frequency ratio is significantly increased, and the event camera captured high-frequency eye saccades and a specific facial micro-expression pattern A, which are highly consistent with the acute anxiety response pattern defined in the psychiatric knowledge base."

[0101] The psychiatric knowledge base can contain patterns defined by psychiatrists that predict a potential deterioration or crisis in mental state (e.g., persistently low HRV, the presence of specific negative keywords in speech, combined with specific micromotor behaviors). When the feature combination output by the inference model triggers these warning rules, the system can alert the user or guardian.

[0102] This invention provides a multi-layered, user-customizable output interface. A standard user interface presents intuitive charts (such as mood wheels and stress dashboards), concise text summaries, and personalized suggestions (based on health guidelines from a psychiatric knowledge base). A professional user interface (for psychiatrists and psychologists) provides more detailed quantitative assessment results (scores for each dimension, confidence intervals), time-series graphs of key physiological / behavioral characteristics, interpretability analysis results of the model, and links to relevant entries in the psychiatric knowledge base to assist in professional judgment and intervention plan development.

[0103] Through the embodiments of this invention, by synergistically fusing the internal physiological information of millimeter-wave radar, the macroscopic behavioral information of RGB cameras, and the high-speed microscopic dynamic information of event cameras, it is possible to capture multi-dimensional and multi-scale physiological and behavioral patterns related to psychological states that are difficult to perceive with a single modality, thereby achieving a more comprehensive and accurate psychological state profile. A high-precision data synchronization mechanism ensures the temporal consistency of multimodal information. The modality reliability assessment and gating weighting module and the multi-layer dynamic cross-modal attention fusion layer in the dynamic cross-modal attention fusion network can dynamically adjust the contribution of each modality based on signal quality and prior knowledge in psychiatry, effectively addressing situations where the signal quality of a single modality deteriorates or information is missing, thus improving the stability and reliability of the system in real and changing environments.

[0104] The event camera's ability to capture rapid and subtle behaviors such as micro-expressions and eye movements, combined with millimeter-wave radar's precise monitoring of physiological indicators like HRV, makes the system capable of identifying early, subclinical, or deliberately concealed changes in psychological states. By systematically integrating a psychiatric knowledge base into feature selection, modality fusion weight adjustment, and the training and calibration of the final inference model, the system's evaluation results more closely align with clinical diagnostic criteria and provide a basis for interpretability. Based on more accurate and comprehensive psychological state representations, it can provide users with more targeted mental health feedback, early warnings, and intervention suggestions.

[0105] The following describes the psychological state assessment system based on multi-source heterogeneous perception provided by the present invention. The psychological state assessment system based on multi-source heterogeneous perception described below can be referred to in correspondence with the psychological state assessment method based on multi-source heterogeneous perception described above.

[0106] refer to Figure 4 , Figure 4 This is a schematic diagram of the modules of the psychological state assessment system based on multi-source heterogeneous perception provided by the present invention.

[0107] The acquisition module 401 is used to acquire multi-source heterogeneous sensing data streams for the target user, wherein the multi-source heterogeneous sensing data streams include data collected by millimeter-wave radar, RGB camera and event camera pre-deployed in the user's daily environment; Alignment module 402 is used to perform hybrid synchronization processing based on multi-source heterogeneous sensing data streams to obtain time-aligned multimodal data sequences; The extraction module 403 is used to extract modality-specific features from multimodal data sequences to obtain millimeter-wave radar features, RGB camera features, and event camera features; The fusion module 404 is used to input millimeter-wave radar features, RGB camera features and event camera features into a dynamic cross-modal attention fusion network guided by a psychiatric knowledge base for adaptive attention fusion, and obtain a deep fusion feature vector output by the dynamic cross-modal attention fusion network. The prediction module 405 is used to input the deep fusion feature vector into the mental state inference model to obtain the user's predicted mental state output by the mental state inference model.

[0108] Specifically, the psychological state assessment system based on multi-source heterogeneous perception provided by the present invention can realize all the method steps implemented in the above-mentioned psychological state assessment method embodiment based on multi-source heterogeneous perception, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0109] Figure 5This is a schematic diagram of the physical structure of an electronic device provided by the present invention, such as... Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other through the communications bus 540. The processor 510 can call logical instructions in the memory 530 to execute a psychological state assessment method based on multi-source heterogeneous perception. This method includes: acquiring a multi-source heterogeneous perception data stream for a target user, wherein the multi-source heterogeneous perception data stream includes data collected by millimeter-wave radar, RGB cameras, and event cameras pre-deployed in the user's daily environment; performing hybrid synchronous processing on the multi-source heterogeneous perception data stream to obtain a time-aligned multimodal data sequence; extracting modality-specific features from the multimodal data sequence to obtain millimeter-wave radar features, RGB camera features, and event camera features; inputting the millimeter-wave radar features, RGB camera features, and event camera features into a dynamic cross-modal attention fusion network guided by a psychiatric knowledge base for adaptive attention fusion to obtain a deep fusion feature vector output by the dynamic cross-modal attention fusion network; and inputting the deep fusion feature vector into a psychological state inference model to obtain the user's predicted psychological state output by the psychological state inference model.

[0110] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0111] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the psychological state assessment method based on multi-source heterogeneous perception provided by the above methods. The method includes: acquiring a multi-source heterogeneous perception data stream for a target user, wherein the multi-source heterogeneous perception data stream includes data collected by millimeter-wave radar, RGB camera, and event camera pre-deployed in the user's daily environment; performing hybrid synchronous processing based on the multi-source heterogeneous perception data stream to obtain a time-aligned multimodal data sequence; extracting modality-specific features from the multimodal data sequence to obtain millimeter-wave radar features, RGB camera features, and event camera features; inputting the millimeter-wave radar features, RGB camera features, and event camera features into a dynamic cross-modal attention fusion network guided by a psychiatric knowledge base for adaptive attention fusion to obtain a deep fusion feature vector output by the dynamic cross-modal attention fusion network; and inputting the deep fusion feature vector into a psychological state inference model to obtain the user's predicted psychological state output by the psychological state inference model.

[0112] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the psychological state assessment method based on multi-source heterogeneous perception provided by the above methods. This method includes: acquiring a multi-source heterogeneous perception data stream for a target user, wherein the multi-source heterogeneous perception data stream includes data collected by millimeter-wave radar, RGB camera, and event camera pre-deployed in the user's daily environment; performing hybrid synchronous processing based on the multi-source heterogeneous perception data stream to obtain a time-aligned multimodal data sequence; extracting modality-specific features from the multimodal data sequence to obtain millimeter-wave radar features, RGB camera features, and event camera features; inputting the millimeter-wave radar features, RGB camera features, and event camera features into a dynamic cross-modal attention fusion network guided by a psychiatric knowledge base for adaptive attention fusion to obtain a deep fusion feature vector output by the dynamic cross-modal attention fusion network; and inputting the deep fusion feature vector into a psychological state inference model to obtain the user's predicted psychological state output by the psychological state inference model.

[0113] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0114] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for assessing psychological states based on multi-source heterogeneous perception, characterized in that, include: Acquire multi-source heterogeneous sensing data streams for target users, wherein the multi-source heterogeneous sensing data streams include data collected by millimeter-wave radar, RGB cameras and event cameras pre-deployed in the user's daily environment; Based on the multi-source heterogeneous sensing data stream, a time-aligned multimodal data sequence is obtained through hybrid synchronization processing. Modality-specific feature extraction is performed on the multimodal data sequence to obtain millimeter-wave radar features, RGB camera features, and event camera features; Based on the millimeter-wave radar features, the RGB camera features, and the event camera features, the data are input into a dynamic cross-modal attention fusion network guided by a psychiatric knowledge base for adaptive attention fusion, resulting in a deep fusion feature vector output by the dynamic cross-modal attention fusion network. The deep fusion feature vector is input into the psychological state inference model to obtain the predicted psychological state of the user output by the psychological state inference model.

2. The psychological state assessment method based on multi-source heterogeneous perception according to claim 1, characterized in that, The process of performing hybrid synchronization processing based on the multi-source heterogeneous sensing data stream to obtain a time-aligned multimodal data sequence includes: The central synchronization controller is invoked to generate periodic global synchronization trigger pulses; The global synchronization trigger pulse is simultaneously distributed to the external trigger input of the millimeter-wave radar, the external frame synchronization input of the RGB camera, and the external timestamp synchronization input of the event camera; Based on the timestamp generated by the global synchronization trigger pulse, time windows are aligned for millimeter-wave radar data, RGB camera data, and event camera data to obtain a time-aligned multimodal data sequence.

3. The psychological state assessment method based on multi-source heterogeneous perception according to claim 1, characterized in that, The multimodal data sequence includes: millimeter-wave radar data sequence, RGB camera data sequence, and event camera data sequence; The modality-specific feature extraction of the multimodal data sequence, yielding millimeter-wave radar features, RGB camera features, and event camera features, includes: Physiological parameters and micro-motion features are extracted from the millimeter-wave radar data sequence to obtain millimeter-wave radar features. Among them, an electrocardiogram waveform is extracted from the millimeter-wave radar data sequence using a feature extraction network based on deep learning, and heart rate variability parameters, respiratory pattern features and micro-motion pattern features related to psychological state are determined based on the electrocardiogram waveform. Visual behavior features are extracted from the RGB camera data sequence to obtain RGB camera features. Specifically, a facial action unit recognition algorithm is used to extract facial action unit intensity sequences based on the RGB camera data sequence, and a three-dimensional human pose estimation algorithm is used to extract body pose features and head movement features based on the RGB camera data sequence. Micro-dynamic behavior features are extracted from the event camera data sequence to obtain event camera features. Specifically, the asynchronous event stream in the event camera data sequence is converted into a synchronized image sequence through an adaptive spatiotemporal density coding algorithm, and micro-expression features and eye movement saccade pattern features are extracted from the synchronized image sequence based on a preset lightweight spatiotemporal graph convolutional network.

4. The psychological state assessment method based on multi-source heterogeneous perception according to claim 1, characterized in that, The dynamic cross-modal attention fusion network includes: a modality-specific embedding and preliminary encoding layer, a modality reliability assessment and gating weighting module, a multi-layer dynamic cross-modal attention fusion layer, and a fusion representation output layer; The features based on the millimeter-wave radar, the RGB camera, and the event camera are input into a dynamic cross-modal attention fusion network guided by a psychiatric knowledge base for adaptive attention fusion, resulting in a deep fusion feature vector output by the dynamic cross-modal attention fusion network, including: The millimeter-wave radar features, RGB camera features, and event camera features are encoded through the modality-specific embedding and preliminary coding layer. Each modality feature is mapped to a unified high-dimensional feature space and modality type embedding and position coding are added to obtain a preliminary encoded feature sequence. The modal reliability assessment and gating weighting module queries the prior importance weights of each modal feature based on a preset psychiatric knowledge base, and determines the reliability weights of each modal feature by combining the real-time signal quality assessment results. Based on the reliability weights of each modal feature, the initially encoded feature sequence is weighted to obtain the weighted feature sequence. The weighted feature sequence is processed by parallel cross-modal attention calculation and intramodal self-attention calculation through the multi-layer dynamic cross-modal attention fusion layer to obtain the updated feature sequence. The updated feature sequence is spliced ​​and fused through the fusion representation output layer to generate a fixed-length deep fusion feature vector.

5. The psychological state assessment method based on multi-source heterogeneous perception according to claim 4, characterized in that, The multi-layer dynamic cross-modal attention fusion layer is composed of multiple identical dynamic cross-modal attention layers stacked together; The step of performing parallel cross-modal attention calculation and intra-modal self-attention calculation on the weighted feature sequence to obtain the updated feature sequence includes: The updated feature sequence is obtained by performing the following operations on the weighted feature sequence at each dynamic cross-modal attention layer: Based on the weighted feature sequence, parallel cross-modal attention calculation is performed to obtain cross-modal attention output. In this process, the feature sequence of each modality is used as the query vector, and the feature sequences of other modalities are used as the key vector and value vector. The cross-modal collaborative adjustment factor defined by the psychiatric knowledge base is incorporated into the attention weight calculation. Based on the weighted feature sequence, parallel intramodal self-attention calculation is performed to obtain the intramodal self-attention output; Based on the cross-modal attention output and the intramodal self-attention output, information is integrated through splicing operations and linear transformations to obtain the integrated feature sequence. The integrated feature sequence is processed by a point-forward network, and combined with residual connections and layer normalization operations, to output the updated feature sequence.

6. The psychological state assessment method based on multi-source heterogeneous perception according to claim 1, characterized in that, Before inputting the deep fusion feature vector into the mental state inference model to obtain the predicted mental state of the user output by the mental state inference model, the method further includes: Obtain a clinical validation dataset of multimodal mental states; Based on the aforementioned multimodal psychological state clinical validation dataset, a pre-trained large-scale multimodal basic model was fine-tuned using a hierarchical learning rate and a stepwise unfreezing strategy to obtain a psychological state inference model.

7. A psychological state assessment system based on multi-source heterogeneous perception, characterized in that, include: The acquisition module is used to acquire multi-source heterogeneous sensing data streams for target users, wherein the multi-source heterogeneous sensing data streams include data collected by millimeter-wave radar, RGB camera and event camera pre-deployed in the user's daily environment; The alignment module is used to perform hybrid synchronization processing based on the multi-source heterogeneous sensing data stream to obtain a time-aligned multimodal data sequence; The extraction module is used to extract modality-specific features from the multimodal data sequence to obtain millimeter-wave radar features, RGB camera features, and event camera features; The fusion module is used to input the features of the millimeter-wave radar, the features of the RGB camera, and the features of the event camera into a dynamic cross-modal attention fusion network guided by a psychiatric knowledge base for adaptive attention fusion, so as to obtain a deep fusion feature vector output by the dynamic cross-modal attention fusion network. The prediction module is used to input the deep fusion feature vector into the psychological state inference model to obtain the predicted psychological state of the user output by the psychological state inference model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the psychological state assessment method based on multi-source heterogeneous perception as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the psychological state assessment method based on multi-source heterogeneous perception as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the psychological state assessment method based on multi-source heterogeneous perception as described in any one of claims 1 to 6.

Images