Depression syndrome detection method and system based on multi-modal visual feature fusion

Abstract

The present application relates to the field of computer vision, and more particularly to a depression syndrome detection method and system based on multi-modal visual feature fusion, which comprises: obtaining facial dynamic visual data and facial static visual data containing optical sensing information, the facial dynamic data outputting pulse diagnosis feature vectors through a dynamic pulse diagnosis feature extraction module, and the facial static data outputting seven emotion feature vectors through a static seven emotion feature extraction module; the two feature vectors are fused into feature vectors through a cross-modal visual fusion module, and the depression syndrome TCM syndrome type is outputted through a syndrome type classification module. The method solves the problems of low efficiency and strong subjectivity caused by relying on artificial inquiry and questionnaire, realizes non-contact detection, improves the accuracy and convenience of syndrome type prediction, and is suitable for daily scenes.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method and system for detecting symptoms of depression based on multimodal visual feature fusion. Background Technology

[0002] Current mainstream diagnostic methods for depression primarily rely on operational or standardized diagnostic criteria, such as the DSM or Zung scales, which have significant limitations: they require long-term professional training for psychiatrists, sustained patient cooperation, and excessive reliance on subjective descriptive materials, resulting in insufficient objectivity and limited information dimensions. While research on computer science-based depression early warning models has made progress, most models still rely on data from questionnaires and electroencephalograms (EEGs). The former has highly subjective data sources, while the latter is difficult to obtain conveniently in everyday home environments, thus limiting their universality and hindering their widespread adoption.

[0003] Traditional Chinese medicine (TCM) possesses unique advantages in the prevention and treatment of emotional disorders. Depression falls under the TCM category of emotional disorders, and its pathogenesis is closely related to "internal injury from the seven emotions." Theories such as "the heart governs the mind" and "the heart's manifestation is in the face" reveal the intrinsic connection between pulse diagnosis and facial visual information in reflecting psychological state. The *Ling Shu* (Spiritual Pivot) states, "When blood vessels are smooth and unobstructed, the spirit resides," indicating that harmonious blood and qi and coordinated organ function are the foundation of mental health. Integrating pulse diagnosis and facial expression characteristics can effectively assist in the TCM diagnosis of depression, and both types of information are readily available and have low device dependence, making them highly suitable for everyday home applications.

[0004] In summary, existing diagnostic methods for depression generally suffer from low accuracy, high subjectivity, and complex operation. There is an urgent need for a scientific, accurate, and easy-to-use method for predicting depression syndromes that is based on traditional Chinese medicine theory, integrates multimodal information, and is convenient for daily use.

[0005] Chinese invention patent application CN119132562A proposes an intelligent TCM diagnostic system based on an AI large-scale model, belonging to the field of TCM-assisted diagnosis and treatment technology. It includes an information collection module, a data processing module, a diagnosis generation module, and a TCM-specific large-scale model diagnostic module. The information collection module collects user information through facial recognition, tongue image recording, patient self-report, and TCM palpation. The data processing module preprocesses the five types of collected information. The diagnosis generation module diagnoses the data output by the data processing module. The TCM-specific large-scale model diagnostic module takes the comprehensive diagnosis generated by the diagnosis generation module as input and outputs corresponding treatment suggestions through a pre-trained TCM-specific large-scale model. Although this technology adaptively combines TCM with an AI large-scale model, it lacks a deep cross-modal fusion mechanism for multimodal features.

[0006] Existing literature CN120260960A proposes a visualization data processing system for pediatric TCM. The system includes the following modules: a data acquisition module for acquiring pediatric physiological time-series datasets; identifying physiological event anchor points in the pediatric physiological time-series datasets to obtain a set of physiological event anchor points; a time calibration module for noise suppression of the pediatric physiological time-series datasets based on the set of physiological event anchor points to obtain a noise-suppressed physiological time-series dataset; performing time calibration on the noise-suppressed physiological time-series dataset to obtain a time-corrected physiological dataset; and a feature extraction module for extracting a set of tongue images from the time-corrected physiological dataset; and generating a set of tongue image feature vectors based on the tongue image set. This technology only performs multimodal data processing for pathological judgment and does not incorporate deep fusion and adaptive design of multimodal features based on depression syndrome types and TCM theory. Summary of the Invention

[0007] The purpose of this invention is to overcome the problems of low accuracy, strong subjectivity, and complex operation in the existing technology, and to provide a method and system for detecting depression symptoms based on multimodal visual feature fusion.

[0008] In a first aspect, the present invention provides a method for detecting symptoms of depression based on multimodal visual feature fusion. The method includes:

[0009] Acquire multimodal visual data of the user containing optical sensing information, including dynamic facial visual data and static facial visual data;

[0010] The above-mentioned facial dynamic visual data is input into the dynamic pulse diagnosis feature extraction module, and then passed through the dynamic noise suppression unit and the temporal pulse diagnosis feature extraction unit of the above-mentioned dynamic pulse diagnosis feature extraction module in sequence to output the pulse diagnosis feature vector.

[0011] The above-mentioned static facial visual data is input into the static seven emotions feature extraction module, and then passed through the static noise suppression unit and the spatial seven emotions feature extraction unit of the static seven emotions feature extraction module in sequence to output the seven emotions feature vector.

[0012] The pulse diagnosis feature vector and the seven emotions feature vector are input into the cross-modal visual fusion module. The feature is fused through the temporal feature map construction unit, spatial feature map construction unit and cross-modal map attention fusion unit of the cross-modal visual fusion module, and the fused feature vector is output.

[0013] Input the above-mentioned fused feature vector into the syndrome classification module to output the user's TCM syndrome type of depression.

[0014] The pulse diagnosis feature vector is a fixed-dimensional numerical vector containing the temporal characteristics of the pulse diagnosis signal obtained by remote photoplethysmography. The seven emotions feature vector is a fixed-dimensional numerical vector containing the spatial characteristics of the seven emotions. The fusion feature vector is a fixed-dimensional numerical vector containing both the temporal characteristics of the pulse diagnosis signal and the spatial characteristics of the seven emotions.

[0015] The seven emotions include the seven emotional states of joy, anger, worry, pensiveness, grief, fear, and fright in traditional Chinese medicine theory. The aforementioned spatial characteristics associated with the seven emotions refer to the facial spatial features (such as facial muscle texture, facial features, and distribution of key points of expression) that are directly related to the seven emotional states. These characteristics are automatically extracted from static facial visual data by the static seven emotions feature extraction module after training on a training dataset containing the seven emotions classification labels (labeled with the categories of joy, anger, worry, pensiveness, grief, fear, and fright).

[0016] Preferably, the dynamic pulse diagnosis feature extraction module and the static seven emotions feature extraction module are trained using an adaptive optimizer combined with a learning rate scheduling strategy. During training, the dynamic pulse diagnosis feature extraction module classifies the pulse diagnosis features in the dynamic facial visual data into at least two categories based on traditional Chinese medicine pulse types, and optimizes the model parameters using a cross-entropy loss function or a temporal classification loss function. Similarly, during training, the static seven emotions feature extraction module classifies the seven emotions features in the static facial visual data into at least two categories based on traditional Chinese medicine seven emotions types, and optimizes the model parameters using a cross-entropy loss function. Finally, during training, the parameters of the dynamic pulse diagnosis feature extraction module and the static seven emotions feature extraction module are frozen, and the model parameters of the cross-modal visual fusion module are optimized using a classification loss function.

[0017] This approach employs a combined training scheme of adaptive optimizer and learning rate scheduling strategy. This addresses the issues of single optimizers easily getting trapped in local optima and fixed learning rates causing training oscillations or slow convergence, ensuring the training stability and efficiency of both dynamic pulse diagnosis and static seven emotions modules. A targeted loss function matches the data characteristics of each module, precisely optimizing model parameters and improving the accuracy of single-modal feature extraction. During the cross-modal fusion stage, the parameters of the basic modules are frozen to prevent changes in the optimized feature extraction capabilities during fusion module training. Simultaneously, a classification loss function focuses on optimizing the modal fusion logic, balancing the stability of basic features with the targeting of fused features, thereby improving the accuracy of syndrome detection.

[0018] Preferably, the dynamic noise suppression unit includes a convolutional layer and a multi-scale spatiotemporal attention model; the multi-scale spatiotemporal attention model includes at least two spatiotemporal convolutional layers with different scales and dilation parameters. The dilation parameters of the spatiotemporal convolutional layers of the multi-scale spatiotemporal attention model are set according to a preset gradient. After the features of different scales are mapped to the same dimension by adaptive pooling, feature integration is achieved by element-level fusion or splicing fusion.

[0019] A dual-layer noise suppression mechanism, employing convolutional layers for initial denoising and multi-scale spatiotemporal attention for refined processing, effectively filters out spatial or temporal noise such as motion artifacts and illumination fluctuations in dynamic facial videos, addressing the problem that traditional single filtering cannot handle multi-scale noise. The dilated convolution of multi-scale spatiotemporal attention can cover spatiotemporal features of different ranges, and adaptive pooling achieves multi-scale feature alignment, avoiding feature dimension confusion. Combined with energy threshold screening of remote photoplethysmography signals, it accurately locates key spatiotemporal regions containing effective pulse diagnosis information, eliminates redundant regions without pulse information, and significantly improves the signal-to-noise ratio and efficiency of subsequent temporal pulse diagnosis feature extraction.

[0020] Preferably, the aforementioned temporal pulse diagnosis feature extraction unit includes a channel attention component and a temporal feature encoding component. The channel attention component enhances the feature weights of the stable channel of the pulse diagnosis signal through multiple sets of spatiotemporal convolution and pooling operations. The temporal feature encoding component extracts the pulse diagnosis signal using spatiotemporal slice encoding and a multilayer temporal encoder.

[0021] In the spatiotemporal slicing encoding operation of the aforementioned temporal feature encoding component, the spatiotemporal tube sampling technique of Vision Transformer for Video is used to segment the facial dynamic visual data into multiple spatiotemporal units. Each spatiotemporal unit is flattened and then transformed into a fixed-dimensional feature vector through a linear projection layer. All feature vectors constitute an embedding matrix. The embedding matrix is ​​concatenated with the position encoding and pulse category encoding and then input into a multilayer temporal encoder. The position encoding is generated by a preset periodic function. The preset periodic function is a spatiotemporal separation encoding function, which includes spatial position encoding and temporal position encoding. The spatial position encoding is generated by a cosine function, and the temporal position encoding is generated by a sine function.

[0022] The channel attention component calculates the signal-to-noise ratio and assigns differentiated weights through a sliding window, which can directionally enhance stable channels of remote optical volumetric imaging signals, reduce interference from noisy channels, and solve the problem of effective signals being masked by redundant information in multi-channel data. The temporal coding component adopts the Vision Transformer for Video spatiotemporal tube sampling technology to divide the video into multiple spatiotemporal units. Combined with spatiotemporal separation coding, it not only preserves the cyclicity and stability of the pulse diagnosis signal, but also accurately captures the temporal dependence, avoiding the defects of traditional temporal coding in losing spatial location information or temporal continuity, and providing high-quality temporal features for subsequent pulse diagnosis feature map construction.

[0023] Preferably, the static noise suppression unit includes a noise filtering component and an edge enhancement component; the noise filtering component of the static noise suppression unit adopts at least one of Gaussian filtering, bilateral filtering, and nonlocal mean filtering; the edge enhancement component adopts at least one of high-pass filtering, Laplacian enhancement, and contrast-limited adaptive histogram equalization.

[0024] When the noise filtering component is a Gaussian low-pass filter, the standard deviation of the Gaussian low-pass filter is adaptively adjusted according to the gray-level variance of the facial static visual data: when the gray-level variance is greater than 50, the standard deviation of the Gaussian low-pass filter is 1.5-2.0; when the gray-level variance is less than or equal to 50, the standard deviation of the Gaussian low-pass filter is 0.5-1.0; when the edge enhancement component is a Gaussian high-pass filter, the standard deviation of the Gaussian high-pass filter is 1.5 times the standard deviation of the Gaussian low-pass filter.

[0025] By designing noise filtering and edge enhancement components, the noise interference in static facial images is resolved, while edge blurring caused by denoising is avoided. The Gaussian low-pass filter adaptively adjusts the standard deviation according to the gray-level variance, which can match images with different noise intensities and avoid incomplete denoising or loss of details caused by fixed parameters. The standard deviation of the Gaussian high-pass filter is 1.5 times that of the low-pass filter, achieving a dynamic balance between denoising and edge enhancement, ensuring that subtle facial textures are clearly preserved, and providing a high-definition image foundation for subsequent extraction of facial features.

[0026] Preferably, the aforementioned spatial seven emotions feature extraction unit includes a channel-spatial attention component and a high-resolution feature extraction component. The channel-spatial attention component filters dual-path features based on depthwise separable convolution, and the high-resolution feature extraction component extracts and fuses facial spatial features of different scales through multi-resolution branches in parallel.

[0027] The aforementioned high-resolution feature extraction component comprises at least two parallel resolution branches. Each branch extracts features through multi-layer convolution, and then uses a bidirectional fusion strategy of downsampling projection and upsampling supplementation to fuse the features extracted by each parallel resolution branch. The downsampling projection achieves feature dimension matching through stride convolution or pooling operations, and the upsampling supplementation restores the feature size through transposed convolution or interpolation operations. During fusion, weights are dynamically allocated based on feature information entropy, with feature branches having higher information entropy having a higher weight ratio.

[0028] The channel-spatial attention component constructs a dual-path structure based on depthwise separable convolution, which automatically locates key facial emotion regions while filtering key channel features, balancing feature selection accuracy and computational efficiency. The high-resolution feature extraction component features bidirectional fusion of multi-resolution branches, which can retain full-scale features from the original resolution to low resolution, avoiding the problem of losing subtle features in traditional single-resolution extraction. Combined with information entropy weighting, it prioritizes the retention of high information density features related to the seven emotions, significantly improving the recognition and distinguishability of the seven emotion features.

[0029] Preferably, the above method first converts the RGB channel features of the pulse diagnosis feature vector into a single-channel grayscale image using a weighted average formula. Then, the time-series feature map construction unit performs time-domain frequency domain transformation on multiple feature channels of the pulse diagnosis feature vector, calculates the frequency domain energy change of each pixel, and concatenates the multi-channel energy information. Finally, a pulse diagnosis feature map is constructed using a clustering algorithm. The clustering algorithm includes at least one of the following: K-means algorithm, DBSCAN algorithm, or hierarchical clustering algorithm.

[0030] Multi-channel pulse diagnosis features are transformed into single-channel grayscale images by RGB channel weighted averaging, unifying feature dimensions while preserving the energy differences of remote photoplethysmography signals. Frequency domain transformation in the time dimension can capture the frequency domain energy changes of pulse diagnosis signals. Combined with clustering algorithms, a structured pulse diagnosis feature map is constructed, transforming abstract temporal pulse diagnosis features into node-edge graph structure data. This not only preserves the energy distribution pattern of pulse diagnosis signals but also provides a structured carrier for subsequent cross-modal fusion, solving the problem that temporal features are difficult to directly fuse with spatial features.

[0031] Preferably, the above-mentioned time-series feature map construction unit uses Fourier transform or wavelet transform for frequency domain transformation, and the frequency range of the frequency domain transformation is 0.1-10Hz;

[0032] The aforementioned spatial feature map construction unit uses a superpixel segmentation algorithm to process the seven emotion feature vectors and construct the seven emotion feature map; the aforementioned cross-modal graph attention fusion unit unifies the node feature representations of the two feature maps through a cross-modal Vision Transformer, and then extracts and fuses cross-modal features through a graph attention mechanism; the aforementioned superpixel segmentation algorithm includes at least one of the SLIC algorithm, SEEDS algorithm, or LSC algorithm;

[0033] The aforementioned cross-modal graph attention fusion unit adopts a multi-attention head mechanism. The feature dimension of a single attention head is a single dimension obtained by equally dividing the total dimension of the fused features according to the number of attention heads. When calculating the edge weight, the final edge weight is obtained by combining the feature space distance and the semantic similarity between modalities through weighted summation. The aforementioned feature space distance is selected from Euclidean distance or Manhattan distance, and the aforementioned semantic similarity between modalities is selected from cosine similarity or Pearson correlation coefficient.

[0034] Frequency domain transformation is limited to the 0.1-10Hz range, accurately covering the frequency range of normal human pulse and avoiding interference from irrelevant frequencies; the superpixel segmentation algorithm constructs a seven-emotion feature map, which can transform facial spatial features into a structured map, forming a unified data format with the pulse diagnosis feature map; the cross-modal Vision Transformer unifies the semantic space of the two modalities through a learnable projection matrix, solving the semantic gap between the heterogeneous feature temporal sequence and the seven emotions in the pulse diagnosis space; the edge weight calculation of multi-attention heads, feature space distance, and semantic similarity can deeply explore the correlation between modalities, avoiding the modal information fragmentation caused by the simple splicing of traditional fusion, and significantly improving the correlation and effectiveness of cross-modal fusion features.

[0035] Preferably, the above-mentioned syndrome classification module adopts a multilayer perceptron or a deep learning classification network; the output of the above-mentioned TCM syndrome of depression includes at least one of the following: liver qi stagnation, liver qi stagnation and spleen deficiency, liver qi stagnation transforming into fire, heart and spleen deficiency, kidney deficiency and liver qi stagnation, and phlegm and qi stagnation.

[0036] In the classification network of the above-mentioned certificate type classification module, a feature attention gating unit is added between the input layer and the first classification layer: the importance weights of each dimension of the fused feature vector are calculated through the activation function.

[0037] Multilayer perceptrons or deep learning classification networks can adapt to the high-dimensionality of fused features and accurately map the correspondence between fused features and TCM syndrome types of depression. Feature attention gating units use the sigmoid function to filter key feature dimensions, effectively filtering redundant fused features and avoiding overfitting or decreased accuracy caused by the curse of dimensionality. Ultimately, this achieves accurate output of TCM syndrome types of depression, solving the problems of strong subjectivity and low accuracy in traditional manual diagnosis.

[0038] In a second aspect, the present invention provides a depression syndrome detection system based on multimodal visual feature fusion.

[0039] The system includes a dynamic pulse diagnosis feature extraction module, a static seven emotions feature extraction module, a cross-modal visual fusion module, and a syndrome classification module;

[0040] It also includes a visual data acquisition module, which is used to acquire multimodal visual data of the user containing optical sensing information;

[0041] When the system was running, the aforementioned method for detecting depression symptoms based on multimodal visual feature fusion was executed.

[0042] All information collected in this method has been agreed upon by the subject. When implementing the above method, this invention strictly follows the "minimum necessity principle" stipulated in the Personal Information Protection Law and anonymizes the collected data. The aforementioned anonymized dataset cannot identify any specific individual and is irreversible. Therefore, its use does not involve personal information processing activities.

[0043] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0044] This invention provides a method for detecting depression symptoms based on multimodal visual feature fusion. By acquiring multimodal visual data containing optical sensor information, it provides a comprehensive and complementary data source for subsequent bimodal feature extraction, avoiding feature bias caused by insufficient coverage of a single visual modality. By inputting dynamic facial visual data into a dynamic pulse diagnosis feature extraction module, interference such as motion artifacts and illumination noise in dynamic videos is filtered out, accurately extracting the temporal features of pulse diagnosis signals, avoiding noise interference in dynamic data and loss of pulse diagnosis temporal information. By inputting static facial visual data into a static seven emotions feature extraction module, Gaussian noise, salt-and-pepper noise, etc., in static images are removed, preserving key spatial features of facial emotions, avoiding blurring of details in static images that could affect the extraction of emotions. The feature recognition rate is too low. By inputting the pulse diagnosis feature vector and the seven emotions feature vector into the cross-modal visual fusion module, the heterogeneous temporal pulse diagnosis features and spatial seven emotions features are transformed into unified graph structure data, obtaining the semantic association between the two modalities and avoiding the semantic gap and insufficient fusion of cross-modal features. By inputting the fused feature vector into the syndrome classification module, the highly correlated cross-modal fused features are accurately mapped to the TCM syndrome of depression, realizing the automated output of syndrome. Finally, it can comprehensively realize non-contact, high accuracy and strong robustness of TCM syndrome detection of depression, overcome the shortcomings of traditional reliance on manual consultation and questionnaire assessment, such as strong subjectivity, low efficiency and large individual differences, and provide a standardized and intelligent technical solution for TCM syndrome detection of depression. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the system modules in Example 1.

[0046] Figure 2 This is a schematic diagram of the dynamic pulse diagnosis feature extraction module in Example 1.

[0047] Figure 3 This is a schematic diagram of the static seven emotions feature extraction module in Example 1.

[0048] Figure 4This is a schematic diagram of the cross-modal visual fusion module in Example 1.

[0049] Figure 5 This is a schematic diagram of the method flow in Example 2. Detailed Implementation

[0050] The present invention will now be described in further detail with reference to specific embodiments. However, this should not be construed as limiting the scope of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.

[0051] Unless otherwise specified, the terms "upper," "lower," "left," "right," "center," "inner," and "outer," etc., used in the description of specific embodiments of the present invention to indicate orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings, or the orientation or positional relationship in which the product / equipment / device is usually placed during use. These terms are merely for the purpose of facilitating the description of the present invention or simplifying the description in specific embodiments, and for enabling those skilled in the art to quickly understand the solution, and do not indicate or imply that a particular device / component / element must have a specific orientation, or be constructed and operated in a specific positional relationship. Therefore, they should not be construed as limitations on the present invention.

[0052] Furthermore, the use of terms such as "horizontal," "vertical," "suspended," "parallel," and "coaxial" does not imply that the corresponding device / component / element must be absolutely horizontal, vertical, suspended, parallel, or coaxial. Slight tilt or deviation is permissible, as long as it does not affect the normal function of the relevant component. For example, "horizontal" simply means that its direction is more horizontal relative to "vertical," not that the structure must be perfectly horizontal; a slight tilt is acceptable. "Coaxial" means that two components are arranged as coaxially as possible, allowing them to move coaxially or approximately coaxially when their relative positions change. Alternatively, it can be simplified to mean that the corresponding device / component / element, when arranged in "horizontal," "vertical," "suspended," "parallel," or "coaxial" directions, can have an error / deviation of ±10% relative to the corresponding direction, more preferably within ±8%, more preferably within ±6%, more preferably within ±5%, and more preferably within ±4%. For example, the deviation in the "coaxial" direction is controlled within 0.2-1mm, preferably within 0.2-0.5mm. As long as the corresponding device / component / element is within the error / deviation range, it can still achieve its function in the solution of the present invention.

[0053] Furthermore, the use of terms such as "first," "second," and "third" in terminology is merely for distinguishing descriptions of identical or similar components and should not be interpreted as emphasizing or implying the relative importance of a particular component.

[0054] Furthermore, in the description of the embodiments of the present invention, "several", "more than", and "a number of" represent at least two. The number can be any number, such as two, three, four, five, six, seven, eight, or nine, and can even exceed nine.

[0055] Furthermore, in the description of the technical solution of this invention, unless otherwise explicitly specified / limited / restricted, the terms "set up," "install," "connect," "link," "provided with," "laid out," and "arranged" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to connection methods commonly used in the art, such as welding, riveting, bolting, and threaded connections. Such connections can be mechanical, electrical, or communication connections; they can be direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components.

[0056] Example 1

[0057] This embodiment introduces a method and system for detecting symptoms of depression based on multimodal visual feature fusion. The system structure is as follows: Figure 1 As shown, the method includes a dynamic pulse diagnosis feature extraction module, a static seven emotions feature extraction module, a cross-modal visual fusion module, and a syndrome classification module. In this embodiment, the dynamic facial visual data is facial video, and the static facial visual data is facial image. The above method includes the following steps:

[0058] Acquire multimodal visual data of the user containing optical sensing information, including dynamic facial visual data and static facial visual data;

[0059] The above-mentioned facial dynamic visual data is input into the dynamic pulse diagnosis feature extraction module, and then passed through the dynamic noise suppression unit and the temporal pulse diagnosis feature extraction unit of the above-mentioned dynamic pulse diagnosis feature extraction module in sequence to output the pulse diagnosis feature vector.

[0060] The above-mentioned static facial visual data is input into the static seven emotions feature extraction module, and then passed through the static noise suppression unit and the spatial seven emotions feature extraction unit of the static seven emotions feature extraction module in sequence to output the seven emotions feature vector.

[0061] The pulse diagnosis feature vector and the seven emotions feature vector are input into the cross-modal visual fusion module. The feature is fused through the temporal feature map construction unit, spatial feature map construction unit and cross-modal map attention fusion unit of the cross-modal visual fusion module, and the fused feature vector is output.

[0062] Input the above-mentioned fused feature vector into the syndrome classification module to output the user's TCM syndrome type of depression.

[0063] 1. The aforementioned dynamic pulse diagnosis feature extraction module includes a dynamic noise suppression unit and a temporal pulse diagnosis feature extraction unit, such as... Figure 2 As shown.

[0064] (1) The dynamic noise suppression unit described above suppresses the spatial and temporal noise introduced during the facial video acquisition process and adaptively focuses on the spatiotemporal region containing key pulse information to extract the user's pulse diagnosis feature vector. This unit includes a Layer-convolutional layer for preliminary noise removal and a spatiotemporal separation pyramid attention module; the spatiotemporal separation pyramid attention module includes three 3D dilated convolutional layers with different scales and dilation rates, which respectively use adaptive average pooling for feature fusion to remove spatial and temporal noise, and adaptively focus on key time periods and spatial locations.

[0065] (2) The above-mentioned temporal pulse feature extraction unit deeply extracts pulse signals with cyclicity, stability and temporal dependence in facial videos. This unit includes a channel attention layer and a multi-layer Transformer layer; the channel attention layer consists of multiple 3D-Dconv and MAXpool operations, which are used to enhance the attention to the relatively stable color channels of the remote photoplethysmography signal and improve the model's anti-interference ability; the multi-layer Transformer layer includes video slice encoding operation and multi-layer Transformer. The video slice encoding operation uses the spatiotemporal tube sampling in the ViVi method to encode the video slices into vectors, and divides the input video into multiple spatiotemporal tubes (Tubelet, dimension C×T×W×H, where C represents the number of channels, T represents the number of frames, W represents the width, and H represents the length). Each spatiotemporal tube is flattened into a one-dimensional vector and transformed into a fixed dimension through a linear projection layer (fully connected layer). Feature vector (Token):

[0066]

[0067] All tokens form an embedding matrix:

[0068]

[0069] in, It is the first A token for a spacetime management system. It is the first A token for a spacetime management system. It is the first A token for a spacetime management system. It is the first A token for a spacetime management system. It is the total number of spacetime tubes. It is the feature dimension of each token. It is the first A spacetime conduit, It is an embedding matrix. For linear projection layers; For flattening operation; for A matrix of dimensionality; It is a matrix with the number of rows representing the total number of spatiotemporal tubes and the number of columns representing the feature dimensions of a single spatiotemporal tube.

[0070] Then, a cosine-sine function is used to generate a positional encoding that incorporates spatial and temporal information, and a pulse category encoding is added to guide the model to focus on target classification information. Finally, the embedding encoding, positional encoding, and category encoding are concatenated and input into a multi-layer Transformer encoder. The Transformer encoder, through its self-attention mechanism, deeply extracts pulse signals with cyclicity, stability, and temporal dependence from facial videos and effectively captures global spatiotemporal context information.

[0071] During training of this module, a Line layer will be added after the temporal pulse diagnosis feature extraction unit. The extracted pulse features will be categorized into six types: floating, deep, rapid, slow, weak, and strong. The parameters will be set using the cross-entropy loss function as follows:

[0072]

[0073] in, For input data and real labels The loss function of the dynamic pulse diagnosis feature extraction module is jointly determined. This represents the total number of pulse diagnosis cases. For the first A real label for each pulse diagnosis case For the first Pulse feature vectors extracted from individual pulse diagnosis instances. Use the Softmax activation function; It is a linear transformation layer.

[0074] Figure 2 M1-M6 in the diagram represent the pulse characteristics labels output by the dynamic pulse extraction module during training, namely, floating, sinking, rapid, slow, weak, and strong.

[0075] 2. The aforementioned static seven emotions feature extraction module includes a static noise suppression unit and a spatial seven emotions feature extraction unit. The structure of the aforementioned static seven emotions feature extraction module is as follows: Figure 3 As shown.

[0076] (1) The static noise suppression unit suppresses noise during the facial image acquisition process and enhances key edge information.

[0077] The static noise suppression unit described above includes a Gaussian low-pass filter for removing image noise and a Gaussian high-pass filter for enhancing image edge sharpness.

[0078] (2) The above-mentioned spatial seven emotions feature extraction unit adaptively focuses on the key facial spatial regions of the "seven emotions" and gradually extracts and integrates semantic information of different scales, thereby effectively capturing the subtle facial features that represent complex "seven emotions" states.

[0079] The spatial seven-emotion feature extraction unit here includes a channel-spatial attention module and an HR-Net module; the channel-spatial attention module, implemented based on depthwise separable convolution, adopts a dual-path structure and outputs a dimension of The channel attention weight vectors automatically extract facial regions of interest while reducing computational complexity; the HR-Net module is built on depthwise separable convolutions to extract multi-level, high-resolution image emotional features.

[0080] During training of this module, a Line layer will be added after the spatial seven emotions feature extraction unit. The extracted seven emotions features will be categorized as: joy, anger, worry, contemplation, grief, fear, and surprise. The parameters will be set using the cross-entropy loss function as follows:

[0081]

[0082] in, For input data and real labels The loss function of the static seven emotions feature extraction module is jointly determined. The total number of cases of the seven emotions. For the first A true label for each of the seven emotions; For the first The feature vectors of the seven emotions extracted from each of the seven emotion instances.

[0083] Figure 3 Q1-Q7 in the text are the seven emotion labels output during the static seven emotion extraction training, namely joy, anger, worry, thought, grief, fear, and surprise.

[0084] 3. The aforementioned cross-modal visual fusion module includes a temporal feature map construction unit, a spatial feature map construction unit, and a cross-modal graph attention fusion unit, such as... Figure 4 As shown.

[0085] (1) The above-mentioned temporal feature map construction unit transforms the temporal features of the pulse into a pulse energy distribution map.

[0086] The temporal feature map construction unit extracts video features Multiple channels undergo time-dimension Fourier transforms respectively. Among them... The input features for constructing units of temporal feature maps. For the dimension space of the feature tensor, In terms of time dimension, For spatial height, For space width, This represents the number of channels.

[0087] The process of calculating the frequency domain energy change of each pixel and concatenating the energy information of each channel can be represented as follows:

[0088]

[0089]

[0090] in, For the first Each channel in spatial coordinates Gray-level frequency characteristics at that location ( ) represents the time Fourier transform operation. ( ) represents the grayscale operation. Units for constructing time-series feature maps The Middle Video feature tensors for each channel This is the result of splicing multi-channel grayscale frequency features. For the first Each channel in spatial coordinates Gray-level frequency characteristics at that location For the first Each channel in spatial coordinates Gray-level frequency characteristics at that location For the first Each channel in spatial coordinates Gray-level frequency characteristics at that location For the first Each channel in spatial coordinates Gray-level frequency characteristics at that location for dimensional space, For the number of channels, The width of the feature map. For feature map height, For channel indexing.

[0091] Then, the K-means algorithm is used to cluster all pixels, with the center of each cluster as a graph node and the average energy value of the pixels within that cluster as the node feature. The Euclidean distance between the center of each original pixel's class and the centers of other classes is calculated as the edge weight to construct a facial pulse energy map. .

[0092] (2) The above spatial feature map construction unit represents the spatial features of the seven emotions in a structured way as a map.

[0093] The aforementioned spatial feature map construction unit uses the SLIC superpixel segmentation algorithm to process facial image features. It uses the center point of each superpixel block as a graph node, the features extracted within that superpixel block through multiple depthwise separable convolutions as node features, and the Euclidean distance between nodes (superpixel centers) as edge weights to construct a facial sentiment map. .

[0094] (3) The above-mentioned cross-modal graph attention fusion unit unifies and fuses heterogeneous information from pulse energy graph and emotion graph.

[0095] The aforementioned cross-modal graph attention fusion unit first uses a cross-modal Transformer to unify the node feature representations of the two graphs, achieved through a learnable projection matrix:

[0096]

[0097]

[0098]

[0099] in, For querying the matrix, The key matrix, For value matrices, To query the projection weight matrix, This is the transpose matrix of the seven facial expressions. The key projection weight matrix, This is the transpose matrix of facial pulse characteristics. The projected weight matrix is ​​the value. This is a characteristic of the evidence type.

[0100] Then, GAT is used to extract features from the unified graph obtained from the cross-modal VIT and then fused. The process can be represented as follows:

[0101]

[0102] in, The combined characteristics of pulse and facial expression This represents the graph attention method.

[0103] 4. The above syndrome classification module obtains the user's TCM syndrome type for depression.

[0104] The above-mentioned certificate type classification module will integrate the following features Input a multilayer perceptron (MLP) and output the probability of TCM syndrome classification for the patient.

[0105] During the training of the cross-modal visual fusion module, it is necessary to combine the syndrome classification module; load the pre-trained dynamic pulse diagnosis feature extraction module and static seven emotions feature extraction module, and freeze the parameters of all layers. Based on the "Guidelines for the Diagnosis and Treatment of Depression Using Integrated Traditional Chinese and Western Medicine," six target TCM syndromes of depression are established: liver qi stagnation, liver qi stagnation and spleen deficiency, liver qi stagnation transforming into fire, heart and spleen deficiency, kidney deficiency and liver qi stagnation, and phlegm-qi stagnation. The following cross-entropy loss function is used to complete the parameter settings:

[0106]

[0107] in, For the loss function of the certificate classification module, For input data, For real labels, For the number of tag categories, The number of training samples, , For indexing, Indicates the first Does the real label of the first instance belong to the...? kind, For the first Training weights for each category, To transform the raw scores into a probability distribution, For the first Fusion features of individual samples Perform a linear transformation.

[0108] This invention discloses a non-contact and convenient method and system for multimodal signal fusion based on TCM syndrome differentiation of depression. Through a dynamic pulse diagnosis feature extraction module, the model utilizes a spatiotemporal separation pyramid attention structure combined with multiple convolutional layers to effectively suppress motion artifact interference, improve input data quality, and adaptively generate facial ROI regions. Furthermore, it employs a Transformer architecture and channel attention mechanism to accurately extract regular cyclic pulse diagnosis signals (such as remote photoplethysmography) contained in facial videos, comprehensively capturing key temporal information. Through a static seven emotions feature extraction module, Gaussian filtering and other preprocessing techniques effectively suppress image noise and enhance the clarity of details and tissue boundaries. The core adopts the HR-Net architecture, combined with an adaptive channel-space attention module and depthwise separable convolutions, highlighting facial features related to the "seven emotions" (joy, anger, worry, thought, grief, fear, and shock), effectively filtering out irrelevant information interference. This cross-modal visual fusion module innovatively maps video pulse diagnosis and image emotional features to a unified semantic space by stitching together features from different modalities across dimensions. Based on the concepts of temporal Fourier transform and superpixel segmentation, it extracts semantically meaningful pixel-level information from both video and image data, reducing computational complexity. The cross-modal Vision Transformer (ViT) technology unifies the semantic representation of the two modalities, and Graph Attention Network (GAT) is used to deeply mine the correlations between modalities, achieving efficient and accurate feature fusion. The fused high-value feature map is then input into the syndrome classification module to accurately determine the TCM syndrome type of depression.

[0109] Example 2

[0110] This embodiment details the operation process of the present invention.

[0111] The above system is deployed on a computer terminal equipped with a camera (such as a personal computer or workstation). The specific process is as follows: Figure 5 As shown. The system workflow is as follows: First, the user's facial data is collected via the computer terminal's camera under standard indoor lighting conditions. All collected facial video and image data are with the user's permission, including a 15-second facial video at a frame rate of 30Hz and a resolution of 1280×720, as well as a 1280×720 resolution static facial image. This non-contact acquisition method effectively overcomes the inconvenience of traditional contact-based pulse diagnosis measurements, achieving unobstructed synchronous acquisition of pulse signals and facial expression signals reflecting the "seven emotions."

[0112] The acquired facial video sequences are input into the dynamic pulse diagnosis feature extraction module to extract temporal feature vectors representing subtle pulse characteristics. Simultaneously, the acquired static facial images are input into the static seven emotions feature extraction module to extract key facial feature vectors highly correlated with the "seven emotions" state.

[0113] Subsequently, the extracted pulse diagnosis time-series feature vector and seven-emotion feature vector are input into the cross-modal visual fusion module. This module first uses a pulse energy map construction method based on time Fourier transform and an emotion map construction method based on superpixel segmentation to transform the original features into semantically related graph structure data. Then, it applies cross-modal VisionTransformer (ViT) technology to unify the semantic representation space of the two graphs, and uses Graph Attention Network (GAT) to deeply explore the intrinsic correlation between modalities, achieving efficient and accurate feature fusion and outputting the fused multimodal feature vector.

[0114] The fused multimodal feature vectors are ultimately input into the syndrome classification module. This module outputs the probability distribution of the user's classification according to various preset TCM syndromes of depression (such as liver qi stagnation, liver qi stagnation and spleen deficiency, liver qi stagnation transforming into fire, heart and spleen deficiency, kidney deficiency and liver qi stagnation, and phlegm-qi stagnation). After processing, the recognition results are displayed on the screen through the graphical user interface (GUI) of a computer terminal, providing users or physicians with intuitive TCM syndrome recognition results for depression.

[0115] Example 3

[0116] This embodiment details the training process of each model.

[0117] The training steps for the dynamic pulse diagnosis feature extraction module are as follows: First, a facial video dataset is collected and labeled by multiple TCM experts into six target pulse categories (floating, deep, rapid, slow, weak, and strong). The dataset is then divided into training, validation, and test sets in a 7:2:1 ratio. During training, batches of video data are input. The data is first input into the dynamic noise suppression unit, which consists of 5 layers and a spatiotemporal attention separation pyramid to obtain noise-suppressed video data. The data is propagated forward sequentially through a dynamic noise suppression unit and a temporal pulse diagnosis feature extraction unit. The batch size is calculated as batch size × number of channels × number of frames × width × height. The data is processed through a multi-layer Transformer layer in the temporal pulse diagnosis feature extraction unit. This layer first utilizes a video slicing encoding submodule (spatiotemporal tube sampling) based on the Vision Transformer for Video concept to segment the input video into multiple spatiotemporal tubes (Tubelets, dimensions 3×5×160×80). Each spatiotemporal tube is flattened into a one-dimensional vector and transformed into a fixed-dimensional feature vector (Token) of 40 through a linear projection layer (fully connected layer).

[0118]

[0119] All tokens form an embedding matrix:

[0120]

[0121] The input is fed into a multi-layer Transformer encoder. The output features of the Transformer encoder are input into a fully connected layer (Linear layer), and the output corresponds to the classification probability distribution of six common pulse types (floating, deep, rapid, slow, weak, and strong).

[0122] The error between the model's prediction (pulse probability) and the expert annotation is calculated using the cross-entropy loss function. The parameters are set using the following formula for the cross-entropy loss function:

[0123]

[0124] An optimizer (such as Adam) combined with a learning rate scheduling strategy (such as Cosine Annealing with Warm Restarts) is used to update model parameters. Model performance is evaluated on the validation set using metrics such as accuracy, F1 score, and precision. Optimization is achieved by adjusting hyperparameters (such as learning rate, batch size, and model depth). Finally, model performance is evaluated on an independent test set.

[0125] The training steps for the static seven emotions feature extraction module are as follows: First, a dataset of static facial images is collected and labeled by multiple experts as seven target "seven emotions" states (joy, anger, worry, contemplation, grief, fear, and surprise). Before training, the image data undergoes data augmentation processing, including random rotation and scaling within the range of 0 to 180 degrees. The augmented dataset is then divided into training, validation, and test sets in a 7:2:1 ratio. During training, batches of image data are input. The data is then passed forward through the image noise suppression module and the seven emotions feature extraction module. The cross-entropy loss function is used to calculate the error between the model's predicted "seven emotions" probabilities and the expert annotations.

[0126]

[0127] An optimizer (such as Adam) combined with a learning rate scheduling strategy (such as Cosine Annealing with Warm Restarts) is used to update model parameters. Model performance is evaluated on the validation set using metrics such as accuracy, F1 score, and precision. Optimization is achieved by adjusting hyperparameters (such as learning rate, kernel size, and attention module parameters). The final model performance is evaluated on an independent test set.

[0128] The training of the overall system, including a cross-modal visual fusion module, aims to target the TCM syndromes of depression. Facial videos and corresponding static facial images of the same subject are collected. Video data must meet the requirements of the dynamic pulse diagnosis feature extraction module, and image data must meet the requirements of the static seven emotions feature extraction module. Based on the video and image information, multiple TCM experts comprehensively labeled the data into six target TCM syndromes of depression: Liver Qi stagnation, Liver Qi stagnation and Spleen deficiency, Liver Qi stagnation transforming into fire, Heart and Spleen deficiency, Kidney deficiency and Liver Qi stagnation, and Phlegm-Qi stagnation.

[0129] Load the pre-trained dynamic pulse diagnosis feature extraction module and static seven emotions feature extraction module, and freeze the parameters of all remaining layers.

[0130] The dynamic pulse diagnosis feature extraction module, after processing facial video, outputs a pulse feature vector; the static seven emotions feature extraction module, after processing facial image, outputs a seven emotions feature vector. Depending on the input requirements of the fusion module, the shape of these feature vectors may be adjusted using an adaptive pooling layer. The adjusted pulse and seven emotions features are then input into the cross-modal visual fusion module, where the temporal feature map construction unit processes the extracted video features. Multiple channels c are subjected to Fourier transforms in the time dimension to calculate the frequency domain energy change of each pixel. The energy information of each channel is then concatenated. The process can be represented as follows:

[0131]

[0132]

[0133] Facial pulse energy map constructed using the K-means algorithm. .

[0134] The Seven Emotions Map network unit uses the SLIC superpixel segmentation algorithm to process facial image features. It uses the center point of each superpixel block as a graph node, the features extracted within that superpixel block through multiple depthwise separable convolutions as node features, and the Euclidean distance between nodes (superpixel centers) as edge weights to construct a facial emotion map. .

[0135] The cross-modal graph attention fusion unit first uses a cross-modal Transformer to unify the node feature representations of the two graphs, achieved through a learnable projection matrix:

[0136]

[0137]

[0138]

[0139] Then, GAT is used to extract features from the unified graph obtained from the cross-modal VIT and then fused. The process can be represented as follows:

[0140]

[0141] The fused cross-modal visual feature vector is obtained. This feature vector is then input into the syndrome classification module, which outputs the classification probability distribution corresponding to the seven target syndromes.

[0142] During training, the input data batch size is set to 30 (e.g., batch_size=30). The error between the model prediction (syndrome probability) and the expert comprehensive annotation is calculated using the cross-entropy loss function.

[0143]

[0144] An optimizer (such as Adam) combined with a learning rate scheduling strategy (such as Cosine Annealing with Warm Restarts) is used to update the parameters of the cross-modal visual fusion module and the syndrome classification module (while the parameters of the video pulse diagnosis and image seven emotions modules are kept frozen).

[0145] The dataset was divided into training, validation, and test sets in a 7:2:1 ratio. On the validation set, the overall model performance was evaluated using metrics such as accuracy, F1 score, and precision. Optimization was achieved by adjusting hyperparameters of the cross-modal visual fusion module and the type classification module, such as the number of GAT heads, attention dimension, number and width of MLP layers, and learning rate.

[0146] Example 4

[0147] This embodiment conducts an experimental evaluation of the depression syndrome detection method and system based on multimodal visual feature fusion proposed in this invention. The evaluation indicators are accuracy, recall, F1 score, and precision.

[0148] The results of the dynamic pulse diagnosis feature extraction module are shown in Table 1:

[0149] Table 1. Comparison of evaluation metrics for dynamic pulse diagnosis feature extraction module under different algorithms

[0150]

[0151] The dynamic pulse diagnosis feature extraction module proposed in this invention achieves a precision of 0.7742, significantly higher than Phynet's 0.7179. Furthermore, the invention's F1 score is 0.7361, recall is 0.7464, surpassing other methods, and accuracy is 0.8936. These results clearly demonstrate that, compared to other methods, this invention exhibits superior performance and robustness in dynamic pulse detection tasks in facial videos, enabling more accurate pulse recognition even in complex data environments.

[0152] The results of the static seven emotions feature extraction module are shown in Table 2:

[0153] Table 2 Comparison of evaluation metrics for static seven emotions feature extraction module under different algorithms

[0154]

[0155] The static seven emotions feature extraction module proposed in this invention achieves a precision of 0.9063, significantly higher than the 0.8825 of 3DCNN. Furthermore, the F1 score of this invention is 0.9352, which is higher than the compared methods. These results demonstrate that, compared to other methods, this invention exhibits superior performance and robustness in static seven emotions recognition tasks.

[0156] The effects of combining the dynamic pulse diagnosis feature extraction module, the static seven emotions feature extraction module, and the cross-modal visual fusion module are shown in Table 3.

[0157] Table 3 Comprehensive Evaluation Indicators of the Invention

[0158]

[0159] The dynamic pulse diagnosis feature extraction module and the static seven emotions feature extraction module proposed in this invention, combined with the cross-modal visual fusion module, have an accuracy of 0.8372, a precision of 0.8596, and an F1 score of 0.8372, all above 0.8. The model proposed in this invention has excellent anti-interference performance and accuracy.

Claims

1. A method for detecting symptoms of depression based on multimodal visual feature fusion, characterized in that, Includes the following steps: Acquire multimodal visual data of the user containing optical sensing information, wherein the multimodal visual data includes dynamic facial visual data and static facial visual data; The facial dynamic visual data is input into the dynamic pulse diagnosis feature extraction module, and then passed through the dynamic noise suppression unit and the temporal pulse diagnosis feature extraction unit of the dynamic pulse diagnosis feature extraction module in sequence to output the pulse diagnosis feature vector. The facial static visual data is input into the static seven emotions feature extraction module, and then passed through the static noise suppression unit and the spatial seven emotions feature extraction unit of the static seven emotions feature extraction module in sequence to output the seven emotions feature vector; The pulse diagnosis feature vector and the seven emotions feature vector are input into the cross-modal visual fusion module. The feature is fused through the temporal feature map construction unit, spatial feature map construction unit and cross-modal map attention fusion unit of the cross-modal visual fusion module, and the fused feature vector is output. The temporal feature map construction unit of the cross-modal visual fusion module first converts the RGB channel features of the pulse diagnosis feature vector into a single-channel grayscale image using a weighted average formula. Then, it performs time-dimensional frequency domain transformation on multiple feature channels of the pulse diagnosis feature vector, calculates the frequency domain energy change of each pixel, and splices the multi-channel energy information. Finally, it constructs a pulse diagnosis feature map using a clustering algorithm. The spatial feature map construction unit of the cross-modal visual fusion module uses a superpixel segmentation algorithm to segment the facial spatial features corresponding to the seven emotion feature vectors to obtain multiple superpixel blocks. The center point of each superpixel block is used as a graph node, the features extracted by multi-layer depthwise separable convolution within the superpixel block are used as the node features of the corresponding node, and the feature spatial distance between the nodes is used as the edge weight to construct the seven emotion feature map. The cross-modal graph attention fusion unit of the cross-modal visual fusion module first uses the cross-modal VisionTransformer to unify the node feature representation of the pulse diagnosis feature map and the seven emotions feature map, and then extracts and fuses the cross-modal features through the graph attention mechanism. The fused feature vector is input into the syndrome classification module, which outputs the user's TCM syndrome type of depression. The pulse diagnosis feature vector is a fixed-dimensional numerical vector containing the temporal characteristics of the pulse diagnosis signal obtained by remote optical volume plethysmography; the seven emotions feature vector is a fixed-dimensional numerical vector containing the spatial characteristics of the seven emotions; and the fusion feature vector is a fixed-dimensional numerical vector containing both the temporal characteristics of the pulse diagnosis signal and the spatial characteristics of the seven emotions.

2. The method for detecting depressive symptoms based on multimodal visual feature fusion according to claim 1, characterized in that, During training, the dynamic pulse diagnosis feature extraction module classifies the pulse diagnosis features in the facial dynamic visual data into at least two categories according to the TCM pulse pattern categories, and optimizes the model parameters of the dynamic pulse diagnosis feature extraction module using a cross-entropy loss function or a temporal classification loss function. During training, the static seven emotions feature extraction module classifies the seven emotions features in the facial static visual data into at least two categories according to the TCM seven emotions categories, and optimizes the model parameters of the static seven emotions feature extraction module using a cross-entropy loss function. During training, the parameters of the dynamic pulse diagnosis feature extraction module and the static seven emotions feature extraction module are frozen, and the model parameters of the cross-modal visual fusion module are optimized using a classification loss function.

3. The method for detecting depressive symptoms based on multimodal visual feature fusion according to claim 1, characterized in that, The dynamic noise suppression unit includes a convolutional layer and a multi-scale spatiotemporal attention model. The multi-scale spatiotemporal attention model contains at least two spatiotemporal convolutional layers with different scales and dilation parameters. The dilation parameters of the spatiotemporal convolutional layers of the multi-scale spatiotemporal attention model are set according to a preset gradient. After the features of different scales are mapped to the same dimension through adaptive pooling, feature integration is achieved by element-level fusion or splicing fusion.

4. The method for detecting depressive symptoms based on multimodal visual feature fusion according to claim 1, characterized in that, The temporal pulse diagnosis feature extraction unit includes a channel attention component and a temporal feature encoding component. The channel attention component enhances the feature weights of the stable channels of the pulse diagnosis signal through multiple sets of spatiotemporal convolution and pooling operations. The temporal feature encoding component extracts the pulse diagnosis signal using spatiotemporal slice encoding and a multilayer temporal encoder. In the spatiotemporal slicing encoding operation of the temporal feature encoding component, the spatiotemporal tube sampling technique of Vision Transformer for Video is used to segment the facial dynamic visual data into multiple spatiotemporal units. Each spatiotemporal unit is flattened and transformed into a fixed-dimensional feature vector through a linear projection layer. All feature vectors constitute an embedding matrix. The embedding matrix is ​​concatenated with the position encoding and pulse category encoding and then input into a multilayer temporal encoder. The position encoding is generated by a preset periodic function. The preset periodic function is a spatiotemporal separation encoding function, which includes spatial position encoding and temporal position encoding. The spatial position encoding is generated by a cosine function, and the temporal position encoding is generated by a sine function.

5. The method for detecting depressive symptoms based on multimodal visual feature fusion according to claim 1, characterized in that, The static noise suppression unit includes a noise filtering component and an edge enhancement component; the noise filtering component of the static noise suppression unit uses at least one of Gaussian filtering, bilateral filtering, and nonlocal mean filtering; the edge enhancement component uses at least one of high-pass filtering, Laplacian enhancement, and contrast-limited adaptive histogram equalization. When the noise filtering component is a Gaussian low-pass filter, the standard deviation of the Gaussian low-pass filter is adaptively adjusted according to the gray-level variance of the facial static visual data: when the gray-level variance is greater than 50, the standard deviation of the Gaussian low-pass filter is 1.5-2.0; when the gray-level variance is less than or equal to 50, the standard deviation of the Gaussian low-pass filter is 0.5-1.0; when the edge enhancement component is a Gaussian high-pass filter, the standard deviation of the Gaussian high-pass filter is 1.5 times the standard deviation of the Gaussian low-pass filter.

6. The method for detecting depressive symptoms based on multimodal visual feature fusion according to claim 1, characterized in that, The spatial seven emotions feature extraction unit includes a channel-spatial attention component and a high-resolution feature extraction component. The channel-spatial attention component filters dual-path features based on depthwise separable convolution, and the high-resolution feature extraction component extracts and fuses facial spatial features of different scales through multi-resolution branches in parallel. The high-resolution feature extraction component includes at least two parallel resolution branches. Each branch extracts features through multi-layer convolution, and then uses a bidirectional fusion strategy of downsampling projection and upsampling supplementation to fuse the features extracted by each parallel resolution branch. The downsampling projection achieves feature dimension matching through stride convolution or pooling operations, and the upsampling supplementation restores the feature size through transposed convolution or interpolation operations. During fusion, weights are dynamically allocated based on feature information entropy, and the feature branch with higher information entropy has a higher weight ratio.

7. The method for detecting depressive symptoms based on multimodal visual feature fusion according to claim 1, characterized in that, The clustering algorithm includes at least one of the following: K-means algorithm, DBSCAN algorithm, or hierarchical clustering algorithm.

8. The method for detecting depressive symptoms based on multimodal visual feature fusion according to claim 1, characterized in that, The time-series feature map construction unit uses Fourier transform or wavelet transform for frequency domain transformation, and the frequency range of the frequency domain transformation is 0.1-10Hz. The cross-modal graph attention fusion unit adopts a multi-attention head mechanism. The feature dimension of a single attention head is a single dimension obtained by equally dividing the total dimension of the fused features according to the number of attention heads. When calculating the edge weight, the final edge weight is obtained by combining the feature space distance and the semantic similarity between modalities through weighted summation. The feature space distance is selected from Euclidean distance or Manhattan distance, and the intermodal semantic similarity is selected from cosine similarity or Pearson correlation coefficient. The superpixel segmentation algorithm includes at least one of the SLIC algorithm, SEEDS algorithm, or LSC algorithm.

9. The method for detecting depressive symptoms based on multimodal visual feature fusion according to claim 1, characterized in that, The syndrome classification module adopts a multilayer perceptron or a deep learning classification network; the output TCM syndrome of depression includes at least one of the following: liver qi stagnation, liver qi stagnation and spleen deficiency, liver qi stagnation transforming into fire, heart and spleen deficiency, kidney deficiency and liver qi stagnation, and phlegm and qi stagnation. In the classification network of the certificate type classification module, a feature attention gating unit is added between the input layer and the first classification layer: the importance weights of each dimension of the fused feature vector are calculated through the activation function.

10. A depression syndrome detection system based on multimodal visual feature fusion, characterized in that, The system includes a dynamic pulse diagnosis feature extraction module, a static seven emotions feature extraction module, a cross-modal visual fusion module, and a syndrome classification module; It also includes a visual data acquisition module, which is used to acquire multimodal visual data of the user containing optical sensing information; When the system is running, it executes the method for detecting depression symptoms based on multimodal visual feature fusion as described in any one of claims 1 to 9.

Images