Video depression state evaluation method and system based on multi-view frequency domain feature fusion

Abstract

The application discloses a video depression state evaluation method based on multi-view frequency domain feature fusion, which comprises the following steps: collecting facial video data of a target object and performing pretreatment, and outputting a standardized facial image sequence; converting the facial image sequence into three parallel views containing different physical characteristics, and explicitly decoupling depression-related micro-texture and instantaneous dynamics from the original video; constructing three independent parallel backbone networks to perform parallel feature coding on the three views respectively, deeply fusing video-level feature vectors of the three independent modes, and mapping to a continuous depression degree score space; training and optimizing the model by using a training data set, and realizing the classification of the final state by predicting the depression degree score. The application also discloses a video depression state evaluation system based on multi-view frequency domain feature fusion. The application can directly filter out high-energy low-frequency interference from the original data level and accurately extract micro-expression high-frequency signals.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical technology, and in particular to a video-based method and system for assessing depressive states based on multi-view frequency domain feature fusion. Background Technology

[0002] With the rapid development of artificial intelligence and affective computing technologies, automated depression assessment based on computer vision has become a research hotspot in the field of intelligent healthcare. This technology mainly uses cameras to collect facial videos of subjects and utilizes deep learning models to analyze behavioral characteristics such as facial expressions, head posture, and eye movements, thereby achieving an objective quantitative assessment of the degree of depression.

[0003] However, such methods have the following technical defects: (1) The original video is easily affected by ambient lighting and smooth backgrounds, and the low signal-to-noise ratio leads to the failure of micro-expression capture: Facial videos contain a large number of lighting components that change with the environment and smooth skin areas, which are low-frequency components in the frequency domain, while depression-related micro-expression changes (such as trembling of the corners of the eyes and twitching of the corners of the mouth) are often high-frequency signals with low energy. The strong low-frequency information makes the signal-to-noise ratio of high-frequency micro-expression signals extremely low, making it difficult for existing models to accurately capture discriminative pathological features under complex lighting or different skin color backgrounds; (2) It is difficult to effectively decouple and capture high-frequency dynamic changes in time sequence: Existing technologies (optical flow method, differential method, 3D convolution) tend to capture macroscopic low-frequency movements such as head rotation. However, the typical pathological features of depression are high-frequency tremors or pauses of facial muscles. Existing methods struggle to explicitly decouple such microscopic high-frequency dynamics from macroscopic head movements, leading to the easy obscuring of crucial diagnostic pathological temporal features by irrelevant head posture movements, severely impacting the accuracy of assessment; (3) A single observation perspective makes it difficult to balance the features of "global semantics" and "local details": Existing technologies typically rely solely on the original RGB video as a single input source. In the RGB space, "explicit" semantic features (such as facial contours, facial features, and large facial expressions) often dominate, while "implicit" pathological features (such as subtle texture changes and instantaneous muscle tremors) have extremely weak signals. In single-viewpoint deep learning networks, strong semantic features easily inhibit the learning of weak pathological features, causing the model to ignore subtle cues that truly reflect depression.

[0004] Therefore, there is an urgent need to provide a novel video-based depressive state assessment method and system based on multi-view frequency domain feature fusion to solve the above problems. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a video depression state assessment method and system based on multi-view frequency domain feature fusion, which can directly filter out high-energy low-frequency interference caused by changes in ambient lighting and smooth skin background from the raw data level, and accurately extract high-frequency signals of micro-expressions.

[0006] To address the aforementioned technical problems, one technical solution adopted by this invention is: to provide a video-based depressive state assessment method based on multi-view frequency domain feature fusion, comprising the following steps:

[0007] S1: Acquire facial video data of the target object, perform preprocessing, and output a standardized facial image sequence;

[0008] S2: The facial image sequence output in step S1 is converted into three parallel views containing different physical properties, including an RGB appearance view, a 2D high-frequency spatial view, and a 3D high-frequency spatial view, to explicitly decouple the depression-related fine textures and instantaneous dynamics from the original video.

[0009] S3: By constructing three independent parallel backbone networks, the three views generated in step S2 are encoded in parallel. The video-level feature vectors of the three independent modalities are then deeply fused and mapped to a continuous depression score space.

[0010] S4: Train the model built in step S3 using the training dataset, and guide the optimization of the model through the loss function to achieve a quantitative regression assessment of the final state by predicting the depression level score.

[0011] In a preferred embodiment of the present invention, step S1, the step of preprocessing the acquired facial video data, includes:

[0012] S101: Face detection: The high-precision face detection framework Facer based on deep convolutional neural networks is used to process video frames. A batch processing method is used to read a single video frame sequence, and dynamic bounding box expansion is used to crop the image. The size of the cropped facial image blocks of different sizes is normalized to construct a standardized facial image sequence.

[0013] S102: Image Denoising and Standardized Output: Gaussian smoothing filtering is applied to each frame of the facial image sequence to smooth isolated pixel noise while preserving the main facial texture structure. The final output consists of an array or tensor of at least 30 geometrically aligned and denoised RGB facial images, which serves as the standard input for subsequent multi-view frequency domain feature generation.

[0014] In a preferred embodiment of the present invention, in step S2, an RGB appearance view is directly constructed using the standardized RGB facial image sequence output in step S1, denoted as... As a baseline data stream It fully preserves the global color distribution, facial topology, and macroscopic facial expression changes of the subjects' faces.

[0015] In a preferred embodiment of the present invention, in step S2, each frame of the facial image sequence output in step S1 is independently subjected to frequency domain filtering to filter out low-frequency components composed of smooth skin and ambient lighting, extract fine texture features, and construct a 2D high-frequency spatial view. Specific steps include:

[0016] S211: Image grayscale conversion: Convert the t-th frame RGB image output in step S1 to grayscale. Convert to single-channel grayscale image The mapping is performed using a standard luminance-weighted formula:

[0017] in, , , Representing pixels The pixel values ​​of the red, green, and blue color channels are converted to grayscale, resulting in image data from... Convert a three-channel RGB image to A single-channel grayscale image;

[0018] S212: Two-dimensional Discrete Fourier Transform: Processing a single-frame grayscale image Transform it from the spatial domain Mapping to the frequency domain For size The transformation formula for the image is:

[0019]

[0020] in, Represents the complex spectral coefficients in the frequency domain. The imaginary unit, and These are the frequency variables in the horizontal and vertical directions, respectively;

[0021] S213: Constructing a high-pass filter: First, calculate any point in the frequency domain. to the frequency domain center distance :

[0022]

[0023] in, , ;

[0024] Subsequently, the filter mask is defined. Set the cutoff frequency radius as :

[0025]

[0026] The original spectrum is multiplied point by point with the filter to obtain the filtered spectrum. :

[0027]

[0028] S214: Two-dimensional Discrete Fourier Inverse Transform: The filtered spectrum... Restore to the spatial domain to generate an edge-enhanced image. :

[0029]

[0030] in This represents taking the real part of a complex number, and finally reconstructing the processed image of t frames to obtain a 2D high-frequency spatial view. .

[0031] In a preferred embodiment of the present invention, in step S2, the entire video sequence is treated as a spatiotemporal tensor, and a joint spatiotemporal high-pass filter is performed using three-dimensional frequency domain transformation to filter out low-frequency components in both the spatial and temporal dimensions, thereby constructing a 3D high-frequency spatial view. Specific steps include:

[0032] S221: Three-dimensional Discrete Fourier Transform: Treating video sequences as three-dimensional signals Perform a joint spatiotemporal transformation to obtain data containing spatial frequencies. and time frequency Three-dimensional spectrum :

[0033]

[0034] in, For time indexing, For time frequency variables;

[0035] S222: Spatiotemporal High-Pass Filtering: First, calculate the value at any point in the three-dimensional frequency domain. to the frequency domain center distance :

[0036]

[0037] in, , , Let the frequency domain center coordinates be used, and set the cutoff frequency radius as... Constructing a filter mask:

[0038]

[0039] The mask is used to filter the three-dimensional spectrum to obtain the filtered spectrum. :

[0040]

[0041] in, It is a three-dimensional high-pass filter;

[0042] S223: 3D Discrete Fourier Inverse Transform: Restores the filtered 3D spectrum back to the spatiotemporal domain, resulting in a video sequence containing only microscopic dynamic changes. :

[0043]

[0044] Final output The view effectively decouples macroscopic head movements from the smooth background, retaining only the dynamic textures of micro-expressions with high-frequency characteristics.

[0045] In a preferred embodiment of the present invention, in step S3, the parallel backbone network uses ResNet-50 as the basic backbone network. Each branch contains four consecutive convolutional stages. As the network depth increases, the spatial resolution of the feature map decreases step by step, and the number of channels increases step by step. Finally, the fourth stage convolution outputs high-dimensional deep features.

[0046] Furthermore, a bidirectional interaction mechanism was constructed in the intermediate layer of the parallel backbone network, which includes two processes: enhancement of the frequency domain view by the RGB view, and enhancement of the RGB view by the frequency domain view.

[0047] Enhancement of the frequency domain view by the RGB view: This process utilizes the clear facial topology in the RGB view to calibrate the 2D high-frequency spatial view and the 3D high-frequency spatial view, and suppresses the pseudo-high-frequency response caused by background noise.

[0048] Enhancement of RGB view by frequency domain view: This process introduces high-frequency prior information in the frequency domain and guides the RGB branch to focus on local subtle changes with diagnostic value through a channel attention mechanism, reducing the model's dependence on the subject's facial identity attributes.

[0049] In a preferred embodiment of the present invention, in step S3, global average pooling is performed on the output features of the three independent parallel backbone networks, transforming the original video tensor data into three sets of time-series feature vectors: RGB frame-level feature sequence, 2D frequency domain frame-level feature sequence, and 3D frequency domain frame-level feature sequence. Then, a recurrent neural network is used to perform temporal modeling on the three sets of time-series feature vectors, aggregating the video-level global representation. After temporal aggregation, three-dimensional video-level feature vectors are output: RGB video feature vector, 2D frequency domain video feature vector, and 3D frequency domain video feature vector. Finally, the three independent modal video-level feature vectors obtained by aggregation are deeply fused and mapped to a continuous depression score space.

[0050] In a preferred embodiment of the present invention, in step S4, the root mean square error (RMSE) is used as the target loss function to guide the end-to-end training of the deep neural network, thereby achieving accurate regression prediction of depression scores.

[0051] To address the aforementioned technical problems, another technical solution adopted by this invention is: providing a video-based depressive state assessment system based on multi-view frequency domain feature fusion, comprising:

[0052] The standardized facial image sequence output module is used to acquire facial video data of the target object, perform preprocessing, and output a standardized facial image sequence.

[0053] The multi-view frequency domain feature generation module is used to convert the facial image sequence output by the standardized facial image sequence output module into three parallel views containing different physical properties, including an RGB appearance view, a 2D high-frequency spatial view, and a 3D high-frequency spatial view, to explicitly decouple depression-related fine textures and instantaneous dynamics from the original video.

[0054] The model building and view processing module is used to construct three independent parallel backbone networks to perform parallel feature encoding on the three views generated by the multi-view frequency domain feature generation module, deeply fuse the video-level feature vectors of the three independent modalities, and map them to a continuous depression score space.

[0055] The model optimization and output module is used to train the model built by the model building and view processing module using the training dataset, and guides the optimization of the model through the loss function, and achieves quantitative regression assessment of the final state by predicting the depression level score.

[0056] The beneficial effects of this invention are:

[0057] (1) In view of the technical problem that the original video features in the prior art are easily affected by high-energy low-frequency components such as ambient light, smooth background and subject identity information, resulting in low signal-to-noise ratio of high-frequency micro-expression signals related to depression, which in turn leads to low accuracy of the model in assessing the depressive state in complex environments and poor anti-interference ability, the present invention uses two-dimensional spatial frequency domain filtering, which can directly filter out high-energy low-frequency interference caused by changes in ambient light and smooth skin background from the original data level, accurately extract high-frequency micro-expression signals, effectively overcome the defects of the prior art that are easily affected by environmental noise, and enable the model to effectively capture depressive cues;

[0058] (2) In view of the technical problem that existing spatiotemporal analysis technology is difficult to explicitly decouple the micro high-frequency tremors (or pauses) of facial muscles from the macro low-frequency movements of the head, which makes it easy for pathological temporal features with key diagnostic value to be masked by large irrelevant head movements, thus causing misjudgment of the patient's depressive state, this invention introduces three-dimensional discrete Fourier transform and spatiotemporal high-pass filtering to explicitly separate the high-frequency micro tremors of facial muscles from the macro low-frequency posture movements of the head, effectively preventing key micro-expression dynamic features from being masked by large body movements;

[0059] (3) In view of the technical problem that existing technologies usually rely on a single observation perspective or simple back-end splicing, it is difficult to achieve deep interaction and complementarity between static appearance and frequency domain details in the feature encoding stage. This results in the lack of joint calibration and constraints of multiple perspectives when facing various manifestations of depressive symptoms, which leads to weak generalization ability and insufficient robustness in cross-scene or cross-individual regression prediction tasks. This invention establishes a bidirectional cross-modal attention interaction mechanism. Through the synergistic effect of spatial attention and channel attention, it realizes deep interaction between RGB view and frequency domain high-frequency view, thereby significantly enhancing the model's comprehensive ability to capture complex depressive pathological features. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating a preferred embodiment of the video depression state assessment method based on multi-view frequency domain feature fusion of the present invention;

[0061] Figure 2 This is a flowchart showing the specific steps of step S2;

[0062] Figure 3 This is a flowchart detailing the steps in step S3;

[0063] Figure 4 This is a schematic diagram of the structure of the parallel backbone network;

[0064] Figure 5 This is a block diagram of a video depression assessment system based on multi-view frequency domain feature fusion. Detailed Implementation

[0065] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0066] Please see Figure 1 The embodiments of the present invention include:

[0067] A video-based depressive state assessment method based on multi-view frequency domain feature fusion includes the following steps:

[0068] S1: Acquire facial video data of the target object, perform preprocessing, and output a standardized facial image sequence;

[0069] This step involves acquiring a normalized facial image sequence through video capture and a series of image preprocessing techniques, and reducing noise interference introduced by environmental, equipment, and individual behavioral differences. Specific steps include:

[0070] S101: Facial Video Data Acquisition

[0071] Use a camera device with a resolution of at least 1080p (1920×1080 pixels) (such as Logitech C920 or Huawei software-defined camera) to acquire facial video data of the target object, extract facial image frames and form a facial image sequence.

[0072] In one specific embodiment, a 4K (3840×2160 pixels) resolution camera was used with a frame rate set to 30fps to ensure that dynamic features related to micro-expressions, eye muscle activity, and subtle changes in the corners of the mouth could be clearly captured.

[0073] The capture time is 30 to 375 seconds to ensure that enough video data containing complete facial expressions and dynamics can be captured.

[0074] To ensure the standardization and consistency of facial video data during the acquisition process, constraints regarding frontal viewpoint, horizontal posture, and shooting distance must be met, including:

[0075] For a frontal view, the angle between the center line of the subject's face (a virtual line formed by the center points of the eyes and the tip of the nose) and the camera's optical axis should be less than 15 degrees.

[0076] The horizontal orientation requirement is that the deviation between the horizontal line connecting the eyes of the target object and the horizontal axis (x-axis) of the pixel coordinate system of the acquired video image frame should be less than 5 degrees.

[0077] The shooting distance requirement is that the distance between the target object and the camera should be maintained between 0.5 meters and 1.0 meter to ensure that the size ratio of the face in the picture is relatively stable.

[0078] In one specific embodiment, the aforementioned angles and distances during the acquisition process can be quantified in real time using a face pose estimation algorithm (such as MediaPipe Face Mesh) to guide the target object to make adjustments and to filter video clips that meet the standards.

[0079] Based on the collected facial video data, facial image frames are extracted through equal-interval sampling and formed into an ordered facial image sequence containing multiple frames of facial images.

[0080] The number of facial image frames is determined by an optimal trade-off between the minimum duration required to capture complete micro-expression dynamics and the computational complexity of subsequent models: too few frames (less than 30 frames) cannot cover statistically significant dynamic patterns, while too many frames introduce redundant information and significantly increase the computational burden. Specifically, 64 facial images are extracted from the acquired facial video data at equal intervals (ten frames apart) to form a facial image sequence.

[0081] S102: Face Detection

[0082] (1) Construction and initialization of the detection model: In this embodiment, the high-precision face detection framework Facer based on deep convolutional neural networks is used to process video frames. Specifically, an end-to-end face detection model based on the RetinaFace architecture is constructed, and the ResNet-50 deep convolutional neural network is used as the feature extractor. This network contains four convolutional stages (Stage 1-Stage 4), and residual connections are used to solve the degradation problem of deep networks.

[0083] (2) Dual-stream processing strategy for video frames: In order to obtain high-quality facial images while ensuring detection speed, this invention adopts a dual-stream processing strategy of "low-resolution detection - high-resolution cropping". First, the video frames are traversed, and the original high-resolution image (size is...) is processed. Perform a 2x downsampling (Resize) to generate a low-resolution image sequence (size: This sequence is used solely as input to the neural network for rapid inference, significantly reducing memory usage and computation time. Simultaneously, the corresponding original high-resolution image sequence is retained in memory for subsequent high-resolution facial region cropping.

[0084] (3) Forward inference and bounding box selection: A single video frame sequence is read using batch processing. The image data is converted from a NumPy array to a PyTorch tensor, and the dimension order is adjusted from (Batch,Height,Width,Channel) to a (Batch,Channel,Height,Width) format suitable for convolutional network input. The tensor obtained above is then input into the detection model for forward propagation, and the output includes the predicted face bounding box. ), confidence level, facial key points.

[0085] (4) Dynamic bounding box expansion and image cropping: In order to ensure that the cropped face image contains the complete facial contour and necessary contextual information, and to adapt to the square input requirements of the subsequent model, the following square dynamic expansion algorithm is adopted:

[0086] The width of the mapped bounding box is calculated based on the face bounding box output by the face detection model. and height .

[0087] Select the larger value (width + height) as the benchmark and multiply it by a preset expansion factor. (In this embodiment) ), to obtain the target trimming side length :

[0088]

[0089] Calculate the geometric center point of the original bounding box. ):

[0090]

[0091] in, These are the coordinates of the face bounding box.

[0092] Based on the center point ( ) and new side length Calculate the coordinates of the expanded top-left corner ( :

[0093]

[0094] Perform a validity check on the generated coordinates to ensure... and No more than the width of the original image. Similarly.

[0095] (5) High-definition cropping and size normalization: using the calculated extended coordinates ( ) and side length Cropping is performed on the original high-resolution image sequence. This ensures that even if detection is performed on low-resolution images, the final face images still retain the high-definition texture details of the original video. The cropped facial image patches of different sizes are then uniformly scaled to a standard size using bilinear interpolation. Pixels are used to construct a standardized sequence of facial images, which serve as input for subsequent feature extraction modules.

[0096] S103: Image Denoising and Normalization Output

[0097] Although step S102 has completed the extraction and cropping of the face region, the original video frame still inevitably contains random noise generated by the photothermal noise of the camera sensor. These noises appear as non-pathological high-frequency components in frequency domain analysis, which can easily cause signal confusion and interference in the subsequent step S2 for extracting the true micro-expression high-frequency signals.

[0098] Therefore, in this embodiment, a kernel size of is applied to each frame of the image sequence. The Gaussian smoothing filter process preserves the main facial texture structure while smoothing isolated pixel noise. The final output is a normalized facial image sequence, which is an array (or tensor) of 64 geometrically aligned and denoised RGB facial images, serving as the standard input for subsequent multi-view frequency domain feature generation.

[0099] S2: The facial image sequence output from step S1 is transformed into three parallel views containing different physical properties, including an RGB appearance view, a 2D high-frequency spatial view, and a 3D high-frequency spatial view. By introducing frequency domain analysis, starting from the underlying logic of signal processing, the subtle textures and instantaneous dynamics related to depression are explicitly decoupled from the original video; combined with Figure 2 The specific steps include:

[0100] S201: Constructing an RGB View

[0101] In this step, the standardized RGB facial image sequence output from step S1 is directly used to construct an RGB appearance view, denoted as... Specifically, the view is defined as a four-dimensional real tensor. The timing length Number of channels Spatial resolution As a baseline data stream The system fully preserves the global color distribution, facial topology, and macroscopic facial expression changes of the subjects. In subsequent multi-stream deep neural networks, it provides necessary anatomical structural references for the frequency domain branch, preventing the loss of facial spatial semantic information due to frequency domain filtering during the later feature extraction process.

[0102] S202: Constructing a 2D high-frequency spatial view

[0103] This step involves independently performing frequency domain filtering on each frame of the video sequence to remove low-frequency components composed of smooth skin and ambient lighting, and to extract fine texture features. Specifically, it includes the following steps:

[0104] (1) Image grayscale conversion: In order to focus on the brightness changes caused by facial texture structure and reduce the computational dimension, the t-th frame RGB image output in step S1 is first converted into grayscale. Convert to single-channel grayscale image Mapping is performed using a standard luminance-weighted formula:

[0105] in, , , Representing pixels The pixel values ​​of the red, green, and blue color channels are converted to grayscale, resulting in image data from... Convert a three-channel RGB image to A single-channel grayscale image;

[0106] (2) Two-dimensional Discrete Fourier Transform (2D-DFT): For the processed single-frame grayscale image Transform it from the spatial domain Mapping to the frequency domain For size The transformation formula for the image is:

[0107]

[0108] in, Represents the complex spectral coefficients in the frequency domain. The imaginary unit, and These are the frequency variables in the horizontal and vertical directions, respectively.

[0109] (3) Constructing a high-pass filter: In order to filter out low-frequency background, an ideal high-pass filter transfer function based on Euclidean distance is constructed. First, calculate any point in the frequency domain. to the frequency domain center distance :

[0110] in, , .

[0111] Subsequently, the filter mask is defined. Set the cutoff frequency radius as :

[0112]

[0113] The original spectrum is multiplied point by point with the filter to obtain the filtered spectrum. :

[0114]

[0115] (4) Two-dimensional discrete inverse Fourier transform (2D-IDFT): The filtered spectrum Restore to the spatial domain to generate an edge-enhanced image. :

[0116]

[0117] in This represents taking the real part of a complex number. Finally, the processed images of frames t are reconstructed to obtain a 2D high-frequency spatial view. Specifically, the view is defined as a four-dimensional real tensor. The timing length Spatial resolution .

[0118] S203: Constructing a 3D high-frequency spatial view

[0119] This step treats the entire video sequence as a spatiotemporal tensor and performs joint spatiotemporal high-pass filtering using a three-dimensional frequency domain transformation. This operation aims to simultaneously filter out low-frequency components in both the spatial and temporal dimensions (such as smoothing the background and skin tone) to address the problem that micro-expressions are easily masked by head posture movements.

[0120] (1) Three-dimensional discrete Fourier transform (3D-DFT): Treating the video sequence as a three-dimensional signal Perform a joint spatiotemporal transformation to obtain data containing spatial frequencies. and time frequency Three-dimensional spectrum :

[0121]

[0122] Where t is the time index. This is a time-frequency variable. In this spectrum... Components with smaller values ​​correspond to slow changes in image content (such as head pose adjustment). Components with larger values ​​correspond to rapid changes (such as muscle tremors).

[0123] (2) Spatiotemporal high-pass filtering: In order to suppress low-frequency interference in both spatiotemporal dimensions, a three-dimensional high-pass filter is constructed. First, calculate the value of any point in the three-dimensional frequency domain. to the frequency domain center distance :

[0124]

[0125] in, , , The coordinates are the center coordinates in the frequency domain. The cutoff frequency radius is set to... Constructing a filter mask:

[0126]

[0127] The mask is used to filter the three-dimensional spectrum to obtain the filtered spectrum. :

[0128]

[0129] (3) Three-dimensional discrete inverse Fourier transform (3D-IDFT): The filtered three-dimensional spectrum is restored back to the spatiotemporal domain to obtain a video sequence containing only microscopic dynamic changes. :

[0130]

[0131] Final output The view effectively decouples macroscopic head movements (low-frequency time signals) from the smooth background (low-frequency spatial signals), retaining only the micro-expression dynamic textures with high-frequency characteristics.

[0132] Based on the above steps, the final output consists of three tuples: (RGB view) (2D high-frequency spatial view) (3D high-frequency spatial view).

[0133] S3: By constructing three independent parallel backbone networks, the three views generated in step S2 are encoded in parallel. The video-level feature vectors of the three independent modalities are then deeply fused and mapped to a continuous depression score space.

[0134] To address the issue of significant differences in the distribution of data across different modalities, this embodiment constructs three independent backbone networks and introduces an interaction mechanism at a critical stage of feature encoding, such as... Figure 3 As shown. The specific steps include:

[0135] S301: Constructing a Parallel Backbone Network

[0136] In this embodiment, three deep convolutional neural network branches with identical structures but independent weight parameters are constructed to process the RGB view, the 2D frequency domain high-frequency view, and the 3D frequency domain high-frequency view, respectively. Given that the three views represent color space, spatial edge texture, and spatiotemporal high-frequency dynamic features, respectively, and follow completely different statistical distributions, forcing shared weights would lead to network convergence difficulties. Therefore, the three branches are designed to initialize and update gradients independently. Specifically, combining... Figure 4 In this embodiment, all three parallel branches use ResNet-50 as the basic backbone network. Each branch includes an initial convolutional layer and a max-pooling layer at the input, as well as four consecutive residual feature extraction stages, namely Stage 1 to Stage 4. ResNet-50 adopts a residual learning structure, with each residual stage consisting of multiple stacked residual bottleneck modules. Each residual bottleneck module includes a 1×1 convolution, a 3×3 convolution, and a 1×1 convolution, where the 1×1 convolution is used for channel compression and channel expansion, and the 3×3 convolution is used for extracting local spatial features. Simultaneously, skip connections in the form of identity mapping or projection mapping are used to fuse the module input and convolution output to improve the training stability and feature representation capability of the deep network. In this example, Stage 1 to Stage 4 contain 3, 4, 6, and 3 residual bottleneck modules, respectively. As the network depth increases, the spatial resolution of the feature map gradually decreases, while the number of channels gradually increases, enabling the network to gradually extract high-level semantic features from shallow edge texture features. Finally, the high-dimensional deep features output by Stage4 are global average pooled to form video-level feature vectors for the corresponding views. The video-level feature vectors for each view are then fed into independent LSTM networks to complete temporal feature aggregation, extract the temporal features of each view, and then the temporal features of all views are concatenated and fused. The concatenated fused features are input into the regression head network, which completes the regression prediction of continuous depression scores.

[0137] To address the differences in input channels across different views, this embodiment employs a differentiated initialization strategy:

[0138] RGB view: Directly load standard three-channel weights pre-trained on large-scale datasets (such as ImageNet) to retain the ability to extract general visual features.

[0139] 2D / 3D High-Frequency Views: For input 2D and 3D high-frequency views, since the input format is single-channel data, it is impossible to directly match the first-layer three-channel convolutional kernel of the standard ResNet-50. Therefore, this embodiment adopts a channel averaging strategy for weight initialization. Specifically, the weight parameters of the first-layer convolutional kernel of the standard pre-trained model are... The average value is calculated along the channel dimension to obtain the weight parameters adapted to single-channel input. The calculation formula is:

[0140]

[0141] in This initialization strategy effectively inherits the prior knowledge of the pre-trained model in low-level edge and texture extraction. Compared with random initialization, it can significantly accelerate the convergence speed of the frequency domain branch and improve the stability of feature extraction.

[0142] S302: Multi-stage cross-view feature interaction

[0143] To break down information silos between different views (RGB view, 2D frequency domain high-frequency view, and 3D frequency domain high-frequency view), this invention constructs a bidirectional interaction mechanism in the intermediate layer of the backbone network (such as the output of Stage2 and Stage3). This mechanism includes two processes: enhancement of the frequency domain view by the RGB view, and enhancement of the RGB view by the frequency domain view.

[0144] (1) Enhancement of the frequency domain view by the RGB view: This process aims to utilize the clear facial topology in the RGB view to calibrate the 2D / 3D frequency domain view and suppress pseudo-high-frequency responses caused by background noise. The specific implementation steps are as follows:

[0145] (1.1) Feature embedding and subspace projection: Construct three independent Convolutional layers (whose weight parameters are respectively) ,in The input frequency domain feature map to be calibrated (with the channel dimension after projection) and provides RGB features for guidance Mapping to a low-dimensional common interaction subspace, query vectors are generated respectively. Key vector Sum value vector The calculation method is as follows:

[0146]

[0147] Thus, the query vector is obtained. key vector Sum value vector .

[0148] (1.2) Spatial Flattening and Serialization: Since the dot product attention mechanism is based on matrix operations, the spatial structure of the feature map needs to be converted into a sequential structure. The query vector... key vector value vector The dimension from Reshape to (in Then a transpose operation is performed, making the tensor dimension change.

[0149] This is to accommodate subsequent dot product operations.

[0150] (1.3) Global semantic correlation modeling: Calculate the semantic correlation strength between each pixel in the frequency domain features and all pixels in the RGB features. The specific steps are as follows:

[0151] (1.3.1) Matrix multiplication: Calculate the query matrix AND key matrix transpose The dot product of these two values ​​yields the original relevance score matrix. :

[0152]

[0153] In this matrix, elements Physically, it represents the first in the frequency domain view. Texture features at the position and in the RGB view The degree of matching between semantic features at each location.

[0154] (1.3.2) Scaling and Normalization: To prevent the softmax gradient from vanishing due to excessively large dot product values, a scaling factor is introduced. Then, the Softmax function is applied to normalize along the row dimension to generate a global attention map. :

[0155]

[0156] (1.3.3) Cross-modal feature calibration and mapping: using the generated attention map For the RGB value matrix Perform weighted aggregation and calculate and The matrix multiplication yields intermediate features that include the context. :

[0157]

[0158] in Then... Dimensional reshaping back to spatial format To restore its two-dimensional image structure.

[0159] (1.3.4) Feature Projection Recovery and Residual Fusion: To adapt the enhanced features to subsequent layers of the backbone network and maintain the stability of gradient propagation, the following post-processing operations are performed:

[0160] (1.3.5) Channel restoration: through an output projection layer ( Convolution, with weights ) Change the number of feature channels from Restore to the original dimension .

[0161] (1.3.6) Residual connection: Introducing learnable scaling parameters (Initialized to 0), the calibrated features are compared with the original frequency domain features. Add them together to output the final enhanced features. :

[0162]

[0163] (2) Enhancement of the RGB view by the frequency domain view: This process aims to introduce high-frequency prior information from the frequency domain and guide the RGB branch to focus on local subtle changes with diagnostic value through a channel attention mechanism, thereby reducing the model's dependence on the subject's facial identity attributes. The specific implementation steps are as follows:

[0164] (2.1) Multi-view feature fusion and compression: First, the RGB view features are fused and compressed. Frequency domain branching characteristics Concatenation is performed along the channel dimension to merge the information from both.

[0165]

[0166] Subsequently, through a Convolutional layers reduce the number of channels from Dimensional reduction This achieves preliminary feature fusion, resulting in a fused feature tensor. :

[0167]

[0168] (2.2) Global context aggregation: In order to obtain the global channel response distribution, the fused features are... Perform global average pooling. This operation will reduce the spatial dimension. Compress to Generate a global channel descriptor :

[0169]

[0170] in .

[0171] (2.3) Generation of high-frequency saliency channel weights: Multilayer perceptron (MLP) is used to learn cross-modal channel dependencies. First, through... The convolution compresses the number of channels to It then undergoes BatchNorm and ReLU activation to reduce model complexity and introduce non-linearity. Finally, convolution restores the number of channels to [previous value]. The data is then processed using BatchNorm. Finally, a Sigmoid activation function is applied to generate channel attention weight vectors ranging from (0,1). :

[0172]

[0173] in For ReLU function, For the Sigmoid function, , for Convolution operation.

[0174] (2.4) Feature reweighting and residual fusion output: First, the generated channel attention weight vector is... Perform channel-wise multiplication with the original RGB features to dynamically reweight the feature channels.

[0175]

[0176] To preserve the semantic integrity of the original RGB and fuse the enhanced detailed features, residual connections and final mapping are performed. First, the original RGB features are... With enhanced RGB features Perform channel splicing, then go through one Convolutional layer The concatenated features are then combined and activated using BatchNorm and ReLU to output the final enhanced RGB features. :

[0177]

[0178] S303: Final Spatial Feature Embedding

[0179] This step aims to compress the high-dimensional spatial feature map output by the backbone network into a compact frame-level feature vector, providing a standardized sequence input for subsequent temporal modeling.

[0180] After the multi-layer convolution processing in step S301 and the cross-view interaction enhancement in step S302, the feature maps of the RGB branch, 2D frequency domain branch, and 3D frequency domain branch at the Stage4 output typically have high channel dimensions and low spatial resolution. In this embodiment, for an input size of... The image, the feature tensor output by Stage4 has a dimension of Therefore, the embodiment performs global average pooling on the output features of the three branches respectively:

[0181]

[0182] in Indicates the first The channels are located in spatial positions ( The response value at ) For the pooled first Each channel value.

[0183] After the above processing, the original video tensor data is transformed into three sets of time-series feature vectors: RGB frame-level feature sequences. 2D frequency domain frame-level feature sequences 3D frequency domain frame-level feature sequence .

[0184] S304: Temporal Feature Aggregation

[0185] Because nonverbal behavioral features associated with depression exhibit significant long-term dependence, this step aims to utilize a recurrent neural network to perform temporal modeling on the feature sequence output from step S303, aggregating a video-level global representation. In this embodiment, a Long Short-Term Memory (LSTM) network is used as the temporal aggregator. For the three parallel feature branches, three independent LSTM units are constructed to independently learn the temporal dynamic patterns of each view. The specific steps are as follows:

[0186] (1) Temporal feature modeling: The frame-level feature sequences of each branch are modeled. (in The corresponding LSTM units are input frame by frame in chronological order. Each LSTM unit dynamically updates its internal cell state through forget gates, input gates, and output gates. and hidden state :

[0187]

[0188] in, Includes from frame 1 to frame 2 Accumulated temporal information of frames. In this embodiment, the hidden layer dimension of the LSTM is set to 1. ,Right now .

[0189] (2) Video-level feature extraction: In order to obtain the global sentiment representation of the entire video segment, the hidden state vector output by the LSTM at the last time step is extracted. .

[0190] After temporal aggregation, a three-dimensional video-level feature vector is output, denoted as: RGB video feature vector. 2D frequency domain video feature vector 3D frequency domain video feature vector These three feature vectors will serve as inputs to the feature fusion and depression score regression module.

[0191] S305: Feature Fusion and Depression Score Regression

[0192] This step aims to deeply fuse the video-level feature vectors of the three independent modalities obtained in step S304 and map them to a continuous depression severity rating space. The specific steps are as follows:

[0193] (1) Feature concatenation and joint representation: In order to make comprehensive use of appearance semantics, spatial texture and dynamic micro-expression information, the RGB video feature vector is first concatenated and jointly represented. 2D frequency domain video feature vector and 3D frequency domain video feature vectors Concatenate the data along the channel dimension to construct a multimodal joint feature vector. :

[0194]

[0195] in, .

[0196] (2) Regression Prediction Network Construction: A multilayer perceptron (MLP) consisting of fully connected layers (FC) is constructed as the regressor. The high-dimensional joint features are progressively reduced in dimensionality using the fully connected layers, and a dropout layer (with a dropout rate of 0.5) is introduced to prevent overfitting. Simultaneously, the ReLU activation function is used to introduce nonlinearity.

[0197]

[0198] The final fully connected layer maps the features to a scalar value. Normalize it to 1 using the Sigmoid activation function. The interval is then multiplied by the maximum score of the target scale. The final predicted score is obtained. :

[0199]

[0200] S306: Loss Function Design

[0201] In this embodiment, in order to achieve accurate regression prediction of depression level scores, the present invention uses root mean square error (RMSE) as the sole objective loss function to guide the end-to-end training of the deep neural network.

[0202] RMSE is used to measure the standard deviation between the predicted depression score sequence and the actual clinical assessment score sequence. The calculation formula is as follows:

[0203]

[0204] in This represents the number of samples in the current training batch (Batch Size). For the model to the first The predicted depression score output from each video sample; This is the actual clinical label corresponding to this sample.

[0205] S4: Train the model built by the model building and view processing module using the training dataset, and guide the optimization of the model through the loss function to achieve a quantitative regression assessment of the final state by predicting the depression level score.

[0206] This step trains the model using the training dataset and guides its optimization through the RMSE loss function. The training process aims to learn all learnable parameters of the model, including the weights of ResNet-50, the weights of the multi-stage cross-view feature interaction module, the weights of the LSTM, and the weights of the regression prediction network, to achieve accurate classification of the final state. The specific training process is as follows:

[0207] Data partitioning:

[0208] In this embodiment, a public dataset (such as AVEC or DAIC-WOZ) is selected as the experimental benchmark. To ensure the objectivity of the model evaluation results and to guarantee fair comparability with existing technologies under the same benchmark, this invention keeps the sample composition of each subset constant.

[0209] Specifically, the training and validation sets pre-defined in the dataset are used directly. The training set contains officially designated training samples for gradient backpropagation, weight updates, and feature representation learning in deep neural networks. The validation set contains officially designated validation samples for real-time monitoring of the model's convergence status and for performing hyperparameter fine-tuning during training.

[0210] Model initialization:

[0211] Before each training session begins, all parameters that need to be learned in the model are initialized.

[0212] Iterative training process:

[0213] For each training set, the maximum number of training epochs is set to 400. Within each training cycle, the following operations are performed:

[0214] (1) Forward propagation: Facial samples from three views in the actual training set are input into the model in batches. The model passes through the backbone network, the multi-stage cross-view feature interaction network, and the temporal feature aggregation network in sequence, and finally outputs the prediction results by the regression head.

[0215] (2) Loss calculation: Calculate the loss value according to the defined loss function to obtain the loss of the current batch.

[0216] (3) Backpropagation and parameter update: Using the Adam optimizer, the gradient of the total loss with respect to all trainable parameters of the model is calculated through the backpropagation algorithm. Then, the model parameters are iteratively updated according to the optimizer's update rule (combining the first and second moments of the gradient) and the preset learning rate to minimize the total loss.

[0217] verify:

[0218] After each training cycle, the model is switched to evaluation mode (without parameter updates). In this mode, gradient calculations and parameter updates are frozen, and the Dropout random deactivation mechanism is disabled to ensure the determinism of the inference results. Forward inference is performed using all samples in the validation set to obtain the predicted depression score sequence. Subsequently, the regression performance metrics of the model on the validation set are calculated, with a focus on monitoring the root mean square error (RMSE) and mean absolute error (MAE). If the validation set RMSE of the current round is lower than the historical low, the current model is determined to be the optimal model, and its weight parameter file is saved.

[0219] Experiments on the publicly available dataset AVEC2014 validated the effectiveness of the proposed video-based depression assessment method based on multi-view frequency domain feature fusion. The mean absolute error (MAE) was used as the core evaluation metric; a lower MAE value indicates that the model's predicted depression level is closer to the actual clinical score.

[0220] (1) Comparative analysis of existing methods

[0221] In comparative experiments with current mainstream methods, the model proposed in this invention achieved a MAE score of 5.32, demonstrating a significant performance advantage (as shown in Table 1). Specifically, compared to the spatiotemporal attention network STA-DRN (MAE 6.00) and AVA-DepressNet (MAE 5.99), the error of this method is reduced by approximately 11%, which strongly demonstrates that relying solely on traditional spatiotemporal attention or simple multimodal alignment is insufficient to capture subtle changes in facial depressive cues in patients with depression. Furthermore, compared to the multiscale temporal network MSN (MAE 5.82), this method explicitly decouples fine textures from overall facial information by introducing a frequency domain view, thereby extracting more discriminative pathological representations. Through end-to-end feature interaction, it achieves performance levels comparable to MSN, exhibiting higher training efficiency and clinical feasibility.

[0222] Table 1 Comparative Experiment Results

[0223] (2) Module ablation experiment analysis

[0224] To verify the contributions of each core module within the model, the experiment further compared the model performance under different configurations (as shown in Table 2). First, a baseline model (Backbone) was set up, which used only ResNet-50 to extract RGB features and combined it with LSTM for temporal regression, with a MAE of 6.20. Based on this, Module 1 was constructed by adding 2D and 3D high-frequency spatial view branches. After the three branches extracted features independently, they were directly fused, and the MAE significantly decreased to 5.90. This proves that frequency domain features can effectively supplement the lack of micro-expression details in the simple RGB view. Subsequently, the "RGB view enhancement of frequency domain view" mechanism (Module 2) from S302 was introduced on the basis of Module 1. The clear facial structure of RGB was used to suppress frequency domain noise, and the MAE was further reduced to 5.60. Finally, the complete model of this invention (Module 3) was constructed by adding the "frequency domain view channel attention enhancement of RGB view" mechanism, which achieved deep complementarity between macro structure and micro texture, and the model achieved optimal performance (MAE 5.32). This progressive performance improvement from 6.20 to 5.32 fully demonstrates the necessity and effectiveness of introducing multi-view and two-way cross-view interaction mechanisms in the assessment of depressive states.

[0225] Table 2 Ablation Experiment Results

[0226] See Figure 5This invention also provides a video depression assessment system based on multi-view frequency domain feature fusion, including a standardized facial image sequence output module, a multi-view frequency domain feature generation module, a model construction and view processing module, and a model optimization and output module.

[0227] The standardized facial image sequence output module is used to acquire facial video data of the target object, perform preprocessing, and output a standardized facial image sequence.

[0228] The multi-view frequency domain feature generation module is used to convert the facial image sequence output by the standardized facial image sequence output module into three parallel views containing different physical properties, including an RGB appearance view, a 2D high-frequency spatial view, and a 3D high-frequency spatial view, to explicitly decouple depression-related fine textures and instantaneous dynamics from the original video.

[0229] The model building and view processing module is used to construct three independent parallel backbone networks to perform parallel feature encoding on the three views generated by the multi-view frequency domain feature generation module, deeply fuse the video-level feature vectors of the three independent modalities, and map them to a continuous depression score space.

[0230] The model optimization and output module is used to train the model built by the model building and view processing module using the training dataset, and to guide the optimization of the model through the loss function, and to achieve a quantitative regression assessment of the final state by predicting the depression level score.

[0231] This example provides a video depression assessment system based on multi-view frequency domain feature fusion, which can execute the video depression assessment method based on multi-view frequency domain feature fusion provided by this invention. It can execute any combination of the steps in the method example and has the corresponding functions and beneficial effects of the method.

[0232] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A video-based depressive state assessment method based on multi-view frequency domain feature fusion, characterized in that, Includes the following steps: S1: Acquire facial video data of the target object, perform preprocessing, and output a standardized facial image sequence; S2: The facial image sequence output from step S1 is converted into three parallel views containing different physical properties, including an RGB appearance view, a 2D high-frequency spatial view, and a 3D high-frequency spatial view, explicitly decoupling the subtle textures and instantaneous dynamics related to depression from the original video; the RGB view of the facial image is sequentially subjected to image grayscale conversion, two-dimensional discrete Fourier transform, high-pass filtering, and two-dimensional discrete Fourier inverse transform to obtain the 2D high-frequency spatial view V. 2d The video sequence is sequentially subjected to 3D discrete Fourier transform, spatiotemporal high-pass filtering, and 3D discrete Fourier transform to obtain a 3D high-frequency spatial view V. 2d ; S3: By constructing three independent parallel backbone networks, the three views generated in step S2 are encoded in parallel. The video-level feature vectors of the three independent modalities are then deeply fused and mapped to a continuous depression score space. S4: Train the model built in step S3 using the training dataset, and guide the optimization of the model through the loss function to achieve a quantitative regression assessment of the final state by predicting the depression level score.

2. The video depression state assessment method based on multi-view frequency domain feature fusion according to claim 1, characterized in that, In step S1, the preprocessing steps for the acquired facial video data include: S101: Face detection: The high-precision face detection framework Facer based on deep convolutional neural networks is used to process video frames. A batch processing method is used to read a single video frame sequence, and dynamic bounding box expansion is used to crop the image. The size of the cropped facial image blocks of different sizes is normalized to construct a standardized facial image sequence. S102: Image Denoising and Standardized Output: Gaussian smoothing filtering is applied to each frame of the facial image sequence to smooth isolated pixel noise while preserving the main facial texture structure. The final output consists of an array or tensor of at least 30 geometrically aligned and denoised RGB facial images, which serves as the standard input for subsequent multi-view frequency domain feature generation.

3. The video depression state assessment method based on multi-view frequency domain feature fusion according to claim 1, characterized in that, In step S2, the standardized RGB facial image sequence output from step S1 is directly used to construct an RGB appearance view, denoted as... As a baseline data stream It fully preserves the global color distribution, facial topology, and macroscopic facial expression changes of the subjects' faces.

4. The video depression state assessment method based on multi-view frequency domain feature fusion according to claim 1, characterized in that, In step S2, each frame of the facial image sequence output in step S1 is independently filtered in the frequency domain to remove low-frequency components composed of smooth skin and ambient lighting, extract fine texture features, and construct a 2D high-frequency spatial view. Specific steps include: S211: Image grayscale conversion: Convert the t-th frame RGB image output in step S1 to grayscale. Convert to single-channel grayscale image The mapping is performed using a standard luminance-weighted formula: Where R(x,y), G(x,y), and B(x,y) represent the pixel values ​​of the red, green, and blue color channels at pixel (x,y), respectively. After grayscale conversion, the image data is converted from a three-channel RGB image of H×W×3 to a single-channel grayscale image of H×W. S212: Two-dimensional Discrete Fourier Transform: Processing a single-frame grayscale image Transform it from the spatial domain Mapping to the frequency domain For a size of The transformation formula for the image is: in, Represents the complex spectral coefficients in the frequency domain. The imaginary unit, and These are the frequency variables in the horizontal and vertical directions, respectively; S213: Constructing a high-pass filter: First, calculate any point in the frequency domain. to the frequency domain center distance : in, , ; Subsequently, the filter mask is defined. Set the cutoff frequency radius as : The original spectrum is multiplied point by point with the filter to obtain the filtered spectrum. : S214: Two-dimensional Discrete Fourier Inverse Transform: The filtered spectrum... Restore to the spatial domain to generate an edge-enhanced image. : in This represents taking the real part of a complex number, and finally reconstructing the processed image of t frames to obtain a 2D high-frequency spatial view. .

5. The video depression state assessment method based on multi-view frequency domain feature fusion according to claim 1, characterized in that, In step S2, the entire video sequence is treated as a spatiotemporal global tensor. A joint spatiotemporal high-pass filter is performed using a three-dimensional frequency domain transformation to filter out low-frequency components in both the spatial and temporal dimensions, thus constructing a 3D high-frequency spatial view. Specific steps include: S221: Three-dimensional Discrete Fourier Transform: Treating video sequences as three-dimensional signals Perform a joint spatiotemporal transformation to obtain data containing spatial frequencies. and time frequency Three-dimensional spectrum : in, For time indexing, For time frequency variables; T is the time series length, W is the width of the original high-resolution image, and H is the height of the original high-resolution image; S222: Spatiotemporal High-Pass Filtering: First, calculate the value at any point in the three-dimensional frequency domain. to the frequency domain center distance : in, , , Let the frequency domain center coordinates be used, and set the cutoff frequency radius as... Constructing a filter mask: The mask is used to filter the three-dimensional spectrum to obtain the filtered spectrum. : in, It is a three-dimensional high-pass filter; S223: 3D Discrete Fourier Inverse Transform: Restores the filtered 3D spectrum back to the spatiotemporal domain, resulting in a video sequence containing only microscopic dynamic changes. : Final output The view effectively decouples macroscopic head movements from the smooth background, retaining only the dynamic textures of micro-expressions with high-frequency characteristics.

6. The video depression state assessment method based on multi-view frequency domain feature fusion according to claim 1, characterized in that, In step S3, the parallel backbone network uses ResNet-50 as the basic backbone network. Each branch contains four consecutive convolutional stages. As the network depth increases, the spatial resolution of the feature map decreases step by step, while the number of channels increases step by step. Finally, the fourth stage convolution outputs high-dimensional deep features.

7. The video depression state assessment method based on multi-view frequency domain feature fusion according to claim 6, characterized in that, A bidirectional interaction mechanism was built in the middle layer of the parallel backbone network, which includes two processes: enhancement of the frequency domain view by the RGB view, and enhancement of the RGB view by the frequency domain view. Enhancement of the frequency domain view by the RGB view: This process utilizes the clear facial topology in the RGB view to calibrate the 2D high-frequency spatial view and the 3D high-frequency spatial view, and suppresses the pseudo-high-frequency response caused by background noise. Enhancement of RGB view by frequency domain view: This process introduces high-frequency prior information in the frequency domain and guides the RGB branch to focus on local subtle changes with diagnostic value through a channel attention mechanism, reducing the model's dependence on the subject's facial identity attributes.

8. The video depression state assessment method based on multi-view frequency domain feature fusion according to claim 1, characterized in that, In step S3, global average pooling is performed on the output features of the three independent parallel backbone networks, transforming the original video tensor data into three sets of time-series feature vectors: RGB frame-level feature sequence, 2D frequency domain frame-level feature sequence, and 3D frequency domain frame-level feature sequence. Then, a recurrent neural network is used to perform temporal modeling on the three sets of time-series feature vectors, aggregating the video-level global representation. After temporal aggregation, three-dimensional video-level feature vectors are output: RGB video feature vector, 2D frequency domain video feature vector, and 3D frequency domain video feature vector. Finally, the three independent modal video-level feature vectors obtained by aggregation are deeply fused and mapped to a continuous depression score space.

9. The video depression state assessment method based on multi-view frequency domain feature fusion according to claim 1, characterized in that, In step S4, the root mean square error (RMSE) is used as the target loss function to guide the end-to-end training of the deep neural network, thereby achieving accurate regression prediction of depression scores.

10. A video-based depressive state assessment system based on multi-view frequency domain feature fusion, characterized in that, include: The standardized facial image sequence output module is used to acquire facial video data of the target object, perform preprocessing, and output a standardized facial image sequence. The multi-view frequency domain feature generation module is used to convert the facial image sequence output by the standardized facial image sequence output module into three parallel views containing different physical properties, including an RGB appearance view, a 2D high-frequency spatial view, and a 3D high-frequency spatial view, to explicitly decouple depression-related fine textures and instantaneous dynamics from the original video. The RGB view of the facial image is sequentially converted to grayscale, subjected to two-dimensional discrete Fourier transform, high-pass filtering, and inverse two-dimensional discrete Fourier transform to obtain a 2D high-frequency spatial view V. 2d The video sequence is sequentially subjected to 3D discrete Fourier transform, spatiotemporal high-pass filtering, and 3D discrete Fourier transform to obtain a 3D high-frequency spatial view V. 3d ; The model building and view processing module is used to construct three independent parallel backbone networks to perform parallel feature encoding on the three views generated by the multi-view frequency domain feature generation module, deeply fuse the video-level feature vectors of the three independent modalities, and map them to a continuous depression score space. The model optimization and output module is used to train the model built by the model building and view processing module using the training dataset, and guides the optimization of the model through the loss function, and achieves quantitative regression assessment of the final state by predicting the depression level score.

Images