A method and device for detecting depression tendency based on dynamic three-dimensional human face

Abstract

The present invention relates to the field of artificial intelligence, and in particular to a method and device for detecting depressive tendencies based on dynamic three-dimensional faces. The present invention uses dynamic three-dimensional face sequence data as a basis, integrates dual-path feature extraction of depth modality and texture modality, and frequency domain analysis of head posture, and realizes multimodal feature adaptive fusion through an attention mechanism. Such a technical solution is the first of its kind in the field of depressive tendency detection and can effectively overcome the problems of insufficient utilization of depth information, incomplete capture of dynamic patterns, and lack of head signal analysis in the existing technology. It provides an objective, dynamic and efficient quantitative method for preliminary screening of depressive tendencies, which can capture subtle changes in behavioral characteristics and is suitable for rapid screening of large populations.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method and device for detecting depression tendency based on dynamic three-dimensional human faces. Background Art

[0002] Early identification of depressive tendencies is crucial for improving patients' quality of life and reducing social burden. However, traditional methods for detecting depressive tendencies have numerous limitations, making it difficult to objectively and efficiently capture these dynamic behavioral characteristics. With the development of artificial intelligence (AI), a growing number of studies are attempting to utilize physiological signals and behavioral data for non-invasive detection.

[0003] Currently, the detection of depression tendency mainly relies on the following traditional methods, but they each have their own shortcomings:

[0004] Standardized questionnaires: Typical tools include the Patient Health Questionnaire-9 (PHQ-9) and the Beck Depression Inventory (BDI). These methods rely on self-reporting, which is limited by the subject's subjective emotions and cultural background. Furthermore, questionnaires struggle to capture dynamic behavioral changes, such as persistent changes in facial expression or low-frequency patterns in head posture. Furthermore, questionnaire assessments typically require specialized personnel to interpret the results, making them difficult to implement on a large scale for initial screening.

[0005] Clinical Observation Interview Method: Clinical psychologists assess signs of depression through face-to-face interviews and observation of nonverbal behaviors (such as facial expressions and body posture). Limitations include low inter-rater consistency (e.g., inter-observer Cohen's Kappa values ​​are often below 0.7, indicating insufficient agreement) due to the method's time-consuming and resource-intensive nature, making it impractical for widespread, routine large-scale screening. Furthermore, manual observation lacks precise tools to quantify continuous, dynamic changes in facial and posture.

[0006] Therefore, there is an urgent need for an objective, dynamic and efficient quantitative method in the initial screening of depressive tendencies that can capture subtle changes in behavioral characteristics and is suitable for rapid screening of large populations. Summary of the Invention

[0007] The purpose of the present invention is to overcome the above-mentioned deficiencies in the prior art and to provide a method for detecting depression tendency based on dynamic three-dimensional human faces.

[0008] In order to achieve the above-mentioned object of the invention, the present invention provides the following technical solutions:

[0009] A method for detecting depression tendency based on dynamic three-dimensional human faces, comprising the following steps:

[0010] a: Obtain the face data to be detected, perform preprocessing, and output standardized RGBD data;

[0011] b: feeding the RGBD data into two feature extraction networks in parallel; wherein the first feature extraction network is used to extract the depth variation features of the RGBD data, and the second feature extraction network is used to extract the texture motion features of the RGBD data;

[0012] c: extracting a head posture signal from the RGBD data, and generating a head posture feature according to the head posture signal;

[0013] d: Fusing the depth change feature, the texture motion feature, and the head posture feature, and inputting the fused comprehensive feature into a feedforward neural network for classification regression, and outputting a depression tendency probability score of the face data to be detected.

[0014] As a preferred embodiment of the present invention, the pretreatment in step a comprises the following steps:

[0015] a1: Taking the nose tip as the reference, perform depth normalization on the depth map of the face data to be detected;

[0016] a2: In the depth map after depth normalization, with the nose tip as the reference, retain the points whose depth is within the preset range and output them as the facial region. Then, perform bilinear interpolation between the depth map of the facial region and the corresponding RGB image.

[0017] a3: Perform frame sequence alignment and smoothing on the time series of the face data to be detected, and output standardized RGBD data.

[0018] As a preferred embodiment of the present invention, a1 comprises the following steps:

[0019] a11: Select a frame of depth map, detect the position of the nose tip, and obtain the depth value d of the point nose (t);

[0020] a12: Subtract the depth value d from the depth value of all pixels in the depth map of this frame nose (t);

[0021] a13: Filter the processed depth map within a preset depth range and output the filtered depth map;

[0022] a14: Loop through steps a11 to a13 until the depth maps of all frames are normalized.

[0023] As a preferred embodiment of the present invention, the first feature extraction network includes four groups of sequentially connected three-dimensional convolutional layers and residual connection layers, and a global average pooling layer;

[0024] The convolution kernel of the three-dimensional convolution layer is 3×3×3, and the stride is 1; the output channels of the first to fourth three-dimensional convolution layers are 16, 32, 64, and 128, respectively.

[0025] As a preferred solution of the present invention, the second feature extraction network includes a first temporal convolution layer, a second temporal convolution layer and a multi-head attention mechanism layer connected in sequence;

[0026] The first temporal convolution layer uses dynamic convolution to extract texture motion features and has an output channel of 32;

[0027] The second temporal convolution layer uses dynamic convolution to extract texture motion features and has 64 output channels;

[0028] The multi-head attention mechanism layer is used to further process the features output by the previous layer and output 128-dimensional features.

[0029] As a preferred solution of the present invention, the first temporal convolution layer and the second temporal convolution layer include a fully connected sub-network; including the following operation process:

[0030] The fully connected sub-network predicts a set of weights based on the overall features of the previous frame; wherein, when the input is the first frame, the prediction is made based on the overall features of the current frame;

[0031] The adaptive convolution kernel of the current frame is obtained according to the weight, and its expression is:

[0032] ,

[0033] Among them, W dyn (t) is the convolution filter customized for the t-th frame, K is the number of basic convolution filters in the temporal convolution layer, B k represents the kth base convolution kernel, a k (t) is the corresponding weight coefficient; a k (t)∈{a1,a2,...,a K}.

[0034] As a preferred embodiment of the present invention, said step c comprises the following steps:

[0035] c1: regressing the head pose angle of each frame from the RGB frame sequence of the RGBD data through the HopeNet model, storing it in a head angle sequence, and performing sliding window averaging on the head angle sequence;

[0036] c2: Convert the head angle sequence into a posture sequence in the frequency domain, perform a discrete Fourier transform on each dimension of the posture sequence to obtain a corresponding frequency spectrum representation, and filter out a set proportion of data with the highest frequency based on the frequency spectrum representation;

[0037] c3: Convert the filtered posture sequence back to the time domain through inverse discrete Fourier transform to obtain a smooth head posture curve P′(n);

[0038] c4: extracting a head posture feature vector of the head posture curve; the head posture feature vector includes statistical features and periodic features.

[0039] As a preferred solution of the present invention, the statistical features include one or more of a mean value, a variance, and an extreme value range; and the periodic features include a frequency domain feature and / or a behavior pattern feature.

[0040] As a preferred embodiment of the present invention, the step d comprises the following steps:

[0041] d1: converting the depth change feature, the texture motion feature, and the head posture feature into feature sequences of the same format through dimensionality adjustment;

[0042] d2: Input the dimension-adjusted feature sequence into the multi-head attention fusion module to obtain the fused comprehensive features;

[0043] d3: Input the comprehensive features into a feedforward neural network for classification regression, and output a depression tendency probability score of the face data to be detected.

[0044] A device for detecting depression tendency based on dynamic three-dimensional human faces comprises at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any of the methods described above.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The present invention uses dynamic three-dimensional facial sequence data as a basis, integrates dual-path feature extraction of depth modality and texture modality, and frequency domain analysis of head posture, and realizes multimodal feature adaptive fusion through the attention mechanism. This technical solution is the first of its kind in the field of depressive tendency detection and can effectively overcome the problems of insufficient utilization of depth information, incomplete capture of dynamic patterns, and lack of head signal analysis in existing technologies. It provides an objective, dynamic and efficient quantitative method for preliminary screening of depressive tendencies, which can capture subtle changes in behavioral characteristics and is suitable for rapid screening of large populations. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flow chart of a method for detecting depression tendency based on dynamic three-dimensional faces according to Example 1 of the present invention;

[0048] Figure 2 This is a schematic structural diagram of dual-path feature extraction in a method for detecting depression tendency based on dynamic three-dimensional faces according to Example 2 of the present invention;

[0049] Figure 3 This is a flow chart of step c in a method for detecting depression tendency based on dynamic three-dimensional faces according to embodiment 2 of the present invention;

[0050] Figure 4 This is a structural schematic diagram of a depressive tendency detection device based on a dynamic three-dimensional face, described in Example 4 of the present invention, which utilizes the depressive tendency detection method based on a dynamic three-dimensional face described in the previous embodiment. DETAILED DESCRIPTION

[0051] The present invention will be further described in detail below in conjunction with test examples and specific embodiments. However, this should not be understood as limiting the scope of the present invention to the following embodiments, and all technologies implemented based on the present invention fall within the scope of the present invention.

[0052] This paper proposes a method for detecting depressive tendencies using dynamic three-dimensional facial data (Red-Green-Blue-Depth, RGBD) and deep learning technology. This method captures subtle behavioral changes in an objective and quantitative manner, serving as an early screening tool to indicate potential risks and recommend professional diagnosis and treatment. The method is based on the following theory:

[0053] Objective behavioral characteristics of depressive tendencies: Depressive tendencies are often manifested through specific facial muscle dynamics and head posture patterns, which can serve as objective behavioral indicators. Specifically, they include:

[0054] Facial muscle dynamics: Depressed individuals often exhibit reduced facial expression activity, such as significantly less frequent smiling and prolonged negative expressions, such as drooping of the mouth corners. These signs may be accompanied by subtle three-dimensional deformations, such as increased amplitude and duration of drooping of the mouth corners. Quantifying these local three-dimensional facial movements is necessary to capture the subtle changes in expression associated with depressive tendencies.

[0055] Head posture patterns: Depressive tendencies may also be reflected in the frequency and amplitude of head movements. For example, depressed individuals often lower their heads frequently and for prolonged periods, with overall head movements exhibiting a low-frequency pattern (maintaining a single posture for extended periods, such as drooping). In contrast, healthy individuals exhibit more frequent and uniform changes in head posture.

[0056] These facial micro-dynamics and low-frequency patterns in head posture are important behavioral clues to depressive tendencies. These characteristics possess both dynamic (evolving over time) and three-dimensional (deforming over space) properties, making them difficult to accurately capture through traditional manual observation or two-dimensional video analysis. Therefore, sophisticated three-dimensional spatiotemporal data analysis techniques are required.

[0057] Advantages of dynamic 3D facial data: 2D videos lack depth information and are easily affected by the environment, making them inadequate for quantifying subtle 3D behavioral characteristics of depressive tendencies.

[0058] Lack of depth information: 2D videos only provide a flat texture, making it impossible to capture the true magnitude of facial movements. For example, the drooping corners of the mouth can only be seen in 2D, but the actual "3D drooping depth" cannot be measured. Depth information can quantify this distance.

[0059] Poor environmental robustness: 2D video analysis is susceptible to changes in lighting, camera angle, and occlusion. In poor lighting or when the face is viewed from a side angle, the accuracy of facial landmark detection drops significantly, reportedly by approximately 30%. The introduction of depth sensing can mitigate the effects of lighting and shadows to a certain extent, as depth maps directly reflect distance regardless of lighting, thereby improving the stability of feature extraction.

[0060] Insufficient dynamic continuity: 2D methods struggle to fully capture the continuous evolution of expressions and postures. For example, a continuous, subtle change in expression may be interrupted in 2D video due to changes in angle or lighting, while in 3D sequences, we can track the depth curve throughout the entire process. Furthermore, slow changes in head posture (low-frequency components) are often overlooked in 2D, while 3D data records the complete three-dimensional motion trajectory, better reflecting these low-frequency behavioral patterns.

[0061] In contrast, dynamic three-dimensional face data (RGBD) combines color images with corresponding depth information, which can more comprehensively capture behavioral characteristics related to depressive tendencies:

[0062] Precise Stereoscopic Quantification: RGBD data provides the distance from each pixel to the camera, enabling us to precisely measure the magnitude of three-dimensional changes in facial expressions. For example, we can quantify the displacement of the corners of the mouth in 3D space and plot this change over time, thereby capturing objective indicators of a diminished smile or facial stiffness. Similarly, changes in head posture can be directly quantified using 3D coordinate trajectories (e.g., the angle and duration of head droop).

[0063] Enhanced environmental robustness: The addition of depth information reduces interference from factors such as lighting and skin color. Even in low light or partial occlusion, as long as the depth camera captures the facial outline, reliable features can still be extracted. This makes detection more robust in non-ideal environments.

[0064] Spatiotemporal continuity: RGBD sequences preserve the continuous deformation and motion of the target object over time, allowing analysis of subtle, long-term changes. For example, RGBD data can capture the slow behaviors of individuals with depressive tendencies (such as prolonged head bowing or a persistent expressionless face), allowing the extraction of low-frequency but persistent abnormal patterns.

[0065] Technical Feasibility: Currently, commercially available RGBD cameras (such as the Intel RealSense series) are low-cost, compact, and easily deployed in devices like mobile phones and tablets. This makes large-scale screening feasible—ordinary users can use devices equipped with depth cameras to perform self-examinations, with the system automatically analyzing dynamic facial and head signals to assess depression risk.

[0066] In summary, the use of dynamic 3D facial data for depression detection is necessary and has significant advantages. This paper leverages these advantages of RGBD data, using innovative techniques such as depth normalization, feature extraction, and signal processing to accurately capture and quantitatively analyze behavioral characteristics associated with depression.

[0067] Example 1

[0068] like Figure 1 As shown, a method for detecting depression tendency based on dynamic three-dimensional human faces includes the following steps:

[0069] a: Obtain the face data to be detected, perform preprocessing, and output standardized RGBD data.

[0070] b: The RGBD data is fed into two feature extraction networks in parallel; wherein the first feature extraction network is used to extract the depth change features of the RGBD data, and the second feature extraction network is used to extract the texture motion features of the RGBD data.

[0071] c: Extracting a head posture signal from the RGBD data, and generating a head posture feature based on the head posture signal.

[0072] d: Fusing the depth change feature, the texture motion feature, and the head posture feature, and inputting the fused comprehensive feature into a feedforward neural network for classification regression, and outputting a depression tendency probability score of the face data to be detected.

[0073] Example 2

[0074] This embodiment is a specific implementation of the method for detecting depression tendency based on dynamic three-dimensional faces described in Example 1, and includes the following steps:

[0075] a: Obtain the face data to be detected, perform preprocessing, and output standardized RGBD data.

[0076] An RGBD camera is used to acquire approximately 30 seconds of dynamic facial sequence data, including a color image and corresponding depth map for each frame. The preprocessing stage normalizes and cleans the data: first, the depth values ​​are standardized and calibrated using the nose tip as a reference to eliminate the effects of overall head displacement; then, the effective facial area is cropped and the image size is unified; then, the frame sequence is temporally aligned and smoothed to ensure spatial and temporal consistency and high quality of the input data. Further steps are included:

[0077] Data Acquisition: An RGBD camera is used to acquire dynamic facial sequence data. For example, this example uses an Intel RealSense D455 camera, capturing video at 1280×720 resolution and 30 frames per second under standard indoor lighting conditions, acquiring approximately 900 frames of data in approximately 30 seconds. Each frame contains a synchronized RGB image (three channels) and a corresponding depth map (one channel). The depth map provides approximately 150,000 three-dimensional coordinates. During acquisition, the subject is ensured to be in a natural state (for example, sitting quietly or engaging in light Q&A with the device), and deliberately exaggerated facial expressions are avoided to capture realistic signals from their daily behavior.

[0078] Preprocessing step: The raw RGBD sequence is directly used for analysis, which may be subject to various interferences (such as head movement, background objects, etc.). To this end, the present invention designs three consecutive preprocessing steps to standardize and clean the data to improve the accuracy of subsequent feature extraction. Specifically, the following steps are included:

[0079] a1: Taking the nose tip as a reference, perform depth normalization on the depth map in the face data to be detected.

[0080] Considering that depressive tendency analysis focuses on local facial muscle movements, and the overall head orientation or forward and backward movement will cause an overall offset in the facial depth data, thereby interfering with the detection of subtle changes in expression, this step proposes a depth normalization method based on the nose tip benchmark:

[0081] a11: Select a frame of depth map, detect the position of the nose tip, and obtain the depth value d of the point nose (t) (distance from the camera).

[0082] a12: Subtract the depth value d from the depth value of all pixels in the depth map of this frame nose (t).

[0083] Using the nose tip depth as a reference, subtract d from the depth values ​​of all pixels in the frame. nose(t), which measures the depth relative to the tip of the nose. After this operation, the depth at the tip of the nose is corrected to 0, and the depths of the remaining facial points are expressed relative to the tip of the nose.

[0084] a13: Filter the processed depth map within a preset depth range and output the filtered depth map;

[0085] To focus on the facial area, this embodiment uses a 300mm depth range as a standardization interval: all pixels farther than 300mm from the tip of the nose are considered background and their depth values ​​are truncated to 0. This fixed range is selected based on statistical analysis of facial dimensions and the range of commonly used depth cameras (for example, in a typical sitting position, the depth variation of facial features relative to the tip of the nose generally does not exceed 300mm). Through this process, each depth image frame is normalized to a 300mm radius centered on the tip of the nose, eliminating the effects of significant head movement and background objects.

[0086] a14: Loop through steps a11 to a13 until the depth maps of all frames are normalized.

[0087] Unlike traditional methods that use global min-max normalization (which can change the normalization scale due to scene and movement), this step introduces the body's own reference point (the tip of the nose) and a fixed physical range for normalization. This not only eliminates the interference of overall head displacement on the depth data, but also ensures consistency in depth measurement standards across individuals and frames. Through this processing step, the depth data now primarily reflects local fluctuations in the face (for example, a muscle lift will cause the depth at that location to decrease relative to the tip of the nose), without causing spurious changes due to the movement of the entire face. This significantly improves the signal-to-noise ratio and targetedness of subsequent feature extraction, providing high signal-to-noise ratio input for subsequent capture of subtle expressions.

[0088] a2: In the depth map after depth normalization, the nose tip is used as the reference, and points with depth within the preset range are retained and output as the facial area. The depth map of the facial area is then bilinearly interpolated with the corresponding RGB image.

[0089] After depth normalization, we further determine the valid facial area and unify the data size. Because the depth map may still contain background outside a certain range near the nose tip (such as non-facial parts like shoulders and hair), this step uses depth information to crop out the pure facial area: Centered around the nose tip, the depth map is expanded outward by a certain range, retaining only points with depth values ​​within a certain threshold.

[0090] Furthermore, in this embodiment, after calculating the depth of the nose tip as 0, an area with a depth value between 0.1 and 1.0 is searched (here 0.1 and 1.0 are normalized values, corresponding to 10% and 100% of the aforementioned 300mm range, with the purpose of excluding the background of the very close point and the farthest edge next to the nose tip). This range roughly covers all points on the facial surface. After finding the boundaries of these points, a margin of about 10% is expanded outward based on the boundaries to ensure that the entire face (from the edge of the forehead to the chin contour) is completely contained in the selected area. After obtaining the facial area, it is cropped from the original image, and bilinear interpolation is used to scale / resample the depth map and the corresponding RGB image to a uniform size, for example, 256×256 pixels are used in this embodiment. Bilinear interpolation calculates the values ​​of unaligned pixels by weighting neighboring pixels, so that the scaled image has a smooth transition and is not distorted. The uniform resolution can ensure the consistency of subsequent neural network inputs.

[0091] This step effectively removes background and non-face areas through cropping based on depth thresholding, retaining only the areas containing important facial information. Interpolation then normalizes all data to a consistent scale, preparing it for batch input to the network while preserving the geometric proportions of facial features.

[0092] a3: Perform frame sequence alignment and smoothing on the time series of the face data to be detected, and output standardized RGBD data.

[0093] Furthermore, the time series of the face data to be detected is aligned and denoised. Because frame loss or frame rate fluctuations may occur during the recording process, this embodiment aligns the sequence to a fixed frame rate of 30 fps. If the number of captured frames is less than expected (for example, fewer than 900), the missing frames are supplemented in the temporal dimension through linear interpolation. Linear interpolation applies a linear gradient to the pixels of adjacent known frames in chronological order, generating intermediate transition frames to ensure uniform length and temporal scale across the entire sequence. Furthermore, the aligned sequence may still contain high-frequency jitter noise (from the device or subtle tremors of the subject). To improve the signal-to-noise ratio, this embodiment applies a Gaussian smoothing filter to each pixel in the time series. For example, a Gaussian filter with a window size of 5 frames and a standard deviation of 1 is used to smooth the sequence representation—this is equivalent to taking a weighted average of frames within a temporal neighborhood using Gaussian weights, reducing isolated sharp fluctuations. After this step, a sized and smooth four-channel data tensor is obtained, with a format of 256×256×900×4. The "4" here represents the three RGB color channels and one depth channel.

[0094] This step ensures that each sample sequence has the same length and time scale through frame alignment, enabling the model to learn timing patterns with a fixed number of frame inputs. It also effectively removes sensor noise and occasional motion through smoothing filtering, providing a cleaner timing data foundation for subsequent weak, continuous signals that need to be captured.

[0095] Through this preprocessing, the facial data to be detected (raw dynamic RGBD facial data) is converted into high-quality, standardized spatiotemporal data, providing a reliable guarantee for subsequent feature extraction. This preprocessed data focuses on local dynamic changes in the face, eliminating interference from environmental, background, and overall motion factors, and preserving the most detailed information related to depressive tendencies.

[0096] b: The RGBD data is fed into two feature extraction networks in parallel; wherein the first feature extraction network is used to extract the depth change features of the RGBD data, and the second feature extraction network is used to extract the texture motion features of the RGBD data.

[0097] like Figure 2 As shown, after the preprocessing is completed, the present invention uses a dual-path parallel network architecture to extract features from facial data. The intuitive motivation for designing a dual path is that depth information and color texture contain expression features of different dimensions, each with its own focus. The depth channel is good at representing the amplitude and spatial structure changes of facial deformation, such as the degree of bulge in the area between the eyebrows when frowning; while the RGB channel retains the texture details and dynamic texture changes of muscle movement, such as slight changes in skin color or movement of shadows. By processing and fusing these two parts of information separately, the expression change patterns related to depressive tendencies can be captured more comprehensively and accurately. Specifically:

[0098] Path 1: The first feature extraction network for depth variation feature extraction (based on improved C3D).

[0099] The first path primarily analyzes preprocessed depth data, aiming to extract the three-dimensional variations of facial muscle movement over time. This implementation utilizes a 3D convolutional neural network (C3D) to process spatiotemporal data, using simultaneous convolutions of time and space to capture temporal patterns. To meet the requirements of this task, this implementation improves the original C3D architecture, specifically introducing residual connections to enhance the ability to learn subtle variations in long sequences.

[0100] The improved C3D (the first feature extraction network) consists of four groups of sequentially connected three-dimensional convolutional layers and residual connection layers and a global average pooling layer.

[0101] The first convolution layer has a kernel size of 3×3×3 (3×3 spatially, 3 frames in temporal window), a stride of 1, and 16 output channels, and is used to extract preliminary spatiotemporal features. The second through fourth convolution layers also use 3×3×3 kernels with a stride of 1, with the number of output channels increasing to 32, 64, and 128, respectively. Each convolution layer is followed by a residual connection layer. This residual connection layer directly adds the input features of that layer to the convolution output before feeding it into the next layer. This means that these convolution layers do not completely discard the input information, but instead superimpose the input on the learned incremental features, thereby alleviating the vanishing gradient problem in deep network training. Finally, the global average pooling layer averages the feature cubes output by the previous layer in both spatial and temporal dimensions to produce a 128-dimensional global depth feature vector. This vector condenses key information about facial deformation in the 900-frame depth sequence.

[0102] The original C3D network (proposed by Tran et al. in 2015 for action video classification) consists of five layers of 3D convolutions, with increasing channels at each layer. However, stacking such a deep structure directly can lead to vanishing gradients and feature attenuation when learning small changes in long sequences. For depression detection, the sequences of our invention are long (900 frames), and the signal of interest varies very little (for example, eyelid ptosis typically varies by only a few millimeters, corresponding to minimal changes in depth). Traditional deep networks may lose these small gradients layer by layer as they propagate, resulting in insufficient sensitivity to these subtle depth changes. The success of residual networks (ResNets) demonstrates that directly passing inputs to outputs through shortcut connections can effectively mitigate the vanishing gradient problem in deep networks while preserving the original information. Therefore, our invention introduces a residual structure in the intermediate layers of C3D, allowing each convolutional layer to learn only small changes relative to the input, rather than the entire mapping. This design makes it easier for the network to capture and amplify small depth changes that would otherwise be easily drowned out.

[0103] The improved C3D network can be viewed as automatically learning the gradient and curvature information of a time series. Gradient refers to the first-order difference of depth over time (reflecting the speed of change); curvature can be understood as the second-order difference of depth (reflecting the acceleration or degree of curvature). For example, in the case of a gradual drooping of the upper eyelid, the first-order difference captures the rate of descent of the eyelid position in each frame, while the second-order difference reflects the change in descent rate—whether it continues to descent or begins to rise. By accumulating small changes layer by layer, the residual C3D network effectively builds internal channels sensitive to these temporal gradients, enabling it to respond to the trend and magnitude of facial expression changes. While the network's internal calculations of gradient and curvature formulas are not explicitly shown, these features are effectively learned (for example, a convolution kernel can be approximated as a first-order difference filter to detect change, while stacking two layers is equivalent to detecting second-order changes). Therefore, compared to manual calculations, this data-driven approach can adaptively extract the most discriminative change signals.

[0104] The first feature extraction network proposed in the present invention is the first to introduce the residual connection mechanism into C3D for the study of depression tendency detection. Through this improvement, the network accumulates change features without losing the original information, greatly enhancing the ability to extract subtle depth change patterns. The experimental results also verify its effectiveness: the improved deep feature extraction network can more accurately capture dynamics such as slight drooping of eyelids and slight drooping of mouth corners. These 128-dimensional deep features provide high-quality input for subsequent decision-making. In the dataset tested by the present invention, this improvement significantly improves the accuracy and sensitivity of depression tendency classification. In summary, Path 1 ensures that the three-dimensional motion information of the face is fully mined, especially for the "subtle and continuous" depth signals related to depression, which are captured with high fidelity.

[0105] Path 2: The second feature extraction network for texture motion feature extraction (based on dynamic convolution and multi-head attention).

[0106] The second path uses preprocessed RGB color data as input to extract dynamic features of facial expressions at the texture level. Color videos contain rich details of facial expression changes, such as facial texture displacement caused by muscle movement, lighting changes, and skin color changes. This paper uses a convolutional neural network combined with an attention mechanism to automatically learn these dynamic texture features. To more effectively focus on changes in key areas of expression, this paper introduces dynamic convolution to enable the convolution kernel to adapt to the input, thereby highlighting the features of key areas such as the corners of the mouth and the area around the eyes.

[0107] The second feature extraction network is generally a shallow 2D convolutional network combined with an attention module. The second feature extraction network of the present invention includes a first temporal convolutional layer, a second temporal convolutional layer, and a multi-head attention mechanism layer connected in sequence.

[0108] First, an RGB sequence of size 256×256×900×3 is input, where 900 represents the number of frames and 3 represents the number of color channels. Using the time series frames as continuous input, texture motion features are extracted through two temporal convolutional layers: the first layer uses dynamic convolution with a kernel size of 3×3 and 32 output channels; the second layer also uses a 3×3 dynamic convolution with 64 output channels. A four-head self-attention layer further processes the features output by the previous layer, outputting 128-dimensional features. Specifically, multi-head attention maps the input feature representation into query (Q), key (K), and value (V) vectors, reweighting and integrating the features by calculating the similarity between different positions. This embodiment uses four parallel attention heads, each of which independently learns a 32-dimensional representation. The outputs of the four heads are then concatenated along the channel dimension to produce a final 128-dimensional texture motion feature. This multi-head mechanism can focus on different aspects of facial expressions from different subspaces. For example, one head focuses on mouth movement, while another head focuses on eye blinking. The high-dimensional representation finally obtained by fusion integrates information from multiple angles.

[0109] Furthermore, the dynamic convolution layer is one of the characteristics of this path. The convolution kernel parameters of traditional convolutional networks are fixed after training and are applied uniformly to all inputs during testing; dynamic convolution allows the convolution kernel to change in real time according to each input to improve adaptability to different facial expressions. The present invention pre-sets a set of benchmark convolution kernels in each dynamic convolution layer (for example, K = 4 3×3 basic convolution filters). For each frame of input features, the network predicts a set of weights {a1, a2, ..., aK} based on the overall features of the previous moment or the current frame through a small fully connected subnetwork. Subsequently, these weights are applied to the benchmark convolution kernel set to calculate the adaptive convolution kernel for the current frame, including the following operation process:

[0110] The fully connected sub-network predicts a set of weights based on the overall features of the previous frame; wherein, when the input is the first frame, the prediction is made based on the overall features of the current frame;

[0111] The adaptive convolution kernel of the current frame is obtained according to the weight, and its expression is:

[0112] ,

[0113] Among them, W dyn (t) is the convolution filter customized for the t-th frame, K is the number of basic convolution filters in the temporal convolution layer, B k represents the kth base convolution kernel, a k (t) is the corresponding weight coefficient; a k (t)∈{a1,a2,...,a K}.

[0114] The obtained W dyn (t) is a convolution filter customized for that frame, which performs a convolution operation with the input feature map to produce the output of the next layer. Through this mechanism, the network can adaptively emphasize changes in specific areas: for example, when the corners of the mouth drop, since the features of the previous frame change significantly in the corner area of ​​the mouth, the fully connected layer may give a higher weight to the convolution kernel for the mouth area, so that the convolution focuses on the corner features of the mouth; conversely, for expressionless frames, different convolution kernels may be averaged or given a more balanced weight. In short, dynamic convolution gives the model the ability to "focus on where" and can adjust the filter according to the input content to extract the most relevant texture changes.

[0115] The multi-head attention mechanism layer further enhances the network's ability to express complex expression patterns. Through self-attention, the features at each position will establish a dynamic relationship calculation with the features at other positions in the sequence. For facial expressions, this means that the model can automatically learn the association between expression changes in different regions: for example, when the eyebrows are detected to be raised, the attention mechanism may be associated with changes in the area around the eyes to capture blinking or eyelid movement; when the corners of the mouth are pulled down, it may be associated with the movement of the chin area. Multi-head attention allows the model to consider these associations from multiple angles simultaneously, so that the extracted texture motion features can more accurately reflect the emotions of facial expressions. In particular, the attention mechanism of this embodiment also works in the time dimension, which can capture repeated or continuous expression patterns over a period of time, as well as the correlation between different time frames. In this way, dynamic features such as continuous frowning or repeated blinking can be portrayed under the action of attention without being overwhelmed by averaging.

[0116] This embodiment combines dynamic convolution and multi-head attention mechanisms for RGB video expression feature extraction, which is a flexibility that traditional convolutional networks cannot achieve. Dynamic convolution gives the model the ability to adapt to input in real time, and multi-head attention gives the model global correlation modeling capabilities. The two work together to accurately capture depression-related expression changes - such as the duration of the mouth corners pulling down, the wandering of eye focus, and other subtle but critical texture dynamics. Compared with traditional networks with fixed convolution kernels, the present invention can more effectively distinguish between normal expression changes and abnormally persistent expression patterns, avoiding the limitations of fixed convolution that is poorly adapted to differences in expressions between different people. Through this enhanced texture feature extraction, the model obtains 128-dimensional high-discriminative texture motion features, providing another important perspective for the final judgment of depressive tendencies.

[0117] Furthermore, the dual-path feature extraction of the present invention utilizes an improved C3D network and a dynamic convolution + attention network to extract dynamic features of both depth and texture modalities, respectively. This architecture innovatively integrates three-dimensional geometric features with texture motion features, making up for the shortcomings of single-channel analysis. In particular, the improved C3D introduces residual connections, which solves the gradient vanishing problem in learning small changes in long sequences, enabling the network to capture extremely subtle facial depth changes (such as slight expressions); while dynamic convolution combined with attention gives texture feature extraction adaptive focusing capabilities, enabling it to focus on changes in key facial areas related to depression. In terms of technical effectiveness, the 128-dimensional depth features and 128-dimensional texture features provided by the two paths complement each other, ensuring that regardless of whether depressive tendencies manifest in facial deformation or facial muscle movement, corresponding features are efficiently extracted, significantly improving overall recognition accuracy.

[0118] c: Extracting a head posture signal from the RGBD data, and generating a head posture feature based on the head posture signal.

[0119] In addition to facial expressions, the present invention analyzes head posture behavior as an independent type of feature. Head posture (such as nodding, turning the head, or keeping the head down) is one of the important clues to depressive tendencies: depressed individuals often show reduced head movement or stiff movements, such as keeping their heads down for a long time without speaking. However, if the original head angle sequence is used directly, it is difficult to extract robust features due to the presence of high-frequency noise and individual differences. Therefore, the present invention introduces a frequency domain analysis method in this section to convert the head angle signal into frequency space for processing to highlight slow changes and filter out fast jitters. Figure 3 As shown in the figure, the whole process includes four steps: HopeNet posture extraction, frequency decomposition, time domain reconstruction and feature extraction:

[0120] c1: Head pose angle extraction and preprocessing: The HopeNet model is used to regress the head pose angle of each frame from the RGB frame sequence of the RGBD data, store it in a head angle sequence, and perform sliding window averaging on the head angle sequence.

[0121] The HopeNet model is a deep pose estimation model trained on a large amount of facial data. It accurately predicts the three Euler angles of the head from static images: pitch (nodding), yaw (turning the head left and right), and roll (sideways tilting of the head). Using HopeNet, this embodiment obtains a 900×3 raw angle time series P(n)=[Pitch(n), Yaw(n), Roll(n)] for a 900-frame sequence, where n=1, 2, ..., 900 represents the frame number. The raw angle sequence may contain slight jitter between frames (e.g., due to algorithm prediction errors or subtle, invisible movements of the subject). To reduce the interference of this high-frequency component on subsequent frequency domain analysis, the present invention preprocesses the angle sequence using a sliding window average: a sliding window of 5 frames is used to calculate the mean of the angles within each 5-frame window, replacing the value of the center frame (equivalent to a simple low-pass filter). This smooths out small, transient fluctuations, allowing the sequence to better reflect the overall posture trend. After this preprocessing, the head posture signal has basically eliminated high-frequency noise and retained the main change trends. For example, if the head continues to tilt downward for a period of time, the curve after sliding average will drop smoothly, and the momentary jitter of a frame will be smoothed out.

[0122] c2: Frequency decomposition (Fourier transform): Convert the head angle sequence to a posture sequence in the frequency domain, then perform a discrete Fourier transform on each dimension of the posture sequence to obtain the corresponding spectrum representation, and filter out a set proportion of the data with the highest frequency based on the spectrum representation.

[0123] The smoothed head angle sequence is converted to the frequency domain to separate different frequency components. The present invention performs a discrete Fourier transform (DFT) on each dimensional posture sequence (pitch, yaw, and roll are processed separately) to obtain the corresponding spectrum representation, which is expressed as follows:

[0124] ,

[0125] Where P(n) represents the value of the sequence in the nth frame, X(k) is the complex coefficient of the kth frequency component in the frequency domain, and N=900 is the total number of frames.

[0126] The DFT result X(k) contains information from frequency 0 (the DC component, corresponding to the average posture) to high-frequency components. For the purposes of this invention, high-frequency components primarily represent noise and rapid, small movements, while depressive behaviors often manifest as low-frequency components (for example, keeping the head down is a near-static state, corresponding to frequency components close to 0 Hz). Therefore, this embodiment sorts the frequency from low to high, retaining the top 80% (this ratio can be adjusted based on actual conditions; this embodiment uses 80%) of the frequency components with concentrated energy (i.e., the low-frequency components), and discarding the top 20% of frequencies to filter out high-frequency noise. Specifically, components with indices k > 0.8N in the spectrum are set to zero, retaining only those with indices k ≤ 0.8N. Here, 0.8N corresponds to a frequency of approximately 0.8 * (frame rate / 2) Hz, or approximately 12 Hz, which covers the primary frequency band of typical conscious head movements. Higher frequencies (such as rapid jitter above 14-15 Hz) are treated as noise and removed. After this step, this embodiment obtains a filtered frequency domain representation X′(k), which contains only information related to low-frequency behaviors.

[0127] c3: Time domain reconstruction and feature extraction: The filtered posture sequence is converted back to the time domain through inverse discrete Fourier transform to obtain a smooth head posture curve P′(n);

[0128] c4: extracting a head posture feature vector of the head posture curve; the head posture feature vector includes statistical features and periodic features.

[0129] The filtered frequency domain signal is converted back to the time domain through the inverse discrete Fourier transform (IDFT) to obtain a smooth head posture curve P′(n). Since the high-frequency noise has been removed, P′(n) is smoother and more continuous than the original sequence, and can intuitively reflect the subject's head movement trend. For example, if someone lowers their head for a long time, the Pitch channel of P′(n) will show a curve that slowly decreases and remains at a low angle, without high-frequency jitter spikes. Next, this embodiment extracts a variety of quantitative features from the reconstructed smooth posture signal for use in depressive tendency discrimination, including two categories: statistical features and periodic features. The statistical features include one or more of the mean, variance, and extreme value range; the periodic features include frequency domain features and / or behavioral pattern features, specifically:

[0130] Statistical features: Calculate global metrics for the posture sequence, such as the average of the three angles over the entire acquisition process (reflecting the overall posture, for example, whether the average head pitch angle is low), and variance (reflecting the magnitude of posture changes; low values ​​may indicate stiffness). Additionally, metrics such as extreme range (the difference between the maximum and minimum angles, indicating the range of motion) can be included. These features provide a quantification of the overall dynamics of head movement.

[0131] Frequency Domain Features: Identify indicators such as dominant frequency in the frequency domain. For example, by looking for peaks in the power spectral density, determine whether there are significant low-frequency periodic movement patterns. For individuals with depressive tendencies, strong peaks near 0 Hz may be present (indicating prolonged maintenance of a fixed posture), or very low dominant frequency components may appear in the pitch channel (corresponding to a low nodding frequency). This method records the peak frequency and power of each posture to indicate the presence of periodic movement and its frequency.

[0132] Behavioral pattern features: Specific pattern indicators are extracted for typical depressive behaviors. For example, the "head-down cycle" measures the average duration of a complete head-up cycle, from raising the head to lowering it and then raising it again. Alternatively, the cumulative time the head is below a certain angle threshold is calculated. A value significantly higher than normal may indicate abnormally long periods of head-down behavior. Similarly, the number and amplitude of left and right head turns can be counted. These features capture the frequency and persistence of specific behavioral patterns.

[0133] The above various features are combined to form a 64-dimensional head posture feature vector (the specific dimension can be adjusted according to the number of sub-features actually selected. The present invention assumes that a total of 64 indicators are extracted). The innovation lies in that the present invention does not simply use the original posture sequence or rough statistical values, but accurately extracts low-frequency behaviors through frequency domain filtering + reconstruction, and then supplemented by rich feature extraction, so that the information of the behavioral dimension of head posture can be fully and targetedly utilized. This method avoids the large amount of redundancy and noise that may be contained in traditional direct time domain analysis, and focuses on slow posture changes related to depression. For example, the present invention can clearly distinguish the abnormal behavior pattern of "keeping the head down and still for 1 minute", while traditional analysis may not be able to recognize this pattern or underestimate its importance due to interference from noise or short-term head-raising movements.

[0134] Through the above processing, the present invention obtains a set of parameters that can characterize the subject's head movement habits and characteristics. This set of head posture feature vectors, together with the aforementioned facial depth features and texture features, constitutes a multimodal information source, preparing for the final fusion decision.

[0135] Furthermore, this invention introduces, for the first time in the field of depression detection, a frequency domain analysis method to process head posture signals. Through DFT spectrum extraction and inverse transform, the present invention can accurately extract low-frequency behavioral patterns unique to individuals with depressive tendencies (such as prolonged periods of head bowing and slow movements), while filtering out high-frequency noise and irrelevant movements. Compared with conventional methods that directly utilize raw posture sequences, this innovation greatly improves the reliability and discriminability of head behavior features. The technical effect is reflected in the following: the model can correctly identify depressed individuals with subtle facial expressions but abnormal head behavior (or vice versa), and will not miss detections due to data noise or irrelevant movements, thereby improving the sensitivity (easier to detect those with depressive tendencies) and specificity (avoiding false positives due to general movements).

[0136] d: Fusing the depth change feature, the texture motion feature, and the head posture feature, and inputting the fused comprehensive feature into a feedforward neural network for classification regression, and outputting a depression tendency probability score of the face data to be detected.

[0137] After completing the facial dual-path feature extraction and head posture feature extraction, the next step of the present invention is to fuse the features of multiple modalities and obtain a comprehensive score for depressive tendencies. The difficulty of fusion lies in the fact that the dimensions and importance of features from different sources are different, and simple connection or averaging may not give full play to their complementary effects. Therefore, the present invention adopts a fusion strategy driven by the attention mechanism to dynamically adjust the weights of various features in the final decision. The fused features are then used through a feedforward neural network to predict the depressive tendency score. The specific process is as follows:

[0138] d1: Multimodal feature arrangement: The depth variation feature, the texture motion feature, and the head posture feature are converted into feature sequences of the same format through dimensionality adjustment.

[0139] The present invention combines three types of feature vectors - 128-dimensional depth feature F depth , 128-dimensional texture feature F texture , 64-dimensional head posture feature F head , adjust the dimensions so that they can be input into the attention module in a unified form. One feasible method is to reshape each feature vector into a sequence of the same length and then apply self-attention on the sequence. For example, a 128-dimensional vector is reshaped into a 16×8 matrix (i.e., a sequence of length 16, each element is 8 dimensions), and a 64-dimensional vector can be first increased to 128 dimensions by zero padding or linear transformation and then reshaped into a 16×8 form. In this way, all modal features are represented by a sequence F={f1,f2,...,f 16}, each element is a sub-feature representation of a uniform dimension (e.g. 8 dimensions).

[0140] d2: Attention fusion mechanism: The dimensionally adjusted feature sequence is input into the multi-head attention fusion module to obtain the fused comprehensive features;

[0141] The sorted sequence features are input into the multi-head attention fusion module. Similar to the self-attention in the texture path mentioned above, this embodiment uses h attention heads to process the sequence. Each attention head calculates its own query Q, key K, value V for the sequence, and then calculates the weighted sum using the following formula:

[0142] ,

[0143] where d is the scaling factor.

[0144] This attention operation will distribute weights between sub-features of different modalities to achieve information interaction and screening. Intuitively, attention fusion will automatically learn which features of which modalities are more useful for the final judgment: for example, for some subjects, the facial expression feature signal is obvious and the head posture does not change much, then the attention may give higher weight to the facial features; on the contrary, if a person has a slight expression but lowers his head for a long time, then the head posture features will be paid more attention. The multi-head mechanism allows the model to integrate from multiple angles at the same time. For example, one attention head may focus on the correlation between depth and texture features, and another head focuses on the complementary relationship between head posture and depth features. After multi-head attention calculation, the present invention obtains a unified comprehensive feature representation after fusion (which can be reshaped back to 128-dimensional vector form for input into subsequent networks).

[0145] d3: Depression tendency score prediction: The comprehensive features are input into the feedforward neural network for classification regression, and the depression tendency probability score of the face data to be detected is output.

[0146] The resulting 128-dimensional comprehensive feature vector is then fed into a feed-forward neural network (FFN) to predict and score depression propensity. The FFN can be designed as a series of fully connected layers with nonlinear activations, such as two hidden layers (with ReLU activation) and one output layer. The output layer uses an appropriate activation function to map numerical values ​​into probabilities between 0 and 1.

[0147] For example, the output score S can be interpreted as a probability or risk index for a subject's depressive tendencies. Values ​​closer to 1 indicate a greater confidence in the model regarding the subject's depressive-related behaviors; values ​​closer to 0 indicate a higher likelihood of normal behavior. This score can be directly used for screening decisions. For example, if the threshold is set at 0.5 and S ≥ 0.5, the subject is considered at high risk for depression and requires further professional evaluation.

[0148] Compared to traditional simple feature fusion (such as direct concatenation or weighted averaging), the attention fusion mechanism introduced in this invention significantly enhances the flexibility and expressiveness of fusion. While linear weighting can be considered a special case of the attention mechanism (i.e., fixed attention weights), the present invention's approach allows weights to be adaptively learned from data, meaning the model can automatically adjust its decision-making basis based on the feature distribution of different individuals. For example, for subjects with abnormal facial and head features, the attention mechanism may give higher weight to both, creating a synergistic effect. For subjects with a pronounced abnormality in only one modality, attention will emphasize the contribution of that modality and mitigate the interference of the other. This adaptive feature fusion ensures that the model achieves excellent recognition results for all manifestations of depressive tendencies. Technically, the present invention observed that the use of attention fusion improved the overall model accuracy, particularly reducing the false positive rate. This is because the model is less likely to be misled by occasional anomalies in a single modality and instead comprehensively considers multimodal information to make robust judgments. The resulting depressive tendency score can be used for preliminary screening in various application scenarios. For example, in a mental health app, users can record a video answering questions via their camera, and the system will generate a depression risk score. This method can also be used as a supplementary tool in clinical pre-screening to help identify individuals who may require special attention. This entire technical solution enables an objective, rapid, and comprehensive assessment of depressive tendencies in a non-invasive manner.

[0149] Example 3

[0150] This embodiment is a specific example of verifying the feasibility and effectiveness of the depression tendency detection method based on dynamic three-dimensional human faces described in the above embodiment. It includes the following contents:

[0151] 1. Source and annotation method:

[0152] The experiment used the self-constructed Mental Disorder 4D (MD4D) dataset, comprising 105 subjects, including 66 in the depression group and 39 in the control group. All subjects underwent a mental health screening using the MINI clinical structured interview tool and received gold-standard labels. Each subject recorded a 30-second video containing RGB images and synchronized depth images at a resolution of 640×640, a frame rate of approximately 17 fps, and a total of 900 frames. Informed consent was obtained for all samples before collection, and ethical review was conducted.

[0153] 2. Data preprocessing:

[0154] The pretreatment process proposed by the present invention includes:

[0155] (1) Normalize the nose tip reference depth and crop the facial area within 300mm;

[0156] (2) After the region is cropped, it is unified to 256×256 pixels using bilinear interpolation;

[0157] (3) Use linear interpolation to fill the frame sequence and Gaussian smoothing to reduce noise;

[0158] (4) Extract the head posture angle sequence (Pitch, Yaw, Roll), smooth it using a sliding window, and perform DFT decomposition and low-frequency reconstruction.

[0159] 3. Network structure and training settings:

[0160] The overall network consists of a residual C3D branch for the depth channel, a dynamic convolution + multi-head attention branch for the RGB channels, and a frequency modeling branch for HopeNet pose sequences. All features are fused and passed through the FFN to output a probability score.

[0161] The training parameters are as follows:

[0162] Optimizer: Adam;

[0163] Initial learning rate: 1e-4, dynamically adjusted using cosine annealing strategy;

[0164] Batch size: 8;

[0165] Epoch: 100 rounds, using early stopping strategy to prevent overfitting;

[0166] Loss function: classification uses cross entropy loss, regression uses MSE loss;

[0167] 4. Comparative Experiments and Results

[0168] A five-fold cross validation was performed on the MD4D dataset, and the results are as follows:

[0169]

[0170] In addition, the test results on an independent external validation set (MD4D+, a total of 53 subjects) are:

[0171] Accuracy: 84.91%; Sensitivity: 88.46%; Specificity: 81.48%.

[0172] 5. Summary of technical effects:

[0173] The method of the present invention performs excellently in the multimodal fusion depressive tendency detection task, especially in capturing micro-expression changes and low-frequency posture behavior patterns, verifying the feasibility and effectiveness of the technical solution.

[0174] Summary table of method improvements, innovations and technical effects

[0175]

[0176] Example 4

[0177] like Figure 4 As shown, a device for detecting depression based on dynamic three-dimensional faces includes at least one processor, a memory communicatively connected to the processor, and at least one input / output interface communicatively connected to the processor. The memory stores instructions executable by the processor, which are executed by the processor to enable the processor to perform the method for detecting depression based on dynamic three-dimensional faces described in the preceding embodiments. The input / output interface may include a display, a keyboard, a mouse, and a USB port for inputting and outputting data.

[0178] Furthermore, the depression tendency detection device based on dynamic three-dimensional face can be a desktop computer, mobile phone, tablet computer, wearable depression tendency detection device based on dynamic three-dimensional face, etc., which can perform depth information recognition.

[0179] Furthermore, the processor may include one or more processing cores. The processor utilizes various interfaces and circuits to connect various components within the dynamic 3D face-based depression detection device. By running or executing instructions, programs, code sets, or instruction sets stored in memory, and accessing data stored in memory, the processor performs various functions and processes data within the dynamic 3D face-based depression detection device. Optionally, the processor may be implemented in at least one hardware form: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem handles wireless communications. It is understood that the modem may also be implemented independently of the processor via a separate communications chip.

[0180] The memory may include random access memory (RAM) or read-only memory (ROM). The memory may be used to store instructions, programs, code, code sets, or instruction sets, such as the instructions or code sets used to implement the method for detecting depression tendency based on a dynamic three-dimensional face provided in an embodiment of the present application. The memory may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing each of the aforementioned method embodiments, and the like. The data storage area may also store data (e.g., image data, spectrogram data), etc., generated during use by the device for detecting depression tendency based on a dynamic three-dimensional face.

[0181] Those skilled in the art will understand that all or part of the steps of the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0182] When the integrated unit described above is implemented as a software functional unit and sold or used as a standalone product, it can also be stored in a computer-readable storage medium containing program code that can be invoked by a processor to execute the methods described in the above-mentioned method embodiments. Based on this understanding, the technical solutions of the embodiments of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (such as a personal computer, server, or network device) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium includes non-transitory computer-readable storage medium. The computer-readable storage medium has storage space for program code for executing any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code can be compressed, for example, in a suitable form.

[0183] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for detecting depression tendency based on dynamic three-dimensional human faces, characterized in that: The following steps are involved: a: Preprocess the acquired face data to be detected and output standardized RGBD data; b: feeding the RGBD data into two feature extraction networks in parallel; wherein the first feature extraction network is used to extract the depth variation features of the RGBD data, and the second feature extraction network is used to extract the texture motion features of the RGBD data; c: extracting a head posture signal from the RGBD data, and generating a head posture feature according to the head posture signal; d: fusing the depth change feature, the texture motion feature, and the head posture feature, and inputting the fused comprehensive feature into a feedforward neural network for classification and regression, thereby outputting a depression tendency probability score for the face data to be detected; Wherein, the pre-processing in a comprises the following steps: a1: Taking the nose tip as the reference, perform depth normalization on the depth map of the face data to be detected; a2: In the depth map after depth normalization, with the nose tip as the reference, retain the points whose depth is within the preset range and output them as the facial region. Then, perform bilinear interpolation between the depth map of the facial region and the corresponding RGB image. a3: Perform frame sequence alignment and smoothing on the time series of the face data to be detected, and output standardized RGBD data.

2. The method for detecting depression tendency based on dynamic three-dimensional face according to claim 1, characterized in that: a1 includes the following steps: a11: Select a frame of depth map, detect the position of the nose tip, and obtain the depth value d of the point nose (t); a12: Subtract the depth value d from the depth value of all pixels in the depth map of this frame nose (t); a13: Filter the processed depth map within a preset depth range and output the filtered depth map; a14: Loop through steps a11 to a13 until the depth maps of all frames are normalized.

3. The method for detecting depression tendency based on dynamic three-dimensional face according to claim 1, characterized in that: The first feature extraction network includes four groups of sequentially connected three-dimensional convolutional layers and residual connection layers, and a global average pooling layer; The convolution kernel of the three-dimensional convolution layer is 3×3×3, and the stride is 1; the output channels of the first to fourth three-dimensional convolution layers are 16, 32, 64, and 128, respectively.

4. The method for detecting depression tendency based on dynamic three-dimensional face according to claim 1, characterized in that: The second feature extraction network includes a first temporal convolution layer, a second temporal convolution layer, and a multi-head attention mechanism layer connected in sequence; The first temporal convolution layer uses dynamic convolution to extract texture motion features and has an output channel of 32; The second temporal convolution layer uses dynamic convolution to extract texture motion features and has 64 output channels; The multi-head attention mechanism layer is used to further process the features output by the previous layer and output 128-dimensional features.

5. The method for detecting depression tendency based on dynamic three-dimensional face according to claim 4, characterized in that: The first temporal convolution layer and the second temporal convolution layer include a fully connected sub-network; The following operating procedures are included: The fully connected sub-network predicts a set of weights based on the overall features of the previous frame; wherein, when the input is the first frame, the prediction is made based on the overall features of the current frame; The adaptive convolution kernel of the current frame is obtained according to the weight, and its expression is: ， Among them, W dyn (t) is the convolution filter customized for the t-th frame, K is the number of basic convolution filters in the temporal convolution layer, B k represents the kth base convolution kernel, a k (t) is the corresponding weight coefficient; a k (t)∈{a1,a2,...,a K }.

6. The method for detecting depression tendency based on dynamic three-dimensional face according to claim 1, characterized in that: Said c comprises the following steps: c1: regressing the head pose angle of each frame from the RGB frame sequence of the RGBD data through the HopeNet model, storing it in a head angle sequence, and performing sliding window averaging on the head angle sequence; c2: Convert the head angle sequence into a posture sequence in the frequency domain, perform a discrete Fourier transform on each dimension of the posture sequence to obtain a corresponding frequency spectrum representation, and filter out a set proportion of data with the highest frequency based on the frequency spectrum representation; c3: Convert the filtered posture sequence back to the time domain through inverse discrete Fourier transform to obtain a smooth head posture curve P′(n); c4: extracting a head posture feature vector of the head posture curve; the head posture feature vector includes statistical features and periodic features.

7. The method for detecting depression tendency based on dynamic three-dimensional face according to claim 6, characterized in that: The statistical features include one or more of a mean value, a variance, and an extreme value range; the periodic features include a frequency domain feature and / or a behavior pattern feature.

8. The method for detecting depression tendency based on dynamic three-dimensional face according to claim 1, characterized in that: Said step d comprises the following steps: d1: converting the depth change feature, the texture motion feature, and the head posture feature into feature sequences of the same format through dimensionality adjustment; d2: Input the dimension-adjusted feature sequence into the multi-head attention fusion module to obtain the fused comprehensive features; d3: Input the comprehensive features into a feedforward neural network for classification regression, and output a depression tendency probability score of the face data to be detected.

9. A depressive tendency detection device based on dynamic three-dimensional human face, characterized in that: The invention comprises at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method according to any one of claims 1 to 8.

Images