Anxiety and depression state recognition method and device based on video analysis and medium

Abstract

This invention discloses a method, device, and medium for identifying anxiety and depression states based on video analysis, comprising: acquiring a facial video of a subject; extracting keyframes from the facial video using a fixed-interval sampling algorithm to obtain a keyframe image set; extracting inter-frame dynamic change features from the keyframe image set based on the difference between adjacent frames; extracting facial key points from the keyframe image set based on facial key point localization technology, calculating the temporal dynamic parameters of the facial key points, and constructing key point dynamic features; fusing the inter-frame dynamic change features and the key point dynamic features, inputting the fused features into a classification network, and outputting the recognition result, wherein the recognition result includes: anxiety state, depression state, or normal state.

Description

Technical Field

[0001] This invention discloses a method, device, and medium for identifying anxiety and depression based on video analysis, belonging to the field of computer-aided technology. Background Technology

[0002] Existing anxiety and depression emotion recognition technologies mainly include three categories: static image recognition, speech signal analysis, and questionnaire assessment. Among them, static image schemes rely on facial geometric features and cannot capture temporal dynamic information and micro-expression changes. Speech schemes are easily affected by environmental noise and cannot reflect facial muscle tremors. Scale schemes are highly subjective and difficult to implement for large-scale rapid screening. Although there has been video-based emotion recognition research in recent years, most of it is used for general emotion classification. There is a lack of automated detection schemes for video dynamic features specifically for anxiety and depression. Moreover, existing video analysis methods usually use random sampling or continuous full-frame input, which fails to balance computational efficiency and the integrity of dynamic information. Summary of the Invention

[0003] The purpose of this application is to provide a method, device, and medium for identifying anxiety and depression based on video analysis, so as to at least solve the technical problem that existing emotion recognition methods are unable to achieve non-contact, real-time automatic identification of anxiety and depression.

[0004] According to one aspect of the present invention, in order to achieve the above-mentioned objective, a video analysis-based method for identifying anxiety and depression states is provided, comprising the following steps: acquiring a facial video of a subject; extracting keyframes from the facial video based on a fixed-interval sampling algorithm to obtain a keyframe image set; extracting inter-frame dynamic change features from the keyframe image set based on the difference between adjacent frames; extracting facial key points from the keyframe image set based on facial key point localization technology, calculating the temporal dynamic parameters of the facial key points, and constructing key point dynamic features; fusing the inter-frame dynamic change features and the key point dynamic features, inputting the fused features into a classification network, and outputting a recognition result, wherein the recognition result includes: anxiety state, depression state, or normal state.

[0005] Furthermore, the fixed-interval sampling algorithm extracts keyframes from the facial video at a preset uniform time interval, and the total number of keyframes is positively correlated with the duration of the facial video.

[0006] Furthermore, the step of extracting inter-frame dynamic change features from the keyframe image set based on the difference between adjacent frames is as follows: calculating the pixel difference value between adjacent frames to obtain the motion amplitude, calculating the local texture feature difference degree between adjacent frames to obtain the texture change amount, and combining the motion amplitude and texture change amount into inter-frame dynamic change features.

[0007] Furthermore, the pixel difference value is calculated using the inter-frame difference method, and the local texture feature difference degree is calculated using the local binary mode operator.

[0008] Furthermore, in the step of extracting facial key points from the keyframe image set based on facial key point localization technology: a deep learning-based target detection model is used to locate the face region, and a multi-point facial key point localization algorithm is used to extract multiple facial key points covering the eyebrows, eyes, nose, mouth and jaw contour.

[0009] Furthermore, in the step of calculating the temporal dynamic parameters of facial key points and constructing key point dynamic features: the spatial position offset of multiple facial key points between adjacent frames is calculated to obtain displacement; the ratio of displacement to the time interval between frames is calculated to obtain velocity; the frequency features of the position of multiple facial key points fluctuating periodically over time are extracted to obtain tremor frequency; and displacement, velocity and tremor frequency are combined into key point dynamic features.

[0010] Furthermore, the tremor frequency is a frequency characteristic within a predetermined physiological frequency band associated with the anxiety state.

[0011] Furthermore, the classification network is a lightweight convolutional neural network, and the fusion specifically involves concatenating the dynamic change features between frames and the dynamic features of key points into a multi-dimensional feature vector, which is then input into the lightweight convolutional neural network.

[0012] According to one embodiment of the present invention, a video analysis-based anxiety and depression state recognition device is also provided, comprising: The acquisition module is used to acquire facial videos of the test subjects and extract keyframes from the facial videos based on a fixed-interval sampling algorithm to obtain a set of keyframe images. The processing module is used to calculate and extract inter-frame dynamic change features from the keyframe image set based on the differences between adjacent frames; The extraction module is used to extract facial key points from the keyframe image set based on facial key point localization technology, calculate the temporal dynamic parameters of the facial key points, and construct the dynamic features of the key points. The analysis module is used to fuse inter-frame dynamic change features and key point dynamic features, input the fused features into the classification network, and output the recognition results, which include: anxiety state, depression state, or normal state.

[0013] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of the present invention.

[0014] The video analysis-based method, device, and medium for identifying anxiety and depression states provided by this invention have the following advantages compared to existing technologies: This application employs the following steps: reducing data redundancy and computational overhead through fixed-interval sampling; effectively capturing temporal dynamic features such as facial micro-expressions and muscle tremors by combining adjacent frame difference calculation with key point dynamic parameter extraction, thereby enhancing robustness against changes in lighting and facial expression faking; and adopting an end-to-end scheme that integrates feature input classification networks, significantly improving recognition accuracy and screening efficiency, making it suitable for large-scale rapid psychological state screening scenarios, thus solving the technical problem that existing emotion recognition methods are unable to achieve non-contact, real-time automatic recognition of anxiety and depression states. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of a video analysis-based method for identifying anxiety and depression states according to one embodiment of the present invention; Figure 2 This is a structural block diagram of an anxiety and depression state recognition device based on video analysis according to one embodiment of the present invention. Detailed Implementation

[0017] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application. It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0018] According to an embodiment of the present invention, an embodiment of a video analysis-based method for identifying anxiety and depression is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0019] This method embodiment can be executed in an electronic device or similar computing device that includes memory and a processor. Taking operation on a vehicle terminal as an example, the vehicle terminal may include one or more processors (processors may include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), digital signal processing (DSP) chips, microcontroller units (MCUs), field-programmable gate arrays (FPGAs), neural network processors (NPUs), tensor processors (TPUs), artificial intelligence (AI) type processors, etc.) and memory for storing data. Optionally, the vehicle terminal may also include transmission devices, input / output devices, and display devices for communication functions. Those skilled in the art will understand that the above structural description is illustrative and does not limit the structure of the vehicle terminal. For example, the vehicle terminal may include more or fewer components than described above, or have a different configuration than described above.

[0020] The memory can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the video analysis-based anxiety and depression state recognition method in this embodiment of the invention. The processor executes various functional applications and data processing by running the computer program stored in the memory, thereby realizing the aforementioned video analysis-based anxiety and depression state recognition method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0021] The transmission device is used to receive or send data via a network. Specific examples of the network mentioned above may include a wireless network provided by the mobile terminal's communication provider. In one example, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0022] Display devices can be, for example, touchscreen liquid crystal displays (LCDs) and touch displays (also referred to as "touchscreens" or "touch displays"). The LCD allows users to interact with the user interface of the mobile terminal. In some embodiments, the mobile terminal has a graphical user interface (GUI), which allows users to interact with the GUI through finger contact and / or gestures on a touch-sensitive surface. Optional human-computer interaction functions include: creating web pages, drawing, word processing, creating electronic documents, playing games, video conferencing, instant messaging, sending and receiving emails, call interfaces, playing digital video, playing digital music, and / or web browsing, etc. Executable instructions for performing the above human-computer interaction functions are configured / stored in one or more processor-executable computer program products or readable storage media.

[0023] Example 1 Figure 1 This is a flowchart of a video analysis-based method for identifying anxiety and depression states according to one embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps: Step S110: Acquire the facial video of the subject, and extract keyframes from the facial video based on a fixed-interval sampling algorithm to obtain a keyframe image set. The specific steps are as follows: In step S110, the first step is to acquire a video of the subject's face. This step uses a high-resolution RGB camera with a resolution of at least 1920×1080 pixels and a frame rate of at least 25 frames per second. The acquisition environment must be set to uniform indoor lighting conditions, avoiding direct sunlight or backlighting to reduce the impact of lighting changes on image quality. The subject must face the camera directly with their face unobstructed, and the distance between their face and the lens must be controlled within 50 to 70 centimeters. Throughout the acquisition process, the subject looks directly at the camera in a natural state, and a 30-second frontal facial video is continuously recorded.

[0024] Next, a fixed-interval sampling algorithm is performed on the acquired 30-second continuous video. Specifically, this algorithm uses a strategy of sampling one frame per second, that is, extracting one frame from the video stream every second. Since the total video duration is 30 seconds, a total of 30 frames are extracted as the initial keyframe candidate set. This fixed-interval sampling method can evenly cover the entire video time sequence, avoid missing key dynamic intervals that may be caused by random sampling, and effectively reduce data redundancy, laying the foundation for subsequent real-time feature extraction.

[0025] After extracting keyframes, each image frame needs to be preprocessed to improve image quality and standardize the input scale. The preprocessing process includes grayscale conversion, Gaussian filtering, histogram equalization, and size normalization. First, the color image is converted to grayscale to reduce computational complexity and preserve brightness information. Then, a Gaussian filter with a kernel size of 3×3 is used to smooth the grayscale image to suppress sensor noise and subtle texture interference. Next, histogram equalization is performed to enhance image contrast and reduce facial appearance differences under different lighting conditions. Finally, all image sizes are normalized to 256×256 pixels for subsequent network processing.

[0026] In practice, the extracted keyframes can also undergo quality screening. Screening criteria include face region integrity, image sharpness, and absence of abnormal occlusion. Face region integrity requires a complete face to be detected in the frame; image sharpness requires no motion blur or blurring; and absence of abnormal occlusion requires the face not to be obscured by hands, hair, or other objects. Frames that fail the screening can be replaced with adjacent qualified frames or simply discarded. Since fixed-interval sampling ensures a sufficient number of 30 keyframes, a small number of unqualified frames will not affect the overall integrity of dynamic feature extraction. Finally, the preprocessed and screened images together constitute a keyframe image set, used for subsequent inter-frame dynamic change calculations and facial landmark tracking.

[0027] Step S120: Based on the differences between adjacent frames, calculate and extract inter-frame dynamic change features from the keyframe image set. The specific steps are as follows: In step S120, the core objective is to capture subtle motion information of facial expressions along a continuous time axis, especially muscle tremors and facial transition features that cannot be captured by static images. In specific implementation, two temporally adjacent frames in the keyframe image set are used as processing units, and the inter-frame differences are calculated from two dimensions: pixel intensity changes and local texture changes.

[0028] First, pixel differencing is performed to obtain motion amplitude information. The pixel values ​​at corresponding positions in two adjacent frames are subtracted to obtain a difference image. The distribution area of ​​non-zero pixels in the difference image represents the area where motion occurs, and the absolute value of the pixel difference reflects the intensity of the motion. To eliminate pseudo-motion caused by noise, the difference image is thresholded, retaining only pixels with differences exceeding a predetermined threshold. Then, the sum of the differences of all valid pixels is calculated and normalized to a predetermined interval to obtain the quantized motion amplitude feature. This motion amplitude feature can effectively characterize typical dynamic manifestations of anxiety and depression, such as rapid eyebrow raising, subtle twitching of the corners of the mouth, and periodic tremors of the periorbital muscles.

[0029] Secondly, a Local Binary Mode (LoBMT) operator is used to compare texture features between adjacent frames to obtain the amount of texture variation. LoBMT is a texture descriptor robust to illumination changes; it generates binary codes by comparing the grayscale values ​​of neighboring pixels with the center pixel. For each frame in the keyframe image set, its LoBMT texture map is first calculated, and then the texture maps of adjacent frames are compared pixel-by-pixel, counting the number of pixels with inconsistent codes. This number directly reflects the stability of the local texture structure between adjacent frames. The number of pixels with inconsistent codes is normalized to obtain the amount of texture variation. This amount of texture variation is highly sensitive to changes in the micro-texture of the skin surface; for example, facial stiffness or subtle twitching caused by muscle tension can be effectively captured through the amount of texture variation.

[0030] After obtaining the motion amplitude and texture change, these two are combined to construct inter-frame dynamic change features. The specific combination method can be either direct concatenation of feature vectors or a weighted summation fusion strategy. To more comprehensively describe the dynamic process, motion amplitude sequences and texture change sequences from multiple pairs of adjacent frames can be further calculated, extracting statistical features, including the average motion amplitude, standard deviation of motion amplitude, average texture change, and standard deviation of texture change. Furthermore, time-frequency analysis can be performed on the motion amplitude sequences to extract motion frequency components, focusing on the tremor frequency band associated with anxiety. These statistical features and time-frequency features are then incorporated into the inter-frame dynamic change feature vector, forming a multi-dimensional dynamic feature representation. This inter-frame dynamic change feature is independent of facial keypoint information, focusing on pixel-level overall motion perception, providing complementary discriminative information for subsequent classification networks.

[0031] Step S130: Extract facial key points from the keyframe image set based on facial key point localization technology, calculate the temporal dynamic parameters of the facial key points, and construct key point dynamic features. The specific steps are as follows: In step 130, the aim is to obtain subtle motion information highly correlated with anxiety and depression by sparsely modeling local areas of the face, including the closing rate of the eyelids, the offset angle of the eyebrows, the tremor frequency of the corners of the mouth, and the slight swaying of the head.

[0032] First, a deep learning-based object detection model is used to locate the face region in each frame of the keyframe image set. Specifically, the YOLOv5s model is selected as the face detector. The size-normalized keyframe images are adjusted to 640×640 pixels and then input into the YOLOv5s model. A confidence threshold of 0.6 and a non-maximum suppression threshold of 0.5 are set, and the face bounding box position is output. For the detected face regions, a 68-point facial keypoint localization algorithm is further used to extract 68 facial keypoints covering the eyebrows, eyes, nose, mouth, and jawline. Each keypoint has two-dimensional coordinates in the image coordinate system and is arranged in a predetermined index order to form a keypoint sequence. For all 30 frames in the keyframe image set, the above detection and localization operations are performed sequentially to obtain 30 sets of facial keypoint coordinate sequences, each containing the positional information of 68 keypoints.

[0033] Next, various temporal dynamic parameters are calculated based on the temporal keypoint coordinate sequence. These parameters include displacement, velocity, and tremor frequency. Displacement is calculated using the Euclidean distance formula; for the same keypoint in two adjacent frames, its spatial offset in the planar coordinate system is calculated. Displacement is calculated sequentially for all 68 keypoints, resulting in displacement vectors. Taking keypoints in the eyebrow region as an example, frequent eyebrow elevation in an anxious state leads to a significant increase in displacement; taking keypoints in the corner of the mouth region as an example, the downward displacement of the corner of the mouth in a depressed state shows a continuous negative shift. Displacement values ​​reflect the degree of positional change in facial muscle activity and are the basis of dynamic features.

[0034] The velocity parameter is obtained by dividing the displacement by the time interval between adjacent frames. Since the fixed-interval sampling algorithm ensures a fixed time interval between every two frames, the velocity is calculated as the ratio of displacement to constant time. The velocity parameter reflects the speed of facial movements; rapid muscle twitching in an anxious state leads to an instantaneous increase in the velocity of local keypoints, while slowed movement in a depressed state is characterized by a general decrease in velocity. To improve the robustness of the features, the average velocity of each keypoint, as well as the maximum velocity, variance, and other statistical parameters, can be calculated.

[0035] Tremor frequency parameters are used to capture periodic muscle tremors and are a key indicator for distinguishing anxiety states. Time-frequency analysis is performed on the displacement sequence of each key point, specifically using short-time Fourier transform to convert the displacement over time to the frequency domain. From the transformed power spectrum, energy peaks within a predetermined physiological frequency band are extracted as tremor frequency features. This predetermined physiological frequency band is preferably 3 Hz to 8 Hz, as this band is highly correlated with anxiety-related muscle tremors. For example, significant oscillations of the orbicularis oculi or orbicularis oris muscles within 3 to 8 Hz usually indicate elevated anxiety levels. For the 68 key points, tremor-sensitive regions such as the lower eyelid, corner of the mouth, and glabella can be selected, and the frequency domain features of key points in these regions can be extracted to form a multidimensional tremor frequency vector.

[0036] The displacement, velocity, and tremor frequency parameters mentioned above are organized in temporal and spatial order to construct keypoint dynamic features. One specific construction method is to concatenate the displacement, velocity, and tremor frequency of all keypoints across all time sequences into a one-dimensional feature vector. Another method is to treat each keypoint as a signal source and extract multiple statistics such as the mean, variance, peak value, and dominant frequency in its displacement sequence to form a compact feature representation. Furthermore, the 68 keypoints can be grouped according to facial functional regions, and the mean and variance of the dynamic parameters within each group can be calculated to obtain region-level dynamic features. These features collectively describe the spatiotemporal motion patterns of the face during video capture, effectively capturing subtle facial expressions and physiological tremors related to anxiety and depression.

[0037] Finally, the constructed keypoint dynamic features are fused with the aforementioned inter-frame dynamic change features. The fusion operation can be performed by directly concatenating feature vectors along their dimensions to form a joint feature vector containing pixel-level motion information and semantic-level keypoint information. This joint feature vector is then fed into a lightweight convolutional neural network for classification, ultimately outputting the recognition result for anxious, depressed, or normal states. Through this method, this implementation fully utilizes the temporal dynamic information of facial key points, improving the accuracy and robustness of identifying anxious and depressed states.

[0038] Step S140: Fuse inter-frame dynamic change features and keypoint dynamic features, input the fused features into the classification network, and output the recognition result. The recognition result includes: anxiety state, depression state, or normal state. The specific steps are as follows: In step 140, after obtaining the inter-frame dynamic change features and key point dynamic features, the two are fused and the fused features are input into the classification network, and finally the three-class recognition results of anxiety state, depression state or normal state are output.

[0039] The fusion operation is performed at the feature level. Specifically, the inter-frame dynamic change feature vector and the keypoint dynamic feature vector are concatenated along the channel dimension to form a joint feature vector with a higher dimension. The inter-frame dynamic change feature focuses on pixel-level overall motion information, including the motion amplitude and texture changes between adjacent frames; the keypoint dynamic feature focuses on semantic-level local motion information, including the displacement, velocity, and tremor frequency of facial key points. The two types of features are complementary, and the joint expression can more comprehensively depict facial dynamics in anxious and depressed states. Let the dimension of the inter-frame dynamic change feature vector be M, and the dimension of the keypoint dynamic feature vector be N, then the dimension of the concatenated joint feature vector is the sum of M and N. This concatenation operation does not involve complex nonlinear transformations, maintains the original expressive power of the features, and reduces the computational cost of multimodal fusion.

[0040] The fused joint feature vector is used as input and fed into a pre-trained classification network for recognition. In this embodiment, the classification network uses a lightweight convolutional neural network. The network structure includes four convolutional layers, two pooling layers, and two fully connected layers connected in sequence. Each convolutional layer is followed by an activation function to introduce non-linearity. The pooling layers use max pooling to reduce feature dimensionality and improve translation invariance. The fully connected layers map high-dimensional features to the output space. The dimension received by the input layer is the same as the dimension of the joint feature vector, and the output layer contains three neurons, corresponding to the three categories of anxiety, depression, and normal state, respectively.

[0041] During the training phase of the classification network, the cross-entropy loss function is used as the optimization objective, and the Adam optimizer is used for parameter updates. Training samples consist of joint feature vectors obtained from collected facial videos after processing in the aforementioned steps, along with their corresponding labels. Labels are divided into three classes, represented using one-hot encoding. During training, the network learns a non-linear mapping from the joint feature vectors to the three class probability distributions. To balance the number of samples in each class, a class-weighted strategy can be used to increase the loss weight of the minority classes. After training, the network parameters are saved for subsequent inference phases.

[0042] In the actual recognition process, the facial video of the subject is processed sequentially according to the aforementioned steps to obtain a joint feature vector, which is then input into the trained classification network. After forward propagation, the network outputs probability values ​​for three categories. The category corresponding to the highest probability value is taken as the final recognition result. If the output probability of the anxious state is the highest, the subject is determined to be in an anxious state; if the output probability of the depressed state is the highest, the subject is determined to be in a depressed state; if the output probability of the normal state is the highest, the subject is determined to be in a normal state.

[0043] To improve the reliability of the recognition results, a probability threshold can be set. When the highest probability value is lower than the predetermined threshold, the output result is marked as uncertain, indicating that the video needs to be re-captured or a comprehensive judgment should be made in conjunction with other information. In addition, the output results can be post-processed, for example, by using a sliding window averaging method to smooth the recognition results of multiple consecutive video segments, in order to eliminate the occasional misjudgments that may occur in a single recognition.

[0044] Through the aforementioned fusion and classification steps, this implementation achieves end-to-end automatic identification from facial videos to states of anxiety and depression. The fusion strategy fully leverages the complementary advantages of pixel-level and keypoint-level dynamic information, while the lightweight convolutional network ensures computational efficiency. The three-classification output meets the practical needs of clinical screening. This implementation requires no contact with the subject and is suitable for large-scale, rapid screening in various scenarios such as hospitals, schools, and communities.

[0045] Based on steps S110 to S140 above, in this embodiment of the invention, data redundancy and computational overhead are reduced by sampling at fixed intervals. The combination of adjacent frame difference calculation and key point dynamic parameter extraction effectively captures temporal dynamic features such as facial micro-expressions and muscle tremors, enhancing robustness against changes in light and facial expression faking. At the same time, the end-to-end scheme of fusion feature input classification network is adopted, which significantly improves recognition accuracy and screening efficiency. It is suitable for large-scale rapid psychological state screening scenarios, thereby solving the technical problem that existing emotion recognition methods are difficult to achieve non-contact real-time automatic recognition of anxiety and depression states.

[0046] In a further embodiment of the present invention, the fixed-interval sampling algorithm extracts keyframes from the facial video at a preset uniform time interval, and the total number of keyframes is positively correlated with the duration of the facial video.

[0047] Furthermore, the step of extracting inter-frame dynamic change features from the keyframe image set based on the difference between adjacent frames is as follows: calculating the pixel difference value between adjacent frames to obtain the motion amplitude, calculating the local texture feature difference degree between adjacent frames to obtain the texture change amount, and combining the motion amplitude and texture change amount into inter-frame dynamic change features.

[0048] Furthermore, the pixel difference value is calculated using the inter-frame difference method, and the local texture feature difference degree is calculated using the local binary mode operator.

[0049] Furthermore, in the step of extracting facial key points from the keyframe image set based on facial key point localization technology: a deep learning-based target detection model is used to locate the face region, and a multi-point facial key point localization algorithm is used to extract multiple facial key points covering the eyebrows, eyes, nose, mouth and jaw contour.

[0050] Furthermore, in the step of calculating the temporal dynamic parameters of facial key points and constructing key point dynamic features: the spatial position offset of multiple facial key points between adjacent frames is calculated to obtain displacement; the ratio of displacement to the time interval between frames is calculated to obtain velocity; the frequency features of the position of multiple facial key points fluctuating periodically over time are extracted to obtain tremor frequency; and displacement, velocity and tremor frequency are combined into key point dynamic features.

[0051] Furthermore, the tremor frequency is a frequency characteristic within a predetermined physiological frequency band associated with the anxiety state.

[0052] Furthermore, the classification network is a lightweight convolutional neural network, and the fusion specifically involves concatenating the dynamic change features between frames and the dynamic features of key points into a multi-dimensional feature vector, which is then input into the lightweight convolutional neural network.

[0053] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0054] This invention also provides a video analysis-based anxiety and depression state recognition device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0055] Example 2 Figure 2 According to one embodiment of the present invention, an anxiety and depression state recognition device based on video analysis includes: The acquisition module 201 is used to acquire the facial video of the subject, extract key frames from the facial video based on a fixed interval sampling algorithm, and obtain a set of key frame images. Processing module 202 is used to calculate and extract inter-frame dynamic change features from the keyframe image set based on the differences between adjacent frames; Extraction module 203 is used to extract facial key points from key frame image set based on facial key point localization technology, calculate the temporal dynamic parameters of facial key points, and construct key point dynamic features; Analysis module 204 is used to fuse inter-frame dynamic change features and key point dynamic features, input the fused features into the classification network, and output the recognition results, which include: anxiety state, depression state or normal state.

[0056] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0057] Example 3 According to one embodiment of the present invention, an electronic device is also provided, comprising: a memory storing an executable program; and a processor for running the program, wherein the program executes the above-described video analysis-based anxiety and depression state recognition method during runtime.

[0058] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program: Step S1: Acquire the facial video of the subject, extract keyframes from the facial video based on a fixed interval sampling algorithm, and obtain a set of keyframe images; Step S2: Calculate and extract inter-frame dynamic change features from the keyframe image set based on the differences between adjacent frames; Step S3: Extract facial key points from the keyframe image set based on facial key point localization technology, calculate the temporal dynamic parameters of the facial key points, and construct the dynamic features of the key points. Step S4: Fuse the inter-frame dynamic change features and key point dynamic features, input the fused features into the classification network, and output the recognition results, which include: anxiety state, depression state, or normal state.

[0059] Example 4 According to one embodiment of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the storage medium is located to execute the above-described video analysis-based anxiety and depression state recognition method.

[0060] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps: Step S1: Acquire the facial video of the subject, extract keyframes from the facial video based on a fixed interval sampling algorithm, and obtain a set of keyframe images; Step S2: Calculate and extract inter-frame dynamic change features from the keyframe image set based on the differences between adjacent frames; Step S3: Extract facial key points from the keyframe image set based on facial key point localization technology, calculate the temporal dynamic parameters of the facial key points, and construct the dynamic features of the key points. Step S4: Fuse the inter-frame dynamic change features and key point dynamic features, input the fused features into the classification network, and output the recognition results, which include: anxiety state, depression state, or normal state.

[0061] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0062] According to one embodiment of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the above-described video analysis-based anxiety and depression state recognition method.

[0063] Example 5 Optionally, in this embodiment, the above-mentioned computer program product can be configured as a computer program that performs the following steps: Step S1: Acquire the facial video of the subject, extract keyframes from the facial video based on a fixed interval sampling algorithm, and obtain a set of keyframe images; Step S2: Calculate and extract inter-frame dynamic change features from the keyframe image set based on the differences between adjacent frames; Step S3: Extract facial key points from the keyframe image set based on facial key point localization technology, calculate the temporal dynamic parameters of the facial key points, and construct the dynamic features of the key points. Step S4: Fuse the inter-frame dynamic change features and key point dynamic features, input the fused features into the classification network, and output the recognition results, which include: anxiety state, depression state, or normal state.

[0064] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0065] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0066] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0067] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0068] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0069] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0070] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for identifying anxiety and depression states based on video analysis, characterized in that, Includes the following steps: Acquire facial video of the subject, extract keyframes from the facial video based on a fixed interval sampling algorithm, and obtain a keyframe image set; Based on the difference between adjacent frames, dynamic change features between frames are extracted from the keyframe image set; Facial key points are extracted from the keyframe image set based on facial key point localization technology, the temporal dynamic parameters of the facial key points are calculated, and the dynamic features of the key points are constructed. The inter-frame dynamic change features and the key point dynamic features are fused together, and the fused features are input into a classification network to output recognition results, wherein the recognition results include: anxiety state, depression state or normal state.

2. The method according to claim 1, characterized in that, The fixed-interval sampling algorithm extracts keyframes from the facial video at a preset uniform time interval, and the total number of keyframes is positively correlated with the duration of the facial video.

3. The method according to claim 1, characterized in that, The step of extracting inter-frame dynamic change features from the keyframe image set based on the difference between adjacent frames is as follows: The pixel difference between adjacent frames is calculated to obtain the motion amplitude, the local texture feature difference between adjacent frames is calculated to obtain the texture change amount, and the motion amplitude and the texture change amount are combined to form the inter-frame dynamic change feature.

4. The method according to claim 3, characterized in that, The pixel difference value is calculated using the inter-frame difference method, and the local texture feature difference degree is calculated using the local binary mode operator.

5. The method according to claim 1, characterized in that, The step of extracting facial key points from the keyframe image set based on facial key point localization technology involves: using a deep learning-based target detection model to locate the face region, and using a multi-point facial key point localization algorithm to extract multiple facial key points covering the eyebrows, eyes, nose, mouth, and jaw contour.

6. The method according to claim 5, characterized in that, In the step of calculating the temporal dynamic parameters of the facial key points and constructing the dynamic features of the key points: the spatial position offset of the multiple facial key points between adjacent frames is calculated to obtain the displacement; the ratio of the displacement to the time interval between frames is calculated to obtain the velocity; the frequency features of the position of the multiple facial key points fluctuating periodically with time are extracted to obtain the tremor frequency; and the displacement, the velocity and the tremor frequency are combined to form the dynamic features of the key points.

7. The method according to claim 6, characterized in that, The tremor frequency is a frequency characteristic within a predetermined physiological frequency band associated with an anxiety state.

8. The method according to claim 1, characterized in that, The classification network is a lightweight convolutional neural network, and the fusion specifically involves concatenating the inter-frame dynamic change features and the keypoint dynamic features into a multi-dimensional feature vector, which is then input into the lightweight convolutional neural network.

9. A video analysis-based anxiety and depression state recognition device, characterized in that, include: The acquisition module is used to acquire the facial video of the subject and extract keyframes from the facial video based on a fixed interval sampling algorithm to obtain a keyframe image set. The processing module is used to calculate and extract inter-frame dynamic change features from the keyframe image set based on the differences between adjacent frames; The extraction module is used to extract facial key points from the keyframe image set based on facial key point localization technology, calculate the temporal dynamic parameters of the facial key points, and construct the dynamic features of the key points. The analysis module is used to fuse the inter-frame dynamic change features and the key point dynamic features, input the fused features into the classification network, and output the recognition results, wherein the recognition results include: anxiety state, depression state, or normal state.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the video analysis-based anxiety and depression state recognition method according to any one of claims 1 to 8.

Images