Human body state detection method, system and device based on facial action and medium

Abstract

The invention relates to a human body state detection method, system and device based on facial actions and a medium, and the method comprises the steps: obtaining an image sequence of a human face, carrying out the human face detection and mark point positioning of the image sequence, and obtaining the coordinates of a face mark point; performing micro-expression action extraction on the image sequence based on the facial mark point coordinates to obtain facial micro-expression action features; performing time sequence analysis on the facial action features to obtain time sequence facial action features; performing multi-scale illumination adjustment on the time sequence facial action features to obtain illumination facial action features; performing multi-modal analysis on the illumination facial action features through a pre-trained long-short term memory network to obtain comprehensive human body state features; and performing real-time state analysis based on the comprehensive human body state characteristics to obtain a human body state detection result. According to the method, different types of facial action features can be comprehensively processed, and the reliability of a detection result is enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human state detection, and particularly relates to a human state detection method, system and device based on facial action and a medium. BACKGROUND

[0002] Human state detection is a key technology in the fields of human-computer interaction, affective computing and health monitoring, and has received extensive attention in recent years. With the rapid development of computer vision and artificial intelligence technology, human state detection methods based on facial action have gradually become a research hotspot. This method can capture more detailed and comprehensive physiological and psychological state information of the human body by analyzing facial expressions and micro-expression changes. However, existing human state detection methods often have some limitations. Most methods only focus on static facial features or a single expression type, ignoring the subtle changes in facial action. In addition, the influence of environmental light changes on facial feature extraction is often ignored, which affects the stability and accuracy of the detection results. SUMMARY

[0003] The main purpose of the present application is to provide a human state detection method, system and device based on facial action, which can comprehensively process different types of facial action features and enhance the reliability of the detection results.

[0004] To achieve the above purpose, the present application provides a human state detection method based on facial action, comprising: obtaining an image sequence of a face, performing face detection and landmark positioning on the image sequence to obtain face landmark coordinates; extracting micro-expression action from the image sequence based on the face landmark coordinates to obtain facial micro-expression action features; performing time series analysis on the facial action features to obtain time series facial action features; performing multi-scale light adjustment on the time series facial action features to obtain light facial action features; performing multi-modal analysis on the light facial action features through a pre-trained long short-term memory network to obtain comprehensive human state features; performing real-time state analysis based on the comprehensive human state features to obtain human state detection results.

[0005] Further, the obtaining an image sequence of a face, performing face detection and landmark positioning on the image sequence to obtain face landmark coordinates comprises: frame extraction is performed on a video stream input by a preset camera device to obtain the image sequence of the face; face detection is performed on the image sequence through a cascade classifier to obtain a face region; cropping the image sequence according to the face region to obtain a face image; extracting features from the face image according to a preset edge detection algorithm to obtain a face contour region; positioning the face contour region according to a preset landmark positioning rule to obtain the face landmark coordinates.

[0006] Further, the micro-expression action extraction from the image sequence based on the face landmark coordinates to obtain face micro-expression action features includes: dividing the face region according to the face landmark coordinates to obtain a plurality of face micro-regions; calculating the optical flow field of the plurality of face micro-regions to obtain face optical flow field features; analyzing the motion of the face micro-regions according to the face optical flow field features to obtain a face micro-region motion information map; segmenting the face micro-region motion information map according to a preset segmentation rule to obtain a face micro-expression action region; extracting three-dimensional features from the face micro-expression action region to obtain action feature description information; performing time series clustering according to the action feature description information to obtain a face micro-expression action class; performing probability statistics on the face micro-expression action class to obtain action probability distribution information; performing feature description processing according to the action probability distribution information to obtain the face micro-expression action features, and performing geometric feature calculation to obtain a face geometric feature vector; analyzing the motion of the face landmark coordinates to obtain a face landmark motion trajectory; performing action description construction according to the face geometric feature vector and the face landmark motion trajectory to obtain corresponding action feature information; performing face action classification on the action feature information according to a preset action classification rule to obtain a face action class; performing feature fusion on the action feature information and the face action class to obtain the face action features.

[0007] Further, the time series analysis on the face action features to obtain time series face action features includes: segmenting the face action features into a plurality of time window face action feature sequences; performing Fourier transform on the face action feature sequence in each time window to obtain frequency domain features; perform time series correlation analysis on the face action feature sequence according to the frequency domain feature, to obtain a time correlation feature of the face action; perform time series interpolation on the face action feature according to the time correlation feature, to obtain the face action feature sequence; perform time series state transition matrix construction on the face action feature sequence, to obtain state transition data of the face action; perform correction on the face action feature sequence according to the state transition data, to obtain the time series face action feature.

[0008] Further, the multi-scale light adjustment on the time series face action feature to obtain a light face action feature comprises: perform wavelet transform on the time series face action feature, to obtain a multi-scale decomposition result; perform subset construction on the multi-scale decomposition result according to a preset feature recognition rule, to obtain face action feature subsets of different scales; perform light-invariant texture feature extraction on each of the face action feature subsets according to a local binary pattern algorithm, to obtain light-invariant feature descriptors; perform equalization processing on each of the face action feature subsets according to the light-invariant feature descriptors, to obtain a plurality of feature compensation subsets after light compensation; perform multi-scale fusion on all the feature compensation subsets, to obtain a fused light feature set; perform light component separation correction on the fused light feature set according to a retinex algorithm, to obtain a light consistency correction result; perform correction processing on the light consistency correction result and the fused light feature set, to obtain the light face action feature.

[0009] Further, the multi-modal analysis on the light face action feature by the pre-trained long short-term memory model to obtain a comprehensive human state feature comprises: perform input sequence segmentation on the light face action feature, to obtain a face action input sequence; perform state matching on the face action input sequence according to a preset multi-modal state database, to obtain corresponding multi-modal state data; input the face action input sequence and the multi-modal state data into a first layer of the long short-term memory model, wherein: in the first layer long short-term memory network, perform feature splicing operation on the face action input sequence and the multi-modal state data, to obtain a fused feature; perform forget gate processing on the fused feature, to obtain a forget gate output; input gate processing is performed on the forgetting gate output to obtain an input gate output; The input gate output is combined with a current candidate memory cell state to obtain an updated memory cell state; Output gate processing is performed on the updated memory cell state to obtain a first layer output feature; The first layer output feature is transmitted as input to a second layer long short-term memory network, and the processing process of the first layer long short-term memory network is repeated; In each layer long short-term memory network, the output feature of the previous layer is subjected to a feature concatenation operation with the face action input sequence and the multi-modal state data to obtain the fusion feature of the corresponding layer, and the processing process of the first layer long short-term memory network is repeated; When the processing process of the last layer long short-term memory network is completed, the output feature of the last layer long short-term memory network is subjected to a fully connected layer processing, and the comprehensive human state feature is output.

[0010] Further, real-time state analysis is performed based on the comprehensive human state feature to obtain a human state detection result, including: The comprehensive human state feature is subjected to a frequency domain transformation to obtain a time window feature and a frequency domain human feature; The frequency domain human feature is subjected to a feature enhancement processing to obtain a corresponding multi-scale human feature; The multi-scale human feature is subjected to a nonlinear transformation to obtain a state feature vector; The state feature vector is classified according to a preset state category to obtain a state classification result; The state classification result is subjected to state prediction according to a preset state transition probability matrix to obtain the human state detection result.

[0011] The application also provides a human state detection system based on facial actions, which is applied to the human state detection method based on facial actions in any of the above aspects, and includes: A collection module is configured to acquire an image sequence of a human face, perform face detection and landmark positioning on the image sequence, and obtain face landmark coordinates; An analysis module is configured to extract micro-expression action features from the image sequence based on the face landmark coordinates; An association module is configured to perform time series analysis on the face action features to obtain time series face action features; A processing module is configured to perform multi-scale illumination adjustment on the time series face action features to obtain illumination face action features; A control module is configured to perform multi-modal analysis on the illumination facial action features by a pre-trained long short-term memory network to obtain comprehensive human state features. An execution module is configured to perform real-time state analysis based on the comprehensive human state features to obtain a human state detection result.

[0012] The application further provides a human state detection device based on facial actions, comprising: A memory is configured to store a program. A processor is configured to execute the program to implement each step of the human state detection method based on facial actions according to any one of the preceding embodiments.

[0013] The application further provides a medium storing computer instructions configured to enable a computer to execute the method according to any one of the preceding embodiments.

[0014] The application provides the following beneficial effects: By locating the landmark points and extracting the micro-expression actions of the face image sequence, more subtle and rich facial expression changes can be captured, thereby improving the accuracy and sensitivity of human state detection. The time series analysis of the facial action features effectively utilizes the dynamic information of the facial expressions, overcomes the limitations of relying only on static features, and makes the state detection result more comprehensive and accurate. The multi-scale illumination adjustment technology effectively reduces the influence of environmental illumination changes on facial feature extraction, improves the robustness and adaptability of the detection method under different illumination conditions. The multi-modal analysis by the pre-trained long short-term memory network realizes the comprehensive processing of different types of facial action features, and enhances the reliability and generalization ability of the detection result. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a human state detection method flowchart based on facial actions provided by the application; Figure 2 is a human state detection system structure diagram based on facial actions provided by the application; Figure 3 is a human state detection device structure diagram based on facial actions provided by the application.

[0016] The application objectives, functional features and advantages will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0017] In order to make the objectives, technical solutions and advantages of the application clearer, the application will be further described in detail below with reference to the embodiments and the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application.

[0018] The application will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0019] Referring to Figure 1 The application provides a human state detection method based on facial actions, including: Step S1: Obtain an image sequence of a human face, perform face detection and landmark positioning on the image sequence, and obtain facial landmark coordinates; Step S2: Extract micro-expression actions based on the facial landmark coordinates, and obtain facial micro-expression action features; Step S3: Perform time series analysis on the facial action features, and obtain time series facial action features; Step S4: Perform multi-scale illumination adjustment on the time series facial action features, and obtain illumination facial action features; Step S5: Perform multi-modal analysis on the illumination facial action features through a pre-trained long short-term memory network, and obtain comprehensive human state features; Step S6: Perform real-time state analysis based on the comprehensive human state features, and obtain human state detection results.

[0020] Based on the above steps, the detailed step process is shown as follows: Step S1: Capture multiple frames of human face images continuously through a camera or other image capture device to form an image sequence. These images need to be captured within a short period of time to ensure the continuity and consistency of facial expressions and actions.

[0021] In the obtained image sequence, a face detection algorithm (such as Haar cascade classifier, Dlib library or deep learning model such as MTCNN) is applied to identify and locate the region of the face. Obtain an image sequence of a human face, perform face detection and landmark positioning on the image sequence, and obtain facial landmark coordinates; Perform landmark positioning on the detected face region, which usually includes key points such as eyes, nose, mouth, eyebrows and facial contours.

[0022] For each image, record the coordinates of each landmark. In this way, the entire image sequence will have a set of time series landmark coordinate data. These data will serve as the basis for extracting micro-expression action features in subsequent steps.

[0023] Step S2: Expression action refers to the short and subtle expression changes of the face, usually lasting within 0.5 seconds. These actions are unconscious and can reflect the true emotions and psychological state of a person.

[0024] Using landmark coordinates, the landmarks in the image sequence are analyzed for temporal changes, observing the movement paths and changes of each landmark over time. By calculating the displacement, velocity, and acceleration of the landmarks, features of micro-expression actions are extracted. These features include local expression changes (such as the raising of the corners of the mouth or the raising of the eyebrows) and global expression changes (such as the contraction of facial muscles).

[0025] The coordinate information of each landmark over time is converted into a feature vector, which includes the displacement, angular change, velocity change, etc. of the landmark. Using a sliding window technique, the changes of the landmarks are analyzed in segments to capture short-lived micro-expression actions.

[0026] By analyzing and extracting landmark changes, feature vectors are generated to describe facial micro-expression actions. These feature vectors will serve as input data for temporal analysis.

[0027] Step S3: Divide the continuous facial action feature data into multiple overlapping time windows. The size of the window is determined according to the typical duration of micro-expressions, usually between 0.5 seconds and 3 seconds.

[0028] Within each time window, statistical quantities of facial action features are calculated. These statistics include: Mean: reflects the average level of the feature within the window, Standard deviation: indicates the degree of variation of the feature, Maximum and minimum: capture the extreme value of the feature, Kurtosis and skewness: describe the shape of the feature distribution.

[0029] Using a simple linear regression method, the trend of facial action features over a longer time range is analyzed. The first and second differences of the features are calculated to capture the rate of change and acceleration of facial actions. These trend information reflects the overall direction and intensity of facial expression changes.

[0030] Perform a Fast Fourier Transform (FFT) on the facial action features to analyze their characteristics in the frequency domain. Extract the main frequency components, which may correspond to different types of facial actions or expression changes. Frequency domain analysis can reveal the periodic patterns and complex temporal structure of facial actions.

[0031] Calculate the correlation coefficients between different facial region action features, such as the correlation between eye and mouth actions. Analyzing the changes of these correlations over time can reveal the patterns of coordinated movement of different parts of the face.

[0032] Integrate the results of the above various temporal analysis to form a comprehensive temporal facial action feature vector. This feature vector should include information in the time and frequency domains, as well as the relationships between different facial regions.

[0033] The final output of temporal facial action features should be a multi-dimensional vector, where each dimension represents a specific characteristic of the facial action over time. These features can fully describe the temporal evolution of facial actions, including short-term micro-expressions and long-term expression changes.

[0034] Step S4: Preliminary assessment of changes in lighting conditions is performed by calculating the brightness histogram of each frame. The brightness histogram shows the distribution of pixel brightness in the image, thus helping to identify uneven or sudden changes in lighting conditions.

[0035] Based on the information from facial landmark locations, we extract brightness histograms of key facial regions (such as the eyes, mouth, and forehead) to further refine the analysis of lighting changes. We also calculate the brightness changes of each frame in the entire image sequence to facilitate lighting adjustments in subsequent steps.

[0036] Decompose the image into an image pyramid of different scales, with each scale representing a different spatial frequency. This allows for separate treatment of detail and overall illumination issues. For each scale, apply local contrast enhancement techniques (such as local histogram equalization) to correct for localized illumination unevenness.

[0037] Global illumination equalization techniques perform corrections on the highest-scale image, such as gamma correction or global histogram equalization, to adjust the overall illumination.

[0038] Within each time window, local illumination equalization is performed on different facial regions (such as eyes and mouth). Local histogram equalization is a commonly used method that adjusts the brightness distribution of an image to make the contrast of the image more uniform.

[0039] Apply global illumination equalization techniques, such as Laplace transform or gamma correction, to ensure uniform illumination distribution across the entire image sequence.

[0040] Extract illumination-invariant temporal facial action features from the resized images. These features should reflect the true variations of facial actions and not be affected by illumination changes.

[0041] Compare the features before and after adjustment to ensure the accuracy of the illumination-invariant features. If necessary, further optimize the illumination adjustment parameters.

[0042] The final output illumination facial action feature is a multi-dimensional vector adjusted by illumination, where each dimension represents a specific characteristic of the facial action in time.

[0043] Step S5: Import a pre-trained long short-term memory (LSTM) model. This model has been trained on a large-scale facial action dataset and is capable of recognizing and classifying various facial action features.

[0044] The light facial action features are input into the LSTM model. Ensure that the time series format of the input data meets the requirements of the LSTM model, usually requiring serialization processing of the data.

[0045] The LSTM model not only processes facial action features, but also integrates information from other modalities, such as audio features, posture features, etc.

[0046] If there is other modal data, input these data together with facial action features into the model for multi-modal feature fusion.

[0047] In the LSTM model, the input light facial action features are learned and represented layer by layer.

[0048] The model will capture the pattern of facial action features changing over time, generating a comprehensive human state feature vector.

[0049] The output of the LSTM model is a multi-dimensional comprehensive human state feature vector, which can be interpreted as different human states, such as emotional state, fatigue state, attention level, etc.

[0050] The final output of the comprehensive human state feature is a high-dimensional vector, each dimension representing a certain human state information.

[0051] Step S6: Real-time analysis of comprehensive human state features.

[0052] Define a set of rules and thresholds for classifying comprehensive human state features into different states. For example, rules can be developed to judge fatigue, stress, emotional fluctuations, etc.

[0053] According to the dimensions of the features, the human state is analyzed in multiple dimensions. For example, combining facial expression features and heart rate changes to judge stress state.

[0054] Real-time monitoring of input facial action data, dynamically updating comprehensive human state features.

[0055] Ensure that the system can quickly respond to changes in input data and update the state detection results in real time. Use sliding window technology to refresh the detection results in time when new data arrives.

[0056] Output the human state detection results, including the category and confidence of the state. For example, output "fatigue", "high stress", "emotional fluctuations", etc.

[0057] Provide corresponding explanations and suggestions, such as reminding to rest, adjusting emotions, etc. The explanation information is based on the dimensions of the detection results, helping users understand their state.

[0058] According to the user's feedback, the state classification rule and threshold are adjusted. According to the accuracy of the detection result and the information of the user's feedback, the classification rule and threshold are automatically adjusted to improve the accuracy and response speed of the detection.

[0059] The final output detection result includes the current state of the human body and its trend. These results can be stored in a database and displayed to the user.

[0060] The present application provides a human state detection method based on facial action, which can capture more subtle and rich facial expression changes by locating landmark points and extracting micro-expression actions from a sequence of face images, thereby improving the accuracy and sensitivity of human state detection. Time series analysis of facial action features effectively utilizes the dynamic information of facial expressions, overcoming the limitations of relying only on static features, making the state detection results more comprehensive and accurate. Multi-scale illumination adjustment technology is adopted to effectively reduce the impact of environmental light changes on facial feature extraction, improving the robustness and adaptability of the detection method under different lighting conditions. Through the pre-trained long short-term memory network for multi-modal analysis, the comprehensive processing of different types of facial action features is realized, enhancing the reliability and generalization ability of the detection results.

[0061] In one embodiment, a sequence of images of a face is obtained, and face detection and landmark point positioning are performed on the sequence of images to obtain face landmark point coordinates, including: A video stream of a face is collected by a camera device. The camera device can be a normal camera, an infrared camera, or a depth camera, etc. A series of image sequences of the face are obtained by frame extraction of the collected video stream. The frame extraction frequency is set according to actual needs, for example, 10 frames or 30 frames per second, etc.

[0062] A cascade classifier is applied to the extracted image sequence for face detection. A cascade classifier is used to detect whether there is a face in the image and to locate the approximate area of the face. The detection result outputs the rectangular box coordinates of the face area.

[0063] Based on the face detection result, the original image sequence is cropped to extract images containing only the face area. The size of the cropped face image can be uniformly adjusted to a preset size, such as 224x224 pixels, etc.

[0064] Edge detection algorithm is used to extract facial contour features from the cropped face image. Common edge detection algorithms include Canny operator, Sobel operator, etc. Edge detection can highlight the contour information of key parts such as face contour, eyes, nose, and mouth, providing a basis for subsequent landmark point positioning.

[0065] After obtaining the face contour region, the position coordinates of the face key points are determined according to a preset landmark point positioning rule. The landmark points usually include eye corners, nose tips, mouth corners and other face feature points, and generally 68 or 81 points are selected. The positioning rule can be based on a shape model such as an active shape model (ASM) or an active appearance model (AAM). The positioning result outputs the two-dimensional coordinate value of each landmark point in the image, which is the face landmark point coordinate.

[0066] The embodiment can realize real-time tracking of the face by frame extraction and face detection on the video stream, adapt to face posture changes in different scenes, and improve the robustness of detection. The use of a cascade classifier for face detection can quickly and accurately locate the face region, improving the efficiency of subsequent processing. Cropping and edge detection of the face image can remove background interference and highlight the facial contour features, providing a reliable foundation for landmark point positioning. The landmark point positioning method based on the preset rule can accurately obtain the face key point coordinates, providing high-quality input data for analyzing face movements and expression changes. The entire detection process is highly automated and does not require human intervention, enabling continuous and real-time human state monitoring.

[0067] In one embodiment, micro-expression action extraction is performed on the image sequence based on the face landmark point coordinates to obtain face micro-expression action features, including: The input image sequence is divided into face regions according to the obtained face landmark point coordinates. Face region division refers to dividing the entire face region into multiple sub-regions, such as eyebrow regions, eye regions, nose regions, and mouth regions. After division, multiple face micro-regions are obtained, each corresponding to a local region of the face.

[0068] The multiple face micro-regions obtained by division are subjected to optical flow field calculation to obtain face optical flow field features. Optical flow field calculation refers to analyzing the motion of pixel points between adjacent frames to obtain the motion vector of each pixel point. The face optical flow field features reflect the motion of each region of the face.

[0069] According to the calculated face optical flow field features, motion analysis is performed on each face micro-region to obtain a face micro-region motion information map. Motion analysis includes calculating the average motion amplitude, direction and other information of the pixel points in each micro-region. The face micro-region motion information map visually displays the motion state of each micro-region.

[0070] According to a preset segmentation rule, the face micro-region motion information map is segmented to obtain a face micro-expression action region. The preset segmentation rule includes motion amplitude threshold, connected region size and other conditions for determining which regions have micro-expression actions. The face micro-expression action region obtained after segmentation is the face region where micro-expression changes occur.

[0071] The three-dimensional feature extraction is performed on the segmented facial micro-expression action region to obtain action feature description information. The three-dimensional feature extraction refers to extracting features such as spatial distribution, time continuity, and motion amplitude of the action region to form a quantitative description of the micro-expression action.

[0072] The time sequence clustering is performed according to the extracted action feature description information to obtain facial micro-expression action categories. The time sequence clustering is to cluster similar action features into a category, and each category corresponds to a micro-expression action type, such as eyebrow raising or mouth corner raising.

[0073] The probability statistics are performed on the obtained facial micro-expression action categories to obtain action probability distribution information. The probability statistics are to calculate the frequency of occurrence of each action category to reflect the occurrence probability of various micro-expression actions.

[0074] The feature description processing is performed according to the action probability distribution information to obtain facial micro-expression action features. The feature description processing is to convert the probability distribution information into a feature vector form as a feature representing the micro-expression action.

[0075] The geometric feature calculation is performed on the facial landmark point coordinates to obtain a facial geometric feature vector. The geometric feature calculation includes calculating geometric relationships such as distances and angles between landmark points to form a feature vector describing the shape of the face.

[0076] The motion analysis is performed on the facial landmark point coordinates to obtain facial landmark point motion trajectories. The motion analysis is to calculate the displacement of the landmark points between consecutive frames to obtain the motion trajectories of the landmark points.

[0077] The action description construction is performed according to the facial geometric feature vector and the facial landmark point motion trajectories to obtain corresponding action feature information. The action description construction is to combine the geometric features and the motion features to form a comprehensive description of the facial action.

[0078] The facial action classification is performed on the action feature information according to a preset action classification rule to obtain facial action categories. The preset action classification rule includes feature templates of various typical facial actions for determining to which category the current action belongs.

[0079] The feature fusion is performed on the action feature information and the facial action categories to obtain facial action features. The feature fusion is to combine the quantitative features of the action and the category labels to form a comprehensive description of the facial action as an input feature for subsequent human body state detection.

[0080] The embodiment can accurately capture the micro-expression changes of the face by face region division and optical flow field calculation on the image sequence, and improve the sensitivity and accuracy of micro-expression action detection. The method combining motion analysis and region segmentation can effectively identify the micro-expression action region of the face, and reduce the probability of false detection and missed detection. Through three-dimensional feature extraction and time series clustering, quantitative description and classification of micro-expression actions are realized, and more rich expression information is provided. Combined with geometric features and motion trajectory analysis, a comprehensive facial action feature is constructed, which can more accurately reflect the human state. The feature fusion method integrates multi-dimensional information and enhances the robustness of human state detection. This multi-level and multi-angle analysis method significantly improves the accuracy and reliability of human state detection, providing strong support for emotion recognition, health monitoring and other applications.

[0081] In one embodiment, the face action feature is analyzed in time series to obtain a time series face action feature, including: The obtained face action feature data is segmented by time window, and the continuous face action feature data is divided into multiple time windows. The length of each time window can be pre-set, for example, set to 1 second or 0.5 second. Through time window segmentation, the face action feature sequence in multiple time windows is obtained.

[0082] The Fourier transform is performed on the face action feature sequence in each time window to convert the time domain signal to the frequency domain signal. The Fourier transform can be realized by using the fast Fourier transform (FFT) algorithm. Through the Fourier transform, the frequency domain feature of the face action feature in each time window is obtained, including the amplitude spectrum and the phase spectrum.

[0083] Based on the obtained frequency domain feature, the time series correlation analysis is performed on the face action feature sequence. Specifically, the correlation coefficient of the frequency domain feature between different time windows is calculated to obtain the correlation feature of the face action in the time dimension.

[0084] According to the time correlation feature, the time series interpolation is performed on the face action feature sequence. For the part with high correlation between adjacent time windows, linear interpolation or other methods are used for interpolation to obtain a more continuous face action feature sequence.

[0085] The state transition matrix is constructed for the interpolated face action feature sequence. The state transition matrix describes the transition probability of the face action feature between different time points. By calculating the transition probability between adjacent time point feature vectors, the state transition data of the face action is obtained.

[0086] According to the state transition data, the face action feature sequence is corrected. Specifically, the original feature sequence is smoothed by using the state transition matrix to eliminate abnormal fluctuations, and the corrected time series face action feature is obtained.

[0087] The embodiment realizes more accurate and reliable detection of human state by time series analysis of facial motion features. Through time window segmentation and Fourier transform, the frequency domain features of facial motion are effectively extracted, laying a foundation for subsequent analysis. The introduction of time series correlation analysis and time series interpolation makes the facial motion features more continuous and smooth in the time dimension, effectively reducing the influence of noise and abnormal fluctuations. The construction of the time series state transition matrix and the feature correction further enhance the time series consistency and stability of the facial motion features. This multi-level time series analysis method not only improves the accuracy of feature extraction, but also captures subtle changes in facial motion, thereby more comprehensively and accurately reflecting human state.

[0088] In one embodiment, the time series facial motion features are subjected to multi-scale illumination adjustment to obtain illumination facial motion features, including: After obtaining the time series facial motion features, wavelet transform is performed to obtain multi-scale decomposition results. Wavelet transform is a time-frequency localized signal analysis method that decomposes signals into components of different frequencies and time scales. Through wavelet transform, the performance of facial motion features at different scales is obtained.

[0089] After obtaining the time series facial motion features, wavelet transform is performed to obtain multi-scale decomposition results. Wavelet transform is a time-frequency localized signal analysis method that decomposes signals into components of different frequencies and time scales. Through wavelet transform, the performance of facial motion features at different scales is obtained.

[0090] Based on the preset feature recognition rules, a subset is constructed from the multi-scale decomposition results to obtain facial motion feature subsets at different scales. The feature recognition rules include frequency bandwidth, energy threshold, etc., which are used to filter out effective feature components from the multi-scale decomposition results. Through this step, the facial motion features are divided into multiple scale subsets.

[0091] By applying the Local Binary Pattern (LBP) algorithm to each facial motion feature subset, illumination invariant texture feature extraction is performed to obtain illumination invariant feature descriptors. LBP algorithm is an effective texture description method with gray and rotation invariance, which extracts texture information in facial motion features that is not affected by illumination.

[0092] According to the obtained illumination invariant feature descriptors, each facial motion feature subset is subjected to equalization processing to obtain multiple illumination compensated feature compensation subsets. Equalization processing adjusts the gray distribution of the feature subset, improves the contrast, and makes the facial motion features under different illumination conditions more consistent.

[0093] The multi-scale fusion is performed on all the feature compensation subsets to obtain a fused illumination feature set. The multi-scale fusion is performed by using a weighted average or principal component analysis to comprehensively integrate the feature information of different scales to form a comprehensive feature representation.

[0094] The retinex algorithm is applied to the fused illumination feature set to separate and correct the illumination component, to obtain an illumination consistency correction result. The retinex algorithm is an image enhancement algorithm that separates the illumination component and the reflection component in an image to achieve an illumination-invariant image representation.

[0095] The illumination consistency correction result is corrected with the fused illumination feature set to obtain an illumination facial action feature. The correction is performed by using a pixel-level multiplication or addition operation to apply the illumination consistency correction result to the fused illumination feature set to obtain a facial action feature that is consistent under different illumination conditions.

[0096] The embodiment effectively solves the consistency problem of facial action feature extraction under different illumination conditions by using a multi-scale illumination adjustment technique. The multi-scale decomposition is performed by using a wavelet transform, and a subset is constructed in combination with a feature recognition rule to fully utilize the information of the facial action feature under different scales. The local binary pattern algorithm is applied to extract an illumination-invariant texture feature, and the robustness of the feature is enhanced by using equalization processing and multi-scale fusion. The retinex algorithm is introduced to separate and correct the illumination component to further reduce the influence of illumination changes. This multi-step and multi-algorithm combination method significantly improves the consistency and comparability of the facial action feature under different illumination environments, provides a more reliable feature input for subsequent human state detection, and improves the accuracy and adaptability of the detection.

[0097] In one embodiment, the illumination facial action feature is subjected to multi-modal analysis by using a pre-trained long short-term memory model to obtain a comprehensive human state feature, including: The illumination facial action feature is subjected to input sequence segmentation to obtain a facial action input sequence. The facial action input sequence refers to dividing continuous facial action features into a plurality of discrete sequence segments in chronological order. Subsequently, the facial action input sequence is subjected to state matching according to a pre-set multi-modal state database to obtain corresponding multi-modal state data. The multi-modal state database contains standard data of various human states corresponding to different facial actions.

[0098] Next, the facial motion input sequence and the multi-modal state data are input to a first layer of the long short-term memory model. In the first layer of the long short-term memory network, a feature concatenation operation is performed on the facial motion input sequence and the multi-modal state data to obtain a fusion feature. The feature concatenation operation refers to merging two different source feature data in the feature dimension. A forget gate processing is performed on the fusion feature to obtain a forget gate output. The forget gate is used to decide whether to retain or discard the information in the cell state of the previous time. An input gate processing is performed on the forget gate output to obtain an input gate output. The input gate is used to control how much of the input information at the current time is saved to the cell state.

[0099] The input gate output is combined with a current candidate memory cell state to obtain an updated memory cell state. The candidate memory cell state refers to the new information calculated by the network at the current time. An output gate processing is performed on the updated memory cell state to obtain a first layer output feature. The output gate is used to control how much information in the cell state is output to the hidden state.

[0100] The first layer output feature is transmitted as input to a second layer of the long short-term memory network. In the second layer of the long short-term memory network, a feature concatenation operation is performed on the first layer output feature, the facial motion input sequence and the multi-modal state data to obtain a second layer fusion feature. A forget gate processing is performed on the second layer fusion feature to obtain a second layer forget gate output. An input gate processing is performed on the second layer forget gate output to obtain a second layer input gate output. The second layer input gate output is combined with a second layer current candidate memory cell state to obtain a second layer updated memory cell state. An output gate processing is performed on the second layer updated memory cell state to obtain a second layer output feature.

[0101] In each layer of the long short-term memory network, the output feature of the previous layer is combined with the facial motion input sequence and the multi-modal state data to obtain a fusion feature of the corresponding layer. A forget gate processing is performed on the fusion feature of the layer to obtain a forget gate output of the layer. An input gate processing is performed on the forget gate output of the layer to obtain an input gate output of the layer. The input gate output of the layer is combined with a current candidate memory cell state of the layer to obtain an updated memory cell state of the layer. An output gate processing is performed on the updated memory cell state of the layer to obtain an output feature of the layer.

[0102] When the last layer of the long short-term memory network completes the processing process, a fully connected layer processing is performed on the output feature of the last layer of the long short-term memory network, and a comprehensive human state feature is output. The fully connected layer is a layer type in the neural network, which is used to connect all neurons of the previous layer with each neuron of the current layer to realize further fusion and mapping of the features.

[0103] The embodiment realizes fine processing of continuous facial action by input sequence segmentation and multi-modal analysis of light facial action features, and improves the timing accuracy of state detection. A pre-trained long short-term memory model is used for multi-level feature extraction, fully utilizing the timing information of facial action and the complementarity of multi-modal data, and enhancing the expression ability of state features. Through feature splicing operation in each layer of network, deep fusion of data from different sources is realized, and the comprehensiveness and robustness of the features are improved. The forgetting gate, input gate and output gate mechanism is introduced to effectively control the flow and update of information, avoid the gradient disappearance problem in long sequence processing, and improve the learning ability of the model. Finally, through full connection layer processing, the multi-level features are further integrated, and the output comprehensive human state features are more comprehensive and accurate, providing a reliable basis for subsequent state judgment.

[0104] In one real-time example, real-time state analysis is performed based on the comprehensive human state features, and human state detection results are obtained, including: The comprehensive human state features are subjected to frequency domain transformation to obtain time window features and frequency domain human features. Specifically, short-time Fourier transform is used to analyze the time-frequency of the comprehensive human state features, and convert the time domain signal to the frequency domain. The short-time Fourier transform slides a fixed length window function on the time domain signal, and performs Fourier transform on the signal in each window to obtain the frequency spectrum corresponding to the window. In this way, the time domain information of the signal is retained, and the frequency domain features are obtained. The time window features reflect the changes of the signal in different time periods, and the frequency domain human features represent the frequency composition of the signal.

[0105] The frequency domain human features are subjected to feature enhancement processing to obtain corresponding multi-scale human features. Here, wavelet transform is used to decompose the frequency domain human features into sub-band signals of different frequency ranges. Wavelet transform has good time-frequency localization characteristics and can analyze the details of the signal at different scales. By selecting appropriate wavelet basis functions and decomposition levels, the frequency components related to human state can be highlighted, and noise interference can be suppressed, thereby enhancing the effective features.

[0106] The multi-scale human features are subjected to nonlinear transformation to obtain state feature vectors. Here, a series of nonlinear functions are used to transform and combine the multi-scale human features. Specifically, the features are normalized, nonlinear mapping is applied using activation functions such as ReLU or sigmoid, and high-order features are generated by feature combination. These operations can enhance the expression ability of the features and capture the complex relationships between the features. Finally, dimension reduction operations such as principal component analysis (PCA) are used to obtain state feature vectors of fixed dimensions.

[0107] The state feature vector is classified according to a preset state category, and a state classification result is obtained. Here, a distance-based classification method is adopted, the Euclidean distance between the state feature vector and the center of each preset state category is calculated, and the feature vector is classified into the category with the closest distance. The preset state categories include normal, fatigue, tension and other typical human body states, and the center vector of each category is calculated by the mean value of a large number of sample data.

[0108] The state classification result is predicted according to a preset state transition probability matrix, and a human body state detection result is obtained. The state transition probability matrix describes the probability of conversion between different states, and can be obtained by a large number of historical data statistics. Using the current state classification result and the state transition probability, the probability distribution of each state at the next time is calculated, and the state with the maximum probability is selected as the prediction result.

[0109] The embodiment can simultaneously obtain the time window feature and the frequency domain human body feature by performing frequency domain transformation on the comprehensive human body state feature, effectively capturing the dynamic change and frequency characteristics of the human body state. The multi-scale decomposition of wavelet transformation further enhances the expression ability of the feature, highlights the key frequency components and suppresses noise interference. The nonlinear transformation and feature combination improve the expression ability of the feature, capturing complex feature relationships. The distance-based classification method is simple and efficient, and is easy to realize real-time processing. The preset state transition probability matrix introduces the time sequence correlation, smooths the instantaneous classification error, and improves the stability and accuracy of the detection result. The entire method combines frequency domain analysis, multi-scale processing, nonlinear transformation and probability model, forming a comprehensive and effective human body state detection system, which can adapt to real-time state monitoring requirements in different scenes.

[0110] Reference Figure 2 The application also provides a human body state detection system based on facial action, which is applied to the human body state detection method based on facial action in any of the above. The acquisition module is used to obtain an image sequence of a face, perform face detection and landmark positioning on the image sequence, and obtain face landmark coordinates. The analysis module is used to extract micro-expression action features based on the face landmark coordinates. The correlation module is used to perform time sequence analysis on the face action features, and obtain time sequence face action features. The processing module is used to perform multi-scale light adjustment on the time sequence face action features, and obtain light face action features. The control module is used to perform multi-modal analysis on the light face action features through a pre-trained long short-term memory network, and obtain comprehensive human body state features. The execution module is used for performing real-time state analysis based on the comprehensive human body state features to obtain a human body state detection result.

[0111] The human body state detection system based on facial actions provided by the application can capture more subtle and rich facial expression changes by locating the landmark points and extracting micro-expression actions of the face image sequence, thereby improving the precision and sensitivity of the human body state detection. The time sequence analysis of the facial action features effectively utilizes the dynamic information of the facial expressions, overcomes the limitations of relying on static features only, and makes the state detection result more comprehensive and accurate. The multi-scale light adjustment technology is adopted to effectively reduce the influence of environmental light changes on the facial feature extraction, thereby improving the robustness and adaptability of the detection method under different light conditions. The long short-term memory network is pre-trained to perform multi-modal analysis, realize comprehensive processing of different types of facial action features, and enhance the reliability and generalization ability of the detection result.

[0112] Reference Figure 3 The application further provides a human body state detection device based on facial actions, which comprises: A memory is configured to store a program. A processor is configured to execute the program to implement each step of the human body state detection method based on facial actions according to any one of the preceding embodiments.

[0113] In this embodiment, the processor and the memory can be connected through a bus or other means. The memory can include a volatile memory such as a random access memory, and the memory can also include a non-volatile memory such as a read-only memory, a flash memory, a hard disk or a solid state disk. The processor can be a general-purpose processor such as a central processing unit, a digital signal processor, an application-specific integrated circuit, or one or more integrated circuits configured to implement the embodiments of the application.

[0114] The application further provides a medium storing computer instructions, which are used to make a computer execute the method according to any one of the preceding embodiments.

[0115] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the system and each module described above can refer to the corresponding processes in the preceding method embodiments, which will not be described here.

[0116] The above description is only the preferred embodiments of the application, and does not limit the patent scope of the application. Any equivalent structure or equivalent process transformation based on the content of the specification and drawings, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the application.

Claims

1. A human body state detection method based on facial movements, characterized in that: include: Acquire a facial image sequence, perform face detection and landmark location on the image sequence, and obtain facial landmark coordinates; Extracting micro-expression movements from the image sequence based on the facial landmark coordinates to obtain facial micro-expression movement features; Performing time series analysis on the facial action features to obtain time series facial action features; performing multi-scale illumination adjustment on the temporal facial action feature to obtain an illumination facial action feature; Performing multimodal analysis on the illuminated facial motion features through a pre-trained long short-term memory network to obtain comprehensive human body state features; Real-time state analysis is performed based on the comprehensive human body state characteristics to obtain a human body state detection result.

2. The method for detecting human body status based on facial movements according to claim 1, wherein: The method of acquiring a facial image sequence, performing face detection and landmark location on the image sequence, and obtaining facial landmark coordinates includes: Extracting frames from a video stream input by a preset camera device to obtain the image sequence of the human face; Performing face detection on the image sequence by using a cascade classifier to obtain a face region; cropping the image sequence according to the face region to obtain a face image; Extracting features from the face image based on a preset edge detection algorithm to obtain a facial contour area; The facial contour area is located according to a preset landmark point location rule to obtain the facial landmark point coordinates.

3. The human body state detection method based on facial movements according to claim 1, characterized in that: The extracting micro-expression movements from the image sequence based on the facial landmark coordinates to obtain facial micro-expression movement features includes: Dividing the image sequence into facial regions according to the facial landmark coordinates to obtain a plurality of facial micro-regions; performing optical flow calculation on the plurality of facial micro-regions to obtain facial optical flow features; Performing motion analysis on the facial micro-region according to the facial optical flow field features to obtain a facial micro-region motion information map; Performing regional segmentation on the facial micro-region motion information map according to a preset segmentation rule to obtain facial micro-expression action regions; Performing three-dimensional feature extraction on the facial micro-expression action area to obtain action feature description information; Performing temporal clustering based on the action feature description information to obtain facial micro-expression action categories; Performing probability statistics on the facial micro-expression action categories to obtain action probability distribution information; Performing feature description processing based on the action probability distribution information to obtain the facial micro-expression action features, performing geometric feature calculation, and obtaining a facial geometric feature vector; Performing motion analysis on the facial landmark coordinates to obtain motion trajectories of the facial landmarks; Constructing an action description based on the facial geometric feature vector and the facial landmark point motion trajectory to obtain corresponding action feature information; Classifying the facial action of the action feature information according to a preset action classification rule to obtain a facial action category; The action feature information is fused with the facial action category to obtain the facial action feature.

4. The method for detecting human body status based on facial movements according to claim 1, wherein: The performing time series analysis on the facial action features to obtain time series facial action features includes: Performing time window segmentation on the facial action features to obtain facial action feature sequences of multiple time windows; Performing Fourier transform on the facial action feature sequence in each time window to obtain frequency domain features; Performing time series correlation analysis on the facial action feature sequence based on the frequency domain features to obtain time correlation features of the facial actions; Performing temporal interpolation on facial action features according to the time correlation features to obtain a facial action feature sequence; Constructing a time series state transfer matrix for the facial action feature sequence to obtain state transfer data of the facial action; The facial action feature sequence is modified according to the state transition data to obtain the temporal facial action feature.

5. The method for detecting human body status based on facial movements according to claim 1, wherein: The performing multi-scale illumination adjustment on the temporal facial action feature to obtain the illumination facial action feature includes: Performing wavelet transform on the temporal facial action features to obtain a multi-scale decomposition result; Subsetting the multi-scale decomposition results according to a preset feature recognition rule to obtain facial action feature subsets of different scales; performing illumination-invariant texture feature extraction on each of the facial action feature subsets according to a local binary pattern algorithm to obtain an illumination-invariant feature descriptor; performing equalization processing on each of the facial action feature subsets according to the illumination-invariant feature descriptor to obtain a plurality of illumination-compensated feature-compensated subsets; Performing multi-scale fusion on all the feature compensation subsets to obtain a fused illumination feature set; Performing illumination component separation correction on the fused illumination feature set according to the retinex algorithm to obtain an illumination consistency correction result; The illumination consistency correction result and the fused illumination feature set are corrected to obtain the illumination facial action feature.

6. The method for detecting human body status based on facial movements according to claim 1, wherein: The multimodal analysis of the illuminated facial action features is performed using a pre-trained long short-term memory model to obtain comprehensive human body state features, including: performing input sequence segmentation on the illumination facial action features to obtain a facial action input sequence; Performing state matching on the facial action input sequence according to a preset multimodal state database to obtain corresponding multimodal state data; The facial action input sequence and the multimodal state data are input into the first layer of the long short-term memory model, wherein: In the first layer of the long short-term memory network, a feature splicing operation is performed on the facial action input sequence and the multimodal state data to obtain a fusion feature; Performing forget gate processing on the fused features to obtain a forget gate output; Performing input gate processing on the forget gate output to obtain an input gate output; Combining the input gate output with the current candidate memory cell state to obtain an updated memory cell state; Performing output gate processing on the updated memory unit state to obtain first layer output features; The output features of the first layer are passed as input to the second layer of long short-term memory network, and the processing process of the first layer of long short-term memory network is repeated; In each layer of the long short-term memory network, the output features of the previous layer are concatenated with the facial action input sequence and the multimodal state data to obtain the fused features of the corresponding layer, and the processing process of the first layer of the long short-term memory network is repeated; When the last layer of long short-term memory network completes the processing process, the output features of the last layer of long short-term memory network are fully connected layer processed, and the comprehensive human body state features are output.

7. The method for detecting human body status based on facial movements according to claim 1, wherein: The real-time state analysis based on the comprehensive human body state characteristics is performed to obtain a human body state detection result, including: Performing frequency domain transformation on the comprehensive human body state feature to obtain a time window feature and a frequency domain human body feature; Performing feature enhancement processing on the frequency domain human features to obtain corresponding multi-scale human features; Performing nonlinear transformation on the multi-scale human body features to obtain a state feature vector; Classifying the state feature vector according to a preset state category to obtain a state classification result; The state classification result is subjected to state prediction according to a preset state transition probability matrix to obtain the human body state detection result.

8. A human body state detection system based on facial movements, characterized in that: The method for detecting a human state based on facial movements, as applied to any one of claims 1 to 7, comprises: An acquisition module is used to acquire a facial image sequence, perform face detection and landmark location on the image sequence, and obtain facial landmark coordinates; An analysis module, configured to extract micro-expression movements from the image sequence based on the facial landmark coordinates to obtain facial micro-expression movement features; an association module, the association module being configured to perform a time series analysis on the facial action features to obtain time series facial action features; a processing module, configured to perform multi-scale illumination adjustment on the temporal facial action feature to obtain an illumination facial action feature; A control module, configured to perform multimodal analysis on the illuminated facial motion features using a pre-trained long short-term memory network to obtain comprehensive human body state features; An execution module is used to perform real-time state analysis based on the comprehensive human body state characteristics to obtain a human body state detection result.

9. A human body state detection device based on facial movements, characterized in that: include: Memory, used to store programs; The processor is used to execute the program to implement the various steps of the human body state detection method based on facial movements as described in any one of claims 1 to 7.

10. A medium, characterized in that Computer instructions are stored, and the computer instructions are used to make a computer execute the method according to any one of claims 1 to 7.

Images