Method and system for stereoscopic face emotion perception and fatigue analysis for sports scenarios

Abstract

The application relates to the technical field of computer vision, and particularly discloses a method and system for stereoscopic facial emotion perception and fatigue analysis in a sports scene, which comprises the following steps: collecting a face video sequence in a sports process, pre-processing the face video sequence to obtain a pre-processed face key point sequence; extracting face geometric dynamic features in a continuous time window based on the face key point sequence; performing correlation analysis on the face geometric dynamic features and a preset motion unit set to obtain face motion unit information, wherein the face motion unit information is used for representing a face muscle activity mode corresponding to the geometric dynamic features; determining a sports state by using sports context correction information, and modeling the face geometric dynamic features and the face motion unit information in combination with the sports state to obtain an emotion semantic index; and outputting fatigue evaluation information based on the emotion semantic index. The application can greatly improve the accuracy of non-contact fatigue evaluation in a sports environment.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a method and system for stereoscopic facial emotion perception and fatigue analysis in motion scenarios. Background Technology

[0002] With the development of smart fitness, sports rehabilitation, home exercise training and sports health management, how to assess the physical and mental state of athletes in real time, objectively and with low interference during exercise has become an important issue in the fields of sports science, intelligent sensing and human-computer interaction.

[0003] During actual exercise, athletes not only experience physiological or kinematic changes such as increased heart rate, altered movement rhythm, accelerated breathing, and physical fatigue, but also significant emotional and subjective changes, including feelings of pleasure, focus, anxiety, frustration, boredom, pain, discomfort, overexertion, and accumulated fatigue. These emotions and subjective experiences directly affect exercise adherence, training effectiveness, rehabilitation outcomes, and exercise safety. Therefore, simply monitoring exercise intensity or physiological load is insufficient to fully reflect the athlete's true state during exercise.

[0004] Therefore, existing vision-based fatigue assessment methods still need to be improved for sports scenarios. Summary of the Invention

[0005] The primary objective of this application is to address at least one of the aforementioned problems by providing a method and system for three-dimensional facial emotion perception and fatigue analysis in motion scenarios.

[0006] To achieve the various objectives of this application, the following technical solution is adopted: A method for three-dimensional facial emotion perception and fatigue analysis in motion scenarios, provided to meet one of the purposes of this application, includes: Collect facial video sequences during movement, preprocess the facial video sequences to obtain preprocessed facial key point sequences; Based on the preprocessed facial key point sequence, facial geometric dynamic features within a continuous time window are extracted. By performing correlation analysis between the facial geometric dynamic features and a preset action unit set, facial action unit information is obtained, wherein the facial action unit information is used to characterize the facial muscle activity pattern corresponding to the facial geometric dynamic features. Acquire motion context correction information and determine motion state based on the motion context correction information. Perform weighted fusion processing on the facial geometric dynamic features and facial action unit information based on the motion state to obtain emotional semantic index. The motion context correction information includes at least one of motion speed, motion intensity, motion stage, motion rhythm, posture stability and motion duration. The emotional semantic indicators are weighted and fused based on their temporal characteristics to output fatigue assessment information. Based on the fatigue assessment information, the fatigue level and fatigue trend information are determined.

[0007] In one feasible implementation, the step of extracting facial geometric dynamic features within a continuous time window based on the facial key point sequence includes: Based on the preprocessed facial key point sequence, local geometric quantities representing local morphology are obtained; Within a continuous time window, the local geometric quantities are represented as temporal variation curves that change along the time sequence, and local feature quantities that can characterize the dynamic process are extracted from the temporal variation curves. The mean, fluctuation amplitude, duration, and slope of change of the local feature quantities within a continuous time window are statistically calculated to obtain the facial geometric dynamic features.

[0008] In one feasible implementation, the step of obtaining facial motion unit information by performing correlation analysis between the facial geometric dynamic features and a preset set of motion units includes: The geometric quantities representing changes in the morphology of local facial regions in the facial geometric dynamic features are associated and matched with the facial action units in the preset action unit set to determine the facial muscle activity pattern corresponding to the geometric dynamic features. Based on the results of the association matching, the facial motion unit information is obtained, which includes the facial muscle activity pattern and the probability of occurrence, activation state or continuous intensity value of the facial muscle activity pattern. The intensity changes and duration of the facial motion unit information within a continuous time window are analyzed to determine the continuity characteristics of the facial muscle activity pattern.

[0009] In one feasible implementation, obtaining motion context correction information and determining the motion state based on the motion context correction information includes: Obtain motion context correction information synchronized with the current time window; The motion context correction information is used to determine the motion state of the current time window, which includes a high-intensity motion phase, a stable motion phase, or a recovery phase. The motion context correction information is obtained by acquiring data output from the motion device and / or by obtaining the human motion analysis results from the face video sequence.

[0010] In one feasible implementation, the weighted fusion processing of the facial geometric dynamic features and facial action unit information based on the motion state to obtain the emotional semantic index includes: The facial geometric dynamic features and the facial action unit information are used as input features. The input features are weighted and interpreted according to the motion state, wherein the interpretation weight of the same facial features is different in different motion states. At least one intermediate score is generated based on the weighted input features, and the intermediate scores are weighted and fused to obtain the sentiment semantic index. The emotional semantic indicators are smoothly updated within a continuous time window, wherein the emotional semantic indicators include at least one of valence, arousal, tension, pain and discomfort, focus and relaxation, and exercise pleasure.

[0011] In one feasible implementation, the preprocessing of the face video sequence to obtain a preprocessed facial key point sequence includes: The face video sequence is subjected to face detection and tracking processing to obtain face region, facial key point sequence and head pose information; Using the head pose information, alignment and jitter compensation are performed on the face region to obtain the facial key point sequence after head pose compensation; The facial key point sequence after head pose compensation is subjected to temporal smoothing to obtain a stable facial key point sequence.

[0012] In one feasible implementation, the step of weighted fusion processing of the emotional semantic indicators based on their temporal characteristics to output fatigue assessment information, and determining the fatigue level and fatigue trend information based on the fatigue assessment information, includes: The sentiment semantic index is statistically analyzed over a continuous time window, and its temporal features are extracted. The temporal features include at least one of the following: mean, slope of change, fluctuation amplitude, duration, and inter-window change trend. Based on the aforementioned temporal features, the changing trend of fatigue-related semantic features is determined. When the negative correlation index continues to increase within multiple consecutive time windows, it is determined that the fatigue-related semantic features show an increasing trend. The emotional semantic indicators and their temporal features are weighted and fused to generate the fatigue assessment information, which includes a fatigue score. The fatigue level is determined based on the comparison result between the fatigue score and the preset threshold. The fatigue level includes low fatigue, moderate fatigue and high fatigue. The fatigue trend information is output based on the direction of change of the fatigue score within multiple consecutive time windows. The fatigue trend information includes an upward trend of fatigue, a stable trend of fatigue, or a trend of fatigue relief. When an abnormal discomfort warning is issued, if the level of pain or discomfort increases rapidly in a short period of time, or if a state of high fatigue persists for more than a preset duration, an abnormal discomfort warning will be issued. The fatigue assessment information is provided to the user through multiple feedback methods, including at least one of display, voice, light, vibration, or motion device control commands.

[0013] On the other hand, a three-dimensional facial emotion perception and fatigue analysis system for motion scenarios, provided to meet one of the purposes of this application, includes: The video acquisition module is used to acquire facial video sequences during movement, and to preprocess the facial video sequences to obtain preprocessed facial key point sequences. The motion extraction module is used to extract facial geometric dynamic features within a continuous time window based on the preprocessed facial key point sequence. By performing correlation analysis between the facial geometric dynamic features and a preset set of motion units, facial motion unit information is obtained, wherein the facial motion unit information is used to characterize the facial muscle activity pattern corresponding to the geometric dynamic features. The motion correction module is used to acquire motion context correction information and determine motion state based on the motion context correction information. Based on the motion state, the facial geometric dynamic features and facial action unit information are weighted and fused to obtain emotional semantic indicators. The motion context correction information includes at least one of motion speed, motion intensity, motion stage, motion rhythm, posture stability and motion duration. The evaluation and analysis module is used to perform weighted fusion processing on the emotional semantic indicators based on the temporal characteristics of the emotional semantic indicators, output fatigue evaluation information, and determine the fatigue level and fatigue trend information based on the fatigue evaluation information.

[0014] On another front, a stereoscopic facial emotion perception and fatigue analysis device for motion scenarios, provided to meet one of the purposes of this application, includes a central processing unit and a memory. The central processing unit is used to call and run a computer program stored in the memory to execute the steps of the stereoscopic facial emotion perception and fatigue analysis method for motion scenarios described in this application.

[0015] In another aspect, a computer-readable storage medium is provided to suit one of the purposes of this application, the computer-readable storage medium storing computer-executable instructions for causing a computer to perform a method for stereoscopic facial emotion perception and fatigue analysis in motion scenarios as disclosed in any of the first aspects of this invention.

[0016] The technical solution of this application has many advantages, including but not limited to the following aspects: This invention acquires facial video sequences during movement and uses stabilization processing to reduce the impact of motion jitter. Further, based on the preprocessed facial key point sequences, it extracts facial geometric dynamic features within a continuous time window. By performing correlation analysis between these facial geometric dynamic features and a preset set of action units, facial action unit information is obtained. This facial action unit information characterizes the facial muscle activity patterns corresponding to the geometric dynamic features. Based on the emotional semantic change trend within the continuous time window, a fatigue assessment result is output, effectively reducing misjudgments caused by instantaneous expressions, short-term exertion, or image noise. Subsequently, motion context correction information is used to determine the motion state, and the facial geometric dynamic features and facial action unit information are modeled together with the motion state to obtain an emotional semantic index. The motion context information is used to describe the current motion stage and motion load background of the expression, thereby improving the accuracy of the emotional semantic index. Finally, fatigue assessment information is output based on this index.

[0017] The core of this invention lies in constructing a three-dimensional perception framework consisting of a "facial key point geometry layer—action unit action layer—emotional semantic layer—fatigue assessment layer," and introducing motion context correction to distinguish between normal facial expression changes and fatigue-related expressions during exercise. This invention eliminates the need for users to wear additional contact sensors, enabling continuous assessment of emotional state and fatigue risk in scenarios such as treadmills, indoor gyms, sports rehabilitation, and home training, thereby improving the naturalness, real-time nature, and personalization of exercise feedback. Attached Figure Description

[0018] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart illustrating the three-dimensional facial emotion perception and fatigue analysis method for sports scenarios proposed in this application. Figure 2 This is a schematic diagram of the architecture of the three-dimensional facial emotion perception and fatigue analysis method for sports scenarios proposed in this application; Figure 3 This is a schematic diagram of the three-dimensional facial emotion perception and fatigue analysis system for motion scenarios used in this application. Detailed Implementation

[0019] Before detailing the specific embodiments of the technical solution of this application, we will first disclose the application scenarios suitable for supporting the implementation of the technical solution of this application.

[0020] Current motion monitoring solutions can be mainly divided into the following categories, and each of them has certain limitations.

[0021] The first category is exercise monitoring solutions based on wearable devices or contact sensors. These solutions typically utilize devices such as heart rate monitors, smartwatches, inertial measurement units, electromyography (EMG) sensors, and breathing belts to collect data on heart rate, acceleration, cadence, energy expenditure, muscle activity, or respiratory changes, and further assess exercise intensity, exercise load, or fatigue status. This type of method can provide relatively direct physiological or kinematic indicators, but it requires the exerciser to wear additional equipment, which may affect the comfort and naturalness of the exercise. Furthermore, wearable devices are easily affected by wearing position, tightness, sweat, exercise impact, and individual differences. In addition, this type of method mainly reflects changes in body load and has difficulty directly identifying the exerciser's emotional state and subjective experience, such as boredom, anxiety, pain, or exercise frustration.

[0022] The second category is video-based motion posture recognition and movement quality assessment solutions. These solutions typically capture human movement videos using cameras and then analyze whether the movements are standard, the rhythm is stable, and whether there are any abnormalities in the posture using posture estimation, skeletal point tracking, or motion recognition models. These solutions are suitable for movement guidance and training correction, but their focus is usually on the body movement itself, rather than the exerciser's emotional state and fatigue experience. Even if the system can identify a decrease in movement speed, unstable posture, or reduced range of motion, it is difficult to further determine whether these changes are caused by normal exercise load, decreased attention, negative emotions, pain or discomfort, or accumulated fatigue.

[0023] The third category is emotion computing solutions based on general facial emotion recognition. Existing facial emotion recognition technologies typically classify basic emotions such as happiness, sadness, anger, surprise, fear, disgust, and neutrality in facial images through face detection, facial landmark localization, facial texture feature extraction, or deep learning models. These methods are commonly used for driver fatigue detection, attention monitoring, or general expression analysis, but they encounter significant difficulties when applied to sports scenarios. During exercise, individuals often exhibit head movements, body undulations, facial occlusion, sweat reflection, changes in lighting, mouth breathing, clenching teeth, and tense expressions. These phenomena may stem from normal exertion or reflect negative emotions, pain, discomfort, or accumulated fatigue. Directly using general emotion recognition models can easily misinterpret "normal exertion expressions" as anger, pain, or negative emotions, leading to unstable emotion recognition results, insufficient interpretability, and inaccurate feedback.

[0024] In summary, the existing technology still has the following shortcomings: (1) Existing exercise monitoring methods mainly focus on physiological load and exercise intensity, making it difficult to directly identify the emotional experience and subjective fatigue state of the exerciser.

[0025] (2) Existing general facial emotion recognition methods lack adaptation to motion scenarios and are easily affected by head shaking, sweat reflection, changes in lighting, mouth breathing, and facial expressions of exertion.

[0026] (3) In addition, existing facial action unit recognition methods are mostly used in static or low motion interference scenarios, lacking contextual correction for motion intensity and motion phase.

[0027] Therefore, it is necessary to propose a three-dimensional facial emotion perception and fatigue analysis method for sports scenarios. By collecting facial videos of athletes through cameras and combining facial key points, action units, head posture information and sports context features, a multi-level emotional semantic representation is constructed, thereby realizing real-time assessment and feedback of athletes' emotional state, fatigue level and abnormal discomfort risk.

[0028] Therefore, such as Figure 1 and Figure 2 As shown, this application proposes a three-dimensional facial emotion perception and fatigue analysis method for sports scenarios. This method uses facial video during exercise as input and achieves real-time assessment of the exerciser's emotional state, fatigue level, and risk of abnormal discomfort through facial geometric dynamic analysis, facial action unit recognition, motion context correction, and emotional semantic modeling. This method can be applied to scenarios such as treadmills, fitness mirrors, indoor cycling, rehabilitation training, and home exercise. The following description uses a treadmill scenario as a preferred embodiment, including the following steps: 101. Acquire facial video sequences during exercise, and preprocess the facial video sequences to obtain preprocessed facial key point sequences; this application acquires facial video sequences of the exerciser during exercise by setting a camera in front of or to the side of the exercise equipment. The camera can be a single-channel RGB camera or a combination of RGB and near-infrared cameras to improve image stability under low light, sweat reflection, and complex lighting conditions.

[0029] Because the movement involves body undulation, head shaking, and changes in perspective, the facial video sequence needs to be preprocessed to obtain a preprocessed facial key point sequence. Specifically, this includes: The face video sequence is subjected to face detection and tracking processing to obtain face region, facial key point sequence and head pose information; the head pose information includes yaw angle, pitch angle and roll angle.

[0030] Using the head pose information, alignment and jitter compensation are performed on the face region to obtain the facial key point sequence after head pose compensation; The facial keypoint sequence after head pose compensation is subjected to temporal smoothing to obtain a stable facial keypoint sequence. Specifically, firstly, based on the yaw angle, pitch angle, and roll angle in the head pose information, rotation correction is performed on the keypoints in each frame of the facial keypoint sequence to eliminate facial orientation differences caused by head rotation; secondly, the rotation-corrected facial keypoint sequence is subjected to temporal filtering within a continuous time window to suppress keypoint jitter caused by motion vibration and head sway. Finally, the preprocessed facial keypoint sequence is output. Through face tracking, jitter compensation, and temporal smoothing, the interference of running vibrations, head movements, and camera perspective changes on facial expression analysis is reduced. Stabilizing the face video sequence enables the acquisition of relatively stable facial structure information in dynamic motion scenes, providing reliable input for subsequent 3D emotional semantic modeling. The facial keypoint sequence can be represented in the form of 68 points, 98 points, or a 3D Face Mesh, and the keypoint extraction model can be a traditional regression model, a cascaded regression model, a lightweight deep learning model, or an edge-deployed model.

[0031] like Figure 2 As shown in the embodiments of this application, a method for three-dimensional facial emotion perception and fatigue analysis in motion scenarios further includes the following steps: 102. Based on the preprocessed facial key point sequence, extract facial geometric dynamic features within a continuous time window. By performing correlation analysis between the facial geometric dynamic features and a preset action unit set, obtain facial action unit information, wherein the facial action unit information is used to characterize the facial muscle activity pattern corresponding to the geometric dynamic features. Specifically, the facial motion unit information includes facial motion units (AUs) associated with the current facial geometric dynamic features and information such as intensity sequences. The purpose of step 102 is to systematically extract facial geometric dynamic features based on the stabilized facial key point sequence to describe the interpretable changes in the facial expressions of the person performing the exercise. The facial geometric dynamic features mainly include the morphological changes and temporal dynamics of the mouth, eyes, eyebrows, and local facial regions.

[0032] Among them, mouth features may include the degree of mouth opening and closing, the distance between the upper and lower lips, the tendency of the corners of the mouth to turn up or down, the lateral stretching of the mouth, and the duration of mouth opening, which are used to reflect mouth breathing, exertion, painful expressions, or fatigue; eye features may include the degree of eye opening and closing, blinking frequency, the degree of eyelid drooping, and the duration of eye closure, which are used to reflect changes in attention and fatigue accumulation; eyebrow features may include lowered eyebrows, raised eyebrows, changes in the distance between eyebrows, and the degree of tension in the eyebrow and eye area, which are used to reflect tension, anxiety, exertion, or pain and discomfort.

[0033] First, this embodiment analyzes the changes of facial key points within a continuous time window based on continuous time-series facial videos, emphasizing the continuity of expression changes rather than the instantaneous state of a single frame. This includes the following steps: Based on the preprocessed facial key point sequence, local geometric quantities representing local morphology are obtained; for example, for the mouth, eyes, eyebrows and local facial regions, geometric quantities such as the relative distance between key points, the degree of opening and closing, and the amplitude of upward or downward pressure are calculated, and the corresponding temporal change curves are further formed within a continuous time window.

[0034] Within a continuous time window, the local geometric quantities are represented as temporal variation curves that change along the time sequence, and local feature quantities that can characterize the dynamic process are extracted from the temporal variation curves; the local feature quantities include local opening and closing amplitude changes, key point displacement change rates, trajectory fluctuation degrees, expression duration, and change trends, etc.

[0035] Statistical analysis of the local feature quantities within a continuous time window yields the facial geometric dynamic features. Instead of judging emotions based on the location of key points in a single frame, this statistical analysis, including calculating the mean, fluctuation amplitude, duration, and slope of change, effectively distinguishes between brief expressions of exertion and persistent expressions of tension, fatigue, or discomfort. For example, short-term mouth opening, momentary frowning, or a single eye closure typically occur only for a short period, while sustained mouth breathing, sustained eyebrow and eye tension, or continuous eyelid drooping exhibit more stable geometric trends within the continuous time window. In this way, the system avoids misjudging brief exertions or momentary expressions as stable emotional states.

[0036] The acquisition of facial geometric dynamic features constitutes the first layer of the three-dimensional facial emotion perception system of this invention, namely the facial geometric dynamic layer. It describes the facial state of the mover through the geometric changes of key points within a continuous time window, providing a more stable and scene-appropriate basic input for subsequent action unit analysis and emotional semantic modeling.

[0037] Secondly, based on the geometric dynamic features of facial key points, step 102 further extracts the occurrence probability or intensity sequence of facial action units (AUs) to describe facial expression changes at the level of facial muscle activity. This specifically includes the following steps: The geometric quantities representing changes in the morphology of local facial regions in the facial geometric dynamic features are associated and matched with facial action units (AUs) in the preset action unit set to determine the facial muscle activity patterns corresponding to the geometric dynamic features; that is, to obtain the occurrence probability or intensity sequence of the associated facial action units (AUs). The facial action units (AUs) can be calculated through keypoint geometric relationships, or estimated through local texture features, convolutional neural networks, Transformer models, temporal neural networks, or multi-task learning models. The action unit set can be selected according to application requirements and is not limited to AU4, AU6, AU7, AU12, AU15, AU17, AU23, AU25, AU26, and AU43 listed in this embodiment.

[0038] Specifically, the geometric dynamic features of facial key points mainly reflect the morphological changes of local areas such as the mouth, eyes, and eyebrows within a continuous time window, such as the degree of mouth opening and closing, mouth corner displacement, changes in eye fissure height, changes in eyebrow distance, and the degree of eyebrow drooping. The facial motion unit (AU) further reflects the facial muscle activity patterns and their intensity corresponding to these geometric changes. In other words, facial key point features focus on describing "external geometric changes," while facial motion unit (AU) features focus on describing "internal muscle movement patterns," and the two are interconnected and complementary. For example, geometric changes in key points such as increased mouth opening, increased distance between the upper and lower lips, and downward displacement of the mandible usually correspond to an increase in AU25 (lip separation) and AU26 (mandible descent), and can be used to characterize mouth breathing, increased respiratory load, or exertion during exercise; geometric changes such as lowered eyebrows, reduced distance between the eyebrows, and tightened eyelids usually correspond to an increase in AU4 (brow down) and AU7 (eyelid tension), and can be used to characterize tension, exertion, alertness, or discomfort; changes such as upturned or downturned corners of the mouth can be associated with AU12 (upturned corners of the mouth) and AU15 (downturned corners of the mouth), respectively, and are used to help characterize positive experiences, feelings of frustration, or negative emotional trends; changes such as increased duration of eye closure and reduced palpebral fissure can be associated with AU43 (eye closure), and are used to help characterize accumulated fatigue, decreased attention, or decreased recovery ability.

[0039] Based on the results of the association matching, the facial motion unit information is obtained, including the occurrence probability, activation state, or continuous intensity value of the facial motion unit. The intensity changes and duration of the facial motion unit information within a continuous time window are analyzed to determine the persistence characteristics of the facial muscle activity pattern. Numerically, the facial motion unit information can be represented by the occurrence probability, activation state, or continuous intensity value of the facial motion unit (AU).

[0040] In some embodiments, a sustained increase in the intensity value of an AU indicates a gradual strengthening of the corresponding facial muscle movements; when it remains at a high level across multiple consecutive time windows, it indicates a persistent facial state. In contrast, if an AU only momentarily increases within a short time window, it is more likely to correspond to brief exertion or respiratory adjustment, rather than being directly considered a stable emotional or fatigued state. For example, a short-term increase in AU25 and AU26 may simply indicate mouth breathing due to increased exercise intensity; if accompanied by a brief increase in mouth opening and closing amplitude that quickly recovers, it can be considered a normal motor response. Furthermore, a sustained increase in AU4, AU7, AU23, and the characteristic of downward pressure of the corners of the mouth across multiple time windows, accompanied by lowered eyebrows, narrowed eye openings, and persistent tension in localized facial expressions, is more likely to indicate tension, pain, discomfort, or accumulated fatigue. For example, a short-term increase in AU12 during low to moderate exercise intensity, accompanied by an upturned mouth and relaxed eyebrows and eyes, can reflect a good level of exercise enjoyment; while a sustained increase in AU15 accompanied by a tendency for facial sagging is more likely to indicate an increase in frustration, fatigue, or negative experiences.

[0041] Therefore, this invention does not directly map a single AU to a fixed emotion label, but instead incorporates the geometric dynamic features of facial key points, the combination patterns of multiple facial action units (AUs), intensity changes, and durations into the analysis, thereby improving the stability and interpretability of facial emotion semantic recognition in motion scenes.

[0042] Among them, AU12 reflects an upturned mouth, which is associated with positive emotions or exercise pleasure; AU4, AU7 and AU23 reflect lowered eyebrows, tightened eyelids and tense lips, which are associated with tension, exertion or pain; AU25 and AU26 reflect lip separation and lowered jaw, which are associated with mouth breathing and increased exercise intensity; AU43 reflects a closed-eye state, which is associated with fatigue or decreased attention.

[0043] The acquisition of facial motion units constitutes the second layer of the three-dimensional facial emotion perception system in this embodiment, namely the facial motion unit layer. Its effect is to enhance the interpretability of emotion analysis from the perspective of facial muscle activity, and avoid relying solely on black-box expression classification models.

[0044] like Figure 2 As shown in the embodiments of this application, a method for three-dimensional facial emotion perception and fatigue analysis in motion scenarios further includes the following steps: 103. Obtain motion context correction information and determine motion state based on the motion context correction information. Perform weighted fusion processing on the facial geometric dynamic features and facial action unit information based on the motion state to obtain emotional semantic index. The motion context correction information includes at least one of motion speed, motion intensity, motion stage, motion rhythm, posture stability and motion duration.

[0045] Among them, the emotional semantic index is used to characterize the user's emotional conclusion; after obtaining the information of facial geometric dynamic features and facial action units (AU) in step 103, the two are input into the emotional semantic calculation module and jointly modeled with motion context information. Specifically: First, obtain motion context correction information synchronized with the current time window. The motion context correction information includes at least one of motion speed, motion intensity, motion phase, movement rhythm, posture stability, and motion duration. The motion state of the current time window is determined based on the motion context correction information, and the motion state includes a high-intensity motion phase, a stable motion phase, or a recovery phase. The motion context correction information is obtained through data output from the motion device and / or the results of human motion analysis in the face video sequence. Specifically, motion speed, incline, resistance level, training duration, and stage information can be directly output from treadmills, exercise bikes, elliptical trainers, rehabilitation equipment, or training control terminals; in implementations where device data cannot be directly obtained, it can also be estimated based on human motion speed, body undulation amplitude, gait rhythm, or scene temporal information in the video. The motion rhythm can be extracted from human posture sequences in continuous videos, for example, by calculating step frequency, body undulation frequency, or motion repetition frequency based on periodic displacement changes of key points of the head, shoulders, torso, or lower limbs; the posture stability can be characterized by the degree of displacement fluctuation, offset variance, or trajectory smoothness of the head position, facial frame center, shoulder key points, or torso key points within a continuous time window.

[0046] The purpose of motion context correction information is to resolve the problem of facial expression confusion in motion scenarios. During exercise, frowning, opening the mouth, clenching the teeth, and eye tension do not necessarily indicate negative emotions; they may also be normal exertion or high-intensity breathing responses.

[0047] Therefore, this embodiment does not directly output emotion conclusions based on facial expression features. Instead, it first obtains motion context information synchronized with the current time window, determines whether the current stage is a high-intensity stage, a stable stage, or a recovery stage, and then incorporates the facial geometric dynamic features and the intensity features of the facial action unit (AU) into the corresponding motion state for weighted interpretation to obtain an emotional semantic index used to represent emotions. This includes the following steps: The facial geometric dynamic features and the facial action unit information are used as input features. The input features are weighted and interpreted according to the motion state, wherein the interpretation weight of the same facial features is different in different motion states. At least one intermediate score is generated based on the weighted input features, and the intermediate scores are weighted and fused to obtain the sentiment semantic index. The emotional semantic indicators are smoothly updated within a continuous time window to avoid fluctuations in results caused by instantaneous facial expressions in a single frame; wherein the emotional semantic indicators include at least one of valence, arousal, tension, degree of pain or discomfort, degree of focus or relaxation, and degree of motor pleasure.

[0048] The sentiment semantics calculation module can employ a rule-weighted model, support vector machine, random forest, gradient boosting tree, or lightweight neural network.

[0049] The purpose of step 103 is to achieve the modeling and mapping from the underlying facial geometric changes and the mid-level facial action unit (AU) muscle activity to the high-level motor emotion semantic indicators, thereby completing the transformation from "facial expression changes" to "motor emotional state".

[0050] In some optional implementations, an individual baseline correction mechanism is proposed. This mechanism establishes an individual facial baseline before exercise or during a low-intensity exercise phase; calculates the dynamic offset of the facial geometric dynamic features and facial motion unit information relative to the individual facial baseline during exercise; and corrects the individual baseline based on this dynamic offset to reduce the impact of individual differences on the modeling results. By establishing an individual facial baseline before exercise or during a low-intensity exercise phase and calculating the dynamic offset of features relative to the individual baseline during exercise, the impact of individual differences on the evaluation results is reduced. This mechanism helps improve adaptability and personalized evaluation capabilities among different exercisers.

[0051] like Figure 2 As shown in the embodiments of this application, a method for three-dimensional facial emotion perception and fatigue analysis in motion scenarios further includes the following steps: 104. The emotional semantic indicators are weighted and fused according to their temporal characteristics to output fatigue assessment information, and fatigue level and fatigue trend information are determined based on the fatigue assessment information.

[0052] Fatigue assessment information is used to characterize information related to the degree of fatigue, and may include fatigue score, fatigue level, fatigue trend, and early warning of abnormal discomfort, including the following steps: The emotional semantic indicators are statistically analyzed over a continuous time window to extract their temporal features, which include at least one of the following: mean, slope of change, fluctuation amplitude, duration, and inter-window trend. Based on these temporal features, the changing trend of fatigue-related semantic features is determined. When negatively correlated indicators continuously increase over multiple continuous time windows, the fatigue-related semantic features are considered to be increasing. Each emotional semantic indicator and its temporal features are weighted and fused to generate fatigue assessment information including a fatigue score. The fatigue level is determined based on a comparison between the fatigue score and a preset threshold, including low fatigue, moderate fatigue, and high fatigue. Fatigue trend information is output based on the changing direction of the fatigue score over multiple consecutive time windows, including an increasing fatigue trend, a stable fatigue trend, or a fatigue relief trend. When a rapid increase in pain or discomfort is detected within a short period, or a high fatigue state persists for more than a preset duration, an abnormal discomfort warning is output. The fatigue assessment information is provided to the user through various feedback methods, including at least one of display, voice, light, vibration, or motion device control commands.

[0053] Specifically, the system first statistically analyzes and updates affective semantic indicators such as valence, arousal, tension, pain / discomfort level, focus / relaxation level, and motor pleasure within consecutive time windows, extracting their temporal features such as mean, slope of change, fluctuation amplitude, duration, and inter-window trend. If negative valence continuously decreases, tension and pain / discomfort level continuously increase, and focus / relaxation level continuously decreases, and these changes remain consistent across multiple consecutive time windows, then fatigue-related semantic features are determined to show an increasing trend.

[0054] Based on this, the system weights and integrates various emotional semantic indicators and their time trends to generate a fatigue score. Preferably, the fatigue score is determined by the degree of negative valence, tension, pain / discomfort, decline in focus, decline in exercise pleasure, and the degree of sustained enhancement of the above indicators within a continuous time window. That is, the system does not directly judge fatigue based on a single semantic indicator at a certain moment, but rather by comprehensively scoring based on the combined changes and evolution trends of multiple semantic indicators over a period of time. When negatively related indicators rise continuously and for a long period of time, the fatigue score is increased; when related indicators fluctuate only briefly and then recover, their contribution weight to the fatigue score is reduced.

[0055] Based on the fatigue score, the system further outputs a fatigue level. Preferably, the fatigue level can be divided into three levels: low fatigue, moderate fatigue, and high fatigue. Specifically, when the fatigue score is below a first threshold, it is determined to be low fatigue; when the fatigue score is between the first and second thresholds, it is determined to be moderate fatigue; and when the fatigue score is above the second threshold, it is determined to be high fatigue. The thresholds can be set and adjusted based on pre-experiment results, group statistical data, or individual baselines. Furthermore, the system can also output fatigue trend information based on the direction of fatigue score changes over multiple consecutive time windows. When the fatigue score continuously increases, an upward fatigue trend is output; when the fatigue score remains relatively stable, a stable fatigue trend is output; and when the fatigue score continuously decreases, a fatigue relief trend is output. If the system detects a rapid increase in pain / discomfort level within a short period, or if a high fatigue state persists for more than a preset duration, it can further output an abnormal discomfort warning. For example, when there is a decrease in eye opening and closing, abnormal blinking frequency, persistent increase in eyebrow and eye tension, increased duration of mouth opening, decreased negative valence, increased pain / discomfort intensity, and accompanied by decreased movement stability, the system can determine that the exerciser's fatigue risk is increased. Conversely, if only a brief frown or short-term mouth opening occurs, it is not directly judged as fatigue, but rather a judgment is made in combination with the exercise context and duration. When the fatigue score exceeds a preset threshold, or when negative emotions, pain, or discomfort continue to increase, the system can provide feedback through display, voice, light, vibration, or exercise equipment control commands, such as prompting to reduce speed, adjust breathing, take a short rest, or notify the coach. At the same time, the system can record emotional semantic curves, fatigue trend curves, and abnormal event points for generating training reports. To protect privacy, the system can save only key feature points, AU intensity, emotional score, and event time points, without saving the original facial video.

[0056] Step 104 constitutes the fatigue assessment and closed-loop feedback layer of this invention. Its technical effect lies in transforming the results of motion emotional semantic perception into operable motion feedback, enabling real-time management of the athlete's emotional state, fatigue risk, and exercise safety.

[0057] Please see Figure 3 According to one aspect of this application, a three-dimensional facial emotion perception and fatigue analysis system for motion scenarios is provided, the system comprising: The video acquisition module 201 is used to acquire a facial video sequence during movement, and to preprocess the facial video sequence to obtain a preprocessed facial key point sequence. The motion extraction module 202 is used to extract facial geometric dynamic features within a continuous time window based on the preprocessed facial key point sequence. By performing correlation analysis between the facial geometric dynamic features and a preset set of motion units, facial motion unit information is obtained, wherein the facial motion unit information is used to characterize the facial muscle activity pattern corresponding to the geometric dynamic features. The motion correction module 203 is used to acquire motion context correction information and determine motion state based on the motion context correction information. Based on the motion state, the facial geometric dynamic features and facial action unit information are weighted and fused to obtain emotional semantic index. The motion context correction information includes at least one of motion speed, motion intensity, motion stage, motion rhythm, posture stability and motion duration. The evaluation and analysis module 204 is used to perform weighted fusion processing on the emotional semantic indicators according to the temporal characteristics of the emotional semantic indicators, output fatigue evaluation information, and determine the fatigue level and fatigue trend information based on the fatigue evaluation information.

[0058] Based on any embodiment of the system in this application, the system of this application further includes: a correction unit, which is used to acquire motion context correction information synchronized with the current time window, the motion context correction information including at least one of motion speed, motion intensity, motion stage, action rhythm, posture stability, and motion duration; and to determine the motion state of the current time window based on the motion context correction information, the motion state including a high-intensity motion stage, a stable motion stage, or a recovery stage; wherein, the motion context correction information is acquired through data output from the motion device and / or human motion analysis results in the face video sequence.

[0059] Based on any embodiment of the system in this application, the system of this application further includes: an emotion analysis unit, which is used to take the facial geometric dynamic features and the facial action unit information as input features, and to perform weighted interpretation of the input features according to the motion state, wherein the interpretation weight of the same facial features is different in different motion states; to generate at least one intermediate score based on the weighted input features, and to perform weighted fusion of the intermediate scores to obtain the emotion semantic index; and to perform smooth updates of the emotion semantic index within a continuous time window, wherein the emotion semantic index includes at least one of valence, arousal, tension, pain and discomfort, focus and relaxation, and motion pleasure.

[0060] Based on any embodiment of the system in this application, the system of this application further includes: a motion jitter correction unit, which performs face detection and tracking processing on the face video sequence to obtain a face region, a facial key point sequence, and head pose information; uses the head pose information to perform alignment processing and jitter compensation processing on the face region to obtain the head pose compensated facial key point sequence; and performs temporal smoothing processing on the head pose compensated facial key point sequence to obtain a stable facial key point sequence.

[0061] Based on any embodiment of the system in this application, the system of this application further includes: a fatigue analysis unit, which is used to perform time-series statistics on the emotional semantic indicators within a continuous time window, extract the time-series features of the emotional semantic indicators, the time-series features including at least one of mean, slope of change, fluctuation amplitude, duration, and inter-window change trend; determine the changing trend of fatigue-related semantic features based on the time-series features, and determine that the fatigue-related semantic features show an increasing trend when the negative correlation indicators continuously increase within multiple continuous time windows; and perform weighted fusion of each emotional semantic indicator and its time-series features to generate the fatigue score. The system provides fatigue assessment information; determines fatigue levels based on comparisons between the fatigue score and preset thresholds, including low fatigue, moderate fatigue, and high fatigue; outputs fatigue trend information based on the changing direction of the fatigue score within multiple consecutive time windows, including an upward trend, a stable trend, or a trend of fatigue relief; outputs an abnormal discomfort warning when a rapid increase in pain or discomfort is detected within a short period, or when a state of high fatigue persists for more than a preset duration; and provides the fatigue assessment information to the user through various feedback methods, including at least one of display, voice, light, vibration, or motion device control commands.

[0062] Another embodiment of this application provides a stereoscopic facial emotion perception and fatigue analysis device for motion scenarios. This device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable, non-volatile storage medium stores an operating system, a database, and computer-readable instructions. The database may store information sequences. When the computer-readable instructions are executed by the processor, the processor can implement a stereoscopic facial emotion perception and fatigue analysis method for motion scenarios.

[0063] The processor of this motion-oriented stereoscopic facial emotion perception and fatigue analysis device provides computational and control capabilities, supporting the operation of the entire device. The device's memory can store computer-readable instructions, which, when executed by the processor, cause the processor to perform the motion-oriented stereoscopic facial emotion perception and fatigue analysis method of this application. The network interface of the device is used for communication with a terminal.

[0064] In this embodiment, the processor is used to execute... Figure 3 The system defines the specific functions of each module, and the memory stores the program code and various data required to execute these modules or submodules. The network interface is used to enable data transmission between user terminals or servers.

[0065] The non-volatile readable storage medium in this embodiment stores the program code and data required to execute all modules in the motion-oriented stereoscopic facial emotion perception and fatigue analysis system of this application. The server can call the program code and data of the server to execute the functions of all modules.

[0066] This application also provides a non-volatile readable storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the stereoscopic facial emotion perception and fatigue analysis method for motion scenes according to any embodiment of this application.

[0067] This application also provides a computer program product, including a computer program / instructions that, when executed by one or more processors, implement the steps of the method described in any embodiment of this application.

[0068] The key innovations of this invention compared to existing technologies mainly include the following aspects: Facial emotion semantic computing for sports scenarios: Existing facial emotion recognition methods are mostly geared towards general static or low-motion-interference scenarios. This invention is designed for sports scenarios such as running, fitness, and rehabilitation training, taking into account special interferences such as head shaking, sweat reflection, lighting changes, mouth breathing, and facial expressions of exertion during exercise, and constructs a facial emotion semantic computing method for sports scenarios.

[0069] This invention employs a multi-level, three-dimensional representation. Instead of simply classifying facial images for emotion, it establishes the following multi-level representation chain: The first layer is a geometric dynamic layer of facial key points, used to describe interpretable motion changes in the mouth, eyes, eyebrows, and other local facial regions; the second layer is a facial action unit layer, used to describe facial muscle activity patterns; the third layer is an emotional semantic layer, used to output valence, arousal, tension, pain / discomfort level, and focus / relaxation level; and the fourth layer is a fatigue assessment layer, used to output fatigue score, fatigue level, and fatigue trend. This multi-level structure improves the interpretability and stability of the system and is a key protective feature of this invention.

[0070] The invention employs a motion context correction mechanism, incorporating motion context information such as motion phase, intensity, speed, stability, head vibration, and duration to correct facial emotional features. This mechanism can distinguish between normal exertion expressions and abnormal negative emotions / fatigue expressions, a key technical feature differentiating it from general facial emotion recognition methods.

[0071] The individual baseline correction mechanism of this invention establishes an individual facial baseline before exercise or during a low-intensity exercise phase, and calculates the dynamic shift of features relative to the individual baseline during exercise, thereby reducing the impact of individual differences on the assessment results. This mechanism helps improve the adaptability and personalized assessment capabilities among different exercisers.

[0072] This invention employs a joint assessment of emotional semantics and fatigue trends. It does not rely on single-frame facial expressions or instantaneous classification results, but rather outputs fatigue assessment results based on the changing trends of emotional semantics within a continuous time window. This method can reduce misjudgments caused by instantaneous facial expressions, short-term physical exertion, or image noise, thereby improving the reliability of fatigue warnings.

Claims

1. A method for three-dimensional facial emotion perception and fatigue analysis in motion scenarios, characterized in that, include: Collect facial video sequences during movement, preprocess the facial video sequences to obtain preprocessed facial key point sequences; Based on the preprocessed facial key point sequence, facial geometric dynamic features within a continuous time window are extracted. By performing correlation analysis between the facial geometric dynamic features and a preset action unit set, facial action unit information is obtained, wherein the facial action unit information is used to characterize the facial muscle activity pattern corresponding to the facial geometric dynamic features. Acquire motion context correction information and determine motion state based on the motion context correction information. Perform weighted fusion processing on the facial geometric dynamic features and facial action unit information based on the motion state to obtain emotional semantic index. The motion context correction information includes at least one of motion speed, motion intensity, motion stage, motion rhythm, posture stability and motion duration. The emotional semantic indicators are weighted and fused based on their temporal characteristics to output fatigue assessment information. Based on the fatigue assessment information, the fatigue level and fatigue trend information are determined.

2. The method for three-dimensional facial emotion perception and fatigue analysis in motion scenarios according to claim 1, characterized in that, The extraction of facial geometric dynamic features within a continuous time window based on the facial key point sequence includes: Based on the preprocessed facial key point sequence, local geometric quantities representing local morphology are obtained; Within a continuous time window, the local geometric quantities are represented as temporal variation curves that change along the time sequence, and local feature quantities that can characterize the dynamic process are extracted from the temporal variation curves. The mean, fluctuation amplitude, duration, and slope of change of the local feature quantities within a continuous time window are statistically calculated to obtain the facial geometric dynamic features.

3. The method for three-dimensional facial emotion perception and fatigue analysis in motion scenarios according to claim 1, characterized in that, The process involves performing correlation analysis between the facial geometric dynamic features and a preset set of action units to obtain facial action unit information, including: The geometric quantities representing changes in the morphology of local facial regions in the facial geometric dynamic features are associated and matched with the facial action units in the preset action unit set to determine the facial muscle activity pattern corresponding to the geometric dynamic features. Based on the results of the association matching, the facial motion unit information is obtained, which includes the facial muscle activity pattern and the probability of occurrence, activation state or continuous intensity value of the facial muscle activity pattern. The intensity changes and duration of the facial motion unit information within a continuous time window are analyzed to determine the continuity characteristics of the facial muscle activity pattern.

4. The method for three-dimensional facial emotion perception and fatigue analysis in motion scenarios according to claim 1, characterized in that, The step of acquiring motion context correction information and determining motion state based on the motion context correction information includes: Obtain motion context correction information synchronized with the current time window; The motion context correction information is used to determine the motion state of the current time window, which includes a high-intensity motion phase, a stable motion phase, or a recovery phase. The motion context correction information is obtained by acquiring data output from the motion device and / or by obtaining the human motion analysis results from the face video sequence.

5. The method for three-dimensional facial emotion perception and fatigue analysis in motion scenarios according to claim 1, characterized in that, The weighted fusion processing of the facial geometric dynamic features and facial action unit information based on the motion state yields an emotional semantic index, including: The facial geometric dynamic features and the facial action unit information are used as input features. The input features are weighted and interpreted according to the motion state, wherein the interpretation weight of the same facial features is different in different motion states. At least one intermediate score is generated based on the weighted input features, and the intermediate scores are weighted and fused to obtain the sentiment semantic index. The emotional semantic indicators are smoothly updated within a continuous time window, wherein the emotional semantic indicators include at least one of valence, arousal, tension, pain and discomfort, focus and relaxation, and exercise pleasure.

6. The method for three-dimensional facial emotion perception and fatigue analysis in motion scenarios according to claim 1, characterized in that, The preprocessing of the face video sequence to obtain a preprocessed facial key point sequence includes: The face video sequence is subjected to face detection and tracking processing to obtain face region, facial key point sequence and head pose information; Using the head pose information, alignment and jitter compensation are performed on the face region to obtain the facial key point sequence after head pose compensation; The facial key point sequence after head pose compensation is subjected to temporal smoothing to obtain a stable facial key point sequence.

7. The method for three-dimensional facial emotion perception and fatigue analysis in motion scenarios according to claim 1, characterized in that, The step of weighted fusion processing of the emotional semantic indicators based on their temporal characteristics to output fatigue assessment information, and determining fatigue level and fatigue trend information based on the fatigue assessment information, includes: The sentiment semantic index is statistically analyzed over a continuous time window, and its temporal features are extracted. The temporal features include at least one of the following: mean, slope of change, fluctuation amplitude, duration, and inter-window change trend. Based on the aforementioned temporal features, the changing trend of fatigue-related semantic features is determined. When the negative correlation index continues to increase within multiple consecutive time windows, it is determined that the fatigue-related semantic features show an increasing trend. The emotional semantic indicators and their temporal features are weighted and fused to generate the fatigue assessment information, which includes a fatigue score. The fatigue level is determined based on the comparison result between the fatigue score and the preset threshold. The fatigue level includes low fatigue, moderate fatigue and high fatigue. The fatigue trend information is output based on the direction of change of the fatigue score within multiple consecutive time windows. The fatigue trend information includes an upward trend of fatigue, a stable trend of fatigue, or a trend of fatigue relief. When an abnormal discomfort warning is issued, if the level of pain or discomfort increases rapidly in a short period of time, or if a state of high fatigue persists for more than a preset duration, an abnormal discomfort warning will be issued. The fatigue assessment information is provided to the user through multiple feedback methods, including at least one of display, voice, light, vibration, or motion device control commands.

8. A three-dimensional facial emotion perception and fatigue analysis system for motion scenarios, characterized in that, The system is used to execute the stereoscopic facial emotion perception and fatigue analysis method for motion scenarios according to any one of claims 1-7, the system comprising: The video acquisition module is used to acquire facial video sequences during movement, and to preprocess the facial video sequences to obtain preprocessed facial key point sequences. The motion extraction module is used to extract facial geometric dynamic features within a continuous time window based on the preprocessed facial key point sequence. By performing correlation analysis between the facial geometric dynamic features and a preset set of motion units, facial motion unit information is obtained, wherein the facial motion unit information is used to characterize the facial muscle activity pattern corresponding to the geometric dynamic features. The motion correction module is used to acquire motion context correction information and determine motion state based on the motion context correction information. Based on the motion state, the facial geometric dynamic features and facial action unit information are weighted and fused to obtain emotional semantic indicators. The motion context correction information includes at least one of motion speed, motion intensity, motion stage, motion rhythm, posture stability and motion duration. The evaluation and analysis module is used to perform weighted fusion processing on the emotional semantic indicators based on the temporal characteristics of the emotional semantic indicators, output fatigue evaluation information, and determine the fatigue level and fatigue trend information based on the fatigue evaluation information.

9. A three-dimensional facial emotion perception and fatigue analysis device for motion scenarios, comprising: At least one processor, and, A memory communicatively connected to the at least one processor; characterized in that the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the stereoscopic facial emotion perception and fatigue analysis method for motion scenes as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores, in the form of computer-readable instructions, a computer program implemented according to any one of claims 1 to 7, which, when invoked by a computer, executes the steps included in the corresponding method.

Images