An AI-based visual emotion analysis method and system

Abstract

The application provides an AI-based visual emotion analysis method and system, relating to the technical field of data processing, the method comprising: obtaining a sequence of original image data containing a target object, and performing preprocessing to obtain a face region and an upper body region by segmentation; calculating the visibility score of face key points; when the score is lower than a preset first threshold, generating occlusion label information for the corresponding frame; performing skeleton key point detection, extracting limb posture features, obtaining limb feature parameters, and converting the limb feature parameters into emotion prior parameters through a mapping function; adaptively adjusting the weight of facial expression features and limb posture features in feature fusion, generating a fusion feature vector, and performing emotion determination to obtain an emotion category label; and binding the emotion category label with its timestamp to form an emotion change record of the target object; the application improves the accuracy of visual emotion analysis.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an AI-based visual emotion analysis method and system. Background Technology

[0002] In existing technologies, visual emotion analysis primarily relies on deep learning models such as convolutional neural networks (CNNs) to extract and recognize features such as facial expressions and body movements in images or videos. A common implementation involves first preprocessing the acquired image data to extract facial regions or related feature regions; then, using a pre-trained neural network model, these feature regions are categorized into emotion types, resulting in emotion labels such as "happy," "angry," and "neutral." Some systems also incorporate time-series analysis to track emotional changes across video frames. Current visual emotion analysis is widely used in scenarios such as security monitoring, driver fatigue detection, and distance education, enabling automatic recognition of users' emotional states.

[0003] However, in practical applications of distance education, existing visual emotion analysis methods are prone to decreased accuracy in emotion recognition due to students' facial expressions obscuring the image. For example, during classroom Q&A sessions, students may be looking down in thought for extended periods or resting their hands on their foreheads, making it difficult for the system to accurately determine their true emotional state based solely on facial features. In such cases, the model may misjudge "focus" as "indifference" or "fatigue," thus affecting the teacher's real-time assessment of student engagement. This deficiency makes emotion analysis based on single visual features insufficient to meet the accuracy requirements for student emotion recognition in practical distance education. Summary of the Invention

[0004] The purpose of this invention is to provide an AI-based visual emotion analysis method and system, which aims to solve the problems mentioned in the background art.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: Firstly, an AI-based visual emotion analysis method, the method comprising: The original image data sequence containing the target object is obtained, and each frame is preprocessed to segment the face region and upper body region, generating the first analysis data and the second analysis data respectively. Local feature integrity detection is performed on the first analysis data, the visibility score of facial key points is calculated, and facial integrity parameters are generated based on the score; when the facial integrity parameters are lower than a preset first threshold, occlusion marker information of the corresponding frame is generated. Based on the occlusion marker information, skeletal keypoint detection is performed on the second analysis data to extract limb posture features, obtain limb feature parameters, and then the limb feature parameters are converted into the emotion prior parameters of the frame through a mapping function. Based on the prior emotion parameters, the weights of facial expression features and body posture features in feature fusion are adaptively adjusted to generate a fused feature vector. The prior emotion parameters are then used to determine the emotion of the fused feature vector to obtain the emotion category label for the frame. By binding the emotion category label to the timestamp of the frame, a record of the target object's emotion changes is formed.

[0006] Preferably, local feature integrity detection is performed on the first analysis data, the visibility score of facial key points is calculated, and a facial integrity parameter is generated based on the score; when the facial integrity parameter is lower than a preset first threshold, occlusion marker information for the corresponding frame is generated, including: Multiple preset facial key points are detected, and the image sharpness value and pixel grayscale variance of each key point are calculated to obtain the facial key point visibility score; All visibility scores are weighted and summed according to preset key point weighting coefficients to obtain the overall facial integrity parameter; When the facial integrity parameter is lower than a preset first threshold, occlusion marker data is generated based on the key point distribution to characterize the degree of occlusion in the facial region.

[0007] Preferably, based on the occlusion marker information, skeletal keypoint detection is performed on the second analysis data to extract limb pose features, obtain limb feature parameters, and then the limb feature parameters are converted into prior emotion parameters for the frame through a mapping function, including: Detect multiple preset skeletal key points to obtain a set of skeletal key point coordinates; Based on the set of skeletal key point coordinates, calculate the spatial distance and angle parameters between adjacent skeletal key points to obtain motion characteristic data reflecting the range of motion and posture changes; Motion feature data is input into a mapping function and transformed into emotion prior parameters, wherein the emotion prior parameters reflect the strength of the association between the target object's limb movements and different emotional states.

[0008] Preferably, based on prior emotion parameters, the weights of facial expression features and body posture features in feature fusion are adaptively adjusted to generate a fused feature vector. The fused feature vector is then used to determine the emotion category of the frame by combining the prior emotion parameters, including: Based on prior emotion parameters, the confidence values ​​of facial expression features and limb feature parameters are weighted and fused to generate a fused feature vector. Based on the fused feature vectors, basic emotion category labels are generated according to preset discrimination criteria; Obtain the corresponding historical time-series sentiment data, and perform statistical analysis on the continuity and volatility of basic sentiment category labels within a preset time window to obtain the analysis results; Based on the analysis results, the basic emotion category labels are revised or refined to generate the final emotion category labels.

[0009] Preferably, the calculation of the facial key point visibility score includes: Obtain the color distribution histogram of the key point region and calculate its correlation score with the color distribution of the standard facial region; Based on the edge gradient distribution of pixels around the key point, local edge features are extracted, and multidimensional feature vectors are constructed by combining the image sharpness value and pixel grayscale variance. The multidimensional feature vectors are fused according to the preset feature weighting coefficients to obtain the visibility score of the key points.

[0010] Preferably, the transformation process of the mapping function includes: Establish multiple sets of association rules between body posture features and emotional states, including the correspondence between movement amplitude, movement duration, movement coordination and emotional type; The motion feature data is normalized to obtain a standardized motion feature vector; Based on the standardized motion feature vector, the degree of matching with each association rule is determined, and the correlation score of each emotional state is calculated. Based on the relevance score, the emotional state with the highest matching degree is selected, and the corresponding emotional prior parameters are generated. The mapping function can adaptively adjust the association rules and parameters according to the actual application scenario.

[0011] Preferably, the statistical analysis of the historical time-series sentiment data includes: Within a preset time window, the historical emotion tag sequence is statistically analyzed to obtain the duration and switching frequency of various emotion tags, and thus generate continuity and volatility parameters of emotion state. Based on continuity and volatility parameters, a trend analysis is performed on the historical sentiment label sequence to determine whether the current basic sentiment category label is in an upward, downward, or stable phase, and a trend determination result is generated. Based on the continuity parameter, volatility parameter, and trend determination results, the analysis results of the emotional state are output to guide the correction or refinement of the basic emotional category labels.

[0012] Secondly, an AI-based visual emotion analysis system, the system comprising: The preprocessing unit is used to acquire the original image data sequence containing the target object, and preprocess each frame of it to segment the face region and the upper body region, and generate the first analysis data and the second analysis data respectively. The facial integrity detection unit is used to perform local feature integrity detection on the first analysis data, calculate the visibility score of facial key points, and generate facial integrity parameters based on the score; when the facial integrity parameters are lower than a preset first threshold, occlusion marker information of the corresponding frame is generated. The limb feature extraction unit is used to perform skeletal key point detection on the second analysis data based on the occlusion marker information, extract limb posture features, obtain limb feature parameters, and convert the limb feature parameters into the emotion prior parameters of the frame through a mapping function. The feature fusion and determination unit is used to adaptively adjust the weights of facial expression features and body posture features in feature fusion based on the prior emotion parameters, generate a fused feature vector, and determine the emotion of the fused feature vector in combination with the prior emotion parameters to obtain the emotion category label of the frame. The emotion recording unit is used to bind the emotion category label with the timestamp of the frame to form a record of the target object's emotion changes.

[0013] The above-described solution of the present invention has at least the following beneficial effects: By preprocessing the original image data sequence, the facial region and the upper body region are respectively generated as first analysis data and second analysis data, enabling the system to simultaneously acquire facial expression features and body posture features, thus avoiding the limitations of existing technologies that rely solely on a single facial feature for emotion recognition.

[0014] By performing local feature integrity detection during the facial analysis process, the visibility score of facial key points is calculated and facial integrity parameters are generated. When the parameter is lower than a preset threshold, occlusion marker information is generated, which can effectively identify the loss of facial features caused by factors such as looking down, hand occlusion, or insufficient lighting, thus avoiding the misjudgment that is easy to occur when the face is occluded in existing methods.

[0015] By utilizing occlusion markers to perform skeletal keypoint detection on the upper body region when facial features are missing or incomplete, and extracting limb posture features, which are then converted into prior parameters for emotion, the system can still determine emotional state through limb movements even when facial expressions are difficult to recognize. For example, when a student is looking down in thought, facial features are difficult to identify, but their forward leaning and hand-supporting-the-head movements can serve as prior signals of "focus," thus ensuring the accuracy of emotion recognition.

[0016] By introducing prior emotion parameters during feature fusion, the weight distribution of facial and limb features is adaptively adjusted to generate a fused feature vector. This vector is then combined with the prior emotion parameters for emotion determination, enabling the system to achieve dynamic balance in different scenarios. For example, when facial features are clearly visible, facial features are prioritized for determination; when the face is occluded, the weight of limb posture features is increased, thus maintaining the stability and robustness of recognition results in complex scenarios.

[0017] By binding the emotion category label of each frame with a timestamp, a record of the target object's emotional changes is formed, enabling the system to output a complete emotional timeline trajectory. This facilitates teachers' real-time monitoring of students' attention and participation during remote education, avoiding the impact of momentary misjudgments on teaching effectiveness.

[0018] In summary, the present invention can effectively overcome the problem of decreased accuracy in emotion recognition due to occlusion of facial features in the prior art, and achieve more accurate and stable emotion recognition in scenarios such as remote education. Attached Figure Description

[0019] Figure 1 This is a flowchart of an AI-based visual emotion analysis method provided by an embodiment of the present invention. Detailed Implementation

[0020] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0021] like Figure 1 As shown, an embodiment of the present invention proposes an AI-based visual emotion analysis method, the method comprising: The original image data sequence containing the target object is obtained, and each frame is preprocessed to segment the face region and upper body region, generating the first analysis data and the second analysis data respectively. Local feature integrity detection is performed on the first analysis data, the visibility score of facial key points is calculated, and facial integrity parameters are generated based on the score; when the facial integrity parameters are lower than a preset first threshold, occlusion marker information of the corresponding frame is generated. Based on the occlusion marker information, skeletal keypoint detection is performed on the second analysis data to extract limb posture features, obtain limb feature parameters, and then the limb feature parameters are converted into the emotion prior parameters of the frame through a mapping function. Based on the prior emotion parameters, the weights of facial expression features and body posture features in feature fusion are adaptively adjusted to generate a fused feature vector. The prior emotion parameters are then used to determine the emotion of the fused feature vector to obtain the emotion category label for the frame. By binding the emotion category label to the timestamp of the frame, a record of the target object's emotion changes is formed.

[0022] In this embodiment of the invention, by acquiring the original image data sequence containing the target object and preprocessing each frame, the image is divided into a facial region and an upper body region, generating first analysis data and second analysis data respectively. This enables the system to simultaneously acquire input information of facial expression features and body posture features, avoiding the limitations of relying solely on a single feature for emotion recognition.

[0023] By performing local feature integrity detection on the first analysis data, the system calculates the visibility score of facial key points and generates facial integrity parameters. When this parameter is lower than a preset first threshold, occlusion marker information is generated, thereby identifying feature loss in facial areas caused by head tilting, obstructions, or insufficient lighting. For example, when a student rests their chin on their hand for an extended period in a remote classroom, the system can automatically mark the facial features of that frame as occluded to avoid misjudgments during emotion recognition.

[0024] Based on the occlusion marker information, skeletal keypoint detection is performed on the second analysis data to extract limb posture features and obtain limb feature parameters. These limb feature parameters are then converted into emotion prior parameters for that frame using a mapping function. This allows for the supplementary use of limb movements to aid emotion recognition even when facial expression features are missing. For example, when a student is looking down while thinking about a problem, facial expression information is difficult to obtain, but their forward leaning and hand support can reflect their focused state. Introducing emotion prior parameters can improve the system's accuracy in making judgments.

[0025] By using prior emotion parameters, the system adaptively adjusts the weights of facial expression features and body posture features in feature fusion to generate a fused feature vector. This fused feature vector is then used to determine the emotion, achieving a dynamic balance between facial and body features. For example, when facial features are clear, the system automatically increases the weight of facial features; when facial features are occluded, the system increases the weight of body posture features, thus obtaining stable emotion recognition results in different scenarios.

[0026] By binding emotion category tags to the timestamp of the frame, a record of the target object's emotional changes is created. This allows the system to output a complete time-series emotion trajectory, facilitating teachers' real-time monitoring of students' attention and engagement in remote education scenarios. For example, when a student's emotion in class gradually shifts from "focused" to "fatigued," the system can accurately reflect this process through continuous emotion change records, helping teachers adjust the teaching pace in a timely manner.

[0027] This process involves acquiring the original image data sequence containing the target object, preprocessing each frame to segment the face region and upper body region, and generating first analysis data and second analysis data respectively. Specifically, this includes: First, a sequence of raw image data containing the target object is acquired using a camera or video capture device. This raw image data sequence consists of multiple consecutive frames, each recording the appearance information of the target object at different points in time.

[0028] Then, preprocessing operations are performed on each frame of the image sequence. Preprocessing operations may include image scaling, illumination equalization, and noise removal to ensure the stability and accuracy of subsequent feature extraction. For example, the input frames can be normalized to a fixed image size, and then histogram equalization can be used to mitigate the impact of illumination differences on the recognition results.

[0029] Next, in each frame of the image, face detection and human detection models are used to segment the target object into regions. Face detection can use existing deep convolutional neural network models (such as MTCNN) to extract the rectangular region of the face; human detection can use human pose estimation algorithms (such as OpenPose) to locate the skeletal key points of the upper body and thereby determine the upper body region.

[0030] After obtaining the region segmentation results, the facial region image is cropped and normalized to generate the first analysis data; simultaneously, the upper body region image or its corresponding set of skeletal coordinates is extracted to generate the second analysis data. Thus, the system can simultaneously obtain the first analysis data for facial expression analysis and the second analysis data for pose analysis within the same frame, providing input for subsequent multi-feature fusion.

[0031] In a preferred embodiment of the present invention, local feature integrity detection is performed on the first analysis data, the visibility score of facial key points is calculated, and a facial integrity parameter is generated based on the score; when the facial integrity parameter is lower than a preset first threshold, occlusion marker information for the corresponding frame is generated, including: Multiple preset facial key points are detected, and the image sharpness value and pixel grayscale variance of each key point are calculated to obtain the facial key point visibility score; All visibility scores are weighted and summed according to preset key point weighting coefficients to obtain the overall facial integrity parameter; When the facial integrity parameter is lower than a preset first threshold, occlusion marker data is generated based on the key point distribution to characterize the degree of occlusion in the facial region.

[0032] In this embodiment of the invention, multiple preset facial key points are detected during the local feature integrity detection process, and the image sharpness value and pixel grayscale variance of each key point are calculated to obtain a facial key point visibility score, thereby enabling a comprehensive assessment of the quality of usable information in the facial region. When parts of the face are blurred, shadowed, or occluded, this score can promptly reflect the degradation of feature quality.

[0033] By weighting and summing all visibility scores according to preset keypoint weighting coefficients, an overall facial integrity parameter is generated, which comprehensively considers the importance of different facial key points in emotion recognition. For example, features of the eyes and corners of the mouth are more critical in expression analysis, so they can be given higher weights during weighting, making the facial integrity parameter more consistent with actual recognition needs.

[0034] When the overall facial integrity parameter is below a preset first threshold, occlusion marker data is generated based on the key point distribution to characterize the degree of occlusion in the facial region. In this way, the system can not only determine whether facial occlusion exists, but also mark the specific occlusion range and degree. For example, in a scenario where students are wearing masks, the system can identify that the lower half of the face is occluded, thereby reducing the weight of mouth-related features in subsequent fusion processes and improving the robustness of emotion recognition.

[0035] Specifically, all visibility scores are weighted and summed according to preset key point weighting coefficients to obtain the overall facial integrity parameters, which include: First, multiple preset key points are selected in the facial region. These key points include areas closely related to facial expression changes, such as the eyes, eyebrows, tip of the nose, and corners of the mouth. For each key point, a visibility score is obtained based on information such as image sharpness and pixel grayscale changes in its neighboring region.

[0036] Next, for the visibility scores of all key points, preset weight coefficients are assigned according to the importance of different key points. For example, key points of the eyes and corners of the mouth have a greater impact on emotion recognition, so their corresponding weight values ​​are set higher; while areas such as the cheeks or jaw, which do not change significantly, have relatively lower weight values.

[0037] Subsequently, the visibility scores of all keypoints are multiplied one by one by their corresponding weight coefficients, and all results are summed to obtain an overall value, which is the facial integrity parameter. The facial integrity parameter comprehensively reflects the visibility and integrity of the entire facial region, providing a basis for judging whether facial features are effective.

[0038] Specifically, when the facial integrity parameter is lower than a preset first threshold, occlusion marker data is generated based on the key point distribution to characterize the degree of occlusion in the facial region, including: First, a threshold is set for the facial integrity parameter to distinguish whether facial features are in a usable state. When the calculated facial integrity parameter value is lower than this threshold, it indicates that there is severe feature loss or occlusion in the overall face.

[0039] If facial integrity is deemed insufficient, the system further analyzes the distribution of key points. For example, if only the visibility scores of key points in the lower half of the face (such as the corners of the mouth and jaw) are low, it can be determined that the lower part of the face is obscured; if the scores of key points in the eye and eyebrow areas are low at the same time, it can be determined that the upper part of the face is obscured.

[0040] Based on this, occlusion marker data is generated. The occlusion marker data can contain two types of information: one type indicates whether the face is occluded, and the other type marks the location and degree of occlusion. For example, the occlusion marker can be output as a graded form such as "no occlusion", "partial occlusion", "severe occlusion", or it can record the specific area that is occluded, such as "lower half is occluded" or "eyes are occluded".

[0041] In this way, the system can accurately generate occlusion marker information when facial feature quality is insufficient, and provide a trigger basis for subsequent compensation analysis using limb posture features.

[0042] In a preferred embodiment of the present invention, based on the occlusion marker information, skeletal keypoint detection is performed on the second analysis data to extract limb posture features, obtain limb feature parameters, and then the limb feature parameters are converted into prior emotion parameters for the frame through a mapping function, including: Detect multiple preset skeletal key points to obtain a set of skeletal key point coordinates; Based on the set of skeletal key point coordinates, calculate the spatial distance and angle parameters between adjacent skeletal key points to obtain motion characteristic data reflecting the range of motion and posture changes; Motion feature data is input into a mapping function and transformed into emotion prior parameters, wherein the emotion prior parameters reflect the strength of the association between the target object's limb movements and different emotional states.

[0043] In this embodiment of the invention, by performing skeletal key point detection on the second analysis data, multiple preset skeletal key points are obtained and a set of skeletal key point coordinates is formed, thereby accurately capturing the upper body posture information of the target object and providing basic data for subsequent motion feature extraction.

[0044] By calculating the spatial distance and angle parameters between adjacent skeletal keypoints based on the set of skeletal keypoint coordinates, motion characteristic data reflecting the amplitude of movement and changes in posture can be obtained. For example, when a student sits upright with their arms naturally placed, the motion characteristic data shows a low amplitude and stable state; when a student frequently shakes their body or makes large hand movements, the motion characteristic data shows a high amplitude and dynamic changes.

[0045] By inputting motion feature data into a mapping function and converting it into prior parameters for emotion, a direct correlation can be established between body movements and emotional states. For example, when motion feature data shows a student leaning forward with their hands supporting their head, the mapping function can associate this state with emotions such as "focus" or "thinking"; when motion feature data shows a student frequently stretching or with their arms hanging down, the mapping function can associate this state with emotions such as "fatigue". In this way, even when facial features are missing, body movements can be used to assist in emotion recognition, thereby improving the overall recognition accuracy.

[0046] This involves detecting multiple preset skeletal keypoints to obtain a set of skeletal keypoint coordinates, specifically including: First, human pose detection is performed on the input upper body region image. This embodiment can use existing open-source human pose estimation algorithms, such as pose detection models based on convolutional neural networks and feature pyramids, or commonly used methods such as OpenPose and HRNet, to identify the main skeletal structures of the human body.

[0047] During the detection process, the system locates multiple preset skeletal key points, which typically include the head, neck, shoulders, elbows, wrists, and upper torso. Each skeletal key point is represented in pixel coordinates, that is, its position is recorded in the input image coordinate system.

[0048] After detection, the coordinate information of all key points is organized and stored in a preset order to form a skeletal key point coordinate set. This set can be viewed as a data structure containing several coordinate points, used to describe the pose state of the target object's upper body in the current frame, providing input for subsequent motion feature calculation.

[0049] Specifically, based on the set of skeletal keypoint coordinates, the spatial distance and angle parameters between adjacent skeletal keypoints are calculated to obtain motion feature data reflecting the amplitude of movement and changes in posture, including: First, for any two adjacent keypoints in the skeletal keypoint coordinate set, calculate the spatial distance between them. This distance can be obtained by measuring the straight line length between the two keypoints on the image plane. For example, the distance between the left shoulder and the left elbow can reflect the range of arm extension; the distance between the neck and the head can reflect changes in head position.

[0050] Next, for the joint structure formed by three adjacent skeletal key points, the corresponding angle parameters are calculated. The angle parameters are used to represent the degree of flexion and range of motion of the joint. For example, the angle formed by the shoulder, elbow, and wrist can reflect the flexion of the arm, while the angle formed by the neck, shoulder, and torso can reflect the tilting posture of the upper body.

[0051] After obtaining the spatial distance and angle parameters, these data are combined to form motion feature data. Motion feature data can comprehensively reflect the range of motion and posture changes of the target object. For example, when multiple joint angles change drastically in a short period, it indicates that the target object is in an active or unstable state; when the spatial distance and angle of most skeletal key points remain stable, it indicates that the target object is in a static or focused state.

[0052] In a preferred embodiment of the present invention, the weights of facial expression features and body posture features in feature fusion are adaptively adjusted according to emotion prior parameters to generate a fused feature vector. The emotion prior parameters are then used to determine the emotion category label of the frame, including: Based on prior emotion parameters, the confidence values ​​of facial expression features and limb feature parameters are weighted and fused to generate a fused feature vector. Based on the fused feature vectors, basic emotion category labels are generated according to preset discrimination criteria; Obtain the corresponding historical time-series sentiment data, and perform statistical analysis on the continuity and volatility of basic sentiment category labels within a preset time window to obtain the analysis results; Based on the analysis results, the basic emotion category labels are revised or refined to generate the final emotion category labels.

[0053] In this embodiment of the invention, by weighting and fusing the confidence values ​​of facial expression features and limb feature parameters according to prior emotion parameters, a fused feature vector is generated, enabling the system to achieve dynamic feature balance under different situations. When facial features are clear, the system automatically increases the weight of facial expression features; when the face is occluded, the weight of limb posture features is increased, thereby ensuring that the fused feature vector can always accurately reflect the emotional state of the target object.

[0054] By generating basic emotion category labels based on the fused feature vector and according to preset discrimination criteria, emotion classification can be completed at the fusion level. For example, when the confidence level of facial features in the fused feature vector is high and the corners of the mouth are upturned, the system identifies it as "happy"; when the limb feature parameters show that the body is drooping and the head is lowered, the system identifies it as "fatigued". This judgment method takes into account both facial and limb features, improving the accuracy and reliability of emotion recognition.

[0055] By acquiring corresponding historical temporal emotion data and performing statistical analysis on the continuity and volatility of basic emotion category labels within a preset time window, instantaneous fluctuations in emotion recognition results can be avoided. For example, when a student briefly lowers their head in class and then returns to a normal state, relying solely on the recognition result of the current frame might lead to a misjudgment of "indifference." However, by combining temporal analysis, it can be identified that the emotion label is only a short-term fluctuation, thereby improving the overall stability of emotion recognition.

[0056] Based on statistical analysis, the basic emotion category labels are revised or refined to generate final emotion category labels, which can more accurately reflect the true emotional state of the target subject. For example, if the basic label is "focused" continuously within a time window, but "indifference" appears briefly in between, the system will automatically correct the label and output a continuous "focused" state, thereby improving the continuity and authenticity of the emotion change records.

[0057] Specifically, based on prior emotion parameters, the confidence values ​​of facial expression features and limb feature parameters are weighted and fused to generate a fused feature vector, which includes: First, facial expression feature information is acquired, and confidence values ​​corresponding to each emotion category are extracted from the model output. These confidence values ​​reflect the degree of matching between different facial expression features and each emotion category. Simultaneously, limb feature parameters of the upper body region are acquired, derived from the spatial distribution and range of motion features of the aforementioned skeletal key points.

[0058] Next, the two types of features are weighted and fused based on the prior emotion parameters. Specifically, when the prior emotion parameters indicate a high degree of facial feature integrity, the system automatically increases the weight of the facial feature confidence value during the fusion process; when the prior emotion parameters indicate significant facial occlusion, the system increases the weight of the limb feature parameters.

[0059] Finally, the weighted facial expression features and body posture features are concatenated or combined into a multi-dimensional vector to form a fused feature vector. This fused feature vector can simultaneously represent emotional information from both the face and the body, and dynamically reflects the moderating effect of prior parameters on the fusion result.

[0060] Specifically, based on the fused feature vector, basic emotion category labels are generated according to preset discrimination criteria, including: First, the fused feature vector is input into the classification model, which can be a pre-trained convolutional neural network, recurrent neural network, or a traditional support vector machine or decision tree classifier.

[0061] Subsequently, the classification model determines the emotion type of the fused feature vector according to a preset discrimination criterion. This discrimination criterion can be a classification probability threshold corresponding to each emotion category, or a discrimination boundary learned through model training.

[0062] When the fused feature vector matches a certain emotion category with the highest degree of matching and exceeds the judgment threshold, the system generates the corresponding basic emotion category label. For example, when the confidence value of facial features in the fused feature vector shows that the corners of the mouth are turned up, the eyes are curved, and the body posture is relaxed, the classification result may correspond to the label "happy"; when facial features are missing but the body posture shows a continuous forward lean, the classification result may correspond to the label "focused".

[0063] Based on the analysis results, the basic emotion category labels are revised or refined to generate the final emotion category labels, specifically including: First, after generating basic emotion category labels, the system obtains a sequence of historical emotion labels within a preset time window and performs continuity and volatility analysis on them. This analysis may include statistically analyzing the duration, frequency, and number of abrupt changes of emotion labels within the window.

[0064] Next, the analysis results are compared with the current base emotion category label. If the analysis results show that the current label is consistent with historical trends, the original judgment is maintained; if there is a significant conflict, the label is corrected. For example, if the emotion label has been "focused" for the past 30 seconds, but the current base label is "indifferent," the system will correct the result based on the continuity of historical data and output "focused."

[0065] Finally, for some ambiguous or borderline emotional states, the system can refine the basic labels. For example, when the fused feature vector has low distinguishability between "focus" and "thinking", the system can combine trend analysis results to refine the basic label "focus" into "deep thinking", thereby outputting a final emotion category label that is more in line with reality.

[0066] In a preferred embodiment of the present invention, the calculation of the facial key point visibility score includes: Obtain the color distribution histogram of the key point region and calculate its correlation score with the color distribution of the standard facial region; Based on the edge gradient distribution of pixels around the key point, local edge features are extracted, and multidimensional feature vectors are constructed by combining the image sharpness value and pixel grayscale variance. The multidimensional feature vectors are fused according to the preset feature weighting coefficients to obtain the visibility score of the key points.

[0067] In this embodiment of the invention, by acquiring the color distribution histogram of the facial key point region and calculating its correlation score with the color distribution of the standard facial region, it is possible to effectively determine whether there are any abnormalities in the key point region. For example, when a part of the target object's face is disturbed by lighting, the color distribution of that region will deviate significantly from the standard distribution, resulting in a lower correlation score and thus reflecting insufficient visibility of the key point.

[0068] By extracting local edge features based on the edge gradient distribution of pixels surrounding keypoints, and combining this with image sharpness values ​​and pixel grayscale variance, a multi-dimensional feature vector is constructed. This allows the system to comprehensively evaluate the visibility of keypoints using multiple metrics. For example, when a student covers part of their face with their hand, the edge gradient distribution in that area will change abnormally. Combined with sharpness and variance metrics, this can accurately identify that the keypoint has been partially occluded.

[0069] By fusing multidimensional feature vectors according to preset feature weighting coefficients, the visibility score of key points is obtained, ensuring that different features have a reasonable influence in visibility assessment. For example, if the edge features of an eye key point are abnormal but the color distribution is still normal, the system can obtain a reasonable visibility score through fusion calculation, thus avoiding misjudgment caused by anomalies in a single indicator. This mechanism can more accurately reflect the usability of each key point in the facial region, providing a reliable basis for subsequent facial integrity parameter calculations.

[0070] This includes obtaining the color distribution histogram of the key point region and calculating its correlation score with the color distribution of the standard facial region, specifically including: First, the neighborhood region of the target key point is determined in the facial region. This region is usually a rectangular window of a certain size centered on the key point, which is used to reflect the color distribution characteristics around the key point.

[0071] Next, color statistics are performed on the neighborhood region to generate a color distribution histogram. The histogram can be based on the RGB color space or the HSV color space, which is more in line with human visual perception. The pixels in the region are grouped and statistically analyzed according to different color channels to obtain the pixel frequency distribution of each channel.

[0072] Subsequently, the color distribution histogram is compared with a pre-established standard facial region color distribution histogram. The standard histogram can be generated from a large number of facial sample images acquired under uniform lighting conditions and is used as a reference benchmark. The comparison between the two uses a correlation measurement method, such as the histogram similarity coefficient. The higher the correlation score, the closer the color distribution of the key point region is to that of the normal facial region, thus indicating that the key point is more visible.

[0073] Specifically, based on the edge gradient distribution of pixels surrounding key points, local edge features are extracted, and combined with image sharpness values ​​and pixel grayscale variances to construct a multi-dimensional feature vector, which includes: First, gradient calculations are performed on pixels within the neighborhood of the target keypoint to obtain the intensity changes in the horizontal and vertical directions. By comparing the gradient distributions in different directions, the edge features of the keypoint region can be extracted, reflecting the clarity of the texture and contour of that region.

[0074] Next, the image sharpness value of that area is used to measure whether there is blur in the local area. For example, the sharpness of the image can be reflected by statistically analyzing the magnitude of pixel grayscale changes within the neighborhood; higher sharpness indicates more obvious details.

[0075] Simultaneously, the pixel grayscale variance of the region is calculated to characterize the dispersion of pixel values. A higher grayscale variance indicates that there are obvious texture or contour changes in the region, while a lower variance indicates that the region is too smooth or lacks effective features.

[0076] Finally, the edge features, image sharpness values, and pixel grayscale variance are combined to form a multidimensional feature vector. This vector can comprehensively reflect the visibility status of the key point in the local area.

[0077] Specifically, the multi-dimensional feature vectors are fused according to preset feature weighting coefficients to obtain the visibility score of key points, which includes: First, pre-defined weighting coefficients are assigned to each feature component in the multi-dimensional feature vector. These weighting coefficients can be set through experimental experience or obtained through statistical learning based on the training dataset, and are used to reflect the importance of different features in determining the visibility of key points. For example, in low-light scenes, the weight of edge features can be appropriately increased, while the weight of color distribution can be relatively decreased.

[0078] Next, each component in the multidimensional feature vector is combined with its corresponding weighting coefficient to obtain a weighted result. Subsequently, all weighted results are accumulated or combined to form a single numerical value, which is the visibility score of the keypoint.

[0079] Finally, the visibility score is used to represent the visibility of the keypoint region in the current frame. A higher score indicates a clearer and more reliable keypoint feature; a lower score indicates that the region is blurred, occluded, or has missing features. In this way, the system can provide reliable input for subsequent calculations of overall facial integrity parameters.

[0080] In a preferred embodiment of the present invention, the transformation process of the mapping function includes: Establish multiple sets of association rules between body posture features and emotional states, including the correspondence between movement amplitude, movement duration, movement coordination and emotional type; The motion feature data is normalized to obtain a standardized motion feature vector; Based on the standardized motion feature vector, the degree of matching with each association rule is determined, and the correlation score of each emotional state is calculated. Based on the relevance score, the emotional state with the highest matching degree is selected, and the corresponding emotional prior parameters are generated. The mapping function can adaptively adjust the association rules and parameters according to the actual application scenario.

[0081] In this embodiment of the invention, by setting multiple sets of association rules between body posture features and emotional states, a correspondence can be established between action features and emotional categories. For example, if the rule stipulates that when the upper body leans forward and the action lasts for a long time, the corresponding emotional state is "focused"; if frequent body movements and low coordination are detected, the corresponding emotional state is "anxious". This rule setting enables the system to quickly determine the emotion type in different scenarios.

[0082] By normalizing motion feature data, standardized motion feature vectors are obtained, which can eliminate differences between individuals caused by factors such as body proportions and camera distance, making the same action comparable among different individuals. For example, students of different heights may have different spatial displacement amplitudes in a hand-raising motion, but after normalization, the system can unify them into standardized amplitudes to ensure the fairness and consistency of the recognition results.

[0083] By determining the degree of matching between standardized motion feature vectors and various association rules, and calculating the correlation scores for each emotional state, parallel determination of multiple emotional states can be achieved. For example, when a student exhibits both "thinking" motion characteristics and "fatigue" body signals, the system can calculate corresponding correlation scores for different emotions, thereby reflecting the complexity of emotional states.

[0084] Based on the relevance score, the emotional state with the highest matching degree is selected to generate the corresponding emotional prior parameter, so that the subsequent feature fusion process can obtain a stable and reliable reference value. For example, when the student's action features match the "focused" emotion most closely, the emotional prior parameter will be set to "focused," and the weight of focus-related features will be increased during the feature fusion process, thereby improving the accuracy of recognition.

[0085] The mapping function supports adaptive adjustment of association rules and parameters based on actual application scenarios, enabling the system to continuously optimize emotion recognition performance. For example, in a remote education environment, the system can dynamically adjust the distinction threshold between "focused" and "indifferent" based on historical data to better reflect students' actual performance characteristics, thereby improving the robustness of emotion judgment.

[0086] This includes establishing association rules between multiple sets of body posture features and emotional states, specifically including: First, a large number of body movement samples containing different emotional states are collected. These samples can be derived from manually labeled video datasets, such as classroom learning scenarios and driver monitoring scenarios. By statistically analyzing these samples, body posture features that are significantly correlated with different emotional states are extracted.

[0087] Next, based on statistical results or expert experience, multiple sets of correspondence rules between body postures and emotional states are established. For example, if a large forward tilt of the head is detected and the duration is relatively long, the action is highly associated with the emotional state of "focus"; if frequent upper body swaying and lack of coordination in arm movements are detected, the action is highly associated with the emotional state of "anxiety" or "impatience".

[0088] When setting rules, each set of rules can include one or more parameter conditions, such as the relative distance between key limb points, the magnitude of the movement, the duration of the movement, and the coordination between movements. Each rule is associated with one or more emotional states, which are used as a reference standard in subsequent judgments.

[0089] Specifically, based on the standardized motion feature vector, the degree of matching with each association rule is determined, and the correlation score of each emotional state is calculated, including: First, the extracted motion feature data is normalized to obtain a standardized motion feature vector, so that differences in height, body shape or shooting distance among different individuals will not affect subsequent judgments.

[0090] Next, the standardized motion feature vector is compared one by one with each preset posture-emotion association rule. The comparison process includes checking whether the parameter values ​​in the standardized feature vector meet the condition range of a certain rule and calculating the degree of satisfaction. For example, when the rule requires "the upper body leaning forward at an angle greater than a certain threshold and lasting for a certain period of time", if the corresponding parameter in the standardized motion feature vector is close to or completely consistent with this condition, it indicates that the feature matches the rule to a high degree.

[0091] Subsequently, a relevance score is assigned to different emotional states based on the degree of matching. The magnitude of the relevance score depends on how closely the standardized feature vector matches the rule conditions: a high score is given for a complete match, a medium score for a partial match, and a low score for a complete mismatch.

[0092] Finally, the matching results of all rules are summarized, and the relevance scores for each emotional state are accumulated or weighted to obtain the overall relevance level between the current target object and each emotional state. This result will serve as an important basis for generating prior emotional parameters in subsequent steps.

[0093] In a preferred embodiment of the present invention, the statistical analysis of the historical time-series sentiment data includes: Within a preset time window, the historical emotion tag sequence is statistically analyzed to obtain the duration and switching frequency of various emotion tags, and thus generate continuity and volatility parameters of emotion state. Based on continuity and volatility parameters, a trend analysis is performed on the historical sentiment label sequence to determine whether the current basic sentiment category label is in an upward, downward, or stable phase, and a trend determination result is generated. Based on the continuity parameter, volatility parameter, and trend determination results, the analysis results of the emotional state are output to guide the correction or refinement of the basic emotional category labels.

[0094] Based on the continuity parameter, volatility parameter, and trend determination results, the analysis results of the emotional state are output to guide the correction or refinement of the basic emotional category labels.

[0095] In this embodiment of the invention, by statistically analyzing historical emotion tag sequences within a preset time window, the duration and transition frequency of various emotion tags are obtained, thereby generating continuity and volatility parameters of the emotional state. This allows the system to quantify the stability and trend of emotions. For example, if a student consistently exhibits "focus" for five minutes, the system can generate a high continuity parameter, indicating that the emotional state is stable.

[0096] By performing trend analysis on historical emotion tag sequences based on continuity and volatility parameters, the system can determine whether the current basic emotion category tag is in an upward, downward, or stable phase, and generate trend judgment results, thus providing a dynamic description of emotion changes. For example, when the frequency of the emotion "fatigue" gradually increases over a period of time, the system can determine that the emotion is in an upward phase, thereby indicating that teachers' and students' attention may be declining.

[0097] Based on continuity parameters, volatility parameters, and trend determination results, the system outputs analysis results of emotional states, providing a basis for subsequent label correction or refinement. Through this analysis, the system can avoid erroneous judgments caused by transient anomalies and provide more stable decision support for emotion recognition. For example, if the emotion label briefly displays as "indifferent" in a certain frame, but the analysis results indicate that the overall trend is still "focused," this anomaly will not directly affect the final label output, thus ensuring the match between the emotion recognition results and the actual emotional expression.

[0098] By outputting the analysis results, a comprehensive assessment of the continuity, volatility, and trend of historical sentiment data was achieved. This not only improved the stability and robustness of sentiment recognition but also provided reliable data support for subsequent label correction and refinement.

[0099] Specifically, "within a preset time window, statistical analysis is performed on historical emotion tag sequences to obtain the duration and transition frequency of various emotion tags, thereby generating continuity and volatility parameters of emotional states," includes: First, a fixed-duration time window, such as 30 or 60 seconds, is set in the system to cache and analyze the target object's emotion tag sequence within that time period. This sequence is generated frame-by-frame by the aforementioned emotion determination step and includes the target object's emotional state at different points in time.

[0100] Next, statistical analysis was performed on the emotion tags within the window. The statistics included: the duration of each emotion tag appearing continuously within the window, i.e., the duration, and the number of times different emotion tags were switched, i.e., the switching frequency.

[0101] Subsequently, two parameters are generated based on the statistical data: Continuity parameter: used to reflect the degree to which an emotional state remains stable over a certain period of time; the longer the duration, the higher the continuity parameter. Volatility parameter: used to reflect the fluctuation of emotional state within a certain period of time. The higher the frequency of transitions, the greater the volatility parameter.

[0102] Through such statistical and parameterized processing, the system can transform the original discrete label sequence into numerical features with greater analytical value, providing input for subsequent trend analysis.

[0103] Specifically, based on continuity and volatility parameters, trend analysis is performed on historical sentiment label sequences to determine whether the current basic sentiment category label is in an upward, downward, or stable phase, and trend determination results are generated, including: First, the continuous and volatile parameters generated within the time window are read and compared with the data from the previous window to obtain the trend of emotional state changes over time.

[0104] Next, by combining the proportion of the sentiment tag appearing within the window, we analyze the trend of the tag over time: If the continuity parameter of a certain emotion shows a gradual increasing trend, while the volatility parameter is low, then the emotion is judged to be in the rising or strengthening phase. If the continuity parameter of a certain emotion shows a downward trend and the volatility parameter increases, then the emotion is determined to be in a downward or weakening phase. If the continuity and volatility parameters remain relatively stable, the emotion is considered to be in a stable phase.

[0105] Finally, the results of the trend analysis are output as trend determination results. These results can be represented by textual or numerical labels, such as "rising," "falling," or "stable," and serve as an important basis for subsequent revision or refinement of the basic sentiment category labels.

[0106] Embodiments of the present invention also provide an AI-based visual emotion analysis system, the system comprising: The preprocessing unit is used to acquire the original image data sequence containing the target object, and preprocess each frame of it to segment the face region and the upper body region, and generate the first analysis data and the second analysis data respectively. The facial integrity detection unit is used to perform local feature integrity detection on the first analysis data, calculate the visibility score of facial key points, and generate facial integrity parameters based on the score; when the facial integrity parameters are lower than a preset first threshold, occlusion marker information of the corresponding frame is generated. The limb feature extraction unit is used to perform skeletal key point detection on the second analysis data based on the occlusion marker information, extract limb posture features, obtain limb feature parameters, and convert the limb feature parameters into the emotion prior parameters of the frame through a mapping function. The feature fusion and determination unit is used to adaptively adjust the weights of facial expression features and body posture features in feature fusion based on the prior emotion parameters, generate a fused feature vector, and determine the emotion of the fused feature vector in combination with the prior emotion parameters to obtain the emotion category label of the frame. The emotion recording unit is used to bind the emotion category label with the timestamp of the frame to form a record of the target object's emotion changes.

[0107] It should be noted that this system is a system corresponding to the above method. All implementation methods in the above method embodiments are applicable to this embodiment and can achieve the same technical effect.

[0108] Embodiments of the present invention also provide a computing device, including: a processor and a memory storing a computer program, wherein the computer program, when executed by the processor, performs the method described above. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0109] Embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described above. All implementations in the above method embodiments are applicable to this embodiment and can achieve the same technical effects.

[0110] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An AI-based visual emotion analysis method, characterized in that, The method includes: The original image data sequence containing the target object is obtained, and each frame is preprocessed to segment the face region and upper body region, generating the first analysis data and the second analysis data respectively. Local feature integrity detection is performed on the first analysis data: multiple preset facial key points are detected, color distribution histograms of key point regions are obtained, and their correlation scores with the color distribution of standard facial regions are calculated; local edge features are extracted based on the edge gradient distribution of pixels around the key points, and multi-dimensional feature vectors are constructed by combining image sharpness values ​​and pixel grayscale variances; the multi-dimensional feature vectors are fused according to preset feature weighting coefficients to obtain key point visibility scores; all key point visibility scores are weighted and accumulated according to preset key point weighting coefficients to obtain overall facial integrity parameters; when the facial integrity parameters are lower than a preset first threshold, occlusion marker data is generated to characterize the degree of facial occlusion based on the key point distribution, and the occlusion marker data serves as the trigger for starting the subsequent skeletal key point detection process; Based on the occlusion marker information, skeletal keypoint detection is performed on the second analysis data to extract limb posture features, obtain limb feature parameters, and then the limb feature parameters are converted into the emotion prior parameters of the frame through a mapping function. Based on the prior emotion parameters, the weights of facial expression features and body posture features in feature fusion are adaptively adjusted to generate a fused feature vector. The prior emotion parameters are then used to determine the emotion of the fused feature vector to obtain the emotion category label for the frame. By binding the emotion category label to the timestamp of the frame, a record of the target object's emotion changes is formed.

2. The AI-based visual emotion analysis method according to claim 1, characterized in that, Based on the occlusion marker information, skeletal keypoint detection is performed on the second analysis data to extract limb pose features, obtain limb feature parameters, and then the limb feature parameters are converted into prior emotion parameters for the frame through a mapping function, including: Detect multiple preset skeletal key points to obtain a set of skeletal key point coordinates; Based on the set of skeletal key point coordinates, calculate the spatial distance and angle parameters between adjacent skeletal key points to obtain motion characteristic data reflecting the range of motion and posture changes; Motion feature data is input into a mapping function and transformed into emotion prior parameters, wherein the emotion prior parameters reflect the strength of the association between the target object's limb movements and different emotional states.

3. The AI-based visual emotion analysis method according to claim 1, characterized in that, Based on prior emotion parameters, the weights of facial expression features and body posture features in feature fusion are adaptively adjusted to generate a fused feature vector. The fused feature vector is then used to determine the emotion category of the frame by combining the prior emotion parameters, resulting in the emotion category label for that frame, including: Based on prior emotion parameters, the confidence values ​​of facial expression features and limb feature parameters are weighted and fused to generate a fused feature vector. Based on the fused feature vectors, basic emotion category labels are generated according to preset discrimination criteria; Obtain the corresponding historical time-series sentiment data, and perform statistical analysis on the continuity and volatility of basic sentiment category labels within a preset time window to obtain the analysis results; Based on the analysis results, the basic emotion category labels are revised or refined to generate the final emotion category labels.

4. The AI-based visual emotion analysis method according to claim 2, characterized in that, The transformation process of the mapping function includes: Establish multiple sets of association rules between body posture features and emotional states, including the correspondence between movement amplitude, movement duration, movement coordination and emotional type; The motion feature data is normalized to obtain a standardized motion feature vector; Based on the standardized motion feature vector, the degree of matching with each association rule is determined, and the correlation score of each emotional state is calculated. Based on the relevance score, the emotional state with the highest matching degree is selected, and the corresponding emotional prior parameters are generated. The mapping function can adaptively adjust the association rules and parameters according to the actual application scenario.

5. The AI-based visual emotion analysis method according to claim 3, characterized in that, The statistical analysis of the historical time-series sentiment data includes: Within a preset time window, the historical emotion tag sequence is statistically analyzed to obtain the duration and switching frequency of various emotion tags, and thus generate continuity and volatility parameters of emotion state. Based on continuity and volatility parameters, a trend analysis is performed on the historical sentiment label sequence to determine whether the current basic sentiment category label is in an upward, downward, or stable phase, and a trend determination result is generated. Based on the continuity parameter, volatility parameter, and trend determination results, the analysis results of the emotional state are output to guide the correction or refinement of the basic emotional category labels.

6. An AI-based visual emotion analysis system, characterized in that, The system, used in the method of any one of claims 1 to 5, comprises: The preprocessing unit is used to acquire the original image data sequence containing the target object, and preprocess each frame of it to segment the face region and the upper body region, and generate the first analysis data and the second analysis data respectively. The facial integrity detection unit is used to perform local feature integrity detection on the first analysis data, calculate the visibility score of facial key points, and generate facial integrity parameters based on the score; when the facial integrity parameters are lower than a preset first threshold, occlusion marker information of the corresponding frame is generated. The limb feature extraction unit is used to perform skeletal key point detection on the second analysis data based on the occlusion marker information, extract limb posture features, obtain limb feature parameters, and convert the limb feature parameters into the emotion prior parameters of the frame through a mapping function. The feature fusion and determination unit is used to adaptively adjust the weights of facial expression features and body posture features in feature fusion based on the prior emotion parameters, generate a fused feature vector, and determine the emotion of the fused feature vector in combination with the prior emotion parameters to obtain the emotion category label of the frame. The emotion recording unit is used to bind the emotion category label with the timestamp of the frame to form a record of the target object's emotion changes.

7. A computing device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that, when executed by a processor, implements the method as described in any one of claims 1 to 5.

Images