Emotion Recognition Methods and Systems Based on Human Micro-Moves

Abstract

This disclosure provides a method and system for emotion recognition based on human micro-movements, belonging to the field of micro-movement recognition technology. The method comprises: performing feature fusion based on target video features, target skeleton features, and target amplitude features of the target to obtain comprehensive features of the target; obtaining micro-movement features of the target based on the comprehensive features; and classifying the micro-movement features to perform emotion recognition on the target. The method and system for emotion recognition based on human micro-movements provided by this disclosure can improve the accuracy of micro-movement recognition.

Description

Technical Field

[0001] This disclosure belongs to the field of micro-motion recognition technology, and more specifically, relates to an emotion recognition method and system based on human micro-motions. Background Technology

[0002] With the rapid development of computer vision technology, Human Action Recognition (HAR) has become an important research direction in this field. Traditional HAR tasks mainly focus on large-amplitude movements in videos, such as walking and jumping. However, in recent years, Micro-Action Recognition (MAR) has emerged as a new research direction and has begun to receive increasing attention. Micro-actions refer to subtle, unconscious movements, usually manifested as brief changes in posture, gestures, or facial expressions. These movements can reflect an individual's emotions, psychological state, or intentions. The challenge of MAR tasks lies in the fact that micro-actions typically have small amplitudes, rapid execution speeds, and often appear intermittently between different body parts, making them difficult for traditional HAR models to accurately capture. Summary of the Invention

[0003] The purpose of this disclosure is to provide a method and system for emotion recognition based on human micro-movements, so as to improve the accuracy of micro-movement recognition.

[0004] A first aspect of this disclosure provides an emotion recognition method based on human micro-movements, comprising:

[0005] Feature fusion is performed based on the target video features, target skeleton features, and target amplitude features to obtain the comprehensive features of the target.

[0006] The micro-motion features of the target are obtained based on the comprehensive features;

[0007] Based on the micro-motion features, the target is classified and its emotion is identified.

[0008] A second aspect of this disclosure provides an emotion recognition system based on human micro-movements, comprising:

[0009] The feature fusion module is used to fuse the target video features, target skeleton features and target amplitude features based on the target to obtain the comprehensive features of the target.

[0010] The micro-motion recognition module is used to obtain the micro-motion features of the target based on the comprehensive features;

[0011] The emotion recognition module is used to classify based on the micro-motion features and to perform emotion recognition on the target.

[0012] A third aspect of this disclosure provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the above-described emotion recognition method based on human micro-movements.

[0013] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described emotion recognition method based on human micro-movements.

[0014] The beneficial effects of the emotion recognition method and system based on human micro-movements provided in this disclosure are as follows: This disclosure achieves accurate capture of the comprehensive features of the target by fusing target video features, target skeleton features, and target amplitude features. First, by integrating target video features, target skeleton features, and target amplitude features, the complementarity of different modal information is fully utilized, enabling comprehensive and accurate capture of the target's action details, avoiding the limitations of single features, and greatly improving the accuracy and reliability of micro-movement recognition. Second, further extraction of micro-movement features based on the comprehensive features helps to focus on the key characteristics of minute movements, highlighting the differences between subtle movements and providing a more discriminative basis for subsequent classification. Finally, efficient emotion recognition based on micro-movement features improves the accuracy and reliability of recognition. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating an embodiment of the emotion recognition method based on human micro-movements provided in this disclosure;

[0017] Figure 2 This is a schematic diagram of a joint motion amplitude image provided in an embodiment of the present disclosure;

[0018] Figure 3 A structural block diagram of an emotion recognition system based on human micro-movements provided in an embodiment of this disclosure;

[0019] Figure 4 This is a schematic block diagram of an electronic device provided according to an embodiment of the present disclosure. Detailed Implementation

[0020] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of the embodiments of this disclosure. However, those skilled in the art will understand that this disclosure may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this disclosure with unnecessary detail.

[0021] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0022] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of an emotion recognition method based on human micro-movements provided in this disclosure. The method includes:

[0023] S101: Based on the target video features, target skeleton features and target amplitude features of the target to be identified, feature fusion is performed to obtain the comprehensive features of the target to be identified.

[0024] In this embodiment, the target video features may include visual information in the video, such as the dynamic changes of actions, color, texture and other characteristics in the time series, which are extracted from the original video data through video processing technology or pre-trained models.

[0025] The target skeleton features refer to the position and connection information of human joints. Skeletal features can reflect the posture and movement trajectory of the human body and help to capture the basic framework of the human body's posture and movements. In this embodiment, key point detection algorithms can be used to extract them from the video.

[0026] Target amplitude features primarily reflect the magnitude of the motion, referring to the displacement amplitude of joints between different frames. These features reflect the intensity and temporal information of micro-movements. By calculating the displacement amplitude of each joint between adjacent frames and encoding it into an image format, rich amplitude features can be obtained. Target amplitude features help capture subtle changes and rapid occurrences in micro-movements.

[0027] In this embodiment, the target video features, target skeleton features, and target amplitude features are fused to obtain the comprehensive features of the identified target.

[0028] This embodiment can automatically learn the correlation and importance weights between different features by concatenating feature vectors, weighted summing, or using the attention mechanism of deep learning models (such as Transformer), thereby obtaining a high-dimensional feature representation that integrates video, skeleton, and amplitude information.

[0029] For example, the comprehensive features of the target can be obtained by fusing the data using a Transformer model.

[0030] Suppose we have a video in which a person is making subtle hand movements. After feature extraction, we obtain the target video features. Target skeleton features and target amplitude characteristics Then , and according to The features are concatenated into a multimodal vector and input into the Transformer model. The Transformer's self-attention mechanism can automatically focus on the correlations between different modal features. For example, when recognizing hand tension micro-movements, it can find close relationships between the tension of hand muscles in video features, the special angle changes of finger joints in skeletal features, and the rapid and subtle changes in the amplitude of motion in amplitude features. After multi-layer processing, a comprehensive feature that can accurately represent hand micro-movements is finally obtained, which can be used for subsequent micro-movement recognition and sentiment analysis tasks.

[0031] S102: Obtain the micro-motion features of the target based on comprehensive features.

[0032] In this embodiment, multi-layer processing can be performed based on the Transformer model to obtain the micro-motion features of the target.

[0033] For example, each Transformer operation includes a multi-head self-attention mechanism, residual connections, layer normalization, and a feedforward neural network. Through the combination of N Transformer operations containing the above operations, the input features are continuously refined and abstracted, gradually extracting the micro-motion features of the target to be identified. Each layer builds upon the previous layer to further explore deeper information about the features, ultimately yielding... It can accurately reflect the micro-motion characteristics of the target, providing strong support for subsequent tasks such as classification and emotion recognition. Micro-motion characteristics of the target Expressed as:

[0034] in, This represents a combination of N Transformer operations. This represents a multimodal vector.

[0035] S103: Classify based on micro-motion features and perform emotion recognition on the target.

[0036] In this embodiment, classification is performed based on extracted micro-motion features to achieve emotion recognition of the target. During the classification process, an emotion classification model can be pre-established. This model is trained on a large amount of labeled data to learn the mapping relationship between different micro-motion features and emotion categories. In practical applications, the extracted micro-motion features are input into the trained model. The model compares the input features with the learned patterns and outputs the most likely corresponding emotion category, thereby completing the judgment of the target's emotional state.

[0037] As can be seen from the above, this embodiment achieves accurate capture of the comprehensive features of the target by fusing target video features, target skeleton features, and target amplitude features. First, by integrating target video features, target skeleton features, and target amplitude features, the complementary nature of different modal information is fully utilized, enabling comprehensive and accurate capture of the target's action details. This avoids the limitations of single features and greatly improves the accuracy and reliability of micro-motion recognition. Second, further extraction of micro-motion features based on the comprehensive features helps to focus on the key characteristics of minute movements, highlighting the differences between subtle movements and providing a more discriminative basis for subsequent classification. Finally, efficient emotion recognition based on micro-motion features improves the accuracy and reliability of the recognition.

[0038] In one embodiment of this disclosure, it further includes:

[0039] Based on a convolutional neural network, feature extraction is performed on the target video to identify the target, and the image features of each frame are obtained.

[0040] The target video features are obtained based on the image features of each frame.

[0041] In this embodiment, the target video for target identification is processed frame by frame and input into a pre-trained convolutional neural network. middle.

[0042] Pre-trained convolutional neural networks learn from large amounts of image data beforehand, giving them powerful image feature extraction capabilities. For example, they can identify basic visual elements such as edges, textures, and shapes in images, and gradually abstract and combine these elements through structures such as convolutional layers and pooling layers to form higher-level image feature representations.

[0043] Processing image frames at time t in a video hour, Based on its internal convolution kernel parameters and network structure, it can perform convolution operations on different regions of the image to extract local features. Then, it performs dimensionality reduction processing through pooling layers to reduce computational cost while retaining key information, ultimately outputting high-dimensional image features. .

[0044] The image features of each frame are represented as follows:

[0045] In this embodiment, the above operation is performed on each frame of the video to obtain an image feature over time. These image feature sets constitute the target video features, which not only contain the static visual information of each frame, but also, because they are extracted in chronological order, contain information about the changes in actions in the video over time, providing important basic data for subsequent tasks such as micro-motion recognition and emotion analysis.

[0046] As can be seen from the above, this embodiment uses a convolutional neural network for feature extraction, which can accurately capture detailed information in each frame of the image, such as object shape and texture, providing rich material for subsequent analysis. Secondly, by integrating the features of each frame to form target video features, the key information of each frame is preserved while incorporating the dynamic changes of the video, making the obtained video features more comprehensive and representative, significantly improving the accuracy and reliability of subsequent micro-motion recognition and related analysis.

[0047] In one embodiment of this disclosure, it further includes:

[0048] Extract key point data of the target based on the target video;

[0049] Two-dimensional skeleton data of the target to be identified is generated based on key point data;

[0050] The two-dimensional skeleton data is analyzed based on the depth estimation algorithm to obtain the three-dimensional skeleton data of the target to be identified.

[0051] Feature extraction is performed on the 3D skeleton data to obtain the target skeleton features for identification.

[0052] In this embodiment, Media Pipe can be used to extract joint point data of the human body in the target video to generate two-dimensional skeleton data. This process involves identifying the positions of human joints in video frames, determining the two-dimensional coordinates of each joint point, and thus constructing the basic structural framework of the human body. This can be represented as:

[0053]

[0054] in, It is the first The two-dimensional coordinates of each key point in the image frame It represents the total number of joints.

[0055] Next, the two-dimensional skeleton data is processed using a depth estimation algorithm based on a convolutional neural network to predict the depth information of each joint, thereby generating three-dimensional skeleton data, represented as follows:

[0056]

[0057] in, It is The three-dimensional coordinates of each joint point It is depth information.

[0058] In this embodiment, the target video can be a traditional monocular RGB video. The RGB video is combined with depth information to obtain RGB+D skeleton data (i.e., three-dimensional skeleton data), which is represented as follows:

[0059]

[0060] in, For RGB video data, This is used to estimate the depth information. This is how it is obtained. This indicates that it provides richer spatial and depth information, which helps to capture the details of micro-movements.

[0061] In this embodiment, based on the RGB+D skeleton data extracted from the Media Pipe, a deep neural network (such as a Transformer or MLP network) can be used to process the skeleton features of each frame. The deep neural network can learn the spatial structural features and temporal variation patterns in the 3D skeleton data, thereby extracting the target skeleton features for object recognition. These target skeleton features reflect the posture changes and joint movement patterns of the human body during micro-movements, providing crucial foundational data for micro-movement recognition and emotion analysis.

[0062] For example, based on the RGB+D skeleton data extracted from the Media Pipe, the RGB+D skeleton data of each frame is processed by a deep neural network (such as a Transformer or a simple MLP network) to obtain the target skeleton features:

[0063] in, It is the 3D skeleton feature of the frame corresponding to time t. These are the extracted spatial structural features. As can be seen from the above, this embodiment, by extracting joint point data and generating two-dimensional and three-dimensional skeleton data, can accurately construct a body structure model of the target and clearly demonstrate the basic framework of the movement. Combined with a depth estimation algorithm, spatial depth information is added, making the understanding of micro-movements more three-dimensional. Further extraction of target skeleton features allows focus on key joint changes in the movement, effectively capturing subtle posture changes, providing crucial and accurate information support for subsequent micro-movement recognition and emotion analysis, and improving the overall accuracy and reliability of the recognition.

[0064] In one embodiment of this disclosure, it further includes:

[0065] Calculate the displacement amplitude of joint data between two adjacent frames in the target video;

[0066] Generate motion amplitude images of the target based on displacement amplitude;

[0067] Feature extraction is performed on the motion amplitude image to obtain the target amplitude features.

[0068] In this embodiment, for the target video, the motion changes of the joints are measured by calculating the displacement amplitude of the joint data between two adjacent frames. The displacement of each joint at different times is calculated to reflect the dynamic characteristics of the action. Then, the displacement amplitude information is encoded into image form, and the motion amplitude of each joint is integrated into the corresponding image to generate a motion amplitude image. The image integrates the motion intensity information of all relevant nodes over time, becoming a key data source for micro-motion recognition.

[0069] Subsequently, convolutional neural networks or Transformer models can be used to extract features from motion amplitude images. This allows for the discovery of hidden patterns and regularities within the images, transforming the information in the motion amplitude images into more representative target amplitude features. These features can capture key information about the amplitude changes of micro-movements, and when combined with other features (such as video features and skeleton features), they can comprehensively improve the ability to recognize micro-movements, providing strong support for subsequent analysis and applications.

[0070] like Figure 2 As shown, for example, to capture the spatiotemporal dynamic information of micro-movements, a representation method can be used for the motion amplitude image of 33 key joints of the human body (using Media PipePose to detect the following key points: nose, left inner corner of the eye, left eye, left outer corner of the eye, right inner corner of the eye, right eye, right outer corner of the eye, left ear, right ear, above the center of the lips, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, heel of the left hand, heel of the right hand, left thumb joint, right thumb joint, left index finger joint, right index finger joint, center of the upper body (hip), left hip, right hip, left knee, right knee, left ankle, right ankle, left heel, right heel, left toe, right toe). For the t-th frame in the video, the displacement amplitude of each joint point between two adjacent frames is calculated. :

[0071]

[0072] in, This represents the x-coordinate of the i-th keypoint in frame t. This represents the x-coordinate of the i-th key point in frame (t-1); This represents the ordinate of the i-th keypoint in frame t. This represents the ordinate of the i-th key point in frame (t-1); This represents the depth coordinates of the i-th keypoint in frame t. This represents the depth coordinates of the i-th keypoint in frame (t-1).

[0073] Then, the motion amplitude of each joint is encoded into an image, represented as follows:

[0074]

[0075] in, For the Each key point is Motion amplitude at each frame. Motion amplitude information from all time steps is combined to generate a two-dimensional image. These amplitude images serve as motion amplitude images from a video. They provide temporal information about motion intensity and are an important feature for micro-motion recognition.

[0076] Joint motion amplitude image for each frame Feature extraction is performed using a convolutional neural network (CNN) or a Transformer model to obtain the target amplitude features for each frame: ,

[0077] in, It is an image of the motion amplitude of each joint point in each frame. It is the target amplitude feature extracted from each frame.

[0078] As can be seen from the above, this embodiment, by calculating the displacement amplitude of joint points and generating motion amplitude images, can intuitively present the dynamic changes of micro-movements in the time dimension, providing a clear basis for amplitude changes in subsequent analysis. Further extraction of target amplitude features can accurately capture the intensity and rhythm information of the movement. These features, combined with other modal features, can significantly enhance the overall perception of micro-movements, improve the accuracy and comprehensiveness of recognition, and provide more reliable data support and analytical foundation for micro-movement recognition applications.

[0079] In one embodiment of this disclosure, classification based on micro-motion features includes:

[0080] Micro-movements are classified based on the first formula;

[0081] The first formula is: ;

[0082] in, The classification output is represented as micro-motion features. Represented as a weight matrix, This is represented as the micro-motion features of the target for identification. Represented as a bias term, It is represented as a probability distribution function.

[0083] In this embodiment, the micro-motion features of the target will be identified. Input into the classification model. Weight matrix. It serves to perform a linear transformation on micro-motion features, adjusting the weight allocation of features across different categories by learning the correlation strength between different micro-motion features and each category. Bias term The transformation result is offset to better fit the data.

[0084] go through After calculation, the result is transformed into a probability distribution using the softmax function. The softmax function normalizes the output value to between 0 and 1, and the sum of the probabilities of all categories is 1. Thus, the obtained... This represents the probability that a micro-movement feature belongs to each category. Therefore, the category to which a micro-movement is most likely to belong can be determined based on the probability, thus achieving the classification of micro-movements.

[0085] As can be seen from the above, this embodiment, by combining micro-motion features with a weight matrix and a bias term, and processing them using a softmax function, can transform micro-motion features into a clear classification probability distribution. This makes the classification process of micro-motions standardized and easy to understand, enabling efficient determination of the category to which a micro-motion belongs.

[0086] In one embodiment of this disclosure, it further includes:

[0087] The micro-motion features are updated by weighting them to obtain the target micro-motion features;

[0088] Based on the target's micro-motion features, the system classifies and identifies the target's emotions.

[0089] In this embodiment, the weights of the micro-motion features are adjusted and updated.

[0090] For example, during training, the weights can be dynamically changed based on factors such as the feature distribution of the samples and the differences from the real labels, highlighting the more critical parts of the micro-motion features for emotion recognition and suppressing some possible noise or irrelevant information, thereby obtaining the target micro-motion features.

[0091] The target micro-motion features, after weight updates, are input into the classification model for emotion recognition. Because the target micro-motion features have undergone weight optimization, they can be more effectively matched and compared with existing emotion category models during the classification process, improving the accuracy and reliability of emotion recognition and making the final emotion judgment of the identified target more consistent with reality.

[0092] In one embodiment of this disclosure, updating the micro-motion features by assigning weights to the micro-motion features includes:

[0093] The micro-motion features are updated based on the weights of the micro-motion features according to the second formula;

[0094] The second formula is:

[0095] in, Represented as the weighted loss function, Let the weight of the i-th micro-motion feature sample be denoted as . It is represented as the cross-entropy loss of the i-th micro-motion feature sample.

[0096] In this embodiment, during the training process of micro-motion recognition, given the locality and fine-grained differences of micro-motions, traditional uniform weight training methods are difficult to effectively focus on key information. By introducing a weighted loss function based on the second formula, the weight of each micro-motion feature sample is dynamically determined according to the joint motion amplitude and time sensitivity. Samples with large amplitude of motion and critical timing are given higher weights. This results in its corresponding cross-entropy loss. In the total loss function The proportion is larger in China.

[0097] During backpropagation, the model can adjust its parameters based on the weighted loss function, thereby updating the weights of micro-motion features. This makes the model more inclined to learn micro-motion features that have a greater impact on the recognition results, gradually optimizing the weight allocation of micro-motion features, thus improving the model's ability to recognize micro-motions. This allows the model to more accurately capture the movement and subtle changes in key areas, while reducing interference from noise and irrelevant information.

[0098] In the process of micro-motion recognition, the range of motion and time sensitivity of joints contain crucial information. Joints with large range of motion often play an important role in micro-motions, serving as the starting point, turning point, or key change point of the movement, and should be given higher weight. In the time dimension, joint movements at certain specific points in time are closely related to the occurrence of key movements and have high time sensitivity; joint data at these moments also require special attention, and their weight should be increased accordingly.

[0099] By employing a dynamic weight calculation method based on the amplitude and time sensitivity of joint motion, the model can be guided to focus on the motion of these key regions during training and recognition. For example, when calculating the loss function, adjusting the weights can make the model pay more attention to the sample data corresponding to the key regions, strengthening the learning of this important information during parameter updates. This improves the model's understanding and recognition accuracy of micro-movements, better captures the detailed features and dynamic changes of micro-movements, and reduces interference caused by irrelevant or secondary information.

[0100] As can be seen from the above, this embodiment dynamically adjusts the weights according to different sample conditions. This allows the model to focus on more critical micro-motion feature samples. Samples that significantly impact the recognition results are given higher weights, thus strengthening the learning of important information during training. This helps improve the model's recognition accuracy, reduces interference from unimportant or noisy samples, and enables the model to more efficiently capture subtle differences in micro-motions, thereby improving the overall performance of the micro-motion recognition system.

[0101] Corresponding to the emotion recognition method based on human micro-movements in the above embodiments, Figure 3 This is a structural block diagram of an emotion recognition system based on human micro-movements, provided as an embodiment of this disclosure. For ease of explanation, only the parts relevant to the embodiment of this disclosure are shown. References Figure 3 The emotion recognition system 20 based on human micro-movements includes: a feature fusion module 21, a micro-movement recognition module 22, and an emotion recognition module 23.

[0102] Among them, the feature fusion module 21 is used to perform feature fusion based on the target video features, target skeleton features and target amplitude features of the target to obtain the comprehensive features of the target;

[0103] The micro-motion recognition module 22 is used to obtain the micro-motion features of the target based on comprehensive features;

[0104] The emotion recognition module 23 is used to classify based on micro-motion features and perform emotion recognition on the target.

[0105] In one embodiment of this disclosure, the feature fusion module 21 is specifically used for:

[0106] Based on a convolutional neural network, feature extraction is performed on the target video to identify the target, and the image features of each frame are obtained.

[0107] The target video features are obtained based on the image features of each frame.

[0108] In one embodiment of this disclosure, the feature fusion module 21 is specifically used for:

[0109] Extract key point data of the target based on the target video;

[0110] Two-dimensional skeleton data of the target to be identified is generated based on key point data;

[0111] The two-dimensional skeleton data is analyzed based on the depth estimation algorithm to obtain the three-dimensional skeleton data of the target to be identified.

[0112] Feature extraction is performed on the 3D skeleton data to obtain the target skeleton features for identification.

[0113] In one embodiment of this disclosure, the feature fusion module 21 is specifically used for:

[0114] Calculate the displacement amplitude of joint data between two adjacent frames in the target video;

[0115] Generate motion amplitude images of the target based on displacement amplitude;

[0116] Feature extraction is performed on the motion amplitude image to obtain the target amplitude features.

[0117] In one embodiment of this disclosure, the emotion recognition module 23 is specifically used for:

[0118] Micro-movements are classified based on the first formula;

[0119] The first formula is: ;

[0120] in, The classification output is represented as micro-motion features. Represented as a weight matrix, This is represented as the micro-motion features of the target for identification. Represented as a bias term, It is represented as a probability distribution function.

[0121] In one embodiment of this disclosure, the emotion recognition system 20 based on human micro-movements further includes: an update module, which is specifically used for:

[0122] The micro-motion features are updated by assigning weights to them to obtain the target micro-motion features;

[0123] Based on the target's micro-motion features, classification is performed to identify the target's emotion.

[0124] In one embodiment of this disclosure, the update module is further configured to:

[0125] The micro-motion features are updated based on the weights of the micro-motion features according to the second formula;

[0126] The second formula is:

[0127] in, Represented as the weighted loss function, Let the weight of the i-th micro-motion feature sample be denoted as . It is represented as the cross-entropy loss of the i-th micro-motion feature sample.

[0128] See Figure 4 , Figure 4 This is a schematic block diagram of an electronic device provided according to an embodiment of the present disclosure. Figure 4 The electronic device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of the modules in the aforementioned system embodiments, for example... Figure 3 The functions of modules 21 to 23 are shown.

[0129] It should be understood that, in the embodiments of this disclosure, the processor 301 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0130] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0131] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store device type information.

[0132] In specific implementations, the processor 301, input device 302, and output device 303 described in this disclosure embodiment can execute the implementation methods described in the first and second embodiments of the emotion recognition method based on human micro-movements provided in this disclosure embodiment, or they can execute the implementation methods of the electronic devices described in this disclosure embodiment, which will not be repeated here.

[0133] In another embodiment of this disclosure, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to implement these processes. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or system capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0134] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD) card, flash card, etc., equipped on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0135] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0136] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0137] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.

[0138] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this disclosure, depending on actual needs.

[0139] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0140] The above are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this disclosure, and these modifications or substitutions should all be covered within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. An emotion recognition method based on human micro-movements, characterized in that, include: Feature fusion is performed based on the target video features, target skeleton features, and target amplitude features to obtain the comprehensive features of the target; The micro-motion features of the target are obtained based on the comprehensive features; The micro-motion features are updated based on the weights of the micro-motion feature samples to obtain the target micro-motion features. A weighted loss function is introduced, and the weight of each micro-motion feature sample is dynamically determined based on the joint movement amplitude and time sensitivity. For samples with larger movement amplitude and time criticality, higher weights are given so that their corresponding cross-entropy loss accounts for a larger proportion of the total loss function. The parameters are adjusted according to the weighted loss function to update the micro-motion features. Based on the target's micro-motion features, classification is performed to identify the target's emotion.

2. The emotion recognition method based on human micro-movements as described in claim 1, characterized in that, Also includes: Based on a convolutional neural network, feature extraction is performed on the target video to identify the target, and the image features of each frame are obtained. The target video features are obtained based on the image features of each frame.

3. The emotion recognition method based on human micro-movements as described in claim 1, characterized in that, Also includes: Extract key point data of the target based on the target video; Two-dimensional skeleton data of the target is generated based on the key point data; The two-dimensional skeleton data is analyzed based on the depth estimation algorithm to obtain the three-dimensional skeleton data of the target to be identified. Feature extraction is performed on the three-dimensional skeleton data to obtain the target skeleton features of the identified target.

4. The emotion recognition method based on human micro-movements as described in claim 3, characterized in that, Also includes: Calculate the displacement amplitude of the joint point data between two adjacent frames in the target video; Generate a motion amplitude image of the target based on the displacement amplitude; Feature extraction is performed on the motion amplitude image to obtain the target amplitude features.

5. The emotion recognition method based on human micro-movements as described in claim 1, characterized in that, The classification based on the micro-motion features includes: The micro-movements are classified based on the first formula; The first formula is: in, The classification output is represented as micro-motion features. Represented as a weight matrix, This is represented as the micro-motion features of the target for identification. Represented as a bias term, It is represented as a probability distribution function.

6. The emotion recognition method based on human micro-movements as described in claim 1, characterized in that, The step of updating the micro-motion features based on the weights of the micro-motion feature samples includes: The micro-motion features are updated based on the second formula; The second formula is: in, Represented as the weighted loss function, Let the weight of the i-th micro-motion feature sample be denoted as . It is represented as the cross-entropy loss of the i-th micro-motion feature sample.

7. An emotion recognition system based on human micro-movements, characterized in that, include: The feature fusion module is used to fuse the target video features, target skeleton features and target amplitude features based on the target to obtain the comprehensive features of the target. The micro-motion recognition module is used to obtain the micro-motion features of the target based on the comprehensive features; The update module is used to update the micro-motion features according to the weights of the micro-motion feature samples to obtain target micro-motion features; a weighted loss function is introduced to dynamically determine the weight of each micro-motion feature sample based on the joint movement amplitude and time sensitivity. For samples with larger movement amplitude and time criticality, higher weights are given so that their corresponding cross-entropy loss accounts for a larger proportion of the total loss function; the module adjusts its own parameters according to the weighted loss function to update the micro-motion features; the emotion recognition module is used to classify based on the target micro-motion features and perform emotion recognition on the target.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images