Emotion recognition method and system based on human body micro-action

Abstract

The invention provides an emotion recognition method and system based on human body micro-motion, and belongs to the technical field of micro-motion recognition, and the method comprises the steps: carrying out the feature fusion based on a target video feature, a target skeleton feature and a target amplitude feature of a recognition target, and obtaining a comprehensive feature of the recognition target; obtaining micro-action features of the recognized target based on the comprehensive features; and performing classification based on the micro-action features, and performing emotion recognition on the recognition target. According to the emotion recognition method and system based on the human body micro-action, the accuracy of micro-action recognition can be improved.

Description

Technical Field

[0001] The present disclosure belongs to the technical field of micro-motion recognition, and more specifically, to an emotion recognition method and system based on human micro-motions. Background Art

[0002] With the rapid development of computer vision technology, human action recognition (HAR) has become an important research direction in this field. Traditional HAR tasks mainly focus on large movements in videos, such as walking and jumping. However, in recent years, micro-action recognition (MAR) has begun to receive more and more attention as a new research direction. Micro-actions refer to those subtle, unconscious movements, usually manifested as short-term changes in body posture, gestures or facial expressions, which can reflect the individual's emotions, psychological state or intentions. The challenge of the MAR task is that micro-actions usually have a small amplitude, a fast execution speed, and often appear intermittently between different body parts, so it is difficult for traditional HAR models to accurately capture them. Summary of the invention

[0003] The purpose of the present invention is to provide an emotion recognition method and system based on human micro-motions to improve the accuracy of micro-motion recognition.

[0004] A first aspect of the embodiments of the present disclosure provides an emotion recognition method based on human micro-motions, comprising: Based on the target video features, target skeleton features and target amplitude features of the identified target, feature fusion is performed to obtain the comprehensive features of the identified target; Obtaining micro-motion features of the identified target based on the comprehensive features; Classification is performed based on the micro-motion features, and emotion recognition is performed on the recognition target.

[0005] A second aspect of the disclosed embodiment provides an emotion recognition system based on human micro-motions, comprising: A feature fusion module is used to fuse the target video features, target skeleton features and target amplitude features of the target to obtain the comprehensive features of the target; A micro-motion recognition module, used for obtaining micro-motion features of an identified target based on the comprehensive features; The emotion recognition module is used to classify the micro-motion features and perform emotion recognition on the recognition target.

[0006] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the above-mentioned method for emotion recognition based on human micro-movements when executing the computer program.

[0007] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned method for emotion recognition based on human micro-movements are implemented.

[0008] The beneficial effect of the emotion recognition method and system based on human micro-movements provided by the embodiments of the present disclosure is that the embodiments of the present disclosure achieve accurate capture of the comprehensive features of the recognition target by fusing the target video features, the target skeleton features and the target amplitude features. First, by integrating the target video features, the target skeleton features and the target amplitude features for fusion, the complementarity of different modal information is fully utilized, and the movement details of the recognition target can be captured comprehensively and accurately, avoiding the limitations of a single feature, and greatly improving the accuracy and reliability of micro-movement recognition. Secondly, further extracting micro-movement features based on comprehensive features helps to focus on the key features of tiny movements, highlight the differences between subtle movements, and provide a more discriminative basis for subsequent classification. Finally, efficient emotion recognition is performed based on micro-movement features to improve the accuracy and reliability of recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0010] Figure 1 A schematic diagram of a flow chart of an emotion recognition method based on human micro-movements provided in an embodiment of the present disclosure; Figure 2 A schematic diagram of a joint point motion amplitude image provided by an embodiment of the present disclosure; Figure 3 A structural block diagram of an emotion recognition system based on human micro-motions provided in an embodiment of the present disclosure; Figure 4 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0011] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0012] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0013] Please refer to Figure 1 , Figure 1 A schematic flow chart of a method for emotion recognition based on human micro-motions provided in an embodiment of the present disclosure, the method comprising: S101: performing feature fusion based on target video features, target skeleton features and target amplitude features of the identified target to obtain comprehensive features of the identified target.

[0014] In this embodiment, the target video features may include visual information in the video, such as dynamic changes in motion, color, texture, etc., expressed in a time series, which are extracted from the original video data through video processing technology or a pre-trained model.

[0015] The target skeleton feature refers to the position and connection information of the human body joints. The skeleton feature can reflect the posture and motion trajectory of the human body, and is helpful to capture the basic framework of the human body posture and movement. In this embodiment, the key point detection algorithm can be used to extract it from the video.

[0016] The target amplitude feature mainly reflects the amplitude of the action, which refers to the displacement amplitude of the joints between different frames. The target amplitude feature reflects the movement intensity and timing information of the micro-action. By calculating the displacement amplitude of each joint between adjacent frames and encoding it into an image form, rich amplitude features can be obtained. The target amplitude feature helps to capture the subtle changes and rapid occurrence characteristics of micro-actions.

[0017] In this embodiment, the target video features, target skeleton features and target amplitude features are fused to obtain comprehensive features for identifying the target.

[0018] This embodiment can automatically learn the association and importance weights between different features by concatenating feature vectors, weighted summing, or using the attention mechanism of a deep learning model (such as Transformer, etc.), thereby obtaining a high-dimensional feature representation that integrates video, skeleton, and amplitude information.

[0019] For example, the comprehensive features of the identified target can be obtained by fusion through the Transformer model.

[0020] Suppose we have a video in which a person is making micro-movements of his hands. After feature extraction, we get the target video features: , target skeleton features and target amplitude characteristics Then , and according to The multimodal vectors are spliced ​​into a multimodal vector and input into the Transformer model. The self-attention mechanism of Transformer can automatically focus on the relationship between different modal features. For example, when identifying nervous micro-movements of the hands, it can be found that there is a close connection between the tense state of the hand muscles in the video features, the special angle changes of the finger joints in the skeleton features, and the fast and small changes in the movement amplitude in the movement amplitude features. After multiple layers of processing, the comprehensive features that can accurately characterize the hand micro-movements are finally obtained, which is used for subsequent tasks such as micro-movement recognition and emotion analysis.

[0021] S102: Obtain micro-motion features of the identified target based on the comprehensive features.

[0022] In this embodiment, multi-layer processing can be performed based on the Transformer model to obtain micro-motion features of the identified target.

[0023] For example, each layer of Transformer operation includes a multi-head self-attention mechanism, residual connection, layer normalization, and feedforward neural network. Through the combination of N layers of Transformer operations containing the above operations, the input features are continuously refined and abstracted, and the micro-motion features of the identified target are gradually extracted. Each layer further explores the deep information of features based on the previous layer, and the final It can accurately reflect the micro-movement characteristics of the target and provide strong support for subsequent classification and emotion recognition tasks. It is expressed as:

[0024] in, Represents a combination of N layers of Transformer operations, Represents a multimodal vector.

[0025] S103: Classify based on micro-motion features and perform emotion recognition on the recognition target.

[0026] In this embodiment, classification is performed based on the extracted micro-motion features to achieve emotion recognition of the recognition target. During the classification process, an emotion classification model can be pre-established. The emotion classification model is trained based on a large amount of labeled data to learn the mapping relationship between different micro-motion features and emotion categories. In practical applications, the extracted micro-motion features are input into the trained model. The model compares the input features with the learned pattern and outputs the most likely corresponding emotion category, thereby completing the judgment of the emotional state of the recognition target.

[0027] It can be concluded from the above that this embodiment achieves accurate capture of the comprehensive features of the identified target by fusing the target video features, target skeleton features and target amplitude features. First, by integrating the target video features, target skeleton features and target amplitude features for fusion, the complementarity of different modal information is fully utilized, and the action details of the identified target can be captured comprehensively and accurately, avoiding the limitations of a single feature, and greatly improving the accuracy and reliability of micro-motion recognition. Secondly, further extracting micro-motion features based on comprehensive features helps to focus on the key features of tiny movements, highlighting the differences between subtle movements, and providing a more discriminative basis for subsequent classification. Finally, efficient emotion recognition is performed based on micro-motion features to improve recognition accuracy and reliability.

[0028] In one embodiment of the present disclosure, it also includes: Based on the convolutional neural network, the target video of the identified target is extracted to obtain the image features of each frame of the image; The target video features are obtained based on the image features of each frame.

[0029] In this embodiment, the target video of the target to be identified is processed with each frame as a processing unit and input into the pre-trained convolutional neural network. middle.

[0030] Pre-trained convolutional neural networks have been trained on a large amount of image data in advance and have powerful image feature extraction capabilities. For example, they can identify basic visual elements such as edges, textures, and shapes in images, and gradually abstract and combine these elements through structures such as convolutional layers and pooling layers to form more advanced image feature representations.

[0031] The image frame at time t in the processed video hour, According to its internal convolution kernel parameters and network structure, convolution operations can be performed on different areas of the image to extract local features, and then dimensionality reduction processing is performed through the pooling layer to reduce the amount of calculation and retain key information, and finally output high-dimensional image features. .

[0032] The image features of each frame are expressed as:

[0033] In this embodiment, the above operation is performed on each frame of the video to obtain an image feature sequence over time. These image feature sets constitute the target video features, which not only contain the static visual information of each frame of the image, but also contain the change information of the action in the video in the time dimension because they are extracted in chronological order, providing important basic data for subsequent tasks such as micro-motion recognition and emotion analysis.

[0034] From the above, it can be concluded that this embodiment can accurately capture the detailed information in each frame of the image, such as the shape and texture of the object, by extracting features through a convolutional neural network, providing rich materials for subsequent analysis. Secondly, by integrating the features of each frame of the image to form the target video features, the key information of the single frame is retained, and the dynamic changes of the video are incorporated, making the obtained video features more comprehensive and representative, which effectively improves the accuracy and reliability of subsequent micro-motion recognition and related analysis.

[0035] In one embodiment of the present disclosure, it also includes: Extract and identify the joint point data of the target based on the target video; Generate two-dimensional skeleton data of the recognition target based on the joint point data; Analyze the two-dimensional skeleton data based on the depth estimation algorithm to obtain the three-dimensional skeleton data of the identified target; Feature extraction is performed on the three-dimensional skeleton data to obtain the target skeleton features of the identified target.

[0036] In this embodiment, Media Pipe can be used to extract the joint point data of the human body in the target video and generate 2D skeleton data. This process identifies the position of the human body joints in the video frame and determines the 2D coordinates of each joint point, thereby constructing the basic structural framework of the human body. It is expressed as:

[0037] in, It is The two-dimensional coordinates of the joint points in the image frame, is the total number of joint points.

[0038] Next, the 2D skeleton data is processed by the depth estimation algorithm of the convolutional neural network to predict the depth information of each joint point, and then generate 3D skeleton data, which is expressed as:

[0039] in, It is The three-dimensional coordinates of the joint points, It is depth information.

[0040] In this embodiment, the target video may be a conventional monocular RGB video. The RGB video is combined with the depth information to obtain RGB+D skeleton data (ie, three-dimensional skeleton data), which is expressed as:

[0041] in, For RGB video data, is the estimated depth information. The representation provides richer spatial and depth information, which helps to capture the details of micro-motions.

[0042] In this embodiment, the skeleton features of each frame can be processed using a deep neural network (such as a Transformer or MLP network) based on the RGB+D skeleton data extracted by the Media Pipe. The deep neural network can learn the spatial structural features and time series change rules in the three-dimensional skeleton data, thereby extracting the target skeleton features for identifying the target. The target skeleton features can reflect the posture changes and joint movement patterns of the human body during micro-movements, providing key basic data for micro-movement recognition and emotion analysis.

[0043] For example, based on the RGB+D skeleton data extracted by Media Pipe, a deep neural network (such as Transformer or a simple MLP network) is used to process the RGB+D skeleton data of each frame to obtain the target skeleton features:

[0044] in, is the 3D skeleton feature of the corresponding frame at time t, It is the extracted spatial structural feature. From the above, it can be concluded that this embodiment can accurately construct a body structure model of the identified target and clearly display the basic framework of the action by extracting joint point data and generating two-dimensional and three-dimensional skeleton data. Combined with the depth estimation algorithm, the spatial depth information is added to make the understanding of micro-movements more three-dimensional. Further extraction of target skeleton features can focus on the key joint changes of the action, effectively capture the posture changes of subtle movements, and provide key and accurate information support for subsequent micro-movement recognition and emotion analysis, thereby improving the accuracy and reliability of overall recognition.

[0045] In one embodiment of the present disclosure, it also includes: Calculate the displacement amplitude of the joint point data between two adjacent frames in the target video; Generate a motion amplitude image of the identified target based on the displacement amplitude; Feature extraction is performed on the motion amplitude image to obtain target amplitude features.

[0046] In this embodiment, for the target video, the movement change of the joint point is measured by calculating the displacement amplitude of the joint point data between two adjacent frames. The displacement of each joint point at different times is calculated to reflect the dynamic characteristics of the action. Then, the displacement amplitude information is encoded into an image form, and the movement amplitude of each joint point is integrated into the corresponding image to generate a motion amplitude image. The image integrates the motion intensity information of all relevant nodes over time and becomes a key data source for micro-motion recognition.

[0047] Subsequently, a convolutional neural network or Transformer model can be used to extract features from the motion amplitude image. This can help to discover hidden patterns and rules in the image and transform the information in the motion amplitude image into more representative target amplitude features. These features can capture key information about the amplitude changes of micro-motions. When combined with other features (such as video features and skeleton features), they can comprehensively improve the ability to recognize micro-motions and provide strong support for subsequent analysis and applications.

[0048] like Figure 2 As shown, for example, in order to capture the spatiotemporal dynamic information of micro-movements, the method of representing the motion amplitude image of 33 key joints of the human body (using Media PipePose to detect the following key points: nose, inner corner of left eye, left eye, outer corner of left eye, inner corner of right eye, right eye, outer corner of right eye, left ear, right ear, above the center of lips, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, base of left palm, base of right palm, left thumb joint, right thumb joint, left index finger joint, right index finger joint, center of upper body (hip), left hip, right hip, left knee, right knee, left ankle, right ankle, left heel, right heel, left toe, right toe) can be used. For the tth frame in the video, the displacement amplitude of each joint point between two adjacent frames is calculated. :

[0049] in, represents the horizontal coordinate of the i-th key point in the t-th frame, Represents the horizontal coordinate of the i-th key point in the (t-1) frame; represents the ordinate of the i-th key point in the t-th frame, Represents the ordinate of the i-th key point in the (t-1) frame; represents the depth coordinate of the i-th key point in the t-th frame, Represents the depth coordinate of the i-th key point in the (t-1)th frame.

[0050] Then, the motion amplitude of each joint point is encoded into an image form, represented as:

[0051] in, For the The joints are The motion amplitude of all time steps is combined to generate a two-dimensional image. As motion amplitude images of the video, these amplitude images provide temporal information of motion intensity, which is an important feature for micro-motion recognition.

[0052] Image of joint motion amplitude for each frame , use convolutional neural network (CNN) or Transformer model to extract features and obtain the target amplitude features of each frame: , in, is the motion amplitude image of each joint point in each frame, is the target amplitude feature extracted for each frame.

[0053] From the above, it can be concluded that this embodiment can intuitively present the dynamic changes of micro-movements in the time dimension by calculating the displacement amplitude of joint points and generating motion amplitude images, providing a clear basis for amplitude changes for subsequent analysis. Further extraction of target amplitude features can accurately capture the intensity and rhythm information of the movement. These features combined with other modal features can significantly enhance the overall perception of micro-movements, improve the accuracy and comprehensiveness of recognition, and provide more reliable data support and analysis basis for micro-movement recognition related applications.

[0054] In one embodiment of the present disclosure, classification based on micro-motion features includes: Classifying micro-movements based on the first formula; The first formula is: ; in, Represented as the classification output of micro-motion features, Expressed as a weight matrix, Represented as the micro-motion feature of the identified target, is represented as a bias term, It is expressed as a probability distribution function.

[0055] In this embodiment, the micro-motion features of the target are identified Input to the classification model. Weight matrix It plays the role of linear transformation of micro-motion features, and adjusts the weight distribution of features in different categories by learning the correlation strength between different micro-motion features and each category. The transformation result is offset to better fit the data.

[0056] go through After calculating , the result is converted into probability distribution using the softmax function. The softmax function can normalize the output value to between 0 and 1, and the sum of the probabilities of all categories is 1. In this way, the obtained It represents the probability that the micro-motion feature belongs to each category, so that the category to which the micro-motion is most likely to belong can be determined according to the probability, thereby realizing the classification of the micro-motion.

[0057] It can be concluded from the above that this embodiment can convert micro-motion features into clear classification probability distribution by combining micro-motion features with weight matrix and bias term and processing them by softmax function. This makes the classification process of micro-motion standardized and easy to understand, and can efficiently determine the category to which micro-motion belongs.

[0058] In one embodiment of the present disclosure, it also includes: The weight of the micro-motion feature is updated to obtain the target micro-motion feature; Classification is performed based on the target's micro-motion features, and emotion recognition is performed on the identified target.

[0059] In this embodiment, the weights of the micro-motion features are adjusted and updated.

[0060] For example, during the training process, the weights are dynamically changed according to factors such as the feature distribution of the samples and the difference from the true labels, highlighting the more critical parts of the micro-motion features for emotion recognition and suppressing some possible noise or less relevant information, thereby obtaining the target micro-motion features.

[0061] The weighted updated target micro-motion features are input into the classification model for emotion recognition. Since the target micro-motion features have been weighted, they can be more effectively matched and compared with the existing emotion category model during the classification process, improving the accuracy and reliability of emotion recognition, making the final emotion judgment of the target more in line with the actual situation.

[0062] In one embodiment of the present disclosure, updating the weight of the micro-motion feature includes: Updating the micro-motion feature based on the weight of the micro-motion feature according to the second formula; The second formula is:

[0063] in, Expressed as a weighted loss function, It is represented as the weight of the i-th micro-motion feature sample, It is expressed as the cross entropy loss of the i-th micro-motion feature sample.

[0064] In this embodiment, during the training process of micro-motion recognition, due to the local and fine-grained differences of micro-motions, the traditional unified weight training method is difficult to effectively focus on key information. By introducing the weighted loss function of the second formula, the weight of each micro-motion feature sample is dynamically determined according to the joint point movement amplitude and time sensitivity. For samples with large motion amplitude and critical time, higher weight is given , so that its corresponding cross entropy loss In the total loss function A larger proportion.

[0065] During the back propagation process, the model can adjust its own parameters according to the weighted loss function, and then update the weights of the micro-motion features. This makes the model more inclined to learn those micro-motion features that have a greater impact on the recognition results during the learning process, and gradually optimizes the weight distribution of micro-motion features, thereby improving the model's ability to recognize micro-motions, enabling it to more accurately capture the movement of key areas and subtle changes in movement, and reduce the interference of noise and irrelevant information.

[0066] In the process of micro-motion recognition, the movement amplitude and time sensitivity of joints contain key information. For joints with large movement amplitudes, they often play an important role in micro-motions. They can be the starting point, turning point or key change point of the action, and should be given a higher weight. In the time dimension, the joint movement at certain specific time points is closely related to the occurrence of key actions and has a high time sensitivity. The joint data at these moments also need to be paid attention to, and their weights should be increased accordingly.

[0067] Through the dynamic weight calculation method based on the movement amplitude and time sensitivity of the joint points, the model can be guided to focus on the movement of these key areas during model training and recognition. For example, when calculating the loss function, the adjustment of the weight can make the model pay more attention to the sample data corresponding to the key areas, and strengthen the learning of these important information during the parameter update process, thereby improving the model's understanding and recognition accuracy of micro-movements, better capturing the detailed features and dynamic changes of micro-movements, and reducing interference caused by irrelevant or secondary information.

[0068] It can be concluded from the above that this embodiment dynamically adjusts the weight according to different situations of the samples , so that the model can focus on more critical micro-motion feature samples. For those samples that have a greater impact on the recognition results, higher weights are given, thereby strengthening the learning of important information during the training process. This helps to improve the recognition accuracy of the model, reduce the interference caused by unimportant or noisy samples, enable the model to more efficiently capture the subtle differences in micro-motions, and improve the performance of the overall micro-motion recognition system.

[0069] Corresponding to the emotion recognition method based on human micro-movements in the above embodiment, Figure 3 This is a structural block diagram of an emotion recognition system based on human micro-motions provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 3 The emotion recognition system 20 based on human micro-motions includes: a feature fusion module 21, a micro-motion recognition module 22, and an emotion recognition module 23.

[0070] The feature fusion module 21 is used to perform feature fusion based on the target video feature, target skeleton feature and target amplitude feature of the identified target to obtain the comprehensive feature of the identified target; A micro-motion recognition module 22, used to obtain micro-motion features of the recognition target based on the comprehensive features; The emotion recognition module 23 is used to classify based on micro-motion features and perform emotion recognition on the recognition target.

[0071] In one embodiment of the present disclosure, the feature fusion module 21 is specifically used for: Based on the convolutional neural network, the target video of the identified target is extracted to obtain the image features of each frame of the image; The target video features are obtained based on the image features of each frame.

[0072] In one embodiment of the present disclosure, the feature fusion module 21 is specifically used for: Extract and identify the joint point data of the target based on the target video; Generate two-dimensional skeleton data of the recognition target based on the joint point data; Analyze the two-dimensional skeleton data based on the depth estimation algorithm to obtain the three-dimensional skeleton data of the identified target; Feature extraction is performed on the three-dimensional skeleton data to obtain the target skeleton features of the identified target.

[0073] In one embodiment of the present disclosure, the feature fusion module 21 is specifically used for: Calculate the displacement amplitude of the joint point data between two adjacent frames in the target video; Generate a motion amplitude image of the identified target based on the displacement amplitude; Feature extraction is performed on the motion amplitude image to obtain target amplitude features.

[0074] In one embodiment of the present disclosure, the emotion recognition module 23 is specifically used to: Classifying micro-movements based on the first formula; The first formula is: ; in, Represented as the classification output of micro-motion features, Expressed as a weight matrix, Represented as the micro-motion feature of the identified target, is represented as a bias term, It is expressed as a probability distribution function.

[0075] In one embodiment of the present disclosure, the emotion recognition system 20 based on human micro-motions further includes: an updating module, which is specifically used to: updating the micro-motion feature according to the weight of the micro-motion feature to obtain a target micro-motion feature; Classification is performed based on the target micro-motion features, and emotion recognition is performed on the identified target.

[0076] In one embodiment of the present disclosure, the update module is further configured to: Updating the micro-motion feature based on the weight of the micro-motion feature according to a second formula; The second formula is:

[0077] in, Expressed as a weighted loss function, It is represented as the weight of the i-th micro-motion feature sample, It is expressed as the cross entropy loss of the i-th micro-motion feature sample.

[0078] See also Figure 4 , Figure 4 A schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 4 The electronic device 300 in the embodiment shown may include: one or more processors 301, one or more input devices 302, one or more output devices 303 and one or more memories 304. The processors 301, input devices 302, output devices 303 and memories 304 communicate with each other via a communication bus 305. The memory 304 is used to store computer programs, which include program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. The processor 301 is configured to call the program instructions to execute the functions of the modules in the above-mentioned system embodiments, such as Figure 3 The functions of modules 21 to 23 are shown.

[0079] It should be understood that in the embodiment of the present disclosure, the processor 301 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0080] The input device 302 may include a touch panel, a fingerprint collection sensor (for collecting the user's fingerprint information and fingerprint direction information), a microphone, etc., and the output device 303 may include a display (LCD, etc.), a speaker, etc.

[0081] The memory 304 may include a read-only memory and a random access memory, and provide instructions and data to the processor 301. A portion of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store information about the device type.

[0082] In a specific implementation, the processor 301, input device 302, and output device 303 described in the embodiments of the present disclosure can execute the implementation methods described in the first and second embodiments of the emotion recognition method based on human micro-movements provided in the embodiments of the present disclosure, and can also execute the implementation methods of the electronic device described in the embodiments of the present disclosure, which will not be repeated here.

[0083] In another embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by the processor, all or part of the processes in the above-mentioned embodiment method are implemented, and the computer program can also be completed by instructing the relevant hardware through the computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or system capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0084] The computer-readable storage medium may be an internal storage unit of the electronic device of any of the aforementioned embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the computer-readable storage medium may also include both an internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0085] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0086] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0087] In the several embodiments provided in the present application, it should be understood that the disclosed electronic device and method can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces or units, or it can be an electrical, mechanical or other form of connection.

[0088] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present disclosure.

[0089] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0090] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for emotion recognition based on human micro-movements, characterized in that: include: Based on the target video features, target skeleton features and target amplitude features of the identified target, feature fusion is performed to obtain the comprehensive features of the identified target; Obtaining micro-motion features of the identified target based on the comprehensive features; Classification is performed based on the micro-motion features, and emotion recognition is performed on the recognition target.

2. The emotion recognition method based on human micro-motions as claimed in claim 1, characterized in that: Also includes: Based on the convolutional neural network, the target video of the identified target is extracted to obtain the image features of each frame of the image; The target video feature is obtained based on the image feature of each frame of image.

3. The emotion recognition method based on human micro-motions as claimed in claim 1, characterized in that: Also includes: Extract and identify the joint point data of the target based on the target video; Generate two-dimensional skeleton data of the recognition target based on the joint point data; Analyzing the two-dimensional skeleton data based on a depth estimation algorithm to obtain three-dimensional skeleton data of the identified target; Feature extraction is performed on the three-dimensional skeleton data to obtain target skeleton features of the identified target.

4. The emotion recognition method based on human micro-motions as claimed in claim 3, characterized in that: Also includes: Calculate the displacement amplitude of the joint point data between two adjacent frames in the target video; generating a motion amplitude image of the recognition target based on the displacement amplitude; Feature extraction is performed on the motion amplitude image to obtain target amplitude features.

5. The emotion recognition method based on human micro-motions as claimed in claim 1, characterized in that: The classifying based on the micro-motion features includes: classifying the micro-motions based on a first formula; The first formula is: in, Represented as the classification output of micro-motion features, Expressed as a weight matrix, Represented as the micro-motion feature of the identified target, is represented as a bias term, It is expressed as a probability distribution function.

6. The emotion recognition method based on human micro-motions as claimed in claim 5, characterized in that: Also includes: updating the micro-motion feature according to the weight of the micro-motion feature to obtain a target micro-motion feature; Classification is performed based on the target micro-motion features, and emotion recognition is performed on the identified target.

7. The emotion recognition method based on human micro-motions as claimed in claim 6, characterized in that: The step of updating the micro-motion feature by weighting the micro-motion feature comprises: Updating the micro-motion feature based on the weight of the micro-motion feature according to a second formula; The second formula is: in, Expressed as a weighted loss function, It is represented as the weight of the i-th micro-motion feature sample, It is expressed as the cross entropy loss of the i-th micro-motion feature sample.

8. An emotion recognition system based on human micro-movements, characterized in that: include: A feature fusion module is used to fuse the target video features, target skeleton features and target amplitude features of the target to obtain the comprehensive features of the target; A micro-motion recognition module, used for obtaining micro-motion features of an identified target based on the comprehensive features; The emotion recognition module is used to classify the micro-motion features and perform emotion recognition on the recognition target.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images