Emotion calculation method and device for child expressions

Abstract

The invention relates to the technical field of child emotion recognition, in particular to an emotion calculation method and device for child expressions, and the method comprises the steps: collecting and marking the multi-modal dynamic expression data of a plurality of Asian children, and constructing an enhanced mixed expression database according to the multi-modal dynamic expression data of the Asian children; performing iterative training on a pre-constructed deep convolutional neural network model by using the enhanced mixed expression database until a preset iterative period is reached, so as to obtain a deep time sequence emotion calculation model; and inputting the multi-modal dynamic expression of the child to be recognized into the depth time sequence emotion calculation model to calculate the current emotion of the child. Therefore, the problems that an existing child emotion recognition model is mostly based on European and American adult database training and mostly adopts single-frame static image analysis, and the complete dynamic process of expressions cannot be captured, so that the accuracy is remarkably reduced, the model robustness is poor, and effective application in real and continuous interaction scenes is difficult are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of child emotion recognition, and particularly relates to a child expression emotion computing method and device. BACKGROUND

[0002] Affective Computing aims to endow computing systems with the ability to understand and respond to human emotions, and its research spans multiple fields such as psychology, computer science, artificial intelligence, and neuroscience. The goal of this field is to build intelligent systems that can recognize, infer, and even mimic human emotions, to achieve more efficient and more empathetic human-computer interaction.

[0003] In affective computing, emotion recognition is a core research direction, which aims to enable machines to perceive human emotions from facial expressions, speech, text, and other modalities. Although there are various ways to express emotions, facial expression-based recognition has become mainstream due to its intuitiveness. Currently, most technical models (including commercial APIs such as Tencent Cloud) are mainly trained based on adult static image databases, and usually use a classification framework of "six basic expressions (joy, anger, sadness, surprise, fear, disgust) + neutral". This method relies heavily on "typical snapshots" (such as a photo of a person laughing) of peak moments of expressions, and classifies them as the closest basic emotion.

[0004] However, this mainstream paradigm has significant limitations. First, it ignores that expressions are a dynamic process and cannot capture complete information from the beginning, development, to the end. Second, since the training data is mainly adult, the model's generalization ability is severely insufficient when identifying child expressions that differ in facial features and expression methods. Finally, it performs much worse on complex or mixed emotions (such as surprise and joy) that are common in reality than on exaggerated, single emotions. These defects collectively result in the current technology's effectiveness and reliability being greatly compromised when applied to real, natural interaction scenarios, especially those involving children.

[0005] Despite the significant progress made in affective computing technology, its application in the field of child emotion recognition still faces severe challenges, which are mainly rooted in its core data foundation and technical methods.

[0006] Firstly, existing models are generally plagued by double biases in training data. On the one hand, they are mainly trained based on adult (especially European and American) expression databases, while the facial structure and expression patterns of children are quite different from adults, resulting in a serious lack of generalization ability in cross-age, cross-ethnic recognition, especially for Asian children. On the other hand, there is significant class imbalance within these datasets, i.e. common emotions such as "happy" and "sad" are over-represented, while rare emotions such as "disgust" and "surprise" are severely under-represented, which directly undermines the accuracy of the model in recognizing atypical emotions.

[0007] Secondly, the reliance on static single-frame images is one of its fundamental flaws. Facial expressions are a complete dynamic process, while current mainstream methods only analyze the instantaneous snapshot of the "peak" of the expression, completely ignoring the dynamic information from the generation to the change of the expression. This static perspective not only easily loses context, leading to misjudgment, but also makes it difficult to effectively deploy in real-world scenarios that require continuous and real-time responses.

[0008] Finally, the feedback mechanism of current systems is too simplified. They usually only provide a single, discrete emotion label (such as "happy"), and cannot handle complex, mixed, or high-similarity expressions in the real world. This rough feedback granularity greatly reduces its practical value in educational or therapeutic scenarios that require fine-grained emotional interpretation. These problems collectively limit the actual application effect of existing technology in the field of children's emotional development support. SUMMARY

[0009] The present application provides a kind of children's expression emotional computing method and device, to solve the emotional computing model of existing emotion multiple based on European and American adult database training, it is accurate when identifying the facial expression of different ethnic (especially Asian) children, and data bias significantly reduces, and it is seriously dependent on unbalanced data set (i.e. the sample of common emotion is much more than rare emotion), and generally adopt single-frame static image analysis, cannot capture the complete dynamic process of expression, leading to poor robustness of model, it is difficult to effectively apply in real, continuous interactive scenarios etc. Problem.

[0010] The first aspect embodiment of the present application provides a kind of children's expression emotional computing method, comprising the following steps: Collect and label a plurality of Asian children's multi-modal dynamic expression data, and construct an enhanced hybrid expression database according to the plurality of Asian children's multi-modal dynamic expression data;Iterative training is carried out on the pre-constructed deep convolutional neural network model using the enhanced hybrid expression database until the preset iteration period is reached to obtain a deep time series emotional computing model;The multi-modal dynamic expression of the child to be identified is input into the deep time series emotional computing model to calculate the current emotion of the child.

[0011] Optionally, the plurality of Asian children multi-modal dynamic expression data comprises a plurality of Asian children expression grayscale images and a plurality of Asian children dynamic expression videos.

[0012] Optionally, the collecting the plurality of Asian children multi-modal dynamic expression data and constructing the enhanced hybrid expression database according to the plurality of Asian children multi-modal dynamic expression data comprises: collecting a plurality of Asian children expression grayscale images representing seven emotions in a preset children emotional facial expression dataset and a preset facial expression recognition dataset; performing interval frame extraction on each Asian children dynamic expression video to form an Asian children expression image sequence; performing data enhancement processing on the plurality of Asian children expression grayscale images and the Asian children expression image sequence to construct the enhanced hybrid expression database.

[0013] Optionally, the seven emotions comprise anger, disgust, fear, happiness, sadness, surprise, and neutrality.

[0014] Optionally, the training the pre-constructed deep convolutional neural network model using the enhanced hybrid expression database until a preset iteration period is reached to obtain a deep temporal emotion computing model comprises: inputting each frame of image in the enhanced hybrid expression database to a 2D convolutional layer in the deep convolutional neural network model to extract spatial features of each frame of image; inputting the spatial features of each frame of image to a 3D convolutional layer or an LSTM layer in the deep convolutional neural network model to learn variation rules of the spatial features of each frame of image; inputting the variation rules of the spatial features of each frame of image to a fully connected layer in the deep convolutional neural network model to output a 7-dimensional similarity ratio vector; calculating a cross-entropy loss between the 7-dimensional similarity ratio vector and a true label of each frame of image corresponding to the 7-dimensional similarity ratio vector; based on an Adam optimizer and a back propagation algorithm, adjusting weights of the deep convolutional neural network model according to the cross-entropy loss, iteratively performing the foregoing process until a preset iteration period is reached to obtain the deep temporal emotion computing model.

[0015] Optionally, the inputting the to-be-recognized children multi-modal dynamic expression into the deep temporal emotion computing model to calculate a current emotion of the children comprises: inputting the to-be-recognized children multi-modal dynamic expression into the deep temporal emotion computing model to perform emotion calculation to obtain a 7-dimensional similarity ratio vector; determining the current emotion of the children according to the 7-dimensional similarity ratio vector.

[0016] The second aspect of the present application provides a child expression emotion calculation device, comprising: a construction module, configured to collect and label a plurality of Asian child multi-modal dynamic expression data, and construct an enhanced hybrid expression database according to the plurality of Asian child multi-modal dynamic expression data; an iterative training module, configured to iteratively train a pre-constructed deep convolutional neural network model using the enhanced hybrid expression database until a preset iteration period is reached to obtain a deep time-series emotion calculation model; and a recognition module, configured to input a to-be-recognized child multi-modal dynamic expression into the deep time-series emotion calculation model to calculate a current emotion of the child.

[0017] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the child expression emotion calculation method as described in the above embodiments.

[0018] The fourth aspect of the present application provides a computer program product, wherein the computer program / instruction is executed by a processor to implement the child expression emotion calculation method as described above.

[0019] The fifth aspect of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the program is executed by a processor to implement the child expression emotion calculation method as described above.

[0020] The child expression emotion calculation method and device provided by the embodiments of the present application effectively overcome the recognition difficulty caused by data bias by introducing targeted Asian child dynamic data and performing data enhancement, greatly improve the recognition accuracy of the target group, and improve the recognition performance of rare expressions; can analyze the complete dynamic process of the expression rather than the static moment, which is more in line with the natural law of human emotional expression, so that the recognition result is more robust and can adapt to complex changes in real application scenarios; the similarity ratio of seven emotions is output, which provides more rich and refined emotional state information than single label for downstream applications (such as special education and psychological intervention), so that it has higher practical value and interpretability.

[0021] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, taken in conjunction with the following drawings, in which: Figure 1A flowchart illustrating a method for calculating the emotion of a child's facial expressions provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating children expressing facial expressions during play, provided as an embodiment of the present invention. Figure 3 This is a schematic diagram of some facial expression data collected from children in a gamified context, as provided in an embodiment of the present invention. Figure 4 A schematic diagram illustrating the average recognition accuracy of the six basic facial expressions provided in this embodiment of the invention; Figure 5 This is an architecture diagram of a gamified online facial expression collection and interaction system for children provided in an embodiment of the present invention; Figure 6 A block diagram illustrating an emotion calculation device for children's facial expressions provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0023] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0024] The following describes a method and apparatus for calculating children's facial expressions according to embodiments of the present invention with reference to the accompanying drawings.

[0025] Figure 1 This is a flowchart illustrating a method for calculating the emotion of a child's facial expressions provided in an embodiment of the present invention.

[0026] like Figure 1 As shown, the method for calculating the emotion of a child's facial expression includes the following steps: In step S101, multiple Asian children's multimodal dynamic facial expression data are collected and labeled, and an enhanced hybrid facial expression database is constructed based on the multiple Asian children's multimodal dynamic facial expression data.

[0027] In some embodiments, multimodal dynamic facial expression data of multiple Asian children are collected, and an enhanced hybrid facial expression database is constructed based on the multimodal dynamic facial expression data of multiple Asian children, including: Multiple grayscale images of Asian children's facial expressions representing seven emotions were collected from a preset children's emotional facial expression dataset and a preset facial expression recognition dataset. Each video segment featuring the dynamic facial expressions of Asian children was periodically extracted to create a sequence of Asian children's facial expression images; Data augmentation is performed on the plurality of Asian child expression grayscale images and the plurality of Asian child expression image sequences to construct an enhanced mixed expression database.

[0028] The seven emotions include anger, disgust, fear, happiness, sadness, surprise, and neutral.

[0029] In actual implementation, the embodiment of the present application pre-integrates two public authoritative databases: a child affective facial expression (CAFE) dataset containing 1200 high-quality photos of more than 100 children aged 2-8 years old; and a facial expression recognition (FER2013) dataset containing more than 35,000 48x48 pixel grayscale images representing seven core emotions (anger, disgust, fear, happiness, sadness, surprise, and neutral); and collects a plurality of images from the 1200 high-quality photos of more than 100 children aged 2-8 years old and the more than 35,000 48x48 pixel grayscale images representing seven core emotions (anger, disgust, fear, happiness, sadness, surprise, and neutral) as the plurality of Asian child expression grayscale images. Further, to solve the problems of lack of Asian child samples and class imbalance in existing databases, the embodiment of the present application performs independent data collection, and abandons the traditional method of taking single, static photos in a laboratory environment, and instead develops a game-based online interactive system in advance. Through the game-based interactive system as a data collection means, more than 100 Chinese children aged 6-12 years old are guided and captured to generate dynamic expression videos in a near-real natural environment in an uncontrolled and near-real natural environment. This method overcomes the problem of low ecological validity of “performance” expression data in a traditional laboratory environment, and can capture more spontaneous and real emotional expression process expression samples, directly solving the key pain points of lack of Asian child samples and imbalance of rare expression data in existing databases. In this way, the embodiment of the present application can collect and label more than 200 high-quality child dynamic expression videos.

[0030] It should be noted that the method of collecting child dynamic expression videos by the embodiment of the present application is not limited to constructing a game-based online interactive system, and any form of interactive application (game, story, task, etc.) can be used, and it can be in an unguided and natural environment.

[0031] After obtaining more than 200 high-quality child dynamic expression videos, interval frame extraction is performed on each Asian child dynamic expression video to form an Asian child expression image sequence. This method makes expression recognition not dependent on single-frame static images, but through interval frame extraction on continuous video streams to form an image sequence, which can be input into a neural network model as a whole for time sequence feature analysis and recognition, so as to capture complete dynamic information of expressions from the beginning, development to the peak, which is more robust than static analysis and more in line with the natural law of human expressions, thereby significantly improving the recognition accuracy.

[0032] Finally, the Asian child expression image sequence and a plurality of Asian child expression grayscale images are merged, and further expanded through image rotation, flipping and other data enhancement techniques, and finally an enhanced mixed expression database containing 34221 training samples and 8141 test samples is formed. In step S102, the pre-constructed deep convolutional neural network model is iteratively trained using the enhanced mixed expression database until a preset iteration period is reached to obtain a deep time sequence emotion computing model.

[0033] In some embodiments, the pre-constructed deep convolutional neural network model is trained using the enhanced mixed expression database until a preset iteration period is reached to obtain a deep time sequence emotion computing model, including: Each frame of image in the enhanced mixed expression database is input into the 2D convolutional layer in the deep convolutional neural network model to extract the spatial features of each frame of image; The spatial features of each frame of image are input into the 3D convolutional layer or the LSTM layer in the deep convolutional neural network model to learn the change rule of the spatial features of each frame of image; The change rule of the spatial features of each frame of image is input into the fully connected layer in the deep convolutional neural network model to output a 7-dimensional similarity ratio vector; The cross-entropy loss between the 7-dimensional similarity ratio vector and the true label of each frame of image corresponding to the 7-dimensional similarity ratio vector is calculated; Based on the Adam optimizer and the back propagation algorithm, the weights of the deep convolutional neural network model are adjusted according to the cross-entropy loss, and the foregoing process is iteratively executed until a preset iteration period is reached to obtain a deep time sequence emotion computing model.

[0034] In actual execution, unlike the traditional research relying on single-frame image detection, the embodiment of the present application analyzes the complete dynamic process of the expression. In the recognition stage, the continuous video stream in the enhanced hybrid expression database is sampled at intervals (the sampling frequency can be customized), and the continuous image sequence is input into the model as a whole for analysis. The specific process is as follows: the spatial features of each frame of image are input into the 3D convolution layer or the LSTM layer in the deep convolutional neural network model to learn the change rule of the spatial features of each frame of image; the change rule of the spatial features of each frame of image is input into the full connection layer in the deep convolutional neural network model to output a 7-dimensional similarity ratio vector; the cross-entropy loss between the 7-dimensional similarity ratio vector and the true label of each frame of image corresponding to the 7-dimensional similarity ratio vector is calculated; based on the Adam optimizer and the back propagation algorithm, the weights of the deep convolutional neural network model are adjusted according to the cross-entropy loss, and the foregoing process is iteratively executed until a preset iteration period is reached, so as to obtain the deep time sequence emotion calculation model. This enables the deep time sequence emotion calculation model obtained after training to capture the complete time sequence information of the expression from the beginning, development to the vertex, thereby significantly improving the accuracy and robustness of recognition.

[0035] In step S103, the multi-modal dynamic expression of the child to be recognized is input into the deep time sequence emotion calculation model to calculate the current emotion of the child.

[0036] In some embodiments, inputting the multi-modal dynamic expression of the child to be recognized into the deep time sequence emotion calculation model to calculate the current emotion of the child includes: inputting the multi-modal dynamic expression of the child to be recognized into the deep time sequence emotion calculation model for emotion calculation to obtain a 7-dimensional similarity ratio vector; determining the current emotion of the child according to the 7-dimensional similarity ratio vector.

[0037] In actual execution, in order to improve the flexibility and fault tolerance of the model in actual application, the embodiment of the present application labels not a single, absolute emotion label during data collection, and then after inputting the multi-modal dynamic expression of the child to be recognized into the deep time sequence emotion calculation model for emotion calculation, a multi-dimensional similarity ratio vector is obtained. The 7-dimensional similarity ratio vector can be processed in the form of a similarity percentage, probability distribution or confidence score vector of a plurality of basic emotions to obtain the specific emotion of the child.

[0038] The vector quantifies the respective similarity degrees of the input expression and the seven basic emotions (six basic emotions + neutral). This probabilistic and refined feedback mechanism can better describe ambiguous, complex or mixed emotional states, and provide more rich information and higher fault tolerance space for downstream applications.

[0039] The emotion calculation method of children's facial expressions is described in detail below through a specific embodiment.

[0040] Step one, first collect multiple gray-scale images of Asian children's expressions representing seven emotions from the Children's Affective Facial Expressions (CAFE) dataset containing 1200 high-quality photos of 100+ children aged 2-8, and the 2013 Facial Expression Recognition (FER2013) dataset containing over 35,000 gray-scale images of 48x48 pixels representing seven core emotions (anger, disgust, fear, happiness, sadness, surprise, neutral); Further, deploy the gamified interactive system of the pre-constructed online interactive application on a standard personal computer or tablet device equipped with a standard front-facing camera (e.g., 720p or 1080p resolution), and invite an 8-year-old Chinese child to participate in the game. The game guides specific emotions through a series of carefully designed levels. For example, the "Happy" level: in combination with a previously watched social animation about "celebrating a birthday with classmates", make a happy expression against the camera, and get 1-3 digital star rewards. The discrimination uses the first generation model trained based on the CAFE and FER2013 datasets as the discrimination tool, as shown in Figure 2 .

[0041] During the entire process of the child playing the game, the front-facing camera of the device continuously records the child's facial expressions at a rate of 30 frames per second (customizable), and saves the video stream in.mp4 format, obtaining multiple segments of Asian children's dynamic expression videos, and each video is associated with a specific game animation scenario (i.e., a pre-set emotion induction target).

[0042] Further, two trained psychology professionals manually review the collected multiple segments of Asian children's dynamic expression videos to accurately intercept and label the video segments in which the child exhibits clear emotions (e.g., a 3-second segment from seeing the gift box to the end of the smile), and label the segment with the final confirmed emotion label (e.g., "happy").

[0043] Using an automated script program to process the labeled video segments, specifically, extracting consecutive image frames from each segment at a rate of 10 frames per second, forming an image sequence. For example, a 3-second "happy" video segment will be converted into a sequence containing 30 consecutive images.

[0044] To expand the dataset, integrate the extracted partial image sequences, and the pre-processed static image samples in the CAFE and FER2013 datasets: such as randomly flipping all images in the sequence horizontally, or performing a ±5-degree slight rotation, forming an enhanced hybrid database containing over 30,000 training samples for model training, as shown inFigure 3 As shown.

[0045] Step two, based on a deep learning server equipped with a high-performance GPU (such as NVIDIA RTX 4090 graphics processor), a spatio-temporal convolutional neural network (Spatio-temporal CNN) is built using Python language and TensorFlow / PyTorch framework. The spatio-temporal convolutional neural network first contains several 2D convolutional layers for extracting spatial features of each frame of image (such as the shape of eyes and mouth). Then, the spatial features of each frame of image are sent to a 3D convolutional layer or an LSTM (Long Short-Term Memory Network) layer to learn the variation of these features in the time dimension. The last layer of the network is a fully connected layer with Softmax activation function, which is used to output a 7-dimensional similarity ratio vector.

[0046] The enhanced hybrid expression database is input into the spatio-temporal convolutional neural network for prediction. The cross-entropy loss between the predicted results and the true labels is calculated, and the Adam optimizer and backpropagation algorithm are used to iteratively adjust the weights inside the network. The entire training process lasts about 50 cycles (epochs), until the accuracy of the model on the test set no longer improves significantly. After training, the optimal model weights are saved as a file, namely the deep temporal sentiment computing model, for subsequent deployment.

[0047] As shown in Figure 4 In the benchmark performance test, the average recognition accuracy of the deep temporal sentiment computing model proposed by the embodiment of the application for the six basic expressions reached 61.42%, which is much higher than the baseline level of random guessing 16.67%. In practical applications, by setting a reasonable confidence tolerance threshold (for example, considering the top two similar emotions as valid recognition), the comprehensive effective recognition rate of the method can be improved to more than 90%, taking into account the accuracy of the model and the flexibility of the application.

[0048] Step four, as shown in Figure 5 The online interactive application "Emotion Kingdom" (iterative version) is used again, and it is loaded with the deep temporal sentiment computing model trained in the previous step. This program can be used by psychologists or special education teachers when interacting with children.

[0049] When the teacher interacts with the child, the application is started. The program captures the child's facial video stream in real time through the computer camera. In the "expression modeling" and "expression recognition" modules, the model is activated, and the process is similar to step one to obtain the image sequence.

[0050] The image sequence is sent into the loaded deep time sequence emotion calculation model for fast calculation (inference) to output and save a 7-dimensional similarity ratio vector, for example: {happy: 0.85, neutral: 0.10, surprise: 0.03, sad: 0.01,...}. The designer and programmer can autonomously determine the mapping relationship between the ratio and the game feedback, for example, when happy ranks first, three stars are awarded. The teacher can judge the performance level of the child by checking the visual feedback or background data, so as to further adjust the education strategy.

[0051] In summary, the emotion calculation method for children's expressions according to the embodiment of the present application has the following beneficial effects: (1) By autonomously collecting dynamic expression data of Chinese children in a natural game situation, a high-quality and targeted data set is created, which not only fills the key blank of the lack of Asian children samples in existing public databases, but also effectively improves the problem of emotion data class imbalance by collecting the dynamic process of rare expressions; (2) By combining the enhanced hybrid database and the dynamic time sequence analysis method, the performance of the children's expression recognition model trained is excellent. The average recognition accuracy of six basic expressions reaches 61.42%, which is much higher than the baseline level of random guessing (16.67%), which strongly proves the superiority of analyzing the complete dynamic process over analyzing single-frame static images; (3) By feeding back the similarity ratio of the seven basic expressions instead of a single determination result, great flexibility and fault tolerance are provided for actual application. For example, in an education or treatment scene, a reasonable confidence threshold can be set (such as considering the two results with the highest similarity as valid), so as to ensure high accuracy while better adapting to complex and ambiguous emotional expressions in the real world, making it have higher real application value.

[0052] Secondly, the emotion calculation device for children's expressions according to the embodiment of the present application is described with reference to the accompanying drawings.

[0053] Figure 6 is a block schematic diagram of an emotion calculation device for children's expressions provided by the embodiment of the present application.

[0054] As shown in Figure 6 , the emotion calculation device for children's expressions 60 comprises a model construction module 601, an iterative training module 602 and a recognition module 603.

[0055] The construction module 601 is configured to collect and label a plurality of Asian child multi-modal dynamic expression data, and construct an enhanced hybrid expression database according to the plurality of Asian child multi-modal dynamic expression data; the iterative training module 602 is configured to iteratively train a pre-constructed deep convolutional neural network model by using the enhanced hybrid expression database until a preset iteration period is reached, so as to obtain a deep time-series emotion computing model; and the recognition module 603 is configured to input a to-be-recognized child multi-modal dynamic expression into the deep time-series emotion computing model, so as to calculate a current emotion of the child.

[0056] In some embodiments, the plurality of Asian child multi-modal dynamic expression data includes a plurality of Asian child expression grayscale images and a plurality of Asian child dynamic expression videos.

[0057] In some embodiments, the construction module 601 includes: A first collection unit configured to collect a plurality of Asian child expression grayscale images representing seven emotions from a preset child emotion facial expression dataset and a preset facial expression recognition dataset; A second collection unit configured to collect and label a plurality of Asian child dynamic expression videos representing the seven emotions through game interaction; A frame extraction unit configured to extract frames from each of the plurality of Asian child dynamic expression videos at intervals to form a sequence of Asian child expression images; An enhanced construction unit configured to perform data enhancement processing on the plurality of Asian child expression grayscale images and the sequence of Asian child expression images to construct the enhanced hybrid expression database.

[0058] In some embodiments, the seven emotions include anger, disgust, fear, happiness, sadness, surprise, and neutrality.

[0059] In some embodiments, the iterative training module 602 includes: An extraction unit configured to input each frame of image in the enhanced hybrid expression database into a 2D convolutional layer in the deep convolutional neural network model to extract spatial features of each frame of image; A learning unit configured to input the spatial features of each frame of image into a 3D convolutional layer or an LSTM layer in the deep convolutional neural network model to learn variation rules of the spatial features of each frame of image; An output unit configured to input the variation rules of the spatial features of each frame of image into a fully connected layer in the deep convolutional neural network model to output a 7-dimensional similarity ratio vector; A calculation unit configured to calculate a cross-entropy loss between the 7-dimensional similarity ratio vector and a true label of each frame of image corresponding to the 7-dimensional similarity ratio vector; An iteration unit is configured to adjust weights of the deep convolutional neural network model according to a cross-entropy loss based on an Adam optimizer and a back propagation algorithm, and iteratively execute the foregoing process until a preset iteration period is reached, to obtain the deep temporal affective computing model.

[0060] In some embodiments, the recognition module 603 comprises: A calculation unit is configured to input the multi-modal dynamic expression of the child to be recognized into the deep temporal affective computing model for affective computing, to obtain a 7-dimensional similarity ratio vector; A determination unit is configured to determine the current emotion of the child according to the 7-dimensional similarity ratio vector.

[0061] It should be noted that the foregoing explanation of the emotion computing method for the expression of the child is also applicable to the emotion computing device for the expression of the child, which will not be described here again.

[0062] The emotion computing device for the expression of the child according to the embodiments of the present application has the following beneficial effects: (1) By autonomously collecting dynamic expression data of Chinese children in a natural game context, a high-quality and targeted dataset is created, which not only fills the key gap of the lack of Asian child samples in existing public databases, but also effectively improves the problem of unbalanced emotion data categories by collecting the dynamic process of rare expressions; (2) By combining the enhanced hybrid database and the dynamic temporal analysis method, the performance of the child expression recognition model is excellent. The average recognition accuracy of six basic expressions reaches 61.42%, which is much higher than the baseline level of random guessing (16.67%), which strongly proves the superiority of analyzing the complete dynamic process over analyzing single-frame static images; (3) By feeding back the similarity ratio of the seven basic expressions instead of a single determination result, great flexibility and fault tolerance are provided for actual application. For example, in an educational or therapeutic scenario, a reasonable confidence threshold can be set (such as considering the two results with the highest similarity as valid), so as to better adapt to complex and ambiguous emotional expressions in the real world while ensuring high accuracy, making it have higher real application value.

[0063] Figure 7 A structural schematic diagram of an electronic device provided by the embodiments of the present application.

[0064] The electronic device can comprise: A memory 701, a processor 702, and a computer program stored in the memory 701 and executable on the processor 702.

[0065] The processor 702 implements the emotion calculation method of the child expression provided in the above embodiments when executing a program.

[0066] Further, the electronic device further comprises: The communication interface 703 is configured to communicate between the memory 701 and the processor 702.

[0067] The memory 701 is configured to store a computer program executable on the processor 702.

[0068] The memory 701 can include a high-speed RAM memory, and can further include a non-volatile memory, for example, at least one disk memory.

[0069] If the memory 701, the processor 702 and the communication interface 703 are implemented independently, the communication interface 703, the memory 701 and the processor 702 can be connected to each other through a bus and complete the communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 7 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0070] Optionally, in a specific implementation, if the memory 701, the processor 702 and the communication interface 703 are integrated on a chip, the memory 701, the processor 702 and the communication interface 703 can complete the communication between each other through an internal interface.

[0071] The processor 702 can be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0072] The embodiments of the present application further provide a computer program product, and the computer program / instruction is executed by the processor to implement the above emotion calculation method of the child expression.

[0073] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for calculating the emotion of a child's facial expressions.

[0074] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0075] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0076] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0077] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0078] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0079] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0080] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0081] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A method of emotion computing of a child's expression, characterized by, The method comprises the following steps: Collect and label a plurality of Asian children's multi-modal dynamic expression data, and construct an enhanced hybrid expression database according to the plurality of Asian children's multi-modal dynamic expression data; Iteratively train a pre-constructed deep convolutional neural network model using the enhanced hybrid expression database until a preset iteration period is reached to obtain a deep time series emotion computing model; Input the to-be-identified children's multi-modal dynamic expression into the deep time series emotion computing model to calculate the current emotion of the children.

2. The method of claim 1, wherein, The plurality of Asian children's multi-modal dynamic expression data comprises a plurality of Asian children's expression grayscale images and a plurality of segments of Asian children's dynamic expression videos. 3.The method of claim 1, wherein, The plurality of Asian children's multi-modal dynamic expression data is collected, and the enhanced hybrid expression database is constructed according to the plurality of Asian children's multi-modal dynamic expression data, comprising: Collect a plurality of Asian children's expression grayscale images representing seven emotions from a preset children's emotional facial expression dataset and a preset facial expression recognition dataset; Collect and label a plurality of segments of Asian children's dynamic expression videos representing seven emotions through game interaction; Interval frame extraction is performed on each segment of Asian children's dynamic expression video to form a sequence of Asian children's expression images; Data enhancement processing is performed on the plurality of Asian children's expression grayscale images and the sequence of Asian children's expression images to construct the enhanced hybrid expression database.

4. The method of claim 3, wherein, The seven emotions include anger, disgust, fear, happiness, sadness, surprise and neutrality. 5.The method of claim 1, wherein, The training of the pre-constructed deep convolutional neural network model using the enhanced hybrid expression database until a preset iteration period is reached to obtain a deep time series emotion computing model, comprising: Input each frame of image in the enhanced hybrid expression database into a 2D convolutional layer in the deep convolutional neural network model to extract the spatial features of each frame of image; Input the spatial features of each frame of image into a 3D convolutional layer or an LSTM layer in the deep convolutional neural network model to learn the change rule of the spatial features of each frame of image; Input the change rule of the spatial features of each frame of image into a fully connected layer in the deep convolutional neural network model to output a 7-dimensional similarity ratio vector; Calculate the cross-entropy loss between the 7-dimensional similarity ratio vector and the true label of each frame of image corresponding to the 7-dimensional similarity ratio vector; Based on the Adam optimizer and the back propagation algorithm, adjust the weights of the deep convolutional neural network model according to the cross-entropy loss, and iteratively execute the foregoing process until a preset iteration period is reached to obtain the deep time series emotion computing model. 6.The method of claim 1, wherein, The input of the to-be-identified children's multi-modal dynamic expression into the deep time series emotion computing model to calculate the current emotion of the children, comprising: Input the to-be-identified children's multi-modal dynamic expression into the deep time series emotion computing model for emotion calculation to obtain a 7-dimensional similarity ratio vector; Determine the current emotion of the children according to the 7-dimensional similarity ratio vector.

7. A device for emotion computing of children's expressions, characterized by, The method comprises: A construction module for collecting and labeling a plurality of Asian children's multi-modal dynamic expression data, and constructing an enhanced hybrid expression database according to the plurality of Asian children's multi-modal dynamic expression data; an iterative training module configured to iteratively train a pre-constructed deep convolutional neural network model using the enhanced mixed expression database until a preset iteration period is reached, to obtain a deep time-series emotion computing model; an identification module configured to input a to-be-identified multi-modal dynamic expression of a child into the deep time-series emotion computing model, to calculate a current emotion of the child.

8. An electronic device, comprising: comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the emotion computing method for a child expression according to any one of claims 1-6.

9. A computer program product, characterised in that, The computer program / instructions, when executed by the processor, implement the emotion computing method for a child expression according to any one of claims 1-6.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the emotion computing method for a child expression according to any one of claims 1-6.

Images