A method, device, medium and electronic device for recognizing facial emotions

Abstract

The embodiments of the present application provide a method, device, computer-readable medium and electronic device for facial emotion recognition, and relate to the field of computer vision technology of artificial intelligence. The method for facial emotion recognition includes: detecting a facial area contained in an image of a facial emotion to be recognized; based on the facial area, recognizing the facial expression type, and identifying the action label of the target facial organ corresponding to the expression type; based on the expression type, identifying the action label of the target facial organ corresponding to the expression type; and determining the facial emotion in the image based on the expression type and the action label of the target facial organ. The technical solution of the embodiment of the present application improves the comprehensiveness and accuracy of facial emotion detection by identifying the expression type in the facial area and determining the action label of the facial organ corresponding to the expression type, so as to obtain comprehensive facial emotion information by integrating the expression type and the action label.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology of artificial intelligence, and more specifically, to a method, device, computer-readable medium and electronic device for facial emotion recognition. Background Art

[0002] Emotion recognition is an important direction for computers to understand human emotions. Through emotion recognition, the emotions of people's faces in a certain scene can be determined. Generally, the emotions corresponding to the face images are obtained by identifying the key point information in the face image. However, the face angles in real business scenarios are rich and varied. Only relying on the coordinates of the key points cannot accurately identify the expression or emotion of the image, especially when the facial image of the person is irregular. Therefore, there is a problem of low accuracy in emotion recognition. Summary of the invention

[0003] The embodiments of the present application provide a method, apparatus, computer-readable medium and electronic device for facial emotion recognition, thereby obtaining comprehensive facial emotion information at least to a certain extent, thereby improving the comprehensiveness and accuracy of facial emotion recognition.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0005] According to one aspect of an embodiment of the present application, a method for facial emotion recognition is provided, including: detecting a facial area contained in an image of a facial emotion to be recognized; identifying a facial expression type based on the facial area; identifying an action label of a target facial organ corresponding to the expression type based on the expression type; and determining the facial emotion in the image based on the expression type and the action label of the target facial organ.

[0006] According to one aspect of an embodiment of the present application, a device for facial emotion recognition is provided, including: a detection unit, used to detect a facial area contained in an image of a facial emotion to be recognized; an expression recognition unit, used to recognize the facial expression type based on the facial area; an action recognition unit, used to recognize the action label of a target facial organ corresponding to the expression type based on the expression type; and a determination unit, used to determine the facial emotion in the image based on the expression type and the action label of the target facial organ.

[0007] In some embodiments of the present application, based on the aforementioned scheme, the device for facial emotion recognition also includes: a first acquisition unit, used to acquire the video file to be processed; a parsing unit, used to parse the video file to obtain a video frame; a filtering unit, used to filter the video frame based on pixel information of the video frame, and use the filtered video frame as the image of the facial emotion to be identified.

[0008] In some embodiments of the present application, based on the aforementioned scheme, the filtering unit includes: a first generating unit, used to generate a histogram corresponding to the video frame based on pixel information of the video frame; a clustering unit, used to cluster the video frames according to the histogram and an edge detection operator to obtain at least one class; a first filtering unit, used to filter repeated video frames in each of the classes.

[0009] In some embodiments of the present application, based on the aforementioned scheme, the first filtering unit includes: a clarity unit, used to determine the clarity of the video frame based on pixel information of the video frame; and a second filtering unit, used to filter video frames whose clarity is less than a clarity threshold.

[0010] In some embodiments of the present application, based on the aforementioned scheme, the first filtering unit includes: a region detection unit, used to detect the face region in the video frame based on the pixel information of the video frame; a third filtering unit, used to filter video frames that do not include the face region or include incomplete face regions.

[0011] In some embodiments of the present application, based on the aforementioned scheme, the action recognition unit includes: a target determination unit, used to determine the target facial organ corresponding to the expression type according to the correspondence between the expression type and the facial organ; and a target recognition unit, used to identify the action label of the target facial organ.

[0012] In some embodiments of the present application, based on the aforementioned scheme, the action recognition unit includes: a label recognition unit, used to identify the action label corresponding to the facial organ in the face area; a label determination unit, used to determine the target facial organ corresponding to the expression type and the action label of the target facial organ according to the correspondence between the expression type and the facial organ.

[0013] In some embodiments of the present application, based on the aforementioned scheme, the label recognition unit includes: a first extraction unit, used to extract features of the face area to obtain a face feature map; an action label unit, used to input the face feature map into classification models corresponding to each facial organ, respectively, to obtain action labels corresponding to each facial organ.

[0014] In some embodiments of the present application, based on the aforementioned scheme, the facial organs include at least one of the following: mouth, eyes and eyebrows; the mouth action label includes at least one of the following: mouth wide open, mouth slightly open, pursed mouth or closed mouth; the eye action label includes squinting, opening eyes or closing eyes; the eyebrow action label includes frowning.

[0015] In some embodiments of the present application, based on the aforementioned scheme, the image of the facial emotion to be identified includes a video frame of a video file, and the facial emotion recognition device also includes: a first recognition unit, used to identify a target image corresponding to a specified facial emotion from the video frame; a second extraction unit, used to extract a video clip containing the target image from the video file according to the position of the target image in the video file; and a second generation unit, used to generate an emoticon package corresponding to the target emotion based on the video clip.

[0016] In some embodiments of the present application, based on the aforementioned scheme, the image of the facial emotion to be identified includes video frames of a video file, and the facial emotion recognition device also includes: a second acquisition unit, used to obtain the facial emotion corresponding to each video frame in the video file; a third generation unit, used to generate a facial emotion catalog according to the facial emotion corresponding to each video frame and the playback time of each video frame in the video file.

[0017] In some embodiments of the present application, based on the aforementioned scheme, the image of the facial emotion to be identified includes video frames of a video file, and the facial emotion recognition device also includes: a third acquisition unit, used to obtain at least two video frames corresponding to the specified facial emotion from the video file; a merging unit, used to merge the at least two video frames to obtain a clipped video corresponding to the specified facial emotion.

[0018] In some embodiments of the present application, based on the aforementioned scheme, the third acquisition unit includes: a search unit, used to search for the target playback time corresponding to the specified facial emotion in the video file from the facial emotion directory corresponding to the video file; a third extraction unit, used to extract the video frame corresponding to the target playback time from the video file based on the target playback time.

[0019] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for facial emotion recognition as described in the above embodiment is implemented.

[0020] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method for facial emotion recognition as described in the above embodiments.

[0021] In the technical solutions provided in some embodiments of the present application, the facial area contained in the image of the facial emotion to be identified is detected to identify the expression type in the facial area; and the action label of the target facial organ corresponding to the expression type in the facial area is identified, so as to obtain comprehensive facial emotion information by combining the expression type and the action label, thereby improving the comprehensiveness and accuracy of facial emotion detection.

[0022] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0024] Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied;

[0025] Figure 2 A schematic diagram schematically shows an exemplary system architecture according to an embodiment of the present application;

[0026] Figure 3 A flowchart of a method for facial emotion recognition according to an embodiment of the present application is schematically shown;

[0027] Figure 4 A flowchart for processing a video frame according to an embodiment of the present application is schematically shown;

[0028] Figure 5 A schematic diagram of obtaining a video file of an external environment according to an embodiment of the present application is shown;

[0029] Figure 6 A schematic diagram schematically shows a facial region contained in an image based on facial emotion to be identified according to an embodiment of the present application;

[0030] Figure 7 A schematic diagram schematically shows a facial region contained in an image based on facial emotion to be identified according to an embodiment of the present application;

[0031] Figure 8 A schematic diagram schematically shows a sub-task for identifying action labels corresponding to various facial organs according to an embodiment of the present application;

[0032] Fig. 9 A schematic diagram schematically shows a sub-task for identifying action labels corresponding to various facial organs according to an embodiment of the present application;

[0033] Fig.10 A schematic diagram of identifying facial emotions in a video according to an embodiment of the present application is schematically shown;

[0034] Fig.11 A schematic diagram of determining facial emotions according to an embodiment of the present application is schematically shown;

[0035] Fig.12 A block diagram of a device for facial emotion recognition according to an embodiment of the present application is schematically shown;

[0036] Fig.13 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0037] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0038] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0039] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0040] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0041] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0042] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0043] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further perform graphics processing so that the computer processes the images into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0044] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive teaching and other technologies.

[0045] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common facial payment, face verification, smart home, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0046] The solution provided in the embodiments of the present application involves artificial intelligence computer vision technology and machine learning technology, which is specifically described by the following embodiments:

[0047] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied is shown.

[0048] like Figure 1 As shown, the system architecture may include terminal devices (such as Figure 1 The embodiment of the present invention is a schematic diagram of a mobile phone 101, a tablet computer 102, a portable computer 103, a desktop computer, etc., a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the terminal device and the server 105. The network 104 may include various connection types, such as a wired communication link, a wireless communication link, etc.

[0049] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0050] The user can use the terminal device to interact with the server 105 through the network 104 to receive or send messages, etc. The server 105 can be a server that provides various services. For example, the user uses the terminal device 103 (or the terminal device 101 or 102) to upload an image of a facial emotion to be identified to the server 105, and the server 105 detects the face area contained in the image of the facial emotion to be identified; identifies the facial expression type based on the facial area, and identifies the action label corresponding to the facial organ in the facial area; determines the facial emotion in the image based on the expression type and the action label of the target facial organ. By identifying the expression type corresponding to the facial area and the action label corresponding to the facial organ, and combining the expression type and action label to obtain comprehensive facial emotion information, the comprehensiveness and accuracy of facial emotion detection are improved.

[0051] It should be noted that the method for facial emotion recognition provided in the embodiment of the present application is generally executed by the server 105, and accordingly, the facial emotion recognition device is generally set in the server 105. However, in other embodiments of the present application, the terminal device may also have similar functions as the server, so as to execute the facial emotion recognition solution provided in the embodiment of the present application.

[0052] Figure 2 A schematic diagram of an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied is shown.

[0053] like Figure 2 As shown, the terminal device 210 in the system architecture collects photos or videos of people to obtain a face image 220, and detects the face area contained in the image of the face emotion to be identified; identifies the expression type of the face according to the face area, and identifies the action label corresponding to the facial organ in the face area; determines the face emotion in the image according to the expression type and the action label of the target facial organ. By identifying the expression type corresponding to the face area and the action label corresponding to the facial organ, and combining the expression type and action label, comprehensive face emotion information is obtained.

[0054] In this embodiment, the image of the facial emotion to be recognized can be directly obtained through the terminal device 210, so as to directly recognize the facial emotion on the image. Furthermore, the method in this embodiment can retake the acquired facial image when the effect is not good, thereby improving the operability and controllability of facial emotion recognition.

[0055] The implementation details of the technical solution of the embodiment of the present application are described in detail below:

[0056] Please refer to Figure 3 and Figure 4 As shown, Figure 3 A flowchart of a method for facial emotion recognition according to an embodiment of the present application is shown. The method for facial emotion recognition can be executed by a server, which can be Figure 1 The server shown in , or Figure 2 Terminal equipment in Figure 4 A flow chart of processing video frames in an embodiment of the present application is shown.

[0057] exist Figure 3 Before the process of detecting the face area contained in the image to be identified in step S310, steps S410 to S430 are also included, which are described in detail as follows:

[0058] In step S410, a video file to be processed is obtained.

[0059] In one embodiment of the present application, the video file can be obtained by shooting with a camera, or by obtaining it from other data storage devices. In addition, it can also be video data in the network, which is not limited here.

[0060] Figure 5 A schematic diagram of obtaining a video file of an external environment provided in an embodiment of the present application.

[0061] like Figure 5 As shown, in this embodiment, a video file of an object can be shot in real time by the terminal device 510 to identify the emotion of the object; video files of multiple objects in the external environment can also be shot to obtain an image 520 containing multiple faces, and the facial emotions in the image 520 of multiple faces can be identified to determine the emotions of most people in the environment, and corresponding processing can be performed based on the emotions of these people.

[0062] Exemplarily, in this embodiment, an image of a facial emotion to be identified can be obtained through a video file to identify the facial emotion in the video file for subsequent editing and the like.

[0063] In step S420, the video file is parsed to obtain video frames.

[0064] In one embodiment of the present application, after the video file to be processed is acquired, the video file is parsed to obtain video frames, so as to perform facial emotion recognition based on the facial images in the single video frames.

[0065] Optionally, in the process of parsing a video file, a video frame rate is first set, and based on the video frame rate and the length of the video file, a period for extracting video frames is determined, and the corresponding position of each video frame in the video file is determined according to the period for extracting video frames; based on the corresponding position of each video frame in the video file, video frames are extracted from the video file.

[0066] Specifically, in this embodiment, the corresponding position of the video frame in the video file may include the corresponding playback time of the video frame in the video file.

[0067] Optionally, in this embodiment, the video file can be directly processed by video processing software to obtain video frames.

[0068] Furthermore, in this embodiment, since the video file is processed to obtain video frames, facial expressions are recognized based on the video frames. Therefore, in this embodiment, the time periods in which faces exist in the video file can be first identified, and then the video contents corresponding to these time periods can be processed to obtain video frames containing face areas.

[0069] In step S430, the video frame is filtered based on the pixel information of the video frame, and the video frame obtained after filtering is used as the image of the facial emotion to be recognized.

[0070] In one embodiment of the present application, after obtaining the video frames, the video frames are filtered based on the pixel information of the video frames to filter out the video frames with poor effects such as blurry or repeated effects in the video frames, so as to obtain concise and better video frames, and the filtered video frames are used as images of facial emotions to be recognized.

[0071] In one embodiment of the present application, the process of filtering the video frame based on the pixel information of the video frame in step S230 includes the following steps:

[0072] Based on the pixel information of the video frame, a histogram corresponding to the video frame is generated;

[0073] Clustering the video frames according to the histogram and the edge detection operator to obtain at least one class;

[0074] Filter duplicate video frames in each class.

[0075] In one embodiment of the present application, pixel information in a video frame is extracted and a histogram is generated to determine the image information corresponding to each video frame; then, video frames belonging to the same type are determined based on the histogram and the edge detection operator to obtain at least one class. Since there may be at least one image frame in each class, and the image frames in each class are similar images, processing all the image frames in a class will consume a long processing time and cost. Therefore, in this embodiment, repeated video frames in each class are filtered so that each class includes only one video frame.

[0076] In this embodiment, by counting the grayscale information in the pixel information, the frequency of occurrence of all pixels in the video frame is counted according to the size of the grayscale value, and a grayscale histogram is generated to determine the video frames belonging to the same category based on the grayscale histogram and the edge detection operator.

[0077] By filtering the video frames, duplicate video frames can be reduced, and a video frame in the same class can be used to represent other video frames in the same class, so as to perform emotion recognition on a video frame in the class, and use the facial emotion corresponding to the identified video frame as the facial emotion of all video frames in the same class. In this way, the number of images to be recognized can be streamlined, the workload and processing time of face recognition can be reduced, and the efficiency of video processing can be improved.

[0078] In one embodiment of the present application, the process of filtering the video frame based on the pixel information of the video frame in step S230 includes the following steps:

[0079] Determine the definition of the video frame based on the pixel information of the video frame;

[0080] Filter the video frames whose definition is less than the definition threshold.

[0081] In one embodiment of the present application, the definition of the video frame is determined by the pixel information of the video frame. For example, the corresponding resolution is determined according to the pixel distribution in each video frame, and the resolution is used as the definition of the video frame. According to the set definition threshold, the video frames with a definition less than the definition threshold are filtered out to obtain a streamlined video frame.

[0082] In this embodiment, by filtering the video frames in the above manner, video frames with lower definition can be filtered out to ensure the accuracy of facial emotion recognition and improve the effect of facial emotion recognition.

[0083] In one embodiment of the present application, the process of filtering the video frame based on the pixel information of the video frame in step S230 includes the following steps:

[0084] Based on the pixel information of the video frame, the face area in the video frame is detected;

[0085] Filter the video frames that do not include the face area or include an incomplete face area.

[0086] In one embodiment of the present application, the video frames extracted from the video file may be a plurality of continuous video frames, which may or may not contain a face region. In this embodiment, the face region in the video frame is identified based on the pixel information of the video frame. If the video frame contains a face region, the video frame is retained; if the video frame does not contain the face region, the video frame is filtered out.

[0087] Furthermore, since a partial face region may be detected in the video frame, the complete ratio of the face region may be determined based on the detected face region in this embodiment. When the complete ratio exceeds a set ratio threshold, the video frame is retained, and when the complete ratio is less than the ratio threshold, the video frame is filtered out.

[0088] In this embodiment, the video frames are filtered out by detecting the face area to ensure the integrity of the recognized face area, thereby improving the accuracy of face recognition.

[0089] Reference Figure 3 As shown, the method for facial emotion recognition includes at least steps S310 to S340, which are described in detail as follows:

[0090] In step S310, a face region contained in an image of a facial emotion to be recognized is detected.

[0091] In one embodiment of the present application, an image for which facial emotion is to be recognized is detected to determine a facial region contained therein, and then, with respect to the facial region, an expression type corresponding to the facial region is determined.

[0092] Optionally, in this embodiment, the face region detection can be realized by a face region model. In the application process, the video can be directly passed through the face region model. When the face region is identified in the video frame, the face region is captured as the input of the next model. Optionally, the face region model in this embodiment is trained by face region detection samples and their corresponding annotation information.

[0093] Figure 6 A schematic diagram of a facial region contained in an image detection based on facial emotions to be identified provided in an embodiment of the present application.

[0094] like Figure 6 As shown, in this embodiment, the face region 620 in the image 610 can be detected by image recognition, and the face region 620 can be captured as an input for subsequent emotion recognition.

[0095] It should be noted that the method for recognizing the face area in this embodiment can be performed by a recognition method based on face feature points, a recognition method based on the entire face image, a recognition method based on a template, or a method using a neural network for recognition, which is not described in detail here.

[0096] In step S320, the facial expression type is identified based on the facial region.

[0097] In one embodiment of the present application, when identifying the expression type of a face region, it can be identified by a classification model obtained through training, such as a classification model obtained through residual network training, which is not limited here. The expression type obtained through model recognition can include happy, sad, shocked, calm or other expression types, which are not limited here.

[0098] In step S330, based on the expression type, an action label of a target facial organ corresponding to the expression type is identified.

[0099] In practical applications, according to the criteria for judging facial emotions, when facial expressions are combined with certain facial features, there may be deeper emotions. For example, when a person is happy, if their mouth opens wide, it may be due to excitement. Considering this, in this embodiment, the correspondence between the expression type and the facial organs is determined based on the connection between facial expressions and facial features. In this embodiment, the correspondence between the expression type and the facial organs is used to represent the connection between different types of expressions and facial organs, and is used to analyze and obtain the corresponding deep emotions based on the facial organs corresponding to a certain expression.

[0100] Exemplarily, the correspondence between expression types and facial organs in this embodiment may include: the facial organ corresponding to the happy expression type may be the mouth or the eye; the facial organ corresponding to the sad expression type may be the mouth; the facial organ corresponding to the shocked expression type may be the mouth and the eye; the facial organ corresponding to the calm expression type may be the mouth or the eyebrow. This is only an example and is not specifically limited.

[0101] In one embodiment of the present application, the process of identifying the action label of the target facial organ corresponding to the expression type in step S330 according to the expression type can be performed in series or in parallel with step S320, wherein the serial steps are as follows:

[0102] Determining the target facial organ corresponding to the expression type according to the correspondence between the expression type and the facial organ;

[0103] Identify the action label of the target facial organ.

[0104] Specifically, in this embodiment, after the expression type is determined, the target facial organ corresponding to the expression type is first determined based on the correspondence between the expression type and the facial organ; then the target facial organ is directly identified to obtain the action label corresponding to the target facial organ. The above-mentioned execution process in series with step S320 can reduce the number of facial organs that need to be identified, thereby reducing the amount of calculation and reducing the computing power requirements of the computer.

[0105] In addition to the above-mentioned execution method in series with step S320, the execution process in parallel with step S320 includes the following steps:

[0106] Identify action labels corresponding to facial organs in the face area;

[0107] According to the correspondence between the expression type and the facial organ, the target facial organ corresponding to the expression type and the action label of the target facial organ are determined.

[0108] In one embodiment of the present application, contrary to the above-mentioned serial method, in a parallel execution process, the present embodiment first identifies the action labels corresponding to the facial organs in the face area, and after obtaining all the action labels, determines the target facial organ corresponding to the expression type according to the correspondence between the expression type and the facial organ, and then determines the action label of the target facial organ.

[0109] In one embodiment of the present application, in the above parallel process, the process of identifying the action labels corresponding to the facial organs in the face area includes step S3301 and step S3302, which are described in detail as follows:

[0110] In step S3301, the features of the face area are extracted to obtain a face feature map.

[0111] In one embodiment of the present application, features in a human face region are identified and extracted to obtain a human face feature map, wherein the human face feature map includes feature information of organs in the human face, such as feature information of eyebrows, eyes, nose, mouth and other regions, and the human face features are not limited here.

[0112] Optionally, in this embodiment, the features of the face area can be extracted through a multi-task model. Considering the background processing efficiency, the facial features recognition model adopts a multi-task model, that is, multiple tasks share a backbone network to pass the face area through the multi-task model to obtain a face feature map containing the features of the facial organs.

[0113] Figure 7 A schematic diagram of extracting a facial feature map based on a shared network is provided in an embodiment of the present application.

[0114] like Figure 7 As shown, the face image 710 may be a processed picture with only the face area; conv_1, conv_2, conv_3, conv_4, and conv_5 respectively represent network modules with residuals; and fc_1 represents a multi-dimensional fully connected layer, for example, 1024 dimensions.

[0115] Exemplarily, in this embodiment, the network layers conv_1, conv_2, conv_3, conv_4, conv_5, and fc_1 are shared network layers, and each task shares the parameters of the above network layers to process the facial image through the shared network; and the processing results are respectively input into the corresponding mouth recognition task 720, eye recognition task 730, and eyebrow recognition task 740 to obtain the corresponding feature recognition results.

[0116] In step S3302, the facial feature map is input into the classification model corresponding to each facial organ to obtain the action label corresponding to each facial organ.

[0117] In one embodiment of the present application, after the face region is processed through a shared network and a face feature map is extracted, the face feature map is input into the classification model corresponding to each facial organ to determine the action label corresponding to each facial organ. This embodiment can reduce the computing power requirement for digital image processing and improve the image recognition efficiency by first processing based on a shared network and then performing distributed processing.

[0118] Figure 8 A schematic diagram of a task-based identification of action labels corresponding to various facial organs provided in an embodiment of the present application.

[0119] like Figure 8 As shown, the facial feature map is input into the classification model corresponding to each facial organ to obtain the action label corresponding to each facial organ. Here, the mouth shape is taken as an example. After passing through the fc_1 layer 810 in the shared network, it passes through the dropout layer 820. The main purpose of the dropout layer is to prevent overfitting. After passing through the fc_2 layer 820, the action label corresponding to the mouth shape task is obtained. Each task in the mouth shape, eye shape, and eyebrow shape has its own fc_2 layer, that is, the fc_2 layer is not shared. After passing through the fc_2 layer, each task is classified and identified to obtain the action label corresponding to each facial organ.

[0120] In one embodiment of the present application, the facial organs include at least one of the following: mouth, eyes and eyebrows, and other facial regions, such as forehead, cheekbone region, etc., which are not limited here. In this embodiment, the mouth action label includes at least one of the following: mouth wide open, mouth slightly open, pursed mouth or closed mouth; the eye action label includes squinting eyes, open eyes or closed eyes; the eyebrow action label includes frowning, etc., which are not limited here.

[0121] It should be noted that, in the above embodiment, the process of identifying the expression type in step S320 is performed in parallel with step S3301 and step S3302, without any order of precedence. In this embodiment, through the parallel execution mode, the action labels corresponding to each facial organ can be identified while identifying the expression type, so as to obtain the recognition results of the expression type and the recognition results of the action label at the same time, and then obtain more vivid facial emotions based on the two recognition results, thereby improving the efficiency and accuracy of emotion recognition.

[0122] In step S340, the facial emotion in the image is determined based on the expression type and the action label of the target facial organ.

[0123] In one embodiment of the present application, after obtaining the expression type and the action label corresponding to the target facial organ, the facial emotion in the image is comprehensively determined based on the expression type and the target action label to reflect the comprehensiveness of facial emotion recognition and improve the accuracy of facial emotion recognition.

[0124] Optionally, in this embodiment, the method of determining facial emotions based on expression types and action labels corresponding to facial organs may include obtaining facial emotions by combining expression types and action labels; or obtaining facial emotions based on expression types as a basis and action labels as auxiliary references; or determining action labels corresponding to the expression types based on the correspondence between expression types and action labels to comprehensively determine facial emotions, and so on, which are not limited here.

[0125] Fig. 9 A schematic diagram of determining facial emotions based on expression types and action labels provided in an embodiment of the present application.

[0126] like Fig. 9 As shown, this embodiment obtains a face image 910 by acquiring a video or image containing a face; recognizes the face image to obtain an expression type 920 corresponding to the entire face, and simultaneously recognizes an action label 930 corresponding to each organ in the face, such as an action label corresponding to the eyebrow, an action label corresponding to the eye, and an action label corresponding to the mouth; finally, the facial emotion 940 is obtained by combining the expression type 920 and the action labels 930 corresponding to each organ.

[0127] Fig.10 A schematic diagram of recognizing facial emotions in a video provided in an embodiment of the present application.

[0128] like Fig.10 As shown, in this embodiment, when processing video images, the video file is first filtered to obtain a filtered video frame 1010; the filtered video frame 1010 is then input into the face area recognition model, that is, the face area is obtained in model 1 (1020), and the face image 1030 is obtained; the processed video frame is passed through the expression recognition model, that is, model 2 to obtain the expression result 1040 of the face; then the action label recognition model is entered, that is, model 3, to obtain the action labels corresponding to the facial features, that is, the facial features results 1050, which may include the action labels corresponding to the eyes 1060, eyebrows 1070 and mouth 1080; finally, the results of the above three models are integrated to determine the complex emotions of the task 1090.

[0129] Fig.11 A schematic diagram of determining facial emotions provided in an embodiment of the present application.

[0130] like Fig.11 As shown, the expression types in this embodiment may include happiness, sadness, shock or calmness, etc., which are not limited here. Based on the expression types and the action labels of the corresponding facial organs, the facial emotions can be obtained.

[0131] Exemplarily, when the facial organ action label corresponding to the happy expression type is mouth wide open, the synthesized facial emotion is excitement; when the facial organ action label corresponding to the happy expression type is squinting, the synthesized facial emotion is expectation; when the facial organ action label corresponding to the sad expression type is pursing the lips, the synthesized facial emotion is aggrieved; when the facial organ action label corresponding to the sad expression type is mouth wide open, the synthesized facial emotion is collapsed; when the facial organ action label corresponding to the sad expression type is mouth closed, the synthesized facial emotion is sad; when the facial organ action label corresponding to the shocked expression type is mouth wide open and eyes open, the synthesized facial emotion is surprise; when the facial organ action label corresponding to the shocked expression type is mouth wide open, the synthesized facial emotion is excitement; when the facial organ action label corresponding to the calm expression type is pursing the lips, the synthesized facial emotion is aggrieved; when the facial organ action label corresponding to the calm expression type is frowning, the synthesized facial emotion is serious.

[0132] After the above process, for the emotion recognition of the characters in the video, not only the basic labels of facial emotions can be output, such as happiness, sadness, shock, calmness, etc.; more complex and deep emotion labels can also be output, such as excitement, expectation, grievance, collapse, sadness, sluggishness, surprise, grievance, seriousness, etc., making the results of facial emotion recognition more accurate and vivid.

[0133] It should be noted that Fig.11 The method of determining facial emotions described in the above is only an example of a method for determining facial emotions for intuitive understanding, and does not represent the determination of facial emotions based on the above formula. In addition, there may be other types of facial expressions, other types of action labels, or more abundant correspondences between facial expressions and facial organs, and there may also be other types of facial emotions and facial organ action labels combined, which will not be elaborated here.

[0134] In one embodiment of the present application, the image of the facial emotion to be recognized includes a video frame of a video file, and the facial emotion recognition method in this embodiment further includes the following steps:

[0135] Identify the target image corresponding to the specified facial emotion from the video frame;

[0136] According to the position of the target image in the video file, extracting the video segment containing the target image from the video file;

[0137] Generate emoticons corresponding to the target emotions based on the video clips.

[0138] In one embodiment of the present application, when processing a video file, a video frame is obtained through the video file, and the video frame is used as an image of a facial emotion to be identified, so as to identify a target image corresponding to a specified facial emotion in the video frame. Then, according to the position of the target image in the video file, the corresponding video clip is extracted as an emoticon package.

[0139] Optionally, the designated facial emotion in this embodiment may be a manually input facial emotion name, such as excitement, expectation, or grievance, etc., which is not limited here.

[0140] Optionally, the position in the video file in this embodiment may include the playback time of the target image in the video file, etc., which is used to determine the storage area corresponding to the target image in the video file based on the position, so as to retrieve the video clip corresponding to the target image based on the position.

[0141] Optionally, the video clip in this embodiment may be a video clip of a preset length, or may be a continuous video clip containing only the specified facial emotion, which is not limited here.

[0142] Optionally, in this embodiment, after the video clip is obtained, the video clip can be processed accordingly, such as adding filters, image modification elements or texts, etc., to generate an emoticon package with better effect.

[0143] In one embodiment of the present application, the image of the facial emotion to be recognized includes a video frame of a video file, and the facial emotion recognition method in this embodiment further includes the following steps:

[0144] Get the facial emotion corresponding to each video frame in the video file;

[0145] A facial emotion catalog is generated according to the facial emotion corresponding to each video frame and the playback time of each video frame in the video file.

[0146] In one embodiment of the present application, when processing a video file, based on the video frames in the video file, the video frames are used as images to be identified, the facial emotions in each video frame are determined, and the corresponding emotion labels are generated. The facial emotion catalog is generated based on the emotion labels of the facial emotions corresponding to each video frame and the playback time of each video frame in the video file.

[0147] In this way, after obtaining a video file, the corresponding position of each emotion type in the video file can be intuitively determined through text and other means to perform corresponding processing. Especially when video data increases dramatically every day, the facial expressions of the characters appearing in each video frame in the video can be directly analyzed without manual intervention, and stored as background information, which can be used for video editing and other projects later, greatly improving the management and processing efficiency of video files.

[0148] In one embodiment of the present application, the image of the facial emotion to be recognized includes a video frame of a video file, and the facial emotion recognition method in this embodiment further includes the following steps:

[0149] Obtain at least two video frames corresponding to the specified facial emotion from the video file;

[0150] Merge at least two video frames to obtain a clipped video corresponding to a specified facial emotion.

[0151] In one embodiment of the present application, a video frame corresponding to a specified facial emotion can be extracted from a video file, and the video frame can be processed or edited to obtain an edited video corresponding to the specified facial emotion. In this way, a specific emotion can be directly processed, thereby improving the efficiency of video editing.

[0152] Specifically, in the process of obtaining at least two video frames corresponding to a specified facial emotion from a video file in this embodiment, the target playback time corresponding to the specified facial emotion in the video file can be searched based on the facial emotion directory corresponding to the video file; based on the target playback time, the video frame corresponding to the target playback time is extracted from the video file.

[0153] Optionally, the number of video frames selected in this embodiment may be two or more, so as to perform editing, merging, and other processing based on multiple video frames to obtain a clipped video.

[0154] The following describes an embodiment of the device of the present application, which can be used to execute the method for facial emotion recognition in the above-mentioned embodiment of the present application. It can be understood that the device can be a computer program (including program code) running on a computer device, for example, the device is an application software; the device can be used to execute the corresponding steps in the method provided in the embodiment of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method for facial emotion recognition in the above-mentioned embodiment of the present application.

[0155] Fig.12 A block diagram of a device for facial emotion recognition according to an embodiment of the present application is shown.

[0156] Reference Fig.12As shown, according to an embodiment of the present application, a facial emotion recognition device 1200 includes: a detection unit 1210, used to detect a facial area contained in an image of a facial emotion to be recognized; an expression recognition unit 1220, used to recognize the facial expression type based on the facial area; an action recognition unit 1230, used to recognize the action label of a target facial organ corresponding to the expression type based on the expression type; and a determination unit 1240, used to determine the facial emotion in the image based on the expression type and the action label of the target facial organ.

[0157] In some embodiments of the present application, based on the aforementioned scheme, the device for facial emotion recognition also includes: a first acquisition unit, used to acquire the video file to be processed; a parsing unit, used to parse the video file to obtain a video frame; a filtering unit, used to filter the video frame based on the pixel information of the video frame, and use the filtered video frame as the image of the facial emotion to be identified.

[0158] In some embodiments of the present application, based on the aforementioned scheme, the filtering unit includes: a first generating unit, used to generate a histogram corresponding to the video frame based on the pixel information of the video frame; a clustering unit, used to cluster the video frames according to the histogram and the edge detection operator to obtain at least one class; a first filtering unit, used to filter repeated video frames in each class.

[0159] In some embodiments of the present application, based on the aforementioned scheme, the first filtering unit includes: a clarity unit, used to determine the clarity of the video frame based on the pixel information of the video frame; and a second filtering unit, used to filter the video frames whose clarity is less than the clarity threshold.

[0160] In some embodiments of the present application, based on the aforementioned scheme, the first filtering unit includes: a region detection unit, used to detect the face area in the video frame based on the pixel information of the video frame; a third filtering unit, used to filter the video frames that do not include the face area or include incomplete face areas.

[0161] In some embodiments of the present application, based on the aforementioned scheme, the action recognition unit 1230 includes: a target determination unit, used to determine the target facial organ corresponding to the expression type according to the correspondence between the expression type and the facial organ; and a target recognition unit, used to identify the action label of the target facial organ.

[0162] In some embodiments of the present application, based on the aforementioned scheme, the action recognition unit 1230 includes: an action label recognition unit, used to identify the action label corresponding to the facial organ in the face area; a label determination unit, used to determine the target facial organ corresponding to the expression type and the action label of the target facial organ according to the correspondence between the expression type and the facial organ.

[0163] In some embodiments of the present application, based on the aforementioned scheme, the facial organs include at least one of the following: mouth, eyes and eyebrows; the mouth action label includes at least one of the following: mouth wide open, mouth slightly open, pursed mouth or closed mouth; the eye action label includes squinting, opening eyes or closing eyes; the eyebrow action label includes frowning.

[0164] In some embodiments of the present application, based on the aforementioned scheme, the determination unit 1330 includes: a first determination unit, used to determine at least one facial organ corresponding to the expression type according to the correspondence between the expression type and the facial organ; a second determination unit, used to determine the facial emotion in the image according to the action label corresponding to at least one facial organ corresponding to the expression type, and the expression type.

[0165] In some embodiments of the present application, based on the aforementioned scheme, the image of the facial emotion to be identified includes a video frame of a video file, and the facial emotion recognition device also includes: a first recognition unit, used to identify a target image corresponding to a specified facial emotion from the video frame; a second extraction unit, used to extract a video clip containing the target image from the video file according to the position of the target image in the video file; and a second generation unit, used to generate an emoticon package corresponding to the target emotion based on the video clip.

[0166] In some embodiments of the present application, based on the aforementioned scheme, the image of facial emotions to be identified includes video frames of a video file, and the facial emotion recognition device also includes: a second acquisition unit, used to obtain the facial emotion corresponding to each video frame in the video file; a third generation unit, used to generate a facial emotion directory according to the facial emotion corresponding to each video frame and the playback time of each video frame in the video file.

[0167] In some embodiments of the present application, based on the aforementioned scheme, the image of the facial emotion to be identified includes video frames of a video file, and the facial emotion recognition device also includes: a third acquisition unit, used to obtain at least two video frames corresponding to the specified facial emotion from the video file; a merging unit, used to merge at least two video frames to obtain a clipped video corresponding to the specified facial emotion.

[0168] In some embodiments of the present application, based on the aforementioned scheme, the third acquisition unit includes: a search unit, used to search for the target playback time corresponding to the specified facial emotion in the video file from the facial emotion directory corresponding to the video file; a third extraction unit, used to extract the video frame corresponding to the target playback time from the video file based on the target playback time.

[0169] Fig.13 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0170] It should be noted that Fig.13 The computer system 1300 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0171] like Fig.13 As shown, the computer system 1300 includes a central processing unit (CPU) 1301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1302 or the program loaded from the storage part 1308 to the random access memory (RAM) 1303, such as executing the method described in the above embodiment. In RAM 1303, various programs and data required for system operation are also stored. CPU 1301, ROM 1302 and RAM 1303 are connected to each other through bus 1304. Input / output (I / O) interface 1305 is also connected to bus 1304.

[0172] The following components are connected to the I / O interface 1305: an input section 1306 including a keyboard, a mouse, etc.; an output section 1307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1308 including a hard disk, etc.; and a communication section 1309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to the I / O interface 1305 as needed. A removable medium 1311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1310 as needed so that a computer program read therefrom is installed into the storage section 1308 as needed.

[0173] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1309, and / or installed from a removable medium 1311. When the computer program is executed by a central processing unit (CPU) 1301, various functions defined in the system of the present application are executed.

[0174] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0175] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0176] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0177] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

[0178] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0179] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0180] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0181] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for facial emotion recognition, characterized in that: include: Detecting the face area contained in the image of the facial emotion to be recognized; Identify the expression type of the facial area through the trained classification model; Determining the target facial organ corresponding to the expression type according to the correspondence between the expression type and the facial organ, and identifying the action label of the target facial organ corresponding to the expression type; The deep emotion of the face in the image is determined according to the expression type and the action label of the target facial organ.

2. The method according to claim 1, characterized in that Before detecting the face region contained in the image to be identified with respect to the facial emotion, the method further comprises: Get the video file to be processed; Parsing the video file to obtain video frames; The video frame is filtered based on pixel information of the video frame, and the video frame obtained after filtering is used as the image of the facial emotion to be recognized.

3. The method according to claim 2, characterized in that Filtering the video frame based on pixel information of the video frame includes: Based on the pixel information of the video frame, generating a histogram corresponding to the video frame; Clustering the video frames according to the histogram and the edge detection operator to obtain at least one class; Filter out duplicate video frames in each of the described classes.

4. The method according to claim 2, characterized in that: Filtering the video frame based on pixel information of the video frame includes: Determining the definition of the video frame based on pixel information of the video frame; Filter the video frames whose definition is less than the definition threshold.

5. The method according to claim 2, characterized in that: Filtering the video frame based on pixel information of the video frame includes: Detecting a face region in the video frame based on pixel information of the video frame; Filter the video frames that do not include the face area or include an incomplete face area.

6. The method according to claim 1, characterized in that The method further comprises: Identify action labels corresponding to facial organs in the face region.

7. The method according to claim 6, characterized in that Identifying action labels corresponding to facial organs in the face region includes: Extracting features of the face area to obtain a face feature map; The facial feature map is respectively input into the classification model corresponding to each facial organ to obtain the action label corresponding to each facial organ.

8. The method according to claim 1, characterized in that The image of the facial emotion to be recognized includes a video frame of a video file, and the method further includes: Identify a target image corresponding to a specified facial emotion from the video frame; Extracting a video segment containing the target image from the video file according to the position of the target image in the video file; Generate an expression package corresponding to the designated facial emotion based on the video clip.

9. The method according to claim 1, characterized in that: The image of the facial emotion to be recognized includes a video frame of a video file, and the method further includes: Obtaining facial emotions corresponding to each video frame in the video file; A facial emotion catalog is generated according to the facial emotion corresponding to each video frame and the playback time of each video frame in the video file.

10. The method according to claim 1, characterized in that The image of the facial emotion to be recognized includes a video frame of a video file, and the method further includes: Acquire at least two video frames corresponding to the specified facial emotion from the video file; The at least two video frames are merged to obtain a clipped video corresponding to the specified facial emotion.

11. The method according to claim 10, characterized in that Acquiring at least two video frames corresponding to a specified facial emotion from the video file includes: From the facial emotion directory corresponding to the video file, searching for the target playback time corresponding to the specified facial emotion in the video file; Based on the target playback time, a video frame corresponding to the target playback time is extracted from the video file.

12. A device for facial emotion recognition, characterized in that: include: A detection unit, used for detecting a face region contained in an image of a face emotion to be identified; An expression recognition unit, used to recognize the expression type of the face region through a classification model obtained through training; An action recognition unit, configured to determine a target facial organ corresponding to the expression type according to a correspondence between the expression type and the facial organ, and to recognize an action label of the target facial organ corresponding to the expression type; A determination unit is used to determine the deep emotion of the face in the image according to the expression type and the action label of the target facial organ.

13. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for facial emotion recognition according to any one of claims 1 to 11 is implemented.

14. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the method for facial emotion recognition as described in any one of claims 1 to 11.

15. A computer program product, characterized in that The computer program product comprises a computer program, and the computer program is suitable for being loaded by a processor and executing the facial emotion recognition method according to any one of claims 1 to 11.

Images