Facial expression quantification and recognition method in image and its device, equipment and medium

Through the combination of convolutional neural network and classifier, the robustness of facial expression recognition and quantification in the prior art is solved, and the accurate recognition and quantification of the degree of facial expression is achieved, and the recognition rate and accuracy are improved.

CN111950481BActive Publication Date: 2024-12-03GUANGZHOU FANGGUI INFORMATION TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010826351.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-08-17
Publication Date
2024-12-03
Estimated Expiration
2040-08-17

AI Technical Summary

Technical Problem

The prior art cannot effectively identify and quantify the degree of human facial expressions, especially in light changes and complex environments, and it is not robust to realize the automatic quantification and classification of human smiling face expressions in image content.

Method used

The corrected facial image is extracted using a trained convolutional neural network, and the quantized amplitude of facial expression is calculated using a classifier. The hit probability is calculated through key feature point correction and SoftMax function, and the face is output. Quantitative amplitude of expressions.

Benefits of technology

It improves the recognition rate and quantification accuracy of facial expressions, can be suitable for the recognition and quantification of multiple facial expressions, has the versatility of different types of expressions, and can accurately express the degree of changes in facial expressions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111950481B_ABST
    Figure CN111950481B_ABST
Patent Text Reader

Abstract

The present application discloses a method for quantifying and identifying facial expressions in an image and its device, equipment and medium. The quantification method includes: detecting a facial image in an image to be identified, and correcting the facial image to a standard posture based on key feature points in the facial image; importing the corrected facial image into a trained convolutional neural network for feature extraction to obtain feature information of the facial expression contained therein; importing the extracted feature information into a classifier, and the classifier calculates the hit probability of each classification label corresponding to the feature information, and the classification label represents the classification attribute of the degree of change of the facial expression, which is defined by the classification attribute of the training sample of the convolutional neural network; according to the summary calculation of the hit probability of each classification label, the quantized amplitude of the facial expression in the facial image is output. The present application can realize accurate quantization of facial expressions in an image. And it is suitable for realizing a variety of applications based on facial expression information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer image processing, and in particular to a method for quantifying facial expressions in images and a method for recognizing facial expressions in images, and also to corresponding devices, equipment and non-volatile storage media for these methods. Background Art

[0002] With the rapid development of artificial intelligence and human-computer interaction, facial expression recognition has become an important research topic. Smiling is one of the basic human expressions and has important research and application value. In the application scenario of online video live broadcast, effective recognition of smiling expressions is becoming more and more practical. It is understandable that the recognition of other types of expressions, such as crying faces, also has this practical significance.

[0003] In the prior art, there is a smile recognition method based on key point and template matching. This method first locates the key points of facial expression, then matches them with the parameters of the smile expression feature point template trained online, and then obtains the smile value. This method solves the problem of effectively identifying smiling expressions, but it requires the construction of complex and redundant parameter templates. Although it claims to be "robust to lighting", in actual measurements, this method has poor robustness in complex environments and postures such as lighting changes and side faces.

[0004] In recent years, with the rapid development of deep learning in image algorithms, researchers have proposed a smile recognition method based on convolutional neural networks (CNN). In the master's thesis "Smile Expression Recognition Based on Deep Convolutional Neural Networks" published by Zheng Xue in 2016, in view of the complexity and one-sidedness of manually designed features, it is proposed to use CNN to extract image features, mainly to extract five key feature points, namely the center of the eyes, the tip of the nose, and the corners of the mouth, and then use the SVM classifier for smile recognition. Although this method is superior to traditional manual features in feature expression ability, the learning of features is independent of the learning of the classifier, and the learned features are not optimal for the classifier, which limits the overall accuracy of the algorithm.

[0005] From the above citations, we can know that, so far, the smile recognition method can only recognize whether a person is smiling, but cannot determine the degree of the smile, for example, it cannot determine the three different states of not smiling, smiling, and laughing. In short, the existing technology cannot automatically quantify human smiling expressions, and thus cannot effectively classify human smiling expressions in image content. Summary of the invention

[0006] The primary purpose of the present application is to provide a method for quantifying facial expressions in an image and its corresponding apparatus, device and non-volatile storage medium, so as to effectively quantify the facial expressions contained in a facial image.

[0007] As another object of the present application, a method for recognizing facial expressions in images and its corresponding devices, equipment and non-volatile storage medium are provided, so as to realize more in-depth technical applications by using the results obtained by the previous object.

[0008] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0009] A method for quantifying facial expressions in an image proposed for the purpose of this application comprises the following steps:

[0010] Detecting a facial image in an image to be recognized, and correcting the facial image to a standard posture based on key feature points in the facial image;

[0011] Importing the corrected facial image into a trained convolutional neural network for feature extraction to obtain feature information of the facial expression contained therein;

[0012] Importing the extracted feature information into a classifier, the classifier calculates the hit probability of each classification label corresponding to the feature information, wherein the classification label represents the classification attribute of the degree of change of the facial expression, which is defined by the classification attribute of the training sample of the convolutional neural network;

[0013] The hit probabilities of each classification label are summarized and calculated, and the quantized amplitude of the facial expression in the facial image is output.

[0014] In some embodiments, the key feature points include the center points of the left and right eyes, the tip of the nose, and the left and right corners of the mouth in the facial image. When implementing the facial image correction, the similarity transformation matrix is ​​calculated between the key feature points and the corresponding feature points of the standard face model, and then the similarity transformation matrix is ​​applied to the facial image to correct it to a standard posture.

[0015] In some embodiments, the convolutional neural network is a residual network ResNet, and includes at least 17 convolutional layers and 1 fully connected layer.

[0016] In a preferred embodiment, the classifier pre-matches a corresponding label coefficient for each classification label, and each label coefficient numerically expresses the relative strength relationship between the degrees of change. When the quantized amplitude of the facial expression in the facial image is calculated and output based on the hit probability of each classification label, the product of the hit probability of each classification label and its corresponding label coefficient is accumulated and summed as the quantized amplitude.

[0017] In a further embodiment, the sum is multiplied by an adjustment coefficient as the quantization amplitude, so that the value range of the quantization amplitude is constrained within a specific target value interval.

[0018] In some embodiments, the classifier is constructed by a SoftMax function, and the function is executed to calculate the hit probability.

[0019] In some embodiments, the convolutional neural network performs data smoothing preprocessing on the classification of training samples based on one-hot encoding during the sample training stage.

[0020] In some embodiments, the facial images included in the training data set of the convolutional neural network have human smiling or crying face features.

[0021] In a further embodiment, the method is implemented as a program interface that runs on a computer and can be called. When called, it receives the image to be identified from the real-time video stream during the live broadcast process and executes various steps, and finally returns the quantized amplitude in response to the call.

[0022] In a further embodiment, the returned quantized amplitude is used by the caller to control the progress of one or more virtual activities performed based on the real-time video stream, or is used by the caller to output to a graphical user interface of a terminal device that generates the real-time video stream.

[0023] A method for recognizing facial expressions in an image proposed for the purpose of this application comprises the following steps:

[0024] Monitor the image frames in the real-time video stream of the live broadcast process, call a preset program interface, and use the image frames as the to-be-recognized images to be imported by the program interface, so as to track and analyze the image frames of the real-time video stream through the program interface and continuously obtain the quantized amplitude returned by the program interface;

[0025] In parallel with the monitoring, playing a pre-provided target video;

[0026] Following the playback of the target video, detecting whether the quantized amplitude meets a preset condition, and executing a computer instruction corresponding to whether the preset condition is met,

[0027] When the program interface is called, the following steps are performed:

[0028] Detecting a facial image in an image to be recognized, and correcting the facial image to a standard posture based on key feature points in the facial image;

[0029] Importing the corrected facial image into a trained convolutional neural network for feature extraction to obtain feature information of the facial expression contained therein;

[0030] Importing the extracted feature information into a classifier, the classifier calculates the hit probability of each classification label corresponding to the feature information, wherein the classification label represents the classification attribute of the degree of change of the facial expression, which is defined by the classification attribute of the training sample of the convolutional neural network;

[0031] The hit probabilities of each classification label are summarized and calculated, and the quantized amplitude of the facial expression in the facial image is output.

[0032] In a preferred embodiment, during the execution of the method, the quantized amplitude or text or graphic symbols corresponding thereto are synchronously output in a graphical user interface where the playback window of the real-time video stream is located.

[0033] A facial expression quantification device in an image proposed for the purpose of this application comprises:

[0034] A detection and correction unit, used for detecting a facial image in an image to be recognized, and correcting the facial image to a standard posture based on key feature points in the facial image;

[0035] A feature extraction unit, used for importing the corrected facial image into a trained convolutional neural network for feature extraction to obtain feature information of the facial expression contained therein;

[0036] A classification calculation unit, which imports the extracted feature information into a classifier, and the classifier calculates the hit probability of each classification label corresponding to the feature information, wherein the classification label represents the classification attribute of the degree of change of the facial expression, and is defined by the classification attribute of the training sample of the convolutional neural network;

[0037] The quantization output unit calculates and summarizes the hit probabilities of various classification labels and outputs the quantized amplitude of the facial expression in the facial image.

[0038] An electronic device proposed to meet the purpose of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the method for quantifying facial expressions in an image, or to execute the steps of the method for recognizing facial expressions in an image.

[0039] A non-volatile storage medium is proposed to meet the purpose of the present application, which stores a computer program implemented according to the method for quantifying facial expressions in images, or stores a computer program implemented according to the method for recognizing facial expressions in images. When the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

[0040] Compared with the prior art, the advantages of this application are as follows:

[0041] First, the present application uses a trained convolutional neural network to extract features from the corrected image to be identified, and then uses a classifier to classify the feature information to obtain the hit probability of each classification label, and then summarizes each hit overview to finally determine the quantized amplitude of the facial expression contained in the facial image in the image to be identified. Since each classification label carries the degree of change of the facial expression, the quantized amplitude obtained by summarizing the degree of change can effectively characterize the different degrees of the same type of facial expression, and can significantly improve the recognition rate and quantization accuracy of a specific type of facial expression.

[0042] Secondly, the present application establishes an association between the control effect of the training data of the convolutional neural network and the image to be recognized by expressing classification labels of different degrees of facial expressions. That is to say, the setting of the classification label depends on the selection of the training data. The classification label to which the training data belongs will be used by the classifier when recognizing the image. Therefore, the method of the present application can be applied to the recognition and quantification of a variety of different specific types of facial expressions. For example, when it is necessary to recognize the quantization amplitude of a smiling face, the control training data is mainly a smiling face image; when it is necessary to recognize the quantization amplitude of a crying face, the control training data is mainly a crying face image. It can be seen that the solution of the present application has the versatility to adapt to the quantization of different types of facial expressions.

[0043] In addition, as a concrete means, the present application, in one embodiment, allows the classification labels to adapt to different label coefficients, and allows the label coefficients to numerically express the degree of change of facial expressions, so that different classification labels can identify the relative strength relationship between each other through their label coefficients, and adapt to the gradual changes in the same human expression, such as the classification changes from not smiling to smiling to laughing. Therefore, the calculated quantization amplitude can further have the meaning of accurately expressing facial expressions.

[0044] Additional aspects and advantages of the present application will be partially given in the following description, which will become apparent from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0046] Figure 1 A schematic diagram of a typical network deployment architecture related to implementing the technical solution of this application;

[0047] Figure 2 A flowchart of a typical embodiment of the method for quantifying facial expressions in an image of the present application;

[0048] Figure 3A schematic diagram of an algorithm flow implemented by the facial expression quantification method in an image of the present application;

[0049] Figure 4 A flowchart of a typical embodiment of the method for recognizing facial expressions in images of the present application;

[0050] Figure 5 and Figure 6 Screenshots of a graphical user interface in a terminal device presented after the application implemented by the method for recognizing facial expressions in images of the present application is run, respectively presenting screen information at different moments during the virtual activity;

[0051] Figure 7 It is a principle block diagram of a typical embodiment of the apparatus for quantifying facial expressions in images of the present application;

[0052] Figure 8 This is a principle block diagram of a typical embodiment of the facial expression recognition device in an image of the present application. DETAILED DESCRIPTION

[0053] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0054] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0055] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0056] It will be understood by those skilled in the art that the "client", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; conventional laptop and / or palmtop computers or other devices, which have and / or include a conventional laptop and / or palmtop computer or other device with and / or including a radio frequency receiver. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable for and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.

[0057] The hardware referred to by the names such as "server", "client", and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0058] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.

[0059] See also Figure 1 , the hardware foundation required for the implementation of the relevant technical solutions of this application can be deployed according to the architecture shown in the figure. The server 80 referred to in this application is deployed in the cloud. As a front-end application server, it can be responsible for further connecting relevant data servers, video streaming servers and other servers that provide relevant support, etc., so as to form a logically associated service cluster to provide services for relevant terminal devices such as the smartphone 81 and personal computer 82 shown in the figure. The smartphone and personal computer can both access the Internet through a well-known network access method, and establish a data communication link with the server 80 in the cloud, so as to run the terminal application related to the service provided by the server. In the relevant technical solutions of this application, the server 80 can be responsible for establishing a short video push service, and the terminal can run the application corresponding to the short video push service. Similarly, the server 80 can also be responsible for implementing services related to facial expression quantification.

[0060] The application described herein refers to an application running on a server or terminal device. This application implements the relevant technical solutions of the present application in a programming manner. Its program code can be stored in a non-volatile storage medium recognizable by a computer in the form of computer executable instructions, and can be loaded into the memory by the central processing unit for execution. The relevant device of the present application is constructed by the operation of the application on the computer.

[0061] For the server, the application is usually constructed as a service process, opening the corresponding program interface for remote calls by applications running on various terminal devices. The relevant technical solutions suitable for running on the server in this application can be implemented in the server in this way.

[0062] For various types of terminal devices that are popular nowadays, especially mobile devices such as tablets and mobile phones, they are usually equipped with built-in imaging devices such as cameras, or personal computers can also be externally connected to the imaging devices. In theory, the application of the terminal device of this application can call the imaging devices in these cases.

[0063] The terminal device equipped with the camera device usually has a corresponding background driver and a front-end camera program built in. The driver is used to drive the camera to obtain video images at the bottom of the system, and the camera program is mainly used to provide a human-computer interaction interface. By calling the driver, its camera function can be realized. Therefore, the method suitable for implementation in the terminal device in this application can be implemented in the camera program, so as to realize the drainage path of expanding the access source of the short video push service platform by using the camera program.

[0064] The technical solution in this application suitable for implementation in terminal devices can also be programmed and built into an application that provides live broadcasting as part of its extended functions. The live broadcasting refers to a live broadcast room network service implemented based on the aforementioned network deployment architecture.

[0065] The live broadcast room referred to in this application refers to a video chat room implemented by Internet technology, which usually has audio and video broadcast control functions, including anchor users and audience users. Audience users can include registered users who have registered on the platform, or unregistered visitor users; they can be registered users who follow anchor users, or registered or unregistered users who do not follow anchor users. The anchor user and the audience user can interact through well-known online interactive methods such as voice, video, and text. Generally, the anchor user performs programs for the audience users in the form of audio and video streams, and economic transactions can also be generated during the interaction. Of course, the application form of the live broadcast room is not limited to online entertainment, but can also be extended to other related scenarios, such as education and training scenarios, video conferencing scenarios, product promotion and sales scenarios, and any other scenarios that require similar interactions.

[0066] The application program running on the terminal device of the present application can generate the behavior of distributing electronic gifts according to the needs of the design of its platform. The electronic gifts referred to in the present application, also known as electronic gifts, are non-physical, electronic marks representing certain tangible or intangible values. The implementation form of this mark is extensive and flexible, and is usually presented to users in a visual form, such as in the form of icons, quantity, and value. Electronic gifts usually require users to purchase and consume, or they can be gifts provided by the Internet service platform. However, once an electronic gift is generated, it can support exchange with real securities or be a non-exchangeable item, depending on the technical implementation of the Internet service platform, which does not essentially affect the implementation of the present application. Correspondingly, the behavior of users purchasing electronic gifts constitutes the behavior of users consuming electronic gifts. The behavior of distributing electronic gifts is manifested at the program level and will trigger corresponding consumption events. Electronic gifts can be purchased by users in the application, and the events triggered by the behavior of purchasing electronic gifts can also be regarded as consumption events. When the platform unconditionally and proactively distributes electronic gifts, it will also send users information about the relevant rewards they have received. Therefore, when users receive reward information, it often means that they have received a certain electronic gift from the platform.

[0067] The application program running on the terminal device of the present application can trigger the behavior of playing special effects according to the needs of its platform design. The special effect is a computer animation display effect or a technical control effect achieved by a similar method, which is usually used to enhance the interactive perception atmosphere in the graphical user interface of the application. When the special effect is triggered to play, the user interface can see the corresponding animation playback effect, thereby perceiving the special effect. There are various forms of implementation of special effects, which can be flexibly implemented by those skilled in the art.

[0068] The application running on the terminal device of this application can open the permission of further clearance links for users according to the needs of its platform design, which generally occurs when the application is implemented in the scenario of clearance game. In this case, after the user successfully meets certain preset conditions in the game link of the previous level, the application will open the access permission of the next level game link for him, so that the user can enter the next level of the game and continue the game.

[0069] The application program running on the terminal device of the present application can change the user's identity characteristic parameters according to the needs of the platform design. Usually, the user of the application will be a registered user of the platform and is often in the state of account login. The user usually has at least one identity characteristic parameter, for example, when he is a host user, the number of user votes he usually performs symbolizes the evaluation value of a certain dimension of the host user. The live broadcast platform can set the relative position of the host user among some or all other host users based on the identity characteristic parameter, for example, according to its specific evaluation value, it identifies a corresponding level of grade label for it. The acquisition of such identity characteristic parameters can be collected or converted by the platform according to its own needs by designing various activities, such as quantitatively determining according to any similar trigger factors such as the number of users followed by the host user, the user activity during live broadcast, and the profitability when holding live broadcast room activities. Such identity characteristic parameters can then be used by the live broadcast platform for secondary data mining, and more complex platform functions can be realized based on these identity characteristic parameters, and various activities in the live broadcast room of the host user can be served when necessary. Of course, in accordance with the technical solution of the present application, these identity feature parameters can also be expressed as, for example, a "personality value". By converting the quantized amplitude of the user's expression into the "personality value" and increasing or decreasing it accordingly, it is obvious that this can also characterize the identity of the user to some extent.

[0070] The application program of the present application running on the terminal device can trigger a pop-up sharing operation interface according to the needs of the platform design. This interface usually also contains relevant information for promotion. Users can share the relevant information to a third-party service platform or application program through further operations on the interface, or share it to relevant users, friends, comment areas, etc. on the same platform, so that more users can access the functions related to the method of the present application through the shared information, thereby realizing the fission promotion of the application program of the present application.

[0071] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.

[0072] In a typical embodiment of a facial expression quantification method in an image of the present application, it is implemented as a computer program running on a server, opening a program interface for external calls, and the program interface can be called by an application program of a server or a terminal device. Figure 2 and Figure 3 In this typical embodiment, the method specifically includes the following steps:

[0073] Step S11: Detect a facial image in the image to be recognized, and correct the facial image to a standard posture based on key feature points in the facial image:

[0074] According to business needs, the image to be identified can be passed in by the caller of this method, or an image frame in a real-time video stream such as generated by a live broadcast can be directly used as the image to be identified. Through the identification means mastered by those skilled in the art, a facial image with facial features can be detected from the image to be identified.

[0075] See also Figure 3 , considering that the facial image presents different angles due to different human body activity postures, the facial image can be corrected in advance to a standard posture. In this embodiment, the facial image is corrected using key feature points on the face. It detects the left eye center point, the right eye center point, the nose tip point, the left mouth corner point, and the right mouth corner point in the facial image, a total of five key feature points, and calculates the similarity transformation matrix with the five corresponding feature points of the standard face model, and then applies the similarity transformation matrix to the image, so that the face is corrected to the standard posture. The algorithm for realizing facial image posture correction is not limited to this. For example, a facial posture correction method based on sine transformation can also be used in a similar way. Those skilled in the art should know that any known means suitable for correcting the face in the facial image from a crooked posture to a standard posture can be used to replace the algorithm based on key feature points in this embodiment, so as to realize the correction of the standard posture, and it should not be regarded as transcending the creative spirit of this application.

[0076] Step S12: Import the corrected facial image into the trained convolutional neural network for feature extraction to obtain feature information of the facial expression contained therein:

[0077] The convolutional neural network is trained in advance, and the materials required for training are usually a large number of training samples, which are usually also facial images, and usually, when selecting materials, facial images containing a certain type of expression are selected as training samples. This specific type of expression can be a smiling face or a crying face, depending on the ability of the convolutional neural network to be trained. Of course, the corresponding training sample image is also required to contain smiling face features or crying face features. For example, in this embodiment, the specific type of expression is a smiling face. To this end, the technician can provide the convolutional neural network with training samples of three different classification attributes (which can be represented as classification labels) such as "not smiling", "smiling" and "laughing" in advance according to subjective wishes. After the convolutional neural network is trained with these massive training samples, it will have the ability to extract facial features from the image and distinguish these three different classification attributes, so that it can identify what classification attributes the expression has from the imported facial image.

[0078] In another embodiment optimized on the basis of this embodiment, considering that the classification boundaries between various classification attributes themselves have fuzzy spaces, it is inevitable that there are errors in the training samples. Therefore, in the stage of training samples, the convolutional neural network performs data smoothing preprocessing on the classification of training samples based on one-hot encoding. Specifically, the label n is the total number of classification attributes, that is, the total number of classifications. For the input sample of the cth classification attribute, the traditional one-hot type label is: (l0,...l i ,...l n ), if and only if i = c, l i 1, otherwise l i 0. In this embodiment of the present application, the one-hot type label is further smoothed, that is, l i =λ,l i-1 =(1-λ) / 2,l i+1 =(1-λ) / 2, where λ=0.9 is recommended. According to actual tests, this improvement can effectively suppress overfitting when calculating the loss value, making the training effect better. Since it can effectively suppress overfitting, in subsequent applications, it can relatively effectively overcome the defects caused by illumination in the image, making the recognition process more robust.

[0079] The convolutional neural network used in this application can preferably be a residual network ResNet, which is required to have 17 convolutional layers and 1 fully connected layer, usually called ResNet-18. This network is preferred. After actual testing, it can support the implementation of this application at a more reasonable cost. However, in order to cope with higher requirements for the implementation effect, networks with more convolutional layers and other known networks that can achieve the same or higher effect can also be used. Those skilled in the art should be able to flexibly play their role based on the disclosure herein.

[0080] See also Figure 3 After the corrected facial image is introduced into the trained convolutional neural network disclosed in the present application, for example, ResNet-18 is responsible for performing feature extraction on it, thereby obtaining feature information about the facial expression contained in the facial image. Similarly, this feature information is computer-readable, but not necessarily human-readable.

[0081] Step S13, importing the extracted feature information into a classifier, the classifier calculates the hit probability of each classification label corresponding to the feature information, and the classification label represents the classification attribute of the degree of change of facial expression, which is defined by the classification attribute of the training sample of the convolutional neural network:

[0082] The feature information extracted by the convolutional neural network is further used by the classifier to classify facial images.

[0083] In this typical embodiment, since the training of the convolutional neural network is performed on the specific expression of "smile", the three different classification attributes of "not smiling", "smile" and "laugh" actually also play the role of characterizing the different degrees of change of the characteristic expression of "smile", and it can be understood that in terms of the amount of "smile", the former is smaller than the latter, which is the basis for the present application to quantify the different degrees of change of specific facial expressions.

[0084] The classification attributes targeted by the convolutional neural network actually also play the role of classification labels, that is, the classification labels are essentially defined naturally or artificially by the convolutional neural network when training data. Although the application uses the specific expression of "smile" and its classification descriptions of different degrees to illustrate, it should also be understood that this classification label is mainly machine-readable, which can be explicit or implicit, and does not require it to have the daily plain text expression of humans, such as a, b, c or 0, 1, 2 (see Figure 3 ) to express these classification labels, or simply set different counting variables corresponding to each classification label. Therefore, those skilled in the art should understand that the introduction of "classification labels" in this application is to facilitate logical understanding, and should not be used to limit the scope of implementation of this application.

[0085] In this embodiment, the classifier is constructed by a SoftMax function, and the function is executed to realize the calculation of the hit probability. The classifier uses the feature information as a parameter, corresponds to each classification label, and calculates the hit probability of the feature information hitting each classification label, so as to further realize subsequent quantification.

[0086] It should be pointed out that, considering that different expression attributes are not either-or, this application does not recommend the use of a binary classifier to implement classification, which is also the advantage of this embodiment over the prior art. However, those skilled in the art should still understand that any related known or unknown algorithm that can achieve the same classification effect as the SoftMax function can achieve the purpose and effect of the invention of this application by replacing the SoftMax in this embodiment, and constitute one of the embodiments of this application, and should not be excluded from the inventive spirit of this application.

[0087] Step S14: According to the hit probability of each classification label, the quantized amplitude of the facial expression in the facial image is output:

[0088] In this typical embodiment, the corresponding hit probabilities of the various classification labels obtained in the previous step can be directly accumulated and summarized to obtain the quantized amplitude, which is the simplest way.

[0089] In an embodiment that is analyzed in depth based on the present embodiment, more detailed technical details are given by taking into account the classification attributes of specific expressions: As mentioned above, there is a quantitative change relationship between the classification attributes corresponding to each classification label. More specifically, for example, the classification attribute of "laughing" can be regarded as a "laugh" with a stronger degree than "smiling". Similarly, "smiling" is also a "laugh" with a stronger degree than "not smiling". Based on this relationship presented by the degree of change, the classifier of the present application matches the corresponding label coefficient for each classification label to realize the configuration of the weight of this degree of change. For example, according to the relationship corresponding to the three classification attributes of "not smiling", "smiling" and "laughing", the three classification labels are respectively equipped with three label coefficients of 0, 1 and 2. As a result, the three label coefficients can play a role in adjusting the weights in the calculation process, thereby more accurately describing the final quantization amplitude.

[0090] For an example of this embodiment, see Figure 3 shown. Figure 3 It can be seen that after the facial image is input, it is detected and corrected and imported into the convolutional neural network to realize feature extraction. The classifier predicts the hit probability corresponding to each classification label. Each classification label is assigned three different label coefficients of 0, 1, and 2 respectively. Each label coefficient is multiplied one by one with the hit probability of each classification label, and three product calculation results of 0.0, 0.1, and 1.6 are obtained accordingly. Then these three products are accumulated to obtain the final sum value as the quantization amplitude of 1.7.

[0091] Theoretically, the quantized amplitudes obtained above can be directly output as the result. However, in another embodiment of the present application, in order to make the result of the quantized amplitude more readable, according to people's reading habits, the above sum is multiplied by an adjustment coefficient before being used as the final output quantized amplitude, thereby constraining the numerical range of the quantized amplitude to a target numerical range, such as 0-100.

[0092] After the above process, the function of identifying a specific type of expression from the image to be identified and outputting the corresponding quantized amplitude of the expression is completed.

[0093] In order to make the technical solutions of the above-mentioned embodiments more convenient to call, in one embodiment, the above-mentioned embodiments can be implemented as an application program running on a computer (generally a server, and can also be set as an SDK of a terminal device), and the program interface that can be called through the operation of such an application program, when it is called, receives the image frame taken from the real-time video stream during the live broadcast of the network (such as a live broadcast room) as the image to be identified and executes the various steps of the embodiment, and finally responds to the call and returns the quantized amplitude. The returned quantized amplitude is used by the caller to control the progress of one or more virtual activities performed based on the real-time video stream (such as the virtual activities initiated by the anchor user in the live broadcast room), or is used by the caller to output to the graphical user interface of the terminal device that generates the real-time video stream. The caller referred to here can be either a terminal device or other server, depending on the actual development needs of the network platform.

[0094] Based on the aforementioned method provided in this application, as a different aspect of the same invention, this application also provides a method for recognizing facial expressions in an image, please refer to Figures 4 to 6 , which is revealed more vividly through a virtual activity. Figure 4 As shown, the method of this embodiment includes the following steps:

[0095] Step S21, monitoring the image frames in the real-time video stream of the live broadcast process, calling the program interface implemented by the facial expression quantization method in the image, using the image frames as the image to be identified to be imported by the program interface, so as to track and analyze the image frames of the real-time video stream through the program interface and continuously obtain the quantization amplitude returned by the program interface:

[0096] When the application program implementing the method is started and each step of the method is executed, the camera device of the local terminal device is first started and then the video image is acquired in real time through the camera device.

[0097] In the scenario where the present method is used, the user is usually the user of the terminal device, such as a handheld terminal device such as a mobile phone or a tablet. Such a device usually has a front camera and a rear camera, or has a camera device that realizes the functions of the front camera and the rear camera by switching the orientation state. Therefore, the camera device turned on here can be turned on by default, or the camera device can be placed in an orientation state suitable for playing the role of the front camera, so as to mainly achieve the purpose of obtaining the video image of the user of the terminal device. Of course, in some special occasions, such as the scene of the split-screen display of the graphical user interface output by the machine, the rear camera of the terminal device can also be used instead of the front camera. It can be understood that the selection of the front camera and the rear camera is determined according to the specific usage situation, and the use of capturing facial expressions interacting with the application should not be the key to affecting the creative spirit of this application.

[0098] As one of the commonly implemented functions of this application, see Figure 5 , the video images captured by the camera device will be displayed in the current graphical user interface of the local device, so that the user of the local terminal device can intuitively know his or her real-time expression state. The video images captured by the camera device can be displayed in the graphical user interface in the form of a half window or a full screen; they can be displayed in the form of a semi-transparent layer or a non-transparent layer, which does not affect the implementation of this application.

[0099] Those skilled in the art are familiar with algorithms for identifying specific types of facial expression images from the video images, including various algorithms obtained by machine learning, and various algorithms based on AI technology, which can all achieve the function and effect of identifying specific facial expression images from video images obtained by camera equipment. In some cases, the relevant algorithms for achieving this function and effect can be programmed into a relevant SDK, which can be subsequently used by developers through method calls. Alternatively, developers can also specifically program known algorithms that achieve equivalent functions and effects in the application of this method. Those skilled in the art should know this and so on. However, in this application, it is hereby clarified that the algorithm implemented by the aforementioned method for quantifying facial expressions in images will be used to implement the functions to be achieved by the method for recognizing facial expressions in images, so as to achieve further application of the previous method.

[0100] Since the quantization method requires external provision of the image to be identified, in the present application, the real-time video stream can be captured by the camera device, and image frames can be extracted from the real-time video stream as the image to be identified, so that the program interface implemented by the previous method can perform expression recognition based on the image frames of the real-time video stream, and finally output the corresponding quantization amplitude.

[0101] It is difficult for the user's facial expression to remain static, especially when the application of the present method plays the target video for the user to watch in order to provide emotional stimulation to the user. Therefore, in this step, it is monitored whether the quantitative amplitude of the user's facial expression remains unchanged or changes, that is, the changing state of the specific expression contained in the facial image is monitored.

[0102] As mentioned above, a video image is composed of image frames, such as 60 frames / second or 25 frames / second. In any case, in order to call the program interface implemented by the previous method, the image frames must be collected for recognition.

[0103] In this typical embodiment, the program interface can be called for each frame to perform the recognition, so that the recognition and quantification of the user's facial expression can be accurate to every subtle expression activity. In other embodiments, considering that the user's facial expression is allowed to change within a certain range, a method such as interval sampling can also be adopted to call the program interface for recognition of non-adjacent image frames. Regardless of the implementation method, it does not affect the creative spirit of this application.

[0104] In an improved embodiment, in order to avoid the uneven nature of the change in the quantization amplitude due to the roughness of the algorithm, when detecting the quantization amplitude, the average value of the quantization amplitude within a fixed time length, such as 1 second, is taken, and the averaged quantization amplitude is provided to the subsequent steps for detecting whether it meets the preset conditions.

[0105] It should be noted that the activity of tracking and analyzing the image frames of the real-time video stream mentioned in the present application refers to the activity implemented based on synchronization with the generation of the image frames, and its purpose is to make the quantization amplitude obtained by calling the program interface reflect the content of the image frame correctly and timely as much as possible. However, it should be understood that such tracking may also be non-real-time, for example, there may be an appropriate time difference, and such time difference is suitable to not cause the user to have a significant perception of the difference between the quantization amplitude and his or her true emotions.

[0106] Step S22, in parallel with the monitoring, playing the pre-provided target video:

[0107] In this typical embodiment, in order to achieve the parallelism of the functions executed by this step and the previous step, and to realize listening and playing the target video at the same time, this step can be implemented as an independent thread. Thus, while the listening action is implemented in the previous step, the target video preset for the virtual activity can be played "synchronously".

[0108] The target video is usually prepared in advance to match the design intention of the program developer, or stored in a remote server or in the terminal device, which does not affect the implementation of the present application.

[0109] A logical association is established between the target video and the facial expressions of the users participating in the activity according to the developer’s design intention. Figure 5 As for the design intention shown, which is intended to stimulate the user (user) to laugh or control sad emotions by watching the target video, the target video preferably contains some comedy clips or other humorous video content for making people laugh; for the design intention that is intended to train the user's facial expression ability by watching the target video, the target video can preferably contain clips with various effects, including comedy clips, tragedy clips, etc. It should be noted that this correlation between the target video and the specific facial expression belongs to the logical category, and can be flexibly matched by those skilled in the art according to the program development intention and the examples here, and the matching relationship between the two is not one-to-one, but can be varied, mainly depending on the program development intention. It can be understood that how the specific content of the target video and the facial expression are matched specifically does not affect the realization of the creative spirit of this application.

[0110] In this typical embodiment, the target video can be pushed by the remote server. Specifically, the application of this method can submit a request to obtain the target video to the remote server in accordance with the design intention of the application and the specifications of the developer when programming. In response to the request, the remote server selects one or more target videos from the preset video library and pushes them to the terminal device. The application of the terminal device can then open a play window and play the target video.

[0111] In an improved embodiment, the current graphical user interface of the terminal device allows the target video to be played in full screen. At the same time, the video image obtained in real time by the camera device is not displayed in the current graphical user interface, but the application still monitors and detects the quantization amplitude in the background, and displays an expression icon displayed in an animated form in the current graphical user interface, so that the animation change state of the expression image is associated with the quantization amplitude returned in the step S21, so that the expression image corresponds to the change of the user's facial expression according to the quantization amplitude. As a result, the user of the terminal device can perceive the degree of change of his or her own facial expression through the expression icon, even if the user cannot watch his or her own video image in real time at this time.

[0112] In another improved embodiment, considering the need for monitoring in the previous step and the need for playing in this step, in order to improve the user experience and increase the interactive perception, the graphical user interface of the terminal device can be divided into two upper and lower half windows for displaying the video image captured in real time by the camera device and playing the target video. Similarly, the emoticon icon described in the previous embodiment can also be implemented in the graphical user interface, so that the video image, emoticon icon and target video are loaded and displayed in the current graphical user interface at the same time.

[0113] It can be further understood from the above disclosure that this step can be executed in parallel with the previous step S21 of implementing monitoring in the memory of the terminal device.

[0114] Step S23, following the playback of the target video, detecting whether the quantized amplitude meets a preset condition, and executing a computer instruction corresponding to whether the preset condition is met:

[0115] This step can be implemented by the main process or other parallel processes of the application of the present method. When the graphical user interface of the terminal device is playing the target video, and the terminal device is simultaneously outputting the image frames of the real-time video stream acquired by the camera device and calling the program interface to acquire the corresponding quantized amplitude, a process of the application of the present application can be responsible for executing the function of this step in parallel.

[0116] Specifically, during the target video playback, step S21 continuously outputs the quantized amplitude in response to the actual facial expression activity of the user, so this step tracks each output quantized amplitude to detect whether it meets the preset condition. This preset condition is given in advance according to the program design intention, and is usually specified in the code of the application, or in one or more rule files or data tables for the application to call.

[0117] Whether the quantization amplitude satisfies the preset conditions is detected. Depending on the different preset conditions, a variety of implementation methods corresponding to the preset conditions are adopted. The following discloses several types of detection methods determined based on different program design intentions:

[0118] In the first type of detection method, the preset condition is configured to include a numerical interval corresponding to the quantized amplitude, and by detecting whether the obtained quantized amplitude exceeds or belongs to this numerical interval, it is determined whether the actual facial expression change of the current user meets the preset condition. Here, the two optional cases of "exceeding" and "belonging" are included, which are suitable for corresponding to different program design intentions.

[0119] For example, for the case of "belonging to", the design intention of the application of this application is to stimulate the user to laugh through the playback of the target video. The betting rules between it and the user are set as the user must control the smile within a certain range of variation. It can be simply and vividly understood that if the user holds back his smile, his challenge is successful, and if he laughs or does not laugh, his challenge fails. Therefore, the numerical interval is set to (30, 60). If the quantized amplitude representing the facial expression of the user is between 30 and 60, and persists until the end of the target video playback, the user's challenge is deemed to be successful, constituting the satisfaction of the preset conditions; otherwise, if the quantized amplitude appears at any time during the playback of the target video is lower than 30 or higher than 60, the terminal device user's challenge fails. In this case, if the callback function related to monitoring returns a quantized amplitude outside the range of 30 to 60, it is determined that the preset conditions are not met.

[0120] For another example, the situation of "exceeding" is similar to the former, except that the way of using the numerical range for comparison is different. It is suitable for situations such as motivating users to achieve large emotional fluctuations. For example, only when the user continues to laugh (the quantization amplitude is higher than 60) or keeps not laughing (the quantization amplitude is lower than 30), the preset condition is met. If the user's facial expression does not change and is in the numerical range of 30 to 60, the preset condition is not met.

[0121] In the second type of detection method, the detection conditions can be further simplified. The difference between its technical means and the previous type is that it replaces the aforementioned numerical range by setting a threshold value. For example, the threshold value is set to 50 corresponding to the specific facial expression of a smile. For this case of setting a threshold value, depending on whether the design intention of the programmer is to pursue the user's laughter or not, the preset condition is configured accordingly. When the quantized amplitudes obtained are greater than the threshold value or less than the threshold value during the entire target video playback until the end, the preset condition is considered to be met. Otherwise, at any time during the target video playback process, as long as the quantized amplitude corresponding to the facial expression detected from one or several frames of images is less than or greater than the threshold value, it is considered that the preset condition is not met.

[0122] It can be understood that based on the above two types of detection methods, a variety of preset conditions can be flexibly designed for different program design purposes and intentions. Those skilled in the art can flexibly configure the preset conditions according to the disclosure of the application principle of the preset conditions herein and flexibly implement them in the program development process.

[0123] In summary, in a typical application of the present application, during the playback of the target video, the quantization amplitude is continuously detected to see whether it meets the preset conditions. If all the quantization amplitudes meet the preset conditions until the end of the playback of the target video, it can be regarded as constituting the generation conditions of the expression instructions for triggering the default and pre-associated events; if the quantization amplitude does not meet the preset conditions at any time during the playback of the target video, other processing can be performed, such as generating another expression instruction for triggering a different pre-associated event, or even terminating the playback of the target video.

[0124] According to the technical means of triggering a pre-associated event when a preset condition is met, it can be understood that meeting the preset condition (or even not meeting the preset condition) is understood as a trigger condition of a pre-set computer event in the method of the present application, and can therefore be understood as a computer instruction in a broad sense, that is, the "expression instruction" referred to in the present application. In other words, the expression instruction described in the present application refers to the result information generated by the processing of the aforementioned process based on whether the quantized amplitude meets the preset condition. The result information is parsed as an expression instruction by the application of the present application, which can be parsed by the method of the present application and trigger a pre-associated computer event accordingly. These pre-associated events can theoretically be any type of computer event that can be run by computer code, such as playing preset special effects, sending reward information to the local user, opening the authority of further clearance links, changing the identity feature parameters of the local user, popping up a sharing operation interface... and so on.

[0125] As for the two results generated by whether the quantized amplitude meets the preset conditions, although when the preset conditions are met, it can be regarded as constituting the generation conditions of the expression instruction and triggering the pre-associated event, which is called the first event, however, when the preset conditions are not met, it can also be regarded as constituting the generation conditions of another expression instruction and triggering another pre-associated event, namely the second event. It can be understood that as long as the two results of meeting and not meeting the preset conditions are preset by the developer during programming to establish the association between the preset different events, then, whether the preset conditions are met or not, this determination result can be used to make the terminal device execute the next step, thereby presenting the execution result of the corresponding event.

[0126] In a typical application scenario, according to the program design intention, the user holds back laughter during the whole process of watching the target video, and the corresponding quantized amplitude is always controlled within a numerical range until the target video ends. This is considered to meet the preset conditions, and the user's challenge to hold back laughter with the application is successful. Then, the application obtains the corresponding result information, regards this result as the generation condition of the expression instruction, and issues the expression instruction. According to the expression instruction, the pre-associated first event is triggered accordingly. The first event first outputs a notification message of "challenge successful" to the graphical user interface of the terminal device, such as Figure 6 As shown, the preset electronic gifts are then distributed to the logged-in user of the application and the added electronic gifts are associated with the personal account of the logged-in user. On the other hand, in the second round of the challenge, the user laughs because of the plot of the target video in the middle process, so that the corresponding quantized amplitude of the facial expression exceeds the numerical range, resulting in the user's facial expression not meeting the preset conditions. Therefore, the application obtains the corresponding result information, and similarly regards this result information as an expression instruction, and triggers the associated second event, which only outputs the notification information of "challenge failed" in the current graphical user interface.

[0127] In order to facilitate integration, the functions implemented by the method are integrated into a functional module. A pre-execution step can be added to the method of the typical embodiment with reference to the following improved embodiment, and an access entry of the method is set in the application program so that the user can call the application instance implemented by the method through the access entry. When the user operates the access entry, an event of calling the application instance of the method through the access entry can be triggered, as follows:

[0128] In one embodiment, the application of the present application is an application based on a video playback service such as short videos. The platform randomly or strategically pushes the target video used to trigger the execution of the relevant code of the present method to the user of the application. When the user of the terminal device "swipes" to such a short video in the process of browsing short videos using this application, the user (user) can watch the target video in the graphical user interface of the terminal device. The target video constitutes an access entry, which automatically invites the user to participate in a "hold back laughter" activity. After the user accepts the invitation, the application instance is executed to realize the use of the method of the present application. For this type of target video, it is generally accompanied by relevant tags, so it can be identified by the tags provided when the server pushes it. Of course, in this embodiment, the target video may not automatically invite the user to participate in the application instance of the present method by default, but provide a corresponding preset control in the graphical user interface as the access entry, or default a preset touch instruction operated in the terminal device as the access entry. In any case, as long as the application instance of the method of the present application can be finally called, it does not affect the implementation of the present application.

[0129] In another embodiment, in order to build the camera program of the terminal device into a drainage entrance to the Internet, the previous embodiment can be referred to, and the camera program can be used as an application program running in the terminal device described in the present application, and an application instance of the present method is integrated into the camera program, and the preset controls are provided in the graphical user interface of the camera program, or the preset touch instructions are recognized in the interface, and the instance of the present method is called accordingly. Similarly, the call to the instance of the present method can also be realized.

[0130] It can be seen that the access entry setting method for calling the instance of the present method is very flexible and can be flexibly implemented by those skilled in the art based on the disclosure of the above embodiments.

[0131] On the other hand, in the process of monitoring in step S21, in order to prevent the user from cheating by illegal means such as shifting the face, closing the eyes and not watching the target video, and providing a static facial expression image, when it is identified during the monitoring process that the specific facial expression image in the video image is not within the specified range of the lens, or it is identified that the specific facial expression image contains any one of the features of the user closing the eyes or the eyes not being focused, an alarm message can be generated, and the application control is notified to terminate or reset the playback of the target video, so as to implement a penalty mechanism for the user (terminal device user) violating the rules. It should be pointed out that the technical means of identifying whether the facial expression image is closed and whether the eyes are focused from the video image are already known to those skilled in the art, so they will not be repeated.

[0132] In addition to the various application scenarios involved in the process disclosed in the above embodiments, the method can also be applied to scenarios that require intelligent identification of the user's subjective attitude to identify the user's delayed decision, thereby deriving embodiments in various other situations.

[0133] In one embodiment, the target videos may be movies, TV series, news, etc., and while playing the target videos, the method starts a camera in the background to obtain video images in real time, recognizes the facial expressions of viewers, and recognizes facial expression images, including smiling faces representing different satisfaction levels or disgust expressions representing different satisfaction levels, etc., and converts them into quantitative amplitudes accordingly. According to the pre-associated relationship, when these quantitative amplitudes meet the preset conditions, corresponding expression instructions are generated to trigger pre-associated events, such as an evaluation event. Thus, the overall satisfaction of users watching the target video can be obtained through this method, and finally the evaluation data for the target video can be obtained. These evaluation data are naturally also suitable for further data mining. Of course, to adapt to this scenario, the preset conditions and their triggering mechanisms may be adaptively modified when necessary. For example, when detecting whether the quantization amplitude satisfies the preset conditions, it is possible to further examine whether the sum or average of the quantization amplitudes of multiple time periods during the entire playback process reaches or exceeds a certain threshold, so as to determine whether the preset conditions are met; and if the sum or average of the quantization amplitudes of one or more time periods fails to reach the threshold, it is deemed that the preset conditions are not met, indicating that the viewer does not like the content played by the target video, and the playback of the target video is terminated or reset.

[0134] As such, those skilled in the art should be able to apply the present method to various application scenarios according to the instructions of the above various embodiments of the present method, so as to maximize the beneficial effects of the present method.

[0135] Furthermore, the relevant devices of the present application can be constructed by functionalizing each step in the method disclosed in the above embodiments.

[0136] See also Figure 7 , Figure 7 The present invention discloses a device for quantifying facial expressions in an image, which comprises:

[0137] A detection and correction unit 11 is used to detect a facial image in the image to be recognized, and correct the facial image to a standard posture based on key feature points in the facial image;

[0138] A feature extraction unit 12 is used to import the corrected facial image into a trained convolutional neural network for feature extraction to obtain feature information of the facial expression contained therein;

[0139] A classification calculation unit 13 imports the extracted feature information into a classifier, and the classifier calculates the hit probability of each classification label corresponding to the feature information, wherein the classification label represents the classification attribute of the degree of change of the facial expression, and is defined by the classification attribute of the training sample of the convolutional neural network;

[0140] The quantization output unit 14 calculates and summarizes the hit probabilities of the various classification labels and outputs the quantized amplitude of the facial expression in the facial image.

[0141] See also Figure 8 , Figure 8 A method for recognizing facial expressions in an image of the present application is disclosed, comprising:

[0142] The monitoring unit 21 is used to monitor the image frames in the real-time video stream of the live broadcast process, call the program interface implemented by the facial expression quantization method in the image, use the image frames as the image to be identified that needs to be imported by the program interface, and track and analyze the image frames of the real-time video stream through the program interface to continuously obtain the quantization amplitude returned by the program interface;

[0143] A playing unit 22, used for playing a pre-provided target video in parallel with the monitoring;

[0144] The trigger unit 23 is used to detect whether the quantization amplitude meets a preset condition following the playing of the target video, and execute a computer instruction corresponding to whether the preset condition is met.

[0145] Furthermore, to facilitate the execution of the present application, the present application provides an electronic device comprising a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the method for quantifying facial expressions in images described in the aforementioned embodiments, or to execute the steps of the method for recognizing facial expressions in images described in the aforementioned embodiments.

[0146] It can be seen that the memory is suitable for using a non-volatile storage medium. By implementing the aforementioned method as a computer program and installing it in an electronic device such as a mobile phone or a computer, the relevant program code and data are stored in the non-volatile storage medium of the electronic device. The program is further run by the central processing unit of the electronic device, and it is transferred from the non-volatile storage medium to the memory for operation, so that the desired purpose of the present application can be achieved. Therefore, it can be understood that in one embodiment of the present application, a non-volatile storage medium can also be provided, in which a computer program implemented according to each embodiment of the method for quantifying facial expressions in images or the method for recognizing facial expressions in images is stored, and when the computer program is called and executed by a computer, the steps included in the method are executed.

[0147] To sum up, the present application has obvious and rich advantages. The present application can realize accurate quantification of facial expressions in images, and realizes the effect of calculating quantization amplitudes based on facial expressions in images for further application. It belongs to basic technology, and can expand and enrich various computer application scenarios. It is suitable for meeting various practical needs based on computer applications, and is suitable for realizing a variety of applications based on facial expression information.

[0148] It will be understood by those skilled in the art that the present application includes devices for performing one or more of the operations and methods described in the present application. These devices may be specially designed and manufactured for the desired purpose, or may also include known devices in general-purpose computers. These devices have computer programs stored in their memory, which are selectively activated or reconstructed. Such computer programs may be stored in a device (e.g., computer) readable medium or in any type of medium suitable for storing electronic instructions and coupled to a bus, respectively, the computer readable medium including but not limited to any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic card or optical card. That is, the readable medium includes any medium that stores or transmits information in a readable form by a device (e.g., computer).

[0149] Those skilled in the art will appreciate that each block in these structure diagrams and / or block diagrams and / or flow charts and combinations of blocks in these structure diagrams and / or block diagrams and / or flow charts can be implemented using computer program instructions. Those skilled in the art will appreciate that these computer program instructions can be provided to a general-purpose computer, a professional computer, or a processor of other programmable data processing methods to implement, thereby executing the scheme specified in the block or multiple blocks of the structure diagrams and / or block diagrams and / or flow charts disclosed in this application through the processor of the computer or other programmable data processing method.

[0150] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, altered, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted.

[0151] The above is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for quantifying facial expressions in an image, characterized in that: The steps include: Detecting a facial image in an image to be recognized in a real-time video stream of a virtual event held in a live broadcast room, and correcting the facial image to a standard posture based on key feature points in the facial image; Importing the corrected facial image into a trained convolutional neural network for feature extraction to obtain feature information of the facial expression contained therein; Importing the extracted feature information into a classifier, the classifier calculates the hit probability of each classification label corresponding to the feature information, wherein the classification label represents the classification attribute of the degree of change of the facial expression, which is defined by the classification attribute of the training sample of the convolutional neural network, and the classifier pre-matches a corresponding label coefficient for each classification label, and each label coefficient numerically expresses the relative strength relationship between the degrees of change; According to the summary calculation of the hit probability of each classification label, the product of the hit probability of each classification label and its corresponding label coefficient is accumulated and the sum is used as the quantization amplitude. The quantization amplitude of the facial expression in the facial image is output for controlling the virtual activities held in the live broadcast room.

2. The method according to claim 1, characterized in that: The key feature points include the center points of the left and right eyes, the tip of the nose, and the left and right corners of the mouth in the facial image. When implementing the facial image correction, the similarity transformation matrix is ​​calculated between the key feature points and the corresponding feature points of the standard face model, and then the similarity transformation matrix is ​​applied to the facial image to correct it to a standard posture.

3. The method according to claim 1, characterized in that The sum is multiplied by an adjustment coefficient as the quantization amplitude, so that the value range of the quantization amplitude is constrained within a specific target value interval.

4. The method according to claim 1, characterized in that The facial images included in the training data set of the convolutional neural network have human smiling or crying face features.

5. The method according to any one of claims 1 to 4, characterized in that: The quantized amplitude is used to control the progress of one or more virtual activities performed based on the real-time video stream, or is used to output to a graphical user interface of a terminal device that generates the real-time video stream.

6. A method for recognizing facial expressions in an image, characterized in that: It includes the following steps: Monitor the image frames in the real-time video stream of the live broadcast process, call a preset program interface, and use the image frames as the to-be-recognized images to be imported by the program interface, so as to track and analyze the image frames of the real-time video stream through the program interface and continuously obtain the quantized amplitude returned by the program interface; In parallel with the monitoring, playing a pre-provided target video; Following the playback of the target video, the quantization amplitude is detected to determine whether it meets a preset condition, and a computer instruction corresponding to whether the preset condition is met is executed. When the program interface is called, the following steps are performed: Detecting a facial image in an image to be recognized in a real-time video stream of a virtual event held in a live broadcast room, and correcting the facial image to a standard posture based on key feature points in the facial image; Importing the corrected facial image into a trained convolutional neural network for feature extraction to obtain feature information of the facial expression contained therein; Importing the extracted feature information into a classifier, the classifier calculates the hit probability of each classification label corresponding to the feature information, wherein the classification label represents the classification attribute of the degree of change of the facial expression, which is defined by the classification attribute of the training sample of the convolutional neural network, and the classifier pre-matches a corresponding label coefficient for each classification label, and each label coefficient numerically expresses the relative strength relationship between the degrees of change; According to the summary calculation of the hit probability of each classification label, the product of the hit probability of each classification label and its corresponding label coefficient is accumulated and the sum is used as the quantization amplitude. The quantization amplitude of the facial expression in the facial image is output for controlling the virtual activities held in the live broadcast room.

7. The method according to claim 6, characterized in that During the execution of the method, the quantized amplitude or text or graphic symbols corresponding to the quantized amplitude are synchronously output in the graphical user interface where the playback window of the real-time video stream is located.

8. A device for quantifying facial expressions in an image, characterized in that: It includes: A detection and correction unit, used to detect a facial image in an image to be identified in a real-time video stream of a virtual activity held in a live broadcast room, and correct the facial image to a standard posture based on key feature points in the facial image; A feature extraction unit, used for importing the corrected facial image into a trained convolutional neural network for feature extraction to obtain feature information of the facial expression contained therein; A classification calculation unit, which imports the extracted feature information into a classifier, and the classifier calculates the hit probability of each classification label corresponding to the feature information, wherein the classification label represents the classification attribute of the degree of change of the facial expression, which is defined by the classification attribute of the training sample of the convolutional neural network, and the classifier pre-matches a corresponding label coefficient for each classification label, and each label coefficient numerically expresses the relative strength relationship between the degrees of change; The quantization output unit calculates the hit probability of each classification label, accumulates the sum of the products of the hit probability of each classification label and its corresponding label coefficient, and outputs the quantization amplitude of the facial expression in the facial image for controlling the virtual activities held in the live broadcast room.

9. An electronic device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run a computer program stored in the memory to execute the steps of the method for quantifying facial expressions in an image as described in any one of claims 1 to 5, or to execute the steps of the method for recognizing facial expressions in an image as described in claim 6 or 7.

10. A non-volatile storage medium, characterized in that: It stores a computer program implemented according to the method for quantifying facial expressions in images as described in any one of claims 1 to 5, or stores a computer program implemented according to the method for recognizing facial expressions in images as described in claim 6 or 7. When the computer program is called and executed by a computer, the steps included in the corresponding method are executed.