Facial emotion recognition method and apparatus, and storage medium and vehicle

Abstract

The present application is applicable to the technical field of image recognition. Provided are a facial emotion recognition method and apparatus, and a storage medium and a vehicle. The method comprises: acquiring a facial image of a user; dividing the facial image, and performing feature extraction on each facial region obtained after division, so as to obtain facial features of the user; obtaining from a pre-established classification feature pool a clustering region closest to the facial features, wherein the classification feature pool comprises clustering regions for facial features corresponding to different emotions; and on the basis of the closest clustering region, determining the emotion of the user. In the present application, the emotion of the user can be determined on the basis of features of each region of the facial image, and the closest clustering region in the classification feature pool, thereby improving the accuracy of recognizing the emotion of the user, and reducing misdetection and false alarms of the emotion of the user.

Description

Facial emotion recognition method, device, storage medium and vehicle

[0001] This application claims priority to Chinese patent application No. CN202311776179.5, filed on December 21, 2023, entitled “Facial Emotion Recognition Method, Apparatus, Storage Medium, and Vehicle.” The disclosure of that prior application is incorporated herein by reference in its entirety. Technical Field

[0002] The present application relates to the field of image recognition technology, and in particular to a facial emotion recognition method, device, storage medium and vehicle. Background Art

[0003] With the development of artificial intelligence, facial emotion recognition has been applied in an increasing number of fields. For example, in the automotive sector, emotion recognition is crucial for safe driving, in-vehicle atmosphere, and the interactive experience between the vehicle and the user. The future development trend of the intelligent cockpit is to understand the emotional state of the relevant personnel in the vehicle, thereby providing emotional interaction and service capabilities based on their emotions, improving safe driving and the user experience of the intelligent cockpit.

[0004] However, due to the differences in faces of different people, related technologies are prone to misdetection and false alarm of user emotions and cannot correctly identify user emotions. Technical issues

[0005] In view of this, the embodiments of the present application provide a facial emotion recognition method, device, storage medium and vehicle to solve the technical problems in related technologies that are prone to misdetection and false reporting of user emotions and cannot correctly identify user emotions. Technical Solutions

[0006] In a first aspect, an embodiment of the present application provides a method for recognizing facial emotions, comprising:

[0007] Get the user's facial image;

[0008] Segmenting the facial image and performing feature extraction on each segmented facial region to obtain facial features of the user;

[0009] Obtaining a cluster region closest to the facial feature from a pre-established classification feature pool; wherein the classification feature pool includes cluster regions of facial features corresponding to different emotions;

[0010] The user's emotion is determined based on the most similar cluster area.

[0011] In a possible implementation of the first aspect, extracting features from each divided facial region to obtain facial features of the user includes:

[0012] Performing feature extraction on each divided facial region to determine local features corresponding to each facial region;

[0013] determining a head pose in a facial image of the user;

[0014] A facial feature matrix is ​​obtained based on the local features and the head posture, and the facial feature matrix is ​​used as the facial features of the user.

[0015] In a possible implementation of the first aspect, after determining the user's emotion based on the closest cluster area, the method further includes:

[0016] determining the identity of the user based on the facial image of the user;

[0017] Obtaining a classification feature pool associated with the identity of the user based on the facial features of the user and the closest cluster region;

[0018] Accordingly, for the next user, the cluster region closest to the facial features is obtained from the pre-established classification feature pool, including:

[0019] The identity of the next user is determined according to the facial image of the next user, and based on a classification feature pool associated with the identity of the next user, a cluster region closest to the facial features of the next user is obtained.

[0020] In a possible implementation of the first aspect, the pre-established classification feature pool includes clustered regions of facial features corresponding to different emotions of different types of people;

[0021] The step of obtaining a classification feature pool associated with the identity of the user based on the facial features of the user and the closest cluster region includes:

[0022] Obtaining a target cluster region from a pre-established classification feature pool; the type of personnel corresponding to the target cluster region is the same as the type of personnel corresponding to the closest cluster region;

[0023] Based on the facial features of the user and the target cluster area, a classification feature pool associated with the identity of the user is obtained.

[0024] In a possible implementation of the first aspect, obtaining the cluster region closest to the facial feature from a pre-established classification feature pool includes:

[0025] Obtaining, based on the vector value of the facial feature and the vector value of the cluster region in the classification feature pool, the Euclidean distance between the facial feature and different cluster regions in the classification feature pool;

[0026] The cluster area corresponding to the minimum Euclidean distance in the Euclidean distances is used as the closest cluster area.

[0027] In a possible implementation of the first aspect, before obtaining the cluster region closest to the facial feature from the pre-established classification feature pool, the method further includes:

[0028] Collecting a plurality of facial images; the plurality of facial images include facial images of different emotions;

[0029] Dividing the plurality of facial images respectively, and performing feature extraction on each facial region after the plurality of facial images are divided, to obtain corresponding facial features;

[0030] Cluster multiple facial features of the same emotion type to obtain multiple cluster regions under the same emotion, and build a classification feature pool based on all the obtained cluster regions.

[0031] In a possible implementation of the first aspect, determining the user's emotion based on the closest cluster area includes:

[0032] Determining the emotion corresponding to the closest cluster region;

[0033] The emotion corresponding to the closest cluster region is used as the emotion of the user.

[0034] In a possible implementation of the first aspect, the pre-established classification feature pool includes clustered regions of facial features corresponding to different emotions of different types of people; after acquiring the user's facial image, the method further includes:

[0035] determining the person type of the user based on the facial image of the user;

[0036] Accordingly, the clustering region closest to the facial features is obtained from the pre-established classification feature pool, including:

[0037] According to the person type of the user, in a pre-established classification feature pool, cluster areas with the same person type as the user are determined, and from the cluster areas with the same person type as the user, cluster areas closest to the facial features are determined.

[0038] In a second aspect, an embodiment of the present application provides a facial emotion recognition device, comprising:

[0039] The acquisition module is used to obtain the user's facial image.

[0040] The extraction module is used to divide the facial image and perform feature extraction on each divided facial area to obtain the facial features of the user.

[0041] The acquisition module is used to obtain the cluster area closest to the facial feature from a pre-established classification feature pool; wherein the classification feature pool includes cluster areas of facial features corresponding to different emotions.

[0042] A determination module is used to determine the user's emotion based on the closest cluster area.

[0043] In a third aspect, an embodiment of the present application provides a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the facial emotion recognition method as described in any one of the first aspects.

[0044] In a fourth aspect, an embodiment of the present application provides a vehicle comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the facial emotion recognition method as described in any one of the first aspects is implemented. Beneficial effects

[0045] The facial emotion recognition method provided in the embodiment of the present application first divides the acquired facial image of the user, extracts features from each divided facial area, and obtains the facial features of the user. Thereafter, based on a pre-established classification feature pool, the cluster area closest to the facial features is determined, wherein the classification feature pool includes cluster areas of facial features corresponding to different emotions. Furthermore, based on the above-mentioned closest cluster areas, the user's emotions are determined. In this way, the user's emotions can be determined based on the features of each area of ​​the facial image and the closest cluster area in the classification feature pool, which can improve the accuracy of identifying the user's emotions and reduce false detections and false alarms of the user's emotions.

[0046] It can be understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0047] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0049] FIG1 is a flow chart of a facial emotion recognition method according to an embodiment of the present application;

[0050] FIG2 is a schematic diagram of a facial region provided by an embodiment of the present application;

[0051] FIG3 is a flowchart of a facial emotion recognition method provided by an embodiment of the present application;

[0052] FIG4 is a flow chart of a facial emotion recognition method provided by another embodiment of the present application;

[0053] FIG5 is a schematic diagram of the structure of a facial emotion recognition device provided by an embodiment of the present application;

[0054] FIG6 is a schematic structural diagram of a vehicle provided in an embodiment of the present application. Modes for Carrying Out the Invention

[0055] The present application will be described more clearly below with reference to specific embodiments. The following embodiments will help those skilled in the art further understand the function of the present application, but are not intended to limit the present application in any form. It should be noted that those skilled in the art may make a number of modifications and improvements without departing from the concept of the present application. These all fall within the scope of protection of the present application.

[0056] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0057] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0058] In the description of this application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0059] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0060] In addition, the “plurality” mentioned in the embodiments of the present application should be interpreted as two or more.

[0061] In the automotive sector, the future development trend of smart cockpits is to understand the emotional state of passengers in the vehicle, thereby providing emotionally-responsive interactions and services based on their emotions, improving safe driving and the user experience of the smart cockpit. However, due to the differences in facial features, related technologies are prone to misdetection and false alarms of user emotions, failing to accurately identify user emotions.

[0062] In response to the above-mentioned problem of not being able to correctly identify user emotions, the inventors have discovered through research that the user's facial image can be divided into different areas, such as the nose, mouth, forehead, and eyes, and the local features of each area can be extracted to form facial features. Thus, based on a pre-established classification feature pool (the classification feature pool includes clustering areas of facial features corresponding to different emotions), the most similar clustering areas are determined according to the facial features, and then the user's emotions are identified, thereby improving the accuracy of identifying the user's emotions and reducing false detections and false alarms of user emotions.

[0063] FIG1 is a flow chart of a facial emotion recognition method provided by an embodiment of the present application. As shown in FIG1 , the method in the embodiment of the present application is applied to a vehicle, and the method may include:

[0064] Step 101: Obtain a user's facial image.

[0065] Illustratively, in this embodiment, the user's facial image may be acquired through a vehicle-mounted camera, or the user may input the facial image through a vehicle-mounted display screen or other terminal device.

[0066] Step 102: segment the facial image and extract features from each segmented facial region to obtain facial features of the user.

[0067] In this embodiment, when the user's emotions are different, the expressions of various areas of the user's face will be different. Therefore, the user's face can be divided into regions, and emotion analysis and recognition can be performed based on the features corresponding to each facial region. Referring to Figure 2, the facial image is divided into multiple facial regions, for example, the forehead, left eyebrow, right eyebrow, left eye, right eye, nose, left cheek, right cheek, mouth, and chin are divided, totaling 10 facial regions. Of course, the facial image can also be divided in other ways to obtain multiple facial regions, and no specific restrictions are made here. In addition, when the user's emotions are different, the user's head posture may also be different. For example, when happy, the head may be tilted. Therefore, in this embodiment, head posture is also used as the basis for emotion analysis and recognition.

[0068] As can be seen from the above, different emotions reflect different facial postures. For example, the mouth corresponds to postures such as relaxed, slightly open, pursed, and raised, and the proportions of these postures vary depending on the emotion. Similarly, different emotions also reflect different head postures. For example, head posture corresponds to pitch and yaw, and the values ​​of these postures vary depending on the emotion. Therefore, facial features include the proportions of each posture in each facial region, as well as the values ​​of pitch and yaw. By extracting features from each facial region and head posture, the user's facial features can be obtained.

[0069] In one possible implementation, referring to FIG3 , when obtaining the user's facial features, this embodiment may perform feature extraction on each facial region, determine the local features corresponding to each facial region, and determine the head posture in the user's facial image. Thereafter, based on the local features and the head posture, a facial feature matrix is ​​obtained, and the facial feature matrix is ​​used as the user's facial features.

[0070] For example, the features of each facial region can be seen in Table 1. The features corresponding to each facial region are not limited to the features listed in Table 1, but can also be other features. In this embodiment, a neural network can be used to extract features from each facial region obtained by division, and the proportions of different features corresponding to each facial region can be obtained to form a regional feature vector. For example, the features corresponding to the forehead can include relaxation, shrug, and wrinkling. After local feature extraction, the proportions corresponding to the features relaxation, shrug, and wrinkling can be obtained as 97%, 0%, and 3%, respectively. In this embodiment, a neural network can also be used to extract features from the head posture in the facial image to obtain head posture feature values. For example, the features corresponding to head posture can include tilting the head, burying the head, tilting the head, and turning the head, and the features corresponding to different head postures can be determined based on the corresponding pitch angle Pitch, yaw angle Yaw, and roll angle Roll of the head. The above-mentioned regional feature vectors and head posture feature values ​​together constitute the user's facial feature matrix.

[0071] Table 1

[0072] Facial features: Forehead relaxed, raised, wrinkled... Eyebrows relaxed, raised, lowered, squeezed... Eyes unfocused, dull, widened, squinted, pupils constricted, looking up, down, sideways, or cross-eyed, tearing... Nose relaxed, expanding outward and contracting inward... Cheeks relaxed, puffed up, contracted, and drooped... Mouth relaxed, slightly open, twisted to the left or right, lips bitten, pursed, corners raised, corners drooping, teeth showing, tongue sticking out, pouting, O-shaped mouth... Chin relaxed, tilted up, pressed down... Head posture: head tilted, head lowered, head tilted, head turned...

[0073] Step 103: Obtain the cluster region closest to the facial features from a pre-established classification feature pool; wherein the classification feature pool includes cluster regions of facial features corresponding to different emotions.

[0074] In one possible implementation, referring to FIG3 , before obtaining the cluster region most similar to the facial features, this embodiment may also collect multiple facial images, segment each of the multiple facial images, and perform feature extraction on each of the segmented facial regions to obtain corresponding facial features. Subsequently, multiple facial features of the same emotion type are clustered to obtain multiple cluster regions for the same emotion, and a classification feature pool is constructed based on all the obtained cluster regions. The multiple facial images may include facial images of different emotions.

[0075] For example, when constructing a classification feature pool, to ensure the comprehensiveness and diversity of the classification feature pool, this embodiment collects facial images of people in different head postures and with different emotions. When collecting different facial images, multiple cameras at different angles can be used to collect facial images in different head postures. Afterwards, this embodiment can perform region division and feature extraction on the multiple collected facial images, and perform feature extraction on the head postures in the multiple facial images to obtain facial features corresponding to each facial image, that is, to obtain a corresponding facial feature matrix. The specific implementation principle and process of obtaining the facial features mentioned above can refer to the relevant parts of step 102 and will not be repeated here.

[0076] Optionally, in this embodiment, when collecting multiple facial images, each facial image corresponds to an emotion type, which may include happiness, appreciation, sadness, anger, fear, surprise, disgust, and neutrality. Subsequently, multiple facial features of the same emotion type are clustered, and facial features with similar features within the same emotion type are divided into a cluster region. The emotion corresponding to this cluster region is the aforementioned emotion type. For example, all facial features with the emotion type of happiness are clustered to obtain multiple cluster regions with the emotion type of happiness. Similarly, all facial features with the emotion type of sadness are clustered to obtain multiple cluster regions with the emotion type of sadness, and so on. Thus, all the obtained cluster regions constitute a classification feature pool. In this way, each cluster region includes multiple facial features with similar features.

[0077] For example, a deep learning algorithm can also be used to cluster multiple facial features of the same emotion type, and divide facial features with similar features in the same emotion type into a cluster area, thereby constructing a classification feature pool based on all the obtained cluster areas.

[0078] In one possible implementation, in this embodiment, the facial features in each cluster region in the classification feature pool that have the smallest feature difference with the user's facial features are determined as the most similar facial features. Furthermore, the cluster region corresponding to the most similar facial features is determined as the most similar cluster region. Since smaller distances between features indicate higher feature similarity, the most similar facial features can be determined by calculating the distances between the user's facial features and different facial features in each cluster region in the classification feature pool.

[0079] For example, when obtaining the cluster area closest to the facial features, this embodiment can obtain the Euclidean distance between the facial features and the different cluster areas in the classification feature pool based on the vector value of the facial features and the vector value of the cluster area in the classification feature pool. Then, the cluster area corresponding to the minimum Euclidean distance in the Euclidean distance is used as the closest cluster area.

[0080] As can be seen from the above, the user's facial features are represented by the corresponding facial feature matrix, and the facial feature matrix is ​​composed of regional feature vectors and head posture feature values, and each clustering area in the classification feature pool is also obtained by clustering facial features. Therefore, this embodiment can perform Euclidean distance calculation based on the vector value of the user's facial features and the vector value of different facial features in each clustering area in the classification feature pool to obtain the Euclidean distance between the user's facial features and the different facial features in each clustering area. After that, the clustering area corresponding to the minimum Euclidean distance can be determined as the clustering area closest to the user's facial features.

[0081] Step 104: Determine the user's emotion based on the closest cluster area.

[0082] Exemplarily, referring to FIG. 3 , this embodiment can obtain the emotion corresponding to the closest cluster region from a pre-established classification feature pool, and use the emotion (ie, emotion type) corresponding to the closest cluster region as the user's emotion.

[0083] The facial emotion recognition method provided in the embodiment of the present application first divides the acquired facial image of the user, extracts features from each divided facial area, and obtains the facial features of the user. Thereafter, based on a pre-established classification feature pool, the cluster area closest to the facial features is determined, wherein the classification feature pool includes cluster areas of facial features corresponding to different emotions. Furthermore, based on the above-mentioned closest cluster areas, the user's emotions are determined. In this way, the user's emotions can be determined based on the features of each area of ​​the facial image and the closest cluster area in the classification feature pool, which can improve the accuracy of identifying the user's emotions and reduce false detections and false alarms of the user's emotions.

[0084] In one possible implementation, to further improve the accuracy and efficiency of user emotion recognition, after acquiring the user's facial image, this embodiment can also determine the user's person type based on the user's facial image. Accordingly, when obtaining the cluster region that is most similar to the facial features, this embodiment can determine, based on the user's person type, cluster regions with the same person type as the user in a pre-established classification feature pool. Furthermore, from cluster regions with the same person type as the user, cluster regions with the most similar facial features can be determined. The classification feature pool includes cluster regions of facial features corresponding to different emotions of different types of people.

[0085] For example, when constructing the classification feature pool, to ensure its comprehensiveness and diversity, this embodiment collects facial images of different people in different head postures and emotions, primarily covering people of different body shapes, ages, and appearances. Appearance features can be primarily distinguished by eyes, nose, mouth, hair, face shape, and skin color, with as many emotion-related appearance features as possible being included. Subsequently, facial features corresponding to each facial image are obtained. The specific implementation principles and process for obtaining facial features described above can be referenced to the relevant sections of step 102 and will not be repeated here.

[0086] As can be seen from the foregoing, when multiple facial images are collected, each facial image corresponds to an emotion type. In this embodiment, each facial image also corresponds to other types, such as age type and body type. Body type types can include fat, average-sized, and thin, while age types can include child, teenager, youth, middle-aged, and elderly. For example, a facial image may correspond to types such as happy, fat, and child. The facial features obtained from the facial image also correspond to emotion types, age types, and body type types. Subsequently, multiple facial features of the same emotion type are clustered. This is because different people, different body types, or different age groups may express the same emotion differently. For example, when expressing happiness, some people may squint, purse their lips, and expand their noses, while others may squint, open their mouths, and tilt their heads. After clustering, facial features with high similarity are clustered into a single cluster region, forming different cluster regions for the same emotion. In other words, the classification feature pool includes multiple cluster regions for different emotions, while also containing multiple cluster regions for the same emotion. This allows for the segmentation of the same emotion, providing support for subsequent accurate recognition of user emotions.

[0087] Exemplarily, in this embodiment, the facial features in each cluster area have the same emotion type, but the age type and body type of each facial feature may be different. Among them, the age type and body type can be defined as the person type. Then for each cluster area, the person type of the cluster area can be determined based on the person types corresponding to the multiple facial features it includes, such as taking the person type with the largest number of facial features as the person type corresponding to the cluster area. For example, a cluster area includes 10 facial features, of which 9 facial features correspond to the age type of child, and 1 facial feature corresponds to the age type of youth, then the age type of the cluster area is determined to be child. Similarly, the body type of the cluster area can be determined based on the body type of the facial features, etc. In this way, the type corresponding to the cluster area can be obtained, for example, the type corresponding to a cluster area is happy, fat and child, and the type corresponding to another cluster area is happy, thin and youth.

[0088] In this way, this embodiment can obtain the user's personnel type (i.e., age type and body type) based on the user's facial image based on the age algorithm and the body shape algorithm. Then, directly based on the user's personnel type, multiple clustering areas with the same personnel type as the user are selected from the classification feature pool. Then, based on the above-selected multiple clustering areas, the clustering area closest to the user's facial features is determined. Then, the user's emotions are determined based on the closest clustering area, which can improve the accuracy of user emotion recognition and at the same time improve recognition efficiency.

[0089] In order to further improve the accuracy and efficiency of user emotion recognition, it is also possible to build a classification feature pool associated with the user's own characteristics and habits based on the user's facial features and the closest clustering area, so that the next time the user's emotion is recognized, the classification feature pool associated with the user can be directly used to determine the user's emotion.

[0090] In a possible implementation, after determining the user's emotion, this embodiment may also determine the user's identity based on the user's facial image, and obtain a classification feature pool associated with the user's identity based on the user's facial features and the closest clustering area.

[0091] For example, in this embodiment, a user's identity is identified based on a facial image of the user. For example, a recognition result is obtained by performing feature recognition on the facial image of the user, where the recognition result may be a biometric recognition result. An identity, such as identity A, may then be assigned to the recognition result, and a corresponding relationship between the recognition result and identity A is established based on the recognition result and the corresponding identity A.

[0092] In one possible implementation, when obtaining a classification feature pool associated with a user's identity, this embodiment may also obtain a target cluster region from a pre-established classification feature pool, and then obtain a classification feature pool associated with the user's identity based on the user's facial features and the target cluster region. The person type corresponding to the target cluster region is the same as the person type corresponding to the closest cluster region.

[0093] As can be seen from the aforementioned embodiments, cluster regions correspond to emotion types and person types (person types include age types and body types). For example, one cluster region corresponds to the person types of youth and thinness, while another cluster region corresponds to the person types of children and obesity. In this embodiment, if the person type corresponding to the closest cluster region is youth and thinness, all cluster regions with the person types of youth and thinness are selected from the pre-established classification feature pool as the target cluster region. Thus, a classification feature pool is constructed based on the user's facial features and the target cluster region, and the constructed classification feature pool is associated with the user's identity A to serve as the classification feature pool associated with the user's identity. In this way, the classification feature pool associated with the user's identity is the same as the user's type and is associated with the user's own characteristics and habits. In subsequent processes, by identifying the user's identity, the classification feature pool associated with the user's identity can be directly called to perform user emotion recognition, more accurately identifying the user's emotions and improving recognition efficiency.

[0094] It should be noted that in order to ensure that the facial features of each cluster area in the classification feature pool associated with the user's identity are as comprehensive and diverse as possible while being associated with the user's own characteristics and habits, this embodiment can periodically update the classification feature pool associated with the user's identity based on the user's facial features and the closest cluster area. For example, after determining that the user's identity is identity A, it is determined whether the time interval between the current identification of identity A and the last identification of identity A is greater than a preset interval threshold. If the above time interval is greater than the preset interval threshold, the classification feature pool associated with the user's identity is updated; otherwise, the classification feature pool associated with the user's identity is not updated.

[0095] It should be noted that in order to improve the relevance of the classification feature pool associated with the user's identity, after determining that the user's identity is identity A, only the personnel type corresponding to the closest clustering area is recorded, and the classification feature pool associated with the user's identity is not constructed. Instead, after identifying identity A multiple times, for example, after identifying identity A three times, the personnel types corresponding to multiple closest clustering areas are recorded. Thus, based on the personnel types corresponding to the above multiple closest clustering areas, the target personnel type is determined, and then the target clustering area is determined based on the target personnel type, and then the classification feature pool associated with the user's identity is determined.

[0096] In this way, this embodiment constructs a classification feature pool associated with the user's own characteristics, habits, etc. based on the user's facial features and the closest clustering area, so that in the subsequent process, by identifying the user's identity, the classification feature pool associated with the user's identity can be directly called to perform user emotion recognition.

[0097] FIG4 is a flow chart of a facial emotion recognition method provided by another embodiment of the present application. As shown in FIG4 , the method in the embodiment of the present application may include:

[0098] Step 401: Obtain the facial image of the next user.

[0099] Step 402: Segment the facial image of the next user, and perform feature extraction on each segmented facial region to obtain facial features of the next user.

[0100] The specific implementation process and principles of step 401 to step 402 of this embodiment can refer to the above embodiment and will not be repeated here.

[0101] Step 403: Determine the identity of the next user according to the facial image of the next user, and obtain a cluster region that is closest to the facial features of the next user based on a classification feature pool associated with the identity of the next user.

[0102] For example, in this embodiment, feature recognition is performed on the next user's facial image to obtain a recognition result, and the next user's identity is determined based on the correspondence between the recognition result and the identity. Thus, based on the next user's identity, a pool of classification features associated with the next user's identity is determined, and further, the cluster region that most closely matches the next user's facial features is determined. The specific implementation process and principles for determining the cluster region that most closely matches the next user's facial features based on the pool of classification features associated with the next user's identity can be found in step 103 of the aforementioned embodiment and will not be further described here.

[0103] It should be noted that if there is no classification feature pool associated with the identity of the next user, that is, the next user is a new user, the clustering area closest to the facial features of the next user is determined according to step 103 in the above embodiment.

[0104] Step 404: Determine the emotion of the next user based on the closest cluster area.

[0105] The specific implementation process and principle of step 404 in this embodiment can refer to the above embodiments and will not be repeated here.

[0106] This embodiment can determine the identity of the user, and directly determine the clustering area closest to the user's facial features based on the classification feature pool associated with the user's identity, and then determine the user's emotions. Since the classification feature pool associated with the user's identity is associated with the user's own characteristics and habits, the classification feature pool associated with the user's identity can more accurately identify the user's emotions, reduce false detection and false alarms of user emotions, and improve recognition efficiency.

[0107] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0108] FIG5 is a schematic diagram of the structure of a facial emotion recognition device provided by an embodiment of the present application. As shown in FIG5 , the facial emotion recognition device provided by this embodiment may include: an acquisition module 501 , an extraction module 502 , an acquisition module 503 , and a determination module 504 .

[0109] The acquisition module 501 is used to acquire a facial image of a user;

[0110] Extraction module 502, configured to segment the facial image and perform feature extraction on each segmented facial region to obtain facial features of the user;

[0111] An acquisition module 503 is configured to obtain a cluster region closest to the facial feature from a pre-established classification feature pool; wherein the classification feature pool includes cluster regions of facial features corresponding to different emotions;

[0112] The determination module 504 is configured to determine the user's emotion based on the closest cluster region.

[0113] Optionally, the extraction module 502 is specifically configured to:

[0114] Performing feature extraction on each facial region to determine local features corresponding to each facial region;

[0115] determining a head pose in a facial image of the user;

[0116] A facial feature matrix is ​​obtained based on the local features and the head posture, and the facial feature matrix is ​​used as the facial features of the user.

[0117] Optionally, the determining module 504 is further configured to:

[0118] determining the identity of the user based on the facial image of the user;

[0119] A classification feature pool associated with the identity of the user is obtained according to the facial features of the user and the closest cluster region.

[0120] Accordingly, for the next user, the obtaining module 503 is specifically configured to:

[0121] The identity of the next user is determined according to the facial image of the next user, and based on a classification feature pool associated with the identity of the next user, a cluster region closest to the facial features of the next user is obtained.

[0122] Optionally, the pre-established classification feature pool includes clustered regions of facial features corresponding to different emotions of different types of people; the determination module 504 is further specifically configured to:

[0123] Obtaining a target cluster region from a pre-established classification feature pool; the type of personnel corresponding to the target cluster region is the same as the type of personnel corresponding to the closest cluster region;

[0124] Based on the facial features of the user and the target cluster area, a classification feature pool associated with the identity of the user is obtained.

[0125] Optionally, the obtaining module 503 is specifically configured to:

[0126] Obtaining, based on the vector value of the facial feature and the vector value of the cluster region in the classification feature pool, the Euclidean distance between the facial feature and different cluster regions in the classification feature pool;

[0127] The cluster area corresponding to the minimum Euclidean distance in the Euclidean distances is used as the closest cluster area.

[0128] Optionally, the obtaining module 503 is further specifically configured to:

[0129] Collecting a plurality of facial images; the plurality of facial images include facial images of different emotions;

[0130] Dividing the plurality of facial images respectively, and performing feature extraction on each facial region after the plurality of facial images are divided, to obtain corresponding facial features;

[0131] Cluster multiple facial features of the same emotion type to obtain multiple cluster regions under the same emotion, and build a classification feature pool based on all the obtained cluster regions.

[0132] Optionally, the determining module 504 is specifically configured to:

[0133] Determining the emotion corresponding to the closest cluster region;

[0134] The emotion corresponding to the closest cluster region is used as the emotion of the user.

[0135] Optionally, the pre-established classification feature pool includes clustered areas of facial features corresponding to different emotions of different types of people; the acquisition module 501 is further specifically used to:

[0136] determining the person type of the user based on the facial image of the user;

[0137] Accordingly, the obtaining module 503 is further specifically configured to:

[0138] According to the person type of the user, in a pre-established classification feature pool, cluster areas with the same person type as the user are determined, and from the cluster areas with the same person type as the user, cluster areas closest to the facial features are determined.

[0139] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0140] FIG6 is a schematic diagram of the structure of a vehicle provided in one embodiment of the present application. As shown in FIG6 , vehicle 600 of this embodiment includes: a processor 610 and a memory 620. The memory 620 stores a computer program 621 executable on the processor 610. When the processor 610 executes the computer program 621, it implements the steps of any of the aforementioned method embodiments, such as steps 101 to 104 shown in FIG1 . Alternatively, when the processor 610 executes the computer program 621, it implements the functions of the modules / units in the aforementioned device embodiments, such as the functions of modules 501 to 504 shown in FIG5 .

[0141] For example, computer program 621 may be divided into one or more modules / units, one or more of which are stored in memory 620 and executed by processor 610 to implement the present application. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of computer program 621 in vehicle 600.

[0142] Those skilled in the art will understand that Figure 6 is only an example of a vehicle and does not constitute a limitation on the vehicle. The vehicle may include more or fewer components than shown in the figure, or a combination of certain components, or different components, such as input and output devices, network access devices, buses, etc.

[0143] The processor 610 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0144] Memory 620 can be an internal storage unit of the vehicle, such as the vehicle's hard drive or memory, or an external storage device, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, or a flash memory card. Memory 620 can also include both internal storage units and external storage devices. Memory 620 is used to store computer programs and other programs and data required by the vehicle. Memory 620 can also be used to temporarily store data that has been output or is about to be output.

[0145] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0146] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0147] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0148] In the embodiments provided by the present invention, it should be understood that the disclosed devices / vehicles and methods can be implemented in other ways. For example, the device / vehicle embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0149] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0150] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0151] If the integrated module / unit is implemented as a software functional unit and sold or used as a standalone product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can also implement all or part of the process steps in the above-mentioned method embodiments by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium.

[0152] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A facial emotion recognition method, characterized in that: include: Get the user's facial image; Dividing the facial image, and extracting features from each divided facial region to obtain facial features of the user; Obtaining a cluster region closest to the facial feature from a pre-established classification feature pool; wherein the classification feature pool includes cluster regions of facial features corresponding to different emotions; Based on the most similar cluster area, the emotion of the user is determined.

2. The facial emotion recognition method according to claim 1, characterized in that: The step of extracting features from each divided facial region to obtain facial features of the user includes: Extracting features from each divided facial region to determine local features corresponding to each facial region; determining a head pose in a facial image of the user; A facial feature matrix is ​​obtained based on the local features and the head posture, and the facial feature matrix is ​​used as the facial features of the user.

3. The facial emotion recognition method according to claim 2, characterized in that: The feature extraction of each facial region after division includes: using a neural network to extract features of each facial region obtained by division, obtaining the proportion of different features corresponding to each facial region, and forming a regional feature vector; Determining the head posture in the facial image of the user includes: extracting features of the head posture in the facial image using a neural network to obtain a head posture feature value; The regional feature vector and the head posture feature value together constitute the facial feature matrix of the user.

4. The facial emotion recognition method according to claim 1, characterized in that: After determining the emotion of the user based on the most similar clustering area, the method further includes: determining the identity of the user based on the facial image of the user; Obtaining a classification feature pool associated with the identity of the user according to the facial features of the user and the closest clustering region; Correspondingly, for the next user, the clustering region closest to the facial feature is obtained from the pre-established classification feature pool, including: The identity of the next user is determined according to the facial image of the next user, and based on the classification feature pool associated with the identity of the next user, a clustering region closest to the facial feature of the next user is obtained.

5. The facial emotion recognition method according to claim 4, characterized in that: The pre-established classification feature pool includes clustered areas of facial features corresponding to different emotions of different types of people; The step of obtaining a classification feature pool associated with the identity of the user according to the facial features of the user and the closest clustering region includes: A target clustering region is obtained from a pre-established classification feature pool; the type of personnel corresponding to the target clustering region is the same as the type of personnel corresponding to the closest clustering region; Based on the facial features of the user and the target clustering region, a classification feature pool associated with the identity of the user is obtained.

6. The facial emotion recognition method according to claim 5, characterized in that: Therefore, according to the facial features of the user and the closest clustering area, a classification feature pool associated with the identity of the user is obtained, which also includes: Periodically updating a classification feature pool associated with the identity of the user according to the facial features of the user and the closest clustering region; After determining the identity of the user, determine whether the time interval between the current identification of the identity and the last identification of the same identity is greater than a preset interval threshold. If the time interval is greater than the preset interval threshold, update the classification feature pool associated with the identity of the user; otherwise, do not update the classification feature pool associated with the identity of the user.

7. The facial emotion recognition method according to claim 1, characterized in that: The step of obtaining the cluster region closest to the facial feature from the pre-established classification feature pool includes: Determine the facial features in each cluster area in the classification feature pool, whose facial features have the smallest difference from the facial features of the user, as the most similar facial features; The cluster region corresponding to the closest facial feature is determined as the closest cluster region.

8. The facial emotion recognition method according to claim 1, characterized in that: The step of obtaining the cluster region closest to the facial feature from the pre-established classification feature pool includes: Based on the vector value of the facial feature and the vector value of the cluster region in the classification feature pool, obtaining the Euclidean distance between the facial feature and different cluster regions in the classification feature pool; The clustering area corresponding to the minimum Euclidean distance among the Euclidean distances is used as the closest clustering area.

9. The method for facial emotion recognition according to any one of claims 1 to 8, characterized in that: Before obtaining the clustering region closest to the facial feature from the pre-established classification feature pool, the method further includes: Collecting a plurality of facial images; the plurality of facial images include facial images of different emotions; Dividing the multiple facial images respectively, and extracting features from each facial region after the multiple facial images are divided, to obtain corresponding facial features; Multiple facial features of the same emotion type are clustered to obtain multiple cluster regions under the same emotion, and a classification feature pool is constructed based on all the obtained cluster regions.

10. The facial emotion recognition method according to claim 9, characterized in that: The collecting of multiple facial images includes: collecting facial images of people in different head postures and with different emotions.

11. The method for facial emotion recognition according to any one of claims 1 to 8, characterized in that: The determining the emotion of the user based on the most similar cluster region includes: Determining the emotion corresponding to the closest cluster region; The emotion corresponding to the most similar cluster region is used as the emotion of the user.

12. The facial emotion recognition method according to claim 1, characterized in that: The pre-established classification feature pool includes clustered areas of facial features corresponding to different emotions of different types of people; after obtaining the facial image of the user, it also includes: Determining the person type of the user based on the facial image of the user; Accordingly, obtaining the clustering region closest to the facial feature from the pre-established classification feature pool includes: According to the person type of the user, a clustering area whose person type is the same as that of the user is determined in a pre-established classification feature pool, and a clustering area closest to the facial feature is determined from among the clustering areas whose person type is the same as that of the user.

13. A facial emotion recognition device, characterized in that: include: An acquisition module, used to acquire a user's facial image; An extraction module, used to divide the facial image and extract features from each divided facial area to obtain facial features of the user; An acquisition module, used to obtain the cluster region closest to the facial feature from a pre-established classification feature pool; wherein the classification feature pool includes cluster regions of facial features corresponding to different emotions; A determination module is used to determine the emotion of the user based on the most similar cluster area.

14. A non-volatile computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the facial emotion recognition method according to any one of claims 1 to 12 is implemented.

15. A vehicle comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, wherein: When the processor executes the computer program, the facial emotion recognition method as described in any one of claims 1 to 12 is implemented.

Images