A method and device for facial expression recognition based on local image data fusion

Abstract

The present application provides an expression recognition method and device based on local image data fusion, including: obtaining a grayscale image and a depth image containing a target face; preprocessing the grayscale image to further determine the image mouth length of the target face; comparing the image mouth length with the reference mouth length corresponding to the target face to determine the comparison result; based on the comparison result, determining a precise fusion area and a rough fusion area, wherein the precise fusion area and the rough fusion area both belong to partial areas of the target face; based on the precise fusion area and the rough fusion area, performing image fusion with differentiated accuracy on the preprocessed grayscale image and the preprocessed depth image; performing expression recognition on the fused face image to determine the expression of the target face at that moment. This can improve the accuracy and stability of facial expression recognition.

Description

Technical Field

[0001] The present application relates to the field of face recognition technology, and in particular to an expression recognition method and device based on local image data fusion. Background Art

[0002] In the field of human-computer interaction, accurate recognition of facial expressions can help machines and computers identify possible human cognition and thinking patterns, thereby accurately understanding human emotions and improving the quality of human-computer interaction. Facial expression data is high-dimensional and complex data, and the data describing facial expressions has a large spatial dimension.

[0003] At present, the expression recognition method based on grayscale facial images is easily affected by changes in lighting; while the expression recognition method based on depth facial images is easily affected by factors such as data collection accuracy and dynamic changes (such as dynamic changes in facial features and changes in position), which hinders the stability and accuracy of the facial expression recognition system. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method and device for expression recognition based on local image data fusion to improve the accuracy and stability of facial expression recognition.

[0005] In order to achieve the above purpose, the embodiments of the present application are implemented in the following manner:

[0006] In a first aspect, an embodiment of the present application provides an expression recognition method based on local image data fusion, comprising: obtaining a grayscale image and a depth image containing a target face, wherein the grayscale image and the depth image are obtained after photographing and processing the target face at the same time; preprocessing the grayscale image, and detecting the preprocessed grayscale image to determine the image mouth length of the target face; comparing the image mouth length with a reference mouth length corresponding to the target face to determine a comparison result, wherein the comparison result is any one of a first result, a second result, and a third result, and the first result indicates that the image The mouth length is greater than the reference mouth length, the second result indicates that the mouth length of the image is consistent with the reference mouth length, and the third result indicates that the mouth length of the image is less than the reference mouth length; based on the comparison result, a precise fusion area and a rough fusion area are determined, wherein the precise fusion area and the rough fusion area both belong to partial areas of the target face; based on the precise fusion area and the rough fusion area, the preprocessed grayscale image and the preprocessed depth image are fused with differentiated accuracy; expression recognition is performed on the fused face image to determine the expression of the target face at that moment.

[0007] In an embodiment of the present application, a grayscale image and a depth image containing a target face taken (after processing) at the same time are obtained, and the grayscale image is preprocessed to determine the image mouth length of the target face. Based on the comparison result of the image mouth length and the reference mouth length, the corresponding precise fusion area and rough fusion area are determined. The precise fusion area and the rough fusion area are then used to perform image fusion with differentiated accuracy on the preprocessed grayscale image and the preprocessed depth image; expression recognition is performed on the fused face image to determine the expression of the target face at that moment. This method uses the image mouth length identified in the grayscale image as the basis for determining the precise fusion area and the rough fusion area before fusing the grayscale image and the depth image. Since the mouth lengths are different in different expressions, they correspond to various types of expressions. However, in expressions with elongated mouth corners and those with unelongated mouth corners, different key judgment areas can be used as the key to expression recognition. For example, when the mouth corners are elongated, the face may be smiling (happy), crying (sad), etc., and the smiling expression is accompanied by changes in eye wrinkles and cheek muscles, while when crying, the changes in eyebrows and eyelids are more obvious, with more distinct characteristics. Therefore, by using the image mouth length as the basis for determining the precise fusion area and the rough fusion area, the local image can be fused in a targeted manner, and relatively insignificant areas can be fused roughly, thereby greatly improving the accuracy of facial expression recognition. The combination of grayscale images and depth images can form a complementary effect, which is conducive to further improving the accuracy and stability of facial expression recognition.

[0008] In combination with the first aspect, in a first possible implementation manner of the first aspect, preprocessing the grayscale image includes: determining a line connecting the center points of both eyes of the target face in the grayscale image, and adjusting the angle of the grayscale image based on the line connecting the center points of both eyes to straighten the target face; and determining a nose tip vertex from the grayscale image; correspondingly, the expression recognition method based on local image data fusion also includes: preprocessing the depth image, specifically including:

[0009] Determine a line connecting the center points of both eyes of the target face in the depth image, and adjust the angle of the depth image based on the line connecting the center points of both eyes to straighten the target face; and determine a nose tip vertex from the depth image.

[0010] In this implementation, the target face is aligned by determining the line connecting the center points of both eyes of the target face in the grayscale image (depth image), and adjusting the angle of the grayscale image (depth image) based on the line connecting the center points of both eyes; and the nose tip vertex is determined from the grayscale image (depth image). By determining these points, the grayscale image and the depth image can be well matched, which is conducive to ensuring the accuracy of fusion. In addition, before fusion, the target face is aligned using the line connecting the center points of both eyes, which can further ensure the accuracy of image fusion.

[0011] In combination with the first possible implementation manner of the first aspect, in a second possible implementation manner of the first aspect, based on the precise fusion area and the rough fusion area, the preprocessed grayscale image and the preprocessed depth image are subjected to image fusion with differentiated accuracy, including: establishing a one-to-one mapping relationship between the center points of both eyes and the apex of the nose tip of the target face in the preprocessed grayscale image and the center points of both eyes and the apex of the nose tip of the target face in the preprocessed depth image; extracting features from the precise fusion area of ​​the target face in the preprocessed grayscale image, and extracting features from the precise fusion area of ​​the target face in the preprocessed depth image; The image feature points extracted based on the grayscale image are matched with the depth feature points extracted based on the depth image, and a mapping relationship is established between the matched image feature points and the depth feature points to achieve registration of the grayscale image and the depth image in the precise fusion area; the contour of the rough fusion area of ​​the target face in the preprocessed grayscale image is determined, and the contour of the rough fusion area of ​​the target face in the preprocessed depth image is determined; the image area contour determined based on the grayscale image is matched with the depth area contour determined based on the depth image, and a mapping relationship is established between the matched image area contour and the image area contour to achieve registration of the grayscale image and the depth image in the rough fusion area.

[0012] In this implementation, the eye center points and nose tip vertex of the target face in the preprocessed grayscale image are mapped to the eye center points and nose tip vertex of the target face in the preprocessed depth image, which can be used as a reference for image fusion of the grayscale image and the depth image. For image fusion in the precise fusion area, feature extraction is performed on the precise fusion area of ​​the target face in the preprocessed grayscale image, and feature extraction is performed on the precise fusion area of ​​the target face in the preprocessed depth image; then, the image feature points extracted based on the grayscale image are matched with the depth feature points extracted based on the depth image, and a mapping relationship is established between the matched image feature points and the depth feature points, so as to achieve registration of the grayscale image and the depth image in the precise fusion area. Thus, image fusion in the precise fusion area can be performed by matching multiple feature points, thereby ensuring the accuracy of image fusion. As for the registration of the rough fusion area, the contour of the rough fusion area of ​​the target face in the preprocessed grayscale image and the contour of the rough fusion area of ​​the target face in the preprocessed depth image are mainly used to achieve image fusion in the area after contour matching, which can greatly improve the image fusion efficiency of the rough fusion area. Therefore, this image fusion method with differentiated precision can effectively ensure the accuracy and efficiency of image fusion, which is conducive to improving the stability and accuracy of expression recognition, and can also improve the operating efficiency of the expression recognition method.

[0013] In combination with the second possible implementation manner of the first aspect, in a third possible implementation manner of the first aspect, the target face includes a forehead area, a brow area, an eye socket area, an ear area, a temporal area, a cheek area, a nose area, a mouth area, and a jaw area, and based on the comparison result, a precise fusion area and a rough fusion area are determined, including: if the comparison result is the first result, the brow area, the eye socket area, the cheek area, and the mouth area are determined to be precise fusion areas, and the rest are rough fusion areas; if the comparison result is the second result, the forehead area, the brow area, the eye socket area, the cheek area, and the mouth area are determined to be precise fusion areas, and the rest are rough fusion areas; if the comparison result is the third result, the brow area, the eye socket area, the mouth area, and the jaw area are determined to be precise fusion areas, and the rest are rough fusion areas.

[0014] In this implementation, the target face is divided into the forehead area, eyebrow area, eye socket area, ear area, temporal area, cheek area, nose area, mouth area, and jaw area. If the comparison result is the first result (the image mouth length is greater than the reference mouth length), the eyebrow area, eye socket area, cheek area, and mouth area are determined to be precise fusion areas, and the rest are rough fusion areas. Since the image mouth length is greater than the reference mouth length, the expression of the face at this moment is usually accompanied by features of the corners of the eyes, eyelids, eyebrows, cheeks, etc. For example, when smiling, the corners of the mouth are raised, usually accompanied by the appearance of eye wrinkles, and the cheeks are partially raised, causing contour changes and depth information changes. When crying in sadness, it is usually accompanied by eyebrows changing into a certain shape (for example, the tail of the eyebrows drooping), eyelids converging, and the mouth presenting an overall convex arc. Therefore, when the comparison result is the first result, the eyebrow area, eye socket area, cheek area, and mouth area are determined as precise fusion areas, and the rest are rough fusion areas. The precise fusion areas determined in this way are helpful for the expression recognition of the subsequent fused face image. Similarly, if the comparison result is the second result (the image mouth length is consistent with the reference mouth length, and the consistency here means that the length difference is within the set range), the forehead area, eyebrow area, eye socket area, cheek area, and mouth area are determined as precise fusion areas, and the rest are rough fusion areas. Since there is more uncertainty in the category of facial expression when the image mouth length is consistent with the reference mouth length, multiple key parts can be determined as precise fusion areas, such as calm expression, surprise, doubt, etc., and the specific expression can be jointly determined by the characteristics of multiple areas such as the forehead (for example, whether there are lines), eyebrows (whether the eyebrows are raised), eyes, and mouth shape to ensure the accuracy of expression recognition. If the comparison result is the third result (the image mouth length is less than the reference mouth length), the eyebrow area, eye socket area, mouth area, and jaw area are determined as precise fusion areas, and the rest are rough fusion areas. In this way, the areas that need precise fusion can be selected based on the expression categories that may appear when the image mouth length is less than the reference mouth length. For example, when the expression is angry, it is often accompanied by closed lips, a pursed mouth, eyebrows pressed down and eyebrows raised, and eyes tilted to the upper eyelids. The expression of unwillingness is usually accompanied by the tightening of the lower jaw, and a small naked eye appears on the skin on the surface of the lower jaw. Therefore, in this way, determining the corresponding precise fusion area for different comparison results can help accurately identify facial expressions.

[0015] In combination with the third possible implementation manner of the first aspect, in a fourth possible implementation manner of the first aspect, the comparison result is the first result, and expression recognition is performed on the fused facial image, including: obtaining eyebrow features of an eyebrow area, mouth features of an oral area, eye features of an eye socket area, and cheek features of a cheek area in the facial image, wherein the eyebrow features include eyebrow contour and eyebrow depth information, the mouth features include mouth length and lip contour, the eye features include eyelid contour and eye corner lines, and the cheek features include cheek contour and cheek depth information; based on the eyebrow features, the mouth features, the eye features, and the cheek features, determine the expression state of the target face at that moment.

[0016] In combination with the third possible implementation manner of the first aspect, in a fifth possible implementation manner of the first aspect, the comparison result is the second result, and expression recognition is performed on the fused facial image, including: obtaining forehead features of the forehead area, eyebrow features of the eyebrow area, mouth features of the mouth area, eye features of the eye socket area, and cheek features of the cheek area in the facial image, wherein the forehead features include forehead contour and forehead wrinkle information, the eyebrow features include eyebrow contour and eyebrow depth information, the mouth features include mouth length and lip contour, the eye features include eyelid contour and eye corner wrinkles, and the cheek features include cheek contour and cheek depth information; based on the forehead features, the eyebrow features, the mouth features, the eye features, and the cheek features, determine the expression state of the target face at that moment.

[0017] In combination with the third possible implementation manner of the first aspect, in a sixth possible implementation manner of the first aspect, the comparison result is the third result, and expression recognition is performed on the fused facial image, including: obtaining eyebrow features of the eyebrow area, mouth features of the mouth area, eye features of the eye socket area, and jaw features of the jaw area in the facial image, wherein the eyebrow features include eyebrow contour and eyebrow depth information, the mouth features include mouth length and lip contour, the eye features include eyelid contour and eye corner lines, and the jaw features include jaw contour and jaw texture; based on the eyebrow features, the mouth features, the eye features, and the jaw features, determine the expression state of the target face at that moment.

[0018] In a second aspect, an embodiment of the present application provides an expression recognition device based on local image data fusion, comprising: an image acquisition unit, used to acquire a grayscale image and a depth image containing a target face, wherein the grayscale image and the depth image are obtained after photographing and processing the target face at the same time; a processing unit, used to preprocess the grayscale image, and detect the preprocessed grayscale image to determine the image mouth length of the target face; the processing unit is also used to compare the image mouth length with a reference mouth length corresponding to the target face to determine a comparison result, wherein the comparison result is any one of a first result, a second result and a third result, and the first result indicates that the image mouth The length is greater than the reference mouth length, the second result indicates that the image mouth length is consistent with the reference mouth length, and the third result indicates that the image mouth length is less than the reference mouth length; the processing unit is also used to determine a precise fusion area and a rough fusion area based on the comparison result, wherein the precise fusion area and the rough fusion area both belong to partial areas of the target face; the processing unit is also used to perform image fusion with differentiated accuracy on the preprocessed grayscale image and the preprocessed depth image based on the precise fusion area and the rough fusion area; the processing unit is also used to perform expression recognition on the fused facial image to determine the expression of the target face at that moment.

[0019] In a third aspect, an embodiment of the present application provides a storage medium, which includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the expression recognition method based on local image data fusion described in any one of the first aspect or possible implementation methods of the first aspect.

[0020] In a fourth aspect, an embodiment of the present application provides an electronic device comprising a memory and a processor, wherein the memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions, and when the program instructions are loaded and executed by the processor, the expression recognition method based on local image data fusion described in any one of the first aspect or the possible implementation methods of the first aspect is implemented.

[0021] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A flowchart of an expression recognition method based on local image data fusion provided in an embodiment of the present application.

[0024] Figure 2 A schematic diagram of the region division of the target face provided in an embodiment of the present application.

[0025] Figure 3 A schematic diagram of an expression recognition device based on local image data fusion is provided for an embodiment of the present application.

[0026] Figure 4 A structural block diagram of an electronic device provided in an embodiment of the present application.

[0027] Icons: 1-forehead area; 2-eyebrow area; 3-nose area; 4-mouth area; 5-jaw area; 6-cheek area; 7-orbital area; 8-ear area; 9-temporal area; 10-expression recognition device based on local image data fusion; 11-image acquisition unit; 12-processing unit; 20-electronic device; 21-memory; 22-communication module; 23-bus; 24-processor. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0029] In this embodiment, the expression recognition method based on local image data fusion can be performed by an electronic device, and the electronic device can be a terminal, such as a laptop, a personal computer, a tablet computer, a smart phone, etc.; it can also be a server, such as a cloud server, a server cluster, etc. Of course, the electronic device can also be a dedicated device, such as a targeted developed device, including a processing device and an image acquisition device, and the image acquisition device includes a set real-time image acquisition component (such as a high-definition camera) and a depth image acquisition component (such as a depth camera), which are respectively used to capture real-time images (corresponding grayscale images can be obtained after grayscale processing) and depth images in real time and synchronously.

[0030] See also Figure 1 , Figure 1 A flowchart of an expression recognition method based on local image data fusion provided in an embodiment of the present application.

[0031] In this embodiment, the expression recognition method based on local image data fusion may include step S10, step S20, step S30, step S40, step S50, and step S60.

[0032] In order to improve the quality of human-computer interaction, facial expressions may be recognized during the interaction process, and based on this, the electronic device may execute step S10.

[0033] Step S10: obtaining a grayscale image and a depth image containing a target face, wherein the grayscale image and the depth image are obtained by photographing the target face at the same time.

[0034] In this embodiment, the electronic device can obtain the real-time image captured by the real-time image acquisition component, and obtain the grayscale image after grayscale processing of the real-time image; and the electronic device can obtain the depth image captured by the depth image acquisition component at the same time. The grayscale image and the depth image both contain the target face.

[0035] After acquiring the grayscale image and the depth image, the electronic device may execute step S20.

[0036] Step S20: preprocessing the grayscale image, and detecting the preprocessed grayscale image to determine the mouth length of the target face.

[0037] In this embodiment, the electronic device may pre-process the grayscale image.

[0038] Exemplarily, the electronic device may determine a line connecting the center points of both eyes of the target face in the grayscale image, and adjust the angle of the grayscale image based on the line connecting the center points of both eyes to straighten the target face.

[0039] Specifically, the electronic device can perform contour detection on the grayscale image to determine the position of the eyes of the target face, and based on the center point of each eye, can determine the line connecting the center points of the eyes of the target face in the grayscale image. Then, by adjusting the line connecting the center points of the eyes to be horizontal (taking the horizontal direction as an example, but not limited to this), the grayscale image can be adjusted in angle so that the target face remains in the correct position. Then, the electronic device can detect the contour of the nose from the grayscale image and determine the vertex of the nose tip.

[0040] Similarly, electronic devices can also pre-process depth images.

[0041] Exemplarily, the electronic device may determine a line connecting the center points of both eyes of the target face in the depth image, and adjust the angle of the depth image based on the line connecting the center points of both eyes to align the target face.

[0042] Specifically, the electronic device can also perform contour detection on the depth image to determine the position of the eyes of the target face. Based on the center point of each eye, the line connecting the center points of the eyes of the target face in the depth image can be determined. Then, by adjusting the line connecting the center points of the eyes to be horizontal (taking the horizontal direction as an example, but not limited to this), the angle of the depth image can be adjusted so that the target face remains in the right position. Then, the electronic device can detect the tip of the nose from the depth image (the depth of the tip of the nose is shallower than the depth of other parts of the target face and is closer to the depth camera).

[0043] By determining the line connecting the center points of the eyes of the target face in the grayscale image (depth image), and adjusting the angle of the grayscale image (depth image) based on the line connecting the center points of the eyes, the target face is straightened; and the nose tip vertex is determined from the grayscale image (depth image). By determining these points, the grayscale image and the depth image can be well matched, which is conducive to ensuring the accuracy of fusion. In addition, before fusion, the target face is straightened using the line connecting the center points of the eyes to further ensure the accuracy of image fusion.

[0044] After preprocessing the grayscale image, the electronic device can detect the preprocessed grayscale image to determine the length of the mouth of the target face. Here, the grayscale image is used to determine the length of the mouth because this method is more stable, because the skin color of the mouth and the surrounding facial skin is obviously different, which is more conducive to contour detection. Of course, in some other possible implementations, the depth image can also be used for mouth detection, which is not limited here.

[0045] The method for determining the mouth length of the target face image may be to take the distance between the mouth corners as the mouth length of the image, which is not limited here. Specifically, the mouth contour may be detected to obtain the mouth corner distance between the mouth corners; then, the distance between the inner corners of the eyes of the target face in the grayscale image is detected as the inner corner distance, and then the mouth corner distance is divided by the inner corner distance to obtain the image mouth length (which is a relative value).

[0046] After determining the mouth length of the target face image, the electronic device may execute step S30.

[0047] Step S30: Compare the image mouth length with the reference mouth length corresponding to the target face to determine a comparison result, wherein the comparison result is any one of a first result, a second result, and a third result, wherein the first result indicates that the image mouth length is greater than the reference mouth length, the second result indicates that the image mouth length is consistent with the reference mouth length, and the third result indicates that the image mouth length is less than the reference mouth length.

[0048] In this embodiment, the electronic device may compare the mouth length of the image with the reference mouth length corresponding to the target face to determine a comparison result.

[0049] The reference mouth length here can be determined based on the image of the target face when it is in a calm expression. For example, in the image of the target face when it is in a calm expression, the distance between the inner corners of the two eyes of the target face is used as a reference to determine the mouth length at this time as the reference mouth length (this reference mouth length is a relative value, not an absolute value. For example, if the distance between the inner corners of the eyes is recorded as 1, then the reference mouth length may be 1.4 or 0.8, which varies from person to person, but it is a stable value for an individual).

[0050] Specifically, the comparison between the image mouth length and the reference mouth length may be performed by determining whether the difference between the image mouth length and the reference mouth length exceeds a set value (eg, 5%).

[0051] If the image mouth length is longer than the reference mouth length by a set value or more, then it can be determined that the image mouth length is greater than the reference mouth length, and thus the comparison result can be determined to be the first result.

[0052] If the difference between the image mouth length and the reference mouth length is within the set value, then it can be determined that the image mouth length is consistent with the reference mouth length, and therefore the comparison result can be determined to be the second result.

[0053] If the image mouth length is shorter than the reference mouth length by a set value or more, then it can be determined that the image mouth length is smaller than the reference mouth length, and thus the comparison result can be determined to be the third result.

[0054] After determining the comparison result between the image mouth length and the reference mouth length, the electronic device may execute step S40.

[0055] Step S40: Based on the comparison result, a precise fusion region and a rough fusion region are determined, wherein the precise fusion region and the rough fusion region both belong to partial regions of the target face.

[0056] In order to accurately recognize facial expressions, the target face can be divided into multiple areas. Figure 2 The target face may include: forehead area 1, eyebrow area 2, eye socket area 7, ear area 8, temporal area 9, cheek area 6, nose area 3, mouth area 4, and jaw area 5.

[0057] In this embodiment, the electronic device can determine corresponding precise fusion areas and rough fusion areas according to different comparison results.

[0058] Exemplarily, for the case where the comparison result is the first result:

[0059] The electronic device can determine the eyebrow area, eye socket area, cheek area, and mouth area as precise fusion areas, and the rest (forehead area, ear area, temporal area, nose area, and jaw area) as rough fusion areas.

[0060] Since the mouth length of the image is greater than the reference mouth length, the facial expression at this moment is usually accompanied by features of the corners of the eyes, eyelids, eyebrows, cheeks, etc. For example, when smiling, the corners of the mouth are raised, usually accompanied by the appearance of eye wrinkles, and the cheeks are partially raised, causing contour changes and depth information changes. When crying in sadness, the eyebrows are usually transformed into a certain shape (for example, the tail of the eyebrows droops), the eyelids converge, and the mouth is generally convex. Therefore, when the comparison result is the first result, the eyebrow area, eye socket area, cheek area, and mouth area are determined as precise fusion areas, and the rest are rough fusion areas. The precise fusion areas determined in this way are helpful for the expression recognition of the subsequent fused face images.

[0061] Exemplarily, for the case where the comparison result is the second result:

[0062] The electronic device can determine the forehead area, eyebrow area, eye socket area, cheek area, and mouth area as precise fusion areas, and the rest (ear area, temporal area, nose area, and jaw area) as rough fusion areas.

[0063] When the mouth length in the image is consistent with the reference mouth length, there is more uncertainty in the category of facial expression. Therefore, multiple key parts can be identified as precise fusion areas, such as calm expression, surprise, confusion, etc. The specific expression can be determined jointly through the features of multiple areas such as the forehead (for example, whether there are wrinkles), eyebrows (whether the eyebrows are raised), eyes, and mouth shape to ensure the accuracy of expression recognition.

[0064] Exemplarily, for the case where the comparison result is the third result:

[0065] The electronic device can determine the eyebrow area, eye socket area, mouth area, and jaw area as precise fusion areas, and the rest (forehead area, ear area, temporal area, cheek area, and nose area) as rough fusion areas.

[0066] In this way, we can focus on selecting the areas that need to be accurately fused for the expression categories that may appear when the mouth length of the image is shorter than the reference mouth length. For example, when the expression is angry, it is often accompanied by closed lips, pursed mouth, depressed eyebrows, raised eyebrows, and eyeballs tilted to the upper eyelids. And the expression of resignation is usually accompanied by tightening of the lower jaw, and a small naked eye appears on the skin on the surface of the lower jaw. Therefore, in this way, determining the corresponding accurate fusion area for different comparison results can help accurately identify facial expressions.

[0067] After determining the precise fusion area and the rough fusion area, the electronic device may execute step S50.

[0068] Step S50: Based on the precise fusion area and the rough fusion area, the preprocessed grayscale image and the preprocessed depth image are fused with different precisions.

[0069] In this embodiment, the electronic device may perform image fusion with differentiated precision on the preprocessed grayscale image and the preprocessed depth image based on the corresponding precise fusion area and the rough fusion area.

[0070] For example, the electronic device can establish a one-to-one mapping relationship between the center points of both eyes and the tip of the nose of the target face in the preprocessed grayscale image and the center points of both eyes and the tip of the nose of the target face in the preprocessed depth image. The mapping relationship established in this way can be used as a reference for subsequent image fusion with differentiating accuracy of the preprocessed grayscale image and the preprocessed depth image.

[0071] First, for the image fusion of the preprocessed grayscale image and the preprocessed depth image in the precise fusion area:

[0072] The electronic device can perform feature extraction on the precise fusion area of ​​the target face in the preprocessed grayscale image, and can perform feature extraction on the precise fusion area of ​​the target face in the preprocessed depth image. Here, the feature extraction of the precise fusion area of ​​the target face in the grayscale image and the precise fusion area of ​​the target face in the depth image adopts a consistent feature extraction method, thereby obtaining two types of feature point sets (respectively sets of image feature points and depth feature points) containing multiple feature points.

[0073] Then, the image feature points extracted based on the grayscale image are matched with the depth feature points extracted based on the depth image. In addition to the similarity matching of the feature points, the main matching method can also be coordinated with the corresponding relationship between the positions of the two feature points in the image (it can be understood as the corresponding relationship between the image coordinates of a certain precise fusion area in the grayscale image and the image coordinates of the precise fusion area in the depth image. The mapping relationship established based on the center points of both eyes and the vertex of the nose tip can be used as a reference. The adjustment parameters are determined by the image coordinate correspondence between the contour of the grayscale image and the contour of the depth image in the precise fusion area, and the high-precision image coordinate correspondence is calculated. It is equivalent to that each pair of matched feature points acts as a point position to adjust the image fusion in the precise fusion area, so that the image fusion in the precise fusion area is more accurate). In this way, high-precision matching of image feature points and depth feature points can be achieved, so that a mapping relationship is established between the matched image feature points and the depth feature points, and high-precision image fusion of the preprocessed grayscale image and the preprocessed depth image in the precise fusion area is achieved (that is, the registration of the grayscale image and the depth image in the precise fusion area is achieved).

[0074] As for the image fusion of the preprocessed grayscale image and the preprocessed depth image in the rough fusion area:

[0075] The electronic device can determine the contour of the rough fusion area of ​​the target face in the preprocessed grayscale image, and determine the contour of the rough fusion area of ​​the target face in the preprocessed depth image. Here, the contour detection of the rough fusion area of ​​the target face in the grayscale image and the rough fusion area of ​​the target face in the depth image can adopt a differentiated contour detection method, so as to quickly and accurately realize the contour detection of the rough fusion area. For example, for the contour detection of the rough fusion area of ​​the target face in the grayscale image, the edge detection operator can be used to detect the target contour; and for the contour detection of the rough fusion area of ​​the target face in the depth image, the edge detection operator can also be used to detect the target contour. Since the accuracy requirement of the rough fusion area is relatively low, the priority is to improve the processing efficiency. Therefore, a simpler detection method can be used to reduce the processing time to a certain extent and improve the detection efficiency of the rough fusion area.

[0076] Then, the electronic device can match the image area contour determined based on the grayscale image with the depth area contour determined based on the depth image, and use the mapping relationship established based on the center points of both eyes and the apex of the nose as a reference, and combine the contour similarity and the coordinate difference value (the smaller the coordinate difference, the better) to achieve the matching of the image area contour determined based on the grayscale image and the depth area contour determined based on the depth image. Using the mapping relationship established based on the center points of both eyes and the apex of the nose as a reference, the accuracy of the matching of the image area contour determined based on the grayscale image and the depth area contour determined based on the depth image can be guaranteed to a large extent. Based on this, the electronic device can further establish a mapping relationship between the matched image area contour and the image area contour to achieve the registration of the grayscale image and the depth image in the rough fusion area.

[0077] The eye center points and nose tip vertex of the target face in the preprocessed grayscale image are mapped to the eye center points and nose tip vertex of the target face in the preprocessed depth image, and the mapping relationship is established one by one, which can be used as a reference for image fusion of grayscale image and depth image. For the image fusion of the precise fusion area, the feature extraction of the precise fusion area of ​​the target face in the preprocessed grayscale image and the feature extraction of the precise fusion area of ​​the target face in the preprocessed depth image are performed; then the image feature points extracted based on the grayscale image are matched with the depth feature points extracted based on the depth image, and the mapping relationship between the matched image feature points and the depth feature points is established to achieve the registration of the grayscale image and the depth image in the precise fusion area. Therefore, the image fusion of the precise fusion area can be performed by matching multiple feature points, thereby ensuring the accuracy of image fusion. For the registration of the rough fusion area, the contour of the rough fusion area of ​​the target face in the preprocessed grayscale image and the contour of the rough fusion area of ​​the target face in the preprocessed depth image are mainly used to achieve image fusion in the area after contour matching, which can greatly improve the image fusion efficiency of the rough fusion area. Therefore, this image fusion method with differentiated precision can effectively ensure the precision and efficiency of image fusion, which is beneficial to improving the stability and accuracy of expression recognition, and can also improve the operating efficiency of the expression recognition method.

[0078] By fusing the preprocessed grayscale image and the preprocessed depth image in the precise fusion area and the preprocessed depth image in the rough fusion area, image fusion with differentiated precision of the preprocessed grayscale image and the preprocessed depth image is achieved, which can take into account the accuracy, reliability and operation efficiency of expression recognition.

[0079] After achieving image fusion with differentiated accuracy for the preprocessed grayscale image and the preprocessed depth image, the electronic device may execute step S60.

[0080] Step S60: performing expression recognition on the fused face image to determine the expression of the target face at that moment.

[0081] In this embodiment, the electronic device can perform expression recognition on the fused facial image.

[0082] Exemplarily, when the comparison result is the first result, the electronic device can obtain eyebrow features of the eyebrow area, mouth features of the mouth area, eye features of the eye socket area, and cheek features of the cheek area in the face image, wherein the eyebrow features include eyebrow contour and eyebrow depth information, the mouth features include mouth length and lip contour, the eye features include eyelid contour and eye corner lines, and the cheek features include cheek contour and cheek depth information.

[0083] When the comparison result is the first result, it may be a smile, disdain, sad crying, etc., and smile, crying, disdain and other expressions have many obvious characteristics. For example, when smiling, the corners of the mouth are raised, the mouth is elongated, the cheek muscles are bulging, and wrinkles at the corners of the eyes appear; while when crying sadly, the corners of the mouth are pulled down, the mouth is elongated, the opening of the upper and lower eyelids is reduced, and the tail of the eyebrows is lowered. For these areas with large differences, we obtain their corresponding features, such as eyebrow contour (which can determine the shape and direction of eyebrows) and eyebrow depth information (which can be used to determine whether eyebrows are wrinkled), mouth length (mouth length, mainly used to determine whether the mouth is elongated) and lip contour (which can be used to determine whether the corners of the mouth are upturned or downturned), eyelid contour (which can determine the expansion and contraction of eyelids, and whether the pupil is biased towards the upper eyelid or the lower eyelid) and eye corner lines (which can be used to determine whether there are eye corner lines, i.e. crow's feet), cheek contour (which can be used to determine the degree of change in cheek shape) and cheek depth information (which can determine whether the cheek is raised, mainly the upper part of the cheek, also called the upper cheek area). Then, the acquisition of these features is conducive to accurately obtaining the expression characteristics of the current face.

[0084] Then, the electronic device can determine the expression state of the target face at that moment based on the eyebrow features, mouth features, eye features and cheek features.

[0085] Similarly, in the case where the comparison result is the second result, the electronic device can obtain the forehead features of the forehead area, the eyebrow features of the eyebrow area, the mouth features of the mouth area, the eye features of the eye socket area, and the cheek features of the cheek area in the face image, wherein the forehead features include the forehead contour and forehead wrinkle information, the eyebrow features include the eyebrow contour and eyebrow depth information, the mouth features include the mouth length and the lip contour, the eye features include the eyelid contour and eye corner wrinkles, and the cheek features include the cheek contour and cheek depth information.

[0086] Since the comparison result is the second result, it may be a variety of expressions such as calm, doubtful, serious, etc., and to distinguish these expressions, as many features as possible are needed to distinguish them in order to accurately identify the expression. Therefore, these key areas are selected to obtain their corresponding features, such as forehead contour (mainly revealing whether the forehead is exposed) and forehead wrinkle information (when the forehead is exposed, whether there are forehead wrinkles on the forehead, and when the eyebrows are raised, forehead wrinkles usually appear), eyebrow contour (can determine the shape and direction of the eyebrows) and eyebrow depth information (can be used to determine whether the eyebrows are wrinkled or raised, etc.), mouth length and lip contour (can be used to determine whether the positions of the two corners of the mouth are consistent), eyelid contour (can determine the expansion and contraction of the eyelids, whether the lower eyelids are lifted, and whether the pupil is biased towards the upper eyelid or the lower eyelid) and eye corner wrinkles (can be used to determine whether there are eye corner wrinkles), cheek contour (can be used to determine the degree of change in the shape of the cheeks) and cheek depth information (can determine whether the cheeks are raised and whether they are symmetrical). Then, the acquisition of these features is conducive to accurately obtaining the expression features of the current face.

[0087] Then, the electronic device can determine the expression state of the target face at that moment based on the forehead features, eyebrow features, mouth features, eye features and cheek features.

[0088] Exemplarily, when the comparison result is the third result, the electronic device can obtain eyebrow features of the eyebrow area, mouth features of the mouth area, eye features of the eye socket area, and jaw features of the jaw area in the facial image, wherein the eyebrow features include eyebrow contour and eyebrow depth information, the mouth features include mouth length and lip contour, the eye features include eyelid contour and eye corner lines, and the jaw features include jaw contour and jaw texture.

[0089] Since the comparison result is the third result, the expression is rich, such as pouting, anger, rage, surprise and other expressions. These expressions can be distinguished mainly by eyebrow features, mouth features, eye features and jaw features. For example, the eyebrow contour (which can determine the shape and direction of the eyebrows) and eyebrow depth information (which can be used to determine whether the eyebrows are wrinkled or raised), mouth length (mouth length, mainly used to determine whether the mouth is shortened) and lip contour (which can be used to determine whether the lips are wrinkled), eyelid contour (which can determine the opening of the eyelids, whether the lower eyelids are lifted, and whether there is a biased relationship between the pupil and the upper and lower eyelids) and eye corner lines (which can be used to determine whether there are eye corner lines, i.e. crow's feet), jaw contour (whether the jaw is deformed) and jaw texture (whether there are small meat pit-shaped textures in the form of flakes, usually accompanied by the corners of the mouth being pressed down, often appearing in sad expressions). Then, the acquisition of these features is conducive to accurately obtaining the expression characteristics of the current face.

[0090] Then, the electronic device can determine the expression state of the target face at that moment based on the eyebrow features, mouth features, eye features and jaw features.

[0091] It should be noted that the electronic device can determine the expression state of the target face at that moment by, for example, assigning one expression to multiple features, with different features accounting for different proportions, scoring the target face at that moment by determining whether the feature exists, determining the similarity between the target face and the expression, and determining the expression with the highest similarity as the expression state of the target face at that moment. Of course, static expressions (i.e., the expression state of a face identified through a single image) can also be identified through a trained model, which is not limited here.

[0092] After determining the expression state of the target face at that moment, the electronic device may further determine the expression of the target face at that moment.

[0093] Exemplarily, in order to ensure the accuracy of expression recognition, the electronic device may also analyze the dynamic changes of the expression state of the target face based on the expression state of the target face at multiple previous moments, thereby more accurately identifying the expression of the target face at that moment.

[0094] See also Figure 3 , Figure 3 A schematic diagram of an expression recognition device 10 based on local image data fusion is provided for an embodiment of the present application.

[0095] In this embodiment, the expression recognition device 10 based on local image data fusion may include:

[0096] The image acquisition unit 11 is used to acquire a grayscale image and a depth image containing a target face, wherein the grayscale image and the depth image are obtained by photographing the target face at the same time.

[0097] The processing unit 12 is used to preprocess the grayscale image and detect the preprocessed grayscale image to determine the mouth length of the target face image.

[0098] The processing unit 12 is further used to compare the image mouth length with the reference mouth length corresponding to the target face to determine a comparison result, wherein the comparison result is any one of a first result, a second result and a third result, the first result indicating that the image mouth length is greater than the reference mouth length, the second result indicating that the image mouth length is consistent with the reference mouth length, and the third result indicating that the image mouth length is less than the reference mouth length.

[0099] The processing unit 12 is further configured to determine a precise fusion region and a rough fusion region based on the comparison result, wherein the precise fusion region and the rough fusion region both belong to partial regions of the target face.

[0100] The processing unit 12 is further configured to perform image fusion with differentiated accuracy on the preprocessed grayscale image and the preprocessed depth image based on the precise fusion area and the rough fusion area.

[0101] The processing unit 12 is further used to perform expression recognition on the fused face image to determine the expression of the target face at that moment.

[0102] In this embodiment, the processing unit 12 is further used to determine a line connecting the center points of the two eyes of the target face in the grayscale image, and to adjust the angle of the grayscale image based on the line connecting the center points of the two eyes to straighten the target face; and to determine the nose tip vertex from the grayscale image; the processing unit 12 is further used to preprocess the depth image, specifically to: determine a line connecting the center points of the two eyes of the target face in the depth image, and to adjust the angle of the depth image based on the line connecting the center points of the two eyes to straighten the target face; and to determine the nose tip vertex from the depth image.

[0103] In this embodiment, the processing unit 12 is further used to establish a one-to-one mapping relationship between the center points of both eyes and the tip of the nose of the target face in the preprocessed grayscale image and the center points of both eyes and the tip of the nose of the target face in the preprocessed depth image; perform feature extraction on the precise fusion area of ​​the target face in the preprocessed grayscale image, and perform feature extraction on the precise fusion area of ​​the target face in the preprocessed depth image; match the image feature points extracted based on the grayscale image with the depth feature points extracted based on the depth image, establish a mapping relationship between the matched image feature points and the depth feature points, and achieve registration of the grayscale image and the depth image in the precise fusion area; determine the contour of the rough fusion area of ​​the target face in the preprocessed grayscale image, and determine the contour of the rough fusion area of ​​the target face in the preprocessed depth image; match the image area contour determined based on the grayscale image with the depth area contour determined based on the depth image, establish a mapping relationship between the matched image area contour and the image area contour, and achieve registration of the grayscale image and the depth image in the rough fusion area.

[0104] In this embodiment, the target face includes a forehead area, a brow area, an eye socket area, an ear area, a temporal area, a cheek area, a nose area, a mouth area, and a jaw area. The processing unit 12 is also used to, when the comparison result is the first result, determine that the brow area, the eye socket area, the cheek area, and the mouth area are precise fusion areas, and the rest are rough fusion areas; when the comparison result is the second result, determine that the forehead area, the brow area, the eye socket area, the cheek area, and the mouth area are precise fusion areas, and the rest are rough fusion areas; when the comparison result is the third result, determine that the brow area, the eye socket area, the mouth area, and the jaw area are precise fusion areas, and the rest are rough fusion areas.

[0105] In this embodiment, the comparison result is the first result, and the processing unit 12 is further used to obtain eyebrow features of the eyebrow area, mouth features of the mouth area, eye features of the eye socket area, and cheek features of the cheek area in the face image, wherein the eyebrow features include eyebrow contour and eyebrow depth information, the mouth features include mouth length and lip contour, the eye features include eyelid contour and eye corner lines, and the cheek features include cheek contour and cheek depth information; based on the eyebrow features, the mouth features, the eye features and the cheek features, the expression state of the target face at that moment is determined.

[0106] In this embodiment, the comparison result is the second result, and the processing unit 12 is further used to obtain the forehead features of the forehead area, the eyebrow features of the eyebrow area, the mouth features of the mouth area, the eye features of the eye socket area, and the cheek features of the cheek area in the face image, wherein the forehead features include forehead contour and forehead wrinkle information, the eyebrow features include eyebrow contour and eyebrow depth information, the mouth features include mouth length and lip contour, the eye features include eyelid contour and eye corner wrinkles, and the cheek features include cheek contour and cheek depth information; based on the forehead features, the eyebrow features, the mouth features, the eye features and the cheek features, the expression state of the target face at that moment is determined.

[0107] In this embodiment, the comparison result is the third result, and the processing unit 12 is also used to obtain eyebrow features of the eyebrow area, mouth features of the mouth area, eye features of the eye socket area, and jaw features of the jaw area in the face image, wherein the eyebrow features include eyebrow contour and eyebrow depth information, the mouth features include mouth length and lip contour, the eye features include eyelid contour and eye corner lines, and the jaw features include jaw contour and jaw texture; based on the eyebrow features, the mouth features, the eye features and the jaw features, the expression state of the target face at that moment is determined.

[0108] See also Figure 4 , Figure 4 This is a structural block diagram of an electronic device 20 provided in an embodiment of the present application.

[0109] In this embodiment, the electronic device 20 may be a server, such as a cloud server, a server cluster, etc.; or may be a terminal, such as a personal computer, a smart phone, etc., which is not limited here.

[0110] Exemplarily, the electronic device 20 may include: a communication module 22 connected to the outside world through a network, one or more processors 24 for executing program instructions, a bus 23, and different forms of memory 21, such as a disk, ROM, or RAM, or any combination thereof. The memory 21, the communication module 22, and the processor 24 may be connected through the bus 23.

[0111] Exemplarily, programs are stored in the memory 21. The processor 24 can call and run these programs from the memory 21, so that the expression recognition method based on local image data fusion can be implemented by running the programs.

[0112] An embodiment of the present application further provides a storage medium, which includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the expression recognition method based on local image data fusion described in this embodiment.

[0113] In summary, the embodiments of the present application provide an expression recognition method and device based on local image data fusion, which obtains a grayscale image and a depth image containing a target face taken (after processing) at the same time, and determines the image mouth length of the target face after preprocessing the grayscale image. Based on the comparison result of the image mouth length and the reference mouth length, the corresponding precise fusion area and rough fusion area are determined. The precise fusion area and the rough fusion area are then used to perform image fusion with differentiated accuracy on the preprocessed grayscale image and the preprocessed depth image; expression recognition is performed on the fused face image to determine the expression of the target face at that moment. This method uses the image mouth length identified in the grayscale image as the basis for determining the precise fusion area and the rough fusion area before fusing the grayscale image and the depth image. Since the mouth lengths are different in different expressions, they correspond to various types of expressions. However, in expressions with elongated mouth corners and those with unelongated mouth corners, different key judgment areas can be used as the key to expression recognition. For example, when the mouth corners are elongated, the face may be smiling (happy), crying (sad), etc., and the smiling expression is accompanied by changes in eye wrinkles and cheek muscles, while when crying, the changes in eyebrows and eyelids are more obvious, with more distinct characteristics. Therefore, by using the image mouth length as the basis for determining the precise fusion area and the rough fusion area, the local image can be fused in a targeted manner, and relatively insignificant areas can be fused roughly, thereby greatly improving the accuracy of facial expression recognition. The combination of grayscale images and depth images can form a complementary effect, which is conducive to further improving the accuracy and stability of facial expression recognition.

[0114] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0115] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0116] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A facial expression recognition method based on local image data fusion, characterized in that: include: Acquire a grayscale image and a depth image containing a target face, wherein the grayscale image and the depth image are obtained by photographing the target face at the same time; Preprocessing the grayscale image, and detecting the preprocessed grayscale image to determine the mouth length of the target face image; Comparing the mouth length of the image with the reference mouth length corresponding to the target face to determine a comparison result, wherein the comparison result is any one of a first result, a second result, and a third result, wherein the first result indicates that the mouth length of the image is greater than the reference mouth length, the second result indicates that the mouth length of the image is consistent with the reference mouth length, and the third result indicates that the mouth length of the image is less than the reference mouth length; Based on the comparison result, determining a precise fusion area and a rough fusion area, wherein both the precise fusion area and the rough fusion area belong to partial areas of the target face; Based on the precise fusion area and the rough fusion area, performing image fusion with differentiated accuracy on the preprocessed grayscale image and the preprocessed depth image; Performing expression recognition on the fused face image to determine the expression of the target face at that moment; The target face includes a forehead area, an eyebrow area, an eye socket area, an ear area, a temporal area, a cheek area, a nose area, a mouth area, and a jaw area. Based on the comparison result, a precise fusion area and a rough fusion area are determined, including: If the comparison result is the first result, the eyebrow area, the eye socket area, the cheek area, and the mouth area are determined to be precise fusion areas, and the rest are rough fusion areas; If the comparison result is the second result, the forehead area, the eyebrow area, the eye socket area, the cheek area, and the mouth area are determined to be precise fusion areas, and the rest are rough fusion areas; If the comparison result is the third result, the eyebrow region, the eye socket region, the mouth region, and the jaw region are determined to be precise fusion regions, and the rest are rough fusion regions.

2. The expression recognition method based on local image data fusion according to claim 1, characterized in that: Preprocessing the grayscale image includes: Determine a line connecting the center points of both eyes of the target face in the grayscale image, and adjust the angle of the grayscale image based on the line connecting the center points of both eyes to straighten the target face; and, determining a nose tip vertex from the grayscale image; Correspondingly, the expression recognition method based on local image data fusion further includes: preprocessing the depth image, specifically including: Determine a line connecting the center points of both eyes of the target face in the depth image, and adjust the angle of the depth image based on the line connecting the center points of both eyes to align the target face; And, determining the nose tip vertex from the depth image.

3. The expression recognition method based on local image data fusion according to claim 2, characterized in that: Based on the precise fusion area and the rough fusion area, the preprocessed grayscale image and the preprocessed depth image are fused with different precisions, including: Establish a one-to-one mapping relationship between the center points of both eyes and the apex of the nose tip of the target face in the preprocessed grayscale image and the center points of both eyes and the apex of the nose tip of the target face in the preprocessed depth image; Extracting features from the precise fusion region of the target face in the preprocessed grayscale image, and extracting features from the precise fusion region of the target face in the preprocessed depth image; Match the image feature points extracted from the grayscale image with the depth feature points extracted from the depth image, establish a mapping relationship between the matched image feature points and the depth feature points, and realize the registration of the grayscale image and the depth image in the precise fusion area; Determine the contour of the rough fusion region of the target face in the preprocessed grayscale image, and determine the contour of the rough fusion region of the target face in the preprocessed depth image; The image area contour determined based on the grayscale image is matched with the depth area contour determined based on the depth image, and a mapping relationship is established between the matched image area contour and the image area contour to achieve registration of the grayscale image and the depth image in the rough fusion area.

4. The expression recognition method based on local image data fusion according to claim 1, characterized in that: The comparison result is the first result, and expression recognition is performed on the fused face image, including: Obtaining eyebrow features of the eyebrow region, mouth features of the mouth region, eye features of the eye socket region, and cheek features of the cheek region in the face image, wherein the eyebrow features include eyebrow contour and eyebrow depth information, the mouth features include mouth length and lip contour, the eye features include eyelid contour and eye corner lines, and the cheek features include cheek contour and cheek depth information; Based on the eyebrow features, the mouth features, the eye features and the cheek features, the expression state of the target face at that moment is determined.

5. The expression recognition method based on local image data fusion according to claim 1, characterized in that: The comparison result is the second result, and expression recognition is performed on the fused face image, including: Acquire forehead features of the forehead region, eyebrow features of the eyebrow region, mouth features of the mouth region, eye features of the eye socket region, and cheek features of the cheek region in the face image, wherein the forehead features include forehead contour and forehead wrinkle information, the eyebrow features include eyebrow contour and eyebrow depth information, the mouth features include mouth length and lip contour, the eye features include eyelid contour and eye corner wrinkles, and the cheek features include cheek contour and cheek depth information; Based on the forehead feature, the eyebrow feature, the mouth feature, the eye feature and the cheek feature, the expression state of the target face at that moment is determined.

6. The expression recognition method based on local image data fusion according to claim 1, characterized in that: The comparison result is the third result, and expression recognition is performed on the fused face image, including: Obtaining eyebrow features of the eyebrow region, mouth features of the mouth region, eye features of the eye socket region, and jaw features of the jaw region in the face image, wherein the eyebrow features include eyebrow contours and eyebrow depth information, the mouth features include mouth length and lip contours, the eye features include eyelid contours and eye corner lines, and the jaw features include jaw contours and jaw textures; Based on the eyebrow features, the mouth features, the eye features and the jaw features, the expression state of the target face at that moment is determined.

7. An expression recognition device based on local image data fusion, characterized in that: include: An image acquisition unit, used to acquire a grayscale image and a depth image containing a target face, wherein the grayscale image and the depth image are obtained by photographing the target face at the same time; A processing unit, used to preprocess the grayscale image, and detect the preprocessed grayscale image to determine the mouth length of the target face image; The processing unit is further used to compare the mouth length of the image with the reference mouth length corresponding to the target face to determine a comparison result, wherein the comparison result is any one of a first result, a second result, and a third result, wherein the first result indicates that the mouth length of the image is greater than the reference mouth length, the second result indicates that the mouth length of the image is consistent with the reference mouth length, and the third result indicates that the mouth length of the image is less than the reference mouth length; The processing unit is further used to determine a precise fusion area and a rough fusion area based on the comparison result, wherein the precise fusion area and the rough fusion area both belong to partial areas of the target face; The processing unit is further used to perform image fusion with differentiated accuracy on the preprocessed grayscale image and the preprocessed depth image based on the precise fusion area and the rough fusion area; The processing unit is further used to perform expression recognition on the fused face image to determine the expression of the target face at that moment; Among them, the target face includes a forehead area, a brow area, an eye socket area, an ear area, a temporal area, a cheek area, a nose area, a mouth area, and a jaw area. The processing unit is also used to, when the comparison result is the first result, determine that the brow area, the eye socket area, the cheek area, and the mouth area are precise fusion areas, and the rest are rough fusion areas; when the comparison result is the second result, determine that the forehead area, the brow area, the eye socket area, the cheek area, and the mouth area are precise fusion areas, and the rest are rough fusion areas; when the comparison result is the third result, determine that the brow area, the eye socket area, the mouth area, and the jaw area are precise fusion areas, and the rest are rough fusion areas.

8. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the expression recognition method based on local image data fusion according to any one of claims 1 to 6.

9. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions, and when the program instructions are loaded and executed by the processor, the expression recognition method based on local image data fusion described in any one of claims 1 to 6 is implemented.

Images