Image processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses an image processing method and device, electronic equipment and a storage medium, and the method comprises the steps: detecting a face image, and determining an initial first key point and head posture information in the face image, the initial first key point comprising a key point used for describing a first five sense organ region of a face in the face image; determining an initial second key point according to the initial first key point, wherein the initial second key point comprises key points used for describing a second five sense organ area in the face image; correcting the initial first key point and the initial second key point according to the head posture information to obtain a corrected target first key point and a corrected target second key point; and according to the target first key point and the target second key point, determining a target state corresponding to the second five sense organ area of the human face. According to the embodiment of the invention, the key points in the face image can be corrected, so that the problem that the detection result of the states of the five sense organs is inaccurate due to the rotation of the head is solved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and device, electronic equipment, and storage medium. Background Art

[0002] With the rapid development of application technology, some applications are developing in the direction of fun, such as virtual avatar applications, which can restore the user's facial expressions and movements to the virtual character in the electronic device according to the user's facial expressions and movements.

[0003] In practice, it is found that when detecting the facial expressions of the user, if the user's head rotates or shifts significantly, the mouth will be deformed or blocked, which will lead to inaccurate detection of the tip of the tongue, and further cause the facial expressions restored by the virtual character to deviate from the actual expressions made by the user. Summary of the invention

[0004] The embodiments of the present application disclose an image processing method and device, an electronic device, and a storage medium, which can overcome the problem of inaccurate facial feature state detection results caused by head rotation and improve the accuracy of the determined facial feature state.

[0005] The first aspect of the embodiment of the present application discloses an image processing method, including:

[0006] Detecting a facial image, and determining an initial first key point and head posture information in the facial image, wherein the initial first key point includes a key point in the facial image for describing a first facial feature region of the human face;

[0007] Determine initial second key points according to the initial first key points, wherein the initial second key points include key points in the facial image for describing a second facial feature region of the human face;

[0008] Correcting the initial first key point and the initial second key point according to the head posture information to obtain a corrected target first key point and a target second key point;

[0009] According to the target first key point and the target second key point, a target state corresponding to the second facial feature area of ​​the face is determined.

[0010] A second aspect of an embodiment of the present application discloses an image processing device, comprising:

[0011] A detection unit, configured to detect a face image, and determine an initial first key point and head posture information in the face image, wherein the initial first key point includes a key point in the face image for describing a first facial feature region of the face;

[0012] A first determining unit, configured to determine an initial second key point according to the initial first key point, wherein the initial second key point is a key point in the face image for describing a second facial feature region of the face;

[0013] A correction unit, configured to correct the initial first key point and the initial second key point according to the head posture information to obtain a corrected target first key point and a target second key point;

[0014] The second determination unit is used to determine the target state corresponding to the second facial feature area of ​​the face according to the target first key point and the target second key point.

[0015] A third aspect of the present application discloses an electronic device, including:

[0016] A memory storing executable program code;

[0017] a processor coupled to the memory;

[0018] The processor calls the executable program code stored in the memory to execute the image processing method disclosed in the first aspect of the embodiment of the present application.

[0019] A fourth aspect of an embodiment of the present application discloses a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the image processing method disclosed in the first aspect of an embodiment of the present application.

[0020] Compared with the related art, the embodiments of the present application have the following beneficial effects:

[0021] In the embodiment of the present application, the face detection operation can be performed on the face image first to determine the initial first key point and head posture information used to describe the first facial features area in the face image; and the initial second key point used to describe the second facial features area can be determined based on the initial first key point; further, the initial first key point and the initial second key point can be corrected based on the head posture information to obtain the corrected target first key point and target second key point. It can be seen that in the embodiment of the present application, the first key point and the second key point can be corrected based on the head posture information, so as to solve the problem of inaccurate key points caused by mouth deformation or occlusion due to head rotation or offset, and then more accurate facial features can be determined based on the corrected target first key point and target second key point. In addition, in the embodiment of the present application, only the two-dimensional key points detected from the face image need to be used, and there is no need to introduce additional three-dimensional key points, so as to reduce the amount of calculation of the device and improve the detection efficiency of the facial features. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 It is a schematic diagram of an application scenario disclosed in an embodiment of the present application;

[0024] Figure 2 It is a flowchart of an image processing method disclosed in an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of an initial first key point disclosed in an embodiment of the present application;

[0026] Figure 4 is a schematic diagram of head posture information disclosed in an embodiment of the present application;

[0027] Figure 5 is a schematic diagram of a horizontal correction disclosed in an embodiment of the present application;

[0028] Figure 6 is a schematic diagram of an X-axis-based correction operation disclosed in an embodiment of the present application;

[0029] Figure 7 is a schematic diagram of a correction operation based on the Y-axis disclosed in an embodiment of the present application;

[0030] Figure 8 It is a flowchart of another image processing method disclosed in an embodiment of the present application;

[0031] Fig. 9 is a schematic diagram of another correction operation based on the X-axis disclosed in an embodiment of the present application;

[0032] Fig.10 is a schematic diagram of another correction operation based on the Y-axis disclosed in an embodiment of the present application;

[0033] Fig.11 It is a flowchart of another image processing method disclosed in an embodiment of the present application;

[0034] Fig.12 is a schematic diagram of a target area disclosed in an embodiment of the present application;

[0035] Fig.13 is a schematic diagram of a tongue tip state disclosed in an embodiment of the present application;

[0036] Fig.14 is a structural schematic diagram of an image processing device disclosed in an embodiment of the present application;

[0037] Fig.15 It is a structural schematic diagram of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0039] It should be noted that the terms "first", "second", "third" and "fourth" in the specification and claims of the present application are used to distinguish different objects rather than to describe a specific order. The terms "including" and "having" in the embodiments of the present application and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device including a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0040] The embodiments of the present application disclose an image processing method and device, an electronic device, and a storage medium, which can overcome the problem of inaccurate facial feature state detection results caused by head rotation and improve the accuracy of the determined facial feature state.

[0041] The technical solution of the present application will be described in detail below in conjunction with specific embodiments.

[0042] In order to more clearly explain the image processing method disclosed in the embodiment of the present application, the application scenarios applicable to the method are first introduced. Optionally, the method can be applied to various electronic devices with image processing capabilities, including but not limited to portable electronic devices such as mobile phones and tablet computers; wearable devices such as smart watches and smart bracelets; or desktop electronic devices such as desktop computers and desktop TVs, without limitation.

[0043] See also Figure 1 , Figure 1 is a schematic diagram of an application scenario disclosed in an embodiment of the present application. Optionally, the method can be applied to a scenario in which the user 100 uses a virtual avatar application in the electronic device 110. The virtual avatar application is an application that can collect user images to determine the facial expressions and movements made by the user, and then restore the facial expressions and movements of the user in the virtual character in the electronic device to achieve an interesting interactive effect. Figure 1As shown in , when the user 100 makes an expression in front of the electronic device 110, the virtual character 120 in the electronic device 110 can also make an expression accordingly. It should be further explained that Figure 1 The electronic device 110 is illustrated by taking a mobile phone as an example for the convenience of explanation, which should not be limited to the embodiment of the present application; in addition, the method can also be applied to other scenarios where the state of the five sense organs needs to be determined. Figure 1 This should not be construed as limiting the embodiments of the present application.

[0044] It is found in practice that when the electronic device 110 detects facial expressions of a user, if the user's head rotates or shifts significantly, the mouth will be deformed or blocked, which will lead to inaccurate detection of the facial features, resulting in the facial expressions restored by the virtual character deviating from the actual expressions made by the user.

[0045] In this regard, the embodiment of the present application provides an image processing method, which can be applied to the above-mentioned electronic device 110. Optionally, the electronic device 110 can first perform a face detection operation on the face image to determine the initial first key point and head posture information used to describe the first facial feature area in the face image; and determine the initial second key point used to describe the second facial feature area based on the initial first key point; further, the initial first key point and the initial second key point can be corrected according to the head posture information to obtain the corrected target first key point and target second key point; it can be understood that the target first key point and the target second key point are the corresponding key points when the face is in a straight-looking forward posture, so there is no problem of mouth deformation or occlusion due to head rotation, and the target state corresponding to the facial feature area can be more accurately determined based on the corrected target first key point and the target second key point, thereby overcoming the problem of inaccurate facial feature state detection results due to head rotation. In addition, in the embodiment of the present application, only the two-dimensional key points detected from the face image need to be used, and no additional three-dimensional key points need to be introduced, thereby reducing the amount of calculation of the electronic device 110 and improving the detection efficiency of the facial feature state.

[0046] It should be noted that, in the embodiment of the present application, the first facial features region and the second facial features region may be facial features regions corresponding to different facial features; wherein the facial features region may include an eye region, a nose region, a mouth region, an ear region, an eyebrow region or a tongue region, which is not limited here. In order to facilitate the introduction of the image processing method disclosed in the embodiment of the present application, the following is an introduction using the example that the first facial features region is the mouth region and the second facial features region is the tongue region, which should not be construed as limiting the embodiment of the present application.

[0047] Based on this, the image processing method and device, electronic device, and storage medium of the embodiments of the present application are introduced below.

[0048] See also Figure 2 , Figure 2 1 is a flowchart of an image processing method disclosed in an embodiment of the present application. Optionally, the method can be applied to the above-mentioned electronic device or other execution subjects. The embodiment of the present application is described by taking the electronic device as an example, which should not be limited to the embodiment of the present application. Optionally, the method can include the following steps:

[0049] 202. Perform a face detection operation on the face image to obtain an initial first key point and head posture information corresponding to the face in the face image.

[0050] In the embodiment of the present application, the face image may include a face image captured by the electronic device in real time through a camera, or may be a face image obtained by the electronic device from an image storage space (e.g., a photo album), or may be a face image obtained by the electronic device from other devices (e.g., a surveillance camera, a mobile phone, etc.), which is not limited here. Optionally, when the electronic device is in a state of running an avatar application, the face image may be a face image captured by the electronic device in real time through a camera.

[0051] Optionally, the detection operation performed by the electronic device on the face image may include a key point detection operation and a head posture detection operation. Optionally, the electronic device may use a face detection library to perform a key point detection operation on the face image to obtain an initial first key point corresponding to the face in the face image; wherein the face detection library may include but is not limited to dlib or face_alignment, etc., which are not limited here. Among them, the initial first key point may include a key point in the face image for describing the mouth area of ​​the face.

[0052] like Figure 3 As shown, Figure 3 3 is a schematic diagram of an initial first key point disclosed in an embodiment of the present application. Optionally, the electronic device can input the face image 310 into the face detection library, and then the face detection library can output the face image 320 marked with key points, wherein the key point 330 is the initial first key point. Of course, other face key points can also be marked in the output face image 330, which is not limited here.

[0053] Optionally, the electronic device may perform a head posture detection operation on the face image through a head posture detection function to obtain head posture information corresponding to the face in the face image. Optionally, the head posture detection function may include but is not limited to the solvePnP function of OpenCV, which is not limited here.

[0054] See also Figure 4 , Figure 4: is a schematic diagram of a head posture information disclosed in an embodiment of the present application. Optionally, the head posture information may include one or more of the roll angle roll, pitch angle pitch and yaw angle yaw corresponding to the head, which are not limited here. Among them, the roll angle roll indicates the amplitude of the head tilting left and right, the pitch angle pitch indicates the amplitude of the head pitching forward and backward, and the yaw angle indicates the amplitude of the head turning left and right.

[0055] 204. Determine an initial second key point according to the initial first key point.

[0056] In an embodiment of the present application, after determining the initial first key point, the electronic device can further determine the corresponding initial second key point through a regression network based on the initial first key point, wherein the initial second key point includes the key point for describing the tongue area in the facial image.

[0057] Optionally, the electronic device can determine the minimum circumscribed square enclosed by the initial first key points, and crop the mouth area of ​​the face in the face image according to the minimum circumscribed square to obtain a sub-area corresponding to the minimum circumscribed square; then the electronic device can scale the sub-area to a target resolution (for example: 128*128, 256*256, etc., not limited here); then a normalization operation can be performed on each pixel in the scaled sub-area to obtain a normalized result; then the electronic device can input the normalized result into a regression network to obtain the corresponding initial second key point.

[0058] Optionally, the electronic device may perform a normalization operation on each pixel point in the scaled sub-region by using the following formula 1, that is:

[0059]

[0060] Among them, x represents any pixel point in the sub-area, x norm represents the normalized result corresponding to any pixel point, μ represents the variance of multiple pixel points included in the sub-region, and σ represents the mean value of multiple pixel points included in the sub-region.

[0061] Optionally, the regression network may include but is not limited to a MobileNet_v2 network. Optionally, the regression network may be subjected to a compression operation to improve the timeliness of the regression network.

[0062] By implementing the above method, the electronic device can directly determine the corresponding second key point based on the initial first key point without detecting the second key point, thereby reducing the detection operation of the second key point and reducing the implementation difficulty and cost of the method.

[0063] Optionally, there may be one or more second key points, without limitation.

[0064] 206. Correct the initial first key point and the initial second key point according to the head posture information to obtain a corrected target first key point and a target second key point.

[0065] It is understandable that in the actual shooting process, due to problems such as jitter and shooting angle, the head in the face image always rotates or shifts to a greater or lesser extent, causing the mouth to deform or be blocked, which will cause the initial first key point and the initial second key point determined by the electronic device to be inaccurate, that is, the initial first key point and the initial second key point cannot accurately describe the position of the mouth area and tongue area in the face, which will cause the subsequent tongue area status detection results to be inaccurate.

[0066] In the related art, a three-dimensional model of the face may be established, and three-dimensional key points may be introduced to accurately determine the first key point and the second key point, thereby improving the accuracy of the state of the tongue area subsequently determined. However, the establishment of a three-dimensional model and the introduction of three-dimensional key points require a large amount of calculation, which will greatly increase the detection time and power consumption of the device. In this regard, in an embodiment of the present application, the initial first key point and the initial second key point collected by the electronic device may be two-dimensional key points, and the electronic device may directly correct the two-dimensional initial first key point and the initial second key point according to the head posture information to obtain the target first key point and the target second key point corresponding to the face when the face is in a forward-looking posture, thereby reducing the amount of calculation of the device and improving the detection efficiency of the tongue tip state.

[0067] As mentioned above, the head posture information may include the roll angle, pitch angle, and yaw angle of the head. Figure 5 , Figure 5 is a schematic diagram of a horizontal correction disclosed in an embodiment of the present application. Optionally, the electronic device can perform horizontal correction operations on the initial first key point and the initial second key point according to the roll angle corresponding to the head to obtain the first key point and the second key point corresponding to the face in a horizontal posture. Among them, Figure 5 510 in the figure indicates that the face is in an initial posture, and 520 indicates that the face is in a horizontal posture.

[0068] See also Figure 6 , Figure 6 is a schematic diagram of an X-axis-based correction operation disclosed in an embodiment of the present application. Optionally, the electronic device can perform an X-axis-based correction operation on the initial first key point and the initial second key point according to the yaw angle corresponding to the head, so as to obtain the first key point and the second key point corresponding to the face facing forward in the X-axis direction. For the convenience of explanation, Figure 6The top view of the head is used as an illustration for explanation, 610 indicates that the head is in an initial posture, and 620 indicates that the head is in a posture facing forward in the X-axis direction.

[0069] See also Figure 7 , Figure 7 is a schematic diagram of a correction operation based on the Y axis disclosed in an embodiment of the present application. Optionally, the electronic device can perform a correction operation based on the Y axis on the initial first key point and the initial second key point according to the pitch angle corresponding to the head, so as to obtain the first key point and the second key point corresponding to the face facing forward in the Y axis direction. For the convenience of explanation, Figure 7 The side view of the head is used as an illustration for explanation, 710 indicates that the head is in an initial posture, and 720 indicates that the head is in a posture facing forward in the Y-axis direction.

[0070] Among them, the X-axis is the horizontal axis of the target two-dimensional coordinate system, the Y-axis is the vertical axis of the target two-dimensional coordinate system, and the target two-dimensional coordinate system is a coordinate system with the center point of the mouth area of ​​the face as the origin, the length of the face image as the horizontal axis, and the width of the face image as the vertical axis.

[0071] Optionally, the electronic device may perform the correction operations described above on the initial first key point and the initial second key point in turn according to the roll angle, pitch angle and yaw angle corresponding to the head, or may only perform one or more of the correction operations described above on the initial first key point and the initial second key point, which is not limited here.

[0072] 208. Determine a target state corresponding to the tongue region of the face according to the target first key point and the target second key point.

[0073] In an embodiment of the present application, after the electronic device accurately determines the target first key point and the target second key point, the electronic device can determine the target state corresponding to the tongue area of ​​the face based on the relative position after the target first key point and the target second key point.

[0074] Optionally, if the target second key point is outside the target area surrounded by the target first key point, it can be determined that the tip of the tongue is in a state of extending outside the mouth; optionally, if the target second key point is inside the target area surrounded by the target first key point, it can be determined that the tip of the tongue is in a state of not extending outside the mouth, which is not limited here.

[0075] When implementing the methods disclosed in the above embodiments, a face detection operation can be first performed on the face image to determine the initial first key point and head posture information for describing the mouth area in the face image; and the initial second key point for describing the tongue area can be determined based on the initial first key point; further, the initial first key point and the initial second key point can be corrected based on the head posture information to obtain the corrected target first key point and target second key point. It can be seen that in the embodiment of the present application, the first key point and the second key point can be corrected based on the head posture information, so as to solve the problem of inaccurate key points caused by deformation or occlusion of the mouth due to head rotation or displacement, and then the more accurate tongue area state can be determined based on the corrected target first key point and target second key point. In addition, in the embodiment of the present application, only the two-dimensional key points detected from the face image need to be used, and there is no need to introduce additional three-dimensional key points, so as to reduce the amount of calculation of the device and improve the detection efficiency of the tongue area state.

[0076] See also Figure 8 , Figure 8 1 is a flowchart of another image processing method disclosed in an embodiment of the present application. Optionally, the method can be applied to the above-mentioned electronic device or other execution subjects. The embodiment of the present application is described by taking the electronic device as an example, which should not be limited to the embodiment of the present application. Optionally, the method can include the following steps:

[0077] 802. Perform a face detection operation on the face image to obtain an initial first key point and head posture information corresponding to the face in the face image.

[0078] 804. Determine an initial second key point according to the initial first key point.

[0079] 806. Determine the center point of the mouth area of ​​the face according to the initial first key point, and correct the initial first key point and the initial second key point according to the head posture information and the center point to obtain a corrected target first key point and a target second key point.

[0080] Optionally, there may be multiple initial first key points; and then the electronic device may calculate the first average value of the horizontal coordinates corresponding to the multiple initial first key points, and determine the first average value as the horizontal coordinate of the center point; and may calculate the second average value of the vertical coordinates corresponding to the multiple initial first key points, and determine the second average value as the vertical coordinate of the center point.

[0081] Optionally, the electronic device may determine the coordinate information of the center point of the mouth area of ​​the face according to the coordinate information of the multiple initial first key points and the following formula 2, that is:

[0082]

[0083] Among them, (C x ,C y ) represents the coordinate information of the center point, represents the horizontal coordinate of the i-th initial first key point, Represents the ordinate of the i-th initial first key point.

[0084] By implementing the above method, the electronic device can determine the center point of the mouth area based on the initial first key point through a simple weighted average operation. The implementation method is simple and efficient, thereby reducing the implementation difficulty and cost of the method.

[0085] In an embodiment of the present application, the electronic device can first determine the center point of the mouth area of ​​the face based on the initial first key point, so as to facilitate subsequent correction operations such as rotating and flipping the initial first key point and the initial second key point relative to the center point based on the center point as a reference point and according to the head posture information, thereby improving the accuracy and efficiency of the correction operation.

[0086] Please refer again Figure 5 As an optional implementation, the electronic device, when determining the center point of the mouth area of ​​the face, can further perform a horizontal correction operation on the first initial key point according to the roll angle and center point corresponding to the head, so as to obtain a first correction key point corresponding to the first initial key point when the face is in a horizontal posture; wherein the first initial key point is any initial first key point or any initial second key point.

[0087] Optionally, the electronic device can determine a first difference between the horizontal coordinate of the first initial key point and the horizontal coordinate of the center point, and determine a second difference between the vertical coordinate of the first initial key point and the vertical coordinate of the center point; and then can determine the first correction key point corresponding to the first initial key point when the face is in a horizontal posture based on the first difference, the second difference, the cosine value of the roll angle, the sine value of the roll angle and the coordinates of the center point.

[0088] Optionally, the electronic device can calculate a first product of the first difference and the cosine value of the roll angle, and calculate a second product of the second difference and the sine value of the roll angle, and then calculate a first sum of the first product, the second product and the horizontal coordinate of the center point as the horizontal coordinate of the first correction key point.

[0089] Optionally, the electronic device can calculate a third product of the second difference and the cosine value of the roll angle, and calculate a fourth product of the first difference and the sine value of the roll angle, and then calculate a third difference subtracted from the third product and the fourth product, and calculate the second sum of the third difference and the vertical coordinate of the center point as the vertical coordinate of the first correction key point.

[0090] Optionally, the electronic device may determine the coordinates of the first correction key point according to the roll angle corresponding to the head, the coordinates of the center point and the coordinates of the first initial key point by using the following formula 3, that is:

[0091]

[0092]

[0093] in, represents the coordinates of the first corrected key point, (C x ,C y ) represents the center point coordinates, represents the coordinates of the i-th first initial key point, cos(roll) represents the cosine value of the roll angle, and sin(roll) represents the sine value of the roll angle.

[0094] By implementing the above method, the electronic device can use the center point of the mouth area of ​​the face as the reference point, and perform a horizontal correction operation on the first initial key point according to the corresponding roll angle of the head, so as to obtain the first correction key point corresponding to the first initial key point when the face is in a horizontal posture, thereby eliminating the influence of the left and right tilt of the face on the accuracy of the detection result.

[0095] Furthermore, the electronic device can perform a correction operation on the first correction key point based on the coordinate axis of the target two-dimensional coordinate system according to the yaw angle and / or pitch angle corresponding to the head, so as to obtain the target key point corresponding to the first initial key point; wherein the target two-dimensional coordinate system is a coordinate system with the center point of the mouth area of ​​the face as the origin, the length of the face image as the horizontal axis, and the width of the face image as the vertical axis.

[0096] It should be noted that, in addition to tilting the head to the left or right, the head in the face image may also be pitched forward and backward and twisted to the left or right. The yaw angle corresponding to the head reflects the left or right twisting of the head to a certain extent, while the pitch angle reflects the front and back pitching of the head to a certain extent. In this regard, the above method can be implemented to perform a correction operation on the first correction key point based on the coordinate axis of the target two-dimensional coordinate system according to the yaw angle and / or pitch angle corresponding to the head, so as to eliminate the influence of the left or right twisting or the front and back pitching of the head on the accuracy of the detection result.

[0097] Optionally, the electronic device may perform an X-axis-based correction operation on the first correction key point according to the yaw angle and center point corresponding to the head to obtain a second correction key point; wherein the X-axis is the horizontal axis of the target two-dimensional coordinate system.

[0098] Please also read Figure 6 and Fig. 9 , Fig. 9is a schematic diagram of another correction operation based on the X-axis disclosed in the embodiment of the present application. Fig. 9 The top view of the head is used as an illustration for explanation. As an optional implementation, the electronic device can determine the first distance d1 between the first correction key point 910 and the center point 920 on the X-axis, and calculate the first depth coordinate of the first correction key point according to the yaw angle 930 and the first distance d1, so as to determine the first pseudo three-dimensional point 940 corresponding to the first correction key point 910 according to the first depth coordinate; optionally, the electronic device can calculate the multiplication result between the tangent value of the yaw angle 930 and the first distance d1 as the first depth coordinate of the first correction key point. The first depth coordinate refers to Fig. 9 The coordinates on the Z axis are used to represent the depth information of the head. The three-dimensional coordinates of the first pseudo three-dimensional point 940 can be determined according to the abscissa, ordinate and first depth coordinates of the first correction key point 910.

[0099] Then, the electronic device can rotate the first pseudo 3D point 940 around the center point 920 to the X-axis to obtain the second correction key point 950. Optionally, the abscissa of the second correction key point 950 can be the first quotient of the first depth coordinate corresponding to the first pseudo 3D point 940 and the sine value of the yaw angle 930; the ordinate of the second correction key point 950 can be the ordinate corresponding to the first pseudo 3D point 940.

[0100] As another optional implementation, the electronic device may perform a Y-axis correction operation on the second correction key point according to the pitch angle and center point corresponding to the head to obtain the target key point corresponding to the first initial key point; wherein the Y-axis is the vertical axis of the target two-dimensional coordinate system.

[0101] Please also read Figure 7 and Fig.10 , Fig.10 is a schematic diagram of another correction operation based on the Y axis disclosed in the embodiment of the present application. Fig.10 The side view of the head is used as an illustration for explanation. Optionally, the electronic device can determine the second distance d2 between the second correction key point 950 and the center point 920 on the Y axis, and calculate the second depth coordinate of the second correction key point 950 according to the pitch angle 1010 and the second distance d2, so as to determine the second pseudo three-dimensional point 1020 corresponding to the second correction key point 950 according to the second depth coordinate. Optionally, the electronic device can calculate the result of multiplying the tangent value of the pitch angle 1010 and the second distance d2 as the second depth coordinate of the second correction key point 950. The second depth coordinate refers to Fig.10The coordinates on the Z axis are used to represent the depth information of the head. The three-dimensional coordinates of the second pseudo three-dimensional point 1020 can be determined according to the abscissa, ordinate and second depth coordinates of the second correction key point 950.

[0102] Then, the electronic device can rotate the second pseudo 3D point 1020 around the center point 920 to the Y axis to obtain a target key point 1030 corresponding to the first initial key point. Optionally, the horizontal coordinate of the target key point 1030 can be the horizontal coordinate corresponding to the second pseudo 3D point 1020; the vertical coordinate of the target key point 1030 can be the result of multiplying the target absolute value and the cosine value of the pitch angle, and the target absolute value is the absolute value of the subtraction result of the vertical coordinate corresponding to the second pseudo 3D point 1020 and the vertical coordinate of the center point.

[0103] By implementing the above method, the electronic device can perform correction operations on the first correction key point based on the X-axis and Y-axis of the target two-dimensional coordinate system according to the yaw angle and pitch angle corresponding to the head in turn, so as to eliminate the influence of the left and right turning of the head or the front and back pitching of the head on the accuracy of the detection result. In addition, the correction key point can be determined by determining a pseudo three-dimensional point method and rotating around a center point, so as to achieve an effect similar to a three-dimensional key point without actually introducing a three-dimensional key point, thereby improving the correction effect.

[0104] 808. Determine a target state corresponding to the tongue region of the face according to the target first key point and the target second key point.

[0105] By implementing the methods disclosed in the above embodiments, a more accurate tongue tip state can be determined based on the corrected target first key point and target second key point, thereby overcoming the problem of inaccurate tongue tip state detection results due to head rotation. In addition, in the embodiment of the present application, only two-dimensional key points detected from the face image need to be used, and there is no need to introduce additional three-dimensional key points, thereby reducing the amount of calculation of the device and improving the detection efficiency of the tongue tip state; and, the center point of the mouth area of ​​the face can be first determined based on the initial first key point, so as to facilitate the subsequent use of the center point as a reference point, and the initial first key point and the initial second key point can be rotated, flipped, and other correction operations relative to the center point according to the head posture information, thereby improving the accuracy and efficiency of the correction operation; and, the center point of the mouth area of ​​the face can be used as a reference point, and the first initial key point can be horizontally corrected according to the roll angle corresponding to the head to obtain the first correction key point corresponding to the first initial key point when the face is in a horizontal posture, thereby eliminating the influence of the left and right tilting of the face on the accuracy of the detection result; and, the first correction key point can be corrected based on the coordinate axis of the target two-dimensional coordinate system according to the yaw angle and / or pitch angle corresponding to the head to eliminate the influence of the left and right twisting of the head or the front and back pitching of the head on the accuracy of the detection result.

[0106] See also Fig.11 , Fig.11 This is a flowchart of another image processing method disclosed in the embodiment of the present application. Optionally, the method can be applied to the above-mentioned electronic device or other execution subjects. The embodiment of the present application is described by taking the electronic device as an example, which should not be limited to the embodiment of the present application. Optionally, the method may include the following steps:

[0107] 1102. Perform a face detection operation on the face image to obtain an initial first key point and head posture information corresponding to the face in the face image.

[0108] 1104. Determine an initial second key point according to the initial first key point.

[0109] 1106. Correct the initial first key point and the initial second key point according to the head posture information to obtain the target first key point and the target second key point corresponding to the face in a forward-looking posture.

[0110] 1108. Determine a target area surrounded by multiple target first key points, and determine a target state corresponding to the tongue area of ​​the face based on relative position information between the target second key point and the target area.

[0111] In the embodiment of the present application, there may be multiple target first key points, and the multiple target first key points may include key points distributed on the mouth contour of the face. Fig.12 , Fig.12 12 is a schematic diagram of a target area disclosed in an embodiment of the present application. Optionally, the electronic device can determine a target area 1220 surrounded by multiple target first key points 1210; and then the electronic device can determine the target state corresponding to the tongue area of ​​the face according to the relative position information between the target second key point 1230 and the target area 1220.

[0112] By implementing the above method, a target area for representing the mouth of a human face can be constructed based on multiple target first key points, which makes it convenient to subsequently determine the positional relationship between the tip of the tongue and the mouth based on the relative position information between the target second key point and the target area, and then obtain the state of the tip of the tongue.

[0113] As an optional implementation, the tongue tip state may include a state where the tongue tip does not extend outside the mouth, and a state where the tongue tip extends outside the mouth, which is not limited here. It is understandable that the target area is roughly equal to the mouth range of the face. Optionally, if the second key point of the target is inside the target area, it means that the tongue tip is inside the mouth, and then it can be determined that the tongue tip of the face is in the first state, and the first state includes the state where the tongue tip does not extend outside the mouth; and if the second key point of the target is outside the target area, it means that the tongue tip is outside the mouth, and then it can be determined that the tongue tip of the face is in the second state, and the second state includes the state where the tongue tip extends outside the mouth.

[0114] By implementing the above method, the electronic device can determine whether the tip of the tongue is inside or outside the mouth based on whether the second key point of the target is inside the target area that roughly represents the mouth range of the human face. This makes it possible to determine the approximate state of the tip of the tongue relatively simply and quickly, thereby improving the efficiency of determining the state of the tip of the tongue.

[0115] See also Fig.13 , Fig.13 13 is a schematic diagram of a tongue tip state disclosed in an embodiment of the present application. As an optional implementation, the second state may include a first sub-state 1310 in which the tongue tip extends to the left outside the mouth, a second sub-state 1320 in which the tongue tip extends to the middle outside the mouth, and a third sub-state 1330 in which the tongue tip extends to the right outside the mouth, which is not limited here.

[0116] Please refer again Fig.12 Optionally, when determining that the target second key point 1230 is outside the target area 1220 surrounded by the target first key point, the electronic device can further determine a third distance d3 between the target second key point 1230 and the center point 1240 of the mouth area of ​​the face, wherein the center point 1240 of the mouth area is determined based on the above-mentioned initial first key point.

[0117] Furthermore, the electronic device can determine the target angle 1260 between the target second key point 1230 and the horizontal line 1250 where the center point 1240 is located; and then the electronic device can determine the state of the tip of the tongue of the human face in the above-mentioned first sub-state, second sub-state and third sub-state based on the third distance d3 and the target angle 1260.

[0118] By implementing the above method, the electronic device can determine the direction in which the tip of the tongue extends out of the mouth based on the positional relationship between the second key point of the target and the center point of the mouth area, thereby obtaining more accurate and detailed information on the state of the tip of the tongue.

[0119] Please refer again Fig.12As an optional implementation, if the third distance d3 is greater than the distance threshold, and the target angle 1260 is greater than or equal to the first angle and less than or equal to the first angle threshold θ1, it is determined that the tip of the tongue of the face is in the first sub-state where the tip of the tongue extends to the left outside the mouth;

[0120] The first angle may be any angle between 180° and 270°, and typical values ​​may include 180°, 181°, or 182°, etc., which are not limited here; the first angle threshold θ1 may be any angle between the first angle and 270°, which are not limited here. Fig.12 As shown, when the target angle 1260 is within the range of the first angle (assumed to be 180°) to the first angle threshold θ1, the tip of the tongue is located on the left side of the mouth, from which it can be determined that the tip of the tongue of the human face is in the first sub-state where the tip of the tongue extends to the left outside the mouth.

[0121] Optionally, the distance threshold may be greater than or equal to the distance between the center point 1240 and any point on the outline of the target area 1220 .

[0122] In another optional embodiment, if the third distance d3 is greater than the distance threshold, the target angle 1260 is greater than or equal to the second angle threshold θ2 and less than or equal to the second angle, it can be determined that the tip of the tongue of the human face is in the third sub-state in which the tip of the tongue extends to the right outside the mouth; wherein the second angle is greater than the first angle.

[0123] Optionally, the second angle may be any angle between 270° and 360°, and typical values ​​may include 360°, 358°, etc., which are not limited here. Optionally, the second angle may be greater than the second angle threshold θ2, and the second angle threshold θ2 may be any angle between 270° and the second angle, which are not limited here. Fig.12 As shown, when the target angle 1260 is within the range of the second angle threshold θ2 to the second angle (assumed to be 360°), the tip of the tongue is located on the right side of the mouth, from which it can be determined that the tip of the tongue of the human face is in the third sub-state in which the tip of the tongue extends to the right outside the mouth.

[0124] In another optional embodiment, if the third distance d3 is greater than the distance threshold, the target angle 1260 is greater than or equal to the first angle threshold θ1 and less than or equal to the second angle threshold θ2, then it is determined that the tip of the tongue of the human face is in the second sub-state where the tip of the tongue extends toward the middle and out of the mouth, and the second angle threshold is greater than the first angle threshold.

[0125] like Fig.12 As shown, when the target angle 1260 is within the range of the first angle threshold θ1 to the second angle threshold θ2, the tip of the tongue is located in the middle of the mouth, from which it can be determined that the tip of the tongue of the human face is in the second sub-state in which the tip of the tongue extends toward the middle and out of the mouth.

[0126] By implementing the above method, the specific state of the tongue tip can be quickly determined based on the third distance and the target angle by customizing the distance threshold and the angle threshold. This method is simple and efficient, thereby reducing the implementation difficulty and cost of the method.

[0127] By implementing the methods disclosed in the above embodiments, a more accurate tongue tip state can be determined according to the corrected target first key point and target second key point, thereby overcoming the problem of inaccurate tongue tip state detection results due to head rotation. In addition, in the embodiment of the present application, only two-dimensional key points detected from the face image are required, and no additional three-dimensional key points need to be introduced, thereby reducing the amount of calculation of the device and improving the detection efficiency of the tongue tip state; and, a target area for representing the mouth of the face can be constructed according to multiple target first key points, thereby facilitating the subsequent determination of the positional relationship between the tongue tip and the mouth according to the relative position information between the target second key point and the target area, thereby obtaining the tongue tip state; and, the direction in which the tongue tip extends to the outside of the mouth can be further determined according to the positional relationship between the target second key point and the center point of the mouth area, thereby obtaining more accurate and detailed tongue tip state information; and, the specific state of the tongue tip can be quickly determined according to the third distance and the target angle by customizing the distance threshold and the angle threshold, and the method is simple and efficient, thereby reducing the implementation difficulty and implementation cost of the method.

[0128] See also Fig.14 , Fig.14 1404, a first determining unit 1404, a correction unit 1406, and a second determining unit 1408, wherein:

[0129] The detection unit 1402 is used to detect the face image and determine the initial first key points and head posture information in the face image, where the initial first key points include key points in the face image for describing the mouth area of ​​the face;

[0130] A first determining unit 1404 is used to determine initial second key points according to the initial first key points, where the initial second key points include key points for describing the tongue area in the face image;

[0131] A correction unit 1406, configured to correct the initial first key point and the initial second key point according to the head posture information to obtain a corrected target first key point and a target second key point;

[0132] The second determining unit 1408 is used to determine the target state corresponding to the tongue area of ​​the face according to the target first key point and the target second key point.

[0133] When implementing the above device, the face detection operation can be performed on the face image first to determine the initial first key point and head posture information corresponding to the face in the face image; and the initial second key point can be determined based on the initial first key point; further, the initial first key point and the initial second key point can be corrected based on the head posture information to obtain the target first key point and the target second key point corresponding to the face in the forward-looking posture; it can be understood that the target first key point and the target second key point are the key points corresponding to the face in the forward-looking posture, so there is no problem of mouth deformation or occlusion due to head rotation or offset, and a more accurate tongue tip state can be determined based on the corrected target first key point and the target second key point, thereby overcoming the problem of inaccurate tongue tip state detection results due to head rotation. In addition, in the embodiment of the present application, only the two-dimensional key points detected from the face image need to be used, and there is no need to introduce additional three-dimensional key points, thereby reducing the amount of calculation of the device and improving the detection efficiency of the tongue tip state.

[0134] As an optional implementation, the correction unit 1406 is also used to determine the center point of the mouth area of ​​the face based on the initial first key point; and, based on the head posture information and the center point, correct the initial first key point and the initial second key point to obtain the target first key point and the target second key point corresponding to the face when the face is in a forward-looking posture.

[0135] By implementing the above device, the center point of the mouth area of ​​the face can be first determined based on the initial first key point, so that it is convenient to subsequently use the center point as a reference point and perform correction operations such as rotating and flipping the initial first key point and the initial second key point relative to the center point based on the head posture information, thereby improving the accuracy and efficiency of the correction operation.

[0136] As an optional implementation, the head posture information includes the roll angle, pitch angle and yaw angle corresponding to the head; the correction unit 1406 is also used to perform a horizontal correction operation on the first initial key point according to the roll angle and center point corresponding to the head, so as to obtain a first correction key point corresponding to the first initial key point when the face is in a horizontal posture, the first initial key point being any initial first key point or any initial second key point; and, according to the yaw angle and / or pitch angle corresponding to the head, perform a correction operation on the first correction key point based on the coordinate axis of the target two-dimensional coordinate system to obtain a target key point corresponding to the first initial key point, the target two-dimensional coordinate system being a coordinate system with the center point as the origin, the length of the face image as the horizontal axis, and the width of the face image as the vertical axis.

[0137] By implementing the above-mentioned device, the center point of the mouth area of ​​the face can be used as the reference point, and a horizontal correction operation can be performed on the first initial key point according to the roll angle corresponding to the head, so as to obtain a first correction key point corresponding to the first initial key point when the face is in a horizontal posture, thereby eliminating the influence of the left and right tilting of the face on the accuracy of the detection result; and, according to the yaw angle and / or pitch angle corresponding to the head, a correction operation based on the coordinate axis of the target two-dimensional coordinate system can be performed on the first correction key point, so as to eliminate the influence of the left and right twisting of the head or the front and back pitching of the head on the accuracy of the detection result.

[0138] As an optional implementation, the correction unit 1406 is further configured to perform an X-axis correction operation on the first correction key point according to the yaw angle and the center point corresponding to the head to obtain a second correction key point; and perform a Y-axis correction operation on the second correction key point according to the pitch angle and the center point corresponding to the head to obtain a target key point corresponding to the first initial key point;

[0139] Among them, the X axis is the horizontal axis of the target two-dimensional coordinate system, and the Y axis is the vertical axis of the target two-dimensional coordinate system.

[0140] By implementing the above device, the first correction key point can be corrected based on the X-axis and Y-axis of the target two-dimensional coordinate system according to the yaw angle and pitch angle corresponding to the head, so as to eliminate the influence of left and right turning of the head or front and back pitching of the head on the accuracy of the detection result.

[0141] As an optional implementation, the correction unit 1406 is also used to determine a first distance between the first correction key point and the center point on the X-axis, and calculate a first depth coordinate of the first correction key point based on the yaw angle and the first distance, so as to determine a first pseudo three-dimensional point corresponding to the first correction key point based on the first depth coordinate; and, rotate the first pseudo three-dimensional point around the center point to the X-axis to obtain a second correction key point.

[0142] By implementing the above device, the first pseudo three-dimensional point method can be determined, and the second correction key point can be determined by rotating around the center point, so as to achieve an effect similar to that of three-dimensional key points without actually introducing three-dimensional key points, thereby improving the correction effect.

[0143] As an optional implementation, the correction unit 1406 is also used to determine a second distance between the second correction key point and the center point on the Y-axis, and calculate the second depth coordinate of the second correction key point based on the pitch angle and the second distance, so as to determine the second pseudo three-dimensional point corresponding to the second correction key point based on the second depth coordinate; and, rotate the second pseudo three-dimensional point around the center point to the Y-axis to obtain the target key point corresponding to the first initial key point.

[0144] By implementing the above device, the second pseudo three-dimensional point method can be determined, and the target key point can be determined by rotating around the center point, so as to achieve an effect similar to that of three-dimensional key points without actually introducing three-dimensional key points, thereby improving the correction effect.

[0145] As an optional implementation, the correction unit 1406 is also used to calculate a first average value of the horizontal coordinates corresponding to the multiple initial first key points, and determine the first average value as the horizontal coordinate of the center point of the mouth area of ​​the face; and calculate a second average value of the vertical coordinates corresponding to the multiple initial first key points, and determine the second average value as the vertical coordinate of the center point of the mouth area of ​​the face.

[0146] By implementing the above device, the center point of the mouth area of ​​the face can be first determined based on the initial first key point, so that it is convenient to subsequently use the center point as a reference point and perform correction operations such as rotating and flipping the initial first key point and the initial second key point relative to the center point based on the head posture information, thereby improving the accuracy and efficiency of the correction operation.

[0147] As an optional implementation, the second determination unit 1408 is also used to determine a target area surrounded by multiple target first key points; and determine the target state corresponding to the tongue area of ​​the face based on the relative position information between the target second key point and the target area.

[0148] By implementing the above device, a target area for representing the mouth of a human face can be constructed based on multiple target first key points, thereby facilitating the subsequent determination of the positional relationship between the tip of the tongue and the mouth based on the relative position information between the target second key point and the target area, thereby obtaining the state of the tip of the tongue.

[0149] As an optional embodiment, the second determination unit 1408 is also used to determine that the tip of the tongue of the human face is in a first state when the target second key point is inside the target area, and the first state includes a state in which the tip of the tongue is not extended outside the mouth; and, when the target second key point is outside the target area, determine that the tip of the tongue of the human face is in a second state, and the second state includes a state in which the tip of the tongue is extended outside the mouth.

[0150] By implementing the above device, it is possible to determine whether the tip of the tongue is inside or outside the mouth based on whether the second key point of the target is inside the target area that roughly represents the mouth range of the human face. This makes it possible to determine the approximate state of the tip of the tongue relatively simply and quickly, thereby improving the efficiency of determining the state of the tip of the tongue.

[0151] As an optional embodiment, the second state includes a first sub-state in which the tip of the tongue extends to the left outside the mouth, a second sub-state in which the tip of the tongue extends to the middle outside the mouth, and a third sub-state in which the tip of the tongue extends to the right outside the mouth; the second determination unit 1408 is also used to determine, when the target second key point is outside the target area, a third distance between the target second key point and the center point of the mouth area of ​​the face, the center point of the mouth area being determined based on the initial first key point; and, determine a target angle between the target second key point and the horizontal line where the center point is located; and, based on the third distance and the target angle, determine the state of the tip of the tongue of the face in the first sub-state, the second sub-state, and the third sub-state.

[0152] By implementing the above device, the direction in which the tongue tip extends out of the mouth can be determined based on the positional relationship between the second key point of the target and the center point of the mouth area, thereby obtaining more accurate and detailed tongue tip status information.

[0153] As an optional implementation, the second determination unit 1408 is also used to determine that the tip of the tongue of the human face is in the first sub-state when the third distance is greater than the distance threshold and the target angle is greater than or equal to the first angle and less than or equal to the first angle threshold; and, to determine that the tip of the tongue of the human face is in the second sub-state when the third distance is greater than the distance threshold and the target angle is greater than or equal to the first angle threshold and less than or equal to the second angle threshold, and the second angle threshold is greater than the first angle threshold; and, to determine that the tip of the tongue of the human face is in the third sub-state, and the second angle is greater than the first angle when the third distance is greater than the distance threshold and the target angle is greater than or equal to the second angle threshold and less than or equal to the second angle.

[0154] By implementing the above device, the specific state of the tongue tip can be quickly determined based on the third distance and the target angle by customizing the distance threshold and the angle threshold. This method is simple and efficient, thereby reducing the implementation difficulty and cost of the method.

[0155] See also Fig.15 , Fig.15 Schematic diagram of the structure of an electronic device disclosed in the embodiment of the present application. Fig.15 As shown, the electronic device may include:

[0156] A memory 1501 storing executable program codes;

[0157] A processor 1502 coupled to the memory 1501;

[0158] The processor 1502 calls the executable program code stored in the memory 1501 to execute the image processing method disclosed in the above embodiments.

[0159] An embodiment of the present application discloses a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the image processing method disclosed in the above embodiments.

[0160] The embodiments of the present application also disclose an application publishing platform, wherein the application publishing platform is used to publish a computer program product, wherein when the computer program product runs on a computer, the computer executes part or all of the steps of the method in the above method embodiments.

[0161] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. Those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required for the present application.

[0162] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the above-mentioned processes does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0163] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0164] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0165] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several requests for a computer device (which can be a personal computer, a server or a network device, etc., specifically a processor in a computer device) to perform some or all of the steps of the above-mentioned methods of various embodiments of the present application.

[0166] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0167] The image processing method and device, electronic device, and storage medium disclosed in the embodiments of the present application are introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Detecting a facial image, and determining an initial first key point and head posture information in the facial image, wherein the initial first key point includes a key point in the facial image for describing a first facial feature region of the human face; Determine initial second key points according to the initial first key points, wherein the initial second key points include key points in the facial image for describing a second facial feature region of the human face; Correcting the initial first key point and the initial second key point according to the head posture information to obtain a corrected target first key point and a target second key point; According to the target first key point and the target second key point, a target state corresponding to the second facial feature area of ​​the face is determined.

2. The method according to claim 1, characterized in that The correcting the initial first key point and the initial second key point according to the head posture information to obtain a corrected target first key point and a target second key point includes: Determine the center point of the first facial feature area of ​​the human face according to the initial first key point; The initial first key point and the initial second key point are corrected according to the head posture information and the center point to obtain a corrected target first key point and a target second key point.

3. The method according to claim 2, characterized in that The head posture information includes a roll angle, a pitch angle, and a yaw angle corresponding to the head; and correcting the initial first key point and the initial second key point according to the head posture information and the center point to obtain a corrected target first key point and a target second key point, including: According to the roll angle corresponding to the head and the center point, a horizontal correction operation is performed on the first initial key point to obtain a first correction key point corresponding to the first initial key point when the face is in a horizontal posture, wherein the first initial key point is any of the initial first key points or any of the initial second key points; According to the yaw angle and / or pitch angle corresponding to the head, the first correction key point is corrected based on the coordinate axis of the target two-dimensional coordinate system to obtain the corrected target key point corresponding to the first initial key point. The target two-dimensional coordinate system is a coordinate system with the center point as the origin, the length of the face image as the horizontal axis, and the width of the face image as the vertical axis.

4. The method according to claim 3, characterized in that The step of performing a correction operation on the first correction key point based on a coordinate axis of a target two-dimensional coordinate system according to the yaw angle and / or pitch angle corresponding to the head to obtain a target key point corresponding to the first initial key point includes: According to the yaw angle corresponding to the head and the center point, performing an X-axis correction operation on the first correction key point to obtain a second correction key point; According to the pitch angle corresponding to the head and the center point, a Y-axis correction operation is performed on the second correction key point to obtain a target key point corresponding to the first initial key point; The X-axis is the horizontal axis of the target two-dimensional coordinate system, and the Y-axis is the vertical axis of the target two-dimensional coordinate system.

5. The method according to claim 4, characterized in that The step of performing an X-axis correction operation on the first correction key point according to the yaw angle corresponding to the head and the center point to obtain a second correction key point includes: Determine a first distance between the first correction key point and the center point on the X-axis, and calculate a first depth coordinate of the first correction key point according to the yaw angle and the first distance, so as to determine a first pseudo three-dimensional point corresponding to the first correction key point according to the first depth coordinate; The first pseudo three-dimensional point is rotated around the center point to the X-axis to obtain a second correction key point.

6. The method according to claim 4, characterized in that The step of performing a Y-axis correction operation on the second correction key point according to the pitch angle corresponding to the head and the center point to obtain a target key point corresponding to the first initial key point includes: Determine a second distance between the second correction key point and the center point on the Y axis, and calculate a second depth coordinate of the second correction key point according to the pitch angle and the second distance, so as to determine a second pseudo three-dimensional point corresponding to the second correction key point according to the second depth coordinate; The second pseudo three-dimensional point is rotated around the center point to the Y axis to obtain a target key point corresponding to the first initial key point.

7. The method according to any one of claims 2 to 6, characterized in that: There are multiple initial first key points, and determining the center point of the first facial feature area of ​​the human face according to the initial first key points includes: Calculate a first average value of the horizontal coordinates corresponding to the plurality of the initial first key points, and determine the first average value as the horizontal coordinate of the center point of the first facial feature region of the human face; Calculate the second average values ​​of the vertical coordinates corresponding to the plurality of the initial first key points, and determine the second average value as the vertical coordinate of the center point of the first facial feature region of the human face.

8. The method according to claim 1, characterized in that: There are multiple first target key points; determining the target state corresponding to the second facial feature area of ​​the face according to the first target key point and the second target key point includes: Determine a target area surrounded by a plurality of the target first key points; The target state corresponding to the second facial feature area of ​​the human face is determined based on the relative position information between the second key point of the target and the target area.

9. The method according to claim 8, characterized in that The first facial feature region includes a mouth region, and the second facial feature region includes a tongue region; and determining the target state corresponding to the second facial feature region of the human face according to the relative position information between the target second key point and the target region includes: If the target second key point is inside the target area, it is determined that the tongue area of ​​the face is in a first state, where the first state includes a state where the tip of the tongue is not extended outside the mouth; If the target second key point is outside the target area, it is determined that the tongue area of ​​the face is in a second state, and the second state includes a state in which the tip of the tongue extends outside the mouth.

10. The method according to claim 9, characterized in that The second state includes a first sub-state in which the tip of the tongue extends to the left outside the mouth, a second sub-state in which the tip of the tongue extends to the middle outside the mouth, and a third sub-state in which the tip of the tongue extends to the right outside the mouth; if the second key point of the target is outside the minimum area, determining that the tongue area of ​​the face is in the second state includes: If the target second key point is outside the target area, determining a third distance between the target second key point and a center point of a mouth area of ​​the face, where the center point of the mouth area is determined based on the initial first key point; Determine a target angle between a line connecting the second key point of the target and the center point and a horizontal line where the center point is located; The state of the tongue area of ​​the face is determined in the first sub-state, the second sub-state, and the third sub-state according to the third distance and the target angle.

11. The method according to claim 10, characterized in that The determining, according to the third distance and the target angle, the state of the tongue region of the face in the first sub-state, the second sub-state, and the third sub-state includes: If the third distance is greater than the distance threshold, and the target angle is greater than or equal to the first angle and less than or equal to the first angle threshold, it is determined that the tongue area of ​​the face is in the first sub-state; If the third distance is greater than the distance threshold, and the target angle is greater than or equal to the first angle threshold and less than or equal to the second angle threshold, it is determined that the tongue area of ​​the face is in the second sub-state, and the second angle threshold is greater than the first angle threshold; If the third distance is greater than the distance threshold and the target angle is greater than or equal to the second angle threshold and less than or equal to the second angle, it is determined that the tongue area of ​​the face is in the third sub-state and the second angle is greater than the first angle.

12. The method according to any one of claims 1 to 6 and 8, characterized in that: The first facial features region includes a mouth region, and the second facial features region includes a tongue region.

13. An image processing device, characterized in that: The device comprises: A detection unit, configured to detect a face image, and determine an initial first key point and head posture information in the face image, wherein the initial first key point includes a key point in the face image for describing a first facial feature region of the face; A first determining unit, configured to determine initial second key points according to the initial first key points, wherein the initial second key points include key points in the facial image for describing a second facial feature region of the human face; A correction unit, used for correcting the initial first key point and the initial second key point according to the head posture information to obtain a corrected target first key point and a target second key point; The second determination unit is used to determine the target state corresponding to the second facial feature area of ​​the face according to the target first key point and the target second key point.

14. An electronic device, characterized in that: It comprises a memory storing executable program code, and a processor coupled to the memory; wherein the processor calls the executable program code stored in the memory to execute the method according to any one of claims 1 to 12.

15. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images