Facial organ closure degree calculation method, device, medium and electronic equipment

Abstract

The present application provides a method and device, medium and electronic device for calculating the closure of facial organs, and relates to the field of data processing. The method includes: determining the head posture and the target facial organ posture in the frontal face state, the frontal face state means that the face is not deflected relative to the camera; calculating the transformation matrix of the head posture in the frontal face state and the target facial organ posture in the frontal face state; obtaining the current head posture of the driver and passenger, and calculating the target facial organ posture under the current head posture based on the transformation matrix and the current head posture; calculating the closure of the target facial organ under the current head posture based on the target facial organ posture under the current head posture and the key point data of the target facial organ under the current head posture. The present application avoids the influence of head rotation on the calculation of the closure of the target facial organ.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and device for calculating the closure degree of facial organs, a medium and an electronic device. Background Art

[0002] In the smart cockpit, it is critical to detect the driver's fatigue state. The mainstream methods include analyzing the steering wheel operation and the driver's facial image. When analyzing facial images, the driver's eye closure and mouth opening are generally calculated to determine whether the driver has closed eyes or yawns. Among them, the eye closure is determined by calculating the ratio of the distance between the eyelids and the corners of the eyes, and the mouth opening is determined by the ratio of the distance between the upper and lower points of the inner edge of the mouth and the left and right corners of the mouth. However, this method of calculating the closure of pixel feature points by their coordinates will be affected by the head posture, and it is difficult to accurately express the driver's current state. Summary of the invention

[0003] In view of this, embodiments of the present application provide a method and device, a medium and an electronic device for calculating the degree of closure of facial organs.

[0004] In a first aspect, an embodiment of the present application provides a method for calculating the closure degree of facial organs, including: determining the head posture and the target facial organ posture in a frontal face state, the frontal face state indicating that the face is not deflected relative to the camera; calculating the transformation matrix of the head posture in the frontal face state and the target facial organ posture in the frontal face state; obtaining the current head posture of the driver and passenger, and calculating the target facial organ posture in the current head posture based on the transformation matrix and the current head posture; calculating the closure degree of the target facial organ in the current head posture based on the target facial organ posture in the current head posture and the key point data of the target facial organ in the current head posture.

[0005] In combination with the first aspect, in certain implementations of the first aspect, the closure degree of the target facial organ in the current head posture is calculated based on the posture of the target facial organ in the current head posture and the key point data of the target facial organ in the current head posture, including: calculating the initial closure degree of the target facial organ in the current head posture based on the key point data of the target facial organ in the current head posture; determining the scaling ratio of the target facial organ in a first direction and a scaling ratio of the second direction from the frontal face state to the current head posture based on the posture of the target facial organ in the current head posture, the first direction and the second direction being related to the calculation standard of the closure degree of the facial organ; correcting the initial closure degree of the target facial organ in the current head posture based on the scaling ratio of the first direction and the scaling ratio of the second direction to obtain the closure degree of the target facial organ in the current head posture.

[0006] In combination with the first aspect, in certain implementations of the first aspect, based on the target facial organ posture under the current head posture, the scaling ratio of the target facial organ in the first direction from the front face state to the current head posture and the scaling ratio of the second direction are determined, including: determining the component length of the first direction of the target facial organ posture under the current head posture as the scaling ratio of the target facial organ in the first direction from the front face state to the current head posture; determining the component length of the second direction of the target facial organ posture under the current head posture as the scaling ratio of the target facial organ in the second direction from the front face state to the current head posture.

[0007] In combination with the first aspect, in certain implementations of the first aspect, based on the scaling ratio of the first direction and the scaling ratio of the second direction, the initial closure degree of the target facial organ in the current head posture is corrected to obtain the closure degree of the target facial organ in the current head posture, including: determining the ratio of the scaling ratio of the first direction and the scaling ratio of the second direction; multiplying the ratio and the initial closure degree of the target facial organ in the current head posture to obtain the closure degree of the target facial organ in the current head posture.

[0008] In combination with the first aspect, in certain implementations of the first aspect, based on the key point data of the target facial organ under the current head posture, the initial closure degree of the target facial organ under the current head posture is calculated, including: based on the key point data of the target facial organ under the current head posture, calculating the opening height and opening width of the target facial organ; determining the ratio of the opening height to the opening width as the initial closure degree of the target facial organ under the current head posture.

[0009] In combination with the first aspect, in some implementations of the first aspect, a calculation formula for the target facial organ posture under the current head posture includes: Among them, T goal represents the target facial organ posture in the front face state, T headpose Indicates the head posture in the front face state, represents the transformation matrix of the target facial organ posture, R represents the current head posture of the driver and passenger, and R goal Indicates the target facial organ pose under the current head pose.

[0010] In combination with the first aspect, in some implementations of the first aspect, the target facial organ includes any one of a left eye, a right eye, and a mouth.

[0011] In a second aspect, an embodiment of the present application provides a device for calculating the closure degree of facial organs, including: a determination module, used to determine the head posture and the target facial organ posture in a frontal face state, the frontal face state indicating a state in which the face is not deflected relative to the camera; a first calculation module, used to calculate the transformation matrix of the head posture in the frontal face state and the target facial organ posture in the frontal face state; a second calculation module, used to obtain the current head posture of the driver and passenger, and calculate the target facial organ posture in the current head posture based on the transformation matrix and the current head posture; a third calculation module, used to calculate the closure degree of the target facial organ in the current head posture based on the target facial organ posture in the current head posture and the key point data of the target facial organ in the current head posture.

[0012] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program for executing the method for calculating the degree of closure of facial organs described in the first aspect.

[0013] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; and the processor is used to execute the method for calculating the degree of closure of facial organs described in the first aspect.

[0014] In the present application, the facial features in the frontal state are relatively stable and can provide reliable standard reference points; and the facial features in the frontal state are usually in a relatively upright and symmetrical state, which makes the definition of the coordinate system and the relationship between the facial features relatively simple, making the calculation of the transformation matrix more convenient. Further, by calculating the transformation matrix and applying this matrix to update the target facial organ posture under the current head posture, the precise position and direction of the target facial organ under different head postures can be obtained. Secondly, through the target facial organ posture under the current head posture, the closure of the target facial organ calculated using the key point data of the target facial organ under the current head posture can be corrected, avoiding the influence of the rotation of the head on the closure calculation, and realizing the accurate calculation of the closure of the target facial organ. On this basis, it is helpful to timely detect signs of fatigue or inattention of the driver and passengers, and take corresponding measures to improve driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 The figure is a flow chart of a method for calculating the degree of closure of facial organs provided in one embodiment of the present application.

[0017] Figure 2 Shown is a schematic diagram of constructing a right eye coordinate system provided by an embodiment of the present application.

[0018] Figure 3 Shown is a schematic diagram of constructing a head coordinate system provided in an embodiment of the present application.

[0019] Figure 4 The figure is a schematic diagram of a flow chart for calculating the closure degree of a target facial organ under a current head posture provided by an embodiment of the present application.

[0020] Figure 5 Shown is a schematic diagram of determining eye key point data provided by an embodiment of the present application.

[0021] Figure 6 Shown is a schematic diagram of determining mouth key point data provided by an embodiment of the present application.

[0022] Figure 7 Shown is a schematic structural diagram of a facial organ closure degree calculation device provided in one embodiment of the present application.

[0023] Figure 8 Shown is a schematic structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] In the smart cockpit, driver fatigue detection is an important part of ensuring driving safety. In theory, there are many ways to assess driver fatigue, among which the analysis of steering wheel angle and driver facial image are two main methods.

[0026] Specifically, by analyzing the steering wheel angle, the driver's mental state can be indirectly inferred. If the steering wheel angle changes abnormally frequently or the amplitude is too large, it may mean that the driver is not paying enough attention and may be in a state of fatigue. However, this method can only provide indirect clues and cannot directly reflect the driver's fatigue level. Therefore, we rely more on analyzing the driver's facial image to assess his fatigue state. In this process, the characteristic changes of the eyes and mouth become key indicators.

[0027] For the eyes, we mainly focus on their closure. Through image processing technology, we can accurately calculate the distance between the upper and lower eyelid edges of the eyes and compare it with the distance at the corners of the eyes to get a ratio. This ratio can reflect the degree of opening and closing of the eyes, and then determine whether the driver is in a closed-eye state. Generally speaking, if the eye closure is high, that is, the ratio is small, then the driver may be dozing off or in a state of fatigue. Similarly, the opening and closing of the mouth is also an important indicator for assessing the driver's fatigue state. We determine the opening and closing of the mouth by calculating the ratio of the distance between the upper and lower points of the inner edge of the mouth and the distance between the left and right corners of the mouth. If this ratio is large, it means that the mouth is open wider, and it may be yawning, which is also a manifestation of fatigue.

[0028] However, the above-mentioned method for calculating the degree of closure is implemented by calculating the coordinates of the pixel feature points, which will be affected by the head posture. For example, when the face turns from the front to the left or right, the width of the eyes or mouth becomes smaller and smaller, and the degree of closure of the eyes when the side face is larger than that when the front face is larger. The existing technology does not reflect the processing method for this part of the error. Therefore, this method can only determine whether the driver's eyes or mouth are completely closed, but cannot determine the degree of closure. Other technologies for determining the degree of closure often require the extraction of richer features. For example, for the eyes, information such as iris edge points is required, and the edges of the eyeball, iris and eyelid are fitted to calculate the iris occlusion ratio. However, such high-resolution images have higher hardware requirements and are more complicated to calculate.

[0029] In view of this, the present application first determines the head posture and the target facial organ posture in the frontal face state, and calculates the transformation matrix of the head posture in the frontal face state and the target facial organ posture in the frontal face state. By calculating the transformation matrix and applying this matrix to update the target facial organ posture in the current head posture, the precise position and direction of the target facial organ in different head postures can be obtained. Secondly, through the target facial organ posture in the current head posture, the closure of the target facial organ calculated using the key point data of the target facial organ in the current head posture can be corrected, avoiding the influence of the rotation of the head on the closure calculation, and realizing the accurate calculation of the closure of the target facial organ. In addition, the method in the present application does not need to extract other complex features of the target facial organ, reducing the complexity of the calculation. On this basis, it is helpful to timely detect signs of fatigue or inattention of the driver and passengers, and take corresponding measures to improve driving safety.

[0030] Figure 1 FIG. 2 is a flow chart of a method for calculating the degree of closure of facial organs provided by an embodiment of the present application. Figure 1 As shown, the method includes the following steps.

[0031] Step S110, determining the head posture and the target facial organ posture in the frontal face state.

[0032] The front face state refers to the state in which the face is not deflected relative to the camera. Specifically, it means that the face is facing the camera without deflection or tilt to the left or right or up or down. In other words, this is a standard face orientation, with the front of the face directly facing the camera lens without any rotation or tilt. The target facial organ posture refers to the state or position of specific facial organs (such as the left eye, right eye, mouth, etc.) used to evaluate the driver's fatigue level in this front face state. Since the relative position of the target facial organs changes when the facial orientation changes, understanding these changes in the front face state can more accurately calculate parameters such as closure.

[0033] Specifically, the head posture in the frontal state refers to the position and direction of the head coordinate system relative to the reference coordinate system in the frontal state; the target facial organ posture refers to the position and direction of the target facial organ coordinate system relative to the reference coordinate system in the frontal state. Since the transformation relationship between the facial features and the head posture is basically the same, the standard human face 3D key point model can also be used here to replace the face in the frontal state, and then calculate the transformation relationship from the head posture to the facial features (including the target facial organs of this application).

[0034] Exemplarily, the method for determining the head posture and the target facial organ posture in the frontal face state includes:

[0035] First, define the reference coordinate system. For example, the reference coordinate system is the world coordinate system, which is usually used to describe the absolute position and direction of the target object in three-dimensional space. For example, the world coordinate system is based on the center of the camera. The horizontal right side of the coordinate system is the X-axis, the upward side is the Y-axis, and the backward side is the Z-axis. It should be noted that the right, upward, and backward mentioned here are all defined from a human perspective. For example, when a person faces forward and stretches his arms to shoulder width, the direction pointed by his right hand is the positive direction of the X-axis; when a person is standing, the direction of the top of the head is the positive direction of the Y-axis; the direction of the Z-axis refers to the direction in front of the person with his back to the person.

[0036] Secondly, define the target facial organ coordinate system and the head coordinate system. For example, the target facial organ is the right eye. Figure 2 FIG. 1 is a schematic diagram of constructing a right eye coordinate system provided by an embodiment of the present application. Figure 3 FIG. 1 is a schematic diagram of constructing a head coordinate system provided by an embodiment of the present application. Figure 2 As shown in the figure, the right eye corner to the left eye corner is defined as the X-axis, the lower eyelid midpoint to the upper eyelid midpoint is defined as the vector on the XY plane, the axis perpendicular to the XY plane and pointing forward is defined as the Z-axis, and then the Y-axis can be calculated from the X-axis and the Z-axis according to the right-hand coordinate system. The right eye coordinate system describes the direction of the right eye in space and reflects some characteristics of the right eye. Similarly, the posture of the target facial organs of other target facial organs in the frontal state in the world coordinate system can be determined. Figure 3 As shown, the right eye center to the left eye center is defined as the X axis, the philtrum to the right eye center is defined as the vector on the XY plane, and the direction perpendicular to the XY plane forward is defined as the Z axis. Then the Y axis can be calculated from the X and Z axes according to the right-hand coordinate system.

[0037] Step S120, calculating the transformation matrix of the head posture in the front face state and the target facial organ posture in the front face state.

[0038] Specifically, the transformation matrix can be represented as follows: Among them, T goal represents the target facial organ posture in the front face state, T headpose Indicates the head posture in the front face state. Since there is no need to care about the impact of translation on closure in the usage scenario, T here goal and T headpose Refers only to the rotation matrix.

[0039] Step S130, obtaining the current head posture of the driver and passenger, and calculating the target facial organ posture under the current head posture based on the transformation matrix and the current head posture.

[0040] The current head posture refers to the rotation direction of the driver's head relative to the reference coordinate system at a certain moment, including angle information such as pitch, yaw and roll of the head. For example, a sensor, camera or other image capture device can be used to capture the driver's head image or data, and then the current head posture can be extracted and calculated through image processing, computer vision or sensor data processing.

[0041] Specifically, the calculation formula of the target facial organ posture under the current head posture includes: in, represents the transformation matrix of the head posture in the front face state and the target facial organ posture in the front face state, R represents the current head posture of the driver, and R goal represents the target facial organ posture under the current head posture. It is understandable that the target facial organ does not move relative to the face, and the conversion matrix of the head posture and the target facial organ posture in the frontal or non-frontal state is unchanged. That is, Suitable for any face orientation.

[0042] Step S140, based on the target facial organ posture under the current head posture and the key point data of the target facial organ under the current head posture, calculate the closure degree of the target facial organ under the current head posture.

[0043] First, determine the key points of the target facial organs. For example, for the eyes, these key points include the corners of the eyes, pupils, and points in the upper and lower eyelids; for the mouth, these key points include the corners of the mouth and points in the upper and lower lips. Secondly, according to the posture of the target facial organs under the current head posture, determine the position of these key points under the current head posture. It should be noted that since the change of head posture will cause the position and direction of the target facial organs in the image to change, it is necessary to further transform and adjust the key points accordingly based on the posture of the target facial organs under the current head posture.

[0044] Furthermore, the position information of these transformed and adjusted key points is used to calculate the closure of the target facial organs. For example, for the eyes, closure usually refers to the degree of eyelid opening, which can be achieved by comparing features such as the distance between the upper and lower eyelids, the visibility of the pupil, or the curvature of the eyelid edge. Alternatively, by training the model to learn the mapping relationship from key point data to closure, the closure can be directly predicted based on the current key point data.

[0045] In this embodiment, the facial features in the frontal state are relatively stable and can provide reliable standard reference points; and the facial features in the frontal state are usually in a relatively upright and symmetrical state, which makes the definition of the coordinate system and the relationship between the facial features relatively simple, making it more convenient to calculate the transformation matrix. Further, by calculating the transformation matrix and applying this matrix to update the target facial organ posture under the current head posture, the direction of the target facial organ under different head postures can be obtained. Secondly, through the target facial organ posture under the current head posture, the closure of the target facial organ calculated using the key point data of the target facial organ under the current head posture can be corrected, avoiding the influence of head rotation on the closure calculation, and realizing the accurate calculation of the closure of the target facial organ. In addition, the method in the present application does not need to extract other complex features of the target facial organ, reducing the complexity of the calculation. On this basis, it is helpful to timely discover signs of fatigue or inattention of the driver and passengers, and take corresponding measures to improve driving safety.

[0046] Figure 4 The figure shows a flow chart of calculating the closure degree of the target facial organ under the current head posture provided by an embodiment of the present application. Figure 1 Based on the embodiment shown, Figure 4 The embodiment shown is described below in detail. Figure 4 The embodiment shown and Figure 1 The differences and similarities of the illustrated embodiments are not described in detail.

[0047] like Figure 4 As shown, in this embodiment, based on the target facial organ posture under the current head posture and the key point data of the target facial organ under the current head posture, the closure degree of the target facial organ under the current head posture is calculated, including the following steps.

[0048] Step S410, based on the key point data of the target facial organ under the current head posture, calculate the initial closure degree of the target facial organ under the current head posture.

[0049] In one example, based on key point data of the target facial organ in the current head posture, the opening height and opening width of the target facial organ are calculated; and the ratio of the opening height to the opening width is determined as the initial closure degree of the target facial organ in the current head posture.

[0050] Figure 5 FIG. 1 is a schematic diagram of determining eye key point data provided by an embodiment of the present application. Figure 5 As shown, the key point data includes coordinate data P1 and P2 of the two eye corners, and three groups of key point data at corresponding positions of the upper and lower eyelids, namely P3 and P4, P5 and P6, and P7 and P8.

[0051] Then, the opening height of the eye

[0052] The opening width of the eye w1 = ‖P1-P2‖;

[0053] Then the initial closure degree of the eye in the current head posture is

[0054] Figure 6 FIG. 1 is a schematic diagram of determining mouth key point data provided by an embodiment of the present application. Figure 6 As shown, the key point data includes coordinate data M1 and M2 of two mouth corners, and three groups of key point data at corresponding positions of upper and lower lips, namely M3 and M4, M5 and M6, and M7 and M8.

[0055] Then, the opening height of the mouth

[0056] The opening width of the mouth w2 = ‖M1-M2‖;

[0057] Then the initial closure degree of the mouth in the current head posture is

[0058] In this Figure 5 and Figure 6 In the example, the degree of opening of the eyelids or mouth is represented by calculating the average distance between key points. In addition, the maximum distance or minimum distance between key points can also be calculated to represent the degree of opening of the eyelids or mouth.

[0059] It should be noted that Figure 5 and Figure 6 Only three groups of key point data are used to calculate the opening height. In actual application, any group can be selected, such as 2 groups, 4 groups, 5 groups, etc., as long as the selected number of groups can provide comprehensive and accurate information. In addition, when selecting these key points, their stability and recognizability need to be considered to ensure accurate extraction under different head postures and expressions.

[0060] In this example, by comprehensively considering the ratio of opening height and opening width, the initial closure degree of the target facial organs can be more comprehensively evaluated, and it can better adapt to the facial features of different individuals and improve the accuracy of the calculation. In addition, the ratio is also easier to normalize and standardize, which is convenient for subsequent comparison and analysis.

[0061] As shown above, in another example, the key point data of the target facial organ may also be input into a pre-trained closure calculation model to obtain the initial closure of the target facial organ under the current head posture.

[0062] Step S420, based on the target facial organ posture under the current head posture, determining the scaling ratio of the target facial organ in the first direction and the scaling ratio of the target facial organ in the second direction from the frontal face state to the current head posture.

[0063] The first direction and the second direction are related to the calculation standard of the closure degree of the facial organs. Exemplarily, the first direction is the X-axis and the second direction is the Y-axis.

[0064] In one example, the projection length of the vector of the first direction of the target facial organ posture under the current head posture on the XY plane is determined as the scaling ratio of the target facial organ from the frontal face state to the first direction under the current head posture; the projection length of the vector of the second direction of the target facial organ posture under the current head posture on the XY plane is determined as the scaling ratio of the target facial organ from the frontal face state to the second direction under the current head posture. The projection length of the vector of the first direction on the XY plane is the component length of the first direction, and the projection length of the vector of the second direction on the XY plane is the component length of the second direction.

[0065] The projection length of the vector of the target facial organ posture in the first direction in the current head posture on the XY plane can be regarded as the size of the target facial organ in the first direction in the current head posture, and this length can reflect the scaling ratio of the target facial organ in different head postures, that is, the scaling degree from the frontal face state to the first direction in the current head posture. Similarly, the projection length of the vector of the target facial organ posture in the second direction in the current head posture on the XY plane can be regarded as the size of the target facial organ in the second direction in the current head posture, and this length can reflect the scaling degree of the target facial organ from the frontal face state to the second direction in the current head posture.

[0066] Specifically, when the face rotates relative to the camera, the size of the face's projection on the image plane changes. The scaling of the target facial organs under the current head posture can be estimated based on the change in the length of their projection in the image. The principle is that the actual size of the target facial organs remains unchanged from the frontal state to the current state, so the change in length in the image can be attributed to the change in perspective caused by the head rotation, that is, the scaling of its projection.

[0067] Taking the first direction as an example, the projection length of the vector of the first direction of the target facial organ posture under the current head posture on the XY plane is determined as the scaling ratio of the target facial organ in the first direction, which means that if the length of the target facial organ in the first direction is shorter in the image of the current head posture than in the image of the frontal face state, then scaling has occurred in the first direction. The projection length of the vector of the second direction of the target facial organ posture under the current head posture on the XY plane is determined as the scaling ratio of the target organ in the second direction, which is the same as the principle of the first direction, except that it is for another dimension perpendicular to the first direction.

[0068] Step S430, based on the scaling ratio of the first direction and the scaling ratio of the second direction, correct the initial closure degree of the target facial organ under the current head posture to obtain the closure degree of the target facial organ under the current head posture.

[0069] In one example, a ratio of a scaling ratio in a first direction to a scaling ratio in a second direction is determined; and the ratio is multiplied by an initial closure degree of the target facial organ in a current head posture to obtain the closure degree of the target facial organ in the current head posture.

[0070] Exemplarily, the ratio of the scaling ratio in the first direction to the scaling ratio in the second direction is a / b;

[0071] but Figure 5 The eye closure degree under the current head posture shown is

[0072] Figure 6 The closure of the mouth in the current head posture is shown as

[0073] In this embodiment, by considering the scaling ratios in different directions, the deformation of the target facial organ in the actual three-dimensional space can be more accurately reflected. This accuracy is crucial for analyzing the characteristics of the target facial organ, such as the degree of closure. In addition, different people have different facial morphologies and features. By calculating the ratio of scaling ratios, this solution can better adapt to the facial changes of different individuals. This correction scheme determines the final degree of closure of the target facial organ through simple mathematical operations, avoiding the complex model training and parameter adjustment process, which not only improves the calculation efficiency, but also reduces the complexity and cost of the system.

[0074] When calculating the closure of a target face, a key issue to consider is the change in the appearance of the face under different head poses. When the head pose of a person changes, the view angle relative to the camera also changes, which causes corresponding changes in the size, shape, and position of the face in the image. In order to calculate the closure as accurately as possible, these geometric deformations need to be taken into account.

[0075] Accordingly, in step S410 to step S423, by considering the changes in the head posture and the posture of the target facial organ, the deformation of the target facial organ at different angles can be more accurately captured when calculating the closure of the target facial organ, thereby improving the accuracy of the closure calculation. Specifically, this solution combines the key point data of the target facial organ under the current head posture data, and uses the scaling ratio of the target facial organ from the frontal state to the current head posture, so that the closure calculation can better correct the scale changes caused by different head postures, improve the accuracy and stability of the closure calculation, thereby more comprehensively evaluating the closure of the target facial organ, and improving the accuracy and comprehensiveness of the closure calculation.

[0076] Combination of the above Figures 1 to 6 , describes in detail the embodiment of the method for calculating the closure degree of facial organs of the present application, and the following is combined with Figure 7 , the embodiment of the facial organ closure calculation device of the present application is described in detail. It should be understood that the description of the facial organ closure calculation method embodiment corresponds to the description of the facial organ closure calculation device embodiment, so the parts not described in detail can refer to the previous method embodiment.

[0077] Figure 7 FIG. 1 is a schematic diagram of the structure of a facial organ closure degree calculation device provided by an embodiment of the present application. Figure 7 As shown, the facial organ closure degree calculation device 70 provided in the embodiment of the present application includes:

[0078] A determination module 710 is used to determine the head posture and the target facial organ posture in a frontal face state, where the frontal face state means that the face is not deflected relative to the camera;

[0079] A first calculation module 720, for calculating a transformation matrix of a head posture in a frontal face state and a target facial organ posture in a frontal face state;

[0080] The second calculation module 730 is used to obtain the current head posture of the driver and passenger, and calculate the target facial organ posture under the current head posture based on the transformation matrix and the current head posture;

[0081] The third calculation module 740 is used to calculate the closure degree of the target facial organ under the current head posture based on the posture of the target facial organ under the current head posture and the key point data of the target facial organ under the current head posture.

[0082] In one embodiment of the present application, the third calculation module 740 is also used to calculate the initial closure degree of the target facial organ under the current head posture based on the key point data of the target facial organ under the current head posture; determine the scaling ratio of the target facial organ in the first direction from the frontal face state to the current head posture and the scaling ratio of the second direction based on the posture of the target facial organ under the current head posture, and the first direction and the second direction are related to the calculation standard of the closure degree of the facial organ; based on the scaling ratio of the first direction and the scaling ratio of the second direction, correct the initial closure degree of the target facial organ under the current head posture to obtain the closure degree of the target facial organ under the current head posture.

[0083] In one embodiment of the present application, the third calculation module 740 is also used to determine the component length of the first direction of the target facial organ posture under the current head posture as the scaling ratio of the target facial organ from the front face state to the first direction under the current head posture; and determine the component length of the second direction of the target facial organ posture under the current head posture as the scaling ratio of the target facial organ from the front face state to the second direction under the current head posture.

[0084] In one embodiment of the present application, the third calculation module 740 is also used to determine the ratio of the scaling ratio in the first direction to the scaling ratio in the second direction; multiply the ratio and the initial closure degree of the target facial organ in the current head posture to obtain the closure degree of the target facial organ in the current head posture.

[0085] In one embodiment of the present application, the third calculation module 740 is also used to calculate the opening height and opening width of the target facial organ based on the key point data of the target facial organ under the current head posture; and determine the ratio of the opening height to the opening width as the initial closure degree of the target facial organ under the current head posture.

[0086] In one embodiment of the present application, the calculation formula of the target facial organ posture under the current head posture includes: Among them, T goal represents the target facial organ posture in the front face state, T headpose Indicates the head posture in the front face state, represents the transformation matrix of the head posture in the front face state and the target facial organ posture in the front face state, R represents the current head posture of the driver, and R goal Indicates the target facial organ pose under the current head pose.

[0087] In one embodiment of the present application, the target facial organ includes any one of a left eye, a right eye, and a mouth.

[0088] Below, reference Figure 8 To describe an electronic device according to an embodiment of the present application. Figure 8 Shown is a schematic structural diagram of an electronic device provided by an exemplary embodiment of the present application.

[0089] like Figure 8 As shown, the electronic device 80 includes one or more processors 801 and a memory 802 .

[0090] The processor 801 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 80 to perform desired functions.

[0091] The memory 802 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 801 may run the program instructions to implement the facial organ closure calculation method of each embodiment of the present application described above and / or other desired functions. Various contents such as head posture in the front face state and target facial organ posture in the front face state, transformation matrix, key point data of the target facial organ, etc. may also be stored in the computer-readable storage medium.

[0092] In one example, the electronic device 80 may further include: an input device 803 and an output device 804 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0093] The input device 803 may include, for example, a keyboard, a mouse, etc.

[0094] The output device 804 can output various information to the outside, including the head posture in the front face state and the target facial organ posture in the front face state, the transformation matrix, the key point data of the target facial organ, etc. The output device 804 can include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.

[0095] Of course, to simplify, Figure 8 Only some of the components related to the present application in the electronic device 80 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 80 may also include any other appropriate components.

[0096] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method for calculating the degree of closure of facial organs according to various embodiments of the present application described above in this specification.

[0097] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0098] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the method for calculating the degree of closure of facial organs according to various embodiments of the present application described above in this specification.

[0099] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0100] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present application. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, not for limitation, and the above details do not limit the present application to being implemented by adopting the above specific details.

[0101] The block diagrams of the devices, apparatuses, equipment, and systems involved in this application are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0102] It should also be noted that in the apparatus, device and method of the present application, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present application.

[0103] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present application. Therefore, the present application is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0104] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for calculating the degree of closure of facial organs, characterized in that: include: Determine the head posture in the front face state and the target facial organ posture in the front face state, wherein the front face state indicates that the face is not deflected relative to the camera, and the target facial organ posture refers to the position and direction of the target facial organ coordinate system relative to the reference coordinate system in the front face state; Calculate the transformation matrix of the head posture in the front face state and the target facial organ posture in the front face state; Acquire the current head posture of the driver and passenger, and calculate the target facial organ posture under the current head posture based on the transformation matrix and the current head posture; Calculating the closure degree of the target facial organ under the current head posture based on the target facial organ posture under the current head posture and the key point data of the target facial organ under the current head posture; The calculation formula of the target facial organ posture under the current head posture includes: Among them, T goal represents the target facial organ posture in the frontal state, T headpose represents the head posture in the front face state, represents the transformation matrix of the head posture in the frontal face state and the target facial organ posture in the frontal face state, R represents the current head posture of the driver and passenger, R goal Indicates the target facial organ posture under the current head posture.

2. The method for calculating the degree of closure of facial organs according to claim 1, characterized in that: The calculating the closure degree of the target facial organ under the current head posture based on the target facial organ posture under the current head posture and the key point data of the target facial organ under the current head posture comprises: Calculating an initial closure degree of the target facial organ under the current head posture based on key point data of the target facial organ under the current head posture; Based on the target facial organ posture under the current head posture, determining a scaling ratio of the target facial organ in a first direction and a scaling ratio of the target facial organ in a second direction from the frontal face state to the current head posture, wherein the first direction and the second direction are related to a calculation standard of the closure degree of the facial organ; Based on the scaling ratio of the first direction and the scaling ratio of the second direction, the initial closure degree of the target facial organ in the current head posture is corrected to obtain the closure degree of the target facial organ in the current head posture.

3. The method for calculating the degree of closure of facial organs according to claim 2, characterized in that: The step of determining a scaling ratio of the target facial organ in a first direction and a scaling ratio of the target facial organ in a second direction from the frontal face state to the current head posture based on the target facial organ in the current head posture comprises: Determine the component length of the first direction of the target facial organ posture under the current head posture as the scaling ratio of the target facial organ from the frontal face state to the first direction under the current head posture; The component length of the second direction of the target facial organ posture under the current head posture is determined as the scaling ratio of the target facial organ from the frontal face state to the second direction under the current head posture.

4. The method for calculating the degree of closure of facial organs according to claim 2, characterized in that: The step of correcting the initial closure degree of the target facial organ in the current head posture based on the scaling ratio of the first direction and the scaling ratio of the second direction to obtain the closure degree of the target facial organ in the current head posture includes: Determining a ratio of a scaling ratio in the first direction to a scaling ratio in the second direction; The ratio is multiplied by an initial closure degree of the target facial organ in the current head posture to obtain the closure degree of the target facial organ in the current head posture.

5. The method for calculating the degree of closure of facial organs according to claim 2, characterized in that: The calculating the initial closure degree of the target facial organ under the current head posture based on the key point data of the target facial organ under the current head posture comprises: Calculating the opening height and opening width of the target facial organ based on the key point data of the target facial organ under the current head posture; The ratio of the opening height to the opening width is determined as the initial closure degree of the target facial organ under the current head posture.

6. The method for calculating the degree of closure of facial organs according to any one of claims 1 to 5, characterized in that: The target facial organ includes any one of a left eye, a right eye, and a mouth.

7. A facial organ closure calculation device, characterized in that: include: A determination module, used to determine the head posture and the target facial organ posture in the frontal face state, wherein the frontal face state indicates that the face is not deflected relative to the camera, and the target facial organ posture refers to the position and direction of the target facial organ coordinate system relative to the reference coordinate system in the frontal face state; A first calculation module is used to calculate the transformation matrix of the head posture in the front face state and the target facial organ posture in the front face state; A second calculation module is used to obtain the current head posture of the driver and passenger, and calculate the target facial organ posture under the current head posture based on the transformation matrix and the current head posture; A third calculation module, configured to calculate the closure degree of the target facial organ under the current head posture based on the posture of the target facial organ under the current head posture and the key point data of the target facial organ under the current head posture; The calculation formula of the target facial organ posture under the current head posture includes: Among them, T goal represents the target facial organ posture in the frontal state, T headpose represents the head posture in the front face state, represents the transformation matrix of the head posture in the frontal face state and the target facial organ posture in the frontal face state, R represents the current head posture of the driver and passenger, R goal Indicates the target facial organ posture under the current head posture.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the method for calculating the degree of closure of facial organs as described in any one of claims 1 to 6 above.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is used to execute the method for calculating the degree of closure of facial organs as described in any one of claims 1 to 6.

Images