Face angle prediction method, device, equipment and readable storage medium

Abstract

The application is suitable for the field of image recognition, and provides a face angle prediction method, device and equipment and readable storage medium. The method comprises the following steps: obtaining a face region of a to-be-tested face image; determining a face feature corresponding to the face region; determining a plurality of angle probabilities of each angle type according to the face feature, wherein the angle types comprise a yaw angle, a pitch angle and a roll angle; and determining a predicted angle of each angle type of the face in the to-be-tested face image relative to a shooting position according to the plurality of angle probabilities of each angle type. Thus, the application obtains angle probabilities corresponding to a plurality of angle intervals respectively, calculates the predicted angle through the angle probabilities corresponding to the plurality of angle intervals respectively, ensures the accuracy of the predicted angle, and is suitable for a plurality of scenes in which an accurate face angle needs to be determined.

Description

Technical Field

[0001] The present application belongs to the field of image recognition technology, and in particular relates to a face angle detection method, device, equipment and readable storage medium. Background Art

[0002] With the development of artificial intelligence, facial recognition is increasingly being used in various industries. However, the quality of facial images has a significant impact on image recognition accuracy. Among them, the angle of the face is a key factor affecting image recognition accuracy.

[0003] Related technologies capture key facial features while a face is in motion, reconstructing the face image into a 3D image. This 3D image is then mapped into a 2D image, and facial pose prediction is performed based on the facial motion features in the 2D image. When a face is identified as being at a large angle, angle correction can be performed before recognition, or even omitted, to improve image recognition accuracy.

[0004] However, since the relevant technology can only predict the posture of the face at the action level through the motion features of the face during movement, it is difficult to accurately predict the angle of the face, and it cannot be applied to scenarios that require higher prediction accuracy of the face angle. Summary of the Invention

[0005] The present application provides a face angle prediction method, apparatus, device and readable storage medium, which can avoid the problem of only being able to predict the approximate posture of the face and having difficulty in accurately predicting the face angle, and can adapt to scenarios where the accuracy of face angle prediction is higher.

[0006] In a first aspect, the present application provides a face angle prediction method, comprising:

[0007] Obtain the face area of ​​the face image to be tested;

[0008] Determining facial features corresponding to the facial region;

[0009] determining, based on the facial features, a plurality of angle probabilities for each of a plurality of angle types, the plurality of angle probabilities for each angle type corresponding to a plurality of angle intervals for each angle type, the plurality of angle types comprising a yaw angle, a pitch angle, and a roll angle;

[0010] According to the multiple angle probabilities of each angle type, the predicted angle of each angle type of the face in the face image to be tested relative to the shooting position is determined.

[0011] This application uses facial features corresponding to the facial region to determine multiple angle probabilities for each of the various angle types. Based on the multiple angle probabilities for each angle type, the predicted angle for each angle type of the face in the face image to be tested relative to the shooting position is then determined. Because the multiple angle probabilities obtained are angle probabilities corresponding to multiple angle intervals, the corresponding predicted angle is calculated using the angle probabilities corresponding to the multiple angle intervals. This ensures the accuracy of the predicted angle, avoids situations where accurate facial angle prediction is difficult, and is suitable for scenarios where facial angle prediction requires high accuracy.

[0012] In a second aspect, the present application provides a face angle prediction device, which is configured to execute the method described in the first aspect or any possible implementation of the first aspect. Specifically, the device may include:

[0013] An acquisition module is used to acquire the face area of ​​the face image to be tested;

[0014] A first determining module, configured to determine facial features corresponding to the facial region;

[0015] a second determining module, configured to determine, based on the facial features, a plurality of angle probabilities for each of a plurality of angle types, wherein the plurality of angle probabilities for each angle type respectively correspond to a plurality of angle intervals for each angle type, the plurality of angle types comprising a yaw angle, a pitch angle, and a roll angle;

[0016] The third determination module is used to determine the predicted angle of each angle type of the face in the face image to be tested relative to the shooting position according to multiple angle probabilities of each angle type.

[0017] In a third aspect, the present application provides an electronic device comprising a memory and a processor. The memory is configured to store instructions; the processor executes the instructions stored in the memory, causing the device to perform the face angle prediction method of the first aspect or any possible implementation of the first aspect.

[0018] In a fourth aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the face angle prediction method in the first aspect or any possible implementation of the first aspect.

[0019] In a fifth aspect, a computer program product comprising instructions is provided. When the instructions are executed on a device, the device executes the face angle prediction method in the first aspect or any possible implementation of the first aspect.

[0020] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative effort.

[0022] Figure 1 is a flowchart of a face angle prediction method provided by an embodiment of the present application;

[0023] Figure 2 is a flowchart of a face angle prediction method provided by an embodiment of the present application;

[0024] Figure 3 is a flowchart of a face angle prediction method provided by an embodiment of the present application;

[0025] Figure 4 is a flowchart of a face angle prediction method provided by an embodiment of the present application;

[0026] Figure 5 is a structural schematic diagram of a face angle prediction apparatus provided by an embodiment of the present application;

[0027] Figure 6 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0028] In the following description, specific details are set forth in order to provide a thorough understanding of the present application. However, persons skilled in the art will understand that the present application can be practiced without these specific details. In other instances, well-known structures, devices, circuits, and methods have not been described in detail in order to avoid obscuring the present application.

[0029] It should be understood that, when used in the specification and the appended claims of the present application, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0030] It should also be understood that the term "and / or" used in the specification and the appended claims of the present application means any combination of one or more of the associated listed items and all possible combinations thereof, and includes these combinations.

[0031] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0032] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0033] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0034] The present application provides a face angle prediction method, device, equipment and readable storage medium. The method can be implemented through an identification device and applied to scenarios such as access control identification, missing person search, case investigation, and intelligent security.

[0035] The face angle includes three types of angles of the face relative to the shooting position, which are pitch angle, yaw angle and roll angle.

[0036] The recognition device refers to the device used by the user to perform face angle prediction. The recognition device can be an access control device, a smartphone, a desktop computer, a laptop computer, a tablet computer, a wearable device, a handheld device, an in-vehicle device, a server, etc. The embodiments of this application do not impose any restrictions on the specific type of recognition device.

[0037] The identification device may include display hardware, or be connected to an external display.

[0038] Regarding the above scenario, the following example illustrates the application of the face angle prediction method:

[0039] 1. When using the face angle prediction method in the access control recognition scenario, the recognition device can predict the face angle based on the face in the image, and determine whether the next action can be taken based on the predicted face angle. For example, when the face angle is too large, a prompt message such as "Face angle too large, cannot be recognized" will be displayed on the display screen of the access control device.

[0040] 2. When using the face angle prediction method in a missing person search scenario, the face image is input into the recognition device, and the face angle corresponding to the face in the face image is predicted. The predicted face angle is used to determine whether the next action can be taken. For example, it is determined whether the recognition device can recognize the face, to determine whether the face image matches the image in the missing person image library, and then to determine whether the person is missing.

[0041] Alternatively, the recognition device may be connected to a surveillance camera for communication, and the recognition device may obtain an image captured by the surveillance camera and predict the facial angle corresponding to the face in the image.

[0042] Based on the above scenario description, below, taking the recognition device as an example, combined with the accompanying drawings and application scenarios, the face angle prediction method provided by the embodiment of the present application is described in detail.

[0043] See also Figure 1 , Figure 1 A flow chart of a face angle prediction method provided in an embodiment of the present application is shown.

[0044] like Figure 1 As shown, the face angle prediction method provided by this application may include:

[0045] S101: Acquire a face region of a face image to be tested.

[0046] The face image to be tested can be directly given by the user or extracted from video data collected by image acquisition devices such as surveillance cameras and video cameras.

[0047] The face region refers to the region containing the face in the face image to be tested.

[0048] In some embodiments, the face area is obtained by performing face detection on the face image to be detected, obtaining a first detection window, and intercepting the image in the first detection window.

[0049] Optionally, the recognition device may perform an expansion process on the first detection window to obtain an expanded second detection window, and intercept an area of ​​a corresponding size of the second detection window as the face area.

[0050] It can be understood that the detection window refers to a wireframe from which a face in the face image to be detected can be extracted.

[0051] Among them, a face detection algorithm can be used to perform face detection on the face to be tested.

[0052] The face detection algorithm may be stored in a storage device.

[0053] The storage device can communicate with the recognition device so that the recognition device can retrieve the face detection algorithm from the storage device. This application does not limit the storage method and specific type of the storage device.

[0054] In some embodiments, the YOLO (you only look once) algorithm is used for face detection. The YOLO algorithm is an object recognition and positioning algorithm based on a deep neural network, and its greatest feature is its high running speed.

[0055] In a specific embodiment, assuming the recognition device is an access control device including a camera, when a face approaches the camera of the access control device, the camera captures a facial image, and the access control device performs face detection on the facial image using a face detection algorithm to obtain a facial region of the image to be detected.

[0056] In another specific embodiment, assuming the recognition device is a mobile phone with a recognition applet installed, the mobile phone communicates with a surveillance camera through the recognition applet to obtain images captured by the surveillance camera. The recognition applet can perform face detection on the images captured by the surveillance camera using a face detection algorithm to obtain the facial region of the image to be tested.

[0057] S102: Determine facial features corresponding to the facial region.

[0058] Based on S101, the recognition device can obtain the face region, thereby performing feature extraction on the face region to obtain facial features corresponding to the face region.

[0059] In some embodiments, the recognition device outputs the facial features by inputting the facial area into a backbone network of a facial angle recognition model.

[0060] The backbone network is used to extract facial features from face images.

[0061] The backbone network is pre-stored in a storage device that communicates with the identification device.

[0062] In a specific embodiment, assuming the recognition device is an access control device, when a face approaches the camera of the access control device, the camera captures the face image. The access control device then performs face detection on the face image using a face detection algorithm. After obtaining the face area of ​​the face image, the backbone network is called to extract features of the face area to obtain facial features.

[0063] In another specific embodiment, assuming the recognition device is a mobile phone with a recognition applet installed on the mobile phone, the recognition applet performs face detection on the image using a face detection algorithm, obtains the face region corresponding to the image, and then calls the backbone network to perform feature extraction on the face region to obtain facial features.

[0064] S103: Determine multiple angle probabilities of each of the multiple angle types based on the facial features.

[0065] Among them, the various angle types include yaw angle, pitch angle, and roll angle.

[0066] The multiple angle probabilities of each angle type correspond to the multiple angle intervals of each angle type.

[0067] The multiple angle intervals refer to the multiple angle intervals obtained by dividing the angle ranges of the yaw angle, the pitch angle, and the roll angle according to a preset rule.

[0068] In some embodiments, the prediction rule is to divide the angle range into intervals of every 5 degrees.

[0069] For example, the angle range of the yaw angle and the pitch angle is [-90, 90]. The angle range of the yaw angle and the pitch angle is divided into intervals of every 5 degrees, and 36 angle intervals are obtained respectively.

[0070] The 36 angle intervals of the yaw angle are [-90, -85), [-85, -80) ... (80, 85], (85, 90].

[0071] The 36 angle intervals of the pitch angle are [-90, -85), [-85, -80) ... (80, 85], (85, 90].

[0072] For example, the angle range of the roll angle is [-180, 180], and the angle range of the roll angle is divided into intervals of every 5 degrees, thereby obtaining 72 angle intervals.

[0073] The 72 angle intervals of the roll angle are [-180, -175), [-175, -170), [-170, -165) ... (165, 170], (170, 175], (175, 180].

[0074] In some embodiments, the recognition device inputs the facial features into a fully connected classification network of the facial angle recognition model and outputs multiple angle probabilities for each angle type.

[0075] Among them, the fully connected classification network is used to predict the angle probabilities corresponding to the facial features in multiple angle intervals of each angle type.

[0076] The backbone network and the fully connected classification network serve as face angle recognition models and are pre-stored in a storage device that communicates with the recognition device.

[0077] Specifically, the fully connected classification network is connected to the output end of the backbone network. After the backbone network of the recognition device extracts the facial features of the face image to be tested, the facial features are input into the fully connected classification network for angle probability prediction.

[0078] The fully connected classification network consists of three fully connected layers: the first fully connected layer, the second fully connected layer, and the third fully connected layer. The first and second fully connected layers are connected to 36 nodes respectively, and the third fully connected layer is connected to 72 support points.

[0079] It can be understood that the 36 nodes of the first fully connected layer and the second fully connected layer are used to predict the angle probabilities of the yaw angle and the pitch angle in their 36 angle intervals, respectively, and the 72 nodes of the third fully connected layer are used to predict the angle probabilities of the roll angle in its 72 angle intervals.

[0080] Therefore, there are 36 multiple angle probabilities for each of the yaw angle and the pitch angle, and there are 72 multiple angle probabilities for the roll angle.

[0081] In a specific embodiment, assuming the recognition device is an access control device, after acquiring facial features, the access control device predicts the angle probabilities of the yaw angle and pitch angle within their 36 angle intervals, and predicts the angle probabilities of the roll angle within their 72 angle intervals based on the facial features.

[0082] In another specific embodiment, assuming the recognition device is a mobile phone with a recognition applet installed on the mobile phone, the recognition applet obtains facial features and then predicts the angle probabilities of the yaw angle and pitch angle within their 36 angle intervals, and the angle probabilities of the roll angle within their 72 angle intervals, based on the facial features.

[0083] S104: Determine, based on the multiple angle probabilities for each angle type, a predicted angle of each angle type of the face in the face image to be tested relative to the shooting position.

[0084] In some embodiments, for each angle type, the angle of the face in the face image to be tested relative to the shooting position is determined based on multiple angle probabilities, the number of the angle intervals and the middle angle of each angle interval.

[0085] The middle angle of each angle interval refers to the middle angle of every 5 angles in an angle interval. For example, if an angle interval is [0, 5), then the corresponding middle angle is 2.5 degrees.

[0086] The calculation formula for determining the angle of the face in the to-be-tested face image relative to the shooting position is as follows:

[0087]

[0088] wherein n represents the number of angle intervals, a i represents the middle angle of the i-th angle interval, pred i represents the angle probability of the i-th angle interval.

[0089] In one specific embodiment, it is assumed that the recognition device is an access control device, and the access control device includes a display screen and a camera. After the access control device predicts the angle probability of the yaw angle and the pitch angle in the 36 angle intervals thereof and the angle probability of the roll angle in the 72 angle intervals thereof according to the face features, the access control device calculates the value corresponding to each angle interval according to the angle probability of the yaw angle and the pitch angle in the 36 angle intervals thereof, calculates the value corresponding to each interval according to the angle probability of the roll angle in the 72 angle intervals thereof, and calculates the final predicted angle of the face according to the value corresponding to each interval. The access control device can determine whether the next action can be performed through the predicted angle of the face, for example, when the face angle is too large, the access control device displays prompt information such as “the face angle is too large and cannot be recognized” on the display screen.

[0090] In another specific embodiment, it is assumed that the recognition device is a mobile phone, and the mobile phone has a recognition applet. After the recognition applet predicts the angle probability of the yaw angle and the pitch angle in the 36 angle intervals thereof and the angle probability of the roll angle in the 72 angle intervals thereof according to the face features according to the face features, the recognition applet calculates the value corresponding to each angle interval according to the angle probability of the yaw angle and the pitch angle in the 36 angle intervals thereof, calculates the value corresponding to each interval according to the angle probability of the roll angle in the 72 angle intervals thereof, and calculates the final predicted angle of the face according to the value corresponding to each interval. The recognition applet can determine whether the next action can be performed through the predicted angle of the face, for example, when the face angle is too large, the to-be-tested face image is corrected.

[0091] The face angle prediction method provided in the present application obtains face features according to the face region of the to-be-tested face image, determines a plurality of angle probabilities of each angle type according to the face features, and finally determines the predicted angle of each angle type of the face in the to-be-tested face image relative to the shooting position according to the plurality of angle probabilities of each angle type. Thus, for each angle type, the plurality of angle probabilities obtained are the angle probabilities corresponding to a plurality of angle intervals, and the predicted angle is calculated through the angle probabilities corresponding to the plurality of angle intervals, thereby ensuring the accuracy of the predicted angle.

[0092] Based on the above Figure 2As described in the S103 embodiment, the recognition device can obtain the maximum value of multiple angle probabilities of the roll angle. However, when the maximum value corresponds to a preset mapping angle range, the calculated predicted angle of the roll angle is inaccurate. Various processing methods can be used to ensure that a more accurate predicted angle is obtained.

[0093] The range of the multiple angle intervals of the roll angle is [-180, 180].

[0094] The mapping angle interval includes multiple angle intervals corresponding to [-180, -90) or multiple angle intervals corresponding to (90, 180].

[0095] Next, combine Figure 2 , which details the specific implementation process of the face angle prediction method of this application.

[0096] See also Figure 2 , Figure 2 A flow chart of a face angle prediction method provided in an embodiment of the present application is shown.

[0097] like Figure 2 As shown, the face angle prediction method provided by this application may include:

[0098] S201: Determine a maximum probability angle interval for the roll angle.

[0099] The maximum probability angle interval is the angle interval corresponding to the maximum value among the multiple angle probabilities.

[0100] For example, the maximum value among the plurality of angle probabilities of the roll angle corresponds to the angle interval [-175, -170).

[0101] S202: When the maximum probability angle interval is within a preset mapping angle interval, linearly map the multiple angle probabilities of the roll angle to obtain multiple mapped angle probabilities.

[0102] Since the roll angle range is [-180, 180], which is a circle, based on the periodicity of the roll angle, when the maximum probability angle interval is within the angle interval corresponding to the edge of [-180, -90) or the angle interval corresponding to the edge of (90, 180), it is considered that the roll angle of the face is large. When the roll angle of the face is large, the recognition device will Figure 1 The predicted angle of the roll angle calculated by the method shown in S103 is inaccurate.

[0103] Therefore, when the recognition device determines that the maximum probability angle interval corresponding to the roll angle is in the angle interval of [-90, 90], it can be calculated according to Figure 1 The method shown in S103 calculates the predicted angle of the roll angle.

[0104] However, when the recognition device determines that the maximum probability angle interval corresponding to the roll angle is within the angle interval corresponding to [-180, -90) or the angle interval corresponding to (90, 180), the recognition device needs to linearly map the multiple angle probabilities of the roll angle to obtain the mapped multiple angle probabilities.

[0105] The recognition device linearly maps multiple angle probabilities so that when the maximum probability angle interval corresponding to the roll angle is in the angle interval of [-180, 90) or (90, 180], the accurate angle value of the roll angle can be obtained.

[0106] In some embodiments, linear mapping refers to:

[0107] The angle probability corresponding to the angle interval greater than or equal to 0 degrees and less than or equal to 180 degrees is replaced by the mapped angle probability corresponding to the angle interval greater than or equal to -180 degrees and less than 0 degrees;

[0108] Furthermore, the angle probability corresponding to the angle interval less than or equal to 0 degrees and greater than or equal to -180 degrees is replaced by the mapped angle probability corresponding to the angle interval less than or equal to 180 degrees and greater than 0 degrees.

[0109] For example, when the maximum value of the multiple angle probabilities of the roll angle corresponds to the angle interval [-175, -170), if we directly follow Figure 1 The method shown in S103 calculates the predicted angle of the roll angle, and the obtained predicted angle value may be -103.5 degrees. Obviously, -103.5 degrees is not in the angle range [-175, -170), which is unreasonable.

[0110] Therefore, the recognition device needs to first map the multiple angle probabilities of the roll angle. For example, the angle probability corresponding to the angle interval [-175, -170) is mapped to the angle interval (-10, -5). That is, the angle probability corresponding to the angle interval [-175, -170) is replaced by the angle probability corresponding to the angle interval (-10, -5).

[0111] S203 : Determine a mapping angle according to the mapped multiple angle probabilities, the number of angle intervals of the roll angle, and the middle angle of each angle interval.

[0112] For example, according to the angle probability corresponding to the angle interval (-10, -5], the calculated predicted angle value (mapping angle) can be -5 degrees. Obviously, -5 degrees is in the angle interval (-10, -5], which is reasonable.

[0113] S204: Perform inverse linear mapping on the mapping angle to obtain a predicted angle of the roll angle.

[0114] The inverse linear mapping refers to mapping the predicted angle value (mapped angle) calculated according to the angle probability corresponding to the angle interval (-10, -5] into the angle interval [-175, -170).

[0115] For example, when the angle value obtained based on S203 is -5 degrees, the inverse mapping of -5 degrees obtains an angle value of -175 degrees, which is in the angle interval [-175, 170), which is reasonable.

[0116] In this application, for the roll angle, the recognition device performs linear mapping on the plurality of angle probabilities when determining that the maximum value of the plurality of angle probabilities corresponds to the preset mapping interval, to obtain a plurality of mapped angle probabilities, and then determines a mapped angle according to the plurality of mapped angle probabilities, the number of angle intervals of the roll angle, and the middle angle of each angle interval, and performs inverse linear mapping on the mapped angle to predict the angle. When the maximum value of the plurality of angle probabilities corresponds to the preset mapping interval, the angle probability corresponding to the mapping angle interval is mapped by means of linear mapping, and then the mapping angle is calculated, and the predicted angle is calculated according to the mapping angle, so that a more accurate predicted angle can be obtained.

[0117] Based on the above Figure 1 As described in the embodiment S101, when determining the face region, the recognition device can perform outer expansion processing on the first detection window to obtain a second detection window, intercept a region corresponding to the size of the second detection window, and determine the region corresponding to the size of the second detection window as the face region.

[0118] Next, the specific implementation process of the face angle prediction method of the present application will be described in detail in combination with Figure 3 .

[0119] Based on the description of S101 in the Figure 1 , more face information corresponding to the face image to be measured can be obtained by performing outer expansion processing on the first detection window, so as to ensure that the face angle obtained finally has higher accuracy.

[0120] Please refer to Figure 3 , Figure 3 The flowchart of the face angle prediction method provided by an embodiment of the present application is shown.

[0121] As Figure 3 shown, the face angle prediction method provided by the present application can include:

[0122] S301, taking a window with the center of the first detection window as the center and the long side of the first detection window as the side length as a third detection window.

[0123] In some embodiments, face detection is performed on the face image to be detected, and the first detection window obtained is a rectangle.

[0124] It is understandable that when the first detection window is a rectangle, the recognition device can obtain the third detection window based on the longer of the length and width of the first detection window as the side length and the center of the first detection window as the center.

[0125] In other embodiments, face detection is performed on the face image to be detected, and the first detection window obtained is a square.

[0126] It is understandable that when the first detection window is a square, step S302 can be directly performed on the first detection window according to the expansion coefficient.

[0127] In the face detection process, the first detection window is a rectangle or a square. The rectangle or square shape of the first detection window is usually determined by the distance from the camera, the facial expression or movement, the angle of the face, and other factors.

[0128] In a specific embodiment, assuming that the first detection window is a rectangle with a length of 60 pixels and a width of 40 pixels, then the third detection window obtained by taking the center of the first detection window as the center and the length of the first detection window as the side length is a square with a side length of 60 pixels.

[0129] S302: Perform an outward expansion process on each side of the third detection window according to a preset outward expansion coefficient to obtain a fourth detection window.

[0130] In some embodiments, the preset expansion coefficient is 0.1. Of course, the expansion coefficient can also be other values, such as 0.15, which can be set according to actual conditions and will not be described in detail here.

[0131] In a specific embodiment, assuming that the side length of the third detection window is 60 pixels and the expansion coefficient is 0.1, the third detection window is expanded to obtain a fourth detection window that is a square with a side length of 66 pixels.

[0132] S303: Remove the side length of the fourth detection window that exceeds the side length of the face image to be detected to obtain a fifth detection window.

[0133] It is understandable that after the recognition device expands the third detection window, the fourth detection window obtained may exceed the original face image to be tested, that is, the fourth detection window exceeds the side length corresponding to the face image to be tested.

[0134] When the recognition device determines that the fourth detection window exceeds the side length corresponding to the face image to be detected, the recognition device removes the excess side length to obtain the fifth detection window.

[0135] In a specific embodiment, assuming that the face image to be tested is 90 pixels long and 60 pixels wide, and the side length of the fourth detection window is 66 pixels, the side length exceeding the side length of the face image to be tested is removed, and the resulting fifth detection window is 66 pixels long and 60 pixels wide.

[0136] S304: Use a window with the center of the fifth detection window as the center and the short side of the fifth detection window as the side length as the second detection window.

[0137] In a specific embodiment, assuming that the fifth detection window is 66 pixels long and 60 pixels wide, then, with the center of the fifth detection window as the center and the width of the fifth detection window as the side length, the obtained second detection window is a square with a side length of 60 pixels.

[0138] In the present application, the recognition device uses the center of the first detection window as the center, and the long side of the first detection window as the side length as the third detection window. According to the preset expansion coefficient, each side length of the third detection window is expanded to obtain the fourth detection window. The side length of the fourth detection window that exceeds the corresponding side length of the face image to be tested is removed to obtain the fifth detection window. The window with the center of the fifth detection window as the center and the short side of the fifth detection window as the side length is used as the second detection window. The recognition device uses the second detection window obtained by expanding the first detection window to select a larger face area for the face image to be tested, and includes more face information. By extracting face features from the face area that includes more face information, the obtained face features are more accurate. By predicting the face angle using more accurate face features, a more accurate prediction angle can be obtained.

[0139] Based on the above Figure 1 In addition to the description of the illustrated embodiment, the present application also provides a process for generating a face angle recognition model including a backbone network and a fully connected classification network.

[0140] Next, combine Figure 4 , which details the specific implementation process of generating a face angle recognition model in this application.

[0141] based on Figure 1 As described in S102, when the recognition device obtains the facial features corresponding to the face area, it obtains them through the backbone network in the face angle recognition model.

[0142] based on Figure 1 As described in S103 , when the recognition device obtains the facial features corresponding to the face area, it obtains them through the fully connected classification network in the face angle recognition model.

[0143] Among them, the generation process of the face angle recognition model can be completed by a model generation device, or it can be generated by other feasible devices, which will not be repeated here.

[0144] See also Figure 4 , Figure 4 A schematic diagram of a process for generating a face angle recognition model provided in an embodiment of the present application is shown.

[0145] like Figure 4 As shown, the process of generating the face angle recognition model includes:

[0146] S401: Obtain a sample face image set.

[0147] The sample face image set includes multiple frames of sample face images and the real angle corresponding to each angle type of the face in each frame of the sample face image relative to the shooting position.

[0148] Optionally, the sample face image set includes at least a group of sample face images and real angles corresponding to each angle type of faces in the sample face images relative to the shooting positions.

[0149] The sample face image set can be selected from an existing image dataset (for example, the public dataset 300W-LP), or can be a face image captured in advance by a camera.

[0150] When capturing facial images with a camera, a camera with high precision needs to be used to capture the sample face from multiple angles in order to obtain sample facial images at any angle.

[0151] The camera that captures the face image may be a camera, a smartphone camera, a laptop camera, or a tablet camera.

[0152] The real angle corresponding to the sample face image can be obtained by using relevant sensors or by manual annotation.

[0153] S402: Perform data enhancement processing on each frame of sample face image to obtain an enhanced sample face image.

[0154] Data enhancement processing may include random cropping, adding random noise, color perturbation, or a combination of the two.

[0155] For example, each frame of the sample face image is subjected to random shearing, random noise addition, and color disturbance processing in sequence to obtain an enhanced sample face image.

[0156] S403: Input the enhanced sample face image into the original angle recognition model, and output multiple angle probabilities for each angle type.

[0157] The original face angle recognition model includes the original backbone network and the original fully connected classification network.

[0158] The output ends of the original backbone network are respectively connected to the three fully connected layers of the original fully connected classification network, and the three fully connected layers are respectively the first original fully connected layer, the second original fully connected layer and the third original fully connected layer.

[0159] The first and second original fully connected layers are connected to 36 nodes respectively, and the third original fully connected layer is connected to 72 branches.

[0160] The 36 nodes of the first and second original fully connected layers are used to predict the angle probabilities of the yaw angle and the pitch angle in their 36 angle intervals, respectively, and the 72 nodes of the third original fully connected layer are used to predict the angle probabilities of the roll angle in its 72 angle intervals.

[0161] Taking the yaw angle as an example, after the model generation device inputs the sample image set into the original backbone network, it outputs the facial features. The first original fully connected layer obtains the angle probabilities corresponding to the 36 angle intervals of the yaw angle based on the facial features.

[0162] S404: Adjust the model parameters of the original angle recognition model according to the multiple angle probabilities of each angle type and the real angle corresponding to each angle type.

[0163] In some embodiments, the model generation device first calculates a loss function based on multiple angle probabilities of each angle type and the actual angle corresponding to each angle type, and then adjusts the model parameters of the original angle recognition model through the loss function.

[0164] The above loss function is the cross entropy loss function. Of course, the loss function can also be other types of loss functions, which will not be described in detail here.

[0165] S405: Determine the adjusted original angle recognition model as the face angle recognition model.

[0166] In some embodiments, the model generation device trains the original angle recognition model according to the loss function through the error back propagation algorithm to obtain a trained face angle recognition model, and determines the trained face angle recognition model as a face angle recognition model.

[0167] In the present application, in the process of generating a face angle recognition model by a model generation device, a sample face image is first obtained, and data enhancement processing is performed on each frame of the sample face image to obtain an enhanced sample face image. The enhanced sample face image is then input into the original angle recognition model, and multiple angle probabilities of each angle type are output. According to the multiple angle probabilities of each angle type and the real angle corresponding to each angle type, the model parameters of the original angle recognition model are adjusted, and the adjusted original angle recognition model is determined as a face angle recognition model. By dividing the angle ranges of the three angle types with the help of preset rules, and adjusting the original angle recognition model by the multiple angle probabilities corresponding to the three angle types and the real angles corresponding to the three angle types, a face angle recognition model that more accurately predicts the face angle can be obtained.

[0168] Corresponding to the above Figure 1 The embodiment shown describes a method for predicting a face angle. This application also provides a device for predicting a face angle.

[0169] Next, combine Figure 5 , a face angle prediction device provided in an embodiment of the present application is described in detail.

[0170] See also Figure 5 , Figure 5 A schematic block diagram of a face angle prediction device provided by an embodiment of the present application is shown.

[0171] like Figure 5 As shown, the face angle prediction device provided by an embodiment of the present application includes an acquisition module 501, a first determination module 502, a second determination module 503 and a third determination module 504.

[0172] An acquisition module 501 is used to acquire a face region of a face image to be tested;

[0173] A first determining module 502 is configured to determine facial features corresponding to the facial region;

[0174] A second determining module 503 is configured to determine, based on the facial features, a plurality of angle probabilities for each of a plurality of angle types, wherein the plurality of angle probabilities for each angle type corresponds to a plurality of angle intervals for each angle type, the plurality of angle types comprising yaw angle, pitch angle, and roll angle;

[0175] The third determining module 504 is configured to determine, based on the multiple angle probabilities of each angle type, a predicted angle of each angle type of the face in the face image to be tested relative to the shooting position.

[0176] In some embodiments, the third determining module 504 is specifically configured to:

[0177] For each angle type, the angle of the face in the face image to be tested relative to the shooting position is determined according to multiple angle probabilities, the number of the angle intervals and the middle angle of each angle interval.

[0178] In some embodiments, the third determining module 504 is specifically configured to:

[0179] For the roll angle, determining a maximum probability angle interval, where the maximum probability angle interval is an angle interval corresponding to a maximum value among multiple angle probabilities;

[0180] When the maximum probability angle interval is within a preset mapping angle interval, linearly mapping the multiple angle probabilities of the roll angle to obtain multiple mapped angle probabilities;

[0181] determining a mapping angle according to the mapped multiple angle probabilities, the number of angle intervals of the roll angle, and a middle angle of each angle interval;

[0182] Perform inverse linear mapping on the mapping angle to obtain a predicted angle of the roll angle.

[0183] In some embodiments, the third determining module 504 is specifically configured to:

[0184] The angle probability corresponding to the angle interval greater than or equal to 0 degrees and less than or equal to 180 degrees is replaced by the mapped angle probability corresponding to the angle interval greater than or equal to -180 degrees and less than 0 degrees;

[0185] The angle probability corresponding to the angle interval less than or equal to 0 degrees and greater than or equal to -180 degrees is replaced by the mapped angle probability corresponding to the angle interval less than or equal to 180 degrees and greater than 0 degrees.

[0186] In some embodiments, the acquisition module 501 is specifically configured to:

[0187] The step of obtaining the face region of the face image to be tested includes:

[0188] Acquire the face image to be tested;

[0189] Performing face detection on the face image to be tested to obtain a first detection window, wherein the first detection window includes at least a portion of the face image to be tested;

[0190] Performing an outward expansion process on the first detection window to obtain a second detection window;

[0191] In the face image to be detected, intercepting an area corresponding to the size of the second detection window;

[0192] In some embodiments, the acquisition module 501 is specifically configured to:

[0193] A window centered at the center of the first detection window and having the longer side of the first detection window as the third detection window;

[0194] Expand each side of the third detection window according to a preset expansion coefficient to obtain a fourth detection window;

[0195] Remove the side length of the fourth detection window that exceeds the side length of the face image to be detected, to obtain a fifth detection window;

[0196] A window with the center of the fifth detection window as its center and the short side of the fifth detection window as its side length is used as the second detection window.

[0197] In some embodiments, the first determining module 502 is specifically configured to:

[0198] Inputting the face area into the backbone network of the face angle recognition model and outputting the face features, wherein the backbone network is used to extract the face features in the face image;

[0199] The facial features are input into the fully connected classification network of the facial angle recognition model, and multiple angle probabilities for each angle type are output. The fully connected classification network is used to predict the angle probabilities corresponding to the facial features in multiple angle intervals of each angle type.

[0200] In some embodiments, the model generating device is configured to:

[0201] Acquire a sample face image set, the sample face image set including multiple frames of sample face images and a real angle corresponding to each angle type of a face in each frame of the sample face image relative to a shooting position;

[0202] Perform data enhancement processing on each frame of sample face image to obtain an enhanced sample face image;

[0203] Inputting the enhanced sample face image into an original angle recognition model and outputting multiple angle probabilities for each angle type, wherein the original face angle recognition model includes an original backbone network and an original fully connected classification network;

[0204] Adjusting model parameters of the original angle recognition model according to multiple angle probabilities of each angle type and the actual angle corresponding to each angle type;

[0205] The adjusted original angle recognition model is determined as the face angle recognition model.

[0206] It should be understood that the apparatus 500 of the present application can be implemented by an application-specific integrated circuit (ASIC), or a programmable logic device (PLD), which can be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. It can also be implemented by software Figure 1 The face angle prediction method shown in the software implementation Figure 1 The face angle prediction method shown in the software implementation

[0207] Figure 6 A structural schematic diagram of an electronic device provided in the present application is shown in FIG. 6. As shown in the figure, the device 600 includes a processor 601, a memory 602, a communication interface 603, and a bus 604. The processor 601, the memory 602, and the communication interface 603 communicate through the bus 604, and can also communicate through wireless transmission and other means. The memory 602 is used to store instructions, and the processor 601 is used to execute the instructions stored in the memory 602. The memory 602 stores program code 6021, and the processor 601 can call the program code 6021 stored in the memory 602 to execute the face angle prediction method shown in the software implementation. Figure 6 The face angle prediction method shown in the software implementation. Figure 2

[0208] It should be understood that in the present application, the processor 601 can be a CPU, and the processor 601 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0209] ​The memory 602 may include a read-only memory and a random access memory, and provides instructions and data to the processor 601. The memory 602 may also include a non-volatile random access memory. The memory 602 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0210] In addition to the data bus, the bus 604 may also include a power bus, a control bus, and a status signal bus. Figure 6 Various buses are labeled as bus 604.

[0211] It should be understood that the electronic device 600 according to the present application may correspond to the apparatus 500 in the present application, and may correspond to the apparatus 500 in the present application. Figure 1 In the method shown, when the device 600 corresponds to Figure 2 When the device is used in the method shown, the above and other operations and / or functions of each module in the device 600 are respectively to implement Figure 2 For the sake of brevity, the operating steps of the method executed by the device are not repeated here.

[0212] The present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the steps in the above-mentioned method embodiments.

[0213] The present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device can implement the steps in the above-mentioned various method embodiments when executing the computer program product.

[0214] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application.

[0215] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0216] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0217] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0218] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0219] In the embodiments provided in this application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely illustrative. For example, the division of the above modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0220] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present application.

[0221] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A face angle prediction method, characterized in that: include: Obtain the face area of ​​the face image to be tested; Determining facial features corresponding to the facial region; determining, based on the facial features, a plurality of angle probabilities for each of a plurality of angle types, the plurality of angle probabilities for each angle type corresponding to a plurality of angle intervals for each angle type, the plurality of angle types comprising a yaw angle, a pitch angle, and a roll angle; Determining, based on the multiple angle probabilities for each angle type, a predicted angle of each angle type of the face in the face image to be tested relative to the shooting position; The step of determining the predicted angle of each angle type of the face in the face image to be tested relative to the shooting position based on the multiple angle probabilities of each angle type includes: For each angle type, determining the angle of the face in the face image to be tested relative to the shooting position according to multiple angle probabilities, the number of the angle intervals and the middle angle of each angle interval; The range of the multiple angle intervals of the roll angle is greater than or equal to -180 degrees and less than or equal to 180 degrees, and the mapping angle interval is an angle interval greater than or equal to -180 degrees and less than or equal to -90 degrees or greater than or equal to 90 degrees and less than or equal to 180 degrees. For the roll angle, determining the predicted angle of each angle type of the face in the face image to be tested relative to the shooting position based on the multiple angle probabilities of each angle type includes: Determine a maximum probability angle interval, where the maximum probability angle interval is an angle interval corresponding to a maximum value among the multiple angle probabilities; When the maximum probability angle interval is within the preset mapping angle interval, the angle probability corresponding to the angle interval greater than or equal to 0 degrees and less than or equal to 180 degrees is replaced by the mapped angle probability corresponding to the angle interval greater than or equal to -180 degrees and less than 0 degrees; and the angle probability corresponding to the angle interval less than or equal to 0 degrees and greater than or equal to -180 degrees is replaced by the mapped angle probability corresponding to the angle interval less than or equal to 180 degrees and greater than 0 degrees; determining a mapping angle according to the mapped multiple angle probabilities, the number of angle intervals of the roll angle, and a middle angle of each angle interval; Perform inverse linear mapping on the mapping angle to obtain a predicted angle of the roll angle.

2. The method according to claim 1, wherein The step of obtaining the face region of the face image to be tested includes: Acquire the face image to be tested; Performing face detection on the face image to be tested to obtain a first detection window, wherein the first detection window includes at least a portion of the face image to be tested; Performing an outward expansion process on the first detection window to obtain a second detection window; In the face image to be detected, intercepting an area corresponding to the size of the second detection window; An area of ​​a size corresponding to the second detection window is determined as the face area.

3. The method according to claim 2, wherein The step of performing an outward expansion process on the first detection window to obtain a second detection window includes: A window centered at the center of the first detection window and having the longer side of the first detection window as the third detection window; Expand each side of the third detection window according to a preset expansion coefficient to obtain a fourth detection window; Remove the side length of the fourth detection window that exceeds the side length of the face image to be detected, to obtain a fifth detection window; A window with the center of the fifth detection window as its center and the short side of the fifth detection window as its side length is used as the second detection window.

4. The method according to any one of claims 1 to 3, wherein The determining of facial features corresponding to the facial region includes: Inputting the face area into the backbone network of the face angle recognition model and outputting the face features, wherein the backbone network is used to extract the face features in the face image; The determining, based on the facial features, a plurality of angle probabilities of each of the plurality of angle types includes: The facial features are input into the fully connected classification network of the facial angle recognition model, and multiple angle probabilities for each angle type are output. The fully connected classification network is used to predict the angle probabilities corresponding to the facial features in multiple angle intervals of each angle type.

5. The method according to claim 4, wherein The process of generating the face angle recognition model includes: Acquire a sample face image set, the sample face image set including multiple frames of sample face images and a real angle corresponding to each angle type of a face in each frame of the sample face image relative to a shooting position; Perform data enhancement processing on each frame of sample face image to obtain an enhanced sample face image; Inputting the enhanced sample face image into an original face angle recognition model and outputting multiple angle probabilities for each angle type, wherein the original face angle recognition model includes an original backbone network and an original fully connected classification network; Adjusting the model parameters of the original face angle recognition model according to the multiple angle probabilities of each angle type and the actual angle corresponding to each angle type; The adjusted original face angle recognition model is determined as the face angle recognition model.

6. A face angle prediction device, characterized in that: include: An acquisition module is used to acquire the face area of ​​the face image to be tested; A first determining module, configured to determine facial features corresponding to the facial region; a second determining module, configured to determine, based on the facial features, a plurality of angle probabilities for each of a plurality of angle types, wherein the plurality of angle probabilities for each angle type respectively correspond to a plurality of angle intervals for each angle type, the plurality of angle types comprising a yaw angle, a pitch angle, and a roll angle; The third determination module is used to determine the predicted angle of each angle type of the face in the face image to be tested relative to the shooting position based on multiple angle probabilities of each angle type; wherein, the determination of the predicted angle of each angle type of the face in the face image to be tested relative to the shooting position based on multiple angle probabilities of each angle type includes: for each angle type, determining the angle of the face in the face image to be tested relative to the shooting position based on multiple angle probabilities, the number of the angle intervals and the middle angle of each angle interval; wherein, the range of the multiple angle intervals of the roll angle is greater than or equal to -180 degrees and less than or equal to 180 degrees, and the mapping angle interval is an angle interval greater than or equal to -180 degrees and less than or equal to -90 degrees or greater than or equal to 90 degrees and less than or equal to 180 degrees. For the roll angle, the determination of the multiple angle probabilities of each angle type is performed. The predicted angle of each angle type of the face in the face image to be tested relative to the shooting position includes: determining a maximum probability angle interval, the maximum probability angle interval being the angle interval corresponding to the maximum value of multiple angle probabilities; when the maximum probability angle interval is within the preset mapping angle interval, replacing the angle probability corresponding to the angle interval greater than or equal to 0 degrees and less than or equal to 180 degrees with the mapped angle probability corresponding to the angle interval greater than or equal to -180 degrees and less than 0 degrees; and replacing the angle probability corresponding to the angle interval less than or equal to 0 degrees and greater than or equal to -180 degrees with the mapped angle probability corresponding to the angle interval less than or equal to 180 degrees and greater than 0 degrees; determining a mapping angle based on the multiple mapped angle probabilities, the number of angle intervals of the roll angle and the middle angle of each angle interval; and performing inverse linear mapping on the mapping angle to obtain the predicted angle of the roll angle.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images