Image processing method and device, electronic equipment and storage medium
By calculating the comprehensive uncertainty of the target face in facial landmark detection, the network training process is optimized, which solves the problems of high network structure complexity and insufficient detection accuracy in the existing technology, and realizes efficient facial landmark detection and image processing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XIAOMI TECH (WUHAN) CO LTD
- Filing Date
- 2022-01-21
- Publication Date
- 2026-05-29
AI Technical Summary
Existing facial landmark detection technologies based on deep neural networks suffer from problems such as high network structure complexity, large computational load, high cost, and difficulty in convergence, especially in complex environments where detection accuracy is insufficient.
By acquiring the face image to be tested, the key point information and first uncertainty of the target face are determined. A pre-trained key point detection network is used for detection. The first uncertainty is calculated by combining the second uncertainty of all key points. The network training process is optimized to reduce regression optimization for the uncertainty of each key point.
It improves the accuracy and effectiveness of facial landmark detection, optimizes the structure and training efficiency of image detection networks, is applicable to various task scenarios, and enhances robustness.
Smart Images
Figure CN116543426B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer vision technology, specifically to an image processing method, apparatus, electronic device, and storage medium. Background Technology
[0002] Currently, face recognition based on deep neural networks (DNNs) is one of the most important applications in the field of computer vision (CV). Facial landmark detection refers to locating key feature points of a face from a face image, such as facial contour landmarks and facial feature landmarks. Due to the influence of factors such as pose, occlusion, or lighting, facial landmark detection is a challenging task. Summary of the Invention
[0003] To improve the detection effect of facial landmarks, this disclosure provides an image processing method, apparatus, electronic device, and storage medium.
[0004] In a first aspect, embodiments of this disclosure provide an image processing method, including:
[0005] Acquire a face image to be tested, wherein the face image to be tested includes the target face;
[0006] Image detection is performed on the face image to be tested to determine key point information of at least one key point of the target face and a first uncertainty of the target face; wherein, the first uncertainty is obtained based on the second uncertainty of all key points of the target face;
[0007] Based on the key point information and the first uncertainty, the detection result of the target face is determined.
[0008] In some implementations, acquiring the face image to be tested includes:
[0009] Obtain an image to be processed, wherein the image to be processed includes at least one human face;
[0010] Image detection is performed on the image to be processed to determine the facial region information of each face in the image to be processed;
[0011] For any given face, the image of the face to be tested is obtained by cropping the face based on the face region information.
[0012] In some implementations, the step of performing image detection on the face image to be tested, determining key point information of at least one key point of the target face and a first uncertainty of the target face, includes:
[0013] Key point detection is performed on the face image to be tested, and the key point information and second uncertainty of each key point of the target face are determined based on the pre-set face key point type.
[0014] The first uncertainty of the target face is determined based on the second uncertainty of each key point of the target face.
[0015] In some implementations, the preset key points include at least one of the following types:
[0016] Key points of facial contours, eyes, eyebrows, nose, mouth, and ears.
[0017] In some implementations, determining the detection result of the target face based on the key point information and the first uncertainty includes:
[0018] Based on the first uncertainty of the target face, a reliability score for the target face is determined;
[0019] In response to the reliability score satisfying the first preset condition, the key points are output on the face image to be tested based on the key point information of each key point.
[0020] In some implementations, determining the detection result of the target face based on the key point information and the first uncertainty includes:
[0021] Based on the first uncertainty of the target face, a reliability score for the target face is determined;
[0022] Based on the reliability score and the pre-established correspondence between the reliability score and the face tracking model, the target face tracking model is determined from a set of multiple pre-set face tracking models.
[0023] The target face is detected and tracked using the target face tracking model to obtain the detection result of the target face.
[0024] In some implementations, determining the detection result of the target face based on the key point information and the first uncertainty includes:
[0025] Based on the first uncertainty of the target face, a reliability score for the target face is determined;
[0026] In response to the reliability score meeting the second preset condition, it is determined that the target face detection of the face image to be tested has passed.
[0027] In some implementations, the step of performing image detection on the face image to be tested, determining key point information of at least one key point of the target face and a first uncertainty of the target face, includes:
[0028] The face image to be tested is input into a pre-trained feature extraction network to obtain the feature map output by the feature extraction network.
[0029] The feature map is input into a pre-trained keypoint detection network to obtain the keypoint information of each keypoint of the target face output by the keypoint detection network and the first uncertainty.
[0030] In some embodiments, the method of this disclosure further includes a training process for training the feature extraction network and the keypoint detection network, the training process including:
[0031] Obtain a sample dataset, wherein each sample data in the sample dataset includes a face sample image and a key point label for each key point of the target face in the face sample image;
[0032] For any sample data, the face sample image is input into the feature extraction network to be trained, and the feature map of the face sample image output by the feature extraction network is obtained.
[0033] The feature map of the face sample image is input into the key point detection network to be trained to obtain the key point information of each key point of the target face, as well as the first uncertainty of the target face;
[0034] Based on the key point information, key point labels, and the first uncertainty, determine the difference between the key point information and the key point labels;
[0035] The network parameters of the feature extraction network and / or the key point detection network are adjusted according to the differences until the convergence condition is met, thereby obtaining the trained feature extraction network and / or the key point detection network.
[0036] In a second aspect, embodiments of this disclosure provide an image processing apparatus, including:
[0037] The acquisition module is configured to acquire a face image to be tested, wherein the face image to be tested includes a target face;
[0038] An image detection module is configured to perform image detection on the face image to be tested, determine key point information of at least one key point of the target face and a first uncertainty of the target face; wherein the first uncertainty is obtained based on the second uncertainty of all key points of the target face;
[0039] The result determination module is configured to determine the detection result of the target face based on the key point information and the first uncertainty.
[0040] In some implementations, the acquisition module is configured to:
[0041] Obtain an image to be processed, wherein the image to be processed includes at least one human face;
[0042] Image detection is performed on the image to be processed to determine the facial region information of each face in the image to be processed;
[0043] For any given face, the image of the face to be tested is obtained by cropping the face based on the face region information.
[0044] In some implementations, the image detection module is configured to:
[0045] Key point detection is performed on the face image to be tested, and the key point information and second uncertainty of each key point of the target face are determined based on the pre-set face key point type.
[0046] The first uncertainty of the target face is determined based on the second uncertainty of each key point of the target face.
[0047] In some implementations, the preset key points include at least one of the following types:
[0048] Key points of facial contours, eyes, eyebrows, nose, mouth, and ears.
[0049] In some implementations, the result determination module is configured to:
[0050] Based on the first uncertainty of the target face, a reliability score for the target face is determined;
[0051] In response to the reliability score satisfying the first preset condition, the key points are output on the face image to be tested based on the key point information of each key point.
[0052] In some implementations, the result determination module is configured to:
[0053] Based on the first uncertainty of the target face, a reliability score for the target face is determined;
[0054] Based on the reliability score and the pre-established correspondence between the reliability score and the face tracking model, the target face tracking model is determined from a set of multiple pre-set face tracking models.
[0055] The target face is detected and tracked using the target face tracking model to obtain the detection result of the target face.
[0056] In some implementations, the result determination module is configured to:
[0057] Based on the first uncertainty of the target face, a reliability score for the target face is determined;
[0058] In response to the reliability score meeting the second preset condition, it is determined that the target face detection of the face image to be tested has passed.
[0059] In some embodiments, the image detection module includes:
[0060] The feature extraction module is configured to input the face image to be tested into a pre-trained feature extraction network to obtain a feature map output by the feature extraction network.
[0061] The key point detection module is configured to input the feature map into a pre-trained key point detection network to obtain the key point information of each key point of the target face output by the key point detection network and the first uncertainty.
[0062] In some embodiments, the apparatus described in this disclosure further includes a training module, which is configured to:
[0063] Obtain a sample dataset, wherein each sample data in the sample dataset includes a face sample image and a key point label for each key point of the target face in the face sample image;
[0064] For any sample data, the face sample image is input into the feature extraction network to be trained, and the feature map of the face sample image output by the feature extraction network is obtained.
[0065] The feature map of the face sample image is input into the key point detection network to be trained to obtain the key point information of each key point of the target face, as well as the first uncertainty of the target face;
[0066] Based on the key point information, key point labels, and the first uncertainty, determine the difference between the key point information and the key point labels;
[0067] The network parameters of the feature extraction network and / or the key point detection network are adjusted according to the differences until the convergence condition is met, thereby obtaining the trained feature extraction network and / or the key point detection network.
[0068] Thirdly, embodiments of this disclosure provide an electronic device, including:
[0069] Processor; and
[0070] The memory stores computer instructions that can be read by the processor, and when the computer instructions are read, the processor executes the method according to any embodiment of the first aspect.
[0071] Fourthly, embodiments of this disclosure provide a storage medium for storing computer-readable instructions for causing a computer to perform the method according to any embodiment of the first aspect.
[0072] The image processing method of this disclosure includes acquiring a face image to be tested, performing image detection on the face image to be tested, determining key point information of at least one key point of the target face and a first uncertainty of the target face, and determining the detection result of the target face based on the key point information and the first uncertainty. This disclosure improves the face detection effect and accuracy by using the first uncertainty of the target face to assist in the detection of facial key points, and is applicable to various task scenarios. Furthermore, it improves network performance and training efficiency by determining a first uncertainty representing the comprehensive error of the target face based on the second uncertainty of all key points of the target face. Attached Figure Description
[0073] To more clearly illustrate the technical solutions in the specific embodiments of this disclosure or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0074] Figure 1 This is a flowchart of an image processing method according to some embodiments of the present disclosure.
[0075] Figure 2 This is a flowchart of an image processing method according to some embodiments of the present disclosure.
[0076] Figure 3 This is a flowchart of an image processing method according to some embodiments of the present disclosure.
[0077] Figure 4 This is a schematic diagram of facial key points according to some embodiments of this disclosure.
[0078] Figure 5 This is a schematic diagram of the structure of an image detection network according to some embodiments of the present disclosure.
[0079] Figure 6 This is a flowchart of an image processing method according to some embodiments of the present disclosure.
[0080] Figure 7 This is a flowchart of an image processing method according to some embodiments of the present disclosure.
[0081] Figure 8 This is a flowchart of an image processing method according to some embodiments of the present disclosure.
[0082] Figure 9 This is a flowchart of an image processing method according to some embodiments of the present disclosure.
[0083] Figure 10 This is a structural block diagram of an image processing apparatus according to some embodiments of the present disclosure.
[0084] Figure 11 This is a structural block diagram of an image processing apparatus according to some embodiments of the present disclosure.
[0085] Figure 12 This is a structural block diagram of an electronic device according to some embodiments of the present disclosure. Detailed Implementation
[0086] The technical solutions of this disclosure will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure. Furthermore, the technical features involved in the different embodiments of this disclosure described below can be combined with each other as long as they do not conflict with each other.
[0087] Facial landmark detection is an essential method for face recognition tasks. Facial landmark detection refers to locating the key features of a face from a face image, such as facial contour landmarks and facial feature landmarks. Facial contour landmarks can include chin landmarks, jaw landmarks, cheek landmarks, etc., while facial feature landmarks can include eye landmarks, eyebrow landmarks, nose landmarks, mouth landmarks, ear landmarks, etc.
[0088] Currently, facial landmark localization based on deep neural networks (DNNs) is the most efficient and commonly used detection method. In related technologies, to improve the accuracy of DNN's landmark prediction and localization, an uncertainty parameter is designed for each landmark. The uncertainty of each facial landmark is predicted using the DNN, and regression prediction of the landmark coordinates is performed based on the uncertainty, thus allowing the DNN to output facial landmarks with relatively high accuracy.
[0089] However, in this approach, because each facial landmark detected by the DNN needs to be regressed with an uncertainty, for networks requiring high detection accuracy, the number of facial landmarks can reach hundreds or even thousands. This results in a very large complexity and computational burden for the DNN network structure, leading to high costs. Furthermore, during DNN training, it is impossible to design a clear optimization objective for the uncertainty of facial landmarks, causing the network to have difficulty converging and resulting in poor performance in actual use.
[0090] In view of the above-mentioned deficiencies, the present disclosure provides an image processing method, apparatus, electronic device and storage medium, which aims to improve the accuracy of facial key point localization and optimize the structure and effect of image detection network.
[0091] In a first aspect, embodiments of this disclosure provide an image processing method that can be applied to electronic devices. In these embodiments, the electronic device can be any suitable type of device, such as a mobile terminal, vehicle-mounted terminal, wearable device, access control system, video surveillance system, cloud platform, and server, etc., and this disclosure does not impose any limitations on this.
[0092] like Figure 1 As shown, in some embodiments, the image processing method of this disclosure includes:
[0093] S110. Obtain the face image to be tested, which includes the target face.
[0094] Specifically, the face image to be tested refers to an image from which a face object is expected to be detected. Thus, the face image to be tested may include one or more face objects, which are the target faces.
[0095] In this embodiment of the disclosure, the face image to be tested can be a single frame image captured by the image acquisition device of an electronic device, or a frame image in a video stream captured by the image acquisition device of an electronic device.
[0096] For example, in one example, the electronic device is a smartphone, which includes a camera that can capture an image including a face, and this image can be used as the face image to be tested as described in this disclosure.
[0097] For example, in another example, the electronic device is a video surveillance system, which includes a surveillance camera. The surveillance camera can capture a video stream including a face in the target scene area, and the frame images in the video stream can be used as the face image to be tested as described in this disclosure.
[0098] In summary, the face image to be tested can be any image from which a face object is expected to be detected. It can be an image acquired in real time or a face image uploaded or downloaded from the network. This disclosure will not elaborate further on this.
[0099] In some implementations, considering that facial images acquired by electronic devices often contain numerous interfering factors, such as multiple facial objects or large non-facial regions, the facial image can be pre-cropped to improve the accuracy of subsequent keypoint detection. The cropped image containing only one facial object can then be used as the facial image to be tested. This will be described in the following implementations, and will not be detailed here.
[0100] S120. Perform image detection on the face image to be tested to determine the key point information of at least one key point of the target face and the first uncertainty of the target face.
[0101] Specifically, facial landmark detection requires detecting multiple facial landmarks from the image of the face to be tested. These facial landmarks can belong to different types. Landmark types can include, for example, facial contour landmarks, eye landmarks, eyebrow landmarks, nose landmarks, mouth landmarks, ear landmarks, etc. Each landmark type can include multiple landmarks; for example, eyebrow landmarks can include 5*2, a total of 10 landmarks.
[0102] In this embodiment of the disclosure, the key point information may include the key point coordinates corresponding to each key point. For example, image detection technology can be used to perform image detection on the face image to be tested, thereby detecting all the key points of the target face in the face image to be tested, as well as the position coordinates of each key point in the image.
[0103] In addition, in this embodiment, key point detection also requires determining a first uncertainty of the target face. The first uncertainty represents the combined error of all key points of the target face; that is, the first uncertainty of the target face is obtained based on the second uncertainty of all key points. The higher the first uncertainty, the greater the error in key point detection of the target face; conversely, the lower the first uncertainty, the smaller the error in key point detection of the target face.
[0104] In some implementations, a pre-trained keypoint detection network can be used to predict the position coordinates of each keypoint on the target face, thereby obtaining keypoint information for each keypoint. Simultaneously, the keypoint detection network can also predict the uncertainty of each keypoint, obtaining a second uncertainty corresponding to each keypoint.
[0105] Taking a keypoint A among the eyebrow keypoints of a target face as an example, the keypoint detection network can predict the position coordinates (x, y) and second uncertainty p of keypoint A based on the eyebrow features of the target face. The second uncertainty p represents the error of the position coordinates (x, y) of the keypoint. The larger the second uncertainty p, the larger the error of the position coordinates (x, y), and vice versa.
[0106] It is understood that the above explanation uses only one key point as an example. For all key points of the target face, each key point corresponds to key point information and a second uncertainty. In this embodiment, the detection result of the target face is not directly determined based on the second uncertainty of each key point. Instead, the first uncertainty of the target face is calculated based on the second uncertainty of all key points, and the first uncertainty is used as the comprehensive uncertainty corresponding to the target face.
[0107] In some embodiments, the root mean square of the second uncertainties of all key points can be used as the first uncertainty corresponding to the target face. In other embodiments, the mean of the second uncertainties of all key points can be used as the first uncertainty corresponding to the target face. Of course, it is understood that other methods can be used to fuse the second uncertainties of all key points to obtain the first uncertainty, as long as the first uncertainty can represent the comprehensive error of the key points, and this disclosure does not impose any limitations on this.
[0108] S130. Based on the key point information and the first uncertainty, determine the detection result of the target face.
[0109] Specifically, after determining the key point information of each key point of the target face and the first uncertainty corresponding to the target face, the corresponding post-processing logic can be set according to different downstream task scenarios, so as to obtain the detection result of the target face based on the key point information and the first uncertainty.
[0110] In one example, taking the input of a facial photo into the database, the facial image uploaded by the user must meet certain requirements, such as not obscuring eyebrows or tilting the head too much. Therefore, through the process described above, image processing can be performed on the user-uploaded facial photo to obtain key point information for each key point of the facial image, as well as the first uncertainty for the target face. When the first uncertainty is greater than a preset threshold, it indicates that the key point detection of the user-uploaded facial image has a large deviation, possibly due to issues such as occlusion of facial features. Therefore, the corresponding detection result can be output to the user as "failed," indicating that a certain facial feature is occluded.
[0111] In another example, taking a face tracking scenario, the operating conditions of electronic devices differ under different lighting conditions. For instance, in extremely low light, the exposure of the face image captured by the electronic device is very low, resulting in a large initial uncertainty in keypoint detection. Conversely, in bright light, the exposure of the face image captured by the electronic device is normal, resulting in a lower initial uncertainty in keypoint detection. Based on this, by setting an appropriate threshold, the initial uncertainty can be used to determine the current lighting environment of the device, thereby enabling the use of a corresponding tracking algorithm model to achieve face tracking.
[0112] Of course, it is understood that the scenarios described in this disclosure are not limited to the examples above, and this will be explained in detail below, but will not be elaborated here.
[0113] It is worth noting that in this embodiment, the first uncertainty representing the comprehensive error of the target face is determined based on the second uncertainty of all key points of the target face. Thus, for the key point detection network, it is not necessary to perform regression optimization on the uncertainty of each key point during training. Instead, the comprehensive uncertainty of the face is optimized, making the network easier to converge, achieving better results, and greatly improving training efficiency.
[0114] As described above, in this embodiment, the detection of facial key points is aided by the first uncertainty of the target face, thereby improving the face detection effect and accuracy. Furthermore, the first uncertainty representing the comprehensive error of the target face is determined based on the second uncertainty of all key points of the target face, improving network performance and training efficiency. Moreover, this method is not limited to any particular application scenario and can be applied to various downstream tasks, such as face image quality detection, face tracking, and key point localization, exhibiting higher robustness.
[0115] like Figure 2 As shown, in some embodiments, the process of acquiring a face image to be tested in the image processing method of this disclosure includes:
[0116] S210. Obtain the image to be processed, which includes at least one human face.
[0117] S220. Perform image detection on the image to be processed to determine the facial region information of each face in the image to be processed.
[0118] S230. For any face, crop the corresponding face image based on the face region information to obtain the face image to be tested.
[0119] Specifically, the image to be processed can be a raw image captured by the image acquisition device of an electronic device, or an uploaded image uploaded to the electronic device by a user. It is understood that the image to be processed may contain one face or multiple faces.
[0120] In this embodiment of the disclosure, image detection technology can be used to perform image detection on the image to be processed, thereby obtaining the face region information of each face in the image to be processed. For example, in one example, image detection can be performed on the image to be processed using, for example, the CenterFace network, thereby obtaining the face detection box of each face region in the image to be processed, which is also the face region information.
[0121] After obtaining the face detection bounding box for each face, the image to be processed can be cropped based on the face detection bounding box to obtain a face image including each face region, which is the face image to be tested.
[0122] In one example, the center point of each face detection box can be used as the origin. Keeping the origin coordinates unchanged, the face detection box is uniformly expanded outward at a preset ratio, and the face image is cropped out along the expanded face detection box.
[0123] It is understandable that when the image to be processed contains multiple faces, by... Figure 2 The implementation process can crop out the facial image of each face, and these facial images can all be used as the face image to be tested as described in this disclosure.
[0124] In some implementations, after obtaining the image of the face to be tested, key point detection can be performed on the target face in the image of the face to be tested based on image detection technology. The following section combines... Figure 3 Please provide an explanation.
[0125] like Figure 3 As shown, in some embodiments, the image processing method of this disclosure, in the process of image detection of the face image to be detected, includes:
[0126] S310. Perform key point detection on the face image to be tested. Based on the pre-set face key point type, determine the key point information and second uncertainty of each key point of the target face.
[0127] S320. Determine the first uncertainty of the target face based on the second uncertainty of each key point of the target face.
[0128] Specifically, when performing keypoint detection on a target face, it is necessary to detect keypoints belonging to one or more keypoint types from the image of the face to be tested. Keypoint types include, for example, eye keypoints, eyebrow keypoints, nose keypoints, mouth keypoints, facial contour keypoints, etc., where each keypoint type may include multiple keypoints.
[0129] For example Figure 4 As shown in Table 1, the pre-defined facial landmark types can include the following:
[0130] Table 1
[0131] Facial landmark types Key point numbering Key points of facial contour 0～32 Key points of eyebrows 33～42 Key points of the nose 43～51 Eye Key Points 52～63 Key points of the mouth 64～83
[0132] It is understood that the types of facial key points are not limited to the examples in Table 1 above, and may include any other key point types suitable for implementation, such as ear key points, cheek key points, etc., and this disclosure does not impose any restrictions on them.
[0133] In this embodiment of the disclosure, the above-mentioned key point type can be detected based on image detection of the face image to be tested, thereby determining the key point information of each key point of the target face and the second uncertainty of each key point.
[0134] As mentioned above, for any keypoint, its keypoint information includes its position coordinates in the image coordinate system, while the second uncertainty represents the degree of uncertainty in the keypoint's localization result. Therefore, for all keypoints of the target face, the first uncertainty corresponding to the target face can be calculated based on the second uncertainty of each keypoint. For example, in one scenario, the root mean square of the second uncertainties of all keypoints can be used as the first uncertainty corresponding to the target face.
[0135] In some implementations, key point detection of the target face in the face image to be tested can be achieved based on a pre-trained image detection network. Figure 5 The image detection network structure of some embodiments of this disclosure is shown below, in conjunction with... Figure 5 Please provide an explanation.
[0136] like Figure 5 As shown, in some embodiments, the image detection network of this disclosure includes a feature extraction network 510 and a keypoint detection network 520.
[0137] The feature extraction network 510 is the backbone network of the image detection network. It is mainly used to extract features from the face image to be tested, thereby obtaining a feature map that includes semantic and texture features of the face. That is, the input of the feature extraction network 510 is the face image to be tested, and the output is the feature map of the face image to be tested.
[0138] In some example implementations, the feature extraction network 510 may employ a learnable network based on a convolutional neural network (CNN) architecture. For example, in one example, to facilitate deployment in a mobile terminal, the feature extraction network 510 may employ a relatively lightweight MobileNet neural network.
[0139] The keypoint detection network 520 is used to predict keypoint information and a first uncertainty based on the feature map output by the feature extraction network 510. For example... Figure 5 In the example, the keypoint detection network 520 has a network structure consisting of two branches, meaning the output layer is divided into two fully connected layers. One branch is for keypoint information prediction, which performs regression prediction on the position coordinates of each keypoint of the target face to obtain the keypoint information for each keypoint. The other branch is for uncertainty prediction, which predicts the first uncertainty of the output target face based on the uncertainty of each keypoint.
[0140] In one example, the pooling layer of the keypoint detection network 520 uses a 7*7 pooling layer, and each fully connected layer uses a 256*1 dimension fully connected layer.
[0141] In some implementations, utilizing Figure 5 Before processing the face image to be tested, the image detection network shown also includes a normalization process. The purpose of the normalization process is to normalize the pixel values of the face image to be tested, so as to obtain an input image that meets the network design requirements and reduce the amount of computation.
[0142] In one example, before inputting the face image to be tested into the image detection network, the face image can first be scaled to a preset size, such as 112 pixels * 112 pixels, using, for example, bilinear interpolation, and the image can be pixel normalized, as shown below:
[0143] I Norm =(I-127.5) / 127.5 Formula (1)
[0144] In equation (1), I Norm The image pixel values are normalized, and I represents the pixel values of the original image. The normalized image is used as the input image for the image detection network.
[0145] In this embodiment of the invention, after obtaining the key point information of the target face predicted by the image detection network and the first uncertainty of the target face, different detection results for the target face can be obtained according to the specific requirements of the downstream task, which will be described below.
[0146] For example, in some scenarios, it is desirable to detect faces from scene images and display the visualization of facial key points within the scene image. Figure 6 As shown, in this scenario, the image processing method of this disclosure determines the detection result of the target face by including:
[0147] S131-1. Determine the reliability score of the target face based on the first uncertainty of the target face.
[0148] S132-2. In response to the reliability score meeting the first preset condition, output key points on the face image to be tested based on the key point information of each key point.
[0149] Specifically, after obtaining the key point information of the target face output by the image detection network and the first uncertainty of the target face, the reliability score of the target face can be calculated based on the first uncertainty.
[0150] It is understandable that the first uncertainty represents the comprehensive error in detecting and locating key points of the target face. It reflects the reliability of the detected key point information, and the reliability score of the target face can be determined based on this.
[0151] In one example, the first uncertainty output by the image detection network is a value between 0 and 1, so the reliability score of the determined target face can be expressed as:
[0152] θ = 1 - α (Equation (2))
[0153] In equation (2), θ represents the reliability score of the target face, and α represents the first uncertainty of the target face.
[0154] In this embodiment, a first preset threshold can be set in advance based on prior knowledge or scenario requirements. The first preset threshold represents the critical value for whether the key point detection result of the target face passes or fails. When the reliability score is greater than the first preset threshold, it indicates that the detection result of the target face is reliable, that is, the detection passes and the first preset condition is met. Conversely, when the reliability score is not greater than the first preset threshold, it indicates that the detection result of the target face is unreliable, that is, the detection fails and the first preset condition is not met.
[0155] Once the reliability score meets the first preset condition, each key point can be marked on the original face image under test based on the key point information of each key point. This allows the user to see the position of each key point on the image, thus realizing the visualization output of the face key points.
[0156] For example, in some scenarios, real-time face tracking often requires different tracking models for different conditions. For instance, in extreme scenarios such as extremely low light, backlighting, or blurriness, a face tracking model suitable for these extreme scenarios is needed; while in ordinary scenarios with good lighting, a face tracking model suitable for ordinary scenarios is sufficient. Therefore, in some embodiments of this disclosure, the complexity of the current scene can be determined based on a first uncertainty, enabling the switching of the face tracking model. The following section will discuss this further. Figure 7 The implementation method is described below.
[0157] like Figure 7 As shown, in some embodiments, the image processing method of this disclosure determines the detection result of the target face by including:
[0158] S132-1. Determine the reliability score of the target face based on the first uncertainty of the target face.
[0159] S132-2. Based on the reliability score and the pre-established correspondence between the reliability score and the face tracking model, the target face tracking model is determined from multiple pre-set face tracking models.
[0160] S132-3. Use the target face tracking model to detect and track the target face to obtain the detection result of the target face.
[0161] Specifically, in this example, it can be based on the foregoing Figure 6 The process of implementing the method and determining the reliability score of the target face will not be described in detail in this disclosure.
[0162] Understandably, the initial uncertainty of keypoint detection should differ for face images under different lighting conditions. For example, in extremely low light, the exposure of the face image captured by the electronic device is very low, resulting in a larger initial uncertainty of keypoint detection and a correspondingly lower reliability score for the target face. Conversely, in bright light, the exposure of the face image captured by the electronic device is normal, resulting in a lower initial uncertainty of keypoint detection and a correspondingly higher reliability score for the target face.
[0163] Based on this, a correspondence between reliability scores and face tracking models can be pre-established using prior knowledge or a limited number of trials. In one example, the pre-established correspondence can be shown in Table 2 below:
[0164] Table 2
[0165] Reliability score Face tracking model Lighting Scene [0，0.6) Model 1 ordinary scene [0.6，1] Model 2 Dark scenes
[0166] Therefore, after determining the reliability score of the target face, the face tracking model corresponding to the reliability score can be determined as the target face tracking model according to the correspondence in Table 2 above. Then, the target face tracking model can be used to detect and track the target face. For example, in one example, the reliability score of the target face in the image to be detected is 0.8. Based on the correspondence in Table 2 above, the current scene can be determined to be a normal scene, and the corresponding target face tracking model is "Model 1". Therefore, Model 1 can be used to track and detect the target face to obtain the face detection result.
[0167] As can be seen from the above, in this example implementation, the current lighting scene can be judged based on the reliability score, thereby selecting the corresponding face tracking model for face tracking and detection, and improving the effect of the detection system.
[0168] For example, in some scenarios, the method disclosed herein can be used to perform quality checks on photos uploaded to a database. For instance, in facial recognition scenarios such as identity verification, users often need to upload compliant facial photos in advance to serve as template photos for subsequent identity verification. In this case, the method disclosed herein can be used to check the uploaded photos to determine whether they are qualified. The following section combines... Figure 8 The implementation method is described below.
[0169] like Figure 8 As shown, in some embodiments, the image processing method of this disclosure determines the detection result of the target face by including:
[0170] S133-1. Determine the reliability score of the target face based on the first uncertainty of the target face.
[0171] S133-2, In response to the reliability score meeting the second preset condition, it is determined that the target face detection of the face image to be tested has passed.
[0172] Specifically, after a user uploads a facial image or a user's facial image is collected through an electronic device, the facial image can be used as the face image to be tested as described in the foregoing embodiments of this disclosure. By performing key point detection on the face image to be tested based on the method of the foregoing embodiments, the key point information of the target face and the first uncertainty can be obtained.
[0173] In this example, it can be based on the foregoing Figure 6 The process of implementing the method and determining the reliability score of the target face will not be described in detail in this disclosure.
[0174] It is understandable that the photos required for facial recognition often need to meet certain requirements, such as no facial obstruction and the facial tilt angle not being too large. These interfering factors can cause the missing or offset of facial key points, resulting in a large uncertainty in the first detection of key points.
[0175] Based on this, a second preset threshold can be pre-set based on prior knowledge or a limited number of trials. This second preset threshold represents the critical value for whether a target face passes detection. When the reliability score is greater than the second preset threshold, it indicates that the target face has passed detection, meets the second preset condition, and can be added to the database. Conversely, when the reliability score is not greater than the second preset threshold, it indicates that the target face has failed detection, does not meet the second preset condition, and cannot be added to the database.
[0176] In some implementations, if the target face detection fails, key points that do not meet the requirements can be identified based on key point information, thereby outputting prompt information to the user, such as "eyebrows are obstructed".
[0177] As can be seen from the above, the method of this disclosure can be applied to various face recognition scenarios, can distinguish image quality or current environmental conditions based on a first uncertainty, has strong practicality and robustness, and improves the effect of face recognition tasks.
[0178] It is worth noting that, in the embodiments of this disclosure, for example... Figure 5 The image detection network shown does not require regression optimization for the uncertainty of each keypoint during training. Instead, it optimizes the overall uncertainty of the face, making the network converge more easily, achieving better results, and significantly improving training efficiency. The following section will combine... Figure 9 The implementation method provides a detailed description of the training process.
[0179] like Figure 9 As shown, in some embodiments, the image processing method of this disclosure, in the process of training an image detection network, includes:
[0180] S910. Obtain the sample dataset.
[0181] Specifically, the sample dataset includes a massive amount of sample data; for example, in one instance, the sample dataset includes 5,000 sample images. For each sample image, it includes a face sample image and keypoint labels for each keypoint of the target face in the pre-annotated face sample image.
[0182] As can be understood, keypoint labels represent the ground truth values of each keypoint in a face sample image. Keypoint labels can be obtained through manual annotation. For example, in one example, the coordinates of N keypoints of the target face in a face sample image can be marked manually to obtain the keypoint label for each face sample image.
[0183] In some implementations, the massive amounts of data in the sample dataset can be preprocessed, and the preprocessing process can be referred to the above. Figure 2 The implementation method involves cropping the face region from the face sample image and using it as the input image for the image detection network.
[0184] S920. For any sample data, input the face sample image into the feature extraction network to be trained, and obtain the feature map of the face sample image output by the feature extraction network.
[0185] In this embodiment of the disclosure, the network structure of the image detection network can be referred to the foregoing. Figure 5 The implementation method is shown below. When training an image detection network using a sample dataset, every n samples can be considered as a batch of training samples, typically n can be 256. The training process is explained below using a single sample dataset as an example.
[0186] In some implementations, the face sample images can be normalized before being input into the image detection network. The normalization process can be referred to the aforementioned equation (1), and will not be repeated here.
[0187] The sample data, including the face sample images, is input into the feature extraction network 510 to be trained, so that the feature extraction network 510 outputs the feature map corresponding to the face sample image.
[0188] S930. Input the feature map of the face sample image into the key point detection network to be trained to obtain the key point information of each key point of the target face, as well as the first uncertainty of the target face.
[0189] Specifically, the feature map output by the feature extraction network 510 is used as the input of the key point detection network 520. After passing through the pooling layer and fully connected layer of the key point detection network 520, the key point information P of the target face and the first uncertainty α of the target face are output respectively.
[0190] In one example, the key point information of the target face output by the key point detection network 520 is represented as follows:
[0191] P = {(x1,y1),(x2,y2),...(x...} i ,y i )} i＝1,2,3,..,N Equation (3)
[0192] In equation (3), P represents key point information, N represents the number of key points, (x i y i ) represents the position coordinates of the i-th key point.
[0193] S940. Based on key point information, key point labels, and the first uncertainty, determine the difference between key point information and key point labels.
[0194] Specifically, keypoint information can include the location coordinates of keypoints predicted by the image detection network, while keypoint labels represent the true coordinates of the keypoints. The difference between the two, i.e., the loss, can be calculated based on a pre-built loss function.
[0195] It is worth noting that in the method of this disclosure, the image detection network is not optimized and trained solely based on the difference between key point information and key point labels. Instead, the first uncertainty is fused and optimized simultaneously, thus eliminating the need to set additional labels for the first uncertainty and making the network more likely to converge.
[0196] In some implementations, the image processing network employs a multi-objective constrained loss function, expressed as follows:
[0197] L = L p +λ*L α Equation (4)
[0198] In equation (4), L represents the loss between key point information and key point labels. p L represents the keypoint error loss function. a The uncertainty error loss function is expressed as follows:
[0199]
[0200] L p =f(σ p Equation (6)
[0201] L α =f(σ p -α) Equation (7)
[0202] f(x) = |x| Equation (8)
[0203] In equations (5) to (8), σ p Let α represent the root mean square error of all key points on the target face, α represent the first uncertainty of the prediction output, f represent the L1 loss function, and x represent the root mean square error of all key points on the target face. i and Let x and y represent the x-coordinate label and predicted value of the i-th keypoint, respectively. i and These represent the y-coordinate label and predicted value of the i-th keypoint, respectively.
[0204] S950. Adjust the network parameters of the feature extraction network and / or key point detection network according to the differences until the convergence condition is met, and obtain the trained feature extraction network and / or key point detection network.
[0205] Specifically, after determining the difference between the predicted and labeled values, the network parameters of the feature extraction network and / or keypoint detection network can be optimized and adjusted through backpropagation based on this difference. This process is repeated iteratively using sample data from the sample dataset to optimize the image detection network until the convergence condition is met, at which point network training is complete.
[0206] It is worth noting that in this embodiment, by constructing a loss function as shown in Equation (4), the first uncertainty of the target face is fused to optimize the network training, thereby improving the performance of the image processing network. Furthermore, the constructed loss function has a simple structure, eliminating the need to set additional labels for the first uncertainty, thus optimizing the first uncertainty and making the network more likely to converge. Moreover, since the first uncertainty is the comprehensive uncertainty of the target face, there is no need to perform separate regression optimization for each key point during training, simplifying computation and improving network training efficiency.
[0207] Secondly, this disclosure provides an image processing apparatus that can be applied to an electronic device. In this disclosure, the electronic device can be any suitable type of device, such as a mobile terminal, vehicle-mounted terminal, wearable device, access control system, video surveillance system, cloud platform, and server, etc., and this disclosure does not impose any limitations on this.
[0208] like Figure 10 As shown, in some embodiments, the image processing apparatus of this disclosure includes:
[0209] The acquisition module 10 is configured to acquire a face image to be tested, wherein the face image to be tested includes a target face;
[0210] The image detection module 20 is configured to perform image detection on the face image to be tested, determine key point information of at least one key point of the target face and a first uncertainty of the target face; wherein the first uncertainty is obtained based on the second uncertainty of all key points of the target face;
[0211] The result determination module 30 is configured to determine the detection result of the target face based on the key point information and the first uncertainty.
[0212] As described above, in this embodiment, the detection of facial key points is aided by the first uncertainty of the target face, thereby improving the face detection effect and accuracy. Furthermore, the first uncertainty representing the comprehensive error of the target face is determined based on the second uncertainty of all key points of the target face, improving network performance and training efficiency. Moreover, this method is not limited to any particular application scenario and can be applied to various downstream tasks, such as face image quality detection, face tracking, and key point localization, exhibiting higher robustness.
[0213] In some embodiments, the acquisition module 10 is configured to:
[0214] Obtain an image to be processed, wherein the image to be processed includes at least one human face;
[0215] Image detection is performed on the image to be processed to determine the facial region information of each face in the image to be processed;
[0216] For any given face, the image of the face to be tested is obtained by cropping the face based on the face region information.
[0217] In some embodiments, the image detection module 20 is configured to:
[0218] Key point detection is performed on the face image to be tested, and the key point information and second uncertainty of each key point of the target face are determined based on the pre-set face key point type.
[0219] The first uncertainty of the target face is determined based on the second uncertainty of each key point of the target face.
[0220] In some implementations, the preset key points include at least one of the following types:
[0221] Key points of facial contours, eyes, eyebrows, nose, mouth, and ears.
[0222] In some implementations, the result determination module 30 is configured to:
[0223] Based on the first uncertainty of the target face, a reliability score for the target face is determined;
[0224] In response to the reliability score satisfying the first preset condition, the key points are output on the face image to be tested based on the key point information of each key point.
[0225] In some implementations, the result determination module 30 is configured to:
[0226] Based on the first uncertainty of the target face, a reliability score for the target face is determined;
[0227] Based on the reliability score and the pre-established correspondence between the reliability score and the face tracking model, the target face tracking model is determined from a set of multiple pre-set face tracking models.
[0228] The target face is detected and tracked using the target face tracking model to obtain the detection result of the target face.
[0229] In some implementations, the result determination module 30 is configured to:
[0230] Based on the first uncertainty of the target face, a reliability score for the target face is determined;
[0231] In response to the reliability score meeting the second preset condition, it is determined that the target face detection of the face image to be tested has passed.
[0232] like Figure 11 As shown, in some embodiments, the image detection module 20 includes:
[0233] Feature extraction module 40 is configured to input the face image to be tested into a pre-trained feature extraction network to obtain a feature map output by the feature extraction network;
[0234] The key point detection module 50 is configured to input the feature map into a pre-trained key point detection network to obtain the key point information of each key point of the target face output by the key point detection network and the first uncertainty.
[0235] In some embodiments, the apparatus described in this disclosure further includes a training module 60, which is configured to:
[0236] Obtain a sample dataset, wherein each sample data in the sample dataset includes a face sample image and a key point label for each key point of the target face in the face sample image;
[0237] For any sample data, the face sample image is input into the feature extraction network to be trained, and the feature map of the face sample image output by the feature extraction network is obtained.
[0238] The feature map of the face sample image is input into the key point detection network to be trained to obtain the key point information of each key point of the target face, as well as the first uncertainty of the target face;
[0239] Based on the key point information, key point labels, and the first uncertainty, determine the difference between the key point information and the key point labels;
[0240] The network parameters of the feature extraction network and / or the key point detection network are adjusted according to the differences until the convergence condition is met, thereby obtaining the trained feature extraction network and / or the key point detection network.
[0241] As described above, in this embodiment, the network is optimized and trained by incorporating the first uncertainty of the target face, thereby improving the performance of the image processing network. Furthermore, the constructed loss function has a simple structure, eliminating the need for additional labels on the first uncertainty, thus enabling optimization and making network convergence easier. Moreover, since the first uncertainty is the comprehensive uncertainty of the target face, individual regression optimization for each keypoint is unnecessary during training, simplifying computation and improving network training efficiency.
[0242] Thirdly, embodiments of this disclosure provide an electronic device, including:
[0243] Processor; and
[0244] The memory stores computer instructions that can be read by the processor, and when the computer instructions are read, the processor executes the method according to any embodiment of the first aspect.
[0245] Fourthly, embodiments of this disclosure provide a storage medium for storing computer-readable instructions for causing a computer to perform the method according to any embodiment of the first aspect.
[0246] Specifically, Figure 12 A schematic diagram of the structure of an electronic device 600 suitable for implementing the method of this disclosure is shown. Figure 12 The electronic device shown can perform the corresponding functions of the processor and storage medium described above.
[0247] like Figure 12 As shown, the electronic device 600 includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a memory 602 or a program loaded into the memory 602 from a storage section 608. The memory 602 also stores various programs and data required for the operation of the electronic device 600. The processor 601 and the memory 602 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0248] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.
[0249] In particular, according to embodiments of this disclosure, the above-described method process can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for performing the above-described methods. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611.
[0250] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0251] Obviously, the above embodiments are merely examples for clear illustration and are not intended to limit the embodiments. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all embodiments here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this disclosure.
Claims
1. An image processing method, characterized in that, include: Acquire a face image to be tested, wherein the face image to be tested includes the target face; Image detection is performed on the face image to be tested to determine key point information of at least one key point of the target face and a first uncertainty of the target face; wherein, the first uncertainty is obtained based on the second uncertainty of all key points of the target face; Based on the key point information and the first uncertainty, the detection result of the target face is determined; Determining the detection result of the target face based on the key point information and the first uncertainty includes: Based on the first uncertainty of the target face, a reliability score for the target face is determined; Based on the reliability score and the pre-established correspondence between the reliability score and the face tracking model, the target face tracking model is determined from a set of multiple pre-set face tracking models. The target face is detected and tracked using the target face tracking model to obtain the detection result of the target face.
2. The method according to claim 1, characterized in that, The acquisition of the face image to be tested includes: Obtain an image to be processed, wherein the image to be processed includes at least one human face; Image detection is performed on the image to be processed to determine the facial region information of each face in the image to be processed; For any given face, the image of the face to be tested is obtained by cropping the face based on the face region information.
3. The method according to claim 1, characterized in that, The step of performing image detection on the face image to be tested, determining key point information of at least one key point of the target face and a first uncertainty of the target face, includes: Key point detection is performed on the face image to be tested, and the key point information and second uncertainty of each key point of the target face are determined based on the pre-set face key point type. The first uncertainty of the target face is determined based on the second uncertainty of each key point of the target face.
4. The method according to claim 3, characterized in that, The facial key points include at least one of the following types: Key points of facial contours, eyes, eyebrows, nose, mouth, and ears.
5. The method according to claim 1, characterized in that, Determining the detection result of the target face based on the key point information and the first uncertainty includes: Based on the first uncertainty of the target face, a reliability score for the target face is determined; In response to the reliability score satisfying the first preset condition, the key points are output on the face image to be tested based on the key point information of each key point.
6. The method according to claim 1, characterized in that, Determining the detection result of the target face based on the key point information and the first uncertainty includes: Based on the first uncertainty of the target face, a reliability score for the target face is determined; In response to the reliability score meeting the second preset condition, it is determined that the target face detection of the face image to be tested has passed.
7. The method according to any one of claims 1 to 6, characterized in that, The step of performing image detection on the face image to be tested, determining key point information of at least one key point of the target face and the first uncertainty of the target face, includes: The face image to be tested is input into a pre-trained feature extraction network to obtain the feature map output by the feature extraction network. The feature map is input into a pre-trained keypoint detection network to obtain the keypoint information of each keypoint of the target face output by the keypoint detection network and the first uncertainty.
8. The method according to claim 7, characterized in that, It also includes a training process for training the feature extraction network and the keypoint detection network, the training process including: Obtain a sample dataset, wherein each sample data in the sample dataset includes a face sample image and a key point label for each key point of the target face in the face sample image; For any sample data, the face sample image is input into the feature extraction network to be trained, and the feature map of the face sample image output by the feature extraction network is obtained. The feature map of the face sample image is input into the key point detection network to be trained to obtain the key point information of each key point of the target face, as well as the first uncertainty of the target face; Based on the key point information, key point labels, and the first uncertainty, determine the difference between the key point information and the key point labels; The network parameters of the feature extraction network and / or the key point detection network are adjusted according to the differences until the convergence condition is met, thereby obtaining the trained feature extraction network and / or the key point detection network.
9. An image processing apparatus, characterized in that, include: The acquisition module is configured to acquire a face image to be tested, wherein the face image to be tested includes a target face; An image detection module is configured to perform image detection on the face image to be tested, determine key point information of at least one key point of the target face and a first uncertainty of the target face; wherein the first uncertainty is obtained based on the second uncertainty of all key points of the target face; The result determination module is configured to determine the detection result of the target face based on the key point information and the first uncertainty; The result determination module is configured as follows: Based on the first uncertainty of the target face, a reliability score for the target face is determined; Based on the reliability score and the pre-established correspondence between the reliability score and the face tracking model, the target face tracking model is determined from a set of multiple pre-set face tracking models. The target face is detected and tracked using the target face tracking model to obtain the detection result of the target face.
10. An electronic device, characterized in that, include: processor; as well as The memory stores computer instructions that can be read by the processor, and when the computer instructions are read, the processor executes the method according to any one of claims 1 to 8.
11. A storage medium, characterized in that, Used to store computer-readable instructions for causing a computer to perform the method according to any one of claims 1 to 8.