A method and device for detecting a face image, a terminal device, and a storage medium

Abstract

The present application discloses a method and apparatus for detecting a facial image, a terminal device, and a storage medium. The detection method includes: obtaining a near-infrared image to be processed and a depth image to be processed, and performing normalization processing on the near-infrared image to be processed to obtain the near-infrared image to be detected; obtaining facial key point information of a target face based on the near-infrared image to be detected, and using the facial key point information and a preset face occlusion model to obtain a face occlusion prediction result of the near-infrared image to be detected; obtaining an average depth value set of the depth image to be processed based on the facial key point information; and determining a face detection result using the face occlusion prediction result and / or the average depth value set, wherein the face detection result indicates whether the target face is occluded or not. Through the above method, the target detection result determined based on the two dimensions can more accurately indicate whether the target facial image is occluded, thereby improving the accuracy of facial image detection.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and apparatus for detecting facial images, a terminal device, and a storage medium. Background Art

[0002] With the increasing application of image processing technology, face occlusion detection is a crucial step in face recognition technology. This detection effectively ensures the quality of the face recognition module's image, thereby improving accuracy and user experience. Currently, subspace regression methods can be used to divide different face categories into different subspaces. Occlusion is an independent subspace, so an occluded face image is the superposition of an unoccluded face and the occlusion. Therefore, the problem of recognizing occluded face images can be considered as regressing the unoccluded face image and the occlusion back to their respective subspaces. However, due to the significant differences between individual faces and the fact that a face image typically contains a wealth of features, it is difficult to construct an accurate occlusion subspace using subspace regression methods. Therefore, how to more accurately detect occluded face images has become a pressing issue. Summary of the Invention

[0003] The embodiments of the present application provide a facial image detection method and apparatus, a terminal device, and a storage medium. The target detection results determined based on two dimensions can more accurately indicate whether the target face is obscured, thereby improving the accuracy of facial image detection.

[0004] In view of this, the first aspect of the present application provides a method for detecting a face image, comprising:

[0005] Acquire the aligned near-infrared image to be processed and the depth image to be processed, and perform normalization processing on the near-infrared image to be processed to obtain the near-infrared image to be detected;

[0006] Obtaining facial key point information of a target face in the near-infrared image to be detected based on the near-infrared image to be detected;

[0007] Input the facial key point information into the preset face occlusion model to obtain the face occlusion prediction result of the target face;

[0008] According to the alignment relationship, the average depth value set of the depth image to be processed is obtained by using the facial key point information; wherein the average depth value set includes the average depth value of the left eye area, the average depth value of the right eye area, the average depth value of the nose area, and the average depth value of the mouth area;

[0009] A face detection result is determined using the face occlusion prediction result and / or the average depth value set; wherein the face detection result is whether the target face is occluded or not.

[0010] A second aspect of the present application provides a facial image detection device, comprising:

[0011] A first acquisition module is used to acquire the aligned near-infrared image to be processed and the depth image to be processed;

[0012] The first processing module is used to perform normalization processing on the near-infrared image to be processed to obtain a near-infrared image to be detected;

[0013] The second acquisition module is used to obtain facial key point information of the target face in the near-infrared image to be detected based on the near-infrared image to be detected;

[0014] The third acquisition module is used to input the facial key point information into a preset face occlusion model to obtain the face occlusion prediction result of the target face;

[0015] A second processing module is configured to obtain, based on the alignment relationship and using the facial key point information, a set of average depth values of the depth image to be processed; wherein the set of average depth values includes an average depth value of the left eye region, an average depth value of the right eye region, an average depth value of the nose region, and an average depth value of the mouth region;

[0016] The determination module is used to determine the face detection result using the face occlusion prediction result and / or the average depth value set; wherein the face detection result is whether the target face is occluded or not.

[0017] The third aspect of the present application provides a terminal device, comprising: a memory and a processor; wherein the memory is used to store computer programs; the processor is used to execute the computer programs in the memory to implement the methods described in the above aspects.

[0018] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the methods provided in the above aspects are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of a system architecture of a facial image detection system in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of an embodiment of a facial image detection method provided in an embodiment of the present application;

[0021] Figure 3 A schematic diagram of the architecture of a preset face detection model provided in an embodiment of the present application;

[0022] Figure 4 A schematic diagram of the architecture of a preset facial key point model provided in an embodiment of the present application;

[0023] Figure 5 A schematic diagram of the architecture of a preset face occlusion model provided in an embodiment of the present application;

[0024] Figure 6 A schematic diagram of the structure of a facial image detection device provided in an embodiment of the present application;

[0025] Figure 7 This is a structural diagram of an embodiment of a terminal device in the embodiments of the present application. DETAILED DESCRIPTION

[0026] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the numbers used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "corresponding to" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] The embodiments of the present application provide a facial image detection method that determines a target face detection result based on a facial occlusion prediction result and a set of average depth values. The target detection result determined by these two dimensions can more accurately indicate whether the target face is occluded, thereby improving the accuracy of facial image detection. It is understood that the facial image detection method in the embodiments of the present application can be executed by a terminal device or a server.

[0028] Figure 1This is a schematic diagram of the system architecture of a facial image detection system in an embodiment of the present application. The system can more accurately indicate whether the target face is occluded through the target face detection results determined by two dimensions, thereby improving the accuracy of facial image detection. In one embodiment, the detection system may include a terminal device deployed in a facial image detection method, which obtains a near-infrared image to be processed and a depth image to be processed through the terminal device, and processes the near-infrared image to be processed to obtain a near-infrared image to be detected; obtains facial key point information of the target face based on the near-infrared image to be detected, and uses the facial key point information and a preset face occlusion model to obtain a face occlusion prediction result of the near-infrared image to be detected; obtains an average depth value set of the depth image to be processed based on the aforementioned facial key point information; and determines a target face detection result of the near-infrared image to be detected based on the face occlusion prediction result and / or the average depth value set.

[0029] In one embodiment, the terminal device includes an acquisition module for synchronously or sequentially acquiring near-infrared images and depth images to be processed. It should be noted that the acquisition module can be a component of the terminal device or an external device independent of the terminal device; the acquisition module can also communicate with the terminal device via wired or wireless means; and the acquisition module can also include any one or more combinations of other cameras, such as infrared cameras, depth cameras, and color cameras, without limitation.

[0030] In another embodiment, the detection system also includes a server, and the facial image detection method is deployed on the server. When the facial image detection method is deployed on the server, the terminal device is only used to collect the near-infrared image to be processed and the depth image to be processed, and the server is used to obtain the near-infrared image to be processed and the depth image to be processed in real time through the terminal device for processing. The processing method is similar to the above and will not be repeated here.

[0031] It should be noted that the aforementioned terminal device and the server communicate via a wireless network, a wired network or a removable storage medium; Figure 1 The server in the example can be a single server or a server cluster or cloud computing center composed of multiple servers, etc., which is not limited here. The terminal device can be Figure 1The tablet computer, laptop computer, door lock, and mobile phone or other terminal device shown in the figure. The above-mentioned wireless network uses standard communication technology and / or protocol. The wireless network is usually the Internet, but it can also be any network, including but not limited to Bluetooth, local area network (LAN), metropolitan area network (MAN), wide area network (WAN), mobile, private network or any combination of virtual private networks). In some embodiments, customized or dedicated data communication technology can be used to replace or supplement the above-mentioned data communication technology. The removable storage medium can be a universal serial bus (USB) flash drive, a mobile hard disk or other removable storage medium, etc.

[0032] Although Figure 1 Only four terminal devices and one server are shown, but it should be understood that Figure 1 The examples are only for understanding this solution. The specific number of terminal devices and servers should be flexibly determined based on actual conditions.

[0033] Figure 2 A schematic diagram of an embodiment of a facial image detection method provided in an embodiment of the present application, the method comprising:

[0034] Step 201 : Acquire an aligned near-infrared image to be processed and a depth image to be processed, and perform normalization processing on the near-infrared image to be processed to obtain a near-infrared image to be detected.

[0035] In this embodiment, a near-infrared image to be processed and a depth image to be processed are obtained. The near-infrared image to be processed is 10 bits (bit), that is, the near-infrared image to be processed is image data of 0-1024, and the standard image for image processing is 8 bits. Therefore, it is necessary to normalize the near-infrared image to be processed to obtain the near-infrared image to be detected. At this time, the near-infrared image to be detected is an 8-bit image, and the near-infrared image to be detected specifically is image data of 0-256.

[0036] It should be noted that step 201 may also include an alignment operation, that is, making a one-to-one correspondence between each pixel of the near-infrared image to be processed and the depth image to be processed, so as to obtain a mapping relationship between each pixel.

[0037] Step 202: Obtain face frame information of the target face in the near-infrared image to be detected using the near-infrared image to be detected and a preset face detection model.

[0038] In this embodiment, the near-infrared image to be detected obtained in step 201 is used as the input of a preset face detection model, and the preset face detection model outputs the face frame information of the target face in the near-infrared image to be detected. The face frame information includes facial feature information such as the left eye information, right eye information, nose information, and mouth information of the target face.

[0039] Specifically, the preset face detection model includes the first convolutional layer, the first batch of normalization layers, the first activation layer, and the first output layer. The first output layer includes the first classification layer and the first regression layer. The target loss function of the first classification layer is the cross entropy loss function, and the target loss function of the first regression layer is the Smooth L1 loss function. For easier understanding, please refer to Figure 3 , Figure 3 An architectural diagram of a preset face detection model provided for an embodiment of the present application, wherein the near-infrared image to be detected is input into the first convolution layer, and then the first convolution layer performs convolution processing on the near-infrared image to be detected, and the near-infrared image to be detected after the convolution processing is input into the first normalization layer. Based on this, the first normalization layer performs batch normalization processing on the near-infrared image to be detected after the convolution processing, and then inputs the near-infrared image to be detected after the batch normalization processing into the first activation layer. Similar to the above, the first activation layer performs activation processing on the near-infrared image to be detected after the batch normalization processing, and then inputs the near-infrared image to be detected after the activation processing into the first classification layer and the first regression layer, and finally outputs the face frame information based on the first classification layer and the first regression layer. It should be understood that Figure 3 The example is only used to understand this solution. In actual applications, the preset face detection model can also include multiple first convolutional layers, multiple first normalization layers, and multiple first activation layers. Figure 3 The examples should not be understood as limiting the present solution.

[0040] It should be noted that the preset face detection model is obtained by pre-training. In one embodiment, the training method is: obtain a near-infrared image sample set and a face frame information set, which face frame information set includes the face frame information of each near-infrared image sample; use the near-infrared image sample set as the input of the face detection model to be trained, and output the face frame prediction information of each near-infrared image sample through the face detection model to be trained; train the face detection model to be trained based on the face frame prediction information of each near-infrared image sample and the face frame information of each near-infrared image sample; update the model parameters of the face detection model to be trained according to the target loss function to obtain the optimal preset face detection model.

[0041] Specifically, the face frame information of each near-infrared image sample is used as the target for iterative training, that is, the loss value of the target loss function is determined according to the difference between the face frame information of each near-infrared image sample and the face frame prediction information of each near-infrared image sample passed by the face detection model to be trained, and it is judged whether the loss value of the target loss function reaches the convergence condition. If the convergence condition is not reached, the loss value of the target loss function is used to update the model parameters of the face detection model to be trained; until the target loss function reaches the convergence condition, the optimal preset face detection model is obtained. The convergence condition of the aforementioned target loss function may be that the value of the target loss function is less than or equal to a first preset threshold. As an example, the value of the first preset threshold may be 0.005, 0.01, 0.02 or other values close to 0; it may also be that the difference between two adjacent values of the target loss function is less than or equal to a second preset threshold. The value of the second threshold may be the same as or different from the value of the first threshold. As an example, the value of the second preset threshold may be 0.005, 0.01, 0.02 or other values close to 0, etc. The server may also adopt other convergence conditions, which are not limited here.

[0042] Furthermore, the following formula (1) is shown as the Smooth L1 loss function:

[0043]

[0044] Among them, Smooth L1 refers to the loss value of the Smooth L1 loss function, and x refers to the difference between the face box information of each near-infrared image sample and the face box prediction information of each near-infrared image sample after the face detection model to be trained.

[0045] Since the first output layer in this scheme includes the first classification layer and the first regression layer, and the target loss function of the first classification layer and the target loss function of the first regression layer play a supervisory role, each layer of the preset face detection model finally obtained has its own parameters. Therefore, it is necessary to update the model parameters of the first classification layer in the face detection model to be trained according to the target loss function of the first classification layer, and update the model parameters of the first regression layer in the face detection model to be trained according to the target loss function of the first regression layer, until the target loss function of the first classification layer and the target loss function of the first regression layer reach the convergence condition, so as to obtain the optimal first classification layer and second regression layer, and thus obtain the optimal preset face detection model.

[0046] It should be noted that if the face detection model does not detect a face, the next step is stopped and the process returns to step 201 to reacquire the image until a face is detected. There is no limitation here.

[0047] Step 203: Map the face frame information to the near-infrared image to be detected to obtain the face ROI region of the near-infrared image to be detected.

[0048] In this embodiment, the face frame information obtained in step 202 is mapped to the near-infrared image to be detected to obtain a face region of interest (ROI). Specifically, the region to be processed is cropped from the near-infrared image to be detected based on the face frame information, and this region is the face ROI region.

[0049] Step 204: Acquire facial key point information of the target face based on the facial ROI area and the preset facial key point model.

[0050] In this embodiment, the facial ROI area of the near-infrared image to be detected obtained in step 203 is used as the input of the preset facial key point model. The preset facial key point model performs key point detection on the facial ROI area to output the facial key point information of the near-infrared image to be detected, and the facial key point information includes the key point information of the left eye, the key point information of the right eye, the key point information of the nose, and the key point information of the mouth.

[0051] Specifically, the preset facial key point model includes the second convolution layer, the second batch normalization layer, the second activation layer and the second output layer. The second output layer is the second regression layer, and the target loss function of the second regression layer is the Smooth L1 loss function. For ease of understanding, please refer to Figure 4 , Figure 4 An architectural diagram of a preset facial key point model provided for an embodiment of the present application, wherein the facial ROI area of the near-infrared image to be detected is input into the second convolutional layer, and then the second convolutional layer performs convolution processing on the facial ROI area of the near-infrared image to be detected, and the facial ROI area of the near-infrared image to be detected after the convolution processing is input into the second batch normalization layer. Based on this, the second batch normalization layer performs batch normalization processing on the facial ROI area of the near-infrared image to be detected after the convolution processing, and then inputs the facial ROI area of the near-infrared image to be detected after the batch normalization processing into the second activation layer. Similar to the above, the second activation layer performs activation processing on the facial ROI area of the near-infrared image to be detected after the batch normalization processing, and then inputs the facial ROI area of the near-infrared image to be detected after the activation processing into the second regression layer, and finally outputs the facial key point information of the near-infrared image to be detected based on the second regression layer. It should be understood that Figure 4 The example is only used to understand this solution. In actual applications, the preset face key point model can also include multiple second convolution layers, multiple second batch normalization layers, and multiple second activation layers. Figure 4 The examples should not be understood as limiting the present solution.

[0052] Furthermore, the preset facial key point model is obtained by pre-training, and the preferred training method is: obtaining a facial ROI region sample set and a facial key point information sample set, wherein the facial key point information sample set includes facial key point information of each facial ROI region sample; using the facial ROI region sample set as the input of the facial key point model to be trained, and outputting key point prediction information of each facial ROI region sample through the facial key point model to be trained; using the key point prediction information of each facial ROI region sample and the facial key point information of each facial ROI region sample to train the facial key point model to be trained; and updating the model parameters of the facial key point model to be trained according to the target loss function to obtain the optimal preset facial key point model. It should be understood that the method of updating the model parameters of the facial key point model to be trained is similar to the method of updating the model parameters of the face detection model to be trained in the aforementioned embodiment, and therefore will not be repeated here.

[0053] Furthermore, the Smooth L1 loss function is similar to the aforementioned formula (1), but x refers to the difference between the key point prediction information of each face ROI region sample output by the face key point detection model to be trained and the face key point information of each face ROI region sample.

[0054] Step 205: Integrate the facial key point information to obtain a target near-infrared image.

[0055] In this embodiment, the facial key point information of the target face in the near-infrared image to be detected is processed to obtain the target near-infrared image. Specifically, based on the facial key point information of the near-infrared image to be detected, the face ROI area is subjected to a cutout process to obtain a set of to-be-processed areas of the face ROI area, wherein the set of to-be-processed areas includes a left eye area to be processed, a right eye area to be processed, a nose area to be processed, and a mouth area to be processed. Then, the sizes of the left eye area to be processed, the right eye area to be processed, the nose area to be processed, and the mouth area to be processed are adjusted to obtain a set of target areas, wherein the set of target areas includes a left eye area, a right eye area, a nose area, and a mouth area. Since it is subsequently necessary to perform integration processing in the channel direction, and the prerequisite for performing integration processing in the channel direction is that the sizes of the areas in the width and height directions must be exactly the same, the sizes of the left eye area, the right eye area, the nose area, and the mouth area in the target area set are the same. Based on this, the left eye area, the right eye area, the nose area, and the mouth area of the same size in the target area set are integrated in the channel direction to obtain the target near-infrared image.

[0056] Specifically, as can be seen from step 201, the format of the normalized near-infrared image is U8C1, i.e., one channel (unsigned char). Since the target region set includes four regions: the left eye region, the right eye region, the nose region, and the mouth region, the format of the target near-infrared image point after integration is U8C4.

[0057] Step 206: Input the target near-infrared image into a preset face occlusion model to obtain a face occlusion prediction result of the target face in the near-infrared image to be detected.

[0058] In this embodiment, the target near-infrared image obtained in step 205 is used as the input of the preset face occlusion model, and the result output by the preset face occlusion model is a face occlusion prediction result. Specifically, the preset face occlusion model specifically outputs a face occlusion probability value, which is the probability value of the left eye, right eye, nose and mouth in the target face being occluded, and the preset face occlusion model is a two-classification model. Therefore, the face occlusion probability value output by the preset face occlusion model is compared with the preset occlusion threshold. In this embodiment, the preset occlusion threshold is 0.5 as an example. If the face occlusion probability value is greater than or equal to 0.5, it means that the face is not occluded, and if the face occlusion probability value is less than 0.5, it means that the face is occluded. It should be understood that the specific value of the preset occlusion threshold needs to be flexibly determined according to actual conditions and should not be understood as a limitation of this application.

[0059] Specifically, the preset face occlusion model includes the third convolutional layer, the third batch normalization layer, the third activation layer, and the third output layer. The third output layer is the second classification layer, and the target loss function of the second classification layer is the cross entropy loss function. For easier understanding, please refer to Figure 5 , Figure 5 An architectural diagram of a preset face occlusion model provided for an embodiment of the present application, wherein the target near-infrared image is input to the third convolution layer, and then the third convolution layer performs convolution processing on the target near-infrared image, and the target near-infrared image after convolution processing is input to the third batch normalization layer. Based on this, the third batch normalization layer performs batch normalization processing on the target near-infrared image after convolution processing, and then inputs the target near-infrared image after batch normalization processing to the third activation layer. Similar to the above, the third activation layer performs activation processing on the target near-infrared image after batch normalization processing, and then inputs the target near-infrared image after activation processing to the second classification layer, and finally outputs the face occlusion probability value based on the second classification layer, and determines the face occlusion prediction result based on the face occlusion probability value. It should be understood that Figure 5 The example is only used to understand this solution. In actual applications, the preset face occlusion model can also include multiple third convolutional layers, multiple third batch normalization layers, and multiple third activation layers. Figure 5 The examples should not be understood as limiting the present solution.

[0060] In one embodiment, the preset face occlusion model is obtained by pre-training, and the preferred training method is: obtaining a set of near-infrared image samples and a set of face occlusion result samples, the face occlusion result sample set including the face occlusion result of each near-infrared image sample; using the set of near-infrared image samples as the input of the face occlusion model to be trained, and outputting the face occlusion prediction result of each near-infrared image sample through the face occlusion model to be trained; using the face occlusion prediction result of each near-infrared image sample and the face occlusion result of each near-infrared image sample to train the face occlusion model to be trained; updating the model parameters of the face occlusion model to be trained according to the target loss function to obtain the optimal preset face occlusion model. The method of updating the model parameters of the face occlusion model to be trained is similar to the method of updating the model parameters of the face detection model to be trained in the aforementioned embodiment, so it will not be repeated here. Furthermore, the loss value of the target loss function is determined based on the difference between the face occlusion prediction result of each near-infrared image sample obtained by the face occlusion model to be trained and the face occlusion result of each near-infrared image sample.

[0061] Step 207: Based on the alignment relationship, the facial key point information is used to obtain a set of average depth values of the depth image to be processed.

[0062] In this embodiment, a depth image to be processed is obtained, and the depth image to be processed is processed according to the facial key point information of the target face in the near-infrared image to be detected to obtain an average depth value set, wherein the average depth value set includes the average depth value of the left eye area, the average depth value of the right eye area, the average depth value of the nose area, and the average depth value of the mouth area.

[0063] Specifically, according to the alignment relationship between the near-infrared image and the depth image, the facial key point information of the near-infrared image to be detected is mapped to the depth image to be processed, and the ROI area set to be processed of the depth image to be processed is obtained. At this time, the ROI area set to be processed includes the ROI area to be processed of the left eye area, the ROI area to be processed of the right eye area, the ROI area to be processed of the nose area, and the ROI area to be processed of the mouth area. Based on this, the ROI area to be processed of the left eye area, the ROI area to be processed of the right eye area, the ROI area to be processed of the nose area, and the ROI area to be processed of the mouth area are subjected to noise reduction processing (i.e., invalid points in each ROI area to be processed are removed) to obtain a target ROI area set. It should be understood that the ROI area set includes the ROI area of the left eye area, the ROI area of the right eye area, the ROI area of the nose area, and the ROI area of the mouth area. Finally, the depths of the ROI area of the left eye area, the ROI area of the right eye area, the ROI area of the nose area, and the ROI area of the mouth area are averaged to obtain an average depth value set.

[0064] Step 208: Determine a face detection result using the face occlusion prediction result and / or the average depth value set.

[0065] In this embodiment, based on the average depth value set obtained in step 207 and / or the face occlusion prediction result obtained in step 206, the face detection result of the target face in the input image (i.e., the near-infrared image to be processed and the depth image to be processed) is determined. At this time, the face detection result is that the target face is occluded, or the target face is not occluded.

[0066] Specifically, a maximum average depth value and a minimum average depth value are selected from the average depth value set, the difference between the maximum depth value and the minimum depth value is calculated, and the difference between the maximum depth value and the minimum depth value and a preset threshold value is determined. It should be understood that the aforementioned preset threshold value is determined through experiments and / or based on statistics of a large amount of data, and the value of the preset threshold value is not specifically limited herein.

[0067] Based on this, if the face occlusion prediction result indicates that the target face is not occluded, and the difference between the maximum depth value and the minimum depth value is less than the preset threshold, then it is determined that the target face is not occluded. Secondly, if the face occlusion prediction result indicates that the target face is occluded and the difference between the maximum depth value and the minimum depth value is greater than or equal to the preset threshold, then it is determined that the target face is occluded. It should be understood that only if the face occlusion prediction result indicates that the target face is occluded or the difference between the maximum depth value and the minimum depth value is greater than or equal to the preset threshold, it can also be determined that the target face is occluded, but this application preferably uses two dimensions for determination, which is not limited here.

[0068] Through the above method, a face occlusion prediction result is obtained based on a target near-infrared image of key areas including the left eye area, right eye area, nose area and mouth area of the target face and a preset face occlusion model. Secondly, based on the facial key point information of the target face and the depth image to be processed, a set of average depth values of the aforementioned key areas is obtained, and a target detection result of the target face is determined based on the face occlusion prediction result and the average depth value set. Finally, the target detection result determined based on the two dimensions can more accurately indicate whether the target face is occluded, thereby improving the accuracy of facial image detection.

[0069] The above describes the face image detection in this solution in detail. The following describes the face image detection device provided by this solution. Figure 6 A schematic diagram of the structure of a facial image detection device provided in an embodiment of the present application is provided. The device 600 includes:

[0070] A first acquisition module 601 is used to acquire the aligned near-infrared image to be processed and the depth image to be processed;

[0071] The first processing module 602 is configured to perform normalization processing on the near-infrared image to be processed to obtain a near-infrared image to be detected;

[0072] The second acquisition module 603 is configured to obtain facial key point information of a target face in the near-infrared image to be detected based on the near-infrared image to be detected;

[0073] The third acquisition module 604 is used to input the facial key point information into a preset face occlusion model to obtain a face occlusion prediction result of the target face;

[0074] The second processing module 605 is configured to obtain an average depth value set of the depth image to be processed using the facial key point information according to the alignment relationship; wherein the average depth value set includes an average depth value of the left eye region, an average depth value of the right eye region, an average depth value of the nose region, and an average depth value of the mouth region;

[0075] The determination module 606 is configured to determine a face detection result using the face occlusion prediction result and / or the average depth value set; wherein the face detection result is whether the target face is occluded or not.

[0076] Optionally, in the above Figure 6 On the basis of the corresponding embodiment, in another embodiment of the facial image detection device 600 provided in the embodiment of the present application, the facial key point information includes the key point information of the left eye, the key point information of the right eye, the key point information of the nose and the key point information of the mouth; the third acquisition module 604 is specifically used to process the facial key point information to obtain a target near-infrared image, wherein the target near-infrared image is obtained by integrating the left eye area, the right eye area, the nose area and the mouth area in the channel direction, wherein the left eye area, the right eye area, the nose area and the mouth area have the same size; based on the target near-infrared image, the face occlusion prediction result of the near-infrared image to be detected is obtained by a preset face occlusion model.

[0077] Optionally, in the above Figure 6 On the basis of the corresponding embodiment, in another embodiment of the facial image detection device 600 provided in the embodiment of the present application, the second acquisition module 603 is specifically used to obtain the facial frame information of the target face in the near-infrared image to be detected through the near-infrared image to be detected and the preset face detection model; map the facial frame information to the near-infrared image to be detected to obtain the facial ROI area of the near-infrared image to be detected; and obtain the facial key point information of the target face according to the facial ROI area and the preset facial key point model.

[0078] Optionally, in the above Figure 6On the basis of the corresponding embodiment, in another embodiment of the facial image detection device 600 provided in the embodiment of the present application, the third acquisition module 604 is specifically used to perform cutout processing on the facial ROI area based on the facial key point information to obtain a set of areas to be processed in the facial ROI area, wherein the set of areas to be processed includes a left eye area to be processed, a right eye area to be processed, a nose area to be processed, and a mouth area to be processed; adjust the size of the area to be processed to obtain a target area set, wherein the target area set includes a left eye area, a right eye area, a nose area, and a mouth area; integrate the target area set in the channel direction to obtain a target near-infrared image.

[0079] Optionally, in the above Figure 6 On the basis of the corresponding embodiment, in another embodiment of the face image detection device 600 provided by the embodiment of the present application, the average depth value set includes the average depth value of the left eye area, the average depth value of the right eye area, the average depth value of the nose area, and the average depth value of the mouth area; the second processing module 605 is specifically used to map the facial key point information to the depth image to be processed, and obtain the ROI area set to be processed of the depth image to be processed, wherein the ROI area set to be processed includes the ROI area to be processed of the left eye area, the ROI area to be processed of the right eye area, the ROI area to be processed of the nose area, and the ROI area to be processed of the mouth area. ROI area to be processed; performing noise reduction processing on the ROI area to be processed of the left eye area, the ROI area to be processed of the right eye area, the ROI area to be processed of the nose area, and the ROI area to be processed of the mouth area to obtain a target ROI area set, wherein the ROI area set includes the ROI area of the left eye area, the ROI area of the right eye area, the ROI area of the nose area, and the ROI area of the mouth area; performing mean calculation on the ROI area of the left eye area, the ROI area of the right eye area, the ROI area of the nose area, and the ROI area of the mouth area to obtain an average depth value set of the depth image to be processed.

[0080] Optionally, in the above Figure 6 On the basis of the corresponding embodiment, in another embodiment of the face image detection device 600 provided in the embodiment of the present application, the preset face occlusion model includes a third convolutional layer, a third batch normalization layer, a third activation layer and a third output layer. The third output layer is the second classification layer, and the target loss function of the second classification layer is the cross entropy loss function.

[0081] Optionally, the determination module 606 is specifically configured to select a maximum average depth value and a minimum average depth value from the set of average depth values; if the face occlusion prediction result indicates that the target face is not occluded and the difference between the maximum depth value and the minimum depth value is less than a preset threshold, then the target face is determined to be not occluded. If the face occlusion prediction result indicates that the target face is occluded and / or the difference between the maximum depth value and the minimum depth value is greater than or equal to a preset threshold, then the target face is determined to be occluded.

[0082] Figure 7 This is a structural diagram of an embodiment of a terminal device in an embodiment of the present application, wherein the terminal device 700 includes a processor 710 and a memory 720 coupled to the processor 710, wherein the processor 710 may be a central processing unit (CPU), a network processor (NP) or a combination of a CPU and an NP. The processor may also be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. Computer-readable instructions are stored in the memory 720, and the computer-readable instructions execute any one of the possible implementation methods described above. After executing the computer-readable instructions, the processor 710 may perform corresponding operations in accordance with the instructions of the computer-readable instructions. In addition, after executing the computer-readable instructions in the memory 720, the processor 710 may perform all operations that the server can perform in accordance with the instructions of the computer-readable instructions, such as the server performing a communication with the server. Figure 2 The operations performed in the corresponding embodiments.

[0083] It should be noted that the terminal device may also include a transceiver 730, which is a port for outputting data and, in some cases, a port for inputting data. The processor 710 may call the transceiver 730 by executing code to obtain a set of lines to be matched and a set of target orders.

[0084] The present application also provides a computer-readable storage medium in which a computer program is stored. When the computer program is run on a computer, the computer executes the above-mentioned Figure 2 The illustrated embodiment describes the steps performed by the server in the method.

[0085] The present application also provides a computer program product including a program, which, when executed on a computer, enables the computer to execute the aforementioned Figure 2 The illustrated embodiment describes the steps performed by the server in the method.

[0086] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0087] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as at least two units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0088] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across at least two network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0089] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0090] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0091] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for detecting a face image, characterized in that: include: Acquire the aligned near-infrared image to be processed and the depth image to be processed, and perform normalization processing on the near-infrared image to be processed to obtain the near-infrared image to be detected; Obtaining face frame information of a target face in the near-infrared image to be detected by using the near-infrared image to be detected and a preset face detection model; mapping the face frame information to the near-infrared image to be detected to obtain a face ROI region of the near-infrared image to be detected; and obtaining face key point information of the target face based on the face ROI region and a preset face key point model; Inputting the facial key point information into a preset face occlusion model to obtain a face occlusion prediction result of the target face; According to the alignment relationship, the facial key point information is mapped to the depth image to be processed to obtain a set of ROI regions to be processed of the depth image to be processed; Performing noise reduction processing on the set of ROI regions to be processed to obtain a set of target ROI regions; Performing mean calculation on the depths of the target ROI region set to obtain a set of average depth values of the depth image to be processed; wherein the set of average depth values includes an average depth value of the left eye region, an average depth value of the right eye region, an average depth value of the nose region, and an average depth value of the mouth region; Select a maximum average depth value and a minimum average depth value from the average depth value set; if the face occlusion prediction result indicates that the target face is not occluded, and the difference between the maximum average depth value and the minimum average depth value is less than a preset threshold, determine that the target face is not occluded; if the face occlusion prediction result indicates that the target face is occluded, and / or the difference between the maximum average depth value and the minimum average depth value is greater than or equal to the preset threshold, determine that the target face is occluded.

2. The detection method according to claim 1, wherein Inputting the facial key point information into a preset face occlusion model to obtain a face occlusion prediction result of the target face includes: Integrating and processing the facial key point information to obtain a target near-infrared image; The target near-infrared image is input into a preset face occlusion model to obtain a face occlusion prediction result of the target face in the near-infrared image to be detected.

3. The detection method according to claim 2, characterized in that The preset face detection model includes a first convolutional layer, a first batch of normalization layers, a first activation layer and a first output layer, the first output layer includes a first classification layer and a first regression layer, the target loss function of the first classification layer is a cross entropy loss function, and the target loss function of the first regression layer is a Smooth L1 loss function; The preset facial key point model includes a second convolutional layer, a second batch normalization layer, a second activation layer and a second output layer, the second output layer is a second regression layer, and the target loss function of the second regression layer is the Smooth L1 loss function; The preset face occlusion model includes a third convolutional layer, a third batch normalization layer, a third activation layer and a third output layer. The third output layer is a second classification layer, and the target loss function of the second classification layer is a cross entropy loss function.

4. The detection method according to claim 3, characterized in that The integrating and processing the facial key point information to obtain a target near-infrared image includes: Based on the facial key point information, the facial ROI region is subjected to a cutout process to obtain a set of to-be-processed regions of the facial ROI region; Adjusting the size of the set of regions to be processed to obtain a set of target regions; The target area set is integrated in the channel direction to obtain the target near-infrared image.

5. A facial image detection device, characterized in that: The facial image detection device comprises: A first acquisition module is used to acquire the aligned near-infrared image to be processed and the depth image to be processed; A first processing module is used to perform normalization processing on the near-infrared image to be processed to obtain a near-infrared image to be detected; A second acquisition module is configured to obtain face frame information of a target face in the near-infrared image to be detected by using the near-infrared image to be detected and a preset face detection model; map the face frame information to the near-infrared image to be detected to obtain a face ROI region of the near-infrared image to be detected; and obtain face key point information of the target face based on the face ROI region and a preset face key point model; A third acquisition module is used to input the facial key point information into a preset face occlusion model to obtain a face occlusion prediction result of the target face; A second processing module is configured to map the facial key point information onto the depth image to be processed based on the alignment relationship to obtain a set of ROI regions to be processed of the depth image to be processed; perform noise reduction processing on the set of ROI regions to be processed to obtain a set of target ROI regions; perform mean calculation on the depth of the set of target ROI regions to obtain a set of average depth values of the depth image to be processed; wherein the set of average depth values includes an average depth value of a left eye region, an average depth value of a right eye region, an average depth value of a nose region, and an average depth value of a mouth region; A determination module is used to select a maximum average depth value and a minimum average depth value from the average depth value set; if the face occlusion prediction result indicates that the target face is not occluded, and the difference between the maximum average depth value and the minimum average depth value is less than a preset threshold, then it is determined that the target face is not occluded; if the face occlusion prediction result indicates that the target face is occluded, and / or the difference between the maximum average depth value and the minimum average depth value is greater than or equal to the preset threshold, then it is determined that the target face is occluded.

6. A terminal device, characterized in that: Comprising a memory and a processor, wherein: The memory is used to store programs; The processor is configured to execute the program in the memory to implement the method according to any one of claims 1 to 4.

7. The terminal device according to claim 6, wherein: The terminal device also includes an acquisition module for acquiring near-infrared images to be processed and depth images to be processed and transmitting them to the processor for processing.

8. A computer storable medium comprising instructions, which, when executed on a computer, causes the computer to perform the method according to any one of claims 1 to 4.

Images