Face Detection Method and Device Based on Foreground and Background Separation

Abstract

This invention provides a face detection method based on foreground and background separation, comprising: separating the foreground and background of a dataset of nighttime face images to obtain a first weight matrix representing the difference in foreground and background separation at various locations in the image; pre-training a pre-trained model using a deep neural network on data from a general face dataset; fusing the nighttime face dataset images with the first weight matrix and then feeding it into the deep neural network for transfer learning based on the pre-trained model to obtain a nighttime face detection model; separating the foreground and background of the nighttime image to be detected to obtain a second weight matrix, and fusing it with the nighttime image to be detected to obtain a nighttime enhanced image; scaling the nighttime enhanced image at different scales and using the nighttime face detection model to detect the nighttime enhanced images at different scales; aggregating the detected results at different scales and selecting the final detection result. This invention can improve the accuracy of nighttime face detection.

Description

Technical Field

[0001] This invention relates to the field of face detection technology, and in particular to a face detection method and apparatus based on foreground and background separation. Background Technology

[0002] Object detection is a crucial task in computer vision. In recent years, with the development of deep learning, object detection techniques based on convolutional neural networks have become a research hotspot. Current deep learning object detection algorithms can be mainly divided into two categories: one-stage object detection algorithms, which do not use candidate regions and directly detect the location and category information of objects in an image through the backbone network; and two-stage object detection algorithms, which first use corresponding candidate region algorithms, such as traditional algorithms or neural networks, to generate suggested target candidate regions from the input image, and then feed all candidate regions into a classifier for classification.

[0003] Face detection, as a type of object detection task, can be accomplished using general object detection algorithms. In recent years, however, many object detection algorithms specifically designed for face detection have also emerged, such as pyramidbox and DSFD.

[0004] When the human eye observes an image, it attaches different levels of importance to different areas, focusing its attention on a specific part of the image, such as a building or a person. This mechanism allows humans to quickly process what they see. The purpose of foreground / background separation is to simulate this human visual attention mechanism, calculating the salience of different areas in the image, thereby separating the foreground with high foreground / background separation from the background with low foreground / background separation.

[0005] The result of image foreground and background separation is a weight matrix with the same dimensions as the image. The value at each position represents the weight of the corresponding image pixel, with the foreground having a higher weight and the background having a lower weight.

[0006] In face detection tasks, variations in the background environment of an image can affect the detection results. Nighttime environments pose a significant challenge to face detection. Images captured in darkness lack sufficient brightness, making it difficult to distinguish between foreground and background, which hinders the use of convolutional neural networks (CNNs) for facial feature extraction and learning. Current object detection algorithms and algorithms specifically designed for face detection do not specifically consider nighttime environments. Due to the difficulty in distinguishing between foreground and background, CNNs struggle to extract and learn facial features, thus failing to achieve high detection accuracy. Summary of the Invention

[0007] The face detection method and apparatus based on foreground and background separation provided by this invention can achieve high detection accuracy for face detection at night.

[0008] In a first aspect, the present invention provides a face detection method based on foreground and background separation, the method comprising:

[0009] Foreground and background separation is performed on the dataset of nighttime face images to obtain the first weight matrix representing the difference in foreground and background separation at each location in the image;

[0010] A pre-trained model is obtained by using a deep neural network to pre-train data from a general face dataset.

[0011] The dataset of nighttime face images is fused with the first weight matrix and then fed into a deep neural network. Transfer learning is performed based on the pre-trained model to obtain a nighttime face detection model.

[0012] The nighttime image to be detected is separated into foreground and background to obtain a second weight matrix, which is then fused with the nighttime image to be detected to obtain a nighttime enhanced image. The nighttime enhanced image is scaled at different scales, and the nighttime face detection model is used to detect the nighttime enhanced images at different scales.

[0013] The detected results at different scales are aggregated, and the final detection results are selected.

[0014] Optionally, the foreground and background separation of the nighttime face dataset images includes: first performing image enhancement on the nighttime face dataset images, and then performing foreground and background separation.

[0015] Optionally, fusing the dataset image of the nighttime face with the first weight matrix includes multiplying the RGB three channels of the dataset image of the nighttime face with the corresponding pixel positions of the first weight matrix.

[0016] Optionally, the foreground and background separation of the nighttime image to be detected includes: first performing image enhancement on the nighttime image to be detected, and then performing foreground and background separation.

[0017] Optionally, the fusion with the night image to be detected includes: multiplying the RGB three channels of the night image to be detected with the corresponding pixel positions of the second weight matrix.

[0018] Optionally, the aggregation of detected results at different scales includes:

[0019] The detected face bounding box coordinates are normalized to the original scale according to the scaling ratio, and then the non-maximum suppression method is used to filter the bounding box.

[0020] Secondly, the present invention provides a face detection device based on foreground and background separation, the device comprising:

[0021] The foreground and background separation unit is used to separate the foreground and background of the dataset images of nighttime faces, and obtain the first weight matrix representing the difference between the foreground and background separation at each position in the image;

[0022] The pre-training unit is used to pre-train data in a general face dataset using a deep neural network to obtain a pre-trained model.

[0023] The detection model building unit is used to fuse the dataset images of the nighttime faces with the first weight matrix, and then feed them into a deep neural network to perform transfer learning based on the pre-trained model to obtain a nighttime face detection model.

[0024] The detection unit is used to separate the foreground and background of the night image to be detected, obtain a second weight matrix, and fuse it with the night image to be detected to obtain a night-enhanced image. The night-enhanced image is scaled at different scales, and the night face detection model is used to detect the night-enhanced images at different scales.

[0025] The aggregation unit is used to aggregate the detected results at different scales and filter out the final detection results.

[0026] Optionally, the foreground and background separation unit is used to first perform image enhancement on the dataset of nighttime faces, and then perform foreground and background separation;

[0027] The detection unit is used to first enhance the nighttime image to be detected, and then perform foreground and background separation.

[0028] Optionally, the detection model building unit is used to multiply the RGB three channels of the nighttime face dataset image with the corresponding pixel positions of the first weight matrix;

[0029] The detection unit is used to multiply the RGB three channels of the night image to be detected with the corresponding pixel positions of the second weight matrix.

[0030] Optionally, the aggregation unit is used to normalize the detected face bounding box coordinates to the original scale according to the scaling ratio, and then use the non-maximum suppression method to filter the bounding box.

[0031] The face detection method and apparatus based on foreground and background separation provided in this invention separates the foreground and background and assigns different weights to different regions of the image, reducing the problem of difficulty in distinguishing the foreground and background in nighttime environments, thereby improving the accuracy of face recognition. As a result, higher detection accuracy and better detection results can be obtained when detecting faces at night. Attached Figure Description

[0032] Figure 1A flowchart of a face detection method based on foreground and background separation provided in an embodiment of the present invention;

[0033] Figure 2 This is a schematic diagram illustrating the fusion of the image and the weight matrix obtained from foreground and background separation, as provided in an embodiment of the present invention.

[0034] Figure 3 This is a schematic diagram of the structure of a face detection device based on foreground and background separation provided in an embodiment of the present invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] To address the problem that current face detection algorithms do not consider insufficient brightness in nighttime images, making it difficult to distinguish between foreground and background, and that convolutional neural networks struggle to extract and learn facial features, thus failing to achieve high detection accuracy in nighttime environments, this invention proposes a nighttime face detection method based on foreground and background separation, supported by image enhancement algorithms and image foreground / background separation techniques. By fusing the enhanced image result with its weight matrix and then feeding it into the neural network for training and testing, a better detection effect can be obtained.

[0037] This invention provides a face detection method based on foreground and background separation, such as... Figure 1 As shown, the method includes:

[0038] S11. Perform foreground and background separation on the dataset images of nighttime faces to obtain the first weight matrix representing the difference between foreground and background separation at each location in the image.

[0039] The LC saliency detection algorithm calculates the global contrast of a pixel across the entire image, which is the sum of the color distances between that pixel and all other pixels in the image. This weight is used to distinguish between foreground and background.

[0040] The process described above for obtaining the first weight matrix allows for the use of image enhancement algorithms to improve the brightness of nighttime images before processing them. Specifically, the LIME (Local Interpretable Model-Agnostic Explanations) algorithm, based on Retinex theory, can be selected.

[0041] The specific process of LIME algorithm to enhance nighttime images is as follows: input a nighttime image, select the maximum value in each pixel channel to initialize the image illumination map, then refine the initial illumination map by strongly adding a structural prior, and finally synthesize the enhanced image according to Retinex theory.

[0042] The foundation of Retinex theory is that the color of an object is determined by its ability to reflect light, not by the absolute value of the reflected light intensity. Image enhancement algorithms based on Retinex theory estimate the incident light L from the image I, and then calculate the image enhancement result, which is the reflecting object R. Specifically, this can be expressed as:

[0043] Image I = Reflecting object R × Incident light L

[0044] Taking the logarithm of the above equation, we get: log(R) = log(I) - log(L)

[0045] By finding the incident light L, we can obtain the information of the reflecting object R according to the above formula. L can be obtained by performing a convolution operation between Gaussian blur and I: L = F * I, where F is a Gaussian blur filter.

[0046] S12. Use a deep neural network to pre-train the data in the general face dataset to obtain a pre-trained model.

[0047] Currently available datasets of nighttime face images have a limited number of samples. If nighttime face images are used directly for training, it is difficult for neural networks to learn to detect facial features. This embodiment addresses the problem of limited sample size by using a general face dataset for pre-training, and then using the pre-trained model weights and nighttime face images for transfer learning.

[0048] A pre-trained general face dataset can be obtained using the WIDER FACE dataset, which is a public dataset with 12,000 samples and sufficient diversity in face size, expression, and occlusion. It can greatly help neural networks learn to detect facial features.

[0049] The training neural network used is YOLOv3. The YOLOv3 network architecture combines YOLOv2, Darknet-19, and residual networks, and has a total of 53 convolutional layers, hence the name Darknet-53. In terms of category prediction, the original single-label classification is improved to multi-label classification; therefore, the softmax layer used for single-label multi-class classification is replaced with a logistic regression layer for multi-label multi-class classification.

[0050] The gradient descent optimization method used during neural network training is the Adam optimization method, which uses momentum and adaptive learning rate to accelerate convergence.

[0051] S13. The dataset images of nighttime faces are fused with the first weight matrix and then fed into a deep neural network. Transfer learning is performed based on the pre-trained model to obtain a nighttime face detection model.

[0052] After obtaining a pre-trained model trained on a general face dataset, the model's weights are loaded, and transfer learning is then performed using nighttime face images based on these weights. First, the information from the nighttime face images and the weight matrix are fused, specifically as follows: Figure 2 As shown, the RGB three-channel pixel values ​​of the image are multiplied by the corresponding weights in the weight matrix, and then multiplied by a reasonable coefficient to adjust the pixel values ​​to an appropriate level, resulting in an image that integrates foreground and background separation information.

[0053] Next, the weights of the pre-trained model obtained in step S12 are loaded into the neural network, and finally the fused image information is fed into the neural network for transfer learning to obtain the nighttime face detection model.

[0054] If an image enhancement algorithm is used to process the image in step S11 to obtain a weight matrix, then in step S13 the enhanced image processed by the image enhancement algorithm and the weight matrix should be fused before transfer learning.

[0055] S14. Separate the foreground and background of the night image to be detected to obtain a second weight matrix, and fuse it with the night image to be detected to obtain a night-enhanced image. Scale the night-enhanced image at different scales, and use the night face detection model to detect the night-enhanced images at different scales.

[0056] After obtaining the nighttime face detection model, foreground and background separation is performed on the image to be detected to obtain a second weight matrix. Using the same fusion method as the training data, the RGB three-channel pixel values ​​of the image are multiplied by the corresponding weights in the second weight matrix, and then multiplied by a reasonable coefficient to adjust the image brightness to be close to the brightness of the dataset used for pre-training, resulting in an image fused with the weight matrix information. This image is then fed into the neural network loaded with the nighttime face detection model for detection. If image enhancement algorithms were used to process the image in steps S11 and S13, the same image enhancement algorithm must be used first during detection before obtaining the weight matrix and fusing it with the enhanced image.

[0057] Since the size of faces varies in images captured in real-world scenes, a multi-scale detection method can be used when using the trained nighttime face detection model to obtain more accurate detection results. Specifically, the fused image information is scaled at different scales before being fed into the neural network for detection, resulting in detection results at different scales. The scaling ratios used in this example include 0.8, 1, 1.5, and 2.

[0058] S15. Aggregate the detected results at different scales and filter out the final detection results.

[0059] The detection results include the coordinates of the detected face bounding boxes and their corresponding scores; higher scores indicate higher confidence. Non-maximum suppression is used to filter the detection results, aiming to remove redundant bounding boxes.

[0060] Non-maximum suppression involves selecting the border with the highest score from the detected border set A, adding it to the final detection result B, and removing it from set A. Then, the remaining borders in set A with an Intersection over Union (IOU) greater than a certain threshold with the highest-scoring border are removed from set A. This process is repeated until A is empty. Here, IOU is the intersection-over-union ratio of borders, representing the proportion of overlapping areas between borders. The threshold is typically set to 0.5.

[0061] If a multi-scale detection method is used in step S14, then when filtering the results, the bounding box coordinates detected at each scale need to be normalized according to the scaling scale. For example, if the scaling scale is 1.5, then the bounding box coordinates detected at that scale are divided by 1.5. Then, the bounding boxes detected at all scales are aggregated, and the final detection result is obtained using non-maximum suppression based on the bounding box scores and intersection-union ratio.

[0062] The face detection method based on foreground and background separation provided in this invention assigns different weights to different regions of the image by separating the foreground and background, reducing the problem of difficulty in distinguishing the foreground and background in nighttime environments, thereby improving the accuracy of face recognition. As a result, higher detection accuracy and better detection results can be obtained when detecting faces at night.

[0063] This invention also provides a face detection device based on foreground and background separation, wherein the device is located in a user device, such as... Figure 3 As shown, the device includes:

[0064] Foreground and background separation unit 11 is used to separate the foreground and background of the dataset images of nighttime faces to obtain a first weight matrix representing the difference between foreground and background separation at each position in the image;

[0065] Pre-training unit 12 is used to pre-train data in a general face dataset using a deep neural network to obtain a pre-trained model.

[0066] The detection model building unit 13 is used to fuse the dataset images of the nighttime faces with the first weight matrix, and then feed them into a deep neural network to perform transfer learning based on the pre-trained model to obtain a nighttime face detection model.

[0067] The detection unit 14 is used to separate the foreground and background of the night image to be detected, obtain a second weight matrix, and fuse it with the night image to be detected to obtain a night-enhanced image. The night-enhanced image is scaled at different scales, and the night face detection model is used to detect the night-enhanced images at different scales.

[0068] Aggregation unit 15 is used to aggregate the detected results at different scales and filter out the final detection results.

[0069] The face detection device based on foreground and background separation provided in this invention separates the foreground and background and assigns different weights to different regions of the image, reducing the problem of difficulty in distinguishing the foreground and background in nighttime environments, thereby improving the accuracy of face recognition. As a result, higher detection accuracy and better detection results can be obtained when detecting faces at night.

[0070] Optionally, the foreground and background separation unit 11 is used to first perform image enhancement on the dataset images of nighttime faces, and then perform foreground and background separation;

[0071] The detection unit 14 is used to first enhance the nighttime image to be detected, and then perform foreground and background separation.

[0072] Optionally, the detection model building unit 13 is used to multiply the RGB three channels of the nighttime face dataset image with the corresponding pixel positions of the first weight matrix;

[0073] The detection unit 14 is used to multiply the RGB three channels of the night image to be detected with the corresponding pixel positions of the second weight matrix.

[0074] Optionally, the aggregation unit 15 is used to normalize the detected face border coordinate information to the original scale according to the scaling ratio, and then use the non-maximum suppression method to filter the border.

[0075] The apparatus in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effects are similar, and will not be repeated here.

[0076] Those skilled in the art will understand that all or part of the processes in the above method embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0077] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A face detection method based on foreground and background separation, characterized in that, The method includes: Foreground and background separation is performed on the dataset of nighttime face images to obtain the first weight matrix representing the difference in foreground and background separation at each location in the image; A pre-trained model is obtained by using a deep neural network to pre-train data from a general face dataset. The dataset of nighttime face images is fused with the first weight matrix and then fed into a deep neural network. Transfer learning is performed based on the pre-trained model to obtain a nighttime face detection model. The nighttime image to be detected is separated into foreground and background to obtain a second weight matrix, which is then fused with the nighttime image to be detected to obtain a nighttime enhanced image. The nighttime enhanced image is scaled at different scales, and the nighttime face detection model is used to detect the nighttime enhanced images at different scales. The detected results at different scales are aggregated, and the final detection results are selected.

2. The method according to claim 1, characterized in that, The process of separating the foreground and background of the nighttime face dataset images includes: first, performing image enhancement on the nighttime face dataset images, and then performing foreground and background separation.

3. The method according to claim 2, characterized in that, The step of fusing the nighttime face dataset image with the first weight matrix includes: multiplying the RGB three channels of the nighttime face dataset image with the corresponding pixel positions of the first weight matrix.

4. The method according to claim 1, characterized in that, The process of separating the foreground and background of the nighttime image to be detected includes: first, performing image enhancement on the nighttime image to be detected, and then performing foreground and background separation.

5. The method according to claim 4, characterized in that, The fusion with the night image to be detected includes multiplying the RGB three channels of the night image to be detected with the corresponding pixel positions of the second weight matrix.

6. The method according to any one of claims 1 to 5, characterized in that, The aggregation of detected results at different scales includes: The detected face bounding box coordinates are normalized to the original scale according to the scaling ratio, and then the non-maximum suppression method is used to filter the bounding box.

7. A face detection device based on foreground and background separation, characterized in that, The device includes: The foreground and background separation unit is used to separate the foreground and background of the dataset images of nighttime faces, and obtain the first weight matrix representing the difference between the foreground and background separation at each position in the image; The pre-training unit is used to pre-train data in a general face dataset using a deep neural network to obtain a pre-trained model. The detection model building unit is used to fuse the dataset images of the nighttime faces with the first weight matrix, and then feed them into a deep neural network to perform transfer learning based on the pre-trained model to obtain a nighttime face detection model. The detection unit is used to separate the foreground and background of the night image to be detected, obtain a second weight matrix, and fuse it with the night image to be detected to obtain a night-enhanced image. The night-enhanced image is scaled at different scales, and the night face detection model is used to detect the night-enhanced images at different scales. The aggregation unit is used to aggregate the detected results at different scales and filter out the final detection results.

8. The apparatus according to claim 7, characterized in that, The foreground and background separation unit is used to first perform image enhancement on the dataset images of nighttime faces, and then perform foreground and background separation. The detection unit is used to first enhance the nighttime image to be detected, and then perform foreground and background separation.

9. The apparatus according to claim 8, characterized in that, The detection model building unit is used to multiply the RGB three channels of the nighttime face dataset image with the corresponding pixel positions of the first weight matrix; The detection unit is used to multiply the RGB three channels of the night image to be detected with the corresponding pixel positions of the second weight matrix.

10. The apparatus according to any one of claims 7 to 9, characterized in that, The aggregation unit is used to normalize the detected face border coordinate information to the original scale according to the scaling ratio, and then use the non-maximum suppression method to filter the border.

Images