A face detection method and system based on deep learning

Abstract

This specification provides a face detection method and system based on deep learning. The method includes: acquiring an image to be detected; preprocessing the image to be detected to determine a preprocessed image; and determining a face recognition result based on the preprocessed image using a face detection model. The face detection model includes at least one recognition sub-model and is a machine learning model.

Description

Technical Field

[0001] This specification relates to the field of face recognition technology, specifically to a face detection method and system based on deep learning. Background Technology

[0002] Facial recognition technology is a biometric identification technology that identifies individuals based on their facial features. It is a novel biometric identification technology that has emerged in recent years with the rapid advancements in computer science, image processing, and pattern recognition. It uses cameras or webcams to capture images or video streams containing human faces, automatically detects and tracks faces within the images, and then applies a series of related technologies to the detected faces to identify different individuals.

[0003] To address the problem of face detection, CN107392182B proposes a deep learning-based face acquisition and recognition method and apparatus. This method compares the input video stream with a learning model trained on a large number of facial photographs to acquire faces and extract facial feature data. However, this application's method of directly learning facial data is subject to many uncertainties, such as different movement states, skin colors, and environments. Under the influence of these uncertainties, the speed and accuracy of face recognition cannot be guaranteed.

[0004] Therefore, we hope to provide a face detection method and system based on deep learning that can ensure the stability of face recognition speed and accuracy, reduce the influence of various factors such as motion state, skin color and environment, and meet user needs. Summary of the Invention

[0005] One embodiment of this specification provides a face detection method based on deep learning. The method includes: acquiring an image to be detected; preprocessing the image to be detected to determine a preprocessed image; and determining a face recognition result based on the preprocessed image using a face detection model. The face detection model includes at least one recognition sub-model and is a machine learning model.

[0006] One embodiment of this specification provides a face detection system based on deep learning. The system includes: an acquisition module for acquiring an image to be detected; a preprocessing module for preprocessing the image to be detected to determine a processed image; and a recognition module for determining a face recognition result based on the processed image using a face detection model. The face detection model includes at least one recognition sub-model and is a machine learning model.

[0007] One embodiment of this specification provides a face detection device based on deep learning. The device includes at least one processor and at least one memory; the at least one memory is used to store computer instructions; the at least one processor is used to execute at least a portion of the computer instructions to implement a face detection method based on deep learning.

[0008] One embodiment of this specification provides a computer-readable storage medium. The storage medium stores computer instructions, and when a computer reads the computer instructions from the storage medium, the computer executes a deep learning-based face detection method. Attached Figure Description

[0009] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein:

[0010] Figure 1 This is an exemplary block diagram of a deep learning-based face detection system according to some embodiments of this specification;

[0011] Figure 2 This is an exemplary flowchart of a deep learning-based face detection method according to some embodiments of this specification;

[0012] Figure 3 This is an exemplary schematic diagram of a face detection model according to some embodiments of this specification. Detailed Implementation

[0013] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this specification. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0014] It should be understood that the terms “system,” “device,” “unit,” and / or “module” used herein are one way to distinguish different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0015] Unless the context clearly indicates an exception, words such as "a," "an," "a kind," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0016] Flowcharts are used in this specification to illustrate the operations performed by the system according to embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0017] Currently, facial recognition technology is being increasingly widely applied in various aspects of life, including identity authentication during account opening in the financial sector (such as banks and securities firms), VIP identification in commercial establishments, and suspect identification in the security field. Facial recognition technology has seen significant performance improvements in recent years, achieving near-human levels of accuracy in non-extreme scenarios. However, in cases of poor facial image quality (e.g., motion blur, excessively bright or dark ambient light), the recognition rate and accuracy of facial recognition will significantly decrease.

[0018] In view of this, in some embodiments of this specification, it is desirable to provide a face detection method based on deep learning, which compensates for the image based on ambient light effect data and motion data, and compensates for the face in the image based on skin color data, thereby improving the recognition speed and accuracy of face recognition.

[0019] Figure 1 This is an exemplary block diagram of a deep learning-based face detection system according to some embodiments of this specification. Figure 1 As shown, the deep learning-based face detection system 100 may include an acquisition module 110, a preprocessing module 120, and a recognition module 130.

[0020] The acquisition module 110 can be used to acquire the image to be detected.

[0021] The preprocessing module 120 can be used to preprocess the image to be detected and determine the processed image.

[0022] The recognition module 130 can be used to determine the face recognition result based on the processed image using a face detection model. The face detection model includes at least one recognition sub-model and is a machine learning model. For further explanation on determining the face recognition result using a face detection model, see [link to documentation]. Figure 2 And related content.

[0023] In some embodiments, at least one recognition sub-model includes at least one coarse recognition model and at least one deep recognition model, wherein the at least one coarse recognition model is used for coarse recognition and the at least one deep recognition model is used for fine recognition.

[0024] In some embodiments, the face detection model is trained, and the loss function used to train the face detection model is constructed based on the output of at least one recognition sub-model.

[0025] For more information on the sub-identification model and loss function, please refer to [link / reference]. Figure 3 And related content.

[0026] In some embodiments, the deep learning-based face detection system 100 may further include a processor, storage devices, etc. The processor can process data and / or information obtained from other devices or system components to execute program instructions based on this data, information, and / or processing results to perform one or more functions described in this application. The processor can retrieve pre-stored data and / or information related to the deep learning-based face detection system 100 from the storage device. In some embodiments, the deep learning-based face detection system 100 may include a network and / or other components connecting the system to external resources. The processor can retrieve data and / or information related to the deep learning-based face detection system 100 via the network.

[0027] It should be noted that the above description of the deep learning-based face detection system 100 and its modules is for convenience only and should not be construed as limiting this specification to the scope of the embodiments described. It is understood that those skilled in the art, after understanding the principles of this system, may arbitrarily combine the various modules or construct subsystems connected to other modules without departing from these principles.

[0028] Figure 2 This is an exemplary flowchart illustrating a deep learning-based face detection method according to some embodiments of this specification. In some embodiments, process 200 may be executed by a processor. Figure 2 As shown, process 200 includes the following steps:

[0029] Step 210: Obtain the image to be detected.

[0030] The image to be detected refers to the image used for face detection.

[0031] The processor can acquire the image to be detected in various ways. In some embodiments, the processor can acquire the image to be detected by capturing it using an image sensing device. An image sensing device refers to a device used to acquire the image to be detected. Image sensing devices may include infrared cameras, white light cameras, warm light full-color cameras, etc.

[0032] Step 220: Preprocess the image to be detected and determine the processed image.

[0033] Preprocessing refers to the processing performed on the image to be detected before recognition. Preprocessing can improve image readability and recognition accuracy. The preprocessed image refers to the image to be detected after preprocessing.

[0034] The processor can determine the processed image in various ways. In some embodiments, the processor can process the image to be detected using image processing techniques to determine the processed image. Image processing techniques may include any technique applicable to image processing, such as grayscale normalization methods and geometric correction.

[0035] Step 230: Based on the processed image, determine the face recognition result using a face detection model.

[0036] A face detection model refers to a model used for face detection. Face detection models can be machine learning models, such as Convolutional Neural Networks (CNN) models, Recurrent Neural Networks (RNN) models, and Deep Belief Networks (DBN) models. In some embodiments, a face detection model may include at least one recognition sub-model.

[0037] A recognition sub-model refers to a model trained with different network structures, parameters, and / or based on different algorithms. In some embodiments, at least one recognition sub-model may include at least one coarse recognition model and at least one deep recognition model. For more information on recognition sub-models, see [link to documentation]. Figure 3 And related content.

[0038] A face recognition result refers to an image containing face bounding boxes and their confidence scores. A processor can determine the face recognition result in various ways. In some embodiments, the processor can input the processed image into a face detection model to obtain the face recognition result output by the face detection model.

[0039] In some embodiments, the face detection model can be trained using a first training sample with a first label.

[0040] In some embodiments, the first training sample may include a processed image, and the first label is the face recognition result of the sample (i.e., the image with a face recognition bounding box corresponding to the processed image). In some embodiments, the first training sample can be obtained through big data analysis, and the first label can be obtained through manual annotation.

[0041] In some embodiments, the face detection model includes at least one recognition sub-model.

[0042] In some embodiments, the face detection model includes at least one recognition sub-model, which is a first recognition sub-model and a second recognition sub-model. The input of the first recognition sub-model is the processed image, and the output is a pre-recognized image. The input of the second recognition sub-model is the pre-recognized image, and the output is the face recognition result. The output of the first recognition sub-model is the input of the second recognition sub-model.

[0043] In some embodiments, the first recognition sub-model and the second recognition sub-model can be obtained by jointly training a first training sample with a first label.

[0044] The processed image is input into the first recognition sub-model to obtain the pre-recognition image output by the first recognition sub-model. This pre-recognition image is then used as training sample data and input into the second recognition sub-model to obtain the image with face recognition bounding boxes output by the second recognition sub-model. A loss function is constructed based on the sample image with face recognition bounding boxes and the image output by the second recognition sub-model with face recognition bounding boxes, and the parameters of both the first and second recognition sub-models are updated synchronously. Through parameter updates, the trained first and second recognition sub-models are obtained.

[0045] In some embodiments of this specification, a face detection model is used to perform face recognition on the preprocessed image to be detected (processed image), which can accurately detect the face in the image to be detected.

[0046] It should be noted that the above description of the process is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art can make various modifications and changes to the process under the guidance of this specification. However, these modifications and changes remain within the scope of this specification.

[0047] Figure 3 This is an exemplary schematic diagram of a face detection model according to some embodiments of this specification.

[0048] In some embodiments, at least one recognition sub-model includes at least one coarse recognition model 320 and at least one deep recognition model 350, wherein at least one coarse recognition model 320 is used for coarse recognition and at least one deep recognition model 350 is used for fine recognition.

[0049] Coarse recognition refers to a rough identification of faces in an image to be detected. For example, coarse recognition can refer to using a model with a network complexity (measured by the number of network layers, network parameters, etc.) lower than a first preset threshold to identify faces in the image to be detected. Another example is identifying regions in the image to be detected where the probability of a face appearing is greater than a second preset threshold as regions where faces may exist. The first and second preset thresholds can be based on empirical estimations.

[0050] Fine-grained recognition refers to the precise identification of faces in an image to be detected. For example, fine-grained recognition can refer to using a model with a network complexity (measured by the number of network layers, network parameters, etc.) greater than a third preset threshold to identify faces in the image. Another example is identifying regions in the image where the probability of a face appearing is greater than a fourth preset threshold as regions where faces may exist. The third and fourth preset thresholds can be based on empirical presets. The third preset threshold can be greater than the first preset threshold, and the fourth preset threshold can be greater than the second preset threshold.

[0051] The coarse recognition model 320 can be a machine learning model, such as a CNN. In some embodiments, the coarse recognition model 320 is a proposal network (P-Net) model.

[0052] The input to the coarse recognition model 320 can be the processed image 312, and the output is the uncompensated first recognition image 313. The uncompensated first recognition image 313 can refer to an image containing face recognition bounding boxes and their confidence levels, output with a low confidence threshold. The face recognition bounding boxes can reflect information such as the specific location and size of the recognized face. The processed image 312 can be determined by preprocessing the image 311 to be detected; more details can be found in [link to relevant documentation]. Figure 2 Related descriptions.

[0053] The deep recognition model 350 can be a machine learning model, such as a convolutional neural network (CNN) model. In some embodiments, the deep recognition model 350 is a refine network (R-Net) model or an output network (O-Net) model.

[0054] The input to the deep recognition model 350 is the uncompensated first recognition image 313, and the output is the face recognition result 360. The face recognition result 360 can refer to the image containing the face recognition box and its confidence level, output with a higher confidence threshold.

[0055] In some embodiments, the output of the coarse recognition model 320 can be the input of the deep recognition model 350, and the coarse recognition model 320 and the deep recognition model 350 can be jointly trained using a second training sample with a second label.

[0056] In some embodiments, the second training sample can be a processed image, and the second label can be the sample face recognition result (i.e., the image with a face recognition bounding box corresponding to the processed image). The joint training process of the coarse recognition model 320 and the deep recognition model 350 is as follows: Figure 2 The joint training process of the first and second recognition sub-models is similar, and can be found in [reference]. Figure 2 Related descriptions.

[0057] In some embodiments, the input to the depth recognition model may further include the compensated first recognition image 315 and / or the second compensated image 316. For details regarding the compensated first recognition image 315 and the second compensated image 316, please refer to the following description of the first and second compensation models.

[0058] When the input to the deep recognition model also includes the compensated first recognition image 315 and / or the second compensated image 316, the second training samples may also include the sample compensated first recognition image and / or the sample second compensated image. The uncompensated first recognition image output by the coarse recognition model is used as a training sample, and the sample compensated first recognition image and / or the sample second compensated image are input into the deep recognition model to obtain an image with a face recognition bounding box output by the deep recognition model. A loss function is constructed based on the sample image with a face recognition bounding box and the image output by the deep recognition model with a face recognition bounding box, and the parameters of the coarse recognition model and the deep recognition model are updated synchronously. Through parameter updates, the trained coarse recognition model 320 and deep recognition model 350 are obtained.

[0059] For more information on constructing the above loss function, please refer to the relevant content below.

[0060] In some embodiments of this specification, the face detection model includes at least one recognition sub-model comprising at least one coarse recognition model and one deep recognition model. The coarse recognition model can quickly detect faces that may exist in the processed image, while the deep recognition model can more accurately identify the specific location and size of the face, thus achieving both coarse and fine recognition of the face. This ensures the stability of the speed and accuracy of face recognition, thereby improving the detection accuracy and meeting user needs.

[0061] In some embodiments, at least one recognition sub-model further includes a compensation model, which includes at least a first compensation model 330 for compensating the image.

[0062] The compensation model is used to address inaccuracies in processed image information. Inaccuracies can include issues such as overly bright or dark facial areas, and image blurring caused by moving subjects. The first compensation model 330 refers to the model used for image correction.

[0063] In some embodiments, the input to the first compensation model 330 is the processed image 312, and the output is the first compensated image 314. The first compensated image 314 may refer to the processed image after compensation for contrast, brightness, etc.

[0064] In some embodiments, the first compensation model can be trained using third training samples with third labels. The training process of the first compensation model is similar to... Figure 2 The training process for face detection models in China is similar; please refer to [link / reference]. Figure 2 .

[0065] In some embodiments, the third training sample may include a processed image, and the third label may be a first compensated image of the processed image after compensation for contrast, brightness, etc. In some embodiments, the third training sample can be obtained through big data analysis, for example, by obtaining a large amount of data from third-party platforms, images captured by multiple image sensing devices, etc., and then performing statistical analysis and other processing. The third label can be obtained through manual annotation.

[0066] Some embodiments in this specification, by adding a compensation model to the face detection model to compensate for the image, can reduce the influence of the external environment, thereby optimizing the performance of the face detection model.

[0067] In some embodiments, the first compensation model 330 may include an ambient light compensation model 331.

[0068] The ambient light compensation model 331 can be a machine learning model. For example, a deep-wise separable convolution (DSC) model, a neural network (NN) model, or any combination thereof.

[0069] In some embodiments, the ambient light compensation model 331 may include a judgment layer and a compensation layer.

[0070] The decision layer refers to the model used to extract light intensity features. The decision layer may include a feature extraction layer and an eigenvalue mapping layer. In some embodiments, the feature extraction layer may be a DSC model, and the eigenvalue mapping layer may be an NN model.

[0071] In some embodiments, the input to the feature extraction layer is the processed image 312, and the output is a light intensity feature map. A light intensity feature map refers to a feature map used to describe the distribution of light intensity in an image.

[0072] In some embodiments, the input to the feature mapping layer is a light intensity feature map, and the output is a light intensity feature value. A light intensity feature value is a feature value that describes the intensity of light in an image.

[0073] In some embodiments, the output of the feature mapping layer can be the input of the feature extraction layer, and the feature extraction layer and the feature mapping layer can be jointly trained using a fourth training sample with a fourth label. The joint training process of the feature extraction layer and the feature mapping layer is similar to... Figure 2 The joint training process of the first and second recognition sub-models is similar, and can be found in [reference]. Figure 2 .

[0074] In some embodiments, the fourth training sample can be a sample image, and the fourth label can be the sample light intensity feature value corresponding to the sample image. In some embodiments, the fourth training sample can be obtained through big data analysis, and the fourth label can be obtained through manually annotated measured light intensity information.

[0075] The processor can determine whether the processed image needs compensation based on ambient light effect data.

[0076] Ambient lighting effect data refers to the feature values ​​in an image used to describe the ambient lighting effect. Ambient lighting effect data may include light intensity and color temperature. Light intensity may refer to the light intensity feature value obtained by the judgment layer based on the processed image. Color temperature may refer to the color temperature corresponding to the processed image. In some embodiments, the processor may calculate the color temperature of the processed image using an automatic white balance algorithm.

[0077] When there is severe light interference, image data can suffer significant problems. To ensure the accuracy of face detection results, if the ambient light effect of the processed image cannot be compensated, the recognition of the current processed image is abandoned, and the compensation layer is no longer input, thus interrupting ambient light compensation. The processor can determine whether the processed image needs ambient light compensation in various ways. In some embodiments, the processor can determine whether the processed image needs ambient light compensation based on a preset table and ambient light effect data. The preset table may include a pre-defined correspondence between different light intensities and color temperatures and whether compensation is required, determined based on prior knowledge or historical data.

[0078] When the processed image requires ambient light compensation, the processed image is input into the compensation layer. The compensation layer refers to the layer structure used for ambient light compensation. In some embodiments, the compensation layer can be implemented based on a processing unit that has the function of performing a reference white compensation algorithm.

[0079] The reference white compensation algorithm is as follows: Count the number of pixels whose grayscale values ​​fall within ±a% of the ambient light intensity's corresponding grayscale value in the ambient light effect data. The processor can then sort the processed image based on the grayscale values, from largest to smallest, and use the top 5% of the pixel grayscale values ​​as the reference white. The choice of a% is determined based on the specific situation. For example, in a black and white image with an intensity of 255, where 75% of the grayscale values ​​are 255 and 25% are 0, a% can be any value, as long as it avoids 0 when selecting the reference white. As another example, in an image with an intensity of 160 and a grayscale value range of 0-170, a% can be determined based on the image's grayscale value distribution, selecting either the top 50% or other proportions of grayscale values.

[0080] The average brightness of the reference white pixel can be calculated using formula (1):

[0081]

[0082] Where aveGray is the average brightness of the reference white pixels, Grayref is the reference white grayscale value, and GrayrefNum is the number of reference white pixels.

[0083] Then, the illumination compensation coefficient is calculated using formula (2):

[0084]

[0085] Where coe is the illumination compensation coefficient.

[0086] Finally, the compensation layer multiplies the original pixel values ​​of the image by the illumination compensation coefficient coe to obtain the illumination-compensated pixel values. The processor can then compensate the processed image 312 based on the illumination-compensated pixel values, outputting the first compensated image 314.

[0087] In some embodiments, the input to the coarse recognition model 320 may also be a first compensated image 314, and the output may be a compensated first recognition image 315. The compensated first recognition image 315 refers to the coarse recognition image after ambient light compensation.

[0088] In some embodiments, the training data for the coarse recognition model may further include a fifth training sample with a fifth label. The training process for the coarse recognition model can be found above.

[0089] In some embodiments, the fifth training sample may include the first compensated image of the sample, and the fifth label may be the first recognized image after sample compensation corresponding to the first compensated image of the sample. In some embodiments, the fifth training sample can be obtained through big data analysis, and the fifth label can be obtained through manual annotation.

[0090] In practical applications, changes in ambient light can affect the quality of face images. Some embodiments in this specification incorporate an ambient light compensation model into the face detection model, which can compensate for images under different ambient lighting conditions, thereby improving recognition accuracy and meeting user needs.

[0091] In some embodiments, the compensation model may further include a second compensation model 340, which is used to compensate for the face.

[0092] The second compensation model 340 refers to a model for correcting faces in an image. Corrections to faces can include adjustments for human motion and skin color, among other things.

[0093] In some embodiments, the input to the second compensation model 340 is a first recognition image, and the output is a second compensated image 316 corresponding to the first recognition image. The first recognition image may include an uncompensated first recognition image 313 and / or a compensated first recognition image 315. The second compensated image 316 refers to the image after face correction. The second compensated image 316 may include multiple images, which are images obtained based on different first recognition images.

[0094] In some embodiments, the second compensation model can be trained using a sixth training sample with a sixth label. The training process of the second compensation model is similar to... Figure 2 The training process for face detection models in China is similar; please refer to [link / reference]. Figure 2 .

[0095] In some embodiments, the sixth training sample may include an uncompensated first recognition image and / or a compensated first recognition image, and the sixth label may be a second compensated image corresponding to the uncompensated first recognition image and / or a compensated first recognition image corresponding to the compensated first recognition image. In some embodiments, the sixth training sample can be obtained through big data analysis, and the sixth label can be obtained through manual annotation.

[0096] Some embodiments in this specification, by adding a second compensation model (such as correction for human movement and correction for human skin color) to the face detection model, can automatically adjust for interference caused by human skin color and movement, thereby optimizing the performance of the face detection model, reducing the influence of various factors such as movement state and skin color, improving recognition accuracy, and meeting user needs.

[0097] In some embodiments, the second compensation model 340 includes a skin color compensation model 341. The input of the skin color compensation model 341 is a first recognition image, and the output is a second compensation image 316.

[0098] Skin color compensation model 341 refers to a layer structure used for skin color compensation. In some embodiments, skin color compensation model 341 can be implemented based on a processing unit with skin color compensation algorithm functionality.

[0099] The skin color compensation algorithm is as follows: Collect a large number of facial image samples. The collected samples should be diverse, including different genders, ages, and skin types. Extract the skin color region of the face for statistical analysis. Convert the color space to YCbCr color space and count the Cb and Cr values ​​of all pixels in the skin color region of the face in the sample. Calculate the mean and variance of all pixel data. Convert the mean to RGB grayscale values ​​and the variance to a pixel value ratio to obtain the skin color value and its deviation b%. Count the number of pixels whose grayscale value corresponds to the skin color value within ± b%, and sort the grayscale values ​​from largest to smallest. Use the top 5% of the grayscale values ​​as reference skin color pixels. (For details on the top 5, see...) Figure 3 The above description.

[0100] The average brightness of the reference skin color pixels can be calculated using formula (3):

[0101]

[0102] Where aveSkin is the average brightness of the reference skin color pixels, Skinref is the reference skin color value, and SkinrefNum is the number of reference skin color pixels.

[0103] Then, the skin color compensation coefficient is calculated using formula (4):

[0104]

[0105] Where coe′ is the skin color compensation coefficient.

[0106] Finally, the skin color compensation model multiplies the original pixel value of pixels whose grayscale value is within ±b% of the corresponding grayscale value of the skin color value by the skin color compensation coefficient coe′ to obtain the skin color compensated pixel value. Pixels whose corresponding grayscale value is outside ±b% are not processed. The processor can compensate the first recognition image based on the skin color compensated pixel value and output the second compensated image 316.

[0107] In practical applications, facial images of different skin tones may be subject to varying degrees of interference. Some embodiments in this specification, by incorporating a skin tone compensation model into the face detection model, can compensate for facial images under different skin tone conditions, thereby improving recognition accuracy.

[0108] In some embodiments, the second compensation model 340 includes a motion compensation model 342.

[0109] In some embodiments, the motion compensation model 342 may include a motion feature extraction layer and a motion compensation layer. The motion compensation model 342 refers to a model used to correct image blurring caused by facial movement. In some embodiments, the motion feature extraction layer and the motion compensation layer may be a Convolutional Neural Network (CNN) model.

[0110] In some embodiments, the input to the motion feature extraction layer is multiple frames of the first recognition image based on adjacent time points (e.g., 1 second), and the output is the facial motion features of the multiple frames of the first recognition image. Facial motion features may include facial movement speed and rotational angular velocity, etc.

[0111] In some embodiments, the motion feature extraction layer can be trained using a seventh training sample with a seventh label. In some embodiments, the seventh training sample can be a multi-frame first recognition image, and the seventh label can be the facial motion features corresponding to the multi-frame first recognition image. The training process of the motion feature extraction layer is similar to... Figure 2 The training process for face detection models in China is similar; please refer to [link / reference]. Figure 2 In some embodiments, the seventh training sample can be obtained from historical shooting data, and the seventh label can be obtained through manual annotation.

[0112] In some embodiments, the input to the motion compensation layer includes the optimal single-frame image, the frame number, and facial motion features, and the output is the motion-compensated first recognition image. The optimal single-frame image refers to the single-frame image with the highest sharpness or facial integrity among the multiple uncompensated first recognition images 313. In some embodiments, the processor can obtain the single-frame image with the highest sharpness using a gradient-based sharpness evaluation algorithm. In some embodiments, the processor can obtain the single-frame image with the highest facial integrity using a semantic segmentation algorithm. The processor can determine the single-frame image with the highest sharpness or the single-frame image with the highest facial integrity obtained above as the optimal single-frame image. The frame number refers to the sequence number of the optimal single-frame image in the sequence of multiple uncompensated first recognition images 313.

[0113] When the second compensation model 340 also includes a motion compensation model 342, the output of the motion compensation model 342 can be the input of the skin color compensation model 341. The first recognized image is input into the motion compensation model 342 to obtain the motion-compensated image, and then the motion-compensated image is input into the skin color compensation model 341 to obtain the second compensated image 316 output by the skin color compensation model 341. In the second compensation model 340, the processing order of the motion compensation model and the skin color compensation model can be preset.

[0114] In some embodiments, the output of the motion feature extraction layer can be the input of the motion compensation layer, and the motion feature extraction layer and the motion compensation layer can be jointly trained using an eighth training sample with an eighth label.

[0115] In some embodiments, each group of training samples in the eighth training sample may include multiple frames of first recognition images, the optimal single frame image, and the sample frame number. The eighth label is the motion-compensated first recognition image corresponding to each group of training samples. The multiple frames of first recognition images are input into the motion feature extraction layer to obtain the face motion features output by the motion feature extraction layer. These face motion features, along with the optimal single frame image and the sample frame number, are input into the motion compensation layer to obtain the motion-compensated first recognition image output by the motion compensation layer. The joint training process of the motion feature extraction layer and the motion compensation layer is as follows: Figure 2 The joint training process for the first and second recognition sub-models is similar. For more information on the joint training process of the motion feature extraction layer and the motion compensation layer, please refer to [link to relevant documentation]. Figure 2 Related descriptions.

[0116] In practical applications, facial movement can cause significant changes in facial images, leading to a decrease in recognition performance. Some embodiments in this specification incorporate a motion compensation model into the face detection model to compensate for interference caused by facial movement, thereby improving recognition accuracy.

[0117] In some embodiments, the face detection model can be trained, and the loss function used to train the face detection model is constructed based on the output of at least one recognition sub-model. For example, the loss function used to train the face detection model can be constructed based on the output of a deep recognition model 350.

[0118] In some embodiments, the loss function used to train the face detection model can be constructed based on the loss term of each of the multiple branches of the face detection model.

[0119] In some embodiments of this specification, the loss function used to train the face detection model is constructed based on the output of at least one recognition sub-model. This can improve the accuracy of the face detection model obtained through training, which is beneficial to improving the accuracy of the face recognition results determined by the face detection model in the future, and thus enabling more accurate face detection.

[0120] In some embodiments, the loss function can be constructed by weighting the loss terms of each of the multiple branches of the face detection model, wherein the multiple branches include at least an uncompensated branch, a branch based on a first compensation, a branch based on a second compensation, and a branch based on both the first and second compensations.

[0121] The uncompensated branch refers to the branch whose input is the uncompensated first recognition image 313. The branch based on the first compensation refers to the branch whose input is the compensated first recognition image 315. The branch based on the second compensation refers to the branch whose input is the compensated image processed only by the second compensation model. The branch based on both first and second compensation refers to the branch whose input is the compensated image processed first by the first compensation model and then by the second compensation model. See the above for explanations regarding the first and second compensation models.

[0122] For example, the loss function can be calculated using formula (5):

[0123] Fk1x1+k2x2+k3x3+k4x4(5)

[0124] Where F represents the loss function, x1 represents the loss term of the uncompensated branch, x2 represents the loss term of the branch based on the first compensation, x3 represents the loss term of the branch based on the second compensation, x4 represents the loss term of the branches based on the first and second compensations, and k1, k2, k3, and k4 are the weights corresponding to the loss terms of the above four branches, respectively. In some embodiments, k1, k2, k3, and k4 can be determined empirically. The values ​​of k1, k2, k3, and k4 vary depending on the samples.

[0125] In some embodiments, the processor can input the first uncompensated recognition image into the deep recognition model, the deep recognition model outputs the face recognition result, and the loss term of the uncompensated branch is determined based on the face recognition result output by the deep recognition model and the label corresponding to the first uncompensated recognition image.

[0126] In some embodiments, the processor can input the sample-compensated first recognition image into the deep recognition model, the deep recognition model outputs a face recognition result, and the processor determines the loss term based on the first compensation branch based on the face recognition result output by the deep recognition model and the label corresponding to the sample-compensated first recognition image.

[0127] In some embodiments, the processor can input the uncompensated first recognition image into a second compensation model, the second compensation model outputs a second compensation image, input the second compensation image output by the second compensation model into a depth recognition model, the depth recognition model outputs a face recognition result, and determine a loss term based on the second compensation branch based on the face recognition result output by the depth recognition model and the label corresponding to the uncompensated first recognition image.

[0128] In some embodiments, the processor can input the sample-compensated first recognition image into a second compensation model, the second compensation model outputs a second compensation image, input the second compensation image output by the second compensation model into a depth recognition model, the depth recognition model outputs a face recognition result, and determine a loss term based on the first compensation and the second compensation based on the face recognition result output by the depth recognition model and the label corresponding to the sample-compensated first recognition image.

[0129] In some embodiments of this specification, by constructing a loss function based on multiple branches of a face detection model, the face detection model can more comprehensively consider the differences between different branches, thereby improving the accuracy of the face detection model obtained through training. This is more conducive to improving the accuracy of the face recognition results determined by the face detection model in the future, and thus can further detect faces more accurately.

[0130] In some embodiments, the weights of the loss term in each of the multiple branches of the face detection model are related to the compensation difference distribution.

[0131] The compensation difference distribution refers to the distribution of multiple differences between the image before and after sample compensation. The image before compensation is the input image of the corresponding compensation model (e.g., the first compensation model, the second compensation model). The image after compensation is the output image of the corresponding compensation model (e.g., the first compensation model, the second compensation model). Different images before and after compensation correspond to different images, therefore the corresponding compensation difference distributions are different, and consequently, the weights of the loss terms corresponding to the images before and after compensation are also different.

[0132] The compensation difference distribution includes the first compensation difference, the second compensation difference, and the third compensation difference.

[0133] The first compensation difference refers to the difference between the processed image 312 and the first compensation image 314 (such as the difference between pixels).

[0134] The second compensation difference refers to the average difference (such as the difference between pixels) between the processed image 312 and the two second compensation images 316. The two second compensation images 316 are: one is the compensation image processed only by the second compensation model, and the other is the compensation image processed first by the first compensation model and then by the second compensation model.

[0135] The third compensation difference refers to the difference between the first compensated image 314 and the compensated image processed by the first compensation model and then the second compensation model (such as the difference between pixels). Since ambient light compensation and skin color compensation and motion compensation are different dimensions, it is meaningless to compare the difference between the first compensated image 314 and the compensated image processed only by the second compensation model.

[0136] The larger the difference in the compensation difference distribution, the greater the weight of the corresponding loss term. In some embodiments, the processor first selects a preset value for k1, for example, k1 is 0.1, and then calculates k2, k3 and k4 according to formulas (6) and (7):

[0137] k2:k3:k4=z1:z2:z3 (6)

[0138] k2+k3+k4=1-k1 (7) Among them, z1, z2 and z3 are the first compensation difference, the second compensation difference and the third compensation difference, respectively.

[0139] In some embodiments, the weight (k1) of the loss term in the uncompensated branch is related to the sum of differences. For example, the weight of the loss term in the uncompensated branch is negatively correlated with the sum of differences (the larger the sum of differences, the smaller the weight of the loss term in the uncompensated branch). The sum of differences is the sum of the first compensated difference, the second compensated difference, and the third compensated difference.

[0140] The difference before and after compensation is greater, so the result after compensation should be used as the recognition result. Some embodiments in this specification, by associating the weight of the loss term of the uncompensated branch with the sum of differences, can make the face detection model pay more attention to branches with large differences caused by factors such as lighting, skin color, and motion. This can further improve the accuracy of the face detection model obtained during training, and is more conducive to improving the accuracy of the face recognition results determined by the face detection model in subsequent use, thereby enabling more accurate face detection.

[0141] In some embodiments of this specification, by associating the weights of the loss term with the compensation difference distribution, the larger the difference, the greater the weight of the loss term. The face detection model will emphasize the branches that have received greater interference, which can improve the accuracy of the face detection model obtained through training. This is beneficial to further improve the accuracy of the face recognition results determined by the face detection model in subsequent use, and thus can detect faces more accurately.

[0142] Some embodiments of this specification provide a face detection device based on deep learning. The device includes at least one processor and at least one memory; the at least one memory is used to store computer instructions; the at least one processor is used to execute at least some of the computer instructions to implement the face detection method based on deep learning.

[0143] Some embodiments of this specification provide a computer-readable storage medium that stores computer instructions. When a computer reads the computer instructions from the storage medium, the computer executes a deep learning-based face detection method.

[0144] The basic concepts have been described above. Obviously, for those skilled in the art, the detailed disclosure above is merely illustrative and does not constitute a limitation of this specification. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this specification. Such modifications, improvements, and corrections are suggested in this specification and therefore remain within the spirit and scope of the exemplary embodiments described herein.

[0145] Furthermore, this specification uses specific terms to describe embodiments thereof. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Moreover, certain features, structures, or characteristics in one or more embodiments of this specification can be appropriately combined.

[0146] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this specification are not intended to limit the order of the processes and methods described herein. Although various examples have been discussed in the foregoing disclosure of some embodiments of the invention that are currently considered useful, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the spirit and scope of the embodiments described herein. For example, while the system components described above can be implemented using hardware devices, they can also be implemented solely using software solutions, such as installing the described system on existing servers or mobile devices.

[0147] Similarly, it should be noted that, in order to simplify the description disclosed herein and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of embodiments in this specification may sometimes combine multiple features into a single embodiment, drawing, or description thereof. However, this method of disclosure does not imply that the subject matter of this specification requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of a single embodiment disclosed above.

[0148] In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of embodiments are modified in some examples with the terms "approximately," "approximately," or "generally." Unless otherwise stated, "approximately," "approximately," or "generally" indicates that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may be changed depending on the characteristics required by individual embodiments. In some embodiments, numerical parameters should take into account specified significant digits and employ a general method of digit reservation. Although the numerical ranges and parameters used to confirm their breadth of range in some embodiments of this specification are approximate values, in specific embodiments, such values ​​are set as precisely as feasible.

[0149] For each patent, patent application, patent application publication, and other material, such as articles, books, specifications, publications, and documents, referenced in this specification, the entire contents of which are incorporated herein by reference. This excludes historical application documents that are inconsistent with or conflict with the content of this specification, as well as documents that limit the broadest scope of the claims in this specification (currently or subsequently appended to this specification). It should be noted that in the event of any inconsistency or conflict between the descriptions, definitions, and / or terminology used in the supplementary materials to this specification and the content of this specification, the descriptions, definitions, and / or terminology used in this specification shall prevail.

[0150] Finally, it should be understood that the embodiments described in this specification are merely illustrative of the principles of the embodiments described herein. Other variations may also fall within the scope of this specification. Therefore, alternative configurations of the embodiments described herein are intended to be illustrative rather than limiting, and should be considered consistent with the teachings of this specification. Accordingly, the embodiments described herein are not limited to those explicitly introduced and described herein.

Claims

1. A face detection method based on deep learning, characterized in that, The method includes: Acquire the image to be detected; The image to be detected is preprocessed to determine the processed image; Based on the processed image, a face recognition result is determined using a face detection model. The face detection model includes at least one recognition sub-model, which is a machine learning model. The at least one recognition sub-model includes at least one coarse recognition model, at least one deep recognition model, and a compensation model. The at least one coarse recognition model is used for coarse recognition, the at least one deep recognition model is used for fine recognition, and the compensation model includes at least a first compensation model and a second compensation model. The first compensation model is used to compensate for the image, and the second compensation model is used to compensate for the face. The first compensation model includes an ambient light compensation model, and the second compensation model includes a skin color compensation model and a motion compensation model. The ambient light compensation model includes a judgment layer and a compensation layer. The input of the judgment layer includes the processed image, and the output of the judgment layer includes light intensity feature values. The input of the compensation layer includes the processed image, and the output of the compensation layer includes a first compensation image. The input of the coarse recognition model includes the first compensated image, and the output of the coarse recognition model includes the first recognized image; The input of the motion compensation model includes multiple frames of the first recognition image, and the output of the motion compensation model includes the motion-compensated first recognition image; the input of the skin color compensation model includes the motion-compensated first recognition image, and the output of the skin color compensation model includes a second compensation image. The input to the depth recognition model includes the second compensated image, and the output of the depth recognition model includes the face recognition result; The face detection model is trained, and the loss function used to train the face detection model is constructed based on the output results of the at least one recognition sub-model, wherein: The loss function is constructed by weighting the loss terms of each of the multiple branches of the face detection model, and the multiple branches include at least an uncompensated branch, a branch based on a first compensation, a branch based on a second compensation, and a branch based on both the first compensation and the second compensation. The weights of the loss term are related to the compensation difference distribution, which includes a first compensation difference, a second compensation difference, and a third compensation difference. The first compensation difference refers to the pixel difference between the processed image and the first compensation image. The second compensation difference refers to the average pixel difference between the processed image and two second compensation images. The two second compensation images include a compensation image processed only by the second compensation model and a compensation image processed first by the first compensation model and then by the second compensation model. The third compensation difference refers to the pixel difference between the first compensation image and the compensation image processed first by the first compensation model and then by the second compensation model. The weight of the loss term in the uncompensated branch is negatively correlated with the sum of differences, which is the sum of the first compensated difference, the second compensated difference, and the third compensated difference.

2. A face detection system based on deep learning, characterized in that, The system includes: The acquisition module is used to acquire the image to be detected; The preprocessing module is used to preprocess the image to be detected and determine the processed image; A recognition module is used to determine a face recognition result based on the processed image using a face detection model. The face detection model includes at least one recognition sub-model, which is a machine learning model. The at least one recognition sub-model includes at least one coarse recognition model, at least one deep recognition model, and a compensation model. The at least one coarse recognition model is used for coarse recognition, and the at least one deep recognition model is used for fine recognition. The compensation model includes at least a first compensation model and a second compensation model. The first compensation model is used to compensate for the image, and the second compensation model is used to compensate for the face. The first compensation model includes an ambient light compensation model, and the second compensation model includes a skin color compensation model and a motion compensation model. The ambient light compensation model includes a judgment layer and a compensation layer. The input of the judgment layer includes the processed image, and the output of the judgment layer includes light intensity feature values. The input of the compensation layer includes the processed image, and the output of the compensation layer includes a first compensation image. The input of the coarse recognition model includes the first compensated image, and the output of the coarse recognition model includes the first recognized image; The input of the motion compensation model includes multiple frames of the first recognition image, and the output of the motion compensation model includes the motion-compensated first recognition image; the input of the skin color compensation model includes the motion-compensated first recognition image, and the output of the skin color compensation model includes a second compensation image. The input to the depth recognition model includes the second compensated image, and the output of the depth recognition model includes the face recognition result; The face detection model is trained, and the loss function used to train the face detection model is constructed based on the output results of the at least one recognition sub-model, wherein: The loss function is constructed by weighting the loss terms of each of the multiple branches of the face detection model, and the multiple branches include at least an uncompensated branch, a branch based on a first compensation, a branch based on a second compensation, and a branch based on both the first compensation and the second compensation. The weights of the loss term are related to the compensation difference distribution, which includes a first compensation difference, a second compensation difference, and a third compensation difference. The first compensation difference refers to the pixel difference between the processed image and the first compensation image. The second compensation difference refers to the average pixel difference between the processed image and two second compensation images. The two second compensation images include a compensation image processed only by the second compensation model and a compensation image processed first by the first compensation model and then by the second compensation model. The third compensation difference refers to the pixel difference between the first compensation image and the compensation image processed first by the first compensation model and then by the second compensation model. The weight of the loss term in the uncompensated branch is negatively correlated with the sum of differences, which is the sum of the first compensated difference, the second compensated difference, and the third compensated difference.

3. A face detection device based on deep learning, characterized in that, The device includes at least one processor and at least one memory; The at least one memory is used to store computer instructions; The at least one processor is configured to execute at least a portion of the computer instructions to implement the method of claim 1.

4. A computer-readable storage medium storing computer instructions, wherein when a computer reads the computer instructions in the storage medium, the computer executes the method as described in claim 1.

Images