A dense small face detection method and system based on convolutional neural network

Abstract

The application discloses a dense small face detection method based on a convolutional neural network, which comprises collecting picture data sets and marking face data, pre-processing the picture data sets, and increasing the number of small faces in the pictures; the network structure is designed, specifically, the backbone network is improved by introducing Ghost convolution and Leaky Relu activation function based on MobilenetV3; the SPP module, FPN and PAN are combined to strengthen the feature extraction network; the SSH module is adopted to observe the features from the global view and fuse the context features; the effective feature layer is output; the prediction result is obtained according to the effective feature layer, the network is trained, and the effective detection of the face is finally realized; the prediction result comprises a classification prediction result, a frame regression prediction result and a face key point regression prediction result. The application shows great flexibility in face detection, especially for the detection scene of serious face occlusion and dense small faces, and also has good effect on the detection under a complex scene.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for dense small face detection based on convolutional neural networks. Background Technology

[0002] Face detection is a type of object detection. Its main task is to determine whether a face exists in a given image, and if so, to determine its location and confidence score. Our brains have a dedicated face detection module, highlighting its importance. Driven by societal needs, face detection is becoming increasingly crucial. It forms the foundation for a range of applications in face image analysis. Excellent face detection algorithms, with slight modifications, can be applied to face recognition and verification, face tracking in surveillance scenarios, facial expression analysis, and facial attribute recognition. Furthermore, face detection is the initial step in all modern vision-based human-computer and human-machine interaction systems. Many social networks now use face detection mechanisms for person tagging, and even the currently popular autonomous driving technology relies heavily on it. Therefore, it's clear that face detection technology will become increasingly important in today's rapidly evolving world.

[0003] Dense small face detection is a challenging task, currently facing the following challenges: limited available features; small faces fall under the category of small object detection, which suffers from low resolution, limited visualization information, and blurred features that are difficult to distinguish; and a small percentage of existing datasets contain small faces. Currently, the most commonly used dataset for face detection is the Wider dataset. The Face dataset contains a limited number of small faces, and the annotation of these small faces is done manually, resulting in inconsistent accuracy. This hinders the detection model's ability to learn about small faces effectively, leading to low detection accuracy. Furthermore, imbalanced sample distribution persists. Current object detection methods often use anchor boxes, generating numerous boxes for each target, but only one is retained (positive sample), while the rest are negative. This imbalance further reduces the accuracy of small face detection. Localization is also challenging. Due to their small size, the bounding boxes of small faces are very small, and even slight deviations can cause regression errors, hindering accurate target localization. Finally, clustering of small targets is another issue. In convolutional neural networks, the feature map output after multiple convolutions is significantly smaller than the original image. This can cause dense clusters of small faces to overlap in the feature map, making it difficult for detection models to separate them correctly. Even the NMS algorithm may filter out correct predicted boxes, resulting in missed detections. Summary of the Invention

[0004] In view of the problems existing in the current dense small face detection methods and systems based on convolutional neural networks, this invention is proposed.

[0005] Therefore, the purpose of this invention is to comprehensively address the currently solvable problem of small-scale face detection, such as strengthening feature extraction to address the problem of limited usable features; improving the loss function to address the problem of imbalanced samples; and data augmentation to address the problem of insufficient samples. Simultaneously, this invention comprehensively considers detection speed and accuracy, avoiding the design of a large network model and prioritizing lightweight design. It aims to create a system that can be deployed on mobile devices for real-time and accurate detection of dense small-scale faces, facilitating the use and deployment of enterprise software.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, embodiments of the present invention provide a dense small face detection method based on convolutional neural networks, which includes collecting image datasets and labeling face data; preprocessing the image datasets to increase the number of small faces in the images; designing the network structure, specifically based on MobileNetV3, introducing Ghost convolution and LeakyReLU activation functions to improve the backbone network; combining the SPP module, FPN, and PAN to strengthen the first-layer feature extraction network; for the second-layer feature extraction network, using the SSH module to observe features globally and fuse contextual features; outputting an effective feature layer; obtaining prediction results based on the effective feature layer; training the network to finally achieve effective face detection; the prediction results include classification prediction results, bounding box regression prediction results, and facial landmark regression prediction results.

[0008] As a preferred embodiment of the dense small face detection method based on convolutional neural networks described in this invention, the face data includes blur level, expression, illumination, occlusion, and pose; the preprocessing of the image dataset specifically involves dividing the dataset into a training set, a validation set, and a test set according to a certain ratio, and randomly cropping and stitching a portion of the images to perform copypasting, with the aim of increasing the number of small faces in the images; and annotating the faces with ground truth bounding boxes and facial landmarks in the training set, wherein the facial landmarks include the eyes, the corners of the mouth, and the tip of the nose.

[0009] As a preferred embodiment of the dense small face detection method based on convolutional neural networks described in this invention, the core of MobilenetV3 is the bneck module, which includes a 1*1 convolution module, a depthwise separable convolution module, and a channel attention mechanism module added to the depthwise separable module. The improvement of the backbone network by introducing Ghost convolution and Leaky ReLU activation function includes changing the 1*1 convolution in the head and the 1*1 convolution in the depthwise separable module (i.e., the tail) of the original bneck structure to Ghost convolution; and improving the ReLU activation in the original depthwise separable convolution module to the Leaky ReLU activation function.

[0010] As a preferred embodiment of the dense small face detection method based on convolutional neural networks described in this invention, the method of combining the SPP module, FPN, and PAN to enhance the feature extraction network includes adding an SPP module before the FPN, with two 3*3 convolutions at the beginning and end, and a fusion of feature channels using three max pooling methods: 5*5, 9*9, and 13*13, respectively.

[0011] As a preferred embodiment of the dense small face detection method based on convolutional neural networks described in this invention, the method employs an SSH module to observe features globally and fuse contextual features. The output effective feature layer includes adding an SSH module to fuse contextual features. Specifically, a LeakyReLU activation function is added after the SSH output module to effectively reduce information loss during feature extraction. Finally, a deformable convolutional module—DCNV1—is added to further refine feature extraction.

[0012] As a preferred embodiment of the dense small face detection method based on convolutional neural networks described in this invention, the output effective feature layer includes the following steps: obtaining three effective feature layers through the SSH module, each feature layer obtaining three prediction results. These three effective feature layers are equivalent to dividing the entire image into grids of different sizes. When the input image is (640, 640, 3), the output images obtained are (80, 80, 64), (40, 40, 64), and (20, 20, 64), respectively. Taking (20, 20, 64) as an example, it is equivalent to dividing the original image into a 20x20 grid, with two prior boxes on each grid, each prior box representing a certain area on the image. The prediction results include face classification prediction results, bounding box regression prediction results, and facial landmark prediction results. The prediction results are used to determine whether the prior boxes contain a face, and to adjust the prior boxes containing faces to obtain prediction boxes and facial landmarks.

[0013] As a preferred embodiment of the dense small-face detection method based on convolutional neural networks described in this invention, the training network ultimately achieves effective face detection by designing a loss function, which is composed of a multi-task loss function, where γ1 and γ2 are balancing factors for the three losses, N is the number of input samples per batch, and N1 is the total number of facial key points in each batch of images; the total loss function is expressed as:

[0014]

[0015] Part One Represents the loss in face classification, among which It is the face area loss balancing factor, t i It is the set of positions of the predicted bounding boxes corresponding to positive samples.

[0016] Secondly, embodiments of the present invention provide a dense small-scale face detection system based on convolutional neural networks, comprising: a dataset preparation and preprocessing module for creating and preprocessing an image dataset containing diverse faces; a network backbone design module for constructing and optimizing the backbone structure of the convolutional neural network, based on MobilenetV3 and improved by Ghost convolution and Leaky ReLU; a feature extraction network optimization module for enhancing the system's feature extraction capability through the fusion of SPP module, FPN, and PAN; and an effective feature layer and prediction module for outputting an effective feature layer for face detection and performing face classification, bounding box regression, and key point localization prediction based on the effective feature layer.

[0017] Thirdly, embodiments of the present invention provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any of the steps of the above-described dense small face detection method based on convolutional neural networks.

[0018] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the above-described dense small face detection method based on convolutional neural networks.

[0019] The beneficial effects of this invention are its high flexibility in face detection, especially in scenarios with severe face occlusion and densely packed small faces, while also demonstrating good performance in complex scenes. The network comprises a three-layer innovative structure. The first layer, the backbone network, is based on the existing MobileNet V3 with further lightweight innovations, applying the latest activation functions to maintain a lightweight backbone network while increasing object recognition accuracy. The second layer, the feature extraction network, innovatively uses a combination of three modules to enhance feature extraction. The third layer, the feature extraction network, fully considers the contextual information in the image, innovatively combining the SSH module and deformable convolution module for refined feature extraction, significantly improving the accuracy of small face detection. When detecting faces in the test image, this invention employs a multi-scale detection method, enabling the acquisition of more information on occluded and low-resolution faces, further improving the accuracy and recall of face detection. Therefore, this invention has the advantages of high speed and high accuracy, making it fully adaptable to mobile deployments, achieving real-time and accurate results. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0021] Figure 1 This is a schematic diagram of the overall process of the present invention.

[0022] Figure 2 This is a schematic diagram of the activation function module in the backbone network of the present invention.

[0023] Figure 3 This is a schematic diagram of Ghost convolution applied in the backbone network of the present invention using MobilenetV3.

[0024] Figure 4 This is a schematic diagram of the backbone network structure of the present invention.

[0025] Figure 5 This is a schematic diagram of the first-layer feature extraction network structure of the present invention.

[0026] Figure 6 This is a schematic diagram of the second-layer feature extraction network structure of the present invention.

[0027] Figure 7 This is a graph showing the decrease in the network training loss function of the present invention.

[0028] Figure 8The detection target accuracy diagram of the present invention.

[0029] Figure 9 This is a schematic diagram illustrating the detection effect of the present invention. Detailed Implementation

[0030] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0031] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0032] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0033] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0034] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0035] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0036] Example 1

[0037] Reference Figures 1-9 This is the first embodiment of the present invention, which provides a dense small-face detection method based on a convolutional neural network, comprising:

[0038] S1: Collect image datasets and label face data. Preprocess the image datasets to increase the number of small faces in the images.

[0039] Specifically, facial data includes blur level, expression, lighting, occlusion, and pose. The image dataset is preprocessed by dividing it into training, validation, and test sets according to a certain ratio. A portion of the images are randomly cropped and stitched together to perform copypasting, which aims to increase the number of small faces in the images. In the training set, ground truth bounding boxes and facial landmarks are annotated for the faces. Facial landmarks include the eyes, the corners of the mouth, and the tip of the nose.

[0040] Furthermore, samples were sourced from the public dataset WIDER FACE, which contains 32,203 images and 393,703 face samples. Of these, 158,989 are labeled faces in the training set, and 39,496 are in the validation set. Each subset includes three levels of testing difficulty: Easy, Medium, and Hard. These faces exhibit significant variations in scale, pose, and lighting. However, the dataset lacks annotations for facial landmarks, so a custom dataset was created. Labelme was used for labeling, marking five points: the two eyes, the nose, and the two corners of the mouth. The created dataset was stored in the `data` folder. Data augmentation was performed using GridMask, which generates a mask of the same size as the input feature map. Gray areas in the mask have a value of 1, and black areas have a value of 0. This mask is then multiplied by the input feature map, effectively removing information from specific locations and regions of the input feature map.

[0041] Furthermore, the image data was preprocessed. The dataset was divided into training, validation, and test sets at a ratio of 40%, 10%, and 50%, respectively. A portion of the images were randomly cropped and stitched together using copypasting to increase the number of small faces in the images. In the training set, ground truth bounding boxes and facial landmarks were annotated for the faces. There are five facial landmarks: the eyes, the corners of the mouth, and the tip of the nose.

[0042] S2: The network structure is designed, specifically based on MobilenetV3, with the introduction of Ghost convolution and LeakyReLU activation functions to improve the backbone network; the SPP module, FPN and PAN are combined to enhance the feature extraction network; the SSH module is used to observe features globally and fuse contextual features; and an effective feature layer is output.

[0043] It should be noted that the backbone network is used as the first step to extract preliminary features from the image, with the aim of enabling the machine to learn what a face is and to make a preliminary identification of faces. Therefore, the backbone network does not need to be very large; it only needs to be lightweight while maintaining good recognition accuracy. Hence, it is designed based on MobilenetV3 with two improvements.

[0044] The first improvement: The core of MobileNetV3 is the bneck module, which consists of a 1x1 convolutional module (for dimensionality increase), a depthwise separable convolutional module, and a channel attention mechanism module added to the depthwise separable module. Considering the increased computational cost and training complexity, it was further lightweighted—the 1x1 convolutions at the head of the original bneck structure and the 1x1 convolutions (i.e., the tail) in the depthwise separable module were replaced with Ghost convolutions. (Ghost convolution: Research has found that the feature maps obtained by ordinary convolution contain "shadow feature maps," that is, one feature map is almost identical to another. However, such feature map pairs can be obtained simply by linear transformation, and there is no need to waste a lot of resources by performing convolution. Therefore, researchers combined convolution and linear transformation to propose Ghost convolution.) Compared with ordinary convolution, Ghost convolution greatly reduces the number of convolutional kernels and reduces the computational cost.

[0045] The second improvement is to replace the ReLU activation in the original depthwise separable convolution module with the Leaky ReLU activation function, which allows for better updates of network weights and makes training smoother and easier.

[0046] Furthermore, the backbone network has three output points, denoted as P1, P2, and P3, with the output locations shown below. Figure 3In stages 1, 2, and 3, when the input image size is 640*640*3, a stride of 1 in the convolutional layer does not change the size of the input feature map. A stride of 2 reduces the size of the input feature map to half its original size. Therefore, the output of P1 is 80*80*64, P2 is 40*40*128, and P3 is 20*20*256. The backbone network is first extracted and trained separately on the ImageNet dataset. The trained weights are then incorporated into the entire network.

[0047] Preferably, the second layer of the network provided in this embodiment is a feature extraction layer, the purpose of which is to perform refined feature extraction. Considering the special characteristic that positional information will be lost during the convolution process of small targets, feature fusion and increasing the receptive field are performed. The advantage of increasing the receptive field is that features can be observed from a larger perspective, and the extraction is more complete and targeted.

[0048] The top-level feature map P3, output from the first step, first passes through the SPP module. This involves three convolutional layers with kernel sizes of 1x1, 3x3, and 1x1. The first convolution transforms 256 channels into 128 channels, the second 3x3 convolution transforms 128 into 256 channels, and the third 1x1 convolution transforms 256 into 128 channels. At this point, the top-level output changes from 20x20x256 to 20x20x128. Next, the SPP layer performs max pooling with kernel sizes of 5x, 9x, and 1x3, followed by a 1x1 convolutional layer. This pads the input feature map. After max pooling, the feature map size remains unchanged; only the number of channels is fused, resulting in an output of 512 channels. Then, a third convolutional layer with the same kernel size is performed, with the number of channels changing from 512 to 128, 128 to 256, and 256 to 128. The final output is a feature map F3 of size 20x20x128.

[0049] Then, the three feature maps P1, P2, and F3 output by the backbone network are processed together using FPN. F3 undergoes a 1x1 convolution and a 2x upsampling to become a 40x40x64 feature map. P2 undergoes a 1x1 convolution to adjust the number of channels from 128 to 64. Then, the two features are fused, that is, the two features are concatenated to obtain 128 channels. After fusion, the same three-layer convolution is performed to eliminate the aliasing effect of upsampling, resulting in output F2 with a feature map size of 40x40x64. F2 is then subjected to a 1x1 convolution and a 2x upsampling to obtain a feature map of 80x80x32. Similarly, P1 is subjected to a 1x1 convolution, and F2 and P1 are fused in the same way to obtain a feature map of 80x80x64. The same three-layer convolution is performed on it to eliminate the aliasing effect of upsampling, resulting in output F1 with a size of 80x80x32. F1 is retained.

[0050] At this point, the three feature maps change from the initial P1-80*80*64, P2-40*40*128, P3-20*20*256 to F1-80*80*32, F2-40*40*64, and F3-20*20*128. Next, a bottom-up reverse fusion is performed on the three feature maps. F1 is convolved with a 1*1 kernel to obtain the first output C1-80*80*64. Simultaneously, F1 is convolved with a 3*3 kernel, a stride of 2, and padding of 1; this is for downsampling. The reason for not using the same method as upsampling is that upsampling enlarges the feature map, generating features that don't actually exist, so it can be directly enlarged. However, downsampling shrinks the feature map, losing some features. Therefore, convolution is used to fuse the features first, so that discarding some features doesn't result in significant information loss. After downsampling, the resulting feature map size is 40*40*64. This is then fused with F2 to obtain a 40*40*128 feature map. This is followed by a three-layer convolution to eliminate aliasing, resulting in another 40*40*64 feature map. A 1*1 convolution is then performed on this 40*40*64 feature map to output C2-40*40*64. Simultaneously, the 40*40*64 feature map is downsampled again to obtain a 20*20*128 feature map. This is then fused with F3 to obtain a 20*20*256 feature map. This is followed by a three-layer convolution to obtain another 20*20*128 feature map. Finally, a 1*1 convolution is performed to adjust the number of channels, resulting in C3-20*20*64. The final output consists of three feature maps: C1, C2, and C3, with sizes of 80*80*64, 40*40*64, and 20*20*64 respectively.

[0051] Preferably, the third layer of the network is a feature extraction layer. Considering that feature fusion has already been performed in the second layer, the third layer considers global feature observation and context feature fusion, using the SSH module. This module consists of two parallel parts: a 3x3 convolution and a context module. The 3x3 convolution does not perform activation, only feature extraction, outputting 32 channels. The context module consists of two serial parts. The first level is a 3x3 convolution that requires an activation function, outputting 16 channels. The second level is another parallel part. Part A is a 3x3 convolution without an activation function, only performing feature extraction, outputting 16 channels. Part B consists of two 3x3 convolutions, the first with an activation function and the second without, both outputting 16 channels. Parts A and B first perform feature fusion, resulting in an output of 32 channels, which is the output of the context module. Then, it is fused with the initial 3x3 convolution, outputting 64 channels. In the context module, the receptive field of two consecutive 3x3 convolutional kernels is equivalent to that of a 5x5 convolutional kernel, and the receptive field of three consecutive 3x3 convolutional kernels is equivalent to that of a 7x7 convolutional kernel. Avoiding the direct use of 5x5 and 7x7 kernels effectively reduces computational cost. This parallel design is a sparse network structure, yet it generates dense data, enhancing neural network performance while ensuring efficient use of computational resources. The three feature layers C1, C2, and C3 output from the second layer are each input into an SSH module, concatenating features with different channel numbers to obtain output feature maps of sizes S1(80, 80, 64), S2(40, 40, 64), and S3(20, 20, 64), respectively. These three feature maps are then activated by Leaky ReLU after SSH, effectively reducing information loss during feature extraction. An improved deformable convolution DCNV2 is added after the Leaky ReLU activation function for further refined feature extraction, effectively improving the accuracy of small face detection. Deformable convolution adds a position offset vector to the traditional convolution kernel, improving the model's ability to model deformable targets.

[0052] The ordinary convolution step involves sampling the feature map using a fixed-size receptive field R, and then weighting the sampled points using a convolution kernel W. For each location P0 in the output feature map, the output feature value y(P0) can be expressed by the formula:

[0053]

[0054] Compared to traditional convolution, deformable convolution adds a positional offset Δp to each sampling point of the convolution kernel. nThe input feature map is then subjected to another convolution to obtain a bias domain with 2N channels. The size of the bias domain is the same as that of the input feature map. The bias matrix of each pixel can be obtained from the bias domain, and the offset Δp can be obtained from the bias matrix. n After learning the offset, deformable convolution adjusts according to the current image content, allowing the kernel to fit different object shapes and extract more detailed features. Especially when processing a large number of densely packed small faces or occluded faces, traditional convolution can only extract features from a fixed-size region, inevitably introducing non-target objects and causing errors. Compared to large faces, small faces have lower resolution and a higher probability of error. Deformable convolution effectively solves this problem. The formula is expressed as:

[0055]

[0056] S3: Obtain the prediction results based on the effective feature layer, and train the network to finally achieve effective detection of faces; the prediction results include classification prediction results, bounding box regression prediction results, and facial landmark regression prediction results.

[0057] In step S2, three effective feature layers, S1, S2, and S3, are obtained. These three feature layers are used to acquire prediction results. There are three types of prediction results: classification prediction results, bounding box regression prediction results, and facial landmark regression prediction results. The classification prediction result is used to determine whether the prior bounding box contains an object. First, a 1x1 convolution is used to adjust the number of channels in the output feature to num_anchors*2, where num_anchors is 2, representing the probability of containing a face. The bounding box regression prediction result is used to adjust the prior bounding boxes to obtain the predicted bounding boxes. Since four parameters are needed to adjust the prior bounding boxes, a 1x1 convolution is used to adjust the number of channels in the output feature to num_anchors*4, representing the adjustment parameters for each prior bounding box. The facial landmark regression prediction result is used to adjust the prior bounding boxes to obtain facial landmarks. As mentioned earlier, there are five facial landmarks, and each landmark requires two adjustment parameters. Therefore, a 1x1 convolution is used to adjust the number of channels in the feature map to num_anchors*10, representing the adjustment of the facial landmarks in each prior bounding box.

[0058] The loss function of this network consists of a multi-task loss function, where λ1 and λ2 are balancing factors that balance the three losses, set to 0.25 and 0.1 respectively during training. The total loss function can be expressed as follows:

[0059]

[0060] Part One Represents the loss in face classification, among which This is the face area loss balancing factor. Considering that faces involve large, medium, and small targets, and the network is designed to detect densely packed small faces, it is desirable for the network to be more effective at detecting small targets. First, the target area *s* is normalized using a normalization function *f(s)*. Then, the normalized area value is subtracted from this value using an adjustable parameter *m*. The resulting balancing factor is negatively correlated with the predicted bounding box area; that is, the balancing factor for predicted bounding boxes of large and medium-sized faces may be less than 1, while that for small faces may be greater than 1. This allows the loss weight for small faces to increase, while the loss weight for large and medium-sized faces remains unchanged or decreases. The formula for the face area loss balancing factor can be expressed as:

[0061]

[0062]

[0063] exist In the diagram, i is the balancing loss factor for the i-th predicted box, and m... c s is an adjustable parameter i It is the area of ​​the i-th prediction box, s max It is the maximum area of ​​all sample prediction boxes.

[0064] exist In the middle, P i This indicates the probability that a face exists in the prediction box. The values ​​are the true values, with 1 for positive bounding boxes and 0 for negative bounding boxes. The classification loss uses the cross-entropy loss function, as shown in the following formula:

[0065]

[0066] Part Two Regression loss representing the face bounding box, where Formula and meaning The same, the difference is that s at this time i s represents the area of ​​the predicted positive bounding box. max This represents the maximum area within all positive sample prediction boxes. In the middle, t i It is the set of positions of the predicted bounding boxes corresponding to positive samples: This represents the location of the ground truth bounding box corresponding to the positive sample, calculated by normalizing the face coordinates. R is the Smooth L1 loss function, calculated as follows:

[0067]

[0068] Part Three Represents the keypoint regression loss, l i ={(l x1 ,l y1 ...(lx5 ,l y5 )} represents the predicted values ​​of five facial landmarks within a positive sample face bounding box. This represents the true values ​​of five facial landmarks within the positive sample face frame. R is the Smooth L1 loss function, and its calculation formula is the same as above.

[0069] The network was trained using the PyTorch deep learning framework, utilizing the training set portion of the dataset. The optimizer employed SGD optimization, with a maximum learning rate of 1e-2, an internal parameter of 0.937, a weight decay of 5e-4, and a learning rate decrease mechanism based on cosine. After 250 epochs of training, the trained network model was finally used for validation and prediction.

[0070] The network outputs three feature layers of sizes 80*80, 40*40, and 20*20, representing the original image being divided into 80*80, 40*40, and 20*20 grids. Each grid contains two prior boxes. First, it determines whether a prior box contains a face. Then, it fine-tunes the prior boxes containing faces to obtain predicted boxes and facial landmarks. Of the four parameters obtained from the prior box regression, the first two are used to adjust the center of the prior box, and the last two are used to adjust its width and height. The two parameters for the facial landmarks are used to adjust the x and y axes of the prior box center to obtain the landmark coordinates. After adjustment and judgment, non-maximum suppression is performed to filter out the boxes with the highest scores of the same category within a certain region, ultimately obtaining the predicted boxes. Figure 9 The prediction results are shown.

[0071] Furthermore, this embodiment also provides a dense small-scale face detection system based on convolutional neural networks, including: a dataset preparation and preprocessing module for creating and preprocessing image datasets containing diverse faces; a network backbone design module for constructing and optimizing the backbone structure of the convolutional neural network, based on MobilenetV3 and improved by Ghost convolution and Leaky ReLU; a feature extraction network optimization module for enhancing the system's feature extraction capability, achieved through the fusion of SPP module, FPN, and PAN; and an effective feature layer and prediction module for outputting an effective feature layer for face detection and performing face classification, bounding box regression, and key point localization prediction based on the effective feature layer.

[0072] This embodiment also provides a computer device applicable to the dense small face detection method based on convolutional neural networks, including a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the dense small face detection method based on convolutional neural networks as proposed in the above embodiment.

[0073] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0074] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the dense small face detection method based on a convolutional neural network as proposed in the above embodiments.

[0075] The storage medium proposed in this embodiment and the data storage method proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0076] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A dense small-scale face detection method based on convolutional neural networks, characterized in that: include, Collect image datasets and label face data; preprocess the image datasets to increase the number of small faces in the images; The network structure is designed based on MobilenetV3, with Ghost convolution and LeakyReLU activation functions introduced to improve the backbone network; the SPP module, FPN and PAN are combined to strengthen the first-layer feature extraction network; for the second-layer feature extraction network, the SSH module is used to observe features globally and fuse contextual features; and the effective feature layer is output. Based on the effective feature layer, the prediction results are obtained, and the network is trained to finally achieve effective face detection; the prediction results include classification prediction results, bounding box regression prediction results, and facial landmark regression prediction results. The core of MobilenetV3 is the bneck module, which includes a 1*1 convolution module, a depthwise separable convolution module, and a channel attention mechanism module added to the depthwise separable module. The improvement of the backbone network by introducing Ghost convolution and LeakyRelu activation function includes changing the 1*1 convolution in the head of the original bneck structure and the 1*1 convolution in the depth-separable module, i.e. the tail, into Ghost convolution. The ReLU activation function in the original depthwise separable convolution module was improved to the LeakyReLU activation function; The method of combining the SPP module, FPN and PAN to enhance the feature extraction network includes adding an SPP module before FPN, with two 3*3 convolutions at the beginning and end, and a fusion of feature channels with three max pooling methods: 5*5, 9*9 and 13*13 respectively in the middle. The SSH module is used to observe features globally and fuse contextual features; the output effective feature layer includes, Add an SSH module to perform context fusion. Specifically, add a LeakyRelu activation function after the SSH output module to effectively reduce information loss during feature extraction. Finally, add a deformable convolution module—DCNV1—to further refine feature extraction. The output effective feature layer includes the following steps: The SSH module obtains three effective feature layers, each of which yields three prediction results. These three effective feature layers are equivalent to dividing the entire image into grids of different sizes. When the input image is (640,640,3), the output images are (80,80,64), (40,40,64), and (20,20,64). Taking (20,20,64) as an example, it is equivalent to dividing the original image into a 20x20 grid. Each grid has two prior boxes, and each prior box represents a certain area on the image. The prediction results include face classification prediction results, bounding box regression prediction results, and face landmark prediction results; the prediction results are used to determine whether the prior bounding box contains a face, and to adjust the prior bounding box containing a face to obtain the prediction bounding box and face landmark.

2. The dense small face detection method based on convolutional neural networks as described in claim 1, characterized in that: The facial data includes blur level, expression, lighting, occlusion, and pose. The preprocessing of the image dataset specifically involves dividing the dataset into a training set, a validation set, and a test set according to a certain ratio, and randomly cropping and stitching a portion of the images to perform copypasting, with the aim of increasing the number of small faces in the images. In the training set, ground truth bounding boxes and facial landmarks are annotated for the faces. The facial landmarks include the eyes, the corners of the mouth, and the tip of the nose.

3. The dense small face detection method based on convolutional neural networks as described in claim 2, characterized in that: The training network ultimately achieves effective face detection by designing a loss function, which consists of a multi-task loss function, where λ1 and λ2 are balancing factors that balance the three losses; the total loss function L is expressed as: Part One Represents the loss in face classification, among which It is the face area loss balancing factor, P i This indicates the probability that a face exists in the prediction box. The actual value; Where i is the balance loss factor for the i-th predicted box, and m c s is an adjustable parameter i It is the area of ​​the i-th prediction box, s max It is the maximum area of ​​all sample prediction boxes; Part Two The regression loss representing the face bounding box, t i It is the set of positions of the predicted bounding boxes corresponding to positive samples. This indicates the position of the ground truth bounding box corresponding to the positive sample. R is the Smooth L1 loss function; Part Three Represents the keypoint regression loss, l i ={(l x1 ,l y1 ...(l x5 ,l y5 )} represents the predicted values ​​of five facial landmarks within a positive sample face bounding box. This represents the true values ​​of five facial landmarks within the face bounding box of a positive sample. R is the Smooth L1 loss function.

4. A dense small face detection system based on convolutional neural networks, based on the dense small face detection method based on convolutional neural networks according to any one of claims 1 to 3, characterized in that: include, The dataset preparation and preprocessing module is used to create and preprocess image datasets containing diverse faces; The network backbone design module is used to build and optimize the backbone structure of convolutional neural networks. It is based on MobilenetV3 and improved by Ghost convolution and LeakyRelu. The feature extraction network optimization module is used to enhance the system's feature extraction capabilities, which is achieved through the fusion of the SPP module, FPN, and PAN. The effective feature layer and prediction module output an effective feature layer for face detection, and perform face classification, bounding box regression and key point localization prediction based on the effective feature layer.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the dense small face detection method based on convolutional neural networks as described in any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the dense small face detection method based on convolutional neural networks as described in any one of claims 1 to 3.

Images