Method, electronic device and medium for determining face occlusion score

Abstract

The present invention relates to computer vision and specifically provides a method, electronic device, and medium for determining a face occlusion score, aiming to address the problem that existing methods cannot accurately reflect the degree of face occlusion. To this end, the present invention inputs a face image into a trained face segmentation model to obtain a segmentation of at least a portion of the face area, and a segmentation of an occluded area within at least a portion of the face area, or a segmentation of a non-occluded area within at least a portion of the face area; obtains the number of pixels of the at least portion of the face area segmentation, and the number of pixels of the occluded area segmentation within at least a portion of the face area, or the number of pixels of the non-occluded area segmentation within at least a portion of the face area; and determines the face occlusion score of the face image based on the number of pixels of the at least portion of the face area segmentation, and the number of pixels of the occluded area segmentation within at least a portion of the face area, or the number of pixels of the non-occluded area segmentation within at least a portion of the face area.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and specifically provides a method for determining face occlusion score, an electronic device, and a medium. Background Art

[0002] In face recognition systems, face quality assessment is an essential preprocessing step, designed to filter out poor-quality facial data before recognition to improve recognition accuracy. Occlusion, as a factor affecting face quality, significantly impacts face recognition, necessitating an accurate estimation. However, the occlusion score is not an intuitive objective, and a specific occlusion score cannot be directly assigned (for example, the face occlusion score ranges from 0 to 1, where 0 represents no occlusion and 1 represents complete occlusion). Instead, it is often necessary to construct the occlusion score using other methods.

[0003] Existing face occlusion score estimation technologies use classification methods to identify whether a face is occluded; others partition the face frame area, estimate the occlusion level of each sub-area, and then weight them to derive the occlusion score for the entire face. However, the construction methods of these face occlusion score estimation technologies do not accurately reflect the degree of face occlusion (for example, the confidence level in the classification method only reflects the confidence level of occlusion; the face frame area includes non-face areas, and the occlusion level of this part should not be used as a factor in determining whether the face is occluded, so partitioning also has an impact). Therefore, the occlusion score estimation based on this method will further incur errors.

[0004] Accordingly, a new method for determining face occlusion score is needed in this field to solve the above problem. Summary of the Invention

[0005] The present invention aims to solve the above technical problem, namely, to solve the problem that the existing face occlusion point construction method cannot accurately reflect the occlusion degree of the face, thereby reducing the user experience.

[0006] To achieve the above objectives, in a first aspect, the present invention provides a method for determining a face occlusion score, the method comprising the following steps:

[0007] Acquire a facial image, and input the facial image into a trained facial segmentation model to obtain segmentation of at least a portion of the facial region, and segmentation of an occluded region within at least a portion of the facial region or segmentation of a non-occluded region within at least a portion of the facial region;

[0008] respectively obtaining the number of pixels of the at least partial face region segmentation, and the number of pixels of the occluded region segmentation within the at least partial face region or the number of pixels of the non-occluded region segmentation within the at least partial face region;

[0009] The face occlusion score of the face image is determined based on the number of pixels of the at least partial face area segmentation, the number of pixels of the occluded area segmentation within the at least partial face area, or the number of pixels of the non-occluded area segmentation within the at least partial face area.

[0010] In an optional technical solution of the above-mentioned method for determining a face occlusion score, the step of “determining the face occlusion score of the face image based on the number of pixels of the at least partial face region segmentation and the number of pixels of the occluded region segmentation within the at least partial face region or the number of pixels of the non-occluded region segmentation within the at least partial face region” includes:

[0011] Obtaining a first ratio of the number of pixels of the occluded region segmented within the at least partial face region to the number of pixels of the at least partial face region segmented, and using the first ratio as a face occlusion score of the face image;

[0012] Alternatively, a second ratio of the number of pixels in the non-occluded area segmented within the at least partial face area to the number of pixels in the at least partial face area segmented is obtained, and the second ratio is used as the face occlusion score of the face image.

[0013] In an optional technical solution of the above-mentioned method for determining the face occlusion score, the method trains the model based on at least the following steps:

[0014] Obtaining a first face training image, annotating the first face training image, and using the annotated first face training image as a training sample for the face segmentation model; constructing a neural network model, and using the neural network model as the face segmentation model to be trained, wherein the face segmentation model to be trained includes a backbone network to be trained, a neck network to be trained, and a head network to be trained;

[0015] The training samples are input into the face segmentation model to be trained to obtain a trained face segmentation model.

[0016] In an optional technical solution of the above-mentioned method for determining a face occlusion score, the first face training image includes a face detection frame area and a non-face detection frame area, and the step of “labeling the first face training image” includes:

[0017] Based on the first face training image, at least a portion of the face region segmentation, and at least a portion of the occluded region segmentation within the face region or at least a portion of the non-occluded region segmentation within the face region are marked.

[0018] In the optional technical solution of the above-mentioned method for determining the face occlusion score, the step of "marking at least a portion of the face region segmentation, and at least a portion of the face region occluded region segmentation or at least a portion of the face region non-occluded region segmentation" includes:

[0019] Based on the first face training image, obtaining a second face training image, wherein the second face training image is an image that only includes the face detection frame area in the first face training image;

[0020] Based on the second face training image and the preset annotation area requirement, at least part of the face area segmentation, and at least part of the occluded area segmentation within the face area or at least part of the non-occluded area segmentation within the face area are labeled, wherein the preset annotation area requirement includes labeling the entire face area or labeling part of the face area.

[0021] In the optional technical solution of the above-mentioned method for determining the face occlusion score, the step of “inputting the training sample into the face segmentation model to be trained to obtain a trained face segmentation model” includes:

[0022] S1. Inputting the training sample into the backbone network to be trained to perform feature extraction to obtain a first feature map of the training sample;

[0023] S2. Inputting the first feature map of the training sample into the neck network to be trained for feature fusion to obtain a second feature map of the training sample;

[0024] S3. Inputting the second feature map of the training sample into the head network to be trained to perform region segmentation to obtain a region segmentation result of the training sample, wherein the region segmentation result is at a pixel level or a grid level;

[0025] S4. Obtain a loss function based on the region segmentation result of the training sample and the training sample, and feed the loss function back to step S1, and loop through steps S1 to S4 until the loss function converges, wherein the loss function at least includes a cross entropy loss function.

[0026] In an optional technical solution of the above-mentioned method for determining the face occlusion score, the head network to be trained includes a head network for segmenting the face region to be trained, a head network for segmenting the occluded region within the face region to be trained, or a head network for segmenting the non-occluded region within the face region to be trained, and the step of “inputting the second feature map of the training sample into the head network to be trained for region segmentation to obtain a region segmentation result of the training sample” includes:

[0027] Inputting the second feature map of the to-be-trained sample into the to-be-trained face region segmentation head network to obtain at least a partial face region segmentation;

[0028] Inputting the second feature map of the to-be-trained sample into the head network for segmenting the occluded region in the to-be-trained face region to obtain the occluded region segmentation in at least part of the face region;

[0029] Alternatively, the second feature map of the sample to be trained is input into a head network for segmenting the non-occluded region in the face region to be trained, so as to obtain the non-occluded region segmentation in at least part of the face region.

[0030] In an optional technical solution of the above-mentioned method for determining the face occlusion score, the method further includes:

[0031] Constructing the backbone network to be trained based on at least one of ResNet, MobileNet, and HRNet;

[0032] The neck network to be trained is constructed based on at least one of FPN, PANet, and Bi-FPN.

[0033] In a second aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the computer program, the method for determining the face occlusion score as described in any one of the above items is implemented.

[0034] In a third aspect, the present invention further provides a readable storage medium, wherein a plurality of program codes are stored in the readable storage medium, and the program codes are suitable for being loaded and run by a processor to execute any one of the above-mentioned methods for determining face occlusion scores.

[0035] Those skilled in the art will understand that, in the technical solution of the present invention, by obtaining a face image and inputting the face image into a trained face segmentation model, at least a portion of the face region segmentation, and at least a portion of the face region occlusion region segmentation or at least a portion of the face region non-occlusion region segmentation are obtained; the number of pixels of the at least portion of the face region segmentation, and the number of pixels of the at least portion of the face region occlusion region segmentation or at least a portion of the face region non-occlusion region segmentation are obtained respectively; based on the number of pixels of the at least portion of the face region segmentation, and the number of pixels of the at least portion of the face region occlusion region segmentation or at least a portion of the face region non-occlusion region segmentation, the face occlusion score of the face image is determined. Such a setting can more accurately reflect the degree of occlusion of the face, thereby further improving the accuracy of face recognition and enhancing the user experience.

[0036] Furthermore, the method further includes: constructing a backbone network to be trained based on at least one of ResNet, MobileNet, and HRNet; and constructing a neck network to be trained based on at least one of FPN, PANet, and Bi-FPN. This setup allows the face segmentation model to be built based on actual user needs, thereby better balancing accuracy and time consumption, further enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The disclosure of the present invention will be more easily understood with reference to the accompanying drawings. Those skilled in the art will readily appreciate that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of the present invention. Furthermore, similar numbers in the drawings represent similar components, wherein:

[0038] Figure 1 1 is a flow chart showing the main steps of a method for determining a face occlusion score according to an embodiment of the present invention;

[0039] Figure 2 1 is a flow chart of the main steps of training a face segmentation model according to an embodiment of the present invention;

[0040] Figure 3 is a schematic diagram of an image with at least a portion of a face region segmented and marked according to an embodiment of the present invention;

[0041] Figure 4 is a schematic diagram of an image showing the segmentation of an occluded region within at least a portion of a face region according to an embodiment of the present invention;

[0042] Figure 5 This is a flow chart of the main steps of inputting training samples into a face segmentation model to be trained for training according to one embodiment of the present invention;

[0043] Figure 6 It is a schematic diagram of the main structural block diagram of an electronic device for executing the face occlusion score determination method of the present invention. DETAILED DESCRIPTION

[0044] Some embodiments of the present invention are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0045] In the description of the present invention, "module" and "processor" may include hardware, software, or a combination of the two. A module may include hardware circuits, various suitable sensors, communication ports, and memory. It may also include software components, such as program code, or a combination of software and hardware. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware, or a combination of the two. Non-transitory computer-readable storage media include any suitable medium that can store program code, such as a magnetic disk, hard disk, optical disk, flash memory, read-only memory, random access memory, etc. The terms "at least one of A or B" or "at least one of A and B" have similar meanings to "A and / or B" and may include only A, only B, or A and B. The singular forms "one" and "the" may also include plural forms.

[0046] As described in the background technology section, in order to address the problem that the existing face occlusion score construction method cannot accurately reflect the degree of face occlusion, thereby reducing the user experience, the present invention provides a face occlusion score determination method.

[0047] See attached Figure 1 , Figure 1 FIG. 1 is a flow chart showing the main steps of a method for determining face occlusion score according to an embodiment of the present invention. Figure 1 As shown, the present invention also provides a method for determining face occlusion score, the method comprising the following steps:

[0048] Step S101: Acquire a face image and input the face image into a trained face segmentation model to obtain at least a partial face region segmentation, at least a partial face region occluded region segmentation, or at least a partial face region unoccluded region segmentation.

[0049] Specifically, the facial area in the usual sense includes the forehead, eyebrows, eyelids, corners of the eyes, eye sockets, nose bridge, nose wings, nose tip, nasolabial grooves, cheeks, lips, maxillary and mandibular areas, and in face recognition technology, recognition can be based on all of the above facial areas or on part of the above facial areas. Therefore, a trained face segmentation model can be used to obtain at least a partial facial area segmentation, at least a partial facial area segmentation, or at least a partial facial area non-occluded area segmentation. For example, considering that the forehead area above the eyebrows basically does not affect face recognition, the at least partial facial area can be a facial area that does not include the forehead and only includes the facial area below the eyebrows. The above-mentioned setting method of at least a partial facial area is only for illustrative purposes and can be selected according to actual needs in actual applications.

[0050] Step S102: respectively obtaining the number of pixels of at least a portion of the face region segmented, the number of pixels of the occluded region segmented within at least a portion of the face region, or the number of pixels of the non-occluded region segmented within at least a portion of the face region.

[0051] Step S103: Determine the face occlusion score of the face image based on the number of pixels of the at least partial face area segmentation, the number of pixels of the occluded area segmentation within at least the partial face area, or the number of pixels of the non-occluded area segmentation within at least the partial face area.

[0052] Based on the above steps S101 to S103, the present invention obtains a facial image and inputs the facial image into a trained facial segmentation model to obtain at least a partial facial region segmentation, as well as an occluded region segmentation within at least a partial facial region or a non-occluded region segmentation within at least a partial facial region; obtains the number of pixels of the at least partial facial region segmentation, as well as the number of pixels of the occluded region segmentation within at least a partial facial region or the number of pixels of the non-occluded region segmentation within at least a partial facial region; and determines the facial occlusion score of the facial image based on the number of pixels of the at least partial facial region segmentation, as well as the number of pixels of the occluded region segmentation within at least a partial facial region or the number of pixels of the non-occluded region segmentation within at least a partial facial region. Such a setting can more accurately reflect the degree of occlusion of the face, thereby further improving the accuracy of facial recognition and enhancing the user experience.

[0053] In some embodiments, determining the face occlusion score of the facial image based on the number of pixels in the at least partial face region segmentation, the number of pixels in the occluded region segmentation within at least the partial face region, or the number of pixels in the non-occluded region segmentation within at least the partial face region comprises the following steps:

[0054] Step S1031: Obtain a first ratio of the number of pixels in the occluded region segmented within at least a portion of the face region to the number of pixels in the at least portion of the face region segmented, and use the first ratio as a face occlusion score of the face image.

[0055] Step S1032: Alternatively, a second ratio of the number of pixels in the non-occluded area segmented within at least a portion of the face area to the number of pixels in the at least a portion of the face area segmented is obtained, and the second ratio is used as the face occlusion score of the face image.

[0056] That is, the first ratio is obtained through the first ratio = (the number of pixels in the occluded area segmented in at least part of the face area) / (the number of pixels in the occluded area segmented in at least part of the face area), and the first ratio is used as the face occlusion score of the face image; or, the second ratio is obtained through the second ratio = (the number of pixels in the non-occluded area segmented in at least part of the face area) / (the number of pixels in the occluded area segmented in at least part of the face area), and the second ratio is used as the face occlusion score of the face image.

[0057] See attached Figure 2 , Figure 2 FIG. 1 is a flow chart showing the main steps of training a face segmentation model according to an embodiment of the present invention. Figure 2 As shown, the method trains the model based on at least the following steps:

[0058] Step S201: obtaining a first face training image, annotating the first face training image, and using the annotated first face training image as a training sample for a face segmentation model.

[0059] In some embodiments, the first face training image includes a face detection frame area and a non-face detection frame area, and labeling the first face training image includes: based on the first face training image, labeling at least part of the face area segmentation, and at least part of the occluded area segmentation within the face area or at least part of the non-occluded area segmentation within the face area.

[0060] Specifically, at least a portion of the face area refers to a portion or all of the area belonging to the face in the first face training image, which can be represented by a set of points at the boundary of the area or a set of points within the area; the occluded area within at least a portion of the face area refers to an area that is occluded within at least a portion of the face area, which can be represented by a set of points at the boundary of the area or a set of points within the area; the unoccluded area within at least a portion of the face area refers to an area that is not occluded within at least a portion of the face area, which can be represented by a set of points at the boundary of the area or a set of points within the area. For example, when performing region segmentation on the first face training image, it can be performed at the pixel level, that is, each pixel in the first face training image is divided into its own region. The above-mentioned region segmentation setting method is only for illustrative purposes and can be selected according to actual needs in actual applications.

[0061] In some embodiments, labeling at least a portion of the face area segmentation, and the occluded area segmentation within at least a portion of the face area or the non-occluded area segmentation within at least a portion of the face area includes: based on the first face training image, obtaining a second face training image, wherein the second face training image is an image that only contains the face detection frame area in the first face training image; based on the second face training image and the preset labeling area requirements, labeling at least a portion of the face area segmentation, the occluded area segmentation within at least a portion of the face area or the non-occluded area segmentation within at least a portion of the face area, wherein the preset labeling area requirements include labeling the entire face area or labeling part of the face area.

[0062] Specifically, the first face training image is an image that includes a face detection frame region and a non-face detection frame region. The non-face detection frame region can be, for example, the neck region, the shoulder region, etc. of the human body. Therefore, in order to more accurately annotate the first face training image, a second face training image that only includes the face detection frame region of the first face training image can be first obtained, and then annotation can be performed based on the second face training image. During annotation, the at least partial face region refers to the partial or entire region of the second face training image that belongs to the face and can be represented by a set of points at the boundary of the region or a set of points within the region; the occluded region within the at least partial face region refers to the region that is occluded within the at least partial face region and can be represented by a set of points at the boundary of the region or a set of points within the region; and the unoccluded region within the at least partial face region refers to the region that is not occluded within the at least partial face region and can be represented by a set of points at the boundary of the region or a set of points within the region. If there is no occluded region within the at least partial face region, the occluded region within the at least partial face region is segmented into an empty set.

[0063] For example, the face detection frame area in the first face training image can be represented by the coordinates of the upper left corner (x1, y1) and the lower right corner (x2, y2) of the face detection frame area. That is, the upper left corner of the first face training image is set as the coordinate origin, the positive direction of the X axis is set to the horizontal right, and the positive direction of the Y axis is set to the vertical downward. Then, the upper left corner coordinates of the face detection frame area under the coordinate axis are (x1, y1), and the lower right corner coordinates are (x2, y2). The above-mentioned representation method of the face detection frame area is only for illustrative purposes. In actual applications, it can be selected according to actual needs.

[0064] In some embodiments, at least a portion of the face area can be customized, that is, it can include the entire face area, or it can include only a portion of the face area, thereby marking the entire face area or marking the portion of the face area. For example, considering that the forehead area above the eyebrows basically does not affect face recognition, the at least part of the face area can be defined as not including the forehead and only including the face area below the eyebrows. According to this definition of at least a portion of the face area, the segmentation of the at least part of the face area based on the annotation of the second face training image can be as shown in 3, and the segmentation of the occluded area within the annotated at least part of the face area can be as shown in Figure 4 The above-mentioned definition method of at least a portion of the face region, the segmented image of the labeled at least a portion of the face region, and the segmented image of the occluded region within the labeled at least a portion of the face region are only for illustrative purposes and can be selected according to actual needs in actual applications.

[0065] Step S202: constructing a neural network model, and using the neural network model as a face segmentation model to be trained, wherein the face segmentation model to be trained includes a backbone network to be trained, a neck network to be trained, and a head network to be trained.

[0066] Step S203: inputting the training samples into the face segmentation model to be trained to obtain a trained face segmentation model.

[0067] S203 is further described below.

[0068] See attached Figure 5 , Figure 5 FIG. 1 is a flow chart showing the main steps of inputting training samples into a face segmentation model to be trained for training according to an embodiment of the present invention. Figure 5 As shown, in some embodiments, inputting the training samples into the face segmentation model to be trained for training to obtain a trained face segmentation model includes the following steps:

[0069] Step S2031: input the training sample into the backbone network to be trained for feature extraction to obtain a first feature map of the training sample.

[0070] Step S2032: inputting the first feature map of the training sample into the neck network to be trained for feature fusion to obtain the second feature map of the training sample.

[0071] Step S2033: Input the second feature map of the training sample into the head network to be trained for region segmentation to obtain a region segmentation result of the training sample, wherein the region segmentation result is at the pixel level or the grid level.

[0072] Step S2034: Obtain a loss function based on the region segmentation result of the training sample and the training sample, and feed the loss function back to step S2031, and loop through steps S2031 to S2034 until the loss function converges, wherein the loss function at least includes a cross entropy loss function.

[0073] Specifically, feature extraction is the foundation of computer vision tasks. A good feature extraction network can significantly improve the performance of the algorithm. In computer vision tasks, the network that extracts features from images is called the backbone network. The receptive field refers to the size of the area on the original image where the pixels on the output feature map are mapped. In previous target detection, detection was performed on the very rough feature maps output by the backbone network. The very rough feature maps output by the backbone network have a large receptive field, which is very friendly for large objects. However, for small objects, an overly large receptive field can easily lead to "loss of focus". To avoid this problem, it is necessary to fuse the feature maps of multiple scales from the bottom up before detection, that is, the neck network needs to fuse the first feature map extracted by the backbone network. After the neck network completes the fusion of the first feature map, the head network can perform detection and positioning based on the features fused by the neck network, that is, to achieve regional segmentation of the training samples.

[0074] Based on the structural settings of the backbone network to be trained, the neck network to be trained, and the head network to be trained, the region segmentation results can be at the pixel level or the grid level. Pixel-level region segmentation is to segment each pixel in the training sample into its own region; grid-level region segmentation is to divide each training sample into multiple grids, and then segment each grid into its own region. The accuracy of pixel-level region segmentation is greater than that of grid-level region segmentation, but the time consumption of pixel-level region segmentation is also longer than that of grid-level region segmentation. Therefore, in actual use, the level of region segmentation to be adopted can be comprehensively considered based on accuracy and time consumption.

[0075] The loss function adjusts the weight of the face segmentation model to be trained by calculating the gap between the region segmentation results of each iteration of the face segmentation model to be trained and the training samples. The calculation formula of the cross entropy loss function is:

[0076]

[0077] Among them, N is the number of samples selected for one training, M is the number of categories, and y ic is the true category of sample i, if the true category of sample i is c, it takes 1, otherwise it takes 0; p ic is the predicted probability that sample i belongs to category c.

[0078] In some embodiments, the head network to be trained includes a head network for segmenting a face region to be trained, a head network for segmenting an occluded region within a face region to be trained, or a head network for segmenting a non-occluded region within a face region to be trained. Inputting the second feature map of the training sample into the head network to be trained for region segmentation to obtain a region segmentation result of the training sample includes the following steps:

[0079] Step S20331: inputting the second feature map of the sample to be trained into the head network for face region segmentation to be trained to obtain at least a partial face region segmentation.

[0080] Step S20332: inputting the second feature map of the sample to be trained into the head network for segmenting the occluded region in the face region to be trained, so as to obtain the segmentation of the occluded region in at least part of the face region.

[0081] Step S20333: Alternatively, the second feature map of the sample to be trained is input into the head network for segmenting the non-occluded area in the face area to be trained, so as to obtain the non-occluded area segmentation in at least part of the face area.

[0082] In some embodiments, the method further includes: constructing a backbone network to be trained based on at least one of ResNet, MobileNet, and HRNet; and constructing a neck network to be trained based on at least one of FPN, PANet, and Bi-FPN.

[0083] Specifically, ResNet (Residual Network) was proposed by Kaiming He and others from Microsoft Labs in 2015. Its main features are its ultra-deep network structure, the proposed residual structure module, and the use of Batch Normalization to accelerate training. MobileNet is based on a streamlined architecture and uses depthwise separable convolution to build a lightweight deep neural network for mobile and embedded vision applications. The network introduces two simple global hyperparameters - width multiplier and resolution multiplier - to effectively balance latency and accuracy. HRNet (High-Resolution Net) was proposed for the task of 2D human pose estimation, and the network is mainly aimed at the pose evaluation of a single individual (i.e., there should be only one human target in the image input to the network).

[0084] FPNs (Feature Pyramid Networks) consist of two pathways: bottom-up and top-down. The bottom-up pathway is a typical convolutional network for feature extraction. From bottom to top, spatial resolution decreases, detecting more high-level structures, and the semantic value of the network layer increases accordingly. PANets (Path Aggregation Networks) use bottom-up path enhancement to enhance the entire feature hierarchy using accurate low-level localization signals, thereby shortening the information path between low-level and top-level features. They also propose adaptive feature pooling, which connects the feature grid and all feature layers, allowing useful information in each feature layer to propagate directly to the underlying proposal subnetwork. Bi-FPNs (bi-directional feature pyramid networks) introduce learnable weights to learn the importance of different input features, while repeatedly applying multi-scale feature fusion from top to bottom and bottom to top.

[0085] The backbone network to be trained includes one of ResNet, MobileNet, and HRNet, and the neck network to be trained includes one of FPN, PANet, and Bi-FPN. This setup allows the face segmentation model to be built based on actual user needs, thereby better balancing accuracy and time consumption, further improving the user experience.

[0086] In some embodiments, the face image and the training sample have the same size. Specifically, the size of the image can be represented by (H, W), where H is the height of the image and W is the width of the image. The face image and the training sample are represented by the same (H, W) size.

[0087] It should be noted that the user images involved in the embodiments of the present disclosure (including but not limited to the first face training image and the second face training image used for training, face images in actual environments, etc.) are all images authorized by the user or fully authorized by all parties.

[0088] The actions of acquiring images involved in the embodiments of the present disclosure are all performed with the authorization of the user or the object or after full authorization by all parties.

[0089] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effects of the present invention, different steps do not have to be performed in such an order. They can be performed simultaneously (in parallel) or in other orders. These changes are within the scope of protection of the present invention.

[0090] Furthermore, the present invention also provides an electronic device.

[0091] See attached Figure 6 , Figure 6 FIG. 1 is a schematic diagram of the main structure of an electronic device for executing the face occlusion score determination method of the present invention. Figure 6 As shown, the present invention further provides an electronic device for executing the face occlusion score determination method of the present invention. The electronic device includes: a processor 11, a memory 12, and a computer program 13 stored in the memory 12 and executable on the processor 11. When the processor 11 executes the computer program 13, the steps of the above-described method embodiments are implemented. Alternatively, when the processor 11 executes the computer program 13, the functions of the modules / units in the above-described embodiments are implemented.

[0092] Illustratively, the processor 11 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0093] Exemplarily, the memory 12 can be an internal storage unit of the electronic device, such as a hard disk or memory of the electronic device. The memory 12 can also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash memory card, etc. equipped on the electronic device. Furthermore, the memory 12 can also include both an internal storage unit of the electronic device and an external storage device. The memory 12 is used to store computer programs and other programs and data required by the electronic device. The memory 12 can also be used to temporarily store data that has been output or is about to be output.

[0094] In some possible implementations, the electronic device may include multiple processors 11 and memories 12. The program for executing the face occlusion score determination method of the above-mentioned method embodiment can be divided into multiple subroutines, and each subroutine can be loaded and run by the processor 11 to execute different steps of the face occlusion score determination method of the above-mentioned method embodiment. Specifically, each subroutine can be stored in a different memory 12 respectively, and each processor 11 can be configured to execute the program in one or more memories 12 to jointly implement the face occlusion score determination method of the above-mentioned method embodiment, that is, each processor 11 executes different steps of the face occlusion score determination method of the above-mentioned method embodiment respectively to jointly implement the face occlusion score determination method of the above-mentioned method embodiment.

[0095] The multiple processors 11 may be processors deployed on the same device. For example, the electronic device may be a high-performance device composed of multiple processors, and the multiple processors 11 may be processors configured on the high-performance device. Furthermore, the multiple processors 11 may be processors deployed on different devices. For example, the electronic device may be a server cluster, and the multiple processors 11 may be processors on different servers in the server cluster.

[0096] The electronic device may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device may include but is not limited to a processor 11 and a memory 12. Those skilled in the art will understand that Figure 6 These are merely examples of electronic devices and do not constitute a limitation of the electronic device. The electronic device may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0097] Furthermore, the present invention also provides a computer-readable storage medium. In one embodiment of a computer-readable storage medium according to the present invention, the computer-readable storage medium can be configured to store a program for executing the face occlusion score determination method of the above-mentioned method embodiment. The program can be loaded and executed by a processor to implement the above-mentioned face occlusion score determination method. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method section of the embodiment of the present invention. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present invention is a non-transitory computer-readable storage medium.

[0098] Furthermore, it should be understood that since the configuration of each module is merely for the purpose of illustrating the functional units of the apparatus of the present invention, the physical devices corresponding to these modules may be the processor itself, or a portion of the software in the processor, a portion of the hardware, or a combination of software and hardware. Therefore, the number of modules in the figure is merely illustrative.

[0099] Those skilled in the art will appreciate that the various modules in the device can be adaptively split or merged. Such splitting or merging of specific modules does not cause the technical solution to deviate from the principles of the present invention. Therefore, the technical solutions after splitting or merging will fall within the scope of protection of the present invention.

[0100] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A method for determining face occlusion score, characterized in that: The method comprises the following steps: Acquire a facial image, and input the facial image into a trained facial segmentation model to obtain segmentation of at least a portion of the facial region, and segmentation of an occluded region within at least a portion of the facial region or segmentation of a non-occluded region within at least a portion of the facial region; respectively obtaining the number of pixels of the at least partial face region segmentation, and the number of pixels of the occluded region segmentation within the at least partial face region or the number of pixels of the non-occluded region segmentation within the at least partial face region; determining a face occlusion score for the facial image based on the number of pixels of the at least partial facial region segmentation and the number of pixels of the occluded region segmentation within the at least partial facial region or the number of pixels of the non-occluded region segmentation within the at least partial facial region; The method trains the model based on at least the following steps: Acquiring a first face training image, annotating the first face training image, and using the annotated first face training image as a training sample for the face segmentation model; Constructing a neural network model and using the neural network model as a face segmentation model to be trained, wherein the face segmentation model to be trained includes a backbone network to be trained, a neck network to be trained, and a head network to be trained, and the head network to be trained includes a head network for face region segmentation to be trained, a head network for occluded region segmentation within the face region to be trained, or a head network for non-occluded region segmentation within the face region to be trained, wherein the head network for face region segmentation to be trained is used to obtain at least a partial face region segmentation, the head network for occluded region segmentation within the face region to be trained is used to obtain at least a partial face region segmentation, and the head network for non-occluded region segmentation within the face region to be trained is used to obtain at least a partial face region segmentation; The training samples are input into the face segmentation model to be trained to obtain a trained face segmentation model.

2. The method for determining face occlusion score according to claim 1, wherein: The step of “determining a face occlusion score of the face image based on the number of pixels of the at least partial face region segmentation and the number of pixels of the occluded region segmentation within the at least partial face region or the number of pixels of the non-occluded region segmentation within the at least partial face region” includes: Obtaining a first ratio of the number of pixels of the occluded region segmented within the at least partial face region to the number of pixels of the at least partial face region segmented, and using the first ratio as a face occlusion score of the face image; Alternatively, a second ratio of the number of pixels in the non-occluded area segmented within the at least partial face area to the number of pixels in the at least partial face area segmented is obtained, and the second ratio is used as the face occlusion score of the face image.

3. The method for determining face occlusion score according to claim 1, wherein: The first face training image includes a face detection frame area and a non-face detection frame area, and the step of "labeling the first face training image" includes: Based on the first face training image, at least a portion of the face region segmentation, and at least a portion of the occluded region segmentation within the face region or at least a portion of the non-occluded region segmentation within the face region are marked.

4. The method for determining face occlusion score according to claim 3, wherein: The step of “marking at least a portion of the face region segmentation, and at least a portion of the face region occluded region segmentation or at least a portion of the face region non-occluded region segmentation” includes: Based on the first face training image, obtaining a second face training image, wherein the second face training image is an image that only includes the face detection frame area in the first face training image; Based on the second face training image and the preset annotation area requirement, at least part of the face area segmentation, and at least part of the occluded area segmentation within the face area or at least part of the non-occluded area segmentation within the face area are labeled, wherein the preset annotation area requirement includes labeling the entire face area or labeling part of the face area.

5. The method for determining face occlusion score according to claim 1, wherein: The step of “inputting the training samples into the face segmentation model to be trained to obtain a trained face segmentation model” includes: S1. Inputting the training sample into the backbone network to be trained to perform feature extraction to obtain a first feature map of the training sample; S2. Inputting the first feature map of the training sample into the neck network to be trained for feature fusion to obtain a second feature map of the training sample; S3. Inputting the second feature map of the training sample into the head network to be trained to perform region segmentation to obtain a region segmentation result of the training sample, wherein the region segmentation result is at a pixel level or a grid level; S4. Obtain a loss function based on the region segmentation result of the training sample and the training sample, and feed the loss function back to step S1, and loop through steps S1 to S4 until the loss function converges, wherein the loss function at least includes a cross entropy loss function.

6. The method for determining face occlusion score according to claim 5, wherein: The step of “inputting the second feature map of the training sample into the head network to be trained for region segmentation to obtain a region segmentation result of the training sample” includes: Inputting the second feature map of the to-be-trained sample into the to-be-trained face region segmentation head network to obtain at least a partial face region segmentation; Inputting the second feature map of the to-be-trained sample into the head network for segmenting the occluded region in the to-be-trained face region to obtain the occluded region segmentation in at least part of the face region; Alternatively, the second feature map of the sample to be trained is input into a head network for segmenting the non-occluded region in the face region to be trained, so as to obtain the non-occluded region segmentation in at least part of the face region.

7. The method for determining face occlusion score according to claim 1, wherein: The method further comprises: Constructing the backbone network to be trained based on at least one of ResNet, MobileNet, and HRNet; The neck network to be trained is constructed based on at least one of FPN, PANet, and Bi-FPN.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for determining a face occlusion score according to any one of claims 1 to 7 is implemented.

9. A readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the face occlusion score determination method according to any one of claims 1 to 7.

Images