Certificate photograph conformity detection method based on multi-label image recognition

Abstract

The application provides a certificate photograph conformity detection method based on multi-label image recognition, a unified neural network model is constructed by connecting the image background replacement model and the certificate photograph conformity detection model in series to solve the detection of many non-conformity items of the certificate photograph, simplify the algorithm process, and improve the response speed and robustness of the system; the image background replacement model changes the to-be-detected certificate photograph image into a pure color background image, so that the subsequent conformity detection model can focus on the portrait area, and the interference of the background on the certificate photograph conformity detection system in an open scene is avoided; the method constructs a task detection branch module for each item of the non-conformity behavior of the certificate photograph, constructs a basic model based on a SwinTransformer model, performs feature extraction and information understanding on the to-be-recognized certificate photograph image with a pure color background through the basic model, and then sends the extracted image features into the task detection branch module.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a method for detecting the compliance of an ID photo based on multi-label image recognition. BACKGROUND

[0002] With the advancement of digital transformation, electronic certificates are becoming more and more popular, which puts higher requirements on the quality and standardization of ID photos. Compliance detection aims to ensure that ID photos meet certain standards and specifications, which usually cover the clarity, size, background, portrait dress, and facial features of the photo.

[0003] In the prior art, the background technology of ID photo compliance detection mainly focuses on image processing, computer vision, and artificial intelligence. Generally speaking, in the steps of ID photo compliance detection, first, through image preprocessing techniques such as cropping and normalization, the detection standards of the photo can be unified; then based on deep learning techniques such as convolutional neural networks, feature extraction is performed on the image for face recognition and analysis; through image segmentation and semantic segmentation techniques, the face and background are separated to detect non-compliant background colors or patterns; in addition, intelligent detection is needed to identify external occlusions in open scenarios, such as wearing a mask, wearing sunglasses, and wearing a hat. However, in actual use, it is found that in the existing ID photo compliance detection method, various interference item detections are executed sequentially, the process is long, and the overall execution efficiency is low, especially in high-concurrency and high-throughput application scenarios. SUMMARY

[0004] In order to solve the problem of long process and low overall execution efficiency of various interference item detections in the prior art ID photo compliance detection method, the present application provides an ID photo compliance detection method based on multi-label image recognition, which can simplify the process of ID photo compliance detection and improve the response speed and robustness of the system.

[0005] The technical solution of the present application is as follows: an ID photo compliance detection method based on multi-label image recognition, characterized in that it comprises the following steps:

[0006] S1: constructing an image background replacement model;

[0007] The image background replacement model performs background removal processing on the ID photo image to be detected, and the ID photo image to be detected is converted into a pure color background image;

[0008] S2: constructing an ID photo compliance detection model;

[0009] The ID photo compliance detection model includes a basic model and a task detection branch module;

[0010] The task detection branch module is constructed respectively for each non-compliance detection task that needs to be detected in compliance detection, and the number of the task detection branch modules is greater than 1;

[0011] The basic model is constructed based on a SwinTransformer model, and the task detection branch module includes three consecutive convolution layers with a kernel size of 3x3;

[0012] The input image is first sent into the basic model, the basic model extracts all features in the input image, and then sends all extracted image features into each of the task detection branch modules respectively; the task detection branch module extracts and identifies task features; each of the task detection branch modules outputs a detection result corresponding to a corresponding detection task;

[0013] The detection result of each branch detection model indicates whether a corresponding feature exists in the input image;

[0014] S3: training the ID photo compliance detection model to obtain a trained ID photo compliance detection model;

[0015] In the training process, the loss function used is:

[0016]

[0017] where m represents the total number of samples, q represents the total number of detection task categories, represents the true value of the jth category of the ith sample, represents the predicted value of the jth category of the ith sample; p ( i ) represents the balancing factor of the ith sample, which is defined as follows:

[0018]

[0019] where n j represents the number of samples of the jth category;

[0020] S4: performing inference on the trained ID photo compliance detection model on an image validation set, and evaluating the inference result; the evaluation index is as follows:

[0021]

[0022] where m represents the total number of samples, q represents the total number of detection task categories, represents the true value of the jth category of the ith sample, represents the predicted value of the jth category of the ith sample;

[0023] S5: The image background replacement model and the ID photo compliance detection model are connected in series, GPU cluster deployment is completed using a microservice architecture, and HTTP concurrent calls are supported;

[0024] S6: After the image background of the to-be-detected ID photo image is unified by the image background replacement model, the to-be-detected ID photo image is sent to the ID photo compliance detection model for compliance detection, and the ID photo compliance detection model outputs the non-compliant features included in the to-be-detected ID photo image.

[0025] It is further characterized in that:

[0026] In step S1, the image background replacement model is based on a lightweight HRNet18 to construct a basic model, uses an alpha channel image of the ID photo, and synthesizes a new image using a new solid color background image according to the following formula to obtain the to-be-detected solid color background image.

[0027] I = alpha * F + (1-alpha) * B;

[0028] Where I represents the synthesized new image, F represents the original ID photo image, and B represents the new solid white background image. The RGB value of each pixel of the image B is (255, 255, 255);

[0029] The training steps of the ID photo compliance detection model include image annotation, model training, model evaluation, and static graph conversion.

[0030] The image annotation corresponds to a label for each detection task that needs to be detected in compliance detection;

[0031] The static graph conversion converts the dynamic graph into a static graph model using Float32 precision;

[0032] In step S3, during the training process of the ID photo compliance detection model, a distributed training method is used to utilize multi-GPU card resources, and multi-task feature sharing is realized based on an end-to-end training framework.

[0033] In step S3, during the training process of the ID photo compliance detection model, the optimizer used is Adam, and the learning rate is set to 0.0001 according to the experimental results.

[0034] In step S3, during the training process of the ID photo compliance detection model, the dynamic graph model weight is saved once every 5 Epochs, and the total number of Epochs is 300.

[0035] This application provides a multi-label image recognition-based method for ID photo compliance detection. It constructs a unified neural network model by concatenating the image background replacement model and the ID photo compliance detection model to solve numerous non-compliance issues in ID photos, simplifying the algorithm process and improving the system's response speed and robustness. The image background replacement model transforms all ID photo images to be detected into solid-color background images, allowing the subsequent compliance detection model to focus on the portrait area, avoiding interference from the background in open scenes, and improving the system's detection accuracy and robustness. This method constructs a task detection branch module for each ID photo non-compliance behavior, builds a basic model based on the SwinTransformer model, and then... The basic model extracts features and understands information from the ID photo image to be identified against a solid-color background. The extracted image features are then fed into the task detection branch module to identify each non-compliant feature. This not only simplifies the algorithm's complexity but also compensates for the shortcomings of single-task models, effectively improving detection efficiency. Furthermore, during the training of the ID photo compliance detection model, a sample balancing factor is proposed to regulate the cross-entropy loss function of each sample, avoiding the attribute "submersion" problem caused by insufficient learning of some non-compliant items, effectively improving the model's recognition accuracy. Additionally, this method sets an evaluation index function to digitally express the model's training results, precisely controlling the training process and ensuring greater practicality. Attached Figure Description

[0036] Figure 1 This is a structural diagram of the certificate photo compliance testing model in this application;

[0037] Figure 2 This is a flowchart illustrating a method for verifying the compliance of ID photos based on multi-label image recognition. Detailed Implementation

[0038] like Figure 2 As shown, the present invention includes a method for detecting the compliance of ID photos based on multi-label image recognition, which includes the following steps.

[0039] S1: Construct an image background replacement model;

[0040] The image background replacement model removes the background from the ID photo images to be detected, transforming them all into solid-color background images. In this embodiment, the image background replacement model unifies the background image of each ID photo to white, allowing the subsequent compliance detection model to focus on the portrait area. This avoids interference from the background in open scenes, improving the system's detection accuracy and robustness.

[0041] Specifically, the image background replacement model constructs a basic model based on a lightweight HRNet18, uses an alpha channel image of a calculated ID photo, synthesizes a new image using a new solid color background image according to the following formula, and obtains the to-be-detected solid color background image.

[0042] I = alpha * F + (1-alpha) * B;

[0043] wherein I represents the synthesized new image, F represents the original ID photo image, and B represents the new solid white background image. The RGB value of each pixel of the image B is (255, 255, 255).

[0044] The matting algorithm in the method takes a lightweight HRNet18 as a basic model, is trained on a 40,000-person portrait photo dataset with pixel-level semantic segmentation labels, can finely segment the portrait foreground and background, has high segmentation accuracy and good adaptability, and is especially suitable for portrait matting tasks in open scenarios such as user self-photography through a mobile phone.

[0045] S2: constructing an ID photo compliance detection model;

[0046] The ID photo compliance detection model comprises a basic model and a task detection branch module.

[0047] The task detection branch module is respectively constructed for each item of non-compliance detection task that needs to be detected in compliance detection, and the number of the task detection branch module is greater than 1.

[0048] The basic model is constructed based on a SwinTransformer model, and the task detection branch module comprises three consecutive convolution layers with a kernel size of 3*3.

[0049] The input image is first sent into the basic model, the basic model extracts all features in the input image, and then sends all extracted image features into each task detection branch module; the task detection branch module extracts and identifies the task features; and each task detection branch module outputs a detection result corresponding to a corresponding detection task.

[0050] The detection result of each branch detection model represents whether the corresponding feature exists in the input image.

[0051] The reference model in the method adopts a SwinTransformer algorithm model pre-trained on an ImageNet dataset, so as to extract multi-task common features, and on this basis, the application constructs a task unique branch module for each label task branch, each task detection branch module adopts three convolution kernels with a size of 3*3 to extract features, and obtains task unique features. In the technical scheme of the method, the SwinTransformer algorithm model and the task detection branch module are combined, the portrait semantic information in the ID photo is fully mined, local detail features can be accurately captured, accurate detection is realized, and each non-compliance item can be fused into a unified deep neural network model for detection, so that the algorithm process is simplified and the system detection speed is improved.

[0052] The application constructs non-compliance behaviors of ID photos as a multi-task label image recognition model, can realize multi-task feature sharing in a unified end-to-end training framework, fully utilizes the advantages of multi-task image feature recognition, not only simplifies the algorithm complexity, but also makes up for the shortcomings of single-task models.

[0053] S3: training the ID photo compliance detection model to obtain a trained ID photo compliance detection model.

[0054] In the training process of the ID photo compliance detection model, a distributed training method is adopted to fully utilize the advantages of multi-GPU card resources and accelerate the training process. In the training process, the loss function used is:

[0055]

[0056] Where m represents the total number of samples, q represents the total number of classes of detection tasks, represents the true value of the jth class of the ith sample, represents the prediction value of the jth class of the ith sample; p ( i ) represents the balance factor of the ith sample, which is defined as follows:

[0057]

[0058] Where n j represents the number of samples of the jth class.

[0059] In the loss function design, the application uses a multi-label corresponding cross-entropy loss function to compare the predicted value and the true value. Considering the class imbalance problem of non-compliance samples, the application proposes a sample balance factor p ( i ) to regulate the cross-entropy loss function of each sample, which effectively avoids the attribute "submersion" problem caused by insufficient learning of part of the non-compliance items.

[0060] In the training process of the certificate photo compliance detection model, the optimizer used is Adam, the learning rate is set to 0.0001 according to the experimental effect, the dynamic graph model weight is saved once every 5 Epochs, and the total number of Epochs is 300.

[0061] The training step of the certificate photo compliance detection model includes image labeling, model training, model evaluation and static graph conversion. Through the image labeling step, the labeled image with labels is used to construct a training data set, and the constructed model is trained.

[0062] The label corresponding to the image labeling is each detection task that needs to be detected in the compliance detection. In this embodiment, the non-compliance detection tasks of the electronic certificate include 23 tasks: "wearing a hat", "wearing a mask", "wearing sunglasses", "wearing a scarf", "wearing a necklace", "wearing earrings", "head accessories", "inappropriate clothing", "inappropriate posture", "inappropriate background", "gesturing", "smoking", "wearing a uniform", "inappropriate expression", "portrait blur", "inappropriate facial light", "makeup or beauty", "graffiti", "reproduction", "hair blocking face", "head abnormality", "face abnormality", "age not meeting the standard".

[0063] In the image labeling method, the labeling tool used is Labelme, and the labeling mode is the checkbox selection mode. The non-compliant images are labeled according to the above 23 labels. One image can be labeled with multiple labels. After each image is labeled, a corresponding json annotation file is generated. After all the training images are labeled, all the annotation files are converted, and each annotation file is converted into a 0, 1 binary string to facilitate subsequent algorithm training, where 0 represents no corresponding label and 1 represents the existence of the corresponding label.

[0064] The static graph conversion converts the dynamic graph into a static graph model with Float32 precision. Considering that the technical solution of the present application is for the application scenario of high concurrency and high throughput of electronic certificate images, the GPU server is used for deployment in specific implementation. In order to maximize the consistency of the inference accuracy of dynamic and static models, Float32 precision is used for conversion.

[0065] S4: Inference of the trained certificate photo compliance detection model on the image verification set, and evaluation of the inference result. The evaluation index is as follows:

[0066]

[0067] Where m represents the total number of samples, q represents the total number of detection task categories, represents the true value of the jth category of the ith sample. This represents the predicted value of the i-th sample in the j-th category.

[0068] The evaluation metric L measures the proportion of mispredicted labels out of all labels in all samples; a smaller value indicates better model performance. The dynamic graph model with the best evaluation metric is selected from all trained dynamic graph models as the final model weights.

[0069] The total number of specific detection task categories q is set adaptively according to the detection tasks of different types of electronic ID photos. In this embodiment, the value of q is set to 23, that is, there are a total of 23 non-compliant items detection tasks for electronic ID photos in this embodiment.

[0070] This method integrates all non-compliant types of ID photos into a single multi-label image recognition model for detection. Therefore, each image only needs to be labeled once, eliminating the need for repeated labeling across multiple sub-tasks, significantly reducing labeling costs. For the 23 non-compliant ID photo categories, the entire detection process only includes image matting and multi-label recognition, resulting in low algorithm complexity, fast detection speed, and easy deployment as it only requires cascading two models. Furthermore, by employing a unified multi-label recognition model to detect the 23 non-compliant categories, the correlation between multiple tasks can be fully explored during model training, enabling feature sharing and improving detection accuracy.

[0071] S5: Connect the image background replacement model and the ID photo compliance detection model in series, and use a microservice architecture to complete the GPU cluster deployment, supporting concurrent HTTP calls.

[0072] S6: As Figure 2 As shown, after the background of the ID photo to be tested is unified by the image background replacement model, it is then sent to the ID photo compliance detection model for compliance detection. The ID photo compliance detection model outputs the non-compliant features included in the ID photo image to be tested.

[0073] By employing the technical solution of this invention, a multi-label image recognition framework is used to construct multi-task labels for non-compliant items in ID photos. Multi-task feature sharing is achieved within a unified end-to-end neural network model, thereby fully leveraging the advantages of multi-task image feature recognition to quickly and accurately analyze non-compliant ID photo situations. This technical solution can provide technical support for the automated management of electronic certificates and has significant value and importance in practical applications.

Claims

1. A method for detecting the compliance of a passport photo based on multi-label image recognition, characterized in that, It comprises the following steps: S1: constructing an image background replacement model; The image background replacement model performs background removal processing on the to-be-detected ID photo image, and all to-be-detected ID photo images become pure color background images; S2: constructing an ID photo compliance detection model; The ID photo compliance detection model comprises a basic model and a task detection branch module; The task detection branch module is constructed for each item of non-compliance detection task that needs to be detected in compliance detection, and the number of the task detection branch module is greater than 1; The basic model is constructed based on a SwinTransformer model, and the task detection branch module comprises three consecutive convolution layers with a kernel size of 3x3; The input image is first sent to the basic model, the basic model extracts all features in the input image, and then sends all extracted image features to each task detection branch module; the task detection branch module extracts and identifies task features; each task detection branch module outputs a detection result corresponding to the corresponding detection task; The detection result of each branch detection model indicates whether the corresponding feature exists in the input image; S3: training the ID photo compliance detection model to obtain a trained ID photo compliance detection model; In the training process, the loss function used is: where m represents the total number of samples, q represents the total number of classes of the detection task, represents the true value of the jth class of the ith sample, represents the predicted value of the jth class of the ith sample; p ( i ) represents the balancing factor of the ith sample, which is defined as follows: where n j represents the number of samples of the jth category; S4: performing inference on the trained ID photo compliance detection model on an image validation set, and evaluating the inference result; the evaluation index is as follows: wherein m represents the total number of samples, q represents the total number of classes of the detection task, represents the true value of the jth class of the ith sample, represents the predicted value of the jth class of the ith sample; S5: connecting the image background replacement model and the ID photo compliance detection model in series, and deploying a GPU cluster using a micro-service architecture to support HTTP concurrent calls; S6: sending the to-be-detected ID photo image to the image background replacement model to unify the image background, and then sending it to the ID photo compliance detection model for compliance detection, and the ID photo compliance detection model outputs the non-compliant features included in the to-be-detected ID photo image.

2. The method of claim 1, wherein the method further comprises: In step S1, the image background replacement model constructs a basic model based on a lightweight HRNet18, uses an alpha channel image of the certificate, and synthesizes a new image using a new pure color background image according to the following formula to obtain the to-be-detected pure color background image; I = alpha * F + (1-alpha) * B; Where I represents the synthesized new image, F represents the original ID photo image, B represents the new pure white background image, and the RGB value of each pixel of the image B is (255, 255, 255).

3. The method of claim 1, wherein the method further comprises: determining a face region in the image; and determining a face region in the image. The training steps of the ID photo compliance detection model include image labeling, model training, model evaluation, and static graph conversion; The label corresponding to the image labeling is each detection task that needs to be detected in compliance detection; The static graph conversion converts the dynamic graph into a static graph model with Float32 precision.

4. The method of claim 1, wherein the method further comprises: In step S3, during the training process of the ID photo compliance detection model, a distributed training method is used to utilize multiple GPU resources based on an end-to-end training framework to realize multi-task feature sharing.

5. The method of claim 1, wherein the method further comprises: In step S3, during the training of the ID photo conformity detection model, the optimizer used is Adam, and the learning rate is set to 0.0001 according to the experimental effect.

6. The method of claim 1, wherein the method further comprises: In step S3, during the training of the ID photo conformity detection model, the dynamic graph model weight is saved once every 5 Epochs, and the total number of Epochs is 300.

Images