Identification photo compliance detection method based on multi-label image recognition

Abstract

According to the identification photo compliance detection method based on multi-label image recognition, the image background replacement model and the identification photo compliance detection model are connected in series, a unified neural network model is constructed to solve the problem of detection of numerous non-compliance items of the identification photo, the algorithm process is simplified, and the detection efficiency is improved. The response speed and robustness of the system are improved; according to the method, identification photo images to be detected are all changed into pure-color background images through an image background replacement model, so that a subsequent compliance detection model can focus on a portrait area, and interference of a background in an open scene on an identification photo compliance detection system is avoided; according to the method, a task detection branch module is constructed for each identification photo non-compliance behavior, a basic model is constructed based on a SwinTransform model, feature extraction and information understanding are performed on a to-be-identified identification photo image with a pure color background through the basic model, and then extracted image features are respectively sent to the task detection branch modules.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for detecting compliance of ID photos based on multi-label image recognition. Background Art

[0002] With the advancement of digital transformation, electronic ID cards are becoming more and more popular, which also puts higher requirements on the quality and standardization of ID photos. Compliance testing aims to ensure that ID photos meet specific standards and specifications, which usually cover the clarity, size, background, portrait clothing, and facial features of the photo.

[0003] In the prior art, the background technology of ID photo compliance detection is mainly concentrated in the fields of image processing, computer vision and artificial intelligence. Generally speaking, the steps of ID photo compliance detection include: first, through image preprocessing technology, such as cropping and normalization, the detection standard of the photo can be unified; then based on deep learning technology, such as convolutional neural network, the image is feature extracted and then face recognition and analysis are performed; then through image segmentation technology and semantic segmentation technology, the face and background are separated, and non-compliant background colors or patterns are detected; in addition, it is necessary to identify external occlusion interference in open scenes through intelligent detection, such as: wearing masks, sunglasses, hats, etc. However, in actual use, it is found that in the ID photo compliance detection method in the prior art, various interference item detections are executed sequentially, and the process is relatively long, especially when used in application scenarios with high concurrency and high throughput, the overall execution efficiency is low. Summary of the invention

[0004] In order to solve the problem that various interference item detections in the ID photo compliance detection method in the prior art are performed sequentially, the process is long and the overall execution efficiency is low, the present invention provides an ID photo compliance detection method based on multi-label image recognition, which can simplify the ID photo compliance detection process and improve the response speed and robustness of the system.

[0005] The technical solution of the present invention is as follows: a method for detecting compliance of ID photos based on multi-label image recognition, characterized in that it comprises the following steps:

[0006] S1: Build image background replacement model;

[0007] The image background replacement model performs background removal processing on the ID photo image to be detected, and converts the ID photo image to a pure color background image;

[0008] S2: Build a compliance detection model for ID photos;

[0009] The ID photo compliance detection model includes: a basic model and a task detection branch module;

[0010] The task detection branch module is constructed for each non-compliance detection task that needs to be detected in the compliance detection, and the number of the task detection branch modules is greater than 1;

[0011] The basic model is built based on the SwinTransformer model, and the task detection branch module includes 3 consecutive convolutional layers with a kernel size of 3×3;

[0012] The input image is first sent to the basic model. After the basic model extracts all the features in the input image, all the extracted image features are sent to each of the task detection branch modules respectively; the task detection branch modules extract and identify task features; each of the task detection branch modules outputs the detection result corresponding to its corresponding detection task;

[0013] The detection result of each branch detection model indicates whether a corresponding feature exists in the input image;

[0014] S3: training the ID photo compliance detection model to obtain a trained ID photo compliance detection model;

[0015] During training, the loss function used is:

[0016]

[0017] Where m represents the total number of samples, q represents the total number of categories of the detection task, represents the true value of the jth category of the i-th sample, represents the predicted value of the jth category of the i-th sample; p ( i ) represents the equalization factor of the i-th sample, which is defined as follows:

[0018]

[0019] where n j represents the number of samples in the jth category;

[0020] S4: The trained ID photo compliance detection model is inferred on the image verification set, and the inference results are evaluated; the evaluation indicators are as follows:

[0021]

[0022] Where m represents the total number of samples, q represents the total number of categories of the detection task, represents the true value of the jth category of the i-th sample, Represents the predicted value of the jth category of the i-th sample;

[0023] S5: Connect the image background replacement model and the ID photo compliance detection model in series, use a microservice architecture to complete GPU cluster deployment, and support HTTP concurrent calls;

[0024] S6: After the ID photo image to be detected is sent to the image background replacement model to unify the image background, it is then sent to the ID photo compliance detection model for compliance detection. The ID photo compliance detection model outputs non-compliant features included in the ID photo image to be detected.

[0025] It is further characterized by:

[0026] In step S1, the image background replacement model builds a basic model based on the lightweight HRNet18, uses the alpha channel image of the calculated ID photo, and synthesizes a new image using the new pure color background image according to the following formula to obtain the pure color background image to be detected;

[0027] I = alpha × F + (1-alpha) × B;

[0028] Where I represents the synthesized new image, F represents the original ID photo image, and B represents the new pure white background image. The RGB value of each pixel in image B is (255, 255, 255);

[0029] The training steps of the ID photo compliance detection model include: image annotation, model training, model evaluation and static image conversion;

[0030] The label corresponding to the image annotation is each detection task that needs to be detected in the compliance detection;

[0031] The static image conversion uses Float32 precision to convert the dynamic image into a static image model;

[0032] In step S3, during the training of the ID photo compliance detection model, a distributed training method is adopted using multi-GPU graphics card resources, and multi-task feature sharing is achieved based on an end-to-end training framework;

[0033] In step S3, during the training of the ID photo compliance detection model, the optimizer used is Adam, and the learning rate is set to 0.0001 according to the experimental results;

[0034] In step S3, during the training of the ID photo compliance detection model, the dynamic graph model weight is saved once every 5 Epochs, and the total number of Epoch training is 300.

[0035] The ID photo compliance detection method based on multi-label image recognition provided by the present application is based on connecting the image background replacement model and the ID photo compliance detection model in series to construct a unified neural network model to solve the detection of many non-compliance items in ID photos, simplify the algorithm process, and improve the response speed and robustness of the system; it uses the image background replacement model to change the ID photo images to be detected into pure color background images, so that the subsequent compliance detection model can focus on the portrait area, avoiding the interference of the background on the ID photo compliance detection system in open scenes, and can improve the detection accuracy and robustness of the system; this method constructs a task detection branch module for each non-compliant behavior of the ID photo, constructs a basic model based on the SwinTransformer model, and The basic model extracts features and understands information of the ID photo image to be identified with a solid color background, and then sends the extracted image features to the task detection branch module respectively, and identifies each non-compliant feature respectively, which not only simplifies the complexity of the algorithm, but also makes up for the shortcomings of the single-task model and effectively improves the detection efficiency; at the same time, in the process of training the ID photo compliance detection model, a sample balancing factor is proposed on the basis of the original loss function to regulate the cross entropy loss function of each sample, avoiding the attribute "flooding" problem caused by insufficient learning of some non-compliant items, and effectively improving the recognition accuracy of the model; at the same time, this method also sets an evaluation index function, which can digitally express the training results of the model, accurately control the training process of the model, and ensure that this method is more practical. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is the structure diagram of the ID photo compliance detection model in this application;

[0037] Figure 2 The figure is a flowchart of a method for detecting compliance of ID photos based on multi-label image recognition. DETAILED DESCRIPTION

[0038] like Figure 2 As shown, the present invention includes a certificate photo compliance detection method based on multi-label image recognition, which includes the following steps.

[0039] S1: Build image background replacement model;

[0040] The image background replacement model performs background removal processing on the ID photo images to be detected, and changes the ID photo images to be detected into pure color background images. In this embodiment, the background image of each ID photo is unified to white through the image background replacement model, so that the subsequent compliance detection model can focus on the portrait area, avoiding the interference of the background on the ID photo compliance detection system in an open scene, and improving the detection accuracy and robustness of the system.

[0041] Specifically, the image background replacement model builds a basic model based on the lightweight HRNet18, uses the alpha channel image of the calculated ID photo, and synthesizes a new image using a new pure color background image according to the following formula to obtain the pure color background image to be detected;

[0042] I=alpha×F+(1-alpha)×B ；

[0043] Where I represents the synthesized new image, F represents the original ID photo image, and B represents the new pure white background image. The RGB value of each pixel in image B is (255, 255, 255) 。

[0044] The cutout algorithm in this method is based on the lightweight HRNet18 model and is trained on a dataset of 40,000 portrait photos with pixel-level semantic segmentation labels. It can finely segment the foreground and background of portraits with high segmentation accuracy and good adaptability. It is especially suitable for portrait cutout tasks in open scenarios such as users taking selfies with their mobile phones.

[0045] S2: Build a compliance detection model for ID photos;

[0046] The ID photo compliance detection model includes: a basic model and a task detection branch module;

[0047] The task detection branch module is constructed for each non-compliance detection task that needs to be detected in the compliance detection, and the number of the task detection branch modules is greater than 1;

[0048] The basic model is built based on the SwinTransformer model, and the task detection branch module includes 3 consecutive convolutional layers with a kernel size of 3×3;

[0049] The input image is first sent to the basic model. After the basic model extracts all the features in the input image, all the extracted image features are sent to each of the task detection branch modules respectively; the task detection branch modules extract and identify task features; each of the task detection branch modules outputs the detection result corresponding to its corresponding detection task;

[0050] The detection result of each branch detection model indicates whether a corresponding feature exists in the input image.

[0051] The benchmark model in this method uses the SwinTransformer algorithm model pre-trained on the ImageNet dataset to extract multi-task common features. On this basis, the present invention constructs a task-unique branch module for each label task branch. Each task detection branch module uses 3 convolutions with a kernel size of 3×3 for feature extraction to obtain task-unique features. In the technical solution of this method, the SwinTransformer algorithm model and the task detection branch module are combined to fully mine the semantic information of the portrait in the ID photo, accurately capture local detail features, and achieve accurate detection. In addition, each non-compliant item can be integrated into a unified deep neural network model for detection, which simplifies the algorithm process and improves the system detection speed.

[0052] This application constructs the non-compliant behavior of ID photos into a multi-task label image recognition model, which can realize multi-task feature sharing in a unified end-to-end training framework, fully utilizing the advantages of multi-task image feature recognition, which not only simplifies the algorithm complexity, but also makes up for the shortcomings of the single-task model.

[0053] S3: training the ID photo compliance detection model to obtain a trained ID photo compliance detection model.

[0054] In the process of training the ID photo compliance detection model, a distributed training method is adopted to fully utilize the advantages of multi-GPU graphics card resources to accelerate the training process. During the training process, the loss function used is:

[0055]

[0056] Where m represents the total number of samples, q represents the total number of categories of the detection task, represents the true value of the jth category of the i-th sample, represents the predicted value of the jth category of the i-th sample; p ( i ) represents the equalization factor of the i-th sample, which is defined as follows:

[0057]

[0058] Where n j Represents the number of samples in the jth category.

[0059] In the loss function design, this method uses the cross entropy loss function corresponding to multiple labels to compare the predicted value and the true value. Considering the imbalance problem of non-compliant samples, this paper proposes a sample balance factor p based on the original loss function. ( i )To regulate the cross entropy loss function of each sample, it effectively avoids the attribute "flooding" problem caused by insufficient learning of some non-compliant items.

[0060] During the training of the ID photo compliance detection model, the optimizer used was Adam, and the learning rate was set to 0.0001 according to the experimental results; the dynamic graph model weight was saved once every 5 Epochs, and the total number of Epoch training was 300.

[0061] The training steps of the ID photo compliance detection model include: image annotation, model training, model evaluation and static image conversion. Through the image annotation step, the labeled images are used to build a training data set to train the built model.

[0062] The label corresponding to the image annotation is each detection task that needs to be detected in the compliance detection. In this embodiment, the non-compliance detection tasks of the electronic certificate include 23 items: "wearing a hat", "wearing a mask", "wearing sunglasses", "wearing a scarf", "wearing a necklace", "wearing earrings", "head accessories", "irregular clothing", "irregular posture", "irregular background", "gesturing", "smoking", "wearing a uniform", "irregular expression", "blurred portrait", "irregular facial lighting", "makeup or beauty", "graffiti", "copying", "hair covering the face", "abnormal head", "abnormal face", "age does not meet the standard".

[0063] In the image annotation method, the annotation tool used is Labelme, and the annotation mode is the checkbox selection mode. The non-compliant images are annotated according to the above 23 labels. One image can be annotated with multiple labels at the same time. After each image is annotated, a corresponding json annotation file is generated. After all training images are annotated, all annotation files are converted, and each annotation file is converted into a 0, 1 binary string to facilitate subsequent algorithm training, where 0 indicates no corresponding label and 1 indicates the existence of a corresponding label.

[0064] The static image conversion uses Float32 precision to convert the dynamic image into a static image model. Considering that the technical solution of this application is aimed at high-concurrency and high-throughput electronic certificate image application scenarios, GPU servers are used for deployment during specific implementation. In order to maximize the consistency of dynamic and static model reasoning accuracy, Float32 precision is used for conversion.

[0065] S4: The trained ID photo compliance detection model is inferred on the image verification set, and the inference results are evaluated; the evaluation indicators are as follows:

[0066]

[0067] Where m represents the total number of samples, q represents the total number of categories of the detection task, represents the true value of the jth category of the i-th sample, Represents the predicted value of the jth category of the i-th sample.

[0068] The evaluation index L measures the proportion of the number of mispredicted labels in the total number of labels in all samples. The smaller the index value, the better the performance of the model. The dynamic graph model with the best evaluation index is selected from all the dynamic graph models trained as the final model weight.

[0069] The value of the total number of categories q of specific detection tasks is adaptively set according to the detection tasks of different types of electronic ID photos. In this embodiment, the value of q is set to 23, that is, there are 23 non-compliant item detection tasks for electronic ID photos in this embodiment.

[0070] This method integrates all types of non-compliance in ID photos into a multi-label image recognition model for detection. Therefore, one image only needs to be annotated once, and there is no need to repeatedly annotate the image according to multiple subtasks, which greatly reduces the annotation cost. For the 23 types of non-compliance items in ID photos, the entire detection process only includes cutouts and multi-label recognition. The algorithm complexity is low, the detection speed is fast, and only two models need to be connected in series to complete the deployment, which is easy to deploy. Moreover, since a unified multi-label recognition model is used to detect 23 non-compliance items, the correlation between multiple tasks can be fully explored during the model training process, feature sharing can be achieved, and detection accuracy can be improved.

[0071] S5: The image background replacement model and the ID photo compliance detection model are connected in series, and a microservice architecture is used to complete GPU cluster deployment, supporting HTTP concurrent calls.

[0072] S6: Figure 2 As shown, the ID photo image to be detected is sent to the image background replacement model to unify the image background, and then sent to the ID photo compliance detection model for compliance detection. The ID photo compliance detection model outputs the non-compliant features included in the ID photo image to be detected.

[0073] After using the technical solution of the present invention, a multi-label image recognition framework is adopted to construct the non-compliant items of the ID photo as multi-task labels, and multi-task feature sharing is realized in a unified end-to-end neural network model, so that the advantages of multi-task image feature recognition can be fully utilized to quickly and accurately analyze the non-compliant situations of the ID photo. The technical solution of this application can provide technical support for the automated management of electronic certificates and has important value and significance in practical applications.

Claims

1. A method for detecting compliance of ID photos based on multi-label image recognition, characterized in that: It includes the following steps: S1: Build image background replacement model; The image background replacement model performs background removal processing on the ID photo image to be detected, and converts the ID photo image to a pure color background image; S2: Build a compliance detection model for ID photos; The ID photo compliance detection model includes: a basic model and a task detection branch module; The task detection branch module is constructed for each non-compliance detection task that needs to be detected in the compliance detection, and the number of the task detection branch modules is greater than 1; The basic model is built based on the SwinTransformer model, and the task detection branch module includes 3 consecutive convolutional layers with a kernel size of 3×3; The input image is first sent to the basic model. After the basic model extracts all the features in the input image, all the extracted image features are sent to each of the task detection branch modules respectively; the task detection branch modules extract and identify task features; each of the task detection branch modules outputs the detection result corresponding to its corresponding detection task; The detection result of each branch detection model indicates whether a corresponding feature exists in the input image; S3: training the ID photo compliance detection model to obtain a trained ID photo compliance detection model; During training, the loss function used is: Where m represents the total number of samples, q represents the total number of categories of the detection task, represents the true value of the jth category of the i-th sample, represents the predicted value of the jth category of the i-th sample; p ( i) represents the equalization factor of the i-th sample, which is defined as follows: Where n j Represents the number of samples in the jth category; S4: The trained ID photo compliance detection model is inferred on the image verification set, and the inference results are evaluated; the evaluation indicators are as follows: Where m represents the total number of samples, q represents the total number of categories of the detection task, represents the true value of the jth category of the i-th sample, Represents the predicted value of the jth category of the i-th sample; S5: Connect the image background replacement model and the ID photo compliance detection model in series, use a microservice architecture to complete GPU cluster deployment, and support HTTP concurrent calls; S6: After the ID photo image to be detected is sent to the image background replacement model to unify the image background, it is then sent to the ID photo compliance detection model for compliance detection. The ID photo compliance detection model outputs non-compliant features included in the ID photo image to be detected.

2. The ID photo compliance detection method based on multi-label image recognition according to claim 1 is characterized by: In step S1, the image background replacement model builds a basic model based on the lightweight HRNet18, uses the alpha channel image of the calculated ID photo, and synthesizes a new image using the new pure color background image according to the following formula to obtain the pure color background image to be detected; I=alpha×F+(1-alpha)×B ； Where I represents the synthesized new image, F represents the original ID photo image, and B represents the new pure white background image. The RGB value of each pixel in image B is (255, 255, 255) 。 3. The ID photo compliance detection method based on multi-label image recognition according to claim 1 is characterized by: The training steps of the ID photo compliance detection model include: image annotation, model training, model evaluation and static image conversion; The label corresponding to the image annotation is each detection task that needs to be detected in the compliance detection; The static image conversion uses Float32 precision to convert the dynamic image into a static image model.

4. The ID photo compliance detection method based on multi-label image recognition according to claim 1 is characterized by: In step S3, during the training of the ID photo compliance detection model, a distributed training method is adopted using multi-GPU graphics card resources, and multi-task feature sharing is achieved based on an end-to-end training framework.

5. The ID photo compliance detection method based on multi-label image recognition according to claim 1 is characterized by: In step S3, during the training of the ID photo compliance detection model, the optimizer used is Adam, and the learning rate is set to 0.0001 according to the experimental results.

6. The ID photo compliance detection method based on multi-label image recognition according to claim 1 is characterized by: In step S3, during the training of the ID photo compliance detection model, the dynamic graph model weight is saved once every 5 Epochs, and the total number of Epoch training is 300.

Images