A method and device for detecting defects in document images

Abstract

The present application relates to the field of image processing technologies, and provides a method and device for detecting defects in document images. The document images are collected from at least one angle under multiple light sources; according to the defect types, a method matching the defect types is adopted to detect the defect states of the document images, thereby improving the accuracy of defect detection in document images.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method and device for detecting defects in document images. Background Art

[0002] Currently, quality inspection personnel generally detect defects in documents. For small defects, the documents need to be placed at different angles and carefully observed to be discovered, resulting in low detection efficiency. As the working time increases, it is easy to get fatigued, leading to low accuracy. Traditional vision methods can only identify a fixed number of types of defects and cannot accurately identify defects with unfixed positions, defect sizes, color spots, and film scratches. Summary of the Invention

[0003] An embodiment of this application provides a method and device for detecting defects in document images to improve the accuracy of detecting defects in document images.

[0004] In a first aspect, an embodiment of this application provides a method for detecting defects in document images, including:

[0005] Collecting a document image from at least one angle under multiple light sources;

[0006] Detecting the defect status of the document image according to the defect type and using a method matching the defect type.

[0007] The above embodiment of this application collects a document image from at least one angle under multiple light sources and detects the defects of the document image using a method matching the defect type according to the defect type, which can improve the accuracy of document detection. Moreover, the detection methods corresponding to each defect type do not affect each other and can run in parallel, improving the defect detection efficiency.

[0008] In some embodiments of this application, the defect type includes anti-counterfeiting defects. Detecting the defect status of the document image using a method matching the defect type includes:

[0009] Determining the anti-counterfeiting area image of the document image collected under the first light source;

[0010] If the anti-counterfeiting area image of the document image is inconsistent with the standard anti-counterfeiting image corresponding to the first light source, it is determined that the document image has anti-counterfeiting defects.

[0011] The above embodiment of this application compares the anti-counterfeiting area image of the collected document image with the standard anti-counterfeiting image under the corresponding light source, and determines whether the document image has anti-counterfeiting defects according to the comparison result, realizing the anti-counterfeiting detection of the document image.

[0012] In some embodiments of the present application, the first light source includes an ultraviolet (UV) light source, and an anti-counterfeiting area image of the document image collected under the first light source is determined; if the anti-counterfeiting area image of the document image is inconsistent with the standard anti-counterfeiting image corresponding to the first light source, it is determined that there is an anti-counterfeiting defect in the document image, including:

[0013] Determine the UV anti-counterfeiting area image of the document image collected at at least one angle under the UV light source; if the UV anti-counterfeiting area image of the document image is inconsistent with the standard UV anti-counterfeiting image, it is determined that there is a UV anti-counterfeiting defect in the document image collected under the UV light source; and / or

[0014] The first light source includes an infrared (IR) light source, and an anti-counterfeiting area image of the document image collected under the first light source is determined; if the anti-counterfeiting area image of the document image is inconsistent with the standard anti-counterfeiting image corresponding to the first light source, it is determined that there is an anti-counterfeiting defect in the document image, including:

[0015] Determine the IR anti-counterfeiting area image of the document image collected from a specified direction under the IR light source; if the IR anti-counterfeiting area image of the document image is inconsistent with the standard IR anti-counterfeiting image, it is determined that there is an IR anti-counterfeiting defect in the document image collected under the IR light source.

[0016] In the above embodiments of the present application, when the light source includes a UV light source, the document images collected at at least one angle under the UV light source are compared and analyzed with the standard UV anti-counterfeiting images under the UV light source, and whether there is a UV anti-counterfeiting defect in the collected document images is detected according to the comparison and analysis results; when the light source includes an IR light source, the document images collected at at least one angle under the IR light source are compared and analyzed with the standard IR anti-counterfeiting images under the IR light source, and whether there is an IR anti-counterfeiting defect in the collected document images is detected according to the comparison and analysis results, so as to realize the anti-counterfeiting detection of the document images.

[0017] In some embodiments of the present application, the defect type includes an etching defect, and the light source includes a common light source;

[0018] A method matching the defect type is used to detect the defect state of the document image, including:

[0019] Intercept the etching area image of the document image collected at at least one angle under the common light source;

[0020] If the etching information extracted from the etching area images corresponding to each angle is inconsistent with the etching information of the standard laser image, it is determined that there is an etching defect in the document image collected under the common light source.

[0021] In the above embodiments of the present application, the etching information of the document images collected at at least one angle under the common light source is compared with the etching information of the standard laser image, and whether there is an etching defect in the document images is detected according to the comparison results, so as to realize the etching detection of the document images.

[0022] In a second aspect, an embodiment of the present application provides a method for detecting defects in a document image, including:

[0023] Collecting a document image from at least one angle under a normal light source;

[0024] Preprocessing the document image to obtain a document image with the background removed;

[0025] Extracting personal information from the document image with the background removed, and generating a standard document image based on the blank document template according to the personal information;

[0026] If the defect degree between the document image to be detected and the standard document image is greater than a set threshold, it is determined that the document image to be detected has a content missing defect.

[0027] The above embodiment of the present application generates a standard document image from the personal information extracted from the collected document image, analyzes and compares it with the document image to be detected, and detects the defects existing in the document image to be detected, improving the detection accuracy.

[0028] In a third aspect, an embodiment of the present application provides a method for detecting defects in a document image, including:

[0029] Preprocessing the document image to be detected to remove noise in the document image to be detected;

[0030] According to the document image to be detected and the standard document image, obtaining a difference image by using the method of morphological subtraction;

[0031] Replacing the black and white pixels in the difference image to obtain a replaced difference image;

[0032] If the contour difference between the difference images before and after replacement is greater than a set threshold, it is determined that the document image to be detected has a black edge defect.

[0033] The above embodiment of the present application obtains a difference image by using the method of morphological subtraction according to the document image to be detected and the standard document image; replaces the black and white pixels in the difference image to obtain a replaced difference image, and realizes the detection of black edge defects according to the comparison result between the contour differences of the difference images before and after replacement and the set threshold.

[0034] In a fourth aspect, an embodiment of the present application provides a method for detecting defects in a document image, including:

[0035] Collecting a document image from at least one angle under a normal light source;

[0036] Preprocessing the document image to obtain a document image with the background removed;

[0037] Extract the portrait outlines of the background-removed ID images in the CMYK color components respectively;

[0038] If the deviation of the portrait outlines in every two color components is greater than the set threshold, it is determined that the ID image collected under normal light source has a ghosting defect.

[0039] The above embodiments of the present application realize the detection of the ghosting defect of the ID image by comparing the deviation of the portrait outlines in the four CMYK color components with the threshold.

[0040] Fifthly, an ID image defect detection method provided by an embodiment of the present application includes:

[0041] Collect the ID image to be detected under normal light source;

[0042] Input the ID image to be detected and the standard ID image into the trained feature extraction network;

[0043] Based on at least one convolutional layer of the trained feature extraction network, extract the image features of the ID image to be detected and the image features of the standard ID image;

[0044] Based on at least one fully connected layer of the trained feature extraction network, perform feature fusion on the extracted image features respectively;

[0045] Based on the trained defect detection network, determine the defect type of the ID image to be detected according to the image features of the ID image to be detected after fusion and the image features of the standard ID image after fusion.

[0046] The above embodiments of the present application use deep learning methods to train the defect detection network, and through the trained defect detection network, realize defect types that cannot be detected by traditional vision algorithms.

[0047] In some embodiments of the present application, the defect detection network is trained according to the following method:

[0048] Obtain multiple ID image training samples;

[0049] Input multiple ID defect training samples and the pre-annotated defect category labels corresponding to each ID image training sample into the initial defect detection network; through the initial defect detection network, perform fusion processing on the image features of the ID image training samples, and obtain the probabilities corresponding to each ID image training sample for indicating that the ID image training samples belong to each preset defect category respectively;

[0050] Determine the detection loss value according to the probability corresponding to each ID image training sample and the pre-annotated defect category label;

[0051] Adjust the model parameters of the initial defect detection network according to the detection loss value until the determined detection loss value is within a preset range, and obtain the trained defect detection network.

[0052] In some embodiments of the present application, the method further includes:

[0053] Use the certificate image training samples of the new defect type and the certificate image training samples of the misdetected ones as new training samples, label the corresponding defect category labels, and input them into the trained defect detection network;

[0054] Retrain the trained defect detection network to obtain an updated defect detection network.

[0055] According to the above embodiments of the present application, the defect detection network is updated according to the certificate image training samples of the new defect type and the certificate image training samples of the misdetected ones, improving the defect detection accuracy.

[0056] In a sixth aspect, an embodiment of the present application provides a certificate image defect detection device, including:

[0057] An acquisition module, configured to acquire certificate images from at least one angle under multiple light sources;

[0058] A defect detection module, configured to determine the defect type to be detected and detect the defect status of the certificate image by using a method matching the defect type.

[0059] In a seventh aspect, an embodiment of the present application provides a certificate image defect detection device, including:

[0060] An acquisition module, configured to acquire certificate images from at least one angle under normal light sources;

[0061] A preprocessing module, configured to preprocess the certificate image to obtain a certificate image with the background removed;

[0062] An image generation module, configured to extract personal information from the certificate image with the background removed, and generate a standard certificate image based on the blank certificate template according to the personal information;

[0063] A defect detection module, configured to determine that there is a content missing defect in the certificate image to be detected if the defect degree between the certificate image to be detected and the standard certificate image is greater than a set threshold.

[0064] In an eighth aspect, an embodiment of the present application provides a certificate image defect detection device, including:

[0065] A preprocessing module, configured to preprocess the certificate image to be detected and remove the noise in the certificate image to be detected;

[0066] A pixel replacement module, configured to obtain a difference image by using a morphological subtraction method based on a document image to be detected and a standard document image; replace black and white pixels in the difference image to obtain a replaced difference image.

[0067] A defect detection module, configured to determine that there is a black edge defect in the document image to be detected if the contour difference between the difference images before and after replacement is greater than a set threshold.

[0068] In a ninth aspect, an embodiment of the present application provides a document image defect detection device, including:

[0069] An acquisition module, configured to acquire a document image from at least one angle under a normal light source;

[0070] A preprocessing module, configured to preprocess the document image to obtain a document image with the background removed;

[0071] A defect detection module, configured to extract the portrait contours of the document image with the background removed under the CMYK color components respectively; if the deviation between the portrait contours under every two color components is greater than a set threshold, determine that there is a ghosting defect in the document image acquired under the normal light source.

[0072] In a tenth aspect, an embodiment of the present application provides a document image defect detection method, including:

[0073] An acquisition module, configured to acquire a document image to be detected under a normal light source;

[0074] A feature extraction module, configured to extract the image features of the document image to be detected and the image features of the standard document image based on the convolutional layer of a trained feature extraction network; perform feature fusion on the extracted image features respectively based on the fully connected layer of the trained feature extraction network;

[0075] A defect detection module, configured to determine the defect type of the document image to be detected based on the fused image features of the document image to be detected and the fused image features of the standard document image, based on a trained defect detection network. Description of the Drawings

[0076] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0077] Figure 1 Exemplarily shows a schematic structural diagram of an image acquisition device provided by an embodiment of the present application;

[0078] Figure 2 Exemplarily shows the flowchart of the document image defect detection method provided by the embodiments of the present application;

[0079] Figure 3 Exemplarily shows the flowchart of the document image defect detection method provided by the embodiments of the present application;

[0080] Figure 4 Exemplarily shows the flowchart of the document image defect detection method provided by the embodiments of the present application;

[0081] Figure 5 Exemplarily shows the flowchart of the document image defect detection method provided by the embodiments of the present application;

[0082] Figure 6 Exemplarily shows the flowchart of the document image defect detection method provided by the embodiments of the present application;

[0083] Figure 7 Exemplarily shows the complete flowchart of the document image defect detection method provided by the embodiments of the present application;

[0084] Figure 8 Exemplarily shows the structural diagram of the document image defect detection device provided by the embodiments of the present application;

[0085] Figure 9 Exemplarily shows the structural diagram of the document image defect detection device provided by the embodiments of the present application;

[0086] Figure 10 Exemplarily shows the structural diagram of the document image defect detection device provided by the embodiments of the present application;

[0087] Figure 11 Exemplarily shows the structural diagram of the document image defect detection device provided by the embodiments of the present application;

[0088] Figure 12 Exemplarily shows the structural diagram of the document image defect detection device provided by the embodiments of the present application;

[0089] Figure 13 Exemplarily shows the structural diagram of the document image defect detection device provided by the embodiments of the present application. Detailed implementation manners

[0090] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the protection scope of the present application.

[0091] Based on the exemplary embodiments shown in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present application. In addition, although the disclosure in the present application is introduced according to one or several exemplary instances, it should be understood that each aspect of these disclosures can also constitute a complete technical solution alone.

[0092] It should be understood that the terms "first", "second", etc. in the specification, claims and the above drawings of the present application are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, for example, it can be implemented in an order other than those given in the illustration or description of the embodiments of the present application.

[0093] In addition, the terms "comprising" and "having" and any variations thereof are intended to cover but not exclude inclusion. For example, a product or device comprising a series of components does not necessarily have to be limited to those components clearly listed, but may include other components not clearly listed or inherent to these products or devices.

[0094] The term "module" used in the present application refers to any known or later-developed hardware, software, firmware, artificial intelligence, fuzzy logic, or a combination of hardware or / and software code that can perform functions related to that element.

[0095] In fields such as insurance, securities, finance, and entry and exit, users are required to provide images of documents (such as ID cards, work permits, electronic passes, etc.) to verify their identities. Since the review of document images is mainly carried out manually, and the image quality varies, it will waste a huge amount of human and material resources, with a high cost, and the accuracy of manual detection is relatively low, unable to meet the business requirements. Moreover, during the document production process, it takes a long time for manual quality inspection of the produced documents. When quality problems are found in the documents, a large number of documents have often been produced, resulting in batch invalidation. If the documents can be automatically detected during the document production process, the yield can be increased. However, the traditional machine vision method has a relatively low detection accuracy for low-quality document images, and can only identify several fixed types of defects, and cannot accurately identify defects with non-fixed positions, defect sizes, color spots, and film scratches.

[0096] To solve the above problems, the embodiments of the present application provide a method and device for detecting defects in document images. By collecting document images from multiple angles under multiple light sources and using methods matching the defect types to detect various defects, the detection accuracy is improved; and a method combining deep learning and machine vision is used to detect defects on the document surface, reducing the labor cost and improving the detection efficiency; also, a standard document image is generated from the personal information extracted from the collected document images and analyzed and compared with the document image to be detected to detect the defects existing in the document image to be detected, improving the detection accuracy; by comparing the contour deviation of the portrait under the four color components of CMYK with a threshold, the detection of ghosting defects in the document image is realized.

[0097] It should be noted that the embodiments of the present application can be applied to electronic devices such as terminal devices, computer systems, servers, etc., and can also operate together with other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, clients, handheld devices, personal notebooks, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, and so on.

[0098] Terminal devices, computer systems, servers and other electronic devices can be described in the general context of computer system-executable instructions (such as program modules) executed by a computer system. Generally, program modules may include routines, programs, target programs, components, logics, data structures, etc., which perform specific tasks or implement specific abstract data types. The computer system / server can be implemented in a distributed cloud computing environment and executed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0099] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0100] Figure 1 Exemplarily shown is a schematic structural diagram of an image acquisition device provided by the embodiments of the present application. As Figure 1As shown, three cameras (101_1 to 101_3) are used to collect images of the document 104 to be detected from at least one angle under three kinds of light sources (102_1 to 102_3). The camera 101_1 is placed at the upper left - 45° direction of the card - holding device 103, the camera 101_2 is placed at the directly - above 0° direction of the card - holding device 103, and the camera 101_3 is placed at the upper right 45° direction of the card - holding device 103. The document image collected by the camera 101_2 contains rich portrait and text information, and the document images collected by the cameras 101_1 and 101_3 can be used for detecting anti - counterfeiting defects. The document 104 to be detected by the camera is placed above the card - holding device 103. The card - holding device 103 can rotate horizontally by - 90° to 90° and vertically by - 90° to 90°, so that the three cameras can collect document images at different angles respectively. The three kinds of light sources include a normal light source 102_1, an ultraviolet (UV) light source 102_2, and an infrared (IR) light source 102_3. The three kinds of light sources can be annular light sources, which can be in a nested relationship or a parallel relationship surrounding the camera 101 - 2; the three kinds of light sources can also be rectangular light sources, placed around the camera 101 - 2, or placed around other cameras. Preferably, the three kinds of light sources are annular light sources and are placed around the camera 101 - 2 in a nested relationship. The inner, middle, and outer layers of the nesting are not restricted (for example, the normal light source is in the inner layer, the UV light source is in the middle layer, and the IR light source is in the outer layer. Only one light source is on each time the camera takes a picture, and the other two light sources are off), and they are switched when collecting document images. Under each kind of light source, the three cameras collect document images from multiple angles. Among them, the camera can be replaced by a scanner (such as CIS). The document image collected by the scanner is clearer and can avoid interference from the external environment.

[0101] It should be noted that, Figure 1 the number and positions of the cameras in are only an example, and there are no restrictive requirements for the number and positions of the cameras. For example, 5 cameras can also be used, which are respectively placed around the card - holding device and at the directly - above direction; and the number and types of the light sources are also only an example. Other numbers and types of light sources can also be used, such as LED light sources.

[0102] It should be noted that after each camera adjusts its shooting position, a camera calibration operation is performed to avoid camera distortion, and the parameters after camera calibration are recorded. Each time the camera adjusts its position, a parameter calibration operation is performed, and the camera is calibrated using these parameters during image collection and defect detection.

[0103] Based on Figure 1 the device shown, Figure 2 the flowchart of the document image defect detection method provided by the embodiment of the present application is exemplarily shown. As Figure 2As shown, this process can be implemented by software or in a combination of software and hardware. This process mainly includes the following steps:

[0104] S201: Capture the document image from at least one angle under multiple light sources.

[0105] In this step, rotate the card-holding device and use at least one camera to capture the document image from at least one angle under at least one light source.

[0106] S202: Detect the defect status of the document image according to the defect type and adopt a method matching the defect type.

[0107] In this step, different defect types correspond to different detection methods. Each detection method can detect the defect status of the document image in parallel and independently, and each detection method does not affect each other, so as to improve the detection speed and efficiency of the document image.

[0108] Generally, an anti-counterfeiting film is pasted on the surface of the document to prevent illegal acts such as forgery, alteration, and cloning. Defects in the anti-counterfeiting film will lead to serious security risks. In an optional embodiment, the defect types detected in the embodiments of the present application include anti-counterfeiting defects, and anti-counterfeiting detection of the document image can be realized. Specifically, determine the anti-counterfeiting area image of the document image collected under the first light source, compare and analyze the anti-counterfeiting area image with the standard anti-counterfeiting image corresponding to the first light source. If they are inconsistent, it is determined that the collected document image has an anti-counterfeiting defect.

[0109] In some embodiments, the first light source includes a UV light source. The UV light source is convenient for detecting the characteristics of the laser anti-counterfeiting film. From the document images collected from multiple angles under the UV light source, select the document images with the clarity of the anti-counterfeiting pattern greater than the set threshold. Adopt morphological transformation and affine transformation methods to determine the UV anti-counterfeiting area image (the area image containing the UV anti-counterfeiting pattern) of the selected document image. Obtain the standard UV anti-counterfeiting image (also called the UV pseudo-sample image) of the document with the standard anti-counterfeiting film under the UV light source. Use the siamese network and template matching methods to analyze and compare whether the UV anti-counterfeiting area image is consistent with the standard UV anti-counterfeiting image. If they are inconsistent, it is determined that the document image collected under the UV light source has a UV anti-counterfeiting defect. If they are consistent, it is determined that the document image collected under the UV light source is a normal image.

[0110] In some other embodiments, the first light source includes an IR light source, which facilitates the detection of special anti-counterfeiting features on the document image. Specifically, a document image is acquired under the IR light source from a specified direction (such as directly above the document). The document image acquired in this specified direction can accurately identify the IR anti-counterfeiting pattern. Using morphological transformation and affine transformation methods, the IR anti-counterfeiting region image (the region image containing the IR anti-counterfeiting pattern) of the document image acquired in the specified direction is determined. The IR anti-counterfeiting region image of the document image acquired in the specified direction is obtained. The standard IR anti-counterfeiting image (also called the IR pseudo-sample image) of the document with a standard anti-counterfeiting film under the IR light source is acquired. Using a siamese network and template matching method, it is analyzed and compared whether the IR anti-counterfeiting region image is consistent with the standard IR anti-counterfeiting image. If they are not consistent, it is determined that there is an IR anti-counterfeiting defect in the document image acquired under the IR light source. If they are consistent, it is determined that the document image acquired under the IR light source is a normal image.

[0111] During the long-term use of the document, the film on the document surface may be missing or have scratches. Variable Laser Image (CLI) and Multiple Laser Image (MLI) can be used to detect the etching defects of the document image acquired under normal light sources. In an alternative embodiment, the document image is acquired from at least one angle under normal light sources. Using morphological transformation and affine transformation methods, the etching region image of the document image acquired from at least one angle under normal light sources is intercepted, and the etching information in the etching region image corresponding to each angle is extracted. The standard MLI / CLI (also called the standard laser image) of the document under normal light sources is obtained, and the etching information in the standard MLI / CLI is extracted. Using a siamese network and template matching method, it is analyzed and compared whether the etching information extracted from the etching region image corresponding to each angle is consistent with the etching information extracted from the standard MLI / CLI. If there is an inconsistency, it is determined that there is an etching defect in the document image acquired under normal light sources. If they are all consistent, it is determined that the document image acquired under normal light sources is a normal image.

[0112] In some embodiments, personal information can also be extracted from the document images acquired from multiple angles under normal light sources to generate a standard document image for detecting the defects of the document image to be detected.

[0113] Figure 3 An exemplary flowchart of the document image defect detection method provided by the embodiments of the present application is shown. As Figure 3 shown, this process mainly includes the following steps:

[0114] S301: Acquire a document image from at least one angle under normal light sources.

[0115] In this step, turn on the normal light source, rotate the card-holding device, and use at least one camera to collect the document image from at least one angle.

[0116] S302: Preprocess the document image to obtain a document image with the background removed.

[0117] In this step, perform grayscale processing on the collected document image to obtain a grayscale document image. The methods of grayscale processing include, but are not limited to, the component method, the maximum value method, the average value method, the weighted average method, etc.; perform filtering processing on the grayscale document image to obtain a denoised grayscale document image. The filtering methods include, but are not limited to, median filtering, mean filtering, Gaussian filtering, etc.; use the Canny edge detection algorithm to extract the edge image of the denoised grayscale document image and perform binaryzation processing to obtain a binary image; obtain a certain range of length-width ratios according to the actual length-width ratio of the document. When the detected contour using the edge detection algorithm conforms to the specified length-width ratio and the area is within the specified area range (since the position of the camera relative to the document is fixed and the area of the captured document image is relatively stable, the area of the document in the image can be measured), it is considered the contour of the document, then intercept this contour and crop to remove other backgrounds to obtain a document image with the background removed. After obtaining the document image with the background removed, operations such as rotation and affine transformation can be performed to reduce the influence caused by the deformation of the document image and make the size ratio of the document image consistent with the actual document.

[0118] It should be noted that the embodiments of the present application do not impose restrictive requirements on the order of the above preprocessing operations. For example, Canny edge detection can be performed first and then binaryzation processing.

[0119] S303: Extract the personal information in the document image with the background removed, and generate a standard document image based on the blank document template according to the personal information.

[0120] In this step, input the document image with the background removed into the trained machine vision detection system. This system extracts the personal information in the document image, generates an information sample map according to the extracted personal information, and combines it with the template of the blank document with background patterns to generate a standard document image for comparison with the document image to be detected. Among them, the personal information includes portrait, name, birthday, expiration date, document number, etc.

[0121] S304: Determine whether the defect degree of the document image to be detected is greater than the set threshold. If so, execute S305; otherwise, execute S306.

[0122] In this step, a siamese network, a morphological subtraction method, and a template matching method are used to determine the defect degree of the document image to be detected and the standard document image. The morphological subtraction method can reveal the content missing in the document image to be detected compared with the standard document image. The more content is missing, the higher the defect degree, and vice versa. Optionally, the defect degree can be determined according to the length of the missing content. For example, if the length of the missing content is greater than 0.5 mm, it is determined that the document image has a content missing defect. The defect degree can also be determined according to the area of the missing content. For example, if the area of the missing content is greater than 0.5 * 0.5 mm, it is determined that the document image has a content missing defect.

[0123] S305: Determine that the document image to be detected has a content missing defect.

[0124] S306: Determine that the document image to be detected is a normal image.

[0125] In the above embodiments of the present application, a standard document image is generated from the personal information extracted from the collected document image, and is analyzed and compared with the document image to be detected to detect whether the document image to be detected has a content missing defect, improving the defect detection accuracy.

[0126] In some embodiments, the black edge defect of the document image to be detected can also be detected according to the difference between the document image to be detected and the standard document image.

[0127] Figure 4 Exemplarily shows the flowchart of the document image defect detection method provided by the embodiments of the present application. As Figure 4 shown, this process mainly includes the following steps:

[0128] S401: Collect document images from at least one angle under normal light sources.

[0129] S402: Preprocess the document image to obtain a document image with the background removed.

[0130] S403: Extract the personal information in the document image with the background removed, and generate a standard document image based on the blank document template according to the personal information.

[0131] For the detailed description of S401 - S403, refer to S301 - S303, which will not be repeated here.

[0132] S404: Preprocess the document image to be detected to remove the noise in the document image to be detected.

[0133] In this step, the document image to be detected is grayscaled, and the adaptive threshold of the document image to be detected is determined to meet different light source brightnesses and different lighting angles. On the basis of not affecting the essence of this application, there are no restrictive requirements for the adaptive threshold in the embodiments of this application. For example, the adaptive threshold includes methods such as the average value of the adjacent area minus the constant C, the Gaussian weighted sum of the neighborhood values minus the constant C, and the Otsu binarization threshold method. Gaussian filtering is performed on the grayscaled document image to be detected to remove the noise in the document image to be detected.

[0134] S405: According to the document image to be detected and the standard document image, a morphological subtraction method is used to obtain a difference image.

[0135] S406: Replace the black and white pixel values in the difference image to obtain the replaced difference image.

[0136] In this step, since the difference image only contains black (0) pixels and white (255) pixels, the black (0) pixel area is the target object (the target to be detected, the information on the document), and the white (255) pixel area is the background. By replacing the black (0) pixels and white (255) pixels in the difference image, the black (0) pixel area in the replaced difference image is the background, and the white (255) pixel area is the target object (the target to be detected, the information on the document).

[0137] S407: Detect the contours in the difference images before and after replacement, and determine the contour differences in the difference images before and after replacement.

[0138] In this step, the Canny edge detection can be used to detect the contours in the difference images before and after replacement.

[0139] S408: Determine whether the contour difference is greater than the set threshold. If so, execute S409; otherwise, execute S410.

[0140] In this step, the length difference of the contours in the difference images before and after replacement can be compared with the set threshold. For example, if the length difference of the contours in the difference images before and after replacement is greater than 0.5 mm, it is determined that there is a black-and-white defect in the document image; the area difference of the contours in the difference images before and after replacement can also be compared with the set threshold. For example, if the area difference of the contours in the difference images before and after replacement is greater than 0.5 * 0.5 mm, it is determined that there is a black-edge defect in the document image.

[0141] S409: Determine that there is a black-edge defect in the document image to be detected.

[0142] S410: Determine that the document image to be detected is a normal image.

[0143] In the above embodiments of the present application, after a series of preprocessings are performed on the document image to be detected, a difference image is obtained based on the standard document image and the document image to be detected, the black and white pixels in the difference image are replaced, and whether there is a black edge defect in the document image to be detected is detected according to the contour difference of the difference image before and after replacement, thereby improving the defect detection accuracy.

[0144] In some embodiments, the ghost defect of the document image can also be detected. Figure 5 Exemplarily shown is the flowchart of the document image defect detection method provided by the embodiments of the present application, as Figure 5 shown, this process mainly includes the following steps:

[0145] S501: Collect the document image from at least one angle under a normal light source.

[0146] S502: Preprocess the document image to obtain a document image with the background removed.

[0147] Among them, the descriptions of S501 and S502 refer to S301 and S302, which will not be repeated here.

[0148] S503: Extract the portrait contours of the document image with the background removed under the CMYK components respectively.

[0149] In this step, the Canny edge detection algorithm is used to intercept the portrait contour image in the document image with the background removed, the portrait contour image is split into 4 color components of CMYK, and the portrait contours under the 4 color components of CMYK are respectively detected using the Canny edge detection algorithm.

[0150] Among them, the 4 color components of CMYK respectively refer to cyan, magenta, yellow, and black.

[0151] S504: Compare the portrait contours under every two color components, and determine whether the deviations of the portrait contours under the 4 color components of CMYK are all greater than the set threshold. If so, execute S405, otherwise execute S406.

[0152] In this step, theoretically, the portrait contours of the standard portrait image under the 4 color components of CMYK should coincide. In specific implementation, the portrait contours under every two color components are compared to determine the contour deviation and compare it with the threshold, so as to determine whether there is a ghost defect in the document image.

[0153] For example, if the deviation of the portrait contour between the C color component and the M color component is greater than the set threshold, the deviation of the portrait contour between the C color component and the Y color component is greater than the set threshold, the deviation of the portrait contour between the C color component and the K color component is greater than the set threshold, the deviation of the portrait contour between the M color component and the Y color component is greater than the set threshold, the deviation of the portrait contour between the M color component and the K color component is greater than the set threshold, and the deviation of the portrait contour between the Y color component and the K color component is greater than the set threshold, it is determined that there is a ghosting defect in the document image; otherwise, it is determined that there is no projection image.

[0154] It should be noted that the sum of the portrait contour deviations under every two color components can also be calculated according to the set weights. If the sum is greater than the set threshold, it is determined that there is a ghosting defect in the document image; otherwise, there is no ghosting defect.

[0155] S505: Determine that there is a ghosting defect in the document image collected under normal light.

[0156] S506: Determine that the document image collected under normal light is a normal image.

[0157] In the above embodiments of the present application, the Canny edge detection algorithm is used to extract the portrait contour in the document image. According to the comparison result between the deviation of the portrait contour under the four color components of CMYK and the threshold, the detection of the ghosting defect in the document image is realized.

[0158] In some embodiments of the present application, a deep learning method is used to train a defect detection network. Through the trained defect detection network, defect types that cannot be detected by traditional vision algorithms can be realized, including but not limited to color patches, impurities (such as dust in the air, oil stains and debris in the surrounding environment), spots, incomplete handwriting, rough edges, missing colors, scratches, etc.

[0159] Based on the trained defect detection network, Figure 6 Exemplarily shows the flowchart of the document image defect detection method provided by the embodiments of the present application. As Figure 6 shown, this process mainly includes the following steps:

[0160] S601: Collect the document image to be detected under normal light.

[0161] In this step, one camera can be used to collect the document image to be detected at at least one angle, or multiple cameras can be used to collect the document image to be detected at at least one angle.

[0162] S602: Input the document image to be detected and the standard document image into the trained feature extraction network.

[0163] In this step, the input layer of the feature extraction network performs preprocessing operations such as normalization and grayscaling on the input image of the document to be detected and the standard document image, so as to improve the speed of defect detection. For the generation process of the standard document image, refer to S303, which will not be repeated here.

[0164] S603: Based on at least one convolutional layer of the trained feature extraction network, extract the image features of the image of the document to be detected.

[0165] In this step, the deep image features and shallow image features of the image of the document to be detected can be extracted through the convolutional layer of the defect detection network. Among them, the parameters of the convolutional layer can be set according to the actual situation. There is a pooling layer (also known as a downsampling layer) after each convolutional layer to reduce the dimensionality of the image features extracted by the convolutional layer, so as to reduce the computational amount and enhance the robustness of the image features. The pooling layer can adopt methods such as max pooling, average pooling, and stochastic pooling.

[0166] S604: Based on at least one fully connected layer of the trained feature extraction network, perform feature fusion on the extracted image features respectively.

[0167] In this step, through at least one fully connected layer of the trained feature extraction network, the shallow image features and deep image features of the image of the document to be detected extracted in S603 are fused, and the shallow image features and deep image features of the extracted standard document image are fused, and the fused image features are normalized.

[0168] S605: Based on the image features of the fused image of the document to be detected and the image features of the fused standard document image, determine the defect type of the image of the document to be detected based on the trained defect detection network.

[0169] In an optional embodiment, the defect detection network can be the first-order detection network YOLO v4, which uses CSPDarkNet53 as the backbone network to extract image features, and the activation function used is the Mish activation function. Compared with the SSD (abbreviation for Single Shot MultiBox Detector) algorithm, the defect detection accuracy is improved, and compared with the second-order detection network, the defect detection speed is improved.

[0170] It should be noted that the deep learning algorithm used by the defect detection network in the embodiments of the present application is not limited. For example, detection networks such as SSD, Faster RCNN, and RetinaNet can also be used to implement defect type detection.

[0171] In an optional embodiment, the defect detection network can be trained in the following manner:

[0172] Collect training samples of document images from at least one angle using at least one camera, input the collected training samples of document images into the input layer of the defect detection network, and perform normalization processing on the obtained training samples of document images to improve the convergence speed of the defect detection network and reduce the training duration. Among them, for the training samples of document images that are not easy to collect, data augmentation can be used to perform various image processing operations such as mirroring, rotation, translation, distortion, filtering, partial occlusion, and contrast adjustment on the defect samples of the original collected document images to obtain more training samples of document images. Combining with the Generative Adversarial Networks (GAN), the generation module in the generative adversarial network generates training samples of document images with defects such as missing content, black edges, impurities, and spots, and the discriminant module in the generative adversarial network predicts the defect types.

[0173] Input multiple document defect training samples and the pre-annotated defect category labels corresponding to each training sample of the document image into the initial defect detection network; perform fusion processing on the image features of the training samples of the document image through the initial defect detection network to obtain the probabilities corresponding to each training sample of the document image indicating that the training sample of the document image belongs to each preset defect category.

[0174] Determine the detection loss value according to the probability corresponding to each training sample of the document image and the pre-annotated defect category label. Specifically, for each training sample of the document image, determine the defect category label corresponding to the maximum probability in the prediction result as the defect category of the training sample of the document image, and determine the detection loss value according to the determined defect category and the pre-annotated defect category label of the training sample of the document image.

[0175] Adjust the model parameters of the initial defect detection network according to the detection loss value until the determined detection loss value is within the preset range to obtain the trained defect detection network.

[0176] In an alternative embodiment, during the process of training the defect detection network, collect the training samples of the document images with defect detection errors, and label the defect category labels for the training samples of the document images with incorrect detection through manual judgment or machine judgment. Moreover, for the newly emerging defect types during the process of training the defect detection network, collect the corresponding training samples of the document images, and label the corresponding new defect category labels for them through manual judgment or machine judgment; input the training samples of the document images of the new defect types and the training samples of the document images with incorrect detection as new training samples and the corresponding defect category labels into the trained defect detection network, and retrain the trained defect detection network to obtain the updated defect detection network.

[0177] During the training of the defect detection network, according to the results of error detection and newly emerging defect types, manual or machine judgment is used for annotation, and the annotated data is re-input into the trained defect detection network to update the model of the defect detection network, thereby improving the accuracy of defect detection. For the case where the number of samples corresponding to the defect type is small, the GAN network and data augmentation method are used to automatically generate training samples of ID card images to strengthen the detection network.

[0178] It should be noted that the defect detection network can be deployed on a server, and with a Graphics Processing Unit (GPU) graphics card, the training and detection speed can be improved. In the industrial field, the defect detection network can also be deployed on an industrial control computer and cooperate with the CPU for defect detection.

[0179] In the above embodiments of the present application, the defect detection network trained by the deep learning method is used to detect defects in ID card images. The deep learning method can detect the position, size, color, etc. of defects such as color patches, spots, and impurities, while traditional computer vision methods cannot accurately detect the above-mentioned defects, improving the accuracy of defect detection.

[0180] In some embodiments, after detecting the defect category of the ID card image, a warning message is displayed on the user interface, which is convenient for timely discovery and resolution of defective ID cards, improving the flexibility and automation level of production, enhancing the intelligence level of the ID card production line, and increasing the yield.

[0181] Based on Figure 2-6 the method flow, Figure 7 Exemplarily, the complete flowchart of the defect detection method provided by the embodiments of the present application is shown. As Figure 7 shown, the process mainly includes the following steps:

[0182] S701: Collect ID card images from at least one angle under multiple light sources.

[0183] In this step, the light sources include UV light sources, IR light sources, and ordinary light sources.

[0184] S702: Preprocess the collected ID card images.

[0185] In this step, the preprocessing includes operations such as grayscale conversion, binarization, background removal, and noise removal. For details, please refer to the foregoing embodiments.

[0186] S703: Perform UV anti-counterfeiting defect detection on the ID card images collected from at least one angle under the UV light source.

[0187] For the detailed process of this step, please refer to the detection process of UV anti-counterfeiting defects in the foregoing embodiments, and will not be repeated here.

[0188] S704: Perform IR anti-counterfeiting defect detection on the document images collected at at least one angle under the IR light source.

[0189] For the detailed process of this step, refer to the detection process of UV anti-counterfeiting defects in the foregoing embodiments, which will not be repeated here.

[0190] S705: Perform etching defect detection on the document images collected at at least one angle under the normal light source.

[0191] For the detailed process of this step, refer to the detection process of etching defects in the foregoing embodiments, which will not be repeated here.

[0192] S706: Perform ghosting defect detection on the document images collected at at least one angle under the normal light source.

[0193] For the detailed process of this step, refer to Figure 5 , which will not be repeated here.

[0194] S707: Extract personal information from the document images collected at at least one angle under the normal light source, and generate a standard document image based on the personal information and the blank document template.

[0195] For the detailed process of this step, refer to S303, which will not be repeated here.

[0196] S708: Use the siamese network, morphological subtraction method, and template matching method to compare the document images collected under the normal light source with the standard document image, and detect the content missing defects of the collected document images.

[0197] For the detailed process of this step, refer to S304 - S306, which will not be repeated here.

[0198] S709: Use the morphological subtraction method to obtain a difference image based on the document images collected under the normal light source and the standard document image, and detect the black edge defects of the document images.

[0199] For the detailed process of this step, refer to Figure 4 , which will not be repeated here.

[0200] S710: Based on the trained feature extraction network, extract the image features of the document images collected under the normal light source and extract the image features of the standard document image.

[0201] For the detailed process of this step, refer to S602 - S604, which will not be repeated here.

[0202] S711: Based on the extracted image features of the document images collected under the normal light source and the extracted image features of the standard document image, and based on the trained defect detection network, determine the defect types of the document images collected under the normal light source.

[0203] For a detailed description of the step defect detection network, refer to the foregoing embodiments and will not be repeated here.

[0204] It should be noted that since the defect detection methods of various types do not affect each other and can be run in parallel, Figure 7 the steps in are not in a strict execution order. For example, S706 can be prior to S703, S703 - S706 can also be executed in parallel, S710 can be prior to S708, and S708 - S710 can also be executed in parallel.

[0205] Based on the above embodiments, an embodiment of the present application provides a document image defect detection device, which can implement the defect detection method in the above embodiments.

[0206] See Figure 8 , the device includes a content missing defect detection module 801, a ghosting defect detection module 802, an etching defect detection module 803, an anti-counterfeiting defect detection module 804, a black edge defect detection module 805, a sample acquisition module 806, a model training module 807, and a deep learning detection module 808.

[0207] The content missing defect detection module 801 is used to generate a standard document image according to personal information, detect the defect degree of the document image to be detected and the standard document image using a siamese network, a morphological subtraction method, and a template matching method, and detect the content missing defect of the printed content of the document image (including portrait and text) according to the defect degree;

[0208] The ghosting defect detection module 802 is used to preprocess the document image collected from at least one angle under normal light to obtain a document image with the background removed, extract the portrait contours of the document image with the background removed under the CMYK color components respectively, and detect the ghosting defect of the document image according to the comparison between the deviation of the portrait contours under every two color components and a set threshold;

[0209] The etching defect detection module 803 is used to intercept the etching area image of the document image collected from at least one angle under normal light, extract the etching information in the etching area image, and compare it with the etching information extracted from the standard laser image to detect the etching defect of the document image;

[0210] The anti-counterfeiting defect detection module 804 is used to compare the document image collected from at least one angle under UV light with the standard UV anti-counterfeiting image to detect the UV anti-counterfeiting defect of the document image, and, compare the document image collected from at least one angle under IR light with the standard IR anti-counterfeiting image to detect the IR anti-counterfeiting defect of the document image;

[0211] The black edge defect detection module 805 is used to obtain a difference image according to the document image to be detected and the standard document image by using the method of morphological subtraction; replace the black and white pixels in the difference image to obtain the replaced difference image, and compare the contour differences between the difference images before and after replacement with a set threshold to detect the black edge defect of the document image to be detected;

[0212] The sample acquisition module 806 is used to acquire document images from at least one angle under at least one light source;

[0213] The model training module 807 is used to train a defect detection network according to the acquired document image training samples, and / or the document image training samples generated by using the data augmentation method, and / or the document image training samples generated by using the GAN network, and label the defect category labels, and update the trained defect detection network according to the document image training samples of the new defect type and the document image training samples of the misdetected ones;

[0214] The deep learning detection module 808 is used to detect the defect type of the document image to be detected by using the defect detection network trained with the deep learning algorithm.

[0215] It should be noted here that the above device provided in the embodiment of the present application can implement the defect detection method steps implemented in the above method embodiment, and can achieve the same technical effect. The same parts and beneficial effects as those in the method embodiment in this embodiment will not be specifically described here.

[0216] Based on the above embodiment, the embodiment of the present application provides a document image defect detection device, and this device can implement the defect detection method in the above embodiment.

[0217] See Figure 9 , this device includes an acquisition module 901 and a defect detection module 902.

[0218] The acquisition module is used to acquire document images from at least one angle under multiple light sources;

[0219] The defect detection module is used to determine the defect type to be detected and detect the defect state of the document image by using a method matching the defect type.

[0220] In some embodiments, the defect type includes an anti-counterfeiting defect, and the defect detection module 902 is specifically used for:

[0221] Determine the anti-counterfeiting area image of the document image acquired under the first light source;

[0222] If the anti-counterfeiting area image of the document image is inconsistent with the standard anti-counterfeiting image corresponding to the first light source, it is determined that the document image has an anti-counterfeiting defect.

[0223] In some embodiments, the first light source includes an ultraviolet (UV) light source, and the defect detection module 702 is specifically configured to:

[0224] Determine the UV anti-counterfeiting area image of the document image collected at at least one angle under the UV light source; if the UV anti-counterfeiting area image of the document image is inconsistent with the standard UV anti-counterfeiting image, determine that the document image collected under the UV light source has a UV anti-counterfeiting defect; and / or

[0225] The first light source includes an infrared (IR) light source, and the defect detection module 902 is specifically configured to:

[0226] Determine the IR anti-counterfeiting area image of the document image collected from a specified direction under the IR light source; if the IR anti-counterfeiting area image of the document image is inconsistent with the standard IR anti-counterfeiting image, determine that the document image collected under the IR light source has an IR anti-counterfeiting defect.

[0227] In some embodiments, the defect type includes an etching defect, the light source includes a normal light source; the defect detection module 902 is specifically configured to:

[0228] Intercept the etching area image of the document image collected at at least one angle under the normal light source;

[0229] If the etching information extracted from the etching area images corresponding to each angle is inconsistent with the etching information of the standard laser image, determine that the document image collected under the normal light source has an etching defect.

[0230] It should be noted here that the above device provided by the embodiments of the present application can implement the defect detection method steps implemented by the above method embodiments, and can achieve the same technical effects. The same parts and beneficial effects as those in the method embodiments will not be specifically described in this embodiment.

[0231] Based on the above embodiments, the embodiments of the present application provide a document image defect detection device, which can implement the defect detection method in the above embodiments.

[0232] See Figure 10 , the device includes a collection module 1001, a preprocessing module 1002, an image generation module 1003, and a defect detection module 1004.

[0233] The collection module 1001 is configured to collect a document image at at least one angle under a normal light source;

[0234] The preprocessing module 1002 is configured to preprocess the document image to obtain a document image with the background removed;

[0235] The image generation module 1003 is configured to extract personal information from the document image with the background removed, and generate a standard document image based on a blank document template according to the personal information;

[0236] A defect detection module 1004 is configured to determine that there is a content missing defect in the document image to be detected if the defect degree of the document image to be detected is greater than a set threshold compared with the standard document image.

[0237] It should be noted here that the above device provided in the embodiment of the present application can implement the defect detection method steps implemented in the above method embodiment and can achieve the same technical effect. The same parts and beneficial effects as those in the method embodiment will not be specifically described herein.

[0238] Based on the above embodiment, the embodiment of the present application provides a document image defect detection device, which can implement the defect detection method in the above embodiment.

[0239] See Figure 11 , the device includes a preprocessing module 1101, a pixel replacement module 1102, and a defect detection module 1103.

[0240] The preprocessing module 1101 is configured to preprocess the document image to be detected and remove the noise in the document image to be detected;

[0241] The pixel replacement module 1102 is configured to obtain a difference image by using the morphological subtraction method according to the document image to be detected and the standard document image; replace the black and white pixels in the difference image to obtain a replaced difference image;

[0242] The defect detection module 1103 is configured to determine that there is a black edge defect in the document image to be detected if the contour difference of the difference image before and after replacement is greater than a set threshold.

[0243] It should be noted here that the above device provided in the embodiment of the present application can implement the defect detection method steps implemented in the above method embodiment and can achieve the same technical effect. The same parts and beneficial effects as those in the method embodiment will not be specifically described herein.

[0244] Based on the above embodiment, the embodiment of the present application provides a document image defect detection device, which can implement the defect detection method in the above embodiment.

[0245] See Figure 12 , the device includes an acquisition module 1201, a preprocessing module 1202, and a defect detection module 1203.

[0246] The acquisition module 1201 is configured to acquire a document image from at least one angle under a normal light source;

[0247] The preprocessing module 1202 is configured to preprocess the document image to obtain a document image with the background removed;

[0248] A defect detection module 1203 is configured to extract the portrait contours of the document image with the background removed under the CMYK color components respectively; if the deviation between the portrait contours under every two color components is greater than a set threshold, it is determined that there is a ghosting defect in the document image collected under normal light.

[0249] It should be noted here that the above device provided by the embodiments of the present application can implement the defect detection method steps implemented by the above method embodiments and can achieve the same technical effects. The same parts and beneficial effects as those in the method embodiments in this embodiment will not be specifically described herein.

[0250] Based on the above embodiments, the embodiments of the present application provide a document image defect detection device, which can implement the defect detection method in the above embodiments.

[0251] See Figure 13 , the device includes a collection module 1301, a feature extraction module 1302, and a defect detection module 1303.

[0252] The collection module 1301 is configured to collect a document image to be detected under normal light;

[0253] The feature extraction module 1302 is configured to extract the image features of the document image to be detected and the image features of the standard document image based on the convolutional layer of the trained feature extraction network; and perform feature fusion on the extracted image features respectively based on the fully connected layer of the trained feature extraction network;

[0254] The defect detection module 1303 is configured to determine the defect type of the document image to be detected based on the fused image features of the document image to be detected and the fused image features of the standard document image, based on the trained defect detection network.

[0255] In some embodiments, the device further includes a sample training module, configured to:

[0256] Obtain a plurality of document image training samples;

[0257] Input a plurality of document defect training samples and the pre-annotated defect category labels corresponding to each document image training sample into the initial defect detection network; perform fusion processing on the image features of the document image training samples through the initial defect detection network to obtain the probabilities corresponding to each document image training sample indicating that the document image training samples respectively belong to each preset defect category;

[0258] Determine the detection loss value according to the probabilities corresponding to each document image training sample and the pre-annotated defect category labels;

[0259] Adjust the model parameters of the initial defect detection network according to the detection loss value until the determined detection loss value is within a preset range, and obtain the trained defect detection network.

[0260] In some embodiments, the device further includes an update module for:

[0261] Use the certificate image training samples of the new defect type and the certificate image training samples of the misdetected ones as new training samples, label the corresponding defect category labels, and input them into the trained defect detection network;

[0262] Retrain the trained defect detection network to obtain an updated defect detection network.

[0263] It should be noted here that the above device provided by the embodiments of the present application can implement the defect detection method steps implemented by the above method embodiments and can achieve the same technical effects. Therefore, the same parts and beneficial effects as those in the method embodiments will not be specifically described in this embodiment.

[0264] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to make a computer execute the method in the above embodiments.

[0265] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0266] The present application is described with reference to the flowcharts and / or block diagrams of the method, device (system), and computer program product according to the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the functions specified in Figure 1 one or more flows and / or blocks Figure 1 one or more blocks.

[0267] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction device that implements the function specified in one or more of the processes Figure 1 a process or processes and / or boxes Figure 1 specified in a box or boxes.

[0268] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable apparatus provide steps for implementing the function specified in one or more of the processes Figure 1 a process or processes and / or boxes Figure 1 specified in a box or boxes.

[0269] It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit and scope of the application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations.

Claims

1. A method for detecting defects in document images, characterized in that, Including: Collecting a certificate image from at least one angle under multiple light sources; Detecting the defect status of the certificate image according to the defect type and adopting a method matching the defect type; Wherein, the defect type includes an etching defect, and the light source includes a normal light source; the detecting the defect status of the certificate image by adopting a method matching the defect type includes: Intercepting an etching area image of the certificate image collected from at least one angle under the normal light source; if the etching information extracted from the etching area images corresponding to each angle is inconsistent with the etching information of the standard laser image, it is determined that the certificate image collected under the normal light source has an etching defect; The defect type includes a ghosting defect, and the light source includes a normal light source; The detecting the defect status of the certificate image by adopting a method matching the defect type includes: Preprocessing the certificate image collected from at least one angle under the normal light source to obtain a certificate image with the background removed; Extracting the portrait contours of the certificate image with the background removed under the CMYK color components respectively; If the deviation of the portrait contours under each two color components is greater than a set threshold, it is determined that the certificate image collected under the normal light source has a ghosting defect.

2. The method according to claim 1, characterized in that, The defect type includes an anti-counterfeiting defect, and the detecting the defect status of the certificate image by adopting a method matching the defect type includes: Determining an anti-counterfeiting area image of the certificate image collected under the first light source; If the anti-counterfeiting area image of the certificate image is inconsistent with the standard anti-counterfeiting image corresponding to the first light source, it is determined that the certificate image has an anti-counterfeiting defect.

3. The method according to claim 2, wherein The first light source includes an ultraviolet (UV) light source. The determining the anti-counterfeiting area image of the certificate image collected under the first light source; if the anti-counterfeiting area image of the certificate image is inconsistent with the standard anti-counterfeiting image corresponding to the first light source, it is determined that the certificate image has an anti-counterfeiting defect, includes: Determining a UV anti-counterfeiting area image of the certificate image collected from at least one angle under the UV light source; if the UV anti-counterfeiting area image of the certificate image is inconsistent with the standard UV anti-counterfeiting image, it is determined that the certificate image collected under the UV light source has a UV anti-counterfeiting defect; and / or The first light source includes an infrared (IR) light source. The determining the anti-counterfeiting area image of the certificate image collected under the first light source; if the anti-counterfeiting area image of the certificate image is inconsistent with the standard anti-counterfeiting image corresponding to the first light source, it is determined that the certificate image has an anti-counterfeiting defect, includes: Determining an IR anti-counterfeiting area image of the certificate image collected from a specified direction under the IR light source; if the IR anti-counterfeiting area image of the certificate image is inconsistent with the standard IR anti-counterfeiting image, it is determined that the certificate image collected under the IR light source has an IR anti-counterfeiting defect.

4. The method according to claim 1, characterized in that The defect type includes a content missing defect, and the light source includes a normal light source; The detecting the defect status of the certificate image by adopting a method matching the defect type includes: preprocessing the certificate image collected from at least one angle under the normal light source to obtain a certificate image with the background removed; Extract personal information from a background-removed ID image, and generate a standard ID image based on the blank ID template according to the personal information. If the defect degree of the ID image to be detected is greater than the set threshold compared with the standard ID image, it is determined that the ID image to be detected has a content missing defect.

5. The method according to claim 1, wherein The defect type includes a black edge defect. The method of detecting the defect state of the ID image by using a method matching the defect type includes: Preprocess the ID image to be detected to remove noise in the ID image to be detected. According to the ID image to be detected and the standard ID image, obtain a difference image by using the method of morphological subtraction. Replace the black and white pixels in the difference image to obtain the replaced difference image. If the contour difference between the difference images before and after replacement is greater than the set threshold, it is determined that the ID image to be detected has a black edge defect.

6. The method according to claim 1, wherein The light source includes a normal light source. The method of detecting the defect state of the ID image by using a method matching the defect type includes: Input the ID image to be detected and the standard ID image collected under the normal light source into the trained feature extraction network. Based on at least one convolutional layer of the trained feature extraction network, extract the image features of the ID image to be detected and the image features of the standard ID image. Based on at least one fully connected layer of the trained feature extraction network, perform feature fusion on the extracted image features respectively. Based on the trained defect detection network, determine the defect type of the ID image to be detected according to the fused image features of the ID image to be detected and the fused image features of the standard ID image.

7. The method according to claim 6, wherein Train the defect detection network according to the following method: Obtain multiple ID image training samples. Input the multiple ID image training samples and the pre-annotated defect category labels corresponding to each ID image training sample into the initial defect detection network. Through the initial defect detection network, perform fusion processing on the image features of the ID image training samples to obtain the probabilities corresponding to each ID image training sample indicating that the ID image training sample belongs to each preset defect category respectively. Determine the detection loss value according to the probability corresponding to each ID image training sample and the pre-annotated defect category label. Adjust the model parameters of the initial defect detection network according to the detection loss value until the determined detection loss value is within the preset range, and obtain the trained defect detection network.

8. The method according to claim 7, wherein The method further includes: Use the ID image training samples of the new defect type and the ID image training samples of the misdetected ones as new training samples, and label the corresponding defect category labels, and input them into the trained defect detection network. Retrain the trained defect detection network to obtain the updated defect detection network.

9. A device for detecting defects in document images, characterized in that, It includes: An acquisition module for acquiring ID images from at least one angle under multiple light sources. A defect detection module for determining the defect type to be detected and using a method matching the defect type to detect the defect state of the ID image. Wherein, the defect type includes etching defects, the light source includes a common light source; and the defect detection module is specifically used for: Intercepting an etched area image of the document image acquired from at least one angle under the ordinary light source; if the etching information extracted from the etched area image corresponding to each angle is inconsistent with the etching information of the standard laser image, it is determined that the document image acquired under the ordinary light source has an etching defect; The defect type includes a ghost defect, and the light source includes a common light source; The defect state of the document image is detected by using a method matching the defect type, and the defect detection module is specifically used for: Preprocessing the document image collected from at least one angle under a common light source to obtain a document image with the background removed; Extract the portrait outline of the document image with the background removed in CMYK color components; If the deviations of the portrait contours under each of the two color components are greater than the set threshold, it is determined that the document image captured under the ordinary light source has a ghosting defect.

Images