An artificial intelligence-based document quality inspection system and method

Abstract

The present invention discloses an identity document quality inspection system and method based on artificial intelligence. The method includes: feeding a card-type identity document into the information collection area, or turning a booklet-type identity document and feeding it into the information collection area; obtaining an identity document image; obtaining identity document chip information; obtaining identity document printing information based on the identity document image, obtaining identity document barcode information based on the identity document image, and performing consistency detection on the chip information, printing information, and barcode information of the identity document; performing printing quality detection on the identity document that has passed the consistency detection based on the identity document image; performing lamination quality detection on the identity document that has passed the printing quality detection based on the identity document image; conveying the identity documents that have not passed all of the above three detections to the unqualified quality inspection area, and conveying the identity documents that have passed the lamination quality detection and do not require further information comparison to the qualified quality inspection area. The system includes functional modules corresponding to implement the above steps. According to the present invention, it is possible to solve the problems of low efficiency and low accuracy of the existing manual identity document quality inspection method.

Description

Technical Field

[0001] The present invention belongs to the field of document quality inspection, and more specifically, relates to a document quality inspection system and method based on artificial intelligence. Background Art

[0002] The existing document quality inspection mainly includes two links: information comparison and printing and lamination quality inspection. For the information comparison link, it is necessary to manually place the passport on the passport reader and the card on the reader-writer. In this way, only the relevant information of one document can be obtained each time, and the most crucial information comparison work needs to be achieved manually. Moreover, it is necessary to manually change the card after the information comparison, resulting in low efficiency. For the printing and lamination quality inspection link, the manual quality inspection method is still adopted, which is easily affected by human fatigue and prone to errors. Generally speaking, both the information comparison and the printing and lamination quality inspection involved in the existing document quality inspection are completed based on manual methods, which not only have low efficiency but also low reliability. Summary of the Invention

[0003] The purpose of the present invention is to solve the problems of low efficiency and low accuracy of the existing manual-based document quality inspection method.

[0004] To achieve the above purpose, the present invention provides a document quality inspection system and method based on artificial intelligence.

[0005] According to the first aspect of the present invention, a document quality inspection system based on artificial intelligence is provided. The system includes the following functional modules:

[0006] An issuing and page-turning module, configured to convey the card-type document to be quality inspected to the information collection area, and page-turn the book-type document to be quality inspected and convey it to the information collection area;

[0007] A photographing module, configured to obtain an image of the document in the information collection area;

[0008] A chip reading module, configured to obtain the chip information of the document in the information collection area;

[0009] A printing information recognition module, configured to obtain the printing information of the document based on the obtained document image, where the printing information includes text information and biometric information;

[0010] A barcode recognition module, configured to obtain the barcode information of the document based on the obtained document image;

[0011] An information comparison module, configured to perform three-way consistency detection on the chip information, the printing information, and the barcode information;

[0012] A printing quality detection module, configured to perform printing quality detection on the document that passes the three-way consistency detection based on the obtained document image;

[0013] A film coating quality detection module, which is used to perform film coating quality detection on the certificates that have passed the printing quality detection based on the obtained certificate images;

[0014] A certificate receiving module, which is used to convey the certificates that have not passed the three-way consistency detection, the printing quality detection or the film coating quality detection to the unqualified quality inspection area, and convey the certificates that have passed the film coating quality detection and do not require further information comparison to the qualified quality inspection area.

[0015] Preferably, the photographing module includes:

[0016] A light source sub-module, which is used to provide white light source, ultraviolet light source and infrared light source;

[0017] A certificate rotation sub-module, which is used to rotate the certificate;

[0018] A first camera module, which is used to take pictures of the certificate at a predetermined rotation angle from below under the white light source, the ultraviolet light source and the infrared light source respectively;

[0019] A second camera module, which is used to take pictures of the certificate at a predetermined rotation angle from above under the white light source, the ultraviolet light source and the infrared light source respectively.

[0020] Preferably, after passing the film coating quality detection, the certificates that need to undergo further information comparison are book-type passports;

[0021] The certificate quality inspection system further includes:

[0022] A certificate return module, which is used to return the book-type passports that have passed the film coating quality detection to the certificate issuing and page turning module;

[0023] The certificate issuing and page turning module is further used to turn the returned book-type passport from the personal information page to the visa page, and re-convey the book-type passport to the information collection area;

[0024] The photographing module is further used to obtain the visa page image of the book-type passport in the information collection area;

[0025] The printed information recognition module is further used to obtain the visa page printed information based on the obtained visa page image;

[0026] The information comparison module is further used to perform four-way consistency detection on the visa page printed information, the chip information, the personal information page printed information and the barcode information of the book-type passport obtained previously;

[0027] The certificate collection module is also used to convey the passport in book form that fails the four-way consistency detection to the unqualified area for quality inspection, and convey the passport in book form that passes the four-way consistency detection to the qualified area for quality inspection.

[0028] Preferably, the print information recognition module, the barcode recognition module, the information comparison module, the print quality detection module, and the film laminating quality detection module are all implemented based on an edge computing device.

[0029] Preferably, the edge computing device implements the print quality detection and the film laminating quality detection based on a predetermined first document quality detection method;

[0030] The first document quality detection method includes:

[0031] Detecting whether the orientation of the input target document image is correct based on a pre-constructed image orientation detection model. If not, adjusting the orientation of the target document image;

[0032] Performing OCR recognition on the target document image with a correct orientation to obtain the actual position information of the target text;

[0033] Obtaining the print offset of the target text according to the actual position information of the target text and the pre-acquired standard position information of the target text;

[0034] If the print offset of the target text exceeds a predetermined print offset threshold, it is determined that the target document fails the print position offset detection;

[0035] Detecting whether the target document image that passes the print position offset detection and is input contains black edges or impurities based on a pre-constructed image semantic segmentation model. If so, it is determined that the target document fails the black edge and impurity detection;

[0036] Detecting whether there are problems such as incomplete transfer of text and portrait, incomplete film laminating, or print ghosting in the target document image that passes the black edge and impurity detection and is input based on a pre-constructed transfer classification model. If so, it is determined that the target document fails the detection of incomplete transfer of text and portrait, incomplete film laminating, and print ghosting;

[0037] Detecting whether there are red bars, color patches, or color dots in the target document image that passes the detection of incomplete transfer of text and portrait, incomplete film laminating, and print ghosting and is input based on a pre-constructed target detection network model. If so, it is determined that the target document fails the detection of red bars, color patches, and color dots;

[0038] Marking the target document that fails the red bar, color patch, and color dot detection as unqualified for quality inspection, and marking the target document that passes the red bar, color patch, and color dot detection as qualified for quality inspection.

[0039] Preferably, the edge computing device implements the print quality detection based on a predetermined second document quality detection method, and the second document quality detection method includes:

[0040] Obtain a standard print image corresponding to the target document image;

[0041] Perform preprocessing and feature point extraction on the standard print image and the target document image;

[0042] Perform feature matching based on the feature points of the standard print image and the feature points of the target document image obtained, and obtain the morphological subtraction result of the standard print image and the target document image;

[0043] Perform similarity detection and defect area calculation on the morphological subtraction result;

[0044] If the detected similarity is higher than a predetermined similarity threshold and the calculated defect area is smaller than a predetermined defect area threshold, it is determined that the target document passes the print quality detection.

[0045] Preferably, the edge computing device implements the print quality detection based on a predetermined third document quality detection method;

[0046] The third document quality detection method is implemented based on a pre-constructed generative adversarial network model;

[0047] The generative adversarial network model includes:

[0048] An encoder GE(x), which is used to obtain a feature vector z based on the input target document image;

[0049] A decoder GD(z), which is used to obtain a reconstructed document image based on the input feature vector z and the chip information of the target document;

[0050] An encoder GE(x'), which is used to obtain a feature vector z' based on the input reconstructed document image;

[0051] If the vector difference between the feature vector z and the feature vector z' exceeds a predetermined vector difference threshold, it is determined that the target document fails the print quality detection.

[0052] Preferably, the edge computing device implements the print quality detection based on a predetermined fourth document quality detection method;

[0053] The fourth document quality detection method is implemented based on a pre-constructed improved generative adversarial network model;

[0054] The improved generative adversarial network model is trained based on data augmentation. The improved generative adversarial network model includes:

[0055] A reconstruction sub-network for reconstructing the input target document image into a document image without anomalies;

[0056] A discrimination sub-network for obtaining the difference between the target document image and the document image without anomalies, and judging whether the target document image passes the print quality inspection based on the difference.

[0057] According to the second aspect of the present invention, there is provided an identity document quality inspection method based on artificial intelligence. This method is implemented based on the above-mentioned identity document quality inspection system based on artificial intelligence. The method includes the following steps:

[0058] Convey the card-type identity document to be quality-inspected to the information collection area, or turn the booklet-type identity document to be quality-inspected and convey it to the information collection area;

[0059] Obtain the image of the identity document in the information collection area;

[0060] Obtain the chip information of the identity document in the information collection area;

[0061] Based on the obtained identity document image, obtain the print information and barcode information of the identity document. The print information includes text information and biometric information;

[0062] Perform a three-way consistency check on the chip information, the print information, and the barcode information;

[0063] Based on the obtained identity document image, perform a print quality inspection on the identity document that passes the three-way consistency check;

[0064] Based on the obtained identity document image, perform a lamination quality inspection on the identity document that passes the print quality inspection;

[0065] Convey the identity document that fails the three-way consistency check, the print quality inspection, or the lamination quality inspection to the unqualified quality inspection area, and convey the identity document that passes the lamination quality inspection and does not require further information comparison to the qualified quality inspection area.

[0066] Preferably, when the booklet-type identity document is a booklet-type passport, after performing the lamination quality inspection on the booklet-type passport that passes the print quality inspection, it further includes:

[0067] Retreat the booklet-type passport that passes the lamination quality inspection;

[0068] Turn the retreated booklet-type passport from the personal information page to the visa page, and convey the booklet-type passport back to the information collection area;

[0069] Obtain the image of the visa page of the booklet passport within the information collection area;

[0070] Obtain the visa page printing information based on the obtained visa page image;

[0071] Perform a four-way consistency check on the visa page printing information, the chip information of the booklet passport obtained previously, the personal data page printing information, and the barcode information;

[0072] Convey the booklet passports that fail the four-way consistency check to the unqualified quality inspection area, and convey the booklet passports that pass the four-way consistency check to the qualified quality inspection area.

[0073] The beneficial effects of the present invention are as follows:

[0074] The artificial intelligence-based document quality inspection system of the present invention conveys the card-type documents to be quality inspected to the information collection area through the document issuance and page turning module, and turns and conveys the booklet-type documents to be quality inspected to the information collection area; obtains the image of the document within the information collection area through the photographing module; obtains the chip information of the document within the information collection area through the chip reading module; obtains the printing information of the document based on the obtained document image through the printing information recognition module; obtains the barcode information of the document based on the obtained document image through the barcode recognition module; performs a three-way consistency check on the chip information, the printing information, and the barcode information through the information comparison module; performs printing quality inspection on the documents that pass the three-way consistency check based on the obtained document image through the printing quality inspection module; performs lamination quality inspection on the documents that pass the printing quality inspection based on the obtained document image through the lamination quality inspection module; conveys the documents that fail the three-way consistency check, the printing quality inspection, or the lamination quality inspection to the unqualified quality inspection area through the document collection module, and conveys the documents that pass the lamination quality inspection and do not require further information comparison to the qualified quality inspection area. Among them, the printing information recognition module, the printing quality inspection module, and the lamination quality inspection module are all implemented based on artificial intelligence technology.

[0075] According to the above content, by adopting the artificial intelligence-based document quality inspection system of the present invention, it is possible to achieve full automation of the information comparison link and the printing and lamination quality inspection link, liberate human resources, and not only have higher efficiency, but also higher reliability and accuracy.

[0076] The artificial intelligence-based document quality inspection method of the present invention belongs to the same general inventive concept as the above-mentioned artificial intelligence-based document quality inspection system, so it has the same beneficial effects as the above-mentioned artificial intelligence-based document quality inspection system, and will not be elaborated one by one here.

[0077] Other features and advantages of the present invention will be described in detail in the following detailed implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] By describing the exemplary embodiments of the present invention in more detail in conjunction with the accompanying drawings, the above and other objects, features, and advantages of the present invention will become more apparent, wherein, in the exemplary embodiments of the present invention, the same reference numerals generally represent the same components.

[0079] Figure 1 FIG. shows a structural block diagram of an artificial intelligence-based document quality inspection system according to an embodiment of the present invention;

[0080] Figure 2 FIG. shows a flowchart of the implementation of an artificial intelligence-based document quality inspection method according to an embodiment of the present invention;

[0081] Figure 3 FIG. shows a system composition diagram of another artificial intelligence-based document quality inspection system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0082] The preferred embodiments of the present invention will be described in more detail below. Although the following describes the preferred embodiments of the present invention, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present invention more thorough and complete, and to fully convey the scope of the present invention to those skilled in the art.

[0083] Embodiment: Figure 1 FIG. shows a structural block diagram of an artificial intelligence-based document quality inspection system according to an embodiment of the present invention. Referring to Figure 1 , the artificial intelligence-based document quality inspection system according to an embodiment of the present invention includes the following functional modules:

[0084] An issuing and page-turning module for conveying the card-type document to be quality-inspected to the information collection area, and turning the page of the book-type document to be quality-inspected and conveying it to the information collection area;

[0085] A photographing module for obtaining an image of the document in the information collection area;

[0086] A chip reading module for obtaining the chip information of the document in the information collection area;

[0087] A printed information recognition module for obtaining the printed information of the document based on the obtained document image, where the printed information includes text information and biometric information, and the biometric information includes portrait information and fingerprint information;

[0088] A barcode recognition module, configured to obtain the barcode information of the certificate based on the acquired certificate image;

[0089] An information comparison module, configured to perform a three-way consistency check on the chip information, the printed information, and the barcode information;

[0090] A printing quality detection module, configured to perform a printing quality detection on the certificates that pass the three-way consistency check based on the acquired certificate image;

[0091] A film laminating quality detection module, configured to perform a film laminating quality detection on the certificates that pass the printing quality detection based on the acquired certificate image;

[0092] A certificate receiving module, configured to convey the certificates that fail the three-way consistency check, the printing quality detection, or the film laminating quality detection to the unqualified inspection area, and convey the certificates that pass the film laminating quality detection and do not require further information comparison to the qualified inspection area.

[0093] Further, in the embodiment of the present invention, the photographing module includes:

[0094] A light source sub-module, configured to provide a white light source, an ultraviolet light source, and an infrared light source;

[0095] A certificate rotation sub-module, configured to rotate the certificate;

[0096] A first camera module, configured to photograph the certificate at a predetermined rotation angle from below under the white light source, the ultraviolet light source, and the infrared light source respectively;

[0097] A second camera module, configured to photograph the certificate at a predetermined rotation angle from above under the white light source, the ultraviolet light source, and the infrared light source respectively.

[0098] Still further, in the embodiment of the present invention, after passing the film laminating quality detection, the certificates that require further information comparison are booklet passports;

[0099] The certificate quality inspection system further includes:

[0100] A certificate return module, configured to return the booklet passports that pass the film laminating quality detection to the certificate issuing and page turning module;

[0101] The certificate issuing and page turning module is further configured to turn the returned booklet passport from the personal information page to the visa page, and convey the booklet passport back to the information collection area;

[0102] The photographing module is further configured to obtain the visa page image of the booklet passport in the information collection area;

[0103] The printing information recognition module is further configured to obtain the visa page printing information based on the acquired visa page image, where the visa page printing information includes visa information;

[0104] The information comparison module is further configured to perform a four-way consistency check on the visa page printing information, the chip information of the booklet passport acquired previously, the personal data page printing information, and the barcode information;

[0105] The certificate receiving module is further configured to convey the booklet passport that fails the four-way consistency check to the unqualified quality inspection area, and convey the booklet passport that passes the four-way consistency check to the qualified quality inspection area.

[0106] Furthermore, in the embodiment of the present invention, the printing information recognition module, the barcode recognition module, the information comparison module, the printing quality detection module, and the film laminating quality detection module are all implemented based on an edge computing device.

[0107] Specifically, in the embodiment of the present invention, the certificate issuing and page turning module is configured to convey the card-type certificate or the booklet certificate to the information collection area in batches. For the booklet certificate, it is necessary to first turn the booklet certificate to the specified page, and then convey the turned booklet certificate to the information collection area. For example, when the booklet certificate is a booklet passport, it is necessary to first turn the booklet passport to the personal data page, and then convey the booklet passport to the information collection area for subsequent personal information comparison and quality detection. When the booklet passport is returned to the certificate issuing and page turning module after passing the film laminating quality detection, the certificate issuing and page turning module is further configured to turn the booklet passport from the personal data page to the visa page, and convey the booklet passport back to the information collection area for subsequent visa information comparison.

[0108] Specifically, in the embodiment of the present invention, the light source sub-module includes a white light source module, an ultraviolet light source module, and an infrared light source module. The document rotation sub-module is implemented by a document holding mechanism. Both the first camera module and the second camera module are implemented by industrial cameras. In the embodiment of the present invention, the white light source module, the ultraviolet light source module, and the infrared light source module respectively provide white light source, ultraviolet light source, and infrared light source. The document holding mechanism can rotate the document at multiple angles. Based on the coordinated operation of the light source sub-module, the document rotation sub-module, the first camera module, and the second camera module, it is possible to take pictures of the front and back of the document at multiple angles under the white light source, ultraviolet light source, and infrared light source to obtain the target document image. In the embodiment of the present invention, the pictures taken under each light source are saved for subsequent identification and detection. The document holding mechanism can rotate at multiple angles because there is a laser anti-counterfeiting film on the personal information page of the document that reflects light. Using angle rotation in combination with the light source is convenient for detecting defects such as incomplete document printing, incomplete film lamination of the anti-counterfeiting film, or scratches.

[0109] Specifically, in the embodiments of the present invention, the printing information recognition module uses OCR character recognition technology to obtain the document character information, the barcode recognition module uses barcode recognition technology to obtain the document barcode information, and the chip reading module supports reading in the BAC mode and connecting to the encryption machine for reading. The printing quality detection module detects the printing quality of the document surface according to the document images under different predetermined light sources and at different angles, and the film laminating quality detection module detects the film laminating quality of the document surface according to the document images under different predetermined light sources and at different angles. For the information comparison module, during the process of performing the three-way consistency detection and the four-way consistency detection, the document-making data of the applicant in the pre-obtained background database can also be added thereto. Correspondingly, the previous three-way consistency detection becomes a four-way consistency detection, and the previous four-way consistency detection becomes a five-way consistency detection.

[0110] Specifically, in the embodiments of the present invention, the model of the edge computing device is JETSON XAVIER NX. In actual implementation, according to different functional requirements, edge computing devices of the JETSON NANO, JETSON TX2 series, or JETSON AGX XAVIER series can be used for replacement. In the embodiments of the present invention, the document issuing and page turning module includes a document issuing sub-module and a page turning sub-module. The edge computing device is connected to the page turning sub-module, the light source sub-module, the document rotation sub-module, the first camera module, the second camera module, and the document retraction module through a network switch, and is connected to the document issuing sub-module, the chip reading module, and the document receiving module through a USB interface. The document quality inspection system further includes a display module, and the display module is connected to the edge computing device through an HDMI data cable.

[0111] Specifically, in the embodiments of the present invention, the operating system of the edge computing device selects the Ubuntu system, which has a GPU and can accelerate deep learning detection. Since the software operation involves various hardware device drivers, when running, a set of deployed systems is packaged and encapsulated as a Docker image, and the IP addresses and identification IDs of each quality inspection device (such as a reader and a camera) are written into the configuration file, which is convenient to modify when needed. When deploying to other devices, the generated Docker image is used for deployment in the container, which is convenient to keep the driver program versions consistent. The trained model is deployed using the ONNX model. The model supports development using frameworks such as Pytorch, TensorFlow, Paddle, MindSpore, and MXNET, and can support cross-platform deployment, supporting Windows, Linux, and Mac operating systems, and supporting development in Python, C++, and JAVA languages. The development and deployment are flexible and convenient. The model file is encrypted and compressed using the tar command. The encryption key is written into the configuration file. When using the model, the encrypted key is first read from the configuration file, the key is decrypted, and the tar file is decompressed using the key to obtain the model file for the program to use.

[0112] Further, in the embodiments of the present invention, the edge computing device implements the printing quality detection and the lamination quality detection based on a predetermined first document quality detection method;

[0113] The first document quality detection method includes:

[0114] Based on a pre-built image orientation detection model, detect whether the orientation of the input target document image is correct. If not, adjust the orientation of the target document image;

[0115] Perform OCR recognition on the target document image with a correct orientation to obtain the actual position information of the target text;

[0116] Obtain the printing offset of the target text according to the actual position information of the target text and the pre-acquired standard position information of the target text;

[0117] If the printing offset of the target text exceeds a predetermined printing offset threshold, it is determined that the target document fails the printing position offset detection;

[0118] Based on a pre-built image semantic segmentation model, detect whether the target document image that has passed the printing position offset detection contains black edges or impurities. If so, it is determined that the target document fails the black edge and impurity detection;

[0119] Based on a pre - built migration classification model, detect whether there are problems such as incomplete transfer of text and portrait, incomplete lamination, or printing ghosting in the input target certificate image that has passed the black edge and impurity detection. If so, determine that the target certificate fails the detection of incomplete transfer of text and portrait, incomplete lamination, and printing ghosting;

[0120] Based on a pre - built object detection network model, detect whether there are red bars, color patches, or color dots in the input target certificate image that has passed the detection of incomplete transfer of text and portrait, incomplete lamination, and printing ghosting. If so, determine that the target certificate fails the detection of red bars, color patches, and color dots;

[0121] Mark the target certificates that fail the detection of red bars, color patches, and color dots as unqualified in quality inspection, and mark the target certificates that pass the detection of red bars, color patches, and color dots as qualified in quality inspection.

[0122] The following provides a more detailed description of the first certificate quality detection method:

[0123] The first certificate quality detection method is a quality inspection method based on segmentation, classification, and object detection networks:

[0124] 1) For certificate orientation judgment, use ResNet or ShuffleNet to classify the image. Each number represents a certificate type and a direction. For example: 1 represents the front of a Hong Kong and Macao permit, 2 represents the front of a Hong Kong and Macao permit rotated 180 degrees, 3 represents the back of a Hong Kong and Macao permit, 4 represents the back of a Hong Kong and Macao permit rotated 180 degrees, 5 represents the front of a passport, 6 represents the front of a passport rotated 180 degrees, 7 represents the back of a passport, and 8 represents the back of a passport rotated 180 degrees. Mark the certificate photos and directions during the training stage and train the network. Use sample pictures to train the model in the early stage and deploy the model for recognition and judgment later.

[0125] 2) For OCR, use the EAST model for text detection and the CRNN network for text recognition. After detecting the position of the specified text, compare it with the position of the standard of this item of text, calculate the offset of text printing. If it is greater than the threshold or coincides with other pre - printed information, it is considered that there is a printing deviation for this item of text.

[0126] 3) For certificate quality detection: For pictures of black edges and impurities, use the image semantic segmentation network ADVENT. During the training stage, manually mark the portrait, background pattern, personal information, anti - counterfeiting film, as well as impurity and black - edge category information for training the segmentation network. The semantic segmentation model can identify the portrait, background pattern, personal information, anti - counterfeiting film, as well as impurity and black - edge category information. For the segmented impurities and black edges, determine that the certificate is unqualified. For pictures of lamination under ultraviolet light, also use a classification network to distinguish between pictures with complete lamination and incomplete lamination.

[0127] 4) For pictures with incomplete text and portrait transfer, incomplete lamination, and double images, use the transfer classification network CDAN. This image classification network is based on transfer learning and can improve the accuracy of classification. During the training phase, manually label the information of normal pictures and pictures with incomplete text and portrait printing, incomplete lamination, and printing double images for training the classification network. This network is used to distinguish normal pictures from pictures with incomplete text and portrait printing, incomplete lamination, and printing double images. For pictures with incomplete portrait and text printing and printing double images, it is determined that the certificate is unqualified. When training the model in the early stage, classify and label normal and pictures with abnormal quality, and provide them to the network for model training.

[0128] 5) For certificates with red bars, color spots, and color dots in printing, use the object detection network to label the defective situations, and use the FCOS object detection network to detect and display the defects. In the early stage of training the model, label the pictures with abnormal situations such as impurities, and provide them to the network for model training.

[0129] When training the model, since the number of samples of some defective certificates is small, use the method of automatically annotating with data augmentation to increase the number of samples. The methods of data augmentation include automatically generating impurities, color spots, and color dots at random positions on the certificate, adding reflection, brightness transformation, image size change, color saturation transformation, photo occlusion, photo stretching, photo distortion, photo cropping, photo deformation, random rotation of photo direction, color perturbation, and adding noise, etc.

[0130] As an optional method, in the embodiments of the present invention, the edge computing device implements the printing quality detection based on a predetermined second certificate quality detection method. The second certificate quality detection method includes:

[0131] Obtain the standard printing image corresponding to the target certificate image;

[0132] Perform preprocessing and feature point extraction on the standard printing image and the target certificate image;

[0133] Perform feature matching according to the feature points of the standard printing image and the feature points of the target certificate image extracted, and obtain the morphological subtraction result of the standard printing image and the target certificate image;

[0134] Perform similarity detection and defective area calculation on the morphological subtraction result;

[0135] If the detected similarity is higher than the predetermined similarity threshold and the calculated defective area is less than the predetermined defective area threshold, it is determined that the target certificate passes the printing quality detection.

[0136] The following provides a more detailed description of the second certificate quality detection method:

[0137] The second document quality detection method is a document quality inspection method based on feature point extraction:

[0138] Obtain the personal information of the document, including all printed information such as photos, names, genders, birthdays, expiration dates, etc. Generate a sample image, that is, a complete printed document, based on the personal information, and combine the background pattern of the blank document with the personal information sample image to generate a printed standard image to be compared with the actual document. The first step is to perform image feature detection, use the SURF algorithm to extract the feature points of the document image to form a feature description. The second step is to perform feature matching. Coarse screening uses the Euclidean distance as a metric, and fine screening uses the Random Sample Consensus (RANSAC) algorithm to remove outliers, that is, incorrect matching points, among the matching points. The third step is to register the standard image with the produced document image using affine transformation. Finally, perform image differencing. The incomplete printed part will appear in the image differencing result, and the result is subjected to similarity detection and contour detection and the defect area is calculated. If the comparison similarity is higher than a preset threshold and the result of morphological subtraction is less than the difference in the preset threshold area, it is determined to be a normal document; otherwise, it is determined to be a defective document. In this way, situations such as incorrect printing of portraits and text and missing printed content can be detected.

[0139] As an alternative, in the embodiments of the present invention, the edge computing device implements the printing quality detection based on a predetermined third document quality detection method;

[0140] The third document quality detection method is implemented based on a pre-constructed generative adversarial network model;

[0141] The generative adversarial network model includes:

[0142] An encoder GE(x), which is used to obtain a feature vector z based on the input target document image;

[0143] A decoder GD(z), which is used to obtain a reconstructed document image based on the input feature vector z and the chip information of the target document;

[0144] An encoder GE(x'), which is used to obtain a feature vector z' based on the input reconstructed document image;

[0145] If the vector difference between the feature vector z and the feature vector z' exceeds a predetermined vector difference threshold, it is determined that the target document fails the printing quality detection.

[0146] The following provides a more detailed description of the third document quality detection method:

[0147] The third document quality detection method is a document quality inspection method based on a generative adversarial network:

[0148] For the situation where there are few defective samples and many normal samples, consider using a generative adversarial network for quality inspection of certificates. The generative adversarial network includes:

[0149] 1) Generative network: It consists of an encoder GE(x) and a decoder GD(z). For the input image x, the vector z is obtained through the encoder GE(x). The vector z, combined with personal basic information, portrait, fingerprint and other information obtained by reading the certificate chip, passes through the decoder GD(z) to obtain the reconstructed data x' of x.

[0150] 2) Discriminator D: It determines the original image x as true and the reconstructed image x' as false, thus continuously optimizing the gap between the reconstructed image and the original image. Ideally, the reconstructed image is indistinguishable from the original image.

[0151] 3) Encoder GE(x'): It encodes the reconstructed image x' again to obtain the vector z' of the reconstructed image x'.

[0152] In the training stage, the entire model is trained with normal certificate samples. Using the personal information in the certificate chip and the scanned certificate photo as input, adding the certificate background pattern and anti-counterfeiting film image based on the personal information, and using the decoder GD(z) to generate the reconstructed photo. Continuously train to reduce the gap between the reconstructed image and the original image.

[0153] During quality inspection, the location and size of the defect are calculated using the morphological differences between the two images x and x'. When the gap between z and z' exceeds the threshold, it is determined that the certificate quality is unqualified; if it is less than the threshold, it is qualified.

[0154] As an optional method, in the embodiments of the present invention, the edge computing device implements the printing quality detection based on a predetermined fourth certificate quality detection method;

[0155] The fourth certificate quality detection method is implemented based on a pre-constructed improved generative adversarial network model;

[0156] The improved generative adversarial network model is trained based on data augmentation. The improved generative adversarial network model includes:

[0157] A reconstruction sub-network for reconstructing the input target certificate image into a certificate image without abnormalities;

[0158] A discriminant sub-network for obtaining the difference between the target certificate image and the certificate image without abnormalities, and judging whether the target certificate image passes the printing quality detection based on this difference.

[0159] The following provides a more detailed description of the fourth certificate quality detection method:

[0160] The fourth certificate quality detection method is a certificate quality inspection method based on an improved generative adversarial network:

[0161] In the case where defective samples are not easily obtained and there are many normal samples, it is necessary to use normal samples to generate abnormal samples for model training. Considering using an improved generative adversarial network for quality inspection of certificates. Among them, the improved generative adversarial network includes a reconstruction sub-network, a discriminator sub-network, and a data augmentation module. The reconstruction sub-network, discriminator sub-network, and data augmentation module are described separately as follows:

[0162] Data augmentation module: mainly used to generate defective images from normal images for model training. The data augmentation module is used to generate defects on the normal sample I image and send the generated defective picture I a to the reconstruction sub-network as input to train the reconstruction sub-network. The steps to generate defective images are as follows:

[0163] 1) First, generate a noise map P using Perlin noise.

[0164] 2) Perform a binarization operation on the noise map P according to a randomly generated threshold to obtain a mapping map M a .

[0165] 3) Obtain an abnormal texture image A from a dataset not related to the certificate.

[0166] 4) Select 2 data augmentation operations from the data augmentation operation set to perform data augmentation on the abnormal texture image A. The data augmentation operation set includes: sharpening, blurring, brightness change, color change, contrast adjustment.

[0167] 5) The training data includes the normal image I, the generated abnormal image I a and the abnormal mask image M a .

[0168] 6) The final data-augmented image consists of three parts: taking the inverse of the mapping map M a to obtain and multiplying it pointwise with the original image I; multiplying the mapping map M a pointwise with the original image I and then multiplying by the mixing coefficient 1 - β; multiplying the matrix of the data-augmented image A and the mapping map M a pointwise and then multiplying by the mixing coefficient β. β is a hyperparameter obtained through learning, and its value range is from 0.01 to 1.

[0169] The formula is as follows:

[0170]

[0171] Finally, obtain the data-augmented abnormal image I a .

[0172] Reconstruction sub-network: It consists of an encoder and a decoder. The reconstruction sub-network is used to detect anomalies in defective images. Its input is the defective image I, and the network reconstructs it into a normal image I r , making the reconstructed image I r as consistent with the normal image I as possible to reduce the image loss. The loss L rec of the reconstruction sub-network consists of two parts. One part is the smooth L1 loss (l1 smooth ), and the other part is the SSIM loss (L SSIM ). The calculation formula is as follows:

[0173]

[0174] L rec (I, I r ) = λL SSIM (I, h) + l 1smooth (I, I r )

[0175] H and W are the height and width of the image I, N p is the number of pixels of the image I, I r is the output of the network, SSIM(I, I r ) (i,j) is the SSIM (structural similarity) value of the images I and I r . λ is a hyperparameter for calculating the loss, learned by the network.

[0176] Discriminative sub-network: It locates defects by detecting the differences between the reconstructed image and the input image. The discriminative sub-network uses the U-Net architecture. The input I c of the discriminative sub-network is the output I r of the reconstruction sub-network and the input image I. The discriminative sub-network checks the difference between the images I r and I. If the input image is an abnormal image, the difference between I r and I in the abnormal area is relatively large, and the defects can be located and segmented based on the difference. The discriminative sub-network automatically learns the distance metric between I r and I. The discriminative sub-network outputs an anomaly score map M o with the same size as the image I. The loss L seg of the discriminative sub-network mainly uses the balanced cross entropy function to enhance the accuracy and robustness of anomaly segmentation.

[0177] The total loss of this method consists of two parts: the reconstruction loss and the defect segmentation loss:

[0178] L(I, I r , M a， M) = Lrec (I ， I r ) + L seg (M a , M)

[0179] M a , M are respectively the abnormal segmentation map output by the discrimination sub-network and the abnormal segmentation map with correct marking (groundtruth) in the data set.

[0180] The output M of the discrimination sub-network o is the mask map for anomaly detection. To better determine the detection result, M o is smoothed by median filtering, then passed through the max-pooling layer, and finally the detection result is determined through the fully connected layer.

[0181] Specifically, in the embodiments of the present invention, the first document quality detection method, the second document quality detection method, the third document quality detection method, and the fourth document quality detection method can be selectively combined and used, as long as the print quality detection and the lamination quality detection can be achieved.

[0182] The document quality inspection system based on artificial intelligence in the embodiments of the present invention has the following beneficial effects:

[0183] 1) Using one quality inspection device to integrate the detection of the personal information page, visa page of documents and passports, and the detection of personal information and endorsements of card-type documents. It can perform one-stop detection on the printed information of documents, including personal information, visa or endorsement information, QR code information, chip information, document printing and lamination quality, integrating multiple quality inspection functions, improving the integrity of quality inspection, and enriching the quality inspection functions.

[0184] 2) When detecting the document quality, multiple light sources are used, combined with the rotation of the holding institution at multiple angles, which is convenient for the detection of document printing and lamination quality.

[0185] 3) The quality inspection and recognition models are deployed using Docker, which can adapt to various drivers of hardware devices, keep the versions of software development libraries consistent, and at the same time facilitate the replication and expansion of the deployment environment.

[0186] 4) The model uses the ONNX format model, supports the deployment of multiple operating systems, and encrypts the model at the same time, improving the security of the program and the model.

[0187] 5) Use industry-leading self-developed deep learning methods to identify and detect using different types of network models according to different detection functions. The network model for the quality inspection function of document printing and lamination quality includes object detection, image classification, and image segmentation methods to detect document printing quality and lamination quality, replacing manual operations and improving the quality inspection efficiency. Use self-developed OCR and printing position detection methods to improve the verification ability of document printing information. Use ShuffleNET to identify the document orientation, facilitating quality inspection judgment and character recognition.

[0188] 6) Use the JETSON XAVIER NX edge computing device to replace the industrial control computer. At the same time, Jetson has graphics card acceleration capabilities, which can improve the operating speed of the quality inspection function based on deep learning, reduce equipment costs, reduce the equipment volume, and reduce the dependence on the server.

[0189] 7) Use a document quality inspection method based on an improved generative adversarial network to detect document quality defects. To address the problem of fewer defective document samples, train the detection network while improving the detection and recognition rate.

[0190] 8) It is possible to perform one-stop detection of the printing information of documents, including personal information, visa or endorsement information, QR code information, chip information, document printing and lamination quality, integrating multiple quality inspection functions, improving the integrity of quality inspection, and enriching the quality inspection functions.

[0191] 9) Use deep learning methods to detect document printing, lamination quality, and printing information, replacing manual operations and improving the quality inspection efficiency and recognition accuracy.

[0192] 10) Use the JETSON XAVIER NX edge computing device to replace the industrial control computer, reducing equipment costs and reducing the equipment volume, making the equipment convenient to carry.

[0193] 11) Solve the problem of difficult acquisition of defective document samples. Use normal document sample images combined with an abnormal image generator to generate defective images, and finally train the recognition network.

[0194] Correspondingly, based on the above artificial intelligence-based document quality inspection system, an embodiment of the present invention further proposes an artificial intelligence-based document quality inspection method. Figure 2 The implementation flowchart of the artificial intelligence-based document quality inspection method according to an embodiment of the present invention is shown. Referring to Figure 2 , the artificial intelligence-based document quality inspection method according to an embodiment of the present invention includes the following steps:

[0195] S100, Convey the card-type document to be quality inspected to the information collection area, or turn the book-type document to be quality inspected and convey it to the information collection area;

[0196] S200. Obtain an image of the document within the information collection area;

[0197] S300. Obtain the chip information of the document within the information collection area;

[0198] S400. Based on the obtained document image, obtain the printing information and barcode information of the document, where the printing information includes text information and biometric information;

[0199] S500. Perform a three-way consistency check on the chip information, the printing information, and the barcode information;

[0200] S600. Based on the obtained document image, perform a printing quality check on the document that has passed the three-way consistency check;

[0201] S700. Based on the obtained document image, perform a lamination quality check on the document that has passed the printing quality check;

[0202] S800. Convey the document that fails the three-way consistency check, the printing quality check, or the lamination quality check to the unqualified quality inspection area, and convey the document that has passed the lamination quality check and does not require further information comparison to the qualified quality inspection area.

[0203] Further, in the embodiment of the present invention, when the booklet-type document is a booklet-type passport, after performing the lamination quality check on the document that has passed the printing quality check based on the obtained document image in step S700, it further includes:

[0204] Retract the booklet-type passport that has passed the lamination quality check;

[0205] Turn the retracted booklet-type passport from the personal information page to the visa page, and convey the booklet-type passport back to the information collection area;

[0206] Obtain an image of the visa page of the booklet-type passport within the information collection area;

[0207] Based on the obtained visa page image, obtain the visa page printing information;

[0208] Perform a four-way consistency check on the visa page printing information and the previously obtained chip information, personal information page printing information, and barcode information of the booklet-type passport;

[0209] Convey the booklet-type passport that fails the four-way consistency check to the unqualified quality inspection area, and convey the booklet-type passport that has passed the four-way consistency check to the qualified quality inspection area.

[0210] Correspondingly, the embodiment of the present invention also proposes another document quality inspection system based on artificial intelligence.Figure 3 The system composition diagram of the document quality inspection system is shown. Refer to Figure 3 , the document quality inspection system includes a hardware device layer, a key technology layer, a function module layer, and an application interface layer. Among them, the hardware devices include industrial cameras, CIS scanners, multi-angle card holders, and various light source illuminations. The key technologies include OpenCV, feature extraction, object detection technology, and image registration technology. The function modules include a content missing defect detection module, a ghosting defect detection module, a film coating defect detection module, a black edge defect detection module, a sample collection module, a model training module, and a deep learning detection module. Among them, the content missing defect detection module is used to detect the content missing defect of the document image (including portrait and text); the ghosting defect detection module is used to detect the ghosting defect of the document image; the film coating defect detection module is used to detect the UV anti-counterfeiting defect of the anti-counterfeiting film of the document image. The black edge defect detection module can detect the black edge defect of the document image to be detected; the sample collection module is used to collect document images under ordinary LED and UV light sources; the model training module is used to train the defect detection network model with sample pictures; the deep learning detection module is used to detect the defect type of the document image to be detected with the defect detection network trained by deep learning algorithms. The application interfaces include: API interfaces, dynamic libraries (including dll dynamic libraries under Windows and.so libraries under Linux), and QT interface programs.

[0211] The embodiments of the present invention have been described above. The above description is exemplary and not exhaustive, and is also not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art in the technical field without departing from the scope and spirit of the described embodiments.

Claims

1. An artificial intelligence-based document quality inspection system, characterized in that, Including: A certificate-issuing and page-turning module, configured to convey the card-type certificates to be quality-inspected to the information collection area, and turn the page of the book-type certificates to be quality-inspected and convey them to the information collection area; A photographing module, configured to acquire an image of the certificate in the information collection area; A chip reading module, configured to acquire the chip information of the certificate in the information collection area; A printed information recognition module, configured to acquire the printed information of the certificate based on the acquired certificate image, where the printed information includes text information and biometric information; A barcode recognition module, configured to acquire the barcode information of the certificate based on the acquired certificate image; An information comparison module, configured to perform a three-way consistency check on the chip information, the printed information, and the barcode information; A printed quality detection module, configured to perform a printed quality detection on the certificates that pass the three-way consistency check based on the acquired certificate image; A film laminating quality detection module, configured to perform a film laminating quality detection on the certificates that pass the printed quality detection based on the acquired certificate image; A certificate collection module, configured to convey the certificates that do not pass the three-way consistency check, the printed quality detection, or the film laminating quality detection to the unqualified quality inspection area, and convey the certificates that pass the film laminating quality detection and do not require further information comparison to the qualified quality inspection area; The printed information recognition module, the barcode recognition module, the information comparison module, the printed quality detection module, and the film laminating quality detection module are all implemented based on an edge computing device; The edge computing device implements the printed quality detection and the film laminating quality detection based on a predetermined first certificate quality detection method; The first certificate quality detection method includes: Detecting whether the orientation of the input target certificate image is correct based on a pre-constructed image orientation detection model, and if not, adjusting the orientation of the target certificate image; Performing OCR recognition on the target certificate image with a correct orientation to obtain the actual position information of the target text; Obtaining the printing offset of the target text according to the actual position information of the target text and the pre-acquired standard position information of the target text; If the printing offset of the target text exceeds a predetermined printing offset threshold, determining that the target certificate fails the printing position offset detection; Detecting whether the input target certificate image that passes the printing position offset detection contains black edges or impurities based on a pre-constructed image semantic segmentation model, and if so, determining that the target certificate fails the black edge and impurity detection; Detecting whether there are problems such as incomplete transfer of text and portrait, incomplete film lamination, or printing ghosting in the input target certificate image that passes the black edge and impurity detection based on a pre-constructed transfer classification model, and if so, determining that the target certificate fails the detection of incomplete transfer of text and portrait, incomplete film lamination, and printing ghosting; Detecting whether there are red bars, color patches, or color dots in the input target certificate image that passes the detection of incomplete transfer of text and portrait, incomplete film lamination, and printing ghosting based on a pre-constructed target detection network model, and if so, determining that the target certificate fails the detection of red bars, color patches, and color dots; Mark the target certificates that fail the red stripe, color patch, and color dot detection as unqualified in quality inspection, and mark the target certificates that pass the red stripe, color patch, and color dot detection as qualified in quality inspection.

2. The artificial intelligence-based document quality inspection system according to claim 1, wherein The photographing module includes: A light source sub-module for providing white light source, ultraviolet light source, and infrared light source; A certificate rotation sub-module for rotating the certificate; A first camera module for photographing the certificate at a predetermined rotation angle from below under the white light source, the ultraviolet light source, and the infrared light source respectively; A second camera module for photographing the certificate at a predetermined rotation angle from above under the white light source, the ultraviolet light source, and the infrared light source respectively.

3. The document quality inspection system based on artificial intelligence according to claim 2, characterized in that, After passing the film covering quality inspection, the certificate that needs to undergo the next information comparison is a passport booklet; The certificate quality inspection system further includes: A certificate return module for returning the passport booklet that passes the film covering quality inspection to the certificate issuing and page turning module; The certificate issuing and page turning module is further used to turn the returned passport booklet from the personal information page to the visa page, and re-transport the passport booklet to the information collection area; The photographing module is further used to obtain the visa page image of the passport booklet in the information collection area; The printed information recognition module is further used to obtain the visa page printed information based on the obtained visa page image; The information comparison module is further used to perform a four-way consistency detection on the visa page printed information, the chip information of the passport booklet obtained previously, the personal information page printed information, and the barcode information; The certificate collection module is further used to transport the passport booklet that fails the four-way consistency detection to the unqualified quality inspection area, and transport the passport booklet that passes the four-way consistency detection to the qualified quality inspection area.

4. The artificial intelligence-based document quality inspection system according to claim 1, wherein The edge computing device also implements the print quality detection based on a predetermined second certificate quality detection method, and the second certificate quality detection method includes: Obtain the standard print image corresponding to the target certificate image; Perform preprocessing and feature point extraction on the standard print image and the target certificate image; Perform feature matching based on the feature points of the standard print image and the feature points of the target certificate image extracted, and obtain the morphological subtraction result of the standard print image and the target certificate image; Perform similarity detection and defect area calculation on the morphological subtraction result; If the detected similarity is higher than a predetermined similarity threshold and the calculated defect area is smaller than a predetermined defect area threshold, it is determined that the target certificate passes the print quality detection.

5. The artificial intelligence-based document quality inspection system according to claim 1, characterized in that, The edge computing device also implements the print quality detection based on a predetermined third certificate quality detection method; The third certificate quality detection method is implemented based on a pre-constructed generative adversarial network model; The generative adversarial network model includes: An encoder GE(x) for obtaining a feature vector z based on the input target certificate image; A decoder GD(z) for obtaining a reconstructed certificate image based on the input feature vector z and the chip information of the target certificate; An encoder GE(x') for obtaining a feature vector z' based on the input reconstructed certificate image; If the vector difference between the feature vector z and the feature vector z' exceeds a predetermined vector difference threshold, it is determined that the target document fails the print quality inspection.

6. The artificial intelligence-based document quality inspection system according to claim 1, characterized in that The edge computing device also implements the print quality inspection based on a predetermined fourth document quality inspection method; The fourth document quality inspection method is implemented based on a pre-constructed improved generative adversarial network model; The improved generative adversarial network model is trained by means of data augmentation. The improved generative adversarial network model includes: A reconstruction sub-network for reconstructing the input target document image into a document image without abnormalities; A discriminant sub-network for obtaining the difference between the target document image and the document image without abnormalities, and determining whether the target document image passes the print quality inspection based on this difference.

7. An identity document quality inspection method based on artificial intelligence, characterized in that, Implemented based on the document quality inspection system according to claim 1; The document quality inspection method includes: Conveying the card-type document to be quality inspected to the information collection area, or turning the page of the book-type document to be quality inspected and conveying it to the information collection area; Obtaining an image of the document in the information collection area; Obtaining the chip information of the document in the information collection area; Obtaining the print information and barcode information of the document based on the obtained document image, where the print information includes text information and biometric information; Performing a three-way consistency check on the chip information, the print information, and the barcode information; Performing a print quality inspection on the document that passes the three-way consistency check based on the obtained document image; Performing a lamination quality inspection on the document that passes the print quality inspection based on the obtained document image; Conveying the document that fails the three-way consistency check, the print quality inspection, or the lamination quality inspection to the non-conforming quality inspection area, and conveying the document that passes the lamination quality inspection and does not require further information comparison to the conforming quality inspection area.

8. The method for quality inspection of certificates based on artificial intelligence according to claim 7, characterized in that, When the book-type document is a book-type passport, after performing the lamination quality inspection on the document that passes the print quality inspection based on the obtained document image, it further includes: Retracting the book-type passport that passes the lamination quality inspection; Turning the retracted book-type passport from the personal information page to the visa page, and conveying the book-type passport back to the information collection area; Obtaining an image of the visa page of the book-type passport in the information collection area; Obtaining the visa page print information based on the obtained visa page image; Performing a four-way consistency check on the visa page print information, the chip information, the personal information page print information, and the barcode information of the book-type passport obtained previously; Conveying the book-type passport that fails the four-way consistency check to the non-conforming quality inspection area, and conveying the book-type passport that passes the four-way consistency check to the conforming quality inspection area.

Images