Certificate integrity detection method and device and storage medium

Abstract

The application provides a certificate integrity detection method and device and a storage medium. The method comprises the following steps: performing edge extraction on a certificate image based on a double-flow convolutional neural network (GSCNN) model to obtain an edge image of the certificate, and / or performing edge corner classification on an edge corner image of the certificate based on an edge corner detection network model to obtain an edge corner detection result; the edge image is used to determine whether the edges of the certificate belong to a closed loop; the edge corner detection result is used to determine whether the edge corners of the certificate are complete; and in the case that the edges of the certificate belong to a closed loop and the edge corners of the certificate are complete, it is determined that the certificate is complete. The certificate integrity detection method and device and the storage medium provided by the application determine whether the certificate is complete according to two automatic detection results of edge detection and edge corner detection, improve the accuracy of certificate integrity determination, improve the certificate acquisition quality, and reduce the manpower and material costs of certificate detection.

Description

Technical Field

[0001] This application relates to the field of edge detection technology, and in particular to a method, apparatus and storage medium for document integrity detection. Background Technology

[0002] With the development of internet finance, remote services provided by financial institutions have brought convenience to people. For example, remote real-person authentication can be achieved by uploading document information online, which greatly facilitates people's business and life.

[0003] However, remotely uploaded document information, such as ID card images, suffers from inconsistent lighting and scene conditions during image capture due to the lack of monitoring. This can lead to variations in resolution and edge integrity. Furthermore, the diverse range of capture devices, including mobile phones, tablets, document scanners, and more, results in numerous problems with uploaded images, many of which fail to meet regulatory requirements. While incomplete image edges have a relatively small impact on text recognition, they severely affect the credibility of the documents. Since the financial industry has extremely high requirements for the authenticity of personal information, relevant departments in the financial sector need to invest significant manpower and resources in manually inspecting uploaded document images, increasing both human and material costs. Summary of the Invention

[0004] This application provides a method, apparatus, and storage medium for document integrity detection, which solves the technical problem of high manpower and material costs in document image detection in the prior art.

[0005] In a first aspect, embodiments of this application provide a method for detecting the integrity of a document, including:

[0006] Edge extraction is performed on the document image based on the dual-stream convolutional neural network GSCNN model to obtain the edge image of the document, and / or the corner image of the document is classified based on the corner detection network model to obtain the corner detection result; the edge image is used to determine whether the edge of the document belongs to a closure; the corner detection result is used to determine whether the corner of the document is complete;

[0007] If the edges of the document are closed and the corners of the document are intact, the document is determined to be complete.

[0008] In some embodiments, the dual-stream convolutional neural network GSCNN model is trained through the following steps:

[0009] In the encoder of the GSCNN model, edge features, multi-scale semantic features, and artificial edge features of the document image are extracted, and feature fusion is performed on the edge features, multi-scale semantic features, and artificial edge features of the document image to obtain high-level edge detection features;

[0010] In the decoder of the GSCNN model, the GSCNN model is trained based on the Feature Pyramid Network (FPN) and the high-level features of edge detection.

[0011] In some embodiments, extracting edge features, multi-scale semantic features, and artificial edge features from the document image includes:

[0012] Edge features of the document image are extracted based on gate and attention mechanisms, multi-scale semantic features of the document image are extracted based on multiple residual convolutional neural networks, and artificial edge features of the document image are extracted using the Canny operator.

[0013] In some embodiments, the model parameters of the GSCNN model are optimized based on dynamic weight cross-entropy loss.

[0014] In some embodiments, the corner detection network model is built based on a lightweight neural network and the Sobel operator, and the corner detection network model is trained through the following steps:

[0015] The lightweight neural network is used to extract the mid-level features of the corner image, and the Sobel operator is used to extract the Sobel features of the corner image;

[0016] High-level semantic features are obtained based on the mid-level features and the Sobel features;

[0017] Based on the aforementioned high-level semantic features, a corner detection network model is trained using a supervised training method.

[0018] In some embodiments, the method further includes:

[0019] The integrity of the edges and corners of the document is determined by the edge and corner detection results and a preset threshold.

[0020] The corner detection result is the confidence level of corner integrity.

[0021] In some embodiments, the method further includes:

[0022] Obtain a scene image containing the identification documents;

[0023] The document detection model is used to obtain the position coordinates of the document in the scene graph;

[0024] The document image is cropped from the scene image based on the location coordinates;

[0025] The document image is cropped at the edges and corners to obtain four original edge and corner images;

[0026] The four original corner images obtained from the cutting are rotated and scaled to obtain the normalized corner images of the document.

[0027] Secondly, embodiments of this application provide a document integrity detection device, comprising:

[0028] The first acquisition module is used to extract edges from the document image based on the dual-stream convolutional neural network GSCNN model to obtain the edge image of the document, and / or to classify the corner image of the document based on the corner detection network model to obtain the corner detection result; the edge image is used to determine whether the edge of the document belongs to a closure; the corner detection result is used to determine whether the corner of the document is complete;

[0029] The determination module is used to determine that the document is complete if the edges of the document are closed and the corners of the document are intact.

[0030] In some embodiments, the dual-stream convolutional neural network GSCNN model is trained through the following steps:

[0031] In the encoder of the GSCNN model, edge features, multi-scale semantic features, and artificial edge features of the document image are extracted, and feature fusion is performed on the edge features, multi-scale semantic features, and artificial edge features of the document image to obtain high-level edge detection features;

[0032] In the decoder of the GSCNN model, the GSCNN model is trained based on the Feature Pyramid Network (FPN) and the high-level features of edge detection.

[0033] In some embodiments, extracting edge features, multi-scale semantic features, and artificial edge features from the document image includes:

[0034] Edge features of the document image are extracted based on gate and attention mechanisms, multi-scale semantic features of the document image are extracted based on multiple residual convolutional neural networks, and artificial edge features of the document image are extracted using the Canny operator.

[0035] In some embodiments, the model parameters of the GSCNN model are optimized based on dynamic weight cross-entropy loss.

[0036] In some embodiments, the corner detection network model is built based on a lightweight neural network and the Sobel operator, and the corner detection network model is trained through the following steps:

[0037] The lightweight neural network is used to extract the mid-level features of the corner image, and the Sobel operator is used to extract the Sobel features of the corner image;

[0038] High-level semantic features are obtained based on the mid-level features and the Sobel features;

[0039] Based on the aforementioned high-level semantic features, a corner detection network model is trained using a supervised training method.

[0040] In some embodiments, the method further includes:

[0041] The judgment module is used to determine whether the corners of the document are complete based on the corner detection results and a preset threshold.

[0042] The corner detection result is the confidence level of corner integrity.

[0043] In some embodiments, the method further includes:

[0044] The second acquisition module is used to acquire a scene image containing the documents;

[0045] The third acquisition module is used to obtain the position coordinates of the document in the scene map using the document detection model;

[0046] A cropping module is used to crop the document image from the scene image based on the position coordinates;

[0047] The fourth acquisition module is used to cut the corners of the document image to obtain four original corner images;

[0048] The normalization module is used to rotate and scale the four original corner images obtained from the cutting process to obtain the normalized corner images of the document.

[0049] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the document integrity detection method as described in the first aspect above.

[0050] Fourthly, embodiments of this application also provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the document integrity detection method as described in the first aspect above.

[0051] Fifthly, embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the document integrity detection method as described in the first aspect above.

[0052] The document integrity detection method, apparatus, and storage medium provided in this application embodiment utilize a dual-stream convolutional neural network for edge detection, and determine whether the document edges belong to a closed loop based on the edge detection results; and / or utilize a corner detection network model for document corner classification, and determine whether the document corners are complete based on the corner detection results; when both the edges belong to a closed loop and the corners are complete, the document is determined to be complete, which improves the accuracy of document integrity detection, increases the credibility of the document, and solves the problem of high manpower and material costs in existing document detection methods. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart illustrating the document integrity detection method provided in the embodiments of this application;

[0055] Figure 2 This is a flowchart illustrating the document integrity detection process in an example scenario provided in this application's embodiments;

[0056] Figure 3 This is a schematic diagram of the edge detection network structure in an example scenario provided in this application embodiment;

[0057] Figure 4 This is a schematic diagram of the edge detection network structure for an example scenario provided in this application embodiment;

[0058] Figure 5 This is a schematic diagram of the structure of a document integrity detection device provided in an embodiment of this application;

[0059] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0061] Figure 1This is a flowchart illustrating the document integrity detection method provided in this application embodiment, as shown below. Figure 1 As shown in the figure, this application provides a method for detecting the integrity of a document. The method includes:

[0062] Step 101: Extract edges from the document image based on the dual-stream convolutional neural network GSCNN model to obtain the edge image of the document, and / or classify the corner images of the document based on the corner detection network model to obtain the corner detection results; the edge image is used to determine whether the edges of the document belong to a closure; the corner detection results are used to determine whether the corners of the document are complete.

[0063] Specifically, in this embodiment, a two-stream convolutional neural network (Gated ShapeConvolutional Neural Networks, GSCNN) model is used to perform edge detection / extraction on the document image. GSCNN mainly consists of an encoder and a decoder. The output of the GSCNN model is the edge image of the document, which is a binary image. Based on this binary image, the outline of the document can be obtained. After obtaining the edge image, the outline is extracted, for example, using the findContours function of OpenCV. Then, it is determined whether the outline belongs to a closure.

[0064] The corner detection network model is used to detect whether the corners in a corner image are complete, and outputs the corner detection result, which can be the probability information of corner completeness. It can also be used to determine whether the corners of an document are complete after obtaining the corner detection result.

[0065] Among them, "complete document" means that the edges of the document are closed and all four corners of the document are intact.

[0066] Optionally, it is not necessary to perform both the action of obtaining the edge image of the document and the action of obtaining the corner detection result.

[0067] For example, after obtaining the edge image of the document based on the GSCNN model, the edge image is contour extracted to obtain the edge contour of the document. It is then determined whether the contour / edge belongs to a closure. If it does, the corner detection network model is used to obtain the corner detection results of the document. Based on the corner detection results, it is determined whether the corners of the document are complete, thereby further determining whether the document is complete. If it does not belong to a closure, there is no need to perform corner detection to obtain the corner detection results.

[0068] For example, after obtaining the corner detection results of the document based on the corner detection network model, it is determined whether the corners of the document are complete based on the corner detection results. If they are complete, the edge image of the document is obtained based on the GSCNN model, and the contour of the edge image of the document is extracted. It is determined whether the contour belongs to a closure, thereby further determining whether the document is complete. If the corners of the document are incomplete, it is not necessary to obtain the edge image of the document.

[0069] For example, the edge image of the document is obtained based on the GSCNN model, and the corner detection results of the document are obtained based on the corner detection network model. Contour extraction is performed on the edge image to obtain the edge contour of the document. It is determined whether the contour / edge belongs to a closure, and the integrity of the document's corners is determined based on the corner detection results. Based on the results of whether the edge contour belongs to a closure and whether the corners are complete, the integrity of the document is further determined.

[0070] Step 102: If the edges of the document are closed and the corners of the document are intact, then the document is determined to be complete.

[0071] Specifically, the integrity of a document is determined based on the results of judging whether the edges of the document belong to a closed loop and whether the corners of the document are complete. Only when the edges of the document belong to a closed loop and the corners of the document are complete can the document be determined to be complete.

[0072] For example, Figure 2 This is a flowchart illustrating the document integrity detection process in an example scenario provided in this application's embodiments, such as... Figure 2 As shown, after obtaining the document image and / or corner images through document detection and cropping, edge detection is performed on the document image to obtain the document's edge image. It is then determined whether the document's edges belong to a closed closure. If not, the result of an incomplete document is output; if the edges belong to a closed closure, corner detection is performed on the corner image. The corner detection results are then used to determine whether the document's corners are complete. If any one or more corners are incomplete, the result of an incomplete document is output; if all four corners are complete, the result of a complete document is output.

[0073] For example, we can first perform corner detection on the corner image to determine whether the corners are complete. If the corners are incomplete, the result of incomplete document is output. If the corners are complete, we can continue to perform edge detection on the document image to determine whether the edge of the document belongs to a closure. If it does not belong to a closure, the result of incomplete document is output; if the edge belongs to a closure, the result of complete document is output.

[0074] For example, corner detection can be performed on the corner image and edge detection on the document image simultaneously, yielding the document's edge image and corner detection results respectively. Then, it is determined whether the document's edges belong to a closed loop and whether the corners are intact. If both the edges belong to a closed loop and the corners are intact, the result of an intact document is output.

[0075] The document integrity detection method provided in this application uses a dual-stream convolutional neural network to perform edge detection on the document image, and determines whether the document edges belong to a closed closure based on the edge detection results; and / or uses a corner detection network model to classify the corners of the corner image, and determines whether the corners of the document are complete based on the corner detection results; if the edges belong to a closed closure and the corners are complete, the document is determined to be complete, which improves the accuracy of document integrity detection, improves the quality of document acquisition, ensures the accuracy of text recognition on the document, improves the credibility of the document, and solves the problem of high manpower and material costs in existing document detection methods.

[0076] In some embodiments, the method further includes:

[0077] Obtain a scene image containing the identification documents;

[0078] The document detection model is used to obtain the position coordinates of the document in the scene graph;

[0079] The document image is cropped from the scene image based on the location coordinates;

[0080] The document image is cropped at the edges and corners to obtain four original edge and corner images;

[0081] The four original corner images obtained from the cutting are rotated and scaled to obtain the normalized corner images of the document.

[0082] Specifically, before performing edge detection, a scene image containing the document is first obtained. The document detection model is used to obtain the position coordinates of the document in the scene image. Then, the document image is cropped based on the position coordinates and perspective transformation.

[0083] For example, a scene image containing the identification document is obtained. An identification document detection model is used to obtain the document's position coordinates within the scene image. If the output of the detection model is empty (None), the process ends or a new scene image containing the document is obtained for detection. Based on the obtained position coordinates, a perspective transformation is used to crop the document image. Furthermore, an identification document orientation model is used to rotate the image according to the target orientation, resulting in a rotated image.

[0084] Before performing edge detection / classification, the obtained document image is cropped at its edges to obtain four original edge images. These four original edge images are then rotated and scaled to obtain the normalized edge images of the document.

[0085] For example, after obtaining the image of the document, the image is cropped at the edges and corners, and different angles are rotated for different edges and corners. Then, the rotated image is uniformly scaled to a size of 64×64 to obtain a normalized edge and corner image.

[0086] The document integrity detection method provided in this application obtains document images by accurately cropping the scene image, and normalizes the corner images based on the document images using operations such as rotation and scaling, thereby improving the image quality of the document images and corner images, thus improving the accuracy of edge detection and edge inspection classification, improving the accuracy of document integrity judgment, and ensuring the credibility of the collected documents.

[0087] In some embodiments, the dual-stream convolutional neural network GSCNN model is trained through the following steps:

[0088] In the encoder of the GSCNN model, edge features, multi-scale semantic features, and artificial edge features of the document image are extracted, and feature fusion is performed on the edge features, multi-scale semantic features, and artificial edge features of the document image to obtain high-level edge detection features;

[0089] In the decoder of the GSCNN model, the GSCNN model is trained based on the Feature Pyramid Network (FPN) and the high-level features of edge detection.

[0090] Specifically, the GSCNN model mainly consists of an encoder and a decoder. In the encoder of the GSCNN model, a multi-branch neural network extracts various features from the document image, including edge features and multi-scale semantic features. Then, the `concat()` function is used to fuse these features with artificial edge features (i.e., Canny features) to obtain high-level edge detection features. During model inference, a pre-trained multi-branch neural network is used to extract mid-level semantic features from the document image.

[0091] In the decoder of the GSCNN model, a Feature Pyramid Network (FPN) is mainly used to fuse high-level edge detection features of different sizes, and the fused features are then used for classification training. After the model training is complete, the trained GSCNN model is used to generate edge images during the model inference stage.

[0092] The document integrity detection method provided in this application extracts shape, content, semantic and other features from document images through a multi-branch network and fuses them (concat) to form new high-level features. For feature maps of different sizes, the PFN feature fusion method is adopted to make full use of more position and detail information from the bottom edge map and more robust semantic information from the high-level features, so that the model learns clearer edges.

[0093] In some embodiments, extracting edge features, multi-scale semantic features, and artificial edge features from the document image includes:

[0094] Edge features of the document image are extracted based on gate and attention mechanisms, multi-scale semantic features of the document image are extracted based on multiple residual convolutional neural networks, and artificial edge features of the document image are extracted using the Canny operator.

[0095] Specifically, the encoder of the GSCNN model contains at least two branches: a document feature extraction branch and an edge feature extraction branch. The document feature extraction branch is mainly composed of multiple residual convolutional neural networks, used to extract high-level multi-scale semantic features of the document image; the edge feature extraction branch is mainly designed based on gate control and attention mechanisms, used to extract edge features of the document image.

[0096] Specifically, artificial edge features, or Canny features, do not need to be calculated separately during model training and inference; they can all be generated online. During the model training phase, the Canny operator is used in GSCNN to directly extract traditional low-level artificial edge features.

[0097] The document integrity detection method provided in this application learns the edge features of document images through gate and attention mechanisms, and extracts high-level semantic features using multiple residual convolutional neural networks. The extracted edge features, high-level semantic features, and artificial edge features are fused to obtain high-level edge detection features with richer and more comprehensive information, thereby improving the accuracy of model training and the accuracy of model edge recognition.

[0098] In some embodiments, the model parameters of the GSCNN model are optimized based on dynamic weight cross-entropy loss.

[0099] Specifically, since edge image generation is a pixel-level classification task (classifying as edge or non-edge), it may lead to a significant imbalance in the pixel ratio between edges and non-edges. To address this issue, a dynamic weighted cross-entropy loss function is designed for iterative training of the model. The expression for the dynamic weighted cross-entropy loss function is:

[0100]

[0101] Among them, F ce (G,P) represents the dynamic weighted cross-entropy loss; W I This represents the weight of edge pixels relative to the total number of pixels; N represents the number of pixels in the document image, and N is a positive integer; G i p represents the actual probability distribution of the i-th pixel belonging to the edge; i This represents the reference probability distribution that the i-th pixel belongs to the edge.

[0102] The document integrity detection methods provided in the above embodiments are further illustrated below with specific examples:

[0103] Example 1:

[0104] Figure 3 This is a schematic diagram of the edge detection network structure in an example scenario provided in this application embodiment, such as... Figure 3 As shown, the specific steps for training and inference of the edge detection network model (i.e., the GSCNN model) are as follows:

[0105] Step 1: Obtain document data: First, obtain the scene image containing the document. Using the detection model, obtain the coordinates of the four vertices (Boxes) of the document / card in the scene image. Based on the vertices (Boxes), use perspective transformation to crop the image and scale the cropped image to the specified size to obtain the document image.

[0106] Step 2: Mark the edges of multiple document images and divide them into a training set and a validation set with a 9:1 ratio.

[0107] Step 3: Obtain Canny features: During the training process, traditional low-level artificial edge features (i.e., Canny features) are directly extracted from the network. These features do not need to be calculated separately during training and inference; they can all be generated online as low-level artificial edge features.

[0108] Step 4: During the training phase, the document image is processed using neural networks with different branches to extract edge features and mid-level multi-scale semantic features (segmentation) of the document. The document feature extraction branch of the encoder contains multiple residual convolutional modules (ResBlock), while the edge feature extraction branch contains gate control and a unique attention module. During inference, the pre-trained multi-branch neural network is directly used to extract the mid-level semantic features of the document image. It should be noted that steps 3 and 4 above are not sequential.

[0109] Step 5: During the training phase, extract the low-level artificial edge features as in Step 3. Then, fuse these features with the edge features (Edge) and multi-scale semantic features (Segmentation) extracted in Step 4 after convolution. Upsample the fused features using FPN in a pyramidal manner, and calculate the loss and gradient for Edge and Segmentation respectively. During the inference phase, use the trained parameters to directly output the edge image.

[0110] In some embodiments, the corner detection network model is built based on a lightweight neural network and the Sobel operator, and the corner detection network model is trained through the following steps:

[0111] The lightweight neural network is used to extract the mid-level features of the corner image, and the Sobel operator is used to extract the Sobel features of the corner image;

[0112] High-level semantic features are obtained based on the mid-level features and the Sobel features;

[0113] Based on the aforementioned high-level semantic features, a corner detection network model is trained using a supervised training method.

[0114] Specifically, a corner detection network model is used to classify the corners of the document's corner images. This corner detection network model is built based on a lightweight neural network and the Sobel operator.

[0115] In this embodiment of the application, the lightweight neural network can be composed of modules such as the most basic Depthwise Separable Convolution (DepthSepConv), the ordinary Convolutional Network Conv2d, and the Channel Attention SE (Squeeze Excitation).

[0116] The lightweight neural network uses the H-Swish activation function, which reduces a significant amount of exponential computation. To maintain a balance between speed and accuracy, the last two layers of the lightweight neural network primarily employ SE modules and networks with 5×5 convolutional kernels; in the preceding layers, each layer uses a convolutional kernel of a specific size; and for the final fully connected layer (FC layer), a network with 1×1 convolutional kernels is used instead, which greatly enhances the model's robustness.

[0117] This lightweight neural network is used to extract mid-level features from corner images, and the Sobel operator is used to extract the edge contours of the corners in the corner images to obtain Sobel features. The Sobel features are fused with the mid-level features of the corner images, and high-level semantic features are extracted from the fused mid-level features. Classification is performed based on these high-level semantic features, and the model is trained under supervision.

[0118] The document integrity detection method provided in this application utilizes a lightweight neural network to extract mid-level features from document corner images, which can fully learn the contour information of the corners. It also fuses Sobel features and mid-level features to further extract high-level semantic features to train the model, thereby improving the accuracy of corner detection network model training and thus improving the accuracy of corner classification.

[0119] The document integrity detection method provided in the above embodiments is further illustrated below with specific examples:

[0120] Example 2:

[0121] Figure 4 This is a schematic diagram of the corner detection network structure in an example scenario provided in this application embodiment, as shown below. Figure 4 As shown, the specific steps for training and inference of the corner detection network model are as follows:

[0122] Step 1: Obtain corner images: Crop the corners according to 0.15 times the shortest side length of the card in the ID image (this multiple is set according to the actual situation and is not limited here), to obtain four corner images: top left, top right, bottom right, and bottom left. Then rotate the top right image by -90°, the bottom right image by 180°, and the bottom left image by 90°, while leaving the top left image unchanged. Then scale the corner images to a uniform size of 64×64 (this size value is set according to the actual situation and is not limited here).

[0123] Step 2: Manually classify and label the corner images, clearly labeling each image with complete or incomplete tags.

[0124] Step 3: Obtain Sobel features: During the training process, traditional low-level artificial features (i.e., Sobel features) are directly extracted from the network. No separate calculation is required during training and inference, and low-level artificial features can be generated online.

[0125] Step 4: In the training phase, extract the low-level artificial edge features according to step (3), and fuse them with the mid-level features of the corner image extracted by the lightweight neural network after convolution. Further extract high-level semantic features, form a classifier through supervised training, and obtain the trained corner detection network model. In the inference phase, the corner detection network model outputs confidence, and judges whether the corner is complete or incomplete by setting a threshold.

[0126] In some embodiments, determining whether the corners of the document are complete based on the corner detection results includes:

[0127] The integrity of the edges and corners of the document is determined by the edge and corner detection results and a preset threshold.

[0128] The corner detection result is the confidence level of corner integrity.

[0129] Specifically, the corner image of the document is input into the trained corner detection network model, which outputs the confidence level of the event that the corner of the document is complete. Then, the completeness or incompleteness of the corner is determined based on the confidence level and a preset threshold.

[0130] For example, if the corner image of a document is input into a trained corner detection network model, and the confidence level of the event that the corner of the document is complete is 0.6, and this confidence level is less than the preset threshold of 0.8, then the document is judged to have incomplete corners.

[0131] For example, if the corner image of a document is input into a trained corner detection network model, and the confidence level of the event that the corner of the document is complete is 0.9, and the confidence level exceeds the preset threshold of 0.8, then the corner of the document is determined to be complete.

[0132] The document integrity detection method provided in this application further determines whether the corners of the document are intact by using confidence level and preset threshold, thereby improving the accuracy of corner integrity determination, which in turn improves the accuracy of document integrity determination, enhances the quality of collected documents, ensures document credibility, and helps improve the accuracy of text recognition. The document integrity detection method provided in this application reduces manpower and material costs and the errors of manual inspection through automated integrity detection.

[0133] Figure 5 This is a schematic diagram of the structure of a document integrity detection device provided in an embodiment of this application, as shown below. Figure 5 As shown in the figure, this application embodiment provides a document integrity detection device, including a first acquisition module 501 and a determination module 502.

[0134] The first acquisition module 501 is used to extract edges from the document image based on the dual-stream convolutional neural network GSCNN model to obtain the edge image of the document, and / or to classify the corner image of the document based on the corner detection network model to obtain the corner detection result; the edge image is used to determine whether the edge of the document belongs to a closure; the corner detection result is used to determine whether the corner of the document is complete.

[0135] The determining module 502 is used to determine that the document is complete if the edges of the document are closed and the corners of the document are intact.

[0136] In some embodiments, the dual-stream convolutional neural network GSCNN model is trained through the following steps:

[0137] In the encoder of the GSCNN model, edge features, multi-scale semantic features, and artificial edge features of the document image are extracted, and feature fusion is performed on the edge features, multi-scale semantic features, and artificial edge features of the document image to obtain high-level edge detection features;

[0138] In the decoder of the GSCNN model, the GSCNN model is trained based on the Feature Pyramid Network (FPN) and the high-level features of edge detection.

[0139] In some embodiments, extracting edge features, multi-scale semantic features, and artificial edge features from the document image includes:

[0140] Edge features of the document image are extracted based on gate and attention mechanisms, multi-scale semantic features of the document image are extracted based on multiple residual convolutional neural networks, and artificial edge features of the document image are extracted using the Canny operator.

[0141] In some embodiments, the model parameters of the GSCNN model are optimized based on dynamic weight cross-entropy loss.

[0142] In some embodiments, the corner detection network model is built based on a lightweight neural network and the Sobel operator, and the corner detection network model is trained through the following steps:

[0143] The lightweight neural network is used to extract the mid-level features of the corner image, and the Sobel operator is used to extract the Sobel features of the corner image;

[0144] High-level semantic features are obtained based on the mid-level features and the Sobel features;

[0145] Based on the aforementioned high-level semantic features, a corner detection network model is trained using a supervised training method.

[0146] In some embodiments, the method further includes:

[0147] The judgment module is used to determine whether the corners of the document are complete based on the corner detection results and a preset threshold.

[0148] The corner detection result is the confidence level of corner integrity.

[0149] In some embodiments, the method further includes:

[0150] The second acquisition module is used to acquire a scene image containing the documents;

[0151] The third acquisition module is used to obtain the position coordinates of the document in the scene map using the document detection model;

[0152] A cropping module is used to crop the document image from the scene image based on the position coordinates;

[0153] The fourth acquisition module is used to cut the corners of the document image to obtain four original corner images;

[0154] The normalization module is used to rotate and scale the four original corner images obtained from the cutting to obtain the normalized corner images of the document.

[0155] Specifically, the document integrity detection device provided in this application embodiment can implement all the method steps implemented in the document integrity detection method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0156] It should be noted that the division of units / modules in the above embodiments of this application is illustrative and only represents one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0157] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 6As shown, the electronic device may include: a processor 601, a communication interface 602, a memory 603, and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other via the communication bus 604. The processor 601 can call logical instructions in the memory 603 to execute a document integrity detection method, which includes:

[0158] Edge extraction is performed on the document image based on the dual-stream convolutional neural network GSCNN model to obtain the edge image of the document, and / or the corner image of the document is classified based on the corner detection network model to obtain the corner detection result; the edge image is used to determine whether the edge of the document belongs to a closure; the corner detection result is used to determine whether the corner of the document is complete;

[0159] If the edges of the document are closed and the corners of the document are intact, the document is determined to be complete.

[0160] Specifically, the processor 601 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a complex programmable logic device (CPLD). The processor may also adopt a multi-core architecture.

[0161] When the logical instructions in memory 603 can be implemented as software functional units and sold or used as independent products, they can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0162] In some embodiments, a computer program product is also provided, the computer program product including a computer program that can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the document integrity detection method provided in the above-described method embodiments, the method including:

[0163] Edge extraction is performed on the document image based on the dual-stream convolutional neural network GSCNN model to obtain the edge image of the document, and / or the corner image of the document is classified based on the corner detection network model to obtain the corner detection result; the edge image is used to determine whether the edge of the document belongs to a closure; the corner detection result is used to determine whether the corner of the document is complete;

[0164] If the edges of the document are closed and the corners of the document are intact, the document is determined to be complete.

[0165] Specifically, the computer program product provided in this application embodiment can implement all the method steps implemented in the above method embodiments and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.

[0166] In some embodiments, a computer-readable storage medium is also provided, the computer-readable storage medium storing a computer program, the computer program being used to cause a computer to execute the document integrity detection method provided in the above method embodiments, the method comprising:

[0167] Edge extraction is performed on the document image based on the dual-stream convolutional neural network GSCNN model to obtain the edge image of the document, and / or the corner image of the document is classified based on the corner detection network model to obtain the corner detection result; the edge image is used to determine whether the edge of the document belongs to a closure; the corner detection result is used to determine whether the corner of the document is complete;

[0168] If the edges of the document are closed and the corners of the document are intact, the document is determined to be complete.

[0169] Specifically, the computer-readable storage medium provided in the embodiments of this application can implement all the method steps implemented in the above method embodiments and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.

[0170] It should be noted that the computer-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic storage (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical storage (e.g., CD, DVD, BD, HVD), and semiconductor storage (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0171] It should also be noted that the terms "first," "second," etc., used in the embodiments of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, and the number of objects is not limited. For example, the first object can be one or more.

[0172] In the embodiments of this application, the term "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following associated objects have an "or" relationship.

[0173] In the embodiments of this application, the term "multiple" refers to two or more, and other quantifiers are similar.

[0174] In this application, "determining B based on A" means that factor A must be considered when determining B. It is not limited to "B can be determined based solely on A," but should also include: "determining B based on A and C," "determining B based on A, C, and E," "determining C based on A, and further determining B based on C," etc. It can also include using A as a condition for determining B, for example, "when A satisfies the first condition, B is determined using the first method"; or "when A satisfies the second condition, B is determined," or "when A satisfies the third condition, B is determined based on the first parameter," etc. Of course, it can also be a condition where A is a factor in determining B, for example, "when A satisfies the first condition, C is determined using the first method, and B is further determined based on C," etc.

[0175] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0176] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-executable instructions. These computer-executable instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0177] These processor-executable instructions may also be stored in a processor-readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the processor-readable memory produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0178] These processors can execute instructions that can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0179] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for detecting the integrity of a document, characterized in that, include: Before performing edge extraction on the document image based on the dual-stream convolutional neural network GSCNN model, perspective transformation correction is performed on the document image; Edge extraction is performed on the document image based on the dual-stream convolutional neural network GSCNN model to obtain the edge image of the document, and / or the corner image of the document is classified based on the corner detection network model to obtain the corner detection result; the edge image is used to determine whether the edge of the document belongs to a closure; the corner detection result is used to determine whether the corner of the document is complete; If the edges of the document are closed and the corners of the document are intact, the document is determined to be complete. The dual-stream convolutional neural network (GSCNN) model is trained through the following steps: In the encoder of the GSCNN model, edge features, multi-scale semantic features, and artificial edge features of the document image are extracted, and feature fusion is performed on the edge features, multi-scale semantic features, and artificial edge features of the document image to obtain high-level edge detection features; In the decoder of the GSCNN model, the GSCNN model is trained based on the feature pyramid network (FPN) and the high-level edge detection features. The model parameters of the GSCNN model are optimized based on dynamic weight cross-entropy loss; The step of extracting edge features, multi-scale semantic features, and artificial edge features of the document image includes: extracting edge features of the document image based on Gate gating and attention mechanisms, extracting multi-scale semantic features of the document image based on multiple residual convolutional neural networks, and extracting artificial edge features of the document image using the Canny operator. The corner detection network model is constructed based on a lightweight neural network and the Sobel operator, and is trained through the following steps: extracting mid-level features of the corner image using the lightweight neural network and extracting Sobel features of the corner image using the Sobel operator; obtaining high-level semantic features based on the mid-level features and the Sobel features; and training the corner detection network model based on the high-level semantic features and through supervised training. The lightweight neural network on which the corner detection network model is constructed uses the H-Swish function as its activation function. Furthermore, the last two layers of the lightweight neural network use SE modules and 5×5 convolutional kernels. Each layer in the preceding layers uses a convolutional kernel of a certain size, and the final fully connected layer is replaced with a network using 1×1 convolutional kernels.

2. The document integrity detection method according to claim 1, characterized in that, The method further includes: The integrity of the edges and corners of the document is determined by the edge and corner detection results and a preset threshold. The corner detection result is the confidence level of corner integrity.

3. The document integrity detection method according to claim 1, characterized in that, The method further includes: Obtain a scene image containing the identification documents; The document detection model is used to obtain the position coordinates of the document in the scene graph; The document image is cropped from the scene image based on the location coordinates; The document image is cropped at the edges and corners to obtain four original edge and corner images; The four original corner images obtained from the cutting are rotated and scaled to obtain the normalized corner images of the document.

4. A document integrity detection device, characterized in that, include: The first acquisition module is used to extract edges from the document image based on the dual-stream convolutional neural network GSCNN model to obtain the edge image of the document, and / or to classify the corner image of the document based on the corner detection network model to obtain the corner detection result; the edge image is used to determine whether the edge of the document belongs to a closure; the corner detection result is used to determine whether the corner of the document is complete; The determination module is used to determine that the document is complete if the edges of the document are closed and the corners of the document are intact; The dual-stream convolutional neural network (GSCNN) model is trained through the following steps: In the encoder of the GSCNN model, edge features, multi-scale semantic features, and artificial edge features of the document image are extracted, and feature fusion is performed on the edge features, multi-scale semantic features, and artificial edge features of the document image to obtain high-level edge detection features; In the decoder of the GSCNN model, the GSCNN model is trained based on the feature pyramid network (FPN) and the high-level edge detection features; The extraction of edge features, multi-scale semantic features, and artificial edge features of the document image includes: extracting edge features of the document image based on gate and attention mechanisms, extracting multi-scale semantic features of the document image based on multiple residual convolutional neural networks, and extracting artificial edge features of the document image using the Canny operator; The corner detection network model is constructed based on a lightweight neural network and the Sobel operator, and is trained through the following steps: extracting mid-level features of the corner image using the lightweight neural network and extracting Sobel features of the corner image using the Sobel operator; obtaining high-level semantic features based on the mid-level features and the Sobel features; and training the corner detection network model based on the high-level semantic features and through supervised training. The lightweight neural network on which the corner detection network model is constructed uses the H-Swish function as its activation function. Furthermore, the last two layers of the lightweight neural network use SE modules and 5×5 convolutional kernels. Each layer in the preceding layers uses a convolutional kernel of a certain size, and the final fully connected layer is replaced with a network using 1×1 convolutional kernels.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the document integrity detection method as described in any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the document integrity detection method as described in any one of claims 1 to 3.

Images