Certificate classification method, electronic device and computer program product

Abstract

The embodiment of the invention provides a certificate classification method, electronic equipment and a computer program product. The method comprises the steps of obtaining a to-be-detected certificate image; extracting image features of the to-be-detected certificate image; performing classification detection on the to-be-detected certificate image based on the image features by using each classification head in the classification network to obtain an initial classification result corresponding to each classification head; and for the initial classification result corresponding to any target classification head, based on the initial classification result corresponding to the target classification head, the certificate image category corresponding to the target classification head and the hierarchical relationship between the classification heads, determining a final classification result of the to-be-detected certificate image. The target classification heads refer to other classification heads except the first level. The multi-level classification head is arranged, so that the problem of insufficient fine feature recognition in a traditional method is solved, the accurate distinguishing capability of the certificates is enhanced, the feature confusion problem is avoided, and the accuracy and reliability of the certificate classification result in a complex scene are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically to a method for classifying documents, an electronic device, a storage medium, and a computer program product. Background Technology

[0002] In current applications of computer vision, document classification tasks play a crucial role in scenarios such as identity verification, security management, and information processing.

[0003] Existing technologies primarily utilize deep learning models to classify documents into multiple categories. For example, document images are input into a convolutional neural network model to distinguish different types of ID cards, such as regular ID cards, temporary ID cards, photocopies, and screen photographs. However, the image quality of original document images deteriorates after processing such as adding watermarks, editing, or photographing. In various complex scenarios, traditional deep learning models lack the ability to distinguish subtle details such as watermarks and editing marks, leading to a significant decrease in classification accuracy.

[0004] Therefore, a new technical solution is urgently needed to solve the above-mentioned technical problems. Summary of the Invention

[0005] This application is made in consideration of the above-mentioned problems. This application provides a method for classifying documents, including:

[0006] Obtain the image of the document to be tested;

[0007] Extract image features from the image of the document to be tested;

[0008] The classification network uses each classification head to classify and detect the image of the document to be tested based on image features, so as to obtain the initial classification result corresponding to each classification head. At least some of the classification heads in the classification network have a hierarchical relationship set in order of their classification granularity from coarse to fine.

[0009] For any initial classification result corresponding to a target classification head, the final classification result of the document image to be tested is determined based on the initial classification result corresponding to the target classification head, the document image category targeted by the target classification head, and the hierarchical relationship between the various classification heads. The target classification head refers to the other classification heads except for the first level.

[0010] For example, each classification head in the classification network includes an independent classification head and multiple second classification heads with hierarchical relationships, and the independent classification head includes a watermark detection classification head;

[0011] The second classification header includes at least one of the following: a specification detection classification header, an attack detection classification header, an editing detection classification header, and a copy detection classification header;

[0012] The classification heads for standard document detection, attack detection, editing, and photocopy detection are hierarchically arranged according to their classification granularity, from coarse to fine. The attack detection classification head is the next level down from the standard document detection classification head and is used to classify non-standard document images included in the initial classification result of the standard document detection classification head. The editing detection classification head is the next level down from the attack detection classification head and is used to classify edited document images included in the initial classification result of the attack detection classification head. The photocopy detection classification head is the next level down from the attack detection classification head and is used to classify document photocopy images included in the initial classification result of the attack detection classification head.

[0013] For example, when the second classification head includes a canonical detection classification head and an attack detection classification head, the classification head in the classification network is used to classify and detect the image of the document under test based on image features to obtain the initial classification result corresponding to each classification head, including:

[0014] Image features are input into the standard detection and classification head to determine the first probability that the image to be tested belongs to a standard image and the second probability that it belongs to a non-standard image. A standard image is an image whose content, format and quality all meet the corresponding preset conditions.

[0015] Image features are input into the watermark detection classification head to determine the third probability that the image to be tested belongs to a watermarked image and the fourth probability that it belongs to an unwatermarked image.

[0016] Image features are input into the attack detection classification head to determine the probability that the image of the document to be tested belongs to each of the various attack types, including at least one of image editing, temporary documents, screen capture, and photocopying.

[0017] For example, in cases where the attack type includes image editing, the second classification head also includes an edit detection classification head.

[0018] The classification network utilizes each classification head in the classification network to classify and detect the image of the document under test based on image features, thereby obtaining the initial classification result corresponding to each classification head. This also includes:

[0019] Image features are input into the editing detection and classification head. The editing detection and classification head determines the probability that the document image to be tested belongs to various editing types based on the image features. The editing types include at least one of image stretching, color channel flipping, image content modification, document content modification, template image use, jagged edges addition, and image stitching.

[0020] For example, in cases where the attack type includes photocopying, the second classification header also includes a photocopy detection classification header.

[0021] The classification network utilizes each classification head in the classification network to classify and detect the image of the document under test based on image features, thereby obtaining the initial classification result corresponding to each classification head. This also includes:

[0022] The image features are input into the copy detection and classification head. Based on the image features, the copy detection and classification head determines the fifth probability that the image of the document to be tested belongs to a color copy image and the sixth probability that it belongs to a black and white copy image.

[0023] For example, for any initial classification result corresponding to a target classification head, based on the initial classification result corresponding to the target classification head, the document image category targeted by the target classification head, and the hierarchical relationship between the various classification heads, the final classification result of the document image to be tested is determined, including:

[0024] The initial classification results of all parent classification heads of the target classification head are determined based on the initial classification result corresponding to the target classification head, the document image category targeted by the target classification head, and the hierarchical relationship between the second classification heads.

[0025] The initial classification result corresponding to the last level classification head in the target classification head and the initial classification results of all the parent classification heads of the target classification head are multiplied to obtain the cumulative multiplication result.

[0026] The cumulative multiplication results are numerically sorted to determine the final classification result of the image of the document to be tested.

[0027] For example, if the final classification result of the image to be tested indicates that the image to be tested is a standard image, the method further includes:

[0028] Using a keypoint regression model, the region where the document is located in a standardized document image is determined;

[0029] Based on the standardized document image, the image corresponding to the area where the document is located is cropped;

[0030] The target detection model is used to perform target detection on the image corresponding to the area where the document is located, so as to obtain the target detection results. The targets include text and / or portraits.

[0031] Based on the target detection results, the final classification result of the document image to be tested is verified.

[0032] For example, before using the object detection model to perform object detection on the image corresponding to the area where the document is located, the method further includes:

[0033] Standardize the image corresponding to the area where the document is located;

[0034] Standardize the image corresponding to the area where the document is located, including:

[0035] Obtain the position coordinates of the first key point on the image corresponding to the document area;

[0036] Based on the positional relationship of the first key point on the image corresponding to the area where the document is located, the positional coordinates of the second key point on the standard graphic are obtained. The standard graphic is the shape of the front view of the document after the image corresponding to the area where the document is located has been standardized.

[0037] Based on the position coordinates of the first key point and the second key point, the relevant parameters of the homography matrix are solved. The homography matrix is ​​used to represent the projection relationship between the image corresponding to the area where the document is located and the front view of the document.

[0038] Based on the homography matrix, each pixel in the image corresponding to the area where the document is located is mapped to a standard graphic to obtain a standardized front view of the document.

[0039] For example, the individual classifier heads in the classification network are obtained through the following training operations:

[0040] Obtain image features from sample document images;

[0041] Each classification head in the initial classification head is used to classify and detect the image features of the sample document image to obtain the sample classification result for each classification head;

[0042] For each classification head in the initial classification heads, the loss function value of the classification head is calculated based on the sample classification result of the classification head and the category labeling information of the sample document image.

[0043] Based on the loss function value corresponding to each classifier in the initial classifier, the total loss function value of the classification network is calculated, and the initial classifier is trained based on the total loss function value to obtain trained classifiers.

[0044] According to another aspect of this application, an electronic device is also provided, including a processor and a memory, wherein the memory stores computer program instructions, which are executed by the processor to perform the document classification method as described above.

[0045] According to another aspect of this application, a storage medium is also provided, on which program instructions are stored, which execute the document classification method as described above when the program instructions are run.

[0046] According to another aspect of this application, a computer program product is also provided, including computer program instructions that, when executed, perform the document classification method as described above.

[0047] According to the above technical solution, a multi-level classification head with progressively decreasing granularity is used to classify and detect image features of document images. The final fusion classification result is obtained based on the hierarchical relationship between the upper and lower level classification heads. Compared with the traditional single classification model, setting up a multi-level classification head solves the problem of insufficient recognition of subtle features in traditional methods, enhances the ability to accurately distinguish documents, avoids feature confusion, and significantly improves the accuracy and reliability of document classification results in complex scenarios.

[0048] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0049] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0050] Figure 1 A schematic flowchart of a document classification method according to an embodiment of this application is shown;

[0051] Figure 2 A schematic diagram illustrating the hierarchical relationship of a multi-level classification head according to an embodiment of this application is shown;

[0052] Figure 3 A schematic block diagram of an electronic device according to one embodiment of this application is shown. Detailed Implementation

[0053] It should be noted that the data obtained in this application, including images of documents and their features, are accessed, collected, stored, and used for subsequent analysis and processing only after the user or relevant data owner has been clearly informed of the content, purpose, and processing method of the data collection. Furthermore, the user or relevant data owner can be provided with access, correction, or deletion of the data, as well as methods to revoke consent or authorization.

[0054] To make the objectives, technical solutions, and advantages of this application more apparent, exemplary embodiments according to this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein. Based on the embodiments of this application described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of this application.

[0055] As mentioned earlier, existing technologies using deep learning models for multi-category classification of documents may suffer from image quality degradation after the original document images have been processed such as adding watermarks, editing, or re-photographing. This results in traditional deep learning models being insufficient in distinguishing minute details such as watermarks and editing marks in various complex scenarios, leading to a significant reduction in document classification accuracy. To address these technical problems, this application provides a document classification method. This method can be applied to classifying any documents, tickets, etc., such as ID cards, business licenses, birth certificates, admission tickets, checks, and transportation tickets. Figure 1 A schematic flowchart of a document classification method 100 according to an embodiment of this application is shown. Figure 1 As shown, method 100 may include the following steps S110 to S170.

[0056] Step S110: Obtain the image of the document to be tested.

[0057] Alternatively, paper documents can be converted into digital images, i.e., images of the documents to be tested, using scanning devices such as high-speed scanners. During the scanning process, the scanner can be set to different resolutions; to ensure image quality and the accuracy of subsequent classification, the resolution can be set to any value between, for example, 300 dpi (dots per inch) and 600 dpi. Alternatively, the documents can be photographed using a camera on an image acquisition device to obtain an image of the document to be tested. This image acquisition device can utilize mobile devices, cameras, etc. During the photographing process, to reduce image distortion and blur, auxiliary tools, such as document photographing frames, can be used to guide the user to accurately place the document in the appropriate position, and to ensure that the shooting light is uniform and sufficient, avoiding image shadows or overexposure due to lighting issues, which would affect the subsequent classification results.

[0058] Step S130: Extract image features from the image of the document to be tested.

[0059] For example, a visual encoder can be used to extract features from the acquired image of the document to be tested. The visual encoder can be a model based on convolutional neural networks (CNNs), such as ResNet or ConvNeXt.

[0060] Taking ResNet as an example, it can extract low-level to high-level features of an image layer by layer by stacking multiple convolutional layers, pooling layers, and residual connection structures. Specifically, when the image of the document to be tested is input into ResNet, it first passes through a convolutional layer. Convolutional kernels of different sizes and numbers slide across the image to extract local features. Then, it is downsampled through a max pooling layer to reduce the amount of feature data while retaining important features. Next, the image is processed through multiple residual blocks. In the residual blocks, the input is directly added to the features after the convolution operation through skip connections, effectively avoiding the gradient vanishing problem that may occur when the network depth increases, thus enabling the extraction of richer image features.

[0061] Optionally, the visual encoder can employ a model based on a transformer structure, such as the Vision Transformer (ViT) or the Swing Transformer. ViT divides the image into multiple fixed-size image patches and maps each patch to an embedding vector. These embedding vectors are then processed using a multi-head self-attention mechanism and a multilayer perceptron to capture long-distance dependencies between different locations in the image, thereby obtaining richer multi-scale visual representations, i.e., image features. The Swing Transformer, building upon ViT, introduces a sliding window mechanism, performing self-attention computation within a local window. This reduces computational complexity while effectively handling large images, improving the efficiency and accuracy of feature extraction.

[0062] Therefore, the aforementioned visual encoder can extract image features containing rich semantic information from the input image of the document to be tested, and these features will serve as the basis for subsequent classification and detection.

[0063] It is understandable that, in addition to the aforementioned visual encoder, any other existing or future-developed algorithms or models can be used to extract image features from the document image under test.

[0064] Step S150: Classify and detect the image of the document to be tested based on image features using each classification head in the classification network to obtain the initial classification result corresponding to each classification head. At least some of the classification heads in the classification network have a hierarchical relationship set in order of their classification granularity from coarse to fine.

[0065] For example, the extracted image features are input into each classifier head in the classification network. A first-level classifier head may include multiple peer classifier heads. Each classifier head can derive its own classification result through its internal neural network structure. For ease of description, this classification result is referred to as the initial classification result below.

[0066] In one embodiment, the classifiers in the classification network may include multiple levels of classifiers, such as a first-level classifier, a second-level classifier, and a third-level classifier. The first-level classifier may be the top-level classifier, the second-level classifier is the next level below the first-level classifier, and the third-level classifier is the next level below the second-level classifier. The classification granularity of the first-level classifier is greater than that of the second-level classifier, and the classification granularity of the second-level classifier is greater than that of the third-level classifier. Classification granularity refers to the level of detail or fineness with which the classifier divides categories. At least some classifiers in the classification network have a hierarchical relationship arranged according to their classification granularity from coarse to fine. At least some classifiers may refer to classifiers with a next-level classifier. For different levels of classifiers, the next-level classifier can further subdivide the image corresponding to at least one category in the initial classification result obtained by the previous-level classifier. It should be noted that as the level of the multi-level classification head decreases, the classification granularity of the classification head gradually decreases, the classification head's ability to recognize minute features in the image gradually increases, the classification accuracy gradually increases, and the classification effect becomes better and better.

[0067] The initial classification result of each classification head can be represented by a probability, which represents the probability that the image features of the document image to be tested belong to two or more categories that this classification head can recognize. For example, when the classification head is a watermark detection classification head, it can obtain two classification results: the probability that the document image to be tested is a watermarked document image and the probability that the document image to be tested is not a watermarked document image.

[0068] Step S170: For any initial classification result corresponding to a target classification head, based on the initial classification result corresponding to the target classification head, the document image category targeted by the target classification head, and the hierarchical relationship between the various classification heads, determine the final classification result of the document image to be tested. The target classification head refers to the other classification heads that are located outside the first level.

[0069] For example, the multi-level classification head includes a first-level classification head and a second-level classification head. The first-level classification head is a canonical detection classification head, and the second-level classification head is an attack detection classification head. The canonical detection classification head is used to determine whether the document in the document image is a canonical document or a non-canonical document. A canonical document image can represent a document image whose image content, image format, and image quality all meet corresponding preset conditions. The judgment of image content can include whether the text, symbols, patterns, etc., on the image are clear, whether there are errors, or whether there are traces of tampering. The judgment of image format can include geometric feature detection of the image, such as whether the image size and proportion meet the corresponding preset conditions, and can also include color space analysis, such as whether the color distribution conforms to the standard, and whether a specific color mode is used. The judgment of image quality can include whether the resolution, sharpness, noise level, and contrast meet the corresponding preset conditions. It should be noted that the aforementioned preset conditions for image content, image format, and image quality can be reasonably set according to actual needs or classification accuracy, and are not limited here. Conversely, non-standard document images can represent document images whose content, format, or quality does not meet preset conditions. After classifying and detecting the image features of the document image under test using the standard detection classification head, an initial classification result can be obtained. This initial classification result can include the probability that the document image under test is a standard document image and the probability that it is a non-standard document image. The attack detection classification head is used to further classify non-standard document images. After classifying and detecting the image features of the document image under test using the attack detection classification head, an initial classification result can be obtained. This initial classification result can include the probability that the document image under test belongs to different types of attack images. The attack detection classification head is a lower-level classification head than the standard detection classification head, used to further classify the non-standard document images included in the initial classification result of the standard detection classification head. In this embodiment, the target classification head is the attack detection classification head. Therefore, when calculating the final classification result, multiple calculation results can be obtained based on the multiple probabilities obtained from the aforementioned attack detection classification head and the probability that the tested document is a non-standard document image obtained from the standard detection classification head. These calculation methods can include multiplication or weighted multiplication. After obtaining multiple calculation results, low-confidence classification results can be filtered out by setting a threshold, retaining only those with a confidence level higher than the preset threshold. The confidence level can be the probability obtained from the aforementioned classification head detection. Optionally, according to user or business needs, the final classification result of the tested document image can be determined by sorting the multiple calculation results by numerical value.

[0070] It can be understood that if the second-level classification head is the last level classification head, then the second-level classification result of the second-level classification head is the final classification result. If the second-level classification head is not the last level classification head, then the final classification result can be determined in a similar way to determining the second-level classification result.

[0071] In the example above, a canonical detection classification head (serving as the first-level classification head) and an attack detection classification head (serving as the second-level classification head) are used to classify and detect the image features of the document image under test. This process determines whether the document image is a canonical document image, and if it is a non-canonical document image, what type of attack it is. This classification result is more accurate than that obtained using traditional deep learning models for document classification and detection.

[0072] According to the above technical solution, image features of document images are classified and detected through a multi-level classification head with progressively decreasing granularity. The final fusion classification result is obtained based on the hierarchical relationship of the classification heads. Compared with the traditional single classification model, setting up a multi-level classification head solves the problem of insufficient recognition of subtle features in traditional methods, enhances the ability to accurately distinguish documents, avoids feature confusion, and significantly improves the accuracy and reliability of document classification results in complex scenarios.

[0073] For example, each classifier head in the classification network can be obtained through the following training operation. This training operation includes steps S141 to S144.

[0074] Step S141: Obtain the image features of the sample document image.

[0075] For example, sample document images can be acquired via a scanning device or the camera of an electronic device. These sample document images can include various types of document images, such as standard document images, watermarked document images, and different types of malicious images, such as edited document images, temporary document images, screen-captured document images, and document photocopies. The number of images of each type should be kept as balanced as possible to avoid overfitting or bias towards a particular category during model training. After acquiring the sample document images, a visual encoder can be used to extract features from them.

[0076] Alternatively, image features of sample document images can be obtained directly from sources such as networks or electronic storage devices.

[0077] Step S142: Use each of the initial classification heads to classify and detect the image features of the sample document image to obtain the sample classification result for each classification head.

[0078] For example, the image features of the extracted sample document images are input into each of the initial classification heads, so that each classification head can output its own sample classification result. Similar to the initial classification result, the sample classification result of each classification head can be represented by a probability, that is, the probability that the image features of the sample document image belong to two or more classification results that this classification head can recognize.

[0079] Step S143: For each classification head in the initial classification heads, calculate the loss function value of the classification head based on the sample classification result of the classification head and the category labeling information of the sample document image.

[0080] For example, each sample ID image can be categorized using manual or machine annotation to obtain category labeling information. The loss function value for each classifier can be calculated by combining the sample classification results and the category labeling information. This loss function value represents the classification accuracy of the classifier. The greater the difference between the sample classification result and the category labeling information, the larger the loss function value; conversely, the smaller the difference, the smaller the loss function. The loss function can include, for example, the cross-entropy loss function.

[0081] Taking the standardization detection classification head as an example, assuming that the sample document image is labeled as a standard document image, and the probability of the standardization detection classification head outputting that it belongs to a standard document image is p, its loss function value can be calculated by the cross-entropy formula, which can measure the degree of difference between the sample classification result and the category labeling information.

[0082] Step S144: Based on the loss function value corresponding to each classification head in the initial classification head, calculate the total loss function value of the classification network, and train the initial classification head based on the total loss function value to obtain each trained classification head.

[0083] After calculating the loss function value of each classifier head using the above method, the total loss function value of the classification network can be obtained by directly summing or weighted summing the loss function values ​​of each classifier head. Using optimization algorithms such as stochastic gradient descent or adaptive moment estimation, the initial classifier heads are trained based on the total loss function value, continuously adjusting the model parameters to gradually reduce the total loss function value. During training, as the parameters are adjusted, the classification ability of each classifier head gradually improves, ultimately resulting in trained classifier heads.

[0084] In existing technologies, image features from all categories of document images are typically input into a single deep learning model for training. However, there may be significant feature similarity between different categories of document images. For example, a standard document image and a copy of a document may have identical features in most areas, differing only in details. Since the model needs to learn multiple categories within a shared feature space, feature competition between different categories may occur, affecting classification performance and reducing the model's generalization ability. This application employs the aforementioned technical solution to train multiple levels of different classification heads. The resulting trained classification heads can classify document images more accurately, avoiding instability in classification results due to inter-class optimization conflicts, and improving the model's classification ability and stability.

[0085] For example, each classification head in the classification network includes an independent classification head and multiple hierarchical second classification heads. The independent classification head includes a watermark detection classification head, and the second classification heads include at least one of a canonical detection classification head, an attack detection classification head, an editing detection classification head, and a copy detection classification head. The canonical detection classification head, attack detection classification head, editing detection classification head, and copy detection classification head have a hierarchical relationship arranged according to their classification granularity from coarse to fine. The canonical detection classification head is the first-level classification head in the classification network. The canonical detection classification head has been described previously and will not be repeated here. The watermark detection classification head is used to determine whether an ID image contains a watermark. The attack detection classification head is the next-level classification head after the canonical detection classification head, used to further classify non-canonical ID images included in the initial classification result of the canonical detection classification head. The attack detection classification head has also been described previously and will not be repeated here. The Edit Detection Classification Head is the next level down from the Attack Detection Classification Head. It further classifies the edited document images included in the initial classification results of the Attack Detection Classification Head, determining the specific editing method used. Examples of editing methods include image stretching and color channel inversion. The Copy Detection Classification Head is the next level down from the Attack Detection Classification Head. It further classifies the document copy images included in the initial classification results of the Attack Detection Classification Head, determining whether the document copy image is a color copy or a black and white copy.

[0086] As mentioned above, most existing technologies employ a single deep learning model for multi-category classification of documents. However, directly performing global classification detection on image features makes it difficult to effectively distinguish between coarse-grained and fine-grained features, leading to the model's inability to accurately identify subtle differences between categories and affecting document classification results. Therefore, this application proposes the aforementioned multi-level classification head to address these issues.

[0087] Figure 2 A schematic diagram illustrating the hierarchical relationship of a multi-level classification head according to an embodiment of this application is shown. Figure 2 As shown, the standardization detection classification header can be used as the first-level classification header in the second classification header. For the independent watermark detection classification header, it can be set to the same level as the standardization detection classification header. The attack detection classification header can be used as the second-level classification header in the second classification header. The edit detection classification header and the copy detection classification header can be used as the third-level classification header in the second classification header; the edit detection classification header and the copy detection classification header are sibling classification headers. See again. Figure 2 The attack detection classification head is a level below the standard detection classification head, used to further classify non-standard document images from the initial classification results of the standard detection classification head. The edit detection classification head and the copy detection classification head are both level below the attack detection classification head. The edit detection classification head is used to further classify edited document images from the initial classification results of the attack detection classification head, while the copy detection classification head is used to further classify document copy images from the initial classification results of the attack detection classification head.

[0088] Therefore, the multi-level classification head set up above can be used to effectively identify the corresponding features for classification at different granularities, thereby improving classification accuracy and the accuracy of classification results.

[0089] In one embodiment, the second classification head includes a canonical detection classification head and an attack detection classification head. In this embodiment, step S150 utilizes each classification head in the classification network to classify and detect the image of the document under test based on image features to obtain an initial classification result corresponding to each classification head, and may include steps S151 to S153.

[0090] Step S151: Input the image features into the standard detection and classification head to determine the first probability that the image to be tested belongs to a standard document image and the second probability that it belongs to a non-standard document image.

[0091] For example, the image features of the document image to be tested extracted in the aforementioned process can be input into the standardization detection and classification head. Through its internal neural network structure, the first probability that the document image to be tested belongs to a standard document image and the second probability that it belongs to a non-standard document image are calculated. That is, the initial classification result of the standardization detection and classification head includes the first probability and the second probability, wherein the sum of the values ​​of the first probability and the second probability is equal to 1.

[0092] Step S152: Input the image features into the watermark detection classification head to determine the third probability that the image to be tested belongs to a watermarked image and the fourth probability that it belongs to an unwatermarked image.

[0093] Adding watermarks to documents is a common practice to enhance security and prevent forgery and tampering. For example, ID cards may contain transparent watermarks or patterns that only appear at specific angles, increasing authenticity and security. The watermark detection classification head, as a peer of the standardized detection classification head, can classify and detect whether a document image has a watermark based on its image features using a specific neural network model. This results in the output of a third probability (the probability that the document image has a watermark) and a fourth probability (the probability that it does not have a watermark). In other words, the initial classification result of the watermark detection classification head includes both the third and fourth probabilities, where the sum of the third and fourth probabilities equals 1.

[0094] Optionally, the standardization detection classification head and the watermark detection classification head can adopt a multilayer perceptron structure. A multilayer perceptron can include an input layer, multiple hidden layers, and an output layer. The input layer receives the image feature vector extracted by the visual encoder; the hidden layers perform nonlinear transformations on the input data from the input layer using activation functions, progressively extracting more abstract features; the output layer outputs two probability values: for the standardization detection classification head, these correspond to the probabilities of standard and non-standard ID card images; for the watermark detection classification head, these correspond to the probabilities of ID card images with and without watermarks.

[0095] Therefore, a standardized detection and classification head is used to perform a coarse classification check to determine whether a document is legitimate or not. Based on the coarse classification results, subsequent finer classification is facilitated, providing a reliable foundation for determining the final document classification result. In addition, a separate watermark detection and classification head is used to classify and detect watermarks. This serves as a useful supplement to the standardized detection and classification results, ensuring the accuracy of document classification.

[0096] Step S153: Input the image features into the attack detection classification head to determine the probability that the image of the document to be tested belongs to various attack types. The attack types include at least one of image editing, temporary documents, screen capture, and photocopy.

[0097] For example, the image features of the document image to be tested extracted in the aforementioned process can be input into the attack detection classification head. Through its internal neural network structure, the probability that the document image belongs to each of the various attack types can be calculated. The attack detection classification head is a multi-label classification task classification head. Multi-label classification task classification heads can provide independent probability estimates for each of multiple categories. Specifically, the attack detection classification head can be a ResNet-based multi-label classification network. Through the analysis of image features by this network, the probability that the document image to be tested belongs to each of the various attack types can be determined. Attack types can include one or more of the following: image editing, temporary documents, screen capture, and photocopies. Image editing refers to editing a document image to obtain an edited document image. Temporary documents refer to photographing a temporary document to obtain a temporary document image. Generally, temporary documents have a shorter validity period compared to official documents, and the two may differ in layout and physical features, such as size, material, color, and pattern details. Temporary ID cards may lack anti-counterfeiting features, or the content information in temporary ID cards may differ from that of official ID cards. Considering these factors, temporary ID cards are also considered a type of attack. Screen capture involves using an electronic device's camera to re-capture an image of the ID card and then obtaining a screen capture image. Photocopying involves photocopying an ID card, photographing the copy, and then obtaining an image of the photocopy. The probabilities of the tested ID card image belonging to different types of attack images are independent and can be any value less than 1. For example, for a given ID card image, the attack detection classification head can predict a probability of 0.8 for an edited ID card image, 0.7 for a temporary ID card image, 0.1 for a screen capture image, and 0.05 for a photocopy image.

[0098] Therefore, an attack detection classification head can be used for fine-grained classification and identification of different attack scenarios, such as editing traces, photocopies, and screen captures. Combining the classification results of the attack detection classification head with those of the aforementioned standardized detection classification head significantly improves the classification and identification effect for different document images and enhances accuracy in complex scenarios.

[0099] For example, when the attack type includes image editing, step S150 uses each classification head in the classification network to classify and detect the test document image based on image features to obtain the initial classification result corresponding to each classification head. This may also include step S154, inputting image features into the editing detection classification head, and determining the probability that the test document image belongs to various editing types based on the image features through the editing detection classification head. The editing types include at least one of image stretching, flipping color channels, modifying image content, modifying document content, using template images, adding jagged edges, and image stitching.

[0100] For example, the image features of the document image to be tested extracted in the aforementioned process can be input into the edit detection classification head. Through its internal neural network structure, the probability that the document image to be tested belongs to different types of edited document images can be calculated. Similar to the attack detection classification head, the edit detection classification head also belongs to the multi-label classification task classification head, and it can be a ResNet-based multi-label classification network. Through the analysis of image features by this network, the probability that the document image to be tested belongs to different edit types can be determined. Edit types can include one or more of the following: image stretching, color channel flipping, image content modification, document content modification, template image use, adding jagged edges, and image stitching. Image stretching can be an image editing method where the image is stretched using image editing software to obtain a stretched document image. Color channel flipping can be an image editing method where the color channels of the image are reversed using image editing software to obtain a document image with flipped color channels. Image content modification can be an image editing method where the content of the document image is modified using image editing software to obtain an image with modified image content. Modifying document content can be achieved through physical methods, such as using stickers to modify or cover the document itself, resulting in an image with the modified content. Using template images involves generating template images with uniform content and texture using software. Adding jagged edges involves using image editing software to add jagged defects to the image edges, resulting in a jagged image. Image stitching involves combining the front and back images of the document to create a stitched image. The probability of obtaining the document image corresponding to each of these editing types is independent and can be any value less than 1.

[0101] Therefore, by employing a hierarchical classification strategy involving standardized detection classification heads, attack detection classification heads, and edit detection classification heads, subcategories of specific categories can be refined. Building upon the classifications of the standardized and attack detection classification heads, the edit detection classification head can be used to further classify and recognize document images that have been edited using various image editing methods such as stretching, stitching, and smearing. This allows the model to more accurately distinguish subtle features, improve feature resolution, and enhance the classification accuracy of the classification method.

[0102] For example, when the attack type includes photocopies, step S150 uses each classification head in the classification network to classify and detect the image of the document to be tested based on image features, so as to obtain the initial classification result corresponding to each classification head. This may also include step S155, inputting image features into the photocopy detection classification head, and using the photocopy detection classification head to determine the probability that the image of the document to be tested belongs to a color photocopy image and the probability that it belongs to a black and white photocopy image based on image features.

[0103] For example, the image features of the document image to be tested extracted in the aforementioned process can be input into the copy detection and classification head. Through its internal neural network structure, the fifth probability that the document image to be tested belongs to a color copy image and the sixth probability that it belongs to a black and white copy image can be calculated, wherein the sum of the values ​​of the fifth probability and the sixth probability is equal to 1.

[0104] Therefore, by standardizing the hierarchical classification strategy of the detection classification head, attack detection classification head, and copy detection classification head, the subcategories of specific categories are refined. Based on the classification of the standard detection and attack detection classification heads, the copy detection classification head can be used to further classify and recognize document copy images, enabling the model to more accurately distinguish the colors of the copies, improve feature resolution, and enhance the classification accuracy of the classification method.

[0105] It should be noted that, according to the embodiments of this application, the execution order of steps S151 to S155 is not limited; they can be executed in any order, or all or some simultaneously. When classifying and detecting the image of the document to be tested, the image features of the document image can be simultaneously input into each of the multi-level classification heads. Then, based on the initial classification results output by each classification head, the final document classification result is obtained. This improves classification efficiency.

[0106] For example, step S170, for any target classification head, determines the final classification result of the document image to be tested based on the initial classification result corresponding to the target classification head, the document image category targeted by the target classification head, and the hierarchical relationship between the classification heads. This may include the following steps S171 and S173.

[0107] Step S171: Determine the initial classification results of all parent classification heads of the target classification head based on the initial classification result corresponding to the target classification head, the document image category targeted by the target classification head, and the hierarchical relationship between the second classification heads.

[0108] For example, the following explanation uses a second classification head that includes a standard detection classification head, an attack detection classification head, and a copy detection classification head. The standard detection classification head is a first-level classification head, and the target classification heads include the attack detection classification head and the copy detection classification head. The document image category targeted by the copy detection classification head is the document copy image in the initial classification result of the attack detection classification head. The document image category targeted by the attack detection classification head is the non-standard document image in the initial classification result of the standard detection classification head. Therefore, in this embodiment, the initial classification results of all parent classification heads of the determined target classification head include the probability of a document copy image and the probability of a non-standard document image.

[0109] Step S172: Multiply the initial classification result corresponding to the last level classification head in the target classification head and the initial classification results of all the parent classification heads of the target classification head to obtain the cumulative multiplication result.

[0110] In step S171, the initial classification results of all parent classification heads of the target classification head have been determined. Now, the initial classification result corresponding to the last level classification head in the target classification head can be multiplied with the initial classification results of all parent classification heads of the target classification head. According to the embodiment described above, the last level classification head in the target classification head is the copy detection classification head, and its corresponding initial classification results include the fifth probability that the image of the document to be tested belongs to a color copy image and the sixth probability that the image of the document to be tested belongs to a black and white copy image. It can be understood that in this example, because the classification result output by the copy detection classification head includes both the fifth and sixth probabilities, the cumulative result includes at least two values. Specifically, the fifth probability can be multiplied by the initial classification results of all parent classification heads of the target classification head, i.e., the probability of a document copy image and the probability of a non-standard document image, thereby obtaining a cumulative result. The sixth probability can also be multiplied by the initial classification results of all the superior classification heads of the target classification head determined above, namely the probability of a photocopy of an ID card and the probability of a non-standard ID card image, to obtain another cumulative result.

[0111] Step S173: The cumulative multiplication results are numerically sorted to determine the final classification result of the image of the document to be tested based on the sorting results.

[0112] Multiple cumulative results can be obtained through the above method. For example, these cumulative results can be sorted according to their numerical values, and cumulative results with values ​​less than a preset threshold can be filtered out. The preset threshold can be reasonably customized according to user or business needs. Optionally, the category corresponding to the initial classification result of the last hierarchical classification head corresponding to the largest value among the filtered cumulative results or the top few values ​​can be used as the final classification result.

[0113] This approach comprehensively considers the joint estimation of classifications from various classifiers at different levels, treating the results of each level of the model as probabilities of independent events. Therefore, the final classification result can represent their joint probability. Consequently, the final classification result is more accurate and reliable. Furthermore, multiplication amplifies deviations from the baseline value. This means that if a classifier at a certain level gives a result far from the expected value, this characteristic will be more pronounced in the final classification result after multiplication. Therefore, it effectively emphasizes the importance or influence of each classifier level.

[0114] Alternatively, step S170, for any target classification head, determines the final classification result of the document image to be tested based on the initial classification result corresponding to the target classification head, the document image category targeted by the target classification head, and the hierarchical relationship between the classification heads. This may include the following steps S174 to S176. Step S174: The initial classification results of all parent classification heads of the target classification head are determined based on the initial classification result corresponding to the target classification head, the document image category targeted by the target classification head, and the hierarchical relationship between the second classification heads. Step S175: The initial classification results corresponding to the last hierarchical classification head in the target classification head and the initial classification results of all parent classification heads of the target classification head are weighted and summed to obtain an accumulated result. Step S176: The accumulated result is numerically sorted to determine the final classification result of the document image to be tested based on the sorting result. Steps S174 to S176 are similar to steps S171 and S173 above, and will not be repeated here for simplicity.

[0115] The above scheme also takes into account the classification results of each level of classification head, and the final classification result is also representative of the type of document image.

[0116] For example, if the final classification result of the document indicates that the image of the document to be tested is a standard document image, that is, after it has been determined that the image of the document to be tested is a standard document image based on the final classification result of the document, the following steps S181 to S184 can be used to verify this classification result.

[0117] Step S181: Use the key point regression model to determine the region where the document is located in the standardized document image.

[0118] For example, a keypoint regression model can learn features of key locations in a standardized document image through training, thereby accurately locating the document's region within the image. For instance, by identifying the four corner points, edge keypoints, and / or center point of the document, the boundary of the document can be determined.

[0119] Step S182: Based on the standardized document image, extract the image corresponding to the area where the document is located.

[0120] For example, after determining the document region based on the keypoint regression model, the region can be extracted from the original image to obtain an image containing only the document content, that is, the image corresponding to the region where the document is located.

[0121] Step S183: Use the object detection model to perform object detection on the image corresponding to the area where the document is located, so as to obtain the object detection result, wherein the object includes text and / or portrait.

[0122] For example, the object detection model can employ a real-time object detection model such as YOLO based on a convolutional neural network. These models are able to detect specific objects in an image, such as text, patterns, or portraits on a document, and output the location and category information of the object.

[0123] Step S184: Based on the target detection results, verify the final classification result of the document image to be tested.

[0124] For example, the object detection model can detect key information on the document, such as name and ID number. If this information matches the format and content of a normal document, the document image is further confirmed as a standard document image. If anomalies are found in the key information area of ​​the document during the object detection process, such as an unclear name area or a missing ID number, the standardization of the document can be reassessed.

[0125] This allows for further verification of the test document image that is ultimately classified as a standardized document image, thus improving the accuracy and reliability of the document classification results.

[0126] For example, before performing target detection on the image corresponding to the area where the document is located using the target detection model in step S183, the method may further include: step S183a, performing standardization processing on the image corresponding to the area where the document is located.

[0127] For example, step S183a, which standardizes the image corresponding to the area where the document is located, may include steps S183a1 to S183a4.

[0128] Step S183a1: Obtain the position coordinates of the first key point on the image corresponding to the document area.

[0129] For example, multiple points can be selected on the image corresponding to the document area as first key points, and their position coordinates can be obtained. For instance, when the image corresponding to the document area is a quadrilateral, the four vertices of the quadrilateral can be selected as the first key points.

[0130] Step S183a2: Based on the positional relationship of the first key point on the image corresponding to the area where the document is located, obtain the position coordinates of the second key point on the standard graphic. The standard graphic is the shape corresponding to the front view of the document after standardizing the image corresponding to the area where the document is located.

[0131] For example, when the selected first keypoints are the four vertices of the image corresponding to the document area, correspondingly, four vertices can be selected on the standard graphic as the second keypoints, and their position coordinates can be obtained. Furthermore, the selection order of each point in the first and second keypoints is the same. For example, when selecting the first keypoints, four vertices are selected clockwise on the image corresponding to the document area, and when selecting the second keypoints, four vertices are also selected clockwise on the standard graphic. The standard graphic is the shape corresponding to the front view of the document after standardizing the image corresponding to the document area. For example, the shape of the front view of the document obtained after standardizing the image (quadrilateral) corresponding to the document area is a rectangle; that is, the second keypoints are the four vertices of this rectangle. It should be noted that the difference between the standard graphic and the front view of the document is that each pixel in the standard graphic does not contain pixel information; it can be considered a blank image composed of the outer contour shape of the front view of the document.

[0132] Step S183a3: Based on the position coordinates of the first key point and the second key point, solve for the relevant parameters of the homography matrix, where the homography matrix is ​​used to represent the projection relationship between the image corresponding to the area where the document is located and the front view of the document.

[0133] For example, the homography matrix can be a 3×3 nonsingular matrix used to describe the projection transformation relationship between two planes. For the first keypoint p = (x, y) on the image corresponding to the area where the document is located and the second keypoint p on the standard graphic... ′ =(x ′ ,y ′ ),satisfy Here, H represents the homography matrix. It should be noted that since H has 8 independent parameters, at least 4 pairs of non-collinear points are needed to uniquely determine it. By substituting the corresponding coordinates of the first and second keypoints in the above embodiment into the formula, the relevant parameters of H can be solved using, for example, the least squares method or a random sampling consensus algorithm.

[0134] Step S183a4: Based on the homography matrix, map each pixel in the image corresponding to the area where the document is located to a standard graphic to obtain a standardized front view of the document.

[0135] For example, after determining the homography matrix, the coordinates of each pixel in the image corresponding to the area on the document can be calculated based on the inverse of the homography matrix by traversing each pixel in the standard image. Then, interpolation is used to fill in the corresponding pixel value for each pixel, ultimately obtaining the front view of the document. The interpolation process can include bilinear interpolation, nearest neighbor interpolation, and bicubic interpolation, among other methods.

[0136] Optionally, standardizing the image corresponding to the area where the document is located may also include operations such as adjusting the image size and normalizing the pixel values ​​of the image. For example, adjusting the image corresponding to the area where the document is located to a fixed size, such as 300×300 pixels, or normalizing the pixel values ​​to the range of [0,1].

[0137] Therefore, the standardized images can better meet the input requirements of the target detection model, obtain reliable target detection results, and thus obtain more accurate verification results.

[0138] According to another aspect of this application, an electronic device is also provided. Figure 3 A schematic block diagram of an electronic device 300 according to one embodiment of this application is shown. Figure 3 As shown, the electronic device 300 may include a processor 310 and a memory 320, wherein the memory 320 stores computer program instructions, which are executed by the processor 310 to perform the document classification method as described above.

[0139] According to another aspect of this application, a storage medium is also provided, on which program instructions are stored, which, when executed, perform the document classification method described above. The storage medium may, for example, include a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0140] According to another aspect of this application, a computer program product is also provided, including computer program instructions that, when executed, perform the document classification method as described above.

[0141] Those skilled in the art can understand the specific implementation schemes and beneficial effects of the above-mentioned electronic devices, storage media, and computer program products by reading the relevant descriptions of the document classification methods. For the sake of brevity, they will not be elaborated further here.

[0142] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.

[0143] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0144] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0145] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0146] Similarly, it should be understood that, in order to simplify this application and aid in understanding one or more of the various inventive aspects, features of this application are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of this application. However, this approach should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, the inventive point lies in solving the corresponding technical problem with fewer features than all of those in a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0147] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0148] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0149] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules in the apparatus according to the embodiments of this application. This application can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0150] It should be noted that the above embodiments are illustrative of this application and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0151] The above description is merely a specific embodiment or illustration of the embodiments of this application. The scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. The scope of protection of this application shall be determined by the scope of the claims.

Claims

1. A method for classifying identification documents, characterized in that, include: Obtain the image of the document to be tested; Extract the image features of the document image to be tested; The image of the document to be tested is classified and detected by each classification head in the classification network based on the image features to obtain the initial classification result corresponding to each classification head. At least some of the classification heads in the classification network have a hierarchical relationship set in order of their classification granularity from coarse to fine. For any initial classification result corresponding to a target classification head, the final classification result of the document image to be tested is determined based on the initial classification result corresponding to the target classification head, the document image category targeted by the target classification head, and the hierarchical relationship between the various classification heads. The target classification head refers to the other classification heads that are located in the first level.

2. The document classification method as described in claim 1, characterized in that, Each classification head in the classification network includes an independent classification head and multiple second classification heads with hierarchical relationships, and the independent classification head includes a watermark detection classification head; The second classification header includes at least one of the following: a specification detection classification header, an attack detection classification header, an edit detection classification header, and a copy detection classification header; The standard detection classification head, the attack detection classification head, the edit detection classification head, and the copy detection classification head are arranged in a hierarchical relationship according to their classification granularity from coarse to fine. The attack detection classification head is the next level classification head after the standard detection classification head, and is used to classify non-standard document images included in the initial classification result of the standard detection classification head. The edit detection classification head is the next level classification head after the attack detection classification head, and is used to classify edited document images included in the initial classification result of the attack detection classification head. The copy detection classification head is the next level classification head after the attack detection classification head, and is used to classify document copy images included in the initial classification result of the attack detection classification head.

3. The document classification method as described in claim 2, characterized in that, When the second classification head includes the standard detection classification head and the attack detection classification head, the step of using each classification head in the classification network to classify and detect the image of the document to be tested based on the image features, to obtain the initial classification result corresponding to each classification head, includes: The image features are input into the standardized detection and classification head to determine the first probability that the image to be tested belongs to a standardized image and the second probability that it belongs to a non-standardized image, wherein the standardized image refers to an image whose image content, image format and image quality all meet the corresponding preset conditions; The image features are input into the watermark detection and classification head to determine the third probability that the image to be tested belongs to a watermarked image and the fourth probability that it belongs to an unwatermarked image. The image features are input into the attack detection and classification head to determine the probability that the image of the document to be tested belongs to various attack types, including at least one of image editing, temporary documents, screen capture, and photocopying.

4. The document classification method as described in claim 3, characterized in that, In the case where the attack type includes image editing, the second classification head also includes the edit detection classification head. The step of classifying and detecting the image of the document to be tested based on the image features using each classifier head in the classification network to obtain the initial classification result corresponding to each classifier head also includes: The image features are input into the edit detection and classification head, and the edit detection and classification head determines the probability that the document image to be tested belongs to various edit types based on the image features. The edit types include at least one of image stretching, color channel flipping, image content modification, document content modification, template image use, jagged edges addition, and image stitching.

5. The document classification method as described in claim 3, characterized in that, In the case where the attack type includes the photocopy, the second classification header also includes the photocopy detection classification header. The step of classifying and detecting the image of the document to be tested based on the image features using each classifier head in the classification network to obtain the initial classification result corresponding to each classifier head also includes: The image features are input into the copy detection and classification head, which then determines, based on the image features, the fifth probability that the image of the document to be tested belongs to a color copy image and the sixth probability that it belongs to a black and white copy image.

6. The document classification method according to any one of claims 1-5, characterized in that, The final classification result of the document image to be tested is determined based on the initial classification result corresponding to any target classification head, the document image category targeted by the target classification head, and the hierarchical relationship between the various classification heads, including: The initial classification results of all parent classification heads of the target classification head are determined based on the initial classification result corresponding to the target classification head, the document image category targeted by the target classification head, and the hierarchical relationship between the second classification heads. The initial classification result corresponding to the last level classification head in the target classification head and the initial classification results of all the parent classification heads of the target classification head are multiplied to obtain the cumulative multiplication result. The cumulative multiplication results are numerically sorted to determine the final classification result of the image of the document to be tested based on the sorting results.

7. The document classification method according to any one of claims 1-5, characterized in that, If the final classification result of the image to be tested indicates that the image to be tested is a standard image, the method further includes: The keypoint regression model is used to determine the region where the document is located in the standardized document image; Based on the standardized document image, the image corresponding to the area where the document is located is cropped; The image corresponding to the area where the document is located is detected using a target detection model to obtain target detection results, wherein the target includes text and / or an image; Based on the target detection results, the final classification result of the document image to be tested is verified.

8. The document classification method as described in claim 7, characterized in that, Before performing target detection on the image corresponding to the region where the document is located using the target detection model, the method further includes: The image corresponding to the area where the document is located is standardized. The standardization process for the image corresponding to the area where the document is located includes: Obtain the position coordinates of the first key point on the image corresponding to the area of ​​the document; Based on the positional relationship of the first key point on the image corresponding to the area where the document is located, the position coordinates of the second key point on the standard graphic are obtained. The standard graphic is the shape corresponding to the front view of the document after the image corresponding to the area where the document is located has been standardized. Based on the position coordinates of the first key point and the position coordinates of the second key point, the relevant parameters of the homography matrix are solved, wherein the homography matrix is ​​used to represent the projection relationship between the image corresponding to the area where the document is located and the front view of the document; Based on the homography matrix, each pixel in the image corresponding to the area where the document is located is mapped to the standard graphic to obtain the standardized front view of the document.

9. The document classification method according to any one of claims 1-6, characterized in that, The classification heads in the classification network are obtained through the following training operations: Obtain image features from sample document images; Each classification head in the initial classification head is used to classify and detect the image features of the sample document image to obtain the sample classification result for each classification head; For each classification head in the initial classification heads, the loss function value of the classification head is calculated based on the sample classification result of the classification head and the category labeling information of the sample document image; Based on the loss function value corresponding to each of the initial classification heads, the total loss function value of the classification network is calculated, and the initial classification heads are trained based on the total loss function value to obtain trained classification heads.

10. An electronic device comprising a processor and a memory, characterized in that, The memory stores computer program instructions, which, when executed by the processor, are used to perform the document classification method as described in any one of claims 1 to 9.

11. A storage medium, characterized in that, The storage medium stores program instructions that, when executed, perform the document classification method as described in any one of claims 1 to 9.

12. A computer program product comprising computer program instructions, characterized in that, The computer program instructions, when executed, are used to perform the document classification method as described in any one of claims 1 to 9.

Images