Document image recognition method and device, electronic equipment and storage medium

Abstract

The application provides a document image recognition method and device, electronic equipment and computer readable storage medium. The document image recognition method comprises: obtaining a target document image to be recognized; performing feature extraction based on the target document image through a feature extraction module in a document recognition model to obtain a sampling feature map of the target document image; performing feature fusion based on the sampling feature map of the target document image through a feature fusion module in the document recognition model to obtain a target fusion feature map of the target document image; performing recognition based on the sampling feature map of the target document image through a first recognition module in the document recognition model to obtain a first recognition result of the target document image; and performing recognition based on the target fusion feature map through a second recognition module in the document recognition model to obtain a second recognition result of the target document image. The application improves the processing speed when multiple function tasks of the document image are processed simultaneously.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, specifically to a method, apparatus, electronic device, and computer-readable storage medium for recognizing document images. Background Technology

[0002] Paper documents are the information transmission medium for people's daily office work. Images captured by photographing documents are called document images. With the popularization of portable mobile devices such as mobile phones and tablets, document images have a wide range of applications in people's lives, such as electronic archives of paper documents such as invoices, business application forms, contracts, and personal files, as well as various image and text recognition software, etc.

[0003] Typically, in computational vision processing tasks involving document images, such as document region detection, document orientation, and document sharpness, different models are used to implement each functional task separately. However, the inventors of this application found in the actual research and development process that using multiple models to implement multiple functional tasks of document images simultaneously results in slow processing speed. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and computer-readable storage medium for recognizing document images, aiming to solve the problem that the existing technology uses multiple models to realize multiple functional tasks of document images, resulting in slow processing speed.

[0005] Firstly, this application provides a method for recognizing document images, the method comprising:

[0006] Obtain the target document image to be identified;

[0007] The feature extraction module in the pre-trained document recognition model extracts features from the target document image to obtain a sampled feature map of the target document image.

[0008] The feature fusion module in the document recognition model performs feature fusion based on the sampled feature map of the target document image to obtain the target fused feature map of the target document image.

[0009] The first recognition module in the document recognition model performs recognition based on the sampled feature map of the target document image to obtain a first recognition result of the target document image, wherein the first recognition result includes at least one of the document attribute recognition result, document orientation recognition result, and document type recognition result of the target document image;

[0010] The second recognition module in the document recognition model performs recognition based on the target fusion feature map to obtain a second recognition result of the target document image, wherein the second recognition result includes at least one of the document clarity recognition result and the document region recognition result of the target document image.

[0011] Secondly, this application provides a document image recognition device, the document image recognition device comprising:

[0012] The acquisition unit is used to acquire the target document image to be identified.

[0013] The extraction unit is used to extract features from the target document image using the feature extraction module in the pre-trained document recognition model, and obtain a sampled feature map of the target document image.

[0014] The fusion unit is used to perform feature fusion based on the sampled feature map of the target document image through the feature fusion module in the document recognition model to obtain the target fused feature map of the target document image;

[0015] The recognition unit is configured to perform recognition based on the sampled feature map of the target document image through the first recognition module in the document recognition model to obtain a first recognition result of the target document image, wherein the first recognition result includes at least one of the document attribute recognition result, document orientation recognition result, and document type recognition result of the target document image;

[0016] The recognition unit is further configured to perform recognition based on the target fusion feature map through the second recognition module in the document recognition model to obtain a second recognition result of the target document image, wherein the second recognition result includes at least one of the document clarity recognition result and the document region recognition result of the target document image.

[0017] Thirdly, this application also provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, it executes the steps of any of the document image recognition methods provided in this application.

[0018] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, the computer program being loaded by a processor to perform the steps in the document image recognition method.

[0019] This application employs a document recognition model to extract features from a target document image, obtaining a sampled feature map. Then, it fuses these sampled feature maps to obtain a target fused feature map. These sampled and fused feature maps are then used to implement different document image recognition functions, such as document attribute recognition, document orientation recognition, document type recognition, document clarity recognition, and document region recognition. Since only one feature extraction and fusion is performed using the same model, the extracted and fused sampled and target fused feature maps can be used for multiple document image processing tasks. This allows the sampled and target fused feature maps from a single model to be shared across multiple document recognition functions. This avoids the need for separate feature extraction and / or fusion for each document recognition function, reducing the number of feature extraction and fusion operations, thereby accelerating the simultaneous implementation of multiple document recognition functions and improving processing speed when handling multiple document image tasks concurrently. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of a scenario for the document image recognition and detection system provided in an embodiment of this application;

[0022] Figure 2 This is a schematic flowchart of a document image recognition method provided in an embodiment of this application;

[0023] Figure 3 This is an illustrative diagram of the document recognition model provided in the embodiments of this application;

[0024] Figure 4 This is an illustrative diagram of the target fusion feature map input by the recognition module provided in the embodiments of this application;

[0025] Figure 5 This is another flowchart illustrating the document image recognition method provided in the embodiments of this application;

[0026] Figure 6 This is a schematic diagram of a sharpness discrimination branch in the document recognition model provided in this application embodiment;

[0027] Figure 7 This is a schematic diagram of an embodiment of the document recognition model training process provided in this embodiment;

[0028] Figure 8 This is a schematic diagram of an embodiment of the document image recognition device provided in this application.

[0029] Figure 9 This is a schematic diagram of an embodiment of the electronic device provided in this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0032] To enable any person skilled in the art to implement and use this application, the following description is provided. In this description, details are set forth for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be implemented without using these specific details. In other instances, well-known processes will not be described in detail to avoid obscuring the description of the embodiments of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in the embodiments of this application.

[0033] This application provides a method, apparatus, electronic device, and computer-readable storage medium for recognizing document images. The document image recognition apparatus can be integrated into an electronic device, which may be a server or a terminal, etc.

[0034] The execution subject of the document image recognition method in this application embodiment can be the document image recognition device provided in this application embodiment, or different types of electronic devices such as server equipment, physical host, or user equipment (UE) that integrate the document image recognition device. The document image recognition device can be implemented in hardware or software. The UE can specifically be a terminal device such as a smartphone, tablet computer, laptop computer, handheld computer, desktop computer, or personal digital assistant (PDA).

[0035] The electronic device can operate independently or in a cluster. By applying the document image recognition method provided in this application, the processing speed can be improved when processing multiple functional tasks involving document images simultaneously.

[0036] See Figure 1 , Figure 1 This is a schematic diagram of a document image recognition system provided in an embodiment of this application. The document image recognition system may include an electronic device 100, which integrates a document image recognition device. For example, the electronic device can acquire a target document image to be recognized; extract features from the target document image using a feature extraction module in a document recognition model to obtain a sampled feature map of the target document image; fuse features based on the sampled feature map of the target document image using a feature fusion module in the document recognition model to obtain a target fused feature map of the target document image; perform recognition based on the sampled feature map of the target document image using a first recognition module in the document recognition model to obtain a first recognition result of the target document image; and perform recognition based on the target fused feature map using a second recognition module in the document recognition model to obtain a second recognition result of the target document image.

[0037] In addition, such as Figure 1 As shown, the document image recognition system may also include a memory 200 for storing data, such as image data and video data.

[0038] It should be noted that, Figure 1 The schematic diagram of the document image recognition system shown is merely an example. The document image recognition system and scenario described in this application are for the purpose of more clearly illustrating the technical solutions of this application and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of document image recognition systems and the emergence of new business scenarios, the technical solutions provided in this invention are also applicable to similar technical problems.

[0039] The document image recognition method provided in the embodiments of this application will now be introduced. In the embodiments of this application, an electronic device is used as the execution subject. For the sake of simplicity and ease of description, the execution subject will be omitted in the subsequent method embodiments.

[0040] Reference Figure 2 , Figure 2 This is a flowchart illustrating a document image recognition method provided in an embodiment of this application. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here. The document image recognition method includes steps 201 to 205, wherein:

[0041] 201. Obtain the target document image to be identified.

[0042] To facilitate the subsequent storage, transmission, or organization of paper documents, it is usually necessary to identify information such as document attributes, document orientation, document clarity, document area, and document type of the document image.

[0043] The target document image refers to an image that requires identification of information such as document attributes, document orientation, document clarity, document area, and document type. The target document image can have various forms. For example, it can be a directly captured document image, or a scanned copy of an image obtained by converting a directly captured image using a scanner or other device or software. The target document image can be an RGB format image or a grayscale image; this application embodiment does not impose specific limitations on the format of the target document image.

[0044] In step 201, there are various ways to obtain the target document image, including, for example:

[0045] (1) In practical applications, the electronic device provided in the embodiments of this application may directly include a camera in hardware. The camera of the electronic device can capture video frames or images of the document in real time and use the video frames or images captured by the camera of the electronic device as the target document image.

[0046] (2) Obtain video frames or images that have been pre-captured and stored inside the electronic device as target document images.

[0047] (3) Electronic devices can establish network connections with other devices, such as scanners or mobile phones, and obtain video frames or images captured by the scanner online from the scanner based on the network connection, or obtain video frames or images captured by the mobile phone online from the mobile phone based on the network connection, as target document images.

[0048] (4) The electronic device reads the images collected by the scanner, mobile phone and other devices from the relevant storage medium that stores the images collected by the scanner, mobile phone and other devices, and uses them as the target document images.

[0049] The method of obtaining the target document image here is only an example and is not limited to this.

[0050] 202. Using the feature extraction module in the pre-trained document recognition model, feature extraction is performed based on the target document image to obtain a sampled feature map of the target document image.

[0051] 203. The feature fusion module in the document recognition model performs feature fusion based on the sampled feature map of the target document image to obtain the target fused feature map of the target document image.

[0052] For ease of understanding, the document recognition model in the embodiments of this application will be introduced below, such as... Figure 3 As shown, the document recognition model can include a feature extraction module, a feature fusion module, and a recognition module. The recognition module can further include a first recognition module and a second recognition module. The architecture and function of each module in the document recognition model are described below:

[0053] (I) Feature extraction module.

[0054] The feature extraction module is used to output a sampled feature map of the target document image based on the target document image. The feature extraction module takes the target document image as input and performs one or more operations on the target document image, including but not limited to convolution and pooling, to extract features from the target document image and obtain a sampled feature map.

[0055] The feature extraction module may include multiple downsampling layers, which are connected in sequence. Each downsampling layer is used to output a sampled feature map at a certain magnification. Figure 3 In this diagram, p0 represents the target document image, and p1, p2, ..., pn represent the sampled feature maps obtained after downsampling the target document image p0 at different magnifications. For example, the feature extraction module has a total of 5 downsampling layers, each with a magnification of 2. The sampled feature maps output by the 5 downsampling layers are p1, p2, p3, p4, and p5, respectively. The length and width of p1, p2, p3, p4, and p5 are 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original input target document image p0, respectively.

[0056] The feature extraction module can be a convolutional neural network (CNN). In some embodiments, the convolutional neural network portion used for feature extraction in an open-source network model can be used as the feature extraction module in this application embodiment. For example, the backbone portion of the MobileNet-v2-1.4 model used for feature extraction can be used as the feature extraction module in this application embodiment.

[0057] (ii) Feature fusion module.

[0058] The feature fusion module is used to output a fused feature map of the target document image based on the sampled feature map of the target document image. The feature fusion module takes the sampled feature map output by the feature fusion module as input, upsamples the sampled feature map, and performs feature fusion to obtain the fused feature map.

[0059] The feature fusion module may include multiple upsampling layers, which are connected in sequence. Each upsampling layer is used to upsample the sampled feature map based on a certain scaling factor to obtain and output the fused feature map. Figure 3 In the diagram, f1, f2, ..., f(n-), and fn represent the fused feature maps obtained after upsampling the feature maps at different magnifications, where fn = pn. For example, the feature fusion module has two upsampling layers, and the fused feature maps output by the four upsampling layers are f1, f2, f3, and f4, respectively. The length and width dimensions of f1, f2, f3, and f4 are the same as the original input sampled feature map p0.

[0060] (III) Identification Module.

[0061] The recognition module is used to output the recognition result of the target document image based on the sampled feature map and / or the fused feature map. The recognition module can be divided into a first recognition module that takes the sampled feature map as input and the recognition result of the target document image as input, and a second recognition module that takes the fused feature map as input and the recognition result of the target document image as input. Further, it may also include a third recognition module that takes the sampled feature map and the fused feature map as input and the recognition result of the target document image as input.

[0062] 1. First recognition module.

[0063] The first recognition module takes the sampled feature map as input, performs prediction processing based on the sampled feature map, and determines the first recognition result of the target document image. For example, it can be used to determine the document attributes, document orientation, document type, etc. of the target document image. The first recognition module can employ a fully connected layer and be connected to the feature extraction module.

[0064] 2. Second identification module.

[0065] The second recognition module takes the fused feature map as input, performs prediction processing based on the fused feature map, and determines the second recognition result of the target document image. For example, it can be used to determine the document clarity, document region (such as the paper area), etc., of the target document image. The second recognition module is connected to the feature fusion module.

[0066] 3. Third recognition module.

[0067] The third recognition module takes the sampled feature map and the fused feature map as input, performs prediction processing based on the sampled feature map and the fused feature map, and determines the third recognition result of the target document image. For example, it can be used to determine whether the paper in the target document image is intact. The third recognition module is connected to the feature extraction module and the feature fusion module.

[0068] In some embodiments, the feature extraction layer includes N downsampling layers. In this case, step 202 may specifically include: downsampling the target document image sequentially according to a preset scaling factor of each downsampling layer in the N upsampling layers; after sampling by each downsampling layer in the N downsampling layers, each downsampling layer outputs a sampled feature map to obtain N sampled feature maps of the target document image.

[0069] 204. The first recognition module in the document recognition model performs recognition based on the sampled feature map of the target document image, and the first recognition result of the target document image.

[0070] The first recognition result includes at least one of the following: document attribute recognition result, document orientation recognition result, and document type recognition result of the target document image.

[0071] Among them, the document attribute recognition result is used to indicate whether the target document image contains a paper area, thereby indicating whether the target document image is a document image.

[0072] The document orientation recognition result is used to indicate the orientation of the paper in the target document image. For example, it can be used to identify whether the paper in the target document image is facing 0°, 90°, 180°, or 270°.

[0073] Document type recognition results are used to indicate the text type, language type, purpose, etc. of the document in the target document image.

[0074] The implementation of document attribute recognition, document orientation recognition, and document type recognition functions will be illustrated with examples below.

[0075] (1) In some embodiments, the first recognition module is used to recognize the document attributes of the target document image. In this case, the first recognition module can be set as follows: the input is the sampled feature map output by the connected feature extraction module, and the output is a fully connected layer including 2 nodes. Step 204 may specifically include: using the first recognition module in the document recognition model, performing binary classification prediction processing based on the sampled feature map to determine whether the target document image contains a paper area, and obtaining the document attribute recognition result of the target document image.

[0076] (2) In some embodiments, the first recognition module is used to identify the document orientation of the target document image. In this case, the first recognition module can be configured to: input the sampled feature map output by the connection feature extraction module, and output a fully connected layer including X1 nodes, where X1 represents the number of nodes, which can be the same as the preset number of orientations; for example, if a total of 4 orientations are preset, then the number of nodes in the fully connected layer X1 = 4. Step 204 may specifically include: using the first recognition module in the document recognition model to perform classification prediction processing based on the sampled feature map, determining the orientation of the paper in the target document image, and obtaining the document orientation recognition result of the target document image.

[0077] The specific principle of "determining the orientation of the paper in the target document image by performing classification prediction processing based on the sampled feature map through the first recognition module in the document recognition model" is as follows: First, the first recognition module predicts the probability that the orientation of the paper in the document image is each preset orientation category based on the sampled feature map; then, the preset orientation category with the highest probability is output as the orientation of the paper in the target document image, thereby obtaining the document orientation recognition result of the target document image.

[0078] For example, if four orientation categories are preset, namely 0°, 90°, 180°, or 270°, the first recognition module sets up a fully connected layer with four nodes. This fully connected layer takes the sampled feature map output by the feature extraction module as input, and the four nodes are used to output the probability that the paper is 0°, 90°, 180°, or 270°. Finally, the preset orientation category with the highest probability, such as 180°, is output as the orientation of the paper in the target document image.

[0079] (3) In some embodiments, the first recognition module is used to identify the document type of the target document image. In this case, the first recognition module can be configured to: input the sampled feature map output by the connection feature extraction module, and output a fully connected layer containing X2 nodes, where X2 represents the number of nodes, which can be the same as the preset number of document types; for example, if a total of 4 document types are preset ("public document", "private document", "confidential document", "top secret document"), then the number of nodes in the fully connected layer X2 = 4. Step 204 may specifically include: using the first recognition module in the document recognition model, performing classification prediction processing based on the sampled feature map to determine the document type on the paper in the target document image, and obtaining the document type recognition result of the target document image.

[0080] The above example illustrates that the first recognition module implements only one document recognition function. It can be understood that the first recognition module can also simultaneously set up multiple functional architectures to share the feature map output by the feature extraction module, thereby achieving different document recognition functions. For example, multiple fully connected layers can be set up simultaneously, each using the feature map output by the feature extraction module as input, and predictions can be made based on the feature map to obtain the recognition results of different document recognition functions. For example, two fully connected layers can be set up for document attributes and document type respectively, forming a first recognition module with two functional architectures. Another example is that two fully connected layers can be set up for document attributes and document orientation respectively, forming a first recognition module with two functional architectures. Yet another example is that three fully connected layers can be set up for document attributes, document orientation, and document type respectively, forming a first recognition module with three functional architectures. In step 204, case (4) will introduce one such case:

[0081] (4) In some embodiments, the first recognition module is also used to recognize the document attributes and document orientation of the target document image. In this case, the first recognition module can be configured as: a first classification submodule and a second classification submodule; both the first and second classification submodules take the sampled feature map of the target document image as input, the first classification submodule outputs the document attribute recognition result of the target document image, and the second classification submodule outputs the document orientation recognition result of the target document image; for example, the first recognition module can be configured as two fully connected layers, one fully connected layer as the first classification submodule, and the other fully connected layer as the second classification submodule. Step 204 may specifically include: classifying the target document image based on the first sampled feature map of the target document image through the first classification submodule of the first recognition module to obtain the document attribute recognition result of the target document image; and classifying the target document image based on the second sampled feature map of the target document image through the second classification submodule of the first recognition module to obtain the document orientation recognition result of the target document image.

[0082] The document attribute recognition result of the target document image determined by the first sub-classification module can be referred to the introduction of case (1) in step 204 above, and the document attribute recognition result of the target document image determined by the second sub-classification module can be referred to the introduction of case (2) in step 204 above. For the sake of simplification, it will not be repeated here.

[0083] The target document image can include multiple sampled feature maps. Here, the first sampled feature map represents the sampled feature map input by the first classification submodule, and the second sampled feature map represents the sampled feature map input by the second classification submodule. In fact, the sampled feature maps input by the first classification submodule and the second classification submodule can be the same or different.

[0084] By setting multiple functional modules simultaneously in the first identification module in step 204 (4), such as setting the first classification sub-module and the second classification sub-module simultaneously, the sampled feature map in one model can be shared across multiple document identification functions. Since the problem of needing to perform feature extraction once for each document identification function can be avoided, the number of feature extractions is reduced, thereby accelerating the simultaneous implementation of multiple document identification functions.

[0085] 205. The second recognition module in the document recognition model performs recognition based on the target fusion feature map to obtain the second recognition result of the target document image.

[0086] The second recognition result includes at least one of the document clarity recognition result and the document region recognition result of the target document image.

[0087] Among them, the document clarity recognition result is used to indicate whether the text in the target document image is clear.

[0088] The document region recognition results are used to indicate the paper area in the target document image.

[0089] The implementation of document clarity recognition and document region recognition functions will be illustrated with examples below.

[0090] (1) In some embodiments, the second recognition module is used to identify the document clarity of the target document image. The second recognition module can be configured as follows: it takes as input the target fused feature map output by the feature fusion module and outputs the document clarity of the target document image. The architecture of the second recognition module can be determined according to the actual situation. For example, it can be configured as follows: a 3*3 convolutional layer with a dimension of 16 is connected after the target fused feature map output by the feature fusion module, and ReLU is used for activation, outputting the document clarity of the target document image. Step 205 may specifically include: using the second recognition module in the document recognition model, performing classification prediction processing based on the sampled feature map to determine whether the target document image is clear, and obtaining the document clarity recognition result of the target document image.

[0091] The specific identification of document clarity results will be described in detail later, and will not be repeated here.

[0092] (2) In some embodiments, the second recognition module is used to recognize the paper area of ​​the target document image. The second recognition module can be configured to: input the target fused feature map output by the connection feature fusion module, and output the structure of the paper area of ​​the target document image. Step 205 may specifically include: using the second recognition module in the document recognition model to perform region detection based on the target fused feature map, and obtaining the document region recognition result of the target document image.

[0093] The specific identification results of the document region will be described in detail later, and will not be repeated here.

[0094] (3) In some embodiments, the second recognition module is used simultaneously to recognize multiple functions such as document clarity and paper area of ​​the target document image. For example, the second recognition module can be configured as a detection submodule and a discrimination submodule; both the detection submodule and the discrimination submodule take the target fusion feature map of the target document image as input, the detection submodule outputs the document region recognition result of the target document image, and the discrimination submodule outputs the document clarity recognition result of the target document image. Step 205 may specifically include: using the detection submodule of the second recognition module to perform region detection based on the first fusion feature map to obtain the document region recognition result of the target document image; using the discrimination submodule of the second recognition module to perform clarity recognition based on the second fusion feature map to obtain the document clarity recognition result of the target document image.

[0095] The target fusion feature map can include multiple maps. Here, the first fusion feature map represents the target fusion feature map input by the detection submodule, and the second fusion feature map represents the target fusion feature map input by the discrimination submodule. In fact, the target fusion feature maps input by the detection submodule and the discrimination submodule can be the same or different.

[0096] For example, such as Figure 3 and Figure 4 As shown, the target fusion feature map includes f1, f2, f3, and f4. The detection submodule uses the first fusion feature map (e.g., f1, f2, f3, f4). Figure 4 As shown in f1), the discrimination submodule uses the second fused feature map (as shown in f1). Figure 4 (as shown in f2).

[0097] By setting multiple functional modules simultaneously in step 205 (3), such as setting both detection and discrimination sub-modules, the target fusion feature map in one model can be shared across multiple document recognition functions. Since this avoids the need to perform feature extraction and fusion for each document recognition function, it reduces the number of feature extractions and feature fusions, thereby accelerating the simultaneous implementation of multiple document recognition functions.

[0098] As can be seen from the above, in this embodiment, a document recognition model is used to extract features from the target document image to obtain a sampled feature map, and then the sampled feature map is fused to obtain a target fused feature map. These sampled feature maps and target fused feature maps are then used to implement different document image recognition functions, such as document attribute recognition, document orientation recognition, document type recognition, document clarity recognition, and document region recognition. Since only the same model is used to extract and fuse features once, the extracted and fused sampled feature maps and target fused feature maps can be used for multiple document image processing tasks, achieving the sharing of sampled features and target fused feature maps across multiple document recognition functions from a single model. This avoids the need for feature extraction and / or fusion for each document recognition function, reducing the number of feature extractions and fusions, thereby accelerating the simultaneous implementation of multiple document recognition functions and improving processing speed when handling multiple document image tasks simultaneously.

[0099] Steps 201 to 205 above described the document recognition model's recognition modules, including a first recognition module and a second recognition module, and how these functional modules share features to achieve multiple document recognition functions. In some embodiments, such as Figure 3 As shown, the document recognition model's recognition module can also include a first recognition module, a second recognition module, and a third recognition module simultaneously.

[0100] like Figure 5 As shown, the document image recognition method at this time may specifically include steps 501 to 506:

[0101] 501. Obtain the target document image to be identified.

[0102] 502. Using the feature extraction module in the pre-trained document recognition model, feature extraction is performed based on the target document image to obtain a sampled feature map of the target document image.

[0103] The sampled feature map of the target document image includes a first sampled feature map and a second sampled feature map.

[0104] 503. The feature fusion module in the document recognition model performs feature fusion based on the sampled feature map of the target document image to obtain the target fused feature map of the target document image.

[0105] The target fusion feature map of the target document image includes a third fusion feature map and a fourth fusion feature map.

[0106] 504. The first recognition module in the document recognition model performs recognition based on the first sampled feature map to obtain the first recognition result of the target document image.

[0107] 505. The second recognition module in the document recognition model performs recognition based on the third fusion feature map to obtain the second recognition result of the target document image.

[0108] The implementation of steps 501 to 505 is the same as that of steps 201 to 205 above. For details, please refer to the above description. It will not be repeated here.

[0109] 506. The third recognition module in the document recognition model performs recognition based on the fourth fused feature map and the second sampled feature map to obtain the third recognition result of the target document image.

[0110] In steps 501 to 506, the target fusion feature map may include multiple maps. Here, the third fusion feature map represents the target fusion feature map input by the detection submodule, and the fourth fusion feature map represents the target fusion feature map input by the discrimination submodule. In fact, the target fusion feature maps input by the detection submodule and the discrimination submodule may be the same or different.

[0111] Similarly, the target document image can include multiple sampled feature maps. Here, the first sampled feature map represents the sampled feature map input by the first classification submodule, and the second sampled feature map represents the sampled feature map input by the second classification submodule. In fact, the sampled feature maps input by the first classification submodule and the second classification submodule can be the same or different.

[0112] The third recognition result can be the recognition result of any document recognition function, such as determining whether the paper in the target document image is complete. In this embodiment, the third recognition result is used to indicate whether the paper in the target document image is complete, i.e., the third recognition module is used to identify whether the paper in the target document image is complete. In this case, the third recognition module can be configured as follows: inputting the sampled feature map (i.e., the second sampled feature map) output by the feature extraction module and the target fused feature map (i.e., the second fused feature map) output by the feature fusion module, and outputting the third recognition result of the target document image.

[0113] For example, the third recognition module includes a fully connected layer with two output nodes. Regression is performed based on this fully connected layer. One node outputs the probability that the paper in the target document image is intact, and the other node outputs the probability that the paper in the target document image is incomplete. Finally, if the probability that the paper in the target document image is intact is greater than or equal to the probability that the paper in the target document image is incomplete, then the paper in the target document image is determined to be intact; if the probability that the paper in the target document image is intact is less than the probability that the paper in the target document image is incomplete, then the paper in the target document image is determined to be intact.

[0114] As can be seen from steps 501 to 506 above, the first recognition module, the second recognition module, and the third recognition module realize the sharing of sampled feature maps and target fusion feature maps. The sharing of features among the functional modules realizes multiple document recognition functions.

[0115] In some embodiments, the above step "by performing region detection based on the first fused feature map through the detection submodule of the second recognition module to obtain the document region recognition result of the target document image" may specifically include steps A1 to A2:

[0116] A1. The detection submodule of the second recognition module performs foreground prediction based on the first fused feature map to obtain the paper foreground region of the target document image.

[0117] Specifically, the first fused feature map is input into the detection submodule of the second recognition module, so that the detection submodule of the second recognition module predicts the probability that each pixel of the target document image is a pixel of the foreground region of the paper based on the first fused feature map; and the pixels with a probability greater than a preset probability threshold are taken as the pixels of the foreground region of the target document image, and the coordinates of each pixel of the foreground region of the target document image are output, thereby obtaining the foreground region of the target document image.

[0118] A2. The foreground area of ​​the target document image is used as the document region recognition result of the target document image.

[0119] In some embodiments, the above step "by performing region detection based on the first fusion feature map through the detection submodule of the second recognition module to obtain the document region recognition result of the target document image" may specifically include steps B1 to B3: B1, by performing foreground prediction based on the first fusion feature map through the detection submodule of the second recognition module to obtain the paper foreground region of the target document image.

[0120] Step B1 is the same as step A1, so it will not be described again here.

[0121] B2. The detection submodule of the second recognition module performs vertex offset prediction based on the first fusion feature map to obtain the offset of each vertex of the target document image relative to each point of the foreground region of the paper.

[0122] Specifically, the first fused feature map is input into the detection submodule of the second recognition module, so that the detection submodule of the second recognition module predicts the distance (including distances in the x-axis and y-axis directions) of each vertex of the document region detection box relative to each pixel of the foreground region of the paper based on the first fused feature map; and the distance of each vertex relative to each pixel of the foreground region of the paper is used as the offset of each vertex of the document region of the target document image relative to each point of the foreground region of the paper.

[0123] B3. The detection submodule of the second recognition module determines the document region recognition result of the target document image based on the offset of the foreground region on the paper and the offset of each document region vertex relative to each point on the foreground region on the paper.

[0124] In this context, document region vertices refer to the vertices of the detection bounding boxes for the paper-based regions of the target document image.

[0125] Finally, based on the coordinates of each pixel in the foreground region of the paper and the offset of each vertex of the document region in the target document image relative to each point in the foreground region of the paper, the coordinates of the four vertices of the document region detection box are calculated. Finally, the coordinates of the four vertices of the document region detection box are used as the document region recognition result of the target document image, thus obtaining the document region recognition result of the target document image.

[0126] By combining the foreground region of the paper with the offset of each document region vertex relative to each point in the foreground region of the paper, the document region in the target document image can be detected more accurately.

[0127] In some embodiments, the above step "by performing clarity recognition based on the second fused feature map through the discrimination submodule of the second recognition module to obtain the document clarity recognition result of the target document image" may specifically include: inputting the second fused feature map into the discrimination submodule of the second recognition module, so that the discrimination submodule of the second recognition module predicts the probability that the target document image is clear and the probability that the target document image is unclear based on the second fused feature map; and when the probability that the target document image is clear is greater than or equal to the probability that the target document image is unclear, the target document image is judged as clear; when the probability that the target document image is clear is less than the probability that the target document image is unclear, the target document image is judged as unclear, thereby obtaining the document region recognition result of the target document image.

[0128] In some embodiments, the above step "performing a sharpness recognition result of the target document image by performing a sharpness recognition based on the second fusion feature map through the discrimination submodule of the second recognition module" may specifically include: performing a sharpness prediction based on the second fusion feature map and the foreground area of ​​the paper through the discrimination submodule of the second recognition module to obtain a document sharpness recognition result of the target document image.

[0129] For easier understanding, please refer to Figure 6 , Figure 6 This is a schematic diagram of the sharpness discrimination branch in the document recognition model provided in this application embodiment. Figure 6 In the second recognition module, there are a detection submodule and a discrimination submodule. The detection submodule is used to predict the foreground based on the first fusion feature map f1 to obtain the paper foreground region Fs of the target document image. The discrimination submodule is used to determine the document clarity recognition result based on the second fusion feature map f1 and the paper foreground region Fs.

[0130] The first feature fusion feature map and the second fusion feature map are mainly used to distinguish whether the feature input is for the detection submodule or the discrimination submodule. In fact, the first feature fusion feature map and the second fusion feature map can be the same or different.

[0131] like Figure 6 As shown, to improve the accuracy of document clarity recognition results, the second recognition module is used to determine the discriminant submodule for document clarity recognition results. Simultaneously, the detection submodule, which determines the foreground region of the paper based on the first fusion feature map f1, can also determine the document clarity recognition result using the second fusion feature map f1 and the foreground region Fs output by the first fusion feature map f1. Specifically, the foreground region Fs acts as a mask, focusing primarily on the foreground region and neglecting the background region during clarity discrimination, thereby improving the accuracy of document clarity recognition results.

[0132] Specifically, the second fused feature map output by the feature fusion module and the paper foreground region output by the detection submodule are input into the discrimination submodule of the second recognition module. This allows the discrimination submodule of the second recognition module to predict the probability that the target document image is clear and the probability that the target document image is unclear based on the second fused feature map and the paper foreground region. When the probability that the target document image is clear is greater than or equal to the probability that the target document image is unclear, the target document image is classified as clear. When the probability that the target document image is clear is less than the probability that the target document image is unclear, the target document image is classified as unclear, thereby obtaining the document region recognition result of the target document image.

[0133] In this embodiment, the foreground area of ​​the paper, input into the discrimination submodule of the second recognition module, acts as a mask in predicting the probability that the target document image is clear and the probability that the target document image is unclear. Since the clarity of the target document image mainly considers the clarity of the document area, by combining the foreground area of ​​the paper for discrimination, without focusing on the background part of the target document image, the accuracy of the discrimination of whether the target document image is clear can be improved.

[0134] The following example illustrates the training process of the document recognition model in this application, using a model comprising a feature extraction module, a feature fusion module, a first recognition module, and a second recognition module. The first recognition module includes a first sub-classification module and a second sub-classification model, while the second recognition module includes a detection sub-module and a discrimination sub-module. (Reference) Figure 7 , Figure 7 This is a schematic diagram of a training process for the document recognition model provided in this embodiment. The training process of the document recognition model may include the following steps 701 to 707:

[0135] 701. Obtain sample images.

[0136] The sample images are labeled with attribute tags, orientation tags, sharpness tags, and region tags.

[0137] The method for obtaining the sample images is similar to that for obtaining the target document images, and can be found in the description in step 201 above. It will not be repeated here.

[0138] This can be achieved by manually labeling sample images to obtain attribute labels, orientation labels, sharpness labels, and region labels. The following examples illustrate the process of manually labeling sample images:

[0139] 1. Label attributes. For an image set I = 1, 2, 3, ..., i, ..., label whether an image i contains at least 70% of the images, using ei = 1 to indicate presence and ei = 0 to indicate absence.

[0140] 2. Rotate the image with ei=1 to make the text orientation conform to normal reading habits.

[0141] 3. Label the sharpness. Label the sharpness of the image with ei=1, based on whether all the text on the paper can be clearly read. Use qi=1 to indicate sharpness and qi=0 to indicate blurriness.

[0142] 4. Label the areas. Label the areas on the paper within the image Ii where ei = 1 using closed quadrilaterals pi, following these rules:

[0143] a. Starting from the top left vertex of the area on the paper, label the remaining three vertices in a clockwise direction.

[0144] b. When both vertices are in the image, the line segment needs to connect the two vertices. When one or neither of the two vertices is in the image, the line segment needs to fit the visible edge of the paper in the image and infer its vertex outside the image.

[0145] c. When an entire edge of the paper is not in the drawing, the image needs to be cut off from the edge of the paper to form the new edge of the paper.

[0146] 5. Label the orientation and augment the dataset. Label the orientation of the original image i (which has been rotated or neutralized) as oi=0, and randomly augment multiple images for each image i:

[0147] a. Randomly rotate the image clockwise by 0 degrees, 90 degrees, 180 degrees, and 270 degrees, and label or update the orientation and region labels of the augmented image: use oi=0 to represent an orientation of 0 degrees, oi=1 to represent an orientation of 90 degrees, oi=2 to represent an orientation of 180 degrees, and oi=3 to represent an orientation of 270 degrees; at the same time, the label pi needs to be adjusted accordingly to ensure that its starting point is the top left vertex of the paper area;

[0148] b. Randomly add slight salt and pepper noise, Gaussian blur, brightness variations, contrast variations, etc. to the image;

[0149] c. Randomly perform small-angle rotations and perspective transformations on the image, and adjust the annotation pi accordingly.

[0150] 702. Using the feature extraction module in the preset recognition model, feature extraction is performed based on the sample image to obtain the sampled feature map of the sample image.

[0151] The preset recognition model is similar in architecture and function to the trained document recognition model, including a feature extraction module, a feature fusion module, a first recognition module, and a second recognition module. The first recognition module includes a first sub-classification module and a second sub-classification model, and the second recognition module includes a detection sub-module and a discrimination sub-module. For details, please refer to the above description of the document recognition model architecture, which will not be repeated here.

[0152] The implementation of step 702 is similar to that of step 202 above, and you can refer to the description of step 202 above for details. The difference is that step 702 uses a preset recognition model to extract features from the sample image to obtain a sample feature map of the sample image, while step 202 uses a trained document recognition model to extract features from the target document image to obtain a sample feature map of the target document image.

[0153] 703. The feature fusion module in the preset recognition model fuses the sampled feature maps of the sample images to obtain the sample fusion feature map of the sample images.

[0154] The implementation of step 703 is similar to that of step 203 above, and you can refer to the description of step 203 above for details. The difference is that step 703 uses a preset recognition model to perform feature fusion on the sample image to obtain a sample fusion feature map of the sample image, while step 203 uses a trained document recognition model to perform feature fusion on the target document image to obtain a target fusion feature map of the target document image.

[0155] 704. The first recognition module in the preset recognition model makes predictions based on the sampled feature map of the sample image to obtain the attribute prediction results and orientation prediction results of the sample image.

[0156] Among them, the attribute prediction result refers to the document attribute of the sample image obtained by recognizing it through a preset recognition model. The attribute prediction result is used to indicate whether the sample image contains paper areas.

[0157] The orientation prediction result refers to the document orientation of the sample image obtained by recognizing it through a preset recognition model. The orientation prediction result is used to indicate the orientation of the paper in the sample image.

[0158] Specifically, the attribute prediction result of the sample image can be obtained by classifying the sample image based on the first sampled feature map of the first recognition module in the preset recognition model through the first classification submodule; and the orientation prediction result of the sample image can be obtained by classifying the sample image based on the second sampled feature map of the sample image through the second classification submodule of the first recognition module in the preset recognition model.

[0159] Since the implementation of step 704 is similar to the implementation of case (4) in step 204 above, step 704 is simplified. For the part of the implementation of the preset recognition model that is not described in detail, please refer to the description of case (4) in step 204 above.

[0160] 705. Using the first recognition module in the preset recognition model, prediction is performed based on the sample fusion feature map to obtain the document region prediction result and the sharpness prediction result of the sample image.

[0161] The clarity prediction result refers to the document clarity of the sample image obtained by recognizing it through a preset recognition model. The clarity prediction result is used to indicate whether the text in the sample image is clear.

[0162] The document region prediction result refers to the paper area of ​​the sample image obtained by recognizing a preset recognition model. The document region prediction result is used to indicate the paper area in the sample document image.

[0163] Specifically, the detection submodule of the second recognition module in the preset recognition model can perform region detection based on the sample fusion feature map to obtain the document region prediction result of the sample image; the discrimination submodule of the second recognition module in the preset recognition model can perform sharpness recognition based on the fusion feature map of the sample image to obtain the sharpness prediction result of the sample image.

[0164] Since the implementation of step 705 is similar to the implementation of case (3) in step 205 above, step 705 is simplified. For the part of the implementation of the preset recognition model that is not described in detail, please refer to the description of case (3) in step 205 above.

[0165] 706. Based on the attribute prediction results, the orientation prediction results, the document region prediction results, the sharpness prediction results, the actual attribute results corresponding to the attribute labels, the actual orientation results corresponding to the orientation labels, the actual sharpness results corresponding to the sharpness labels, and the actual document region results corresponding to the region labels, determine the training loss of the preset recognition model.

[0166] The first classification submodule can be configured with a first loss function. Then, based on the first loss function, the attribute prediction results, and the actual attribute results corresponding to the attribute labels, the attribute classification loss of the first classification submodule is determined. For example, the first loss function is shown in formula (1) below:

[0167]

[0168] In formula (1), e_loss is the attribute classification loss of the first classification submodule, n is the number of training samples, Fe is the probability value of whether there is a paper area with a proportion greater than 70% in the image predicted by the model, and e is the label value of whether there is a paper area with a proportion greater than 70% in the image, which is 1 or 0.

[0169] The second classification sub-model can be configured with a second loss function. Then, based on the second loss function, the orientation prediction result, and the actual orientation result corresponding to the orientation label, the orientation classification loss of the second classification sub-module is determined. For example, the second loss function is shown in formula (2) below:

[0170]

[0171] In formula (2), o_loss is the orientation classification loss of the second classification submodule, n is the number of training samples, and Fo k Let o be the probability that the paper orientation predicted by the model is the k-th direction. k The value is 1 or 0, indicating whether the paper orientation is the k-th direction. e is 1 or 0, indicating whether there is a paper area in the image that accounts for more than 70%. ε = 1e-3 is the zero-removal factor.

[0172] The detection submodule can be configured with a third loss function. Then, based on the third loss function, the document region prediction results, and the actual document region results corresponding to the region labels, the detection loss of the detection submodule is determined. For example, the third loss function is shown in formula (3) below:

[0173]

[0174] In formula (3), d_loss is the detection loss of the detection submodule, s_loss is the document region prediction loss, g_loss is the prediction loss of the offset of the four vertices of the document relative to the current position, e is the label value of whether there is a paper area in the image that accounts for more than 70%, which is 1 or 0, and ε=1e-3 is the zero-removal factor. Among them, s_loss and g_loss can be determined by the following formulas (4) and (5) respectively.

[0175] s_loss=1-2*(∑s*Fs) / (∑s+∑Fs+ε) Formula (4)

[0176]

[0177] In formula (4), s_loss is the document region prediction loss, s is the document region annotation map, Fs is the paper foreground region predicted by the model, and ε=1e-3 is the zero factor.

[0178] In formula (5), g_loss is the prediction loss of the offset of the four vertices of the document relative to the current position, g is the labeled value of the offset of the four vertices of the document relative to the current position, and Fg is the offset of the four vertices of the document relative to the current position predicted by the model.

[0179] The discrimination submodule can be configured with a fourth loss function. Then, based on the fourth loss function, the sharpness prediction result, and the actual sharpness result corresponding to the sharpness label, the sharpness discrimination loss of the discrimination submodule is determined. For example, the fourth loss function is shown in formula (6) below:

[0180]

[0181] In formula (6), q_loss is the sharpness discrimination loss of the discrimination submodule, Fq is the probability value of the paper being sharp as predicted by the model, q is its corresponding label value, e is the label value of whether there is a paper area in the image that accounts for more than 70%, which is 1 or 0, and ε=1e-3 is the zero factor.

[0182] There are several ways to set the training loss of the preset recognition model. For example, the training loss can be set as the direct sum of the attribute classification loss of the first classification submodule, the orientation classification loss of the second classification submodule, the detection loss of the detection submodule, and the sharpness discrimination loss of the discriminator submodule. Alternatively, the training loss can be set as a weighted sum of the attribute classification loss of the first classification submodule, the orientation classification loss of the second classification submodule, the detection loss of the detection submodule, and the sharpness discrimination loss of the discriminator submodule.

[0183] 707. Based on the training loss, adjust the model parameters of the preset recognition model until a preset training stop condition is reached, and use the preset recognition model as the document recognition model.

[0184] The preset training stop conditions can be set according to actual needs. For example, it could be when the training loss is less than a preset value, or when the training loss basically no longer changes, that is, when the difference in training loss between adjacent training iterations is less than the preset value; or when the preset number of iterations for training the recognition model reaches the maximum number of iterations.

[0185] Since the document recognition model is trained on the preset recognition model based on the attribute classification loss of the first classification submodule, the orientation classification loss of the second classification submodule, the detection loss of the detection submodule, and the sharpness discrimination loss of the discriminant submodule, it can ensure that the document recognition model can accurately identify the document attributes, document orientation, document type, document sharpness, and document region of the target document image.

[0186] To better implement the document image recognition method in the embodiments of this application, based on the document image recognition method, the embodiments of this application also provide a document image recognition device, such as... Figure 8 The diagram shown is a schematic representation of an embodiment of a document image recognition device according to this application. The document image recognition device 800 includes:

[0187] Acquisition unit 801 is used to acquire the target document image to be identified;

[0188] The extraction unit 802 is used to extract features from the target document image using the feature extraction module in the pre-trained document recognition model, and obtain a sampled feature map of the target document image.

[0189] The fusion unit 803 is used to perform feature fusion based on the sampled feature map of the target document image through the feature fusion module in the document recognition model to obtain the target fused feature map of the target document image;

[0190] The recognition unit 804 is used to perform recognition based on the sampled feature map of the target document image through the first recognition module in the document recognition model to obtain a first recognition result of the target document image, wherein the first recognition result includes at least one of the document attribute recognition result, document orientation recognition result, and document type recognition result of the target document image;

[0191] The recognition unit 804 is further configured to perform recognition based on the target fusion feature map through the second recognition module in the document recognition model to obtain a second recognition result of the target document image, wherein the second recognition result includes at least one of the document clarity recognition result and the document region recognition result of the target document image.

[0192] In some embodiments, the target fusion feature map includes a first fusion feature map and a second fusion feature map, and the recognition unit 804 is specifically used for:

[0193] The detection submodule of the second recognition module performs region detection based on the first fused feature map to obtain the document region recognition result of the target document image;

[0194] The discrimination submodule of the second recognition module performs clarity recognition based on the second fused feature map to obtain the document clarity recognition result of the target document image.

[0195] In some embodiments, the identification unit 804 is specifically used for:

[0196] The detection submodule of the second recognition module performs foreground prediction based on the first fused feature map to obtain the paper foreground region of the target document image;

[0197] The detection submodule of the second recognition module performs vertex offset prediction based on the first fused feature map to obtain the offset of each vertex of the target document image relative to each point of the foreground region of the paper.

[0198] The detection submodule of the second recognition module determines the document region recognition result of the target document image based on the offset of the foreground region on the paper and the offset of each document region vertex relative to each point in the foreground region on the paper.

[0199] In some embodiments, the identification unit 804 is specifically used for:

[0200] The discrimination submodule of the second recognition module performs sharpness prediction based on the second fused feature map and the foreground region of the paper to obtain the document sharpness recognition result of the target document image.

[0201] In some embodiments, the sampling feature map of the target document image includes a first sampling feature map and a second sampling feature map, and the recognition unit 804 is specifically used for:

[0202] The first classification submodule of the first recognition module classifies the target document image based on the first sampled feature map to obtain the document attribute recognition result of the target document image, wherein the document attribute recognition result is used to indicate whether the target document image is a document image;

[0203] The second classification submodule of the first recognition module performs classification based on the second sampled feature map to obtain the document orientation recognition result of the target document image.

[0204] In some embodiments, the sampled feature map of the target document image includes a first sampled feature map and a second sampled feature map, and the target fusion feature map includes a third fusion feature map and a fourth fusion feature map. The recognition unit 804 is specifically used for:

[0205] The first recognition module in the document recognition model performs recognition based on the first sampled feature map to obtain the first recognition result of the target document image.

[0206] The second recognition module in the document recognition model performs recognition based on the third fusion feature map to obtain a second recognition result of the target document image.

[0207] The third recognition module in the document recognition model performs recognition based on the fourth fused feature map and the second sampled feature map to obtain the third recognition result of the target document image.

[0208] In some embodiments, the document image recognition device further includes a training unit (not shown in the figure), which is specifically used for:

[0209] Acquire sample images, which are labeled with attribute tags, orientation tags, sharpness tags, and region tags;

[0210] The feature extraction module in the preset recognition model extracts features from the sample image to obtain a sampled feature map of the sample image.

[0211] The feature fusion module in the preset recognition model fuses the sampled feature maps of the sample images to obtain the sample fusion feature map of the sample images.

[0212] The first recognition module in the preset recognition model makes predictions based on the sampled feature map of the sample image to obtain the attribute prediction results and orientation prediction results of the sample image.

[0213] The document region prediction result and sharpness prediction result of the sample image are obtained by using the first recognition module in the preset recognition model based on the sample fusion feature map.

[0214] Based on the attribute prediction results, the orientation prediction results, the document region prediction results, the sharpness prediction results, the actual attribute results corresponding to the attribute labels, the actual orientation results corresponding to the orientation labels, the actual sharpness results corresponding to the sharpness labels, and the actual document region results corresponding to the region labels, the training loss of the preset recognition model is determined.

[0215] Based on the training loss, the model parameters of the preset recognition model are adjusted until a preset training stop condition is reached, and the preset recognition model is used as the document recognition model.

[0216] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0217] Because the document image recognition device can perform the functions described in this application, such as... Figures 1 to 7 Corresponding to the steps in the document image recognition method in any embodiment, this application can be implemented as described above. Figures 1 to 7The beneficial effects that the document image recognition method in any embodiment can achieve are detailed in the preceding description and will not be repeated here.

[0218] Furthermore, to better implement the document image recognition method in the embodiments of this application, based on the document image recognition method, the embodiments of this application also provide an electronic device, see below. Figure 9 , Figure 9 This illustration shows a structural diagram of an electronic device according to an embodiment of this application. Specifically, the electronic device provided in this embodiment includes a processor 901, which executes a computer program stored in a memory 902 to implement, for example... Figures 1 to 7 The steps of the document image recognition method in any embodiment correspond to the following; or, when the processor 901 executes the computer program stored in the memory 902, it implements the following... Figure 7 The functions of each unit in the corresponding embodiment.

[0219] For example, a computer program can be divided into one or more modules / units, one or more of which are stored in memory 902 and executed by processor 901 to complete the embodiments of this application. One or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in a computer device.

[0220] The electronic device may include, but is not limited to, processor 901 and memory 902. Those skilled in the art will understand that the illustrations are merely examples of an electronic device and do not constitute a limitation on the device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc., with processor 901, memory 902, input / output devices, and network access devices connected via a bus.

[0221] The processor 901 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the electronic device, connecting various parts of the electronic device through various interfaces and lines.

[0222] The memory 902 can be used to store computer programs and / or modules. The processor 901 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 902 and by calling data stored in the memory 902. The memory 902 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device (such as audio data, video data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0223] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the document image recognition device, electronic equipment, and its corresponding units described above can be found in, for example... Figures 1 to 7 The description of the document image recognition method corresponding to any embodiment will not be repeated here.

[0224] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0225] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute the present application. Figures 1 to 7 For the steps in the document image recognition method corresponding to any embodiment, please refer to the following for specific operations: Figures 1 to 7 The description of the document image recognition method corresponding to any embodiment will not be repeated here.

[0226] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0227] Because of the instructions stored in the computer-readable storage medium, the present application can be executed as described above. Figures 1 to 7 Corresponding to the steps in the document image recognition method in any embodiment, this application can be implemented as described above. Figures 1 to 7The beneficial effects that the document image recognition method in any embodiment can achieve are detailed in the preceding description and will not be repeated here.

[0228] The foregoing has provided a detailed description of a document image recognition method, apparatus, electronic device, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for recognizing document images, characterized in that, The method includes: Obtain the target document image to be identified; The feature extraction module in the pre-trained document recognition model extracts features from the target document image to obtain a sampled feature map of the target document image. The feature fusion module in the document recognition model performs feature fusion based on the sampled feature map of the target document image to obtain a target fusion feature map of the target document image, wherein the target fusion feature map includes a first fusion feature map; The document recognition model uses a first recognition module to identify the target document image based on its sampled feature map, resulting in a first recognition result. This first recognition result includes at least one of the following: document attribute recognition result, document orientation recognition result, and document type recognition result. The document attribute recognition result indicates whether the target document image is a document image. The document type recognition result indicates the text type, language type, and purpose of the document in the target document image. The second recognition module in the document recognition model performs recognition based on the target fusion feature map to obtain a second recognition result of the target document image. This second recognition result includes at least one of a document clarity recognition result and a document region recognition result. The detection submodule of the second recognition module performs foreground prediction based on the first fusion feature map to obtain the paper foreground region of the target document image. This paper foreground region is used as the document region recognition result of the target document image. Alternatively, vertex offset prediction is performed based on the first fusion feature map to obtain the offset of each document region vertex relative to each point in the paper foreground region. Based on the paper foreground region and the offsets of each document region vertex relative to each point in the paper foreground region, the document region recognition result of the target document image is determined. The document region recognition result is used to indicate the paper region in the target document image.

2. The document image recognition method according to claim 1, characterized in that, The target fusion feature map further includes a second fusion feature map. The second recognition module in the document recognition model performs recognition based on the target fusion feature map to obtain a second recognition result for the target document image, including: The detection submodule of the second recognition module performs region detection based on the first fused feature map to obtain the document region recognition result of the target document image; The discrimination submodule of the second recognition module performs clarity recognition based on the second fused feature map to obtain the document clarity recognition result of the target document image.

3. The document image recognition method according to claim 1, characterized in that, The discrimination submodule of the second recognition module performs sharpness recognition based on the second fused feature map to obtain the document sharpness recognition result of the target document image, including: The discrimination submodule of the second recognition module performs sharpness prediction based on the second fused feature map and the foreground region of the paper to obtain the document sharpness recognition result of the target document image.

4. The document image recognition method according to claim 1, characterized in that, The sampled feature map of the target document image includes a first sampled feature map and a second sampled feature map. The first recognition module in the document recognition model performs recognition based on the sampled feature map of the target document image. The first recognition result of the target document image includes: The first classification submodule of the first recognition module performs classification based on the first sampled feature map to obtain the document attribute recognition result of the target document image; The second classification submodule of the first recognition module performs classification based on the second sampled feature map to obtain the document orientation recognition result of the target document image.

5. The document image recognition method according to claim 1, characterized in that, The sampled feature map of the target document image includes a first sampled feature map and a second sampled feature map, and the target fusion feature map includes a third fusion feature map and a fourth fusion feature map; The step of using the first recognition module in the document recognition model to perform recognition based on the sampled feature map of the target document image to obtain a first recognition result of the target document image includes: The first recognition module in the document recognition model performs recognition based on the first sampled feature map to obtain the first recognition result of the target document image. The step of obtaining a second recognition result of the target document image by performing recognition based on the target fusion feature map through the second recognition module in the document recognition model includes: The second recognition module in the document recognition model performs recognition based on the third fusion feature map to obtain a second recognition result of the target document image. The method further includes: The third recognition module in the document recognition model performs recognition based on the fourth fused feature map and the second sampled feature map to obtain the third recognition result of the target document image.

6. The document image recognition method according to any one of claims 1 to 5, characterized in that, The method further includes: Acquire sample images, which are labeled with attribute tags, orientation tags, sharpness tags, and region tags; The feature extraction module in the preset recognition model extracts features from the sample image to obtain a sampled feature map of the sample image. The feature fusion module in the preset recognition model fuses the sampled feature maps of the sample images to obtain the sample fusion feature map of the sample images. The first recognition module in the preset recognition model makes predictions based on the sampled feature map of the sample image to obtain the attribute prediction results and orientation prediction results of the sample image. The document region prediction result and sharpness prediction result of the sample image are obtained by using the first recognition module in the preset recognition model based on the sample fusion feature map. Based on the attribute prediction results, the orientation prediction results, the document region prediction results, the sharpness prediction results, the actual attribute results corresponding to the attribute labels, the actual orientation results corresponding to the orientation labels, the actual sharpness results corresponding to the sharpness labels, and the actual document region results corresponding to the region labels, the training loss of the preset recognition model is determined. Based on the training loss, the model parameters of the preset recognition model are adjusted until a preset training stop condition is reached, and the preset recognition model is used as the document recognition model.

7. A document image recognition device, characterized in that, The document image recognition device includes: The acquisition unit is used to acquire the target document image to be identified. The extraction unit is used to extract features from the target document image using the feature extraction module in the pre-trained document recognition model to obtain a sampled feature map of the target document image, wherein the target fusion feature map includes a first fusion feature map; The fusion unit is used to perform feature fusion based on the sampled feature map of the target document image through the feature fusion module in the document recognition model to obtain the target fused feature map of the target document image; The recognition unit is configured to perform recognition based on the sampled feature map of the target document image using the first recognition module in the document recognition model, and obtain a first recognition result of the target document image, wherein the first recognition result includes at least one of the document attribute recognition result, document orientation recognition result, and document type recognition result of the target document image; the document attribute recognition result is used to indicate whether the target document image is a document image; the document type recognition result is used to indicate the text type, language type, and purpose of the document in the target document image; The recognition unit is further configured to: perform recognition based on the target fusion feature map using the second recognition module in the document recognition model to obtain a second recognition result of the target document image, wherein the second recognition result includes at least one of the document clarity recognition result and the document region recognition result of the target document image; perform foreground prediction based on the first fusion feature map using the detection submodule of the second recognition module to obtain the paper foreground region of the target document image; use the paper foreground region as the document region recognition result of the target document image; or, perform vertex offset prediction based on the first fusion feature map to obtain the offset of each document region vertex of the target document image relative to each point of the paper foreground region, and determine the document region recognition result of the target document image based on the paper foreground region and the offset of each document region vertex relative to each point of the paper foreground region; the document region recognition result is used to indicate the paper region in the target document image.

8. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the document image recognition method as described in any one of claims 1 to 6 when it invokes the computer program in the memory.

9. A computer-readable storage medium, characterized in that, It stores a computer program, which is loaded by a processor to perform the steps in the document image recognition method according to any one of claims 1 to 6.

Images