Document image processing method, device and equipment and storage medium thereof

Abstract

The embodiment of the invention belongs to the technical field of image processing, is applied to a scanning image missing supplementing scene, and relates to a document image processing method and device, equipment and a storage medium thereof. Inputting into an image missing detection model to obtain a missing detection result; identifying whether the missing part information of the scanned image is a null value; if so, determining that the scanned image is an integral scanned image; if not, judging the missing type of the scanned image; if the type is the first missing type, sending a re-scanning prompt to the scanning end; if the type is the second missing type, inputting the scanned image into an image splicing processing model to obtain a spliced image; and inputting the integrity scanning image or the spliced image to an image post-processing end for image post-processing. It is ensured that the scanned image of the target document is preprocessed before the image post-processing end processes the scanned image, so that the integrity of the scanned image processed by the image post-processing end is ensured.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and is applied to the scenario of missing scanned images, and in particular to a document image processing method, device, equipment and storage medium thereof. Background Art

[0002] In financial business, there are often various documents that need to be reviewed. Since the size of the documents is not always consistent with the scanner, some documents may not be fully scanned.

[0003] When faced with missing documents, enterprises usually simply record them on paper materials or in spreadsheets such as Excel. They cannot be associated with existing financial systems, nor can they be easily queried and counted through the Internet, which increases the difficulty of tracking missing documents and the tax risks caused by the failure to complete the bills in a timely manner. Whether it is the extraction of document information or the review of document content, it is inevitable to stitch the documents in the process of digitizing documents. The existing image stitching technology is based on rules and mathematical methods, with many parameters set, resulting in poor robustness. How to timely complete the document scan image when there is missing content in the document scan image and ensure the integrity of the image post-processing has become an urgent problem to be solved. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a document image processing method, device, equipment and storage medium thereof, so as to solve the existing problem that when there is missing content in the document scan image, the document scan image cannot be completed in time, resulting in the inability to ensure the integrity of the image post-processing.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a document image processing method, which adopts the following technical solutions:

[0006] A document image processing method comprises the following steps:

[0007] Step 201, obtaining a scanned image of a target document;

[0008] Step 202, inputting the scanned image into a pre-trained image defect detection model, and obtaining a defect detection result output by the image defect detection model, wherein the defect detection result includes defect location information;

[0009] Step 203, identifying whether the missing part information of the scanned image is a null value according to the missing part detection result;

[0010] Step 204: if the missing part information is a null value, the scanned image is a complete scanned image;

[0011] Step 205, if the missing part information is not a null value, determining the missing type of the scanned image according to the missing part information;

[0012] Step 206, if the missing type of the scanned image is the first missing type, sending a rescanning prompt to the scanning end, and repeating steps 201 to 203;

[0013] Step 207: if the missing type of the scanned image is the second missing type, input the scanned image into a pre-trained image stitching processing model to obtain a stitched image output by the image stitching processing model;

[0014] Step 208: Send the integrity scan image or the stitched image to a preset image post-processing end for image post-processing.

[0015] Furthermore, before executing the step of inputting the scanned image into a pre-trained image missing detection model to obtain a missing detection result output by the image missing detection model, the method further includes:

[0016] Setting first type detection anchor points at all fixed text block positions of the scanned image, and distinguishingly marking the first type detection anchor points;

[0017] Setting second type detection anchor points at all input text block positions of the scanned image, and distinguishingly marking all second type detection anchor points;

[0018] Obtaining detection anchor point distinction marking results corresponding to the scanned image;

[0019] According to the document type of the target document, acquiring at least one integrity scan image corresponding to the document type from the historical scan images;

[0020] Setting a first type of standard anchor point at all fixed text block positions of the integrity scan image, and distinguishingly marking the first type of standard anchor point;

[0021] Setting a second type of standard anchor point at all input text block positions of the integrity scan image, and distinguishingly marking the second type of standard anchor point;

[0022] Obtaining a standard anchor point distinguishing marking result corresponding to the integrity scan image;

[0023] The integrity scan image and the standard anchor point distinction marking result are input together into the image missing detection model to be trained, and the image missing detection model to be trained is subjected to anchor point recognition learning training to obtain a pre-trained image missing detection model.

[0024] Furthermore, the step of inputting the scanned image into a pre-trained image missing detection model to obtain a missing detection result output by the image missing detection model specifically includes:

[0025] Comparing the difference between the detection anchor point distinguishing mark result and the standard anchor point distinguishing mark result;

[0026] According to the difference comparison result, identifying all missing anchor points of the scanned image compared with the integrity scanned image;

[0027] Obtaining the distinguishing marking results of the standard anchor points corresponding to the integrity scan image for all the missing anchor points;

[0028] Based on the standard anchor point distinguishing marking results corresponding to all the missing anchor points, identifying the text block position information corresponding to all the missing anchor points and the type of the text block, wherein the type of the text block includes a template fixed type text block and a content input type text block;

[0029] The image missing detection model outputs the text block position information corresponding to all missing anchor points and the type of the text block.

[0030] Furthermore, before executing the step of determining the missing type of the scanned image according to the missing part information, the method further includes:

[0031] Set the judgment priority for the missing type in advance,

[0032] The content of determining the priority is as follows:

[0033] If the missing part information contains at least one input information other than the fixed template information in the document, the missing type is the first missing type;

[0034] If the missing part information contains only fixed template information in the document, the missing type is the second missing type;

[0035] The step of determining the missing type of the scanned image according to the missing part information specifically includes:

[0036] Determine the text block position information corresponding to all missing anchor points in the missing part information and the type of the text block;

[0037] If there is at least one text block of a content input type, the missing part information includes input information other than the fixed template information in the document, and the missing type is a first missing type;

[0038] If the types of all text blocks are template-fixed text blocks, the missing part information is fixed template information in the document, and the missing type is the second missing type.

[0039] Furthermore, before executing the step of inputting the scanned image into the pre-trained image stitching processing model to obtain the stitched image output by the image stitching processing model, the method further includes:

[0040] According to the detection anchor point distinguishing marking result, the text content at the position of the input text block corresponding to all the second type detection anchor points in the scanned image is masked to obtain a scanned image containing only the fixed template information in the document;

[0041] According to the distinguishing marking result of the standard anchor points, masking the text content at the position of the input text block corresponding to all the second type standard anchor points in the integrity scan image corresponding to the document type is performed to obtain an integrity scan image containing only the fixed template information in the document;

[0042] Randomly segment the integrity scanned image containing only the fixed template information in the document to obtain a segmented image set as a training image set;

[0043] The training image set is input into the image stitching processing model to be trained, and the image feature extraction component, feature overlap recognition component and image feature decoding component in the image stitching processing model to be trained are trained to obtain a pre-trained image stitching processing model.

[0044] Furthermore, the image stitching processing model includes an image feature extraction component, a feature overlap recognition component and an image feature decoding component. The step of inputting the training image set into the image stitching processing model to be trained, and performing learning and training on the image feature extraction component, the feature overlap recognition component and the image feature decoding component in the image stitching processing model to be trained to obtain a pre-trained image stitching processing model specifically includes:

[0045] Input all images in the training image set into the image feature extraction component, perform learning and training on the image feature extraction component through the Transformer architecture encoding and decoding method, obtain the trained image feature extraction component and obtain the image feature extraction results of all images in the training image set, wherein the image feature extraction results include the location information and content information of all text blocks;

[0046] The position information and content information corresponding to all the text blocks are used as a pair of input information and input into the feature overlap recognition component, and the feature overlap recognition component is trained according to the consistency rule of content information in the same text block, so that the feature overlap recognition component learns the overlapping parts between all images in the training image;

[0047] According to the overlapping position information corresponding to the overlapping parts between all the images in the training image, position adjustment is performed on all the images in the training image, position information and content information of all the text blocks after the position adjustment is obtained, and the position information and content information of all the text blocks after the position adjustment are deployed as recognition knowledge to the feature overlap recognition component;

[0048] The position information and content information of all the text blocks after position adjustment are input as feature information into the image feature decoding component, and the image feature decoding component is trained to obtain a trained image feature decoding component and an integrity scan image output by the image feature decoding component that only contains fixed template information in the document, wherein the image feature decoding component includes a feature decoder based on the Transformer architecture pattern.

[0049] Furthermore, the step of inputting the scanned image into a pre-trained image stitching processing model to obtain a stitched image output by the image stitching processing model specifically includes:

[0050] Performing feature extraction on the scanned image through the trained image feature extraction component to obtain an image feature extraction result, wherein the image feature extraction result includes position information and content information of all text blocks in the scanned image;

[0051] Inputting the position information and content information respectively corresponding to all text blocks in the scanned image into the trained feature overlap recognition component, and comparing them with the recognition knowledge in the trained feature overlap recognition component;

[0052] According to the comparison result, identifying the missing text block in the scanned image and the position information and content information of the missing text block;

[0053] The position information and content information corresponding to all text blocks in the scanned image, as well as the position information and content information of the text blocks missing in the scanned image, are input into the trained image feature decoding component to obtain a spliced ​​image output by the trained image feature decoding component.

[0054] In order to solve the above technical problems, the embodiment of the present application also provides a document image processing device, which adopts the following technical solution:

[0055] A document image processing device, comprising:

[0056] A scanning image acquisition module, used to acquire a scanning image of a target document;

[0057] A missing detection module, used to input the scanned image into a pre-trained image missing detection model, and obtain a missing detection result output by the image missing detection model, wherein the missing detection result includes missing part information;

[0058] A missing detection result identification module, used to identify whether the missing part information of the scanned image is a null value according to the missing detection result;

[0059] A first branch processing module, configured to determine that if the missing part information is a null value, the scanned image is a complete scanned image;

[0060] A missing type judgment module, configured to judge the missing type of the scanned image according to the missing part information if the missing part information is not a null value;

[0061] A second branch processing module, configured to send a rescanning prompt to the scanning end if the missing type of the scanned image is the first missing type, and repeatedly execute steps 201 to 203;

[0062] An image stitching processing module, configured to input the scanned image into a pre-trained image stitching processing model if the missing type of the scanned image is a second missing type, and obtain a stitched image output by the image stitching processing model;

[0063] The image post-processing sending module is used to send the integrity scan image or the stitched image to a preset image post-processing end for image post-processing.

[0064] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0065] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the document image processing method described above when executing the computer-readable instructions.

[0066] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0067] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the document image processing method as described above.

[0068] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0069] The document image processing method described in the embodiment of the present application obtains the scanned image of the target document; inputs the scanned image into the image missing detection model to obtain the missing detection result; identifies whether the missing part information of the scanned image is a null value; if the missing part information is a null value, the scanned image is a complete scanned image; if the missing part information is not a null value, the missing type of the scanned image is determined; if the missing type of the scanned image is the first missing type, a rescan prompt is sent to the scanning end; if the missing type of the scanned image is the second missing type, the scanned image is input into the image stitching processing model to obtain a stitched image; the complete scanned image or the stitched image is input into the image post-processing end for image post-processing. This ensures that the scanned image of the target document is pre-processed before the image post-processing end processes the scanned image, so that the scanned image processed by the image post-processing end is complete. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0071] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0072] Figure 2 is a flow chart of an embodiment of a document image processing method according to the present application;

[0073] Figure 3 yes Figure 2 A flowchart of a specific embodiment of the pre-processing steps before the image missing detection in step 202 is shown;

[0074] Figure 4 yes Figure 2 A flowchart of a specific embodiment of step 202 is shown;

[0075] Figure 5 yes Figure 2 A flowchart of a specific embodiment of determining the type of missing information of the scanned image in step 205 is shown;

[0076] Figure 6 yes Figure 2 A flowchart of a specific embodiment of the pre-processing step before the image stitching process in step 207 is shown;

[0077] Figure 7 yes Figure 6 A flowchart of a specific embodiment of step 604 is shown;

[0078] Figure 8 yes Figure 2 A flowchart of a specific embodiment of performing stitching processing on the scanned images in step 207 is shown;

[0079] Fig. 9 is a schematic structural diagram of an embodiment of a document image processing device according to the present application;

[0080] Fig.10 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0081] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0082] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0083] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0084] like Figure 1As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0085] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0086] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, the tablet computer 1012 or the mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer and a desktop computer, etc.

[0087] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0088] It should be noted that the document image processing method provided in the embodiment of the present application is generally executed by a terminal device, and accordingly, the document image processing device is generally arranged in the terminal device.

[0089] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0090] Continue to refer Figure 2 , shows a flow chart of an embodiment of a document image processing method according to the present application. The document image processing method comprises the following steps:

[0091] Step 201, obtaining a scanned image of a target document.

[0092] In this embodiment, the target document is, for example, a document that needs to be filled in when handling financial business, including a personal information filling form, a transaction voucher document, a financial statement document, etc. Of course, the target document can also be a fee detail flow document, a medical expense payment document, etc. in a medical business scenario.

[0093] By scanning the target document, a scanned image of the target document is obtained, and the target document is converted into an electronic scanned image, which is convenient for subsequent cache management of the information contained in the image and for future data reuse and viewing.

[0094] Step 202: input the scanned image into a pre-trained image defect detection model to obtain a defect detection result output by the image defect detection model, wherein the defect detection result includes defect location information.

[0095] An image missing detection model is used to perform missing detection on the scanned image, so as to facilitate subsequent identification of whether the scanned image is a complete scanned image of the target document.

[0096] Step 203: Identify whether the missing part information of the scanned image is a null value according to the missing part detection result.

[0097] Step 204: If the missing part information is a null value, the scanned image is a complete scanned image.

[0098] Step 205: If the missing part information is not a null value, determine the missing type of the scanned image according to the missing part information.

[0099] By presetting the missing type and determining the missing type of the scanned image by the missing part information, it is possible to identify whether it is a template-type missing or a non-template-type missing when the scanned image is missing, so as to facilitate the execution of subsequent steps.

[0100] Step 206: If the missing type of the scanned image is the first missing type, a rescanning prompt is sent to the scanning end, and steps 201 to 203 are repeated.

[0101] In this embodiment, the first missing type can be understood as a non-template missing type, that is, the missing content detected in the scanned image is non-fixed content information, such as personal information entered by the customer, calculated expense information, etc. For the missing content, the target document needs to be rescanned to ensure the integrity of the scanned image subsequently sent to the image post-processing end.

[0102] Step 207: If the missing type of the scanned image is the second missing type, the scanned image is input into a pre-trained image stitching processing model to obtain a stitched image output by the image stitching processing model.

[0103] In this embodiment, the second missing type can be understood as a template-type missing, that is, the missing content detected in the scanned image is fixed content information, for example: the preset header (attribute) field content in the document. For the missing part of the content, there is no need to rescan the target document. The image stitching method can also ensure the integrity of the scanned image subsequently sent to the image post-processing end.

[0104] Step 208: Send the integrity scan image or the stitched image to a preset image post-processing end for image post-processing.

[0105] In this embodiment, the scanned image of the target document is obtained; the scanned image is input into the image missing detection model to obtain the missing detection result; it is identified whether the missing part information of the scanned image is a null value; if the missing part information is a null value, the scanned image is a complete scanned image; if the missing part information is not a null value, the missing type of the scanned image is determined; if the missing type of the scanned image is the first missing type, a rescan prompt is sent to the scanning end; if the missing type of the scanned image is the second missing type, the scanned image is input into the image stitching processing model to obtain a stitched image; the complete scanned image or the stitched image is input into the image post-processing end for image post-processing. It is ensured that the scanned image of the target document is pre-processed before the image post-processing end processes the scanned image, so that the scanned image processed by the image post-processing end is complete.

[0106] Continue to refer Figure 3 , Figure 3 yes Figure 2 The flowchart of a specific embodiment of the preprocessing step before the image missing detection in step 202 is shown. In some optional implementations, the preprocessing before the image missing detection includes the following steps:

[0107] Step 301, setting first type detection anchor points at all fixed text block positions of the scanned image, and distinguishing and marking the first type detection anchor points;

[0108] Step 302, setting second type detection anchor points at all input text block positions of the scanned image, and distinguishingly marking all second type detection anchor points;

[0109] Step 303, obtaining the detection anchor point distinction marking result corresponding to the scanned image;

[0110] Step 304, according to the document type of the target document, obtaining at least one integrity scan image corresponding to the document type from the historical scan images;

[0111] Step 305, setting a first type of standard anchor point at all fixed text block positions of the integrity scan image, and distinguishingly marking the first type of standard anchor point;

[0112] Step 306, setting a second type of standard anchor point at all input text block positions of the integrity scan image, and distinguishingly marking the second type of standard anchor point;

[0113] Step 307, obtaining the standard anchor point distinguishing marking result corresponding to the integrity scan image;

[0114] Step 308: input the integrity scan image and the standard anchor point distinguishing marking result together into the image missing detection model to be trained, perform anchor point recognition learning training on the image missing detection model to be trained, and obtain a pre-trained image missing detection model.

[0115] By setting a first type of detection anchor point at all fixed text block positions of the scanned image, and setting a second type of detection anchor point at all input text block positions of the scanned image as the detection anchor point; setting a first type of standard anchor point at all fixed text block positions of the integrity scanned image, and setting a second type of standard anchor point at all input text block positions of the integrity scanned image as the standard anchor point, and using the integrity scanned image and the standard anchor point distinction marking results to perform anchor point recognition learning and training on the image missing detection model, so that the detection anchor point can be detected by the image missing detection model in the future, thereby identifying the missing part of the image.

[0116] Continue to refer Figure 4 , Figure 4 yes Figure 2 The flowchart of a specific embodiment of step 202 is shown. In some optional implementations, step 202 includes the following steps:

[0117] Step 401, comparing the detection anchor point distinguishing marking result with the standard anchor point distinguishing marking result;

[0118] Step 402, identifying all missing anchor points of the scanned image compared to the integrity scanned image according to the difference comparison result;

[0119] Step 403, obtaining the distinguishing marking results of the standard anchor points corresponding to the integrity scan image for all the missing anchor points;

[0120] Step 404, based on the standard anchor point distinction marking results corresponding to all the missing anchor points, identify the text block position information corresponding to all the missing anchor points and the type of the text block, wherein the type of the text block includes a template fixed type text block and a content input type text block;

[0121] Step 405: Output the text block position information corresponding to all missing anchor points and the type of the text block through the image missing detection model.

[0122] The pre-trained image missing detection model is used to perform anchor point missing identification on the detection anchor points, so that all missing anchor points of the scanned image compared with the complete scanned image can be identified, and according to the distinguishing marks corresponding to the missing anchor points, the text block position information corresponding to all the missing anchor points and the type of the text block are identified, which is more intelligent and automated, and improves the efficiency of scanned image missing detection.

[0123] In this embodiment, before executing the step of determining the missing type of the scanned image through the missing part information, the method further includes: pre-setting a judgment priority for the missing type, wherein the content of the judgment priority is: if the missing part information contains at least one input information other than the fixed template information in the document, then the missing type is the first missing type; if the missing part information contains only the fixed template information in the document, then the missing type is the second missing type.

[0124] By pre-setting the priority for determining the missing type, when the missing content corresponding to all missing anchor points includes both fixed template information in the document and input information outside the fixed template information in the document, the missing type can be determined based on the priority, thereby avoiding the problem of being unable to determine the missing type of the scanned image.

[0125] Continue to refer Figure 5 , Figure 5 yes Figure 2 The flowchart of a specific embodiment of determining the type of missing information of the scanned image in step 205 is shown. In some optional implementations, determining the type of missing information of the scanned image in step 205 includes the following steps:

[0126] Step 501, determining the text block position information corresponding to all missing anchor points in the missing part information and the type of the text block;

[0127] Step 502: If there is at least one text block of a content input type, the missing part information includes input information other than the fixed template information in the document, and the missing type is a first missing type;

[0128] Step 503: If the types of all text blocks are template-fixed text blocks, the missing part information is all fixed template information in the document, and the missing type is the second missing type.

[0129] The missing type of the scanned image is determined by the text block position information corresponding to all missing anchor points in the missing part information and the type of the text block, so as to facilitate the subsequent identification of whether the scanned image can be spliced ​​and completed by the missing type. If splicing and completion cannot be performed, step 206 is executed, otherwise, step 207 is executed.

[0130] Continue to refer Figure 6 , Figure 6 yes Figure 2 The flowchart of a specific embodiment of the preprocessing step before the image stitching process in step 207 is shown. In some optional implementations, the preprocessing before the image stitching process includes the following steps:

[0131] Step 601, according to the detection anchor point distinguishing marking result, masking the text content at the position of the input text block corresponding to all the second type detection anchor points in the scanned image, to obtain a scanned image containing only the fixed template information in the document;

[0132] Step 602, according to the standard anchor point distinguishing marking result, masking the text content at the position of the input text block corresponding to all the second type standard anchor points in the integrity scan image corresponding to the document type, to obtain an integrity scan image containing only the fixed template information in the document;

[0133] Step 603, randomly segmenting the integrity scanned image containing only the fixed template information in the document to obtain a segmented image set as a training image set;

[0134] By randomly segmenting the integrity scanned image that only contains the fixed template information in the document, a segmented image set is obtained and used as a training image set. When the image stitching processing model to be trained is learned and trained, the scanned images used are all scanned images with missing content, which not only increases the number of samples, but also ensures the applicability of the image stitching processing model obtained after learning and training.

[0135] Step 604, input the training image set into the image stitching processing model to be trained, perform learning and training on the image feature extraction component, feature overlap recognition component and image feature decoding component in the image stitching processing model to be trained, and obtain a pre-trained image stitching processing model.

[0136] By learning and training the image stitching processing model to be trained, when there is a template missing in the scanned image in the future, the image stitching processing model that has been learned and trained can be directly used to fill in the missing content of the scanned image, which is more intelligent and automated, and reduces the consumption of manpower and material resources during manual screening.

[0137] In this embodiment, the image stitching processing model includes an image feature extraction component, a feature overlap recognition component and an image feature decoding component.

[0138] Continue to refer Figure 7 , Figure 7 yes Figure 6 The flowchart of a specific embodiment of step 604 is shown. In some optional implementations, step 604 includes the following steps:

[0139] Step 701, input all images in the training image set into the image feature extraction component, perform learning and training on the image feature extraction component through the Transformer architecture encoding and decoding method, obtain the trained image feature extraction component and obtain the image feature extraction results of all images in the training image set, wherein the image feature extraction results include the location information and content information of all text blocks;

[0140] Step 702: taking the position information and content information corresponding to all the text blocks as a pair of input information, inputting them into the feature overlap recognition component, and performing learning and training on the feature overlap recognition component according to the consistency rule of content information in the same text block, so that the feature overlap recognition component learns the overlapping parts between all the images in the training image;

[0141] Step 704, adjusting the positions of all images in the training image according to the overlapping position information corresponding to the overlapping parts between all images in the training image, obtaining the position information and content information of all text blocks after the position adjustment, and deploying the position information and content information of all text blocks after the position adjustment as recognition knowledge to the feature overlap recognition component;

[0142] Step 704, input the position information and content information of all the text blocks after position adjustment as feature information into the image feature decoding component, perform learning and training on the image feature decoding component, obtain the trained image feature decoding component and obtain the integrity scan image output by the image feature decoding component that only contains the fixed template information in the document, wherein the image feature decoding component includes a feature decoder based on the Transformer architecture pattern.

[0143] The availability of the image splicing processing model is ensured by pre-training the image feature extraction component, the feature overlap recognition component and the image feature decoding component in the image splicing processing model in sequence.

[0144] Continue to refer Figure 8 , Figure 8 yes Figure 2 The flowchart of a specific embodiment of stitching the scanned images in step 207 is shown. In some optional implementations, stitching the scanned images in step 207 includes the following steps:

[0145] Step 801, extracting features from the scanned image by using the trained image feature extraction component to obtain an image feature extraction result, wherein the image feature extraction result includes position information and content information of all text blocks in the scanned image;

[0146] Step 802, inputting the position information and content information corresponding to all text blocks in the scanned image into the trained feature overlap recognition component, and comparing them with the recognition knowledge in the trained feature overlap recognition component;

[0147] Step 803, identifying the missing text block in the scanned image and the position information and content information of the missing text block according to the comparison result;

[0148] Step 804, inputting the position information and content information corresponding to all text blocks in the scanned image, as well as the position information and content information of the text blocks missing in the scanned image, into the trained image feature decoding component to obtain a spliced ​​image output by the trained image feature decoding component.

[0149] The present application obtains the scanned image of the target document; inputs the scanned image into the image missing detection model to obtain the missing detection result; identifies whether the missing part information of the scanned image is a null value; if the missing part information is a null value, the scanned image is a complete scanned image; if the missing part information is not a null value, the missing type of the scanned image is determined; if the missing type of the scanned image is the first missing type, a rescan prompt is sent to the scanning end; if the missing type of the scanned image is the second missing type, the scanned image is input into the image stitching processing model to obtain a stitched image; the complete scanned image or the stitched image is input into the image post-processing end for image post-processing. This ensures that the scanned image of the target document is pre-processed before the image post-processing end processes the scanned image, so that the scanned image processed by the image post-processing end is complete.

[0150] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0151] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0152] In the embodiment of the present application, a scanned image of a target document is obtained; the scanned image is input into an image missing detection model to obtain missing detection results; it is identified whether the missing part information of the scanned image is a null value; if the missing part information is a null value, the scanned image is a complete scanned image; if the missing part information is not a null value, the missing type of the scanned image is determined; if the missing type of the scanned image is the first missing type, a rescan prompt is sent to the scanning end; if the missing type of the scanned image is the second missing type, the scanned image is input into an image stitching processing model to obtain a stitched image; the complete scanned image or the stitched image is input into an image post-processing end to perform image post-processing. This ensures that the scanned image of the target document is pre-processed before the image post-processing end performs scanned image processing, so that the scanned image processed by the image post-processing end is complete.

[0153] Further references Fig. 9 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of a document image processing device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0154] like Fig. 9 As shown, the document image processing device 900 described in this embodiment includes: a scanned image acquisition module 901, a missing detection module 902, a missing detection result recognition module 903, a first branch processing module 904, a missing type judgment module 905, a second branch processing module 906, an image stitching processing module 907 and an image post-processing sending module 908. Among them:

[0155] The scanned image acquisition module 901 is used to acquire the scanned image of the target document;

[0156] A missing detection module 902 is used to input the scanned image into a pre-trained image missing detection model to obtain a missing detection result output by the image missing detection model, wherein the missing detection result includes missing part information;

[0157] A missing detection result identification module 903 is used to identify whether the missing part information of the scanned image is a null value according to the missing detection result;

[0158] A first branch processing module 904 is configured to determine that the scanned image is a complete scanned image if the missing part information is a null value;

[0159] A missing type judgment module 905 is used to judge the missing type of the scanned image according to the missing part information if the missing part information is not a null value;

[0160] The second branch processing module 906 is used for sending a rescanning prompt to the scanning end if the missing type of the scanned image is the first missing type, and repeating steps 201 to 203;

[0161] An image stitching processing module 907 is used for inputting the scanned image into a pre-trained image stitching processing model to obtain a stitched image output by the image stitching processing model if the missing type of the scanned image is the second missing type;

[0162] The image post-processing sending module 908 is used to send the integrity scan image or the stitched image to a preset image post-processing end for image post-processing.

[0163] The present application obtains the scanned image of the target document; inputs the scanned image into the image missing detection model to obtain the missing detection result; identifies whether the missing part information of the scanned image is a null value; if the missing part information is a null value, the scanned image is a complete scanned image; if the missing part information is not a null value, the missing type of the scanned image is determined; if the missing type of the scanned image is the first missing type, a rescan prompt is sent to the scanning end; if the missing type of the scanned image is the second missing type, the scanned image is input into the image stitching processing model to obtain a stitched image; the complete scanned image or the stitched image is input into the image post-processing end for image post-processing. This ensures that the scanned image of the target document is pre-processed before the image post-processing end processes the scanned image, so that the scanned image processed by the image post-processing end is complete.

[0164] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0165] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0166] To solve the above technical problems, the present application also provides a computer device. Fig.10 , Fig.10 This is a basic structural block diagram of the computer device in this embodiment.

[0167] The computer device 10 includes a memory 10a, a processor 10b, and a network interface 10c which are interconnected and communicated via a system bus. It should be noted that: Fig.10 Only a computer device 10 having a component memory 10a, a processor 10b, and a network interface 10c is shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.

[0168] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0169] The memory 10a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10a can be an internal storage unit of the computer device 10, such as a hard disk or memory of the computer device 10. In other embodiments, the memory 10a can also be an external storage device of the computer device 10, such as a plug-in hard disk equipped on the computer device 10, a smart memory card (Smart Med ia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10a can also include both the internal storage unit of the computer device 10 and its external storage device. In this embodiment, the memory 10a is generally used to store an operating system and various application software installed on the computer device 10, such as a computer-readable instruction of a document image processing method. In addition, the memory 10a can also be used to temporarily store various data that have been output or are to be output.

[0170] The processor 10b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 10b is generally used to control the overall operation of the computer device 10. In this embodiment, the processor 10b is used to run the computer-readable instructions stored in the memory 10a or process data, such as running the computer-readable instructions of the document image processing method.

[0171] The network interface 10c may include a wireless network interface or a wired network interface. The network interface 10c is generally used to establish a communication connection between the computer device 10 and other electronic devices.

[0172] The computer device proposed in this embodiment belongs to the field of image processing technology and is applied to the scenario of missing scanned images. This application obtains a scanned image of the target document; inputs the scanned image into an image missing detection model to obtain missing detection results; identifies whether the missing part information of the scanned image is a null value; if the missing part information is a null value, the scanned image is a complete scanned image; if the missing part information is not a null value, the missing type of the scanned image is judged; if the missing type of the scanned image is the first missing type, a rescan prompt is sent to the scanning end; if the missing type of the scanned image is the second missing type, the scanned image is input into the image stitching processing model to obtain a stitched image; the complete scanned image or the stitched image is input into the image post-processing end for image post-processing. It ensures that the scanned image of the target document is pre-processed before the image post-processing end processes the scanned image, so that the integrity of the scanned image processed by the image post-processing end is ensured.

[0173] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the document image processing method as described above.

[0174] The computer-readable storage medium proposed in this embodiment belongs to the field of image processing technology and is applied to the scenario of missing scanned images. This application obtains a scanned image of the target document; inputs the scanned image into an image missing detection model to obtain missing detection results; identifies whether the missing part information of the scanned image is a null value; if the missing part information is a null value, the scanned image is a complete scanned image; if the missing part information is not a null value, the missing type of the scanned image is judged; if the missing type of the scanned image is the first missing type, a rescan prompt is sent to the scanning end; if the missing type of the scanned image is the second missing type, the scanned image is input into the image stitching processing model to obtain a stitched image; the complete scanned image or the stitched image is input into the image post-processing end for image post-processing. It ensures that the scanned image of the target document is pre-processed before the image post-processing end processes the scanned image, so that the integrity of the scanned image processed by the image post-processing end is ensured.

[0175] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0176] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A document image processing method, characterized in that: The steps include: Step 201, obtaining a scanned image of a target document; Step 202, inputting the scanned image into a pre-trained image defect detection model, and obtaining a defect detection result output by the image defect detection model, wherein the defect detection result includes defect location information; Step 203, identifying whether the missing part information of the scanned image is a null value according to the missing part detection result; Step 204: if the missing part information is a null value, the scanned image is a complete scanned image; Step 205, if the missing part information is not a null value, determining the missing type of the scanned image according to the missing part information; Step 206, if the missing type of the scanned image is the first missing type, sending a rescanning prompt to the scanning end, and repeating steps 201 to 203; Step 207: if the missing type of the scanned image is the second missing type, input the scanned image into a pre-trained image stitching processing model to obtain a stitched image output by the image stitching processing model; Step 208: Send the integrity scan image or the stitched image to a preset image post-processing end for image post-processing.

2. The document image processing method according to claim 1, characterized in that: Before executing the step of inputting the scanned image into a pre-trained image missing detection model to obtain a missing detection result output by the image missing detection model, the method further includes: Setting first type detection anchor points at all fixed text block positions of the scanned image, and distinguishingly marking the first type detection anchor points; Setting second type detection anchor points at all input text block positions of the scanned image, and distinguishingly marking all second type detection anchor points; Obtaining detection anchor point distinction marking results corresponding to the scanned image; According to the document type of the target document, acquiring at least one integrity scan image corresponding to the document type from the historical scan images; Setting a first type of standard anchor point at all fixed text block positions of the integrity scan image, and distinguishingly marking the first type of standard anchor point; Setting a second type of standard anchor point at all input text block positions of the integrity scan image, and distinguishingly marking the second type of standard anchor point; Obtaining a standard anchor point distinguishing marking result corresponding to the integrity scan image; The integrity scan image and the standard anchor point distinction marking result are input together into the image missing detection model to be trained, and the image missing detection model to be trained is subjected to anchor point recognition learning training to obtain a pre-trained image missing detection model.

3. The document image processing method according to claim 2, characterized in that: The step of inputting the scanned image into a pre-trained image missing detection model to obtain a missing detection result output by the image missing detection model specifically includes: Comparing the detection anchor point distinguishing marking result with the standard anchor point distinguishing marking result; According to the difference comparison result, identifying all missing anchor points of the scanned image compared with the integrity scanned image; Obtaining the distinguishing marking results of the standard anchor points corresponding to the integrity scan image for all the missing anchor points; Based on the standard anchor point distinguishing marking results corresponding to all the missing anchor points, identifying the text block position information corresponding to all the missing anchor points and the type of the text block, wherein the type of the text block includes a template fixed type text block and a content input type text block; The image missing detection model outputs the text block position information corresponding to all missing anchor points and the type of the text block.

4. The document image processing method according to claim 3, characterized in that: Before executing the step of determining the missing type of the scanned image according to the missing part information, the method further includes: Set the judgment priority for the missing type in advance, The content of determining the priority is as follows: If the missing part information contains at least one input information other than the fixed template information in the document, the missing type is the first missing type; If the missing part information contains only fixed template information in the document, the missing type is the second missing type; The step of determining the missing type of the scanned image according to the missing part information specifically includes: Determine the text block position information corresponding to all missing anchor points in the missing part information and the type of the text block; If there is at least one text block of a content input type, the missing part information includes input information other than the fixed template information in the document, and the missing type is a first missing type; If the types of all text blocks are template-fixed text blocks, the missing part information is fixed template information in the document, and the missing type is the second missing type.

5. The document image processing method according to claim 2, characterized in that: Before executing the step of inputting the scanned image into the pre-trained image stitching processing model to obtain the stitched image output by the image stitching processing model, the method further includes: According to the detection anchor point distinguishing marking result, the text content at the position of the input text block corresponding to all the second type detection anchor points in the scanned image is masked to obtain a scanned image containing only the fixed template information in the document; According to the distinguishing marking result of the standard anchor points, masking the text content at the position of the input text block corresponding to all the second type standard anchor points in the integrity scan image corresponding to the document type is performed to obtain an integrity scan image containing only the fixed template information in the document; Randomly segment the integrity scanned image containing only the fixed template information in the document to obtain a segmented image set as a training image set; The training image set is input into the image stitching processing model to be trained, and the image feature extraction component, feature overlap recognition component and image feature decoding component in the image stitching processing model to be trained are trained to obtain a pre-trained image stitching processing model.

6. The document image processing method according to claim 5, characterized in that: The image stitching processing model includes an image feature extraction component, a feature overlap recognition component and an image feature decoding component. The step of inputting the training image set into the image stitching processing model to be trained, and performing learning and training on the image feature extraction component, the feature overlap recognition component and the image feature decoding component in the image stitching processing model to be trained to obtain a pre-trained image stitching processing model specifically includes: Input all images in the training image set into the image feature extraction component, perform learning and training on the image feature extraction component through the Transformer architecture encoding and decoding method, obtain the trained image feature extraction component and obtain the image feature extraction results of all images in the training image set, wherein the image feature extraction results include the location information and content information of all text blocks; The position information and content information corresponding to all the text blocks are used as a pair of input information and input into the feature overlap recognition component, and the feature overlap recognition component is trained according to the consistency rule of content information in the same text block, so that the feature overlap recognition component learns the overlapping parts between all images in the training image; According to the overlapping position information corresponding to the overlapping parts between all the images in the training image, position adjustment is performed on all the images in the training image, position information and content information of all the text blocks after the position adjustment is obtained, and the position information and content information of all the text blocks after the position adjustment are deployed as recognition knowledge to the feature overlap recognition component; The position information and content information of all the text blocks after position adjustment are input as feature information into the image feature decoding component, and the image feature decoding component is trained to obtain a trained image feature decoding component and an integrity scan image output by the image feature decoding component that only contains fixed template information in the document, wherein the image feature decoding component includes a feature decoder based on the Transformer architecture pattern.

7. The document image processing method according to claim 6, characterized in that: The step of inputting the scanned image into a pre-trained image stitching processing model to obtain a stitched image output by the image stitching processing model specifically includes: Performing feature extraction on the scanned image through the trained image feature extraction component to obtain an image feature extraction result, wherein the image feature extraction result includes position information and content information of all text blocks in the scanned image; Inputting the position information and content information respectively corresponding to all text blocks in the scanned image into the trained feature overlap recognition component, and comparing them with the recognition knowledge in the trained feature overlap recognition component; According to the comparison result, identifying the missing text block in the scanned image and the position information and content information of the missing text block; The position information and content information corresponding to all text blocks in the scanned image, as well as the position information and content information of the text blocks missing in the scanned image, are input into the trained image feature decoding component to obtain a spliced ​​image output by the trained image feature decoding component.

8. A document image processing device, characterized in that: include: A scanning image acquisition module, used to acquire a scanning image of a target document; A missing detection module, used to input the scanned image into a pre-trained image missing detection model, and obtain a missing detection result output by the image missing detection model, wherein the missing detection result includes missing part information; A missing detection result identification module, used to identify whether the missing part information of the scanned image is a null value according to the missing detection result; A first branch processing module, configured to determine that if the missing part information is a null value, the scanned image is a complete scanned image; A missing type judgment module, configured to judge the missing type of the scanned image according to the missing part information if the missing part information is not a null value; A second branch processing module, configured to send a rescanning prompt to the scanning end if the missing type of the scanned image is the first missing type, and repeatedly execute steps 201 to 203; An image stitching processing module, configured to input the scanned image into a pre-trained image stitching processing model if the missing type of the scanned image is a second missing type, and obtain a stitched image output by the image stitching processing model; The image post-processing sending module is used to send the integrity scan image or the stitched image to a preset image post-processing end for image post-processing.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the document image processing method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the document image processing method according to any one of claims 1 to 7.

Images