Document reconstruction method and device, equipment and storage medium

Abstract

The invention discloses a document reconstruction method and device, equipment and a storage medium, and relates to the field of artificial intelligence, and the method comprises the steps: obtaining a target document image corresponding to a to-be-reconstructed document, and carrying out the image segmentation of the target document image through a target image segmentation model, so as to determine a first region document and a second region document of the target document image; generating a target structured prompt based on a character recognition result corresponding to the first region document, the first layout position and the target geometric feature; the target geometric feature is a geometric feature corresponding to the shielding pattern; and performing semantic reconstruction on the second region document based on the target structured prompt by using the target large language model to obtain a target reconstructed document, and generating a target reconstructed text corresponding to the target document image based on the character recognition result, the first region document and the target reconstructed document. According to the method and the device, high-precision recovery of the shielded characters is realized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a document reconstruction method, apparatus, device, and storage medium. Background Technology

[0002] Existing OCR (Optical Character Recognition) engines often discard the obscured area or display garbled text when encountering large areas of occlusion, such as seals, fingerprints, or handwritten signatures, leading to the loss of crucial fields. Some methods attempt to use GANs (Generative Adversarial Networks) or diffusion models for image-level restoration of the obscured area, but lack semantic priors, easily producing "illusory characters" inconsistent with the context. Furthermore, seals / fingerprints possess legal credential attributes and cannot be easily erased; their integrity must be preserved, making pure image restoration insufficient for compliance requirements. Simultaneously, traditional document reconstruction schemes do not utilize the semantic reasoning capabilities of large language models, failing to jointly infer the obscured text based on context and seal shape priors.

[0003] In conclusion, how to achieve high-precision restoration of obscured text while preserving the integrity of visual obscuring patterns such as seals / fingerprints is a problem that urgently needs to be solved. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a document reconstruction method, apparatus, device, and storage medium that can achieve high-precision recovery of obscured text while preserving the integrity of visual occlusion patterns such as stamps / fingerprints. The specific solution is as follows: Firstly, this application discloses a document reconstruction method, including: The target document image corresponding to the document to be reconstructed is obtained, and the target document image is segmented using a target image segmentation model to determine the first region document and the second region document of the target document image; the first region document is the document in the region where the text is not occluded by the occlusion pattern, and the second region document is the document in the region where the text is occluded by the occlusion pattern; the occlusion pattern includes a stamp pattern, a watermark pattern, and a fingerprint pattern. Based on the text recognition results corresponding to the first region document, the first page position, and the target geometric features, a target structured prompt is generated; the target geometric features are the geometric features corresponding to the occlusion pattern. Using a target large language model, semantic reconstruction is performed on the second region document based on the target structured prompts to obtain the target reconstructed document. Based on the text recognition results, the first region document, and the target reconstructed document, the target reconstructed text corresponding to the target document image is generated.

[0005] Optionally, the step of segmenting the target document image using a target image segmentation model to determine a first region document and a second region document of the target document image includes: The target document image is segmented and edge-aligned using a target image segmentation model to obtain a target binary mask corresponding to the target document image; Based on the pixel values ​​of the target binary mask, the contours of connected components are determined, and the first region document and the second region document of the target document image are determined based on the contours of the connected components.

[0006] Optionally, before generating the target reconstructed text corresponding to the target document image based on the text recognition result, the first region document, and the target reconstructed document, the method further includes: The occlusion ratio of the target document image is determined based on the connected component contour; Based on the proportion of the occluded image, determine whether the document in the second region meets the conditions for generating glyphs; If the second region document meets the glyph generation conditions, then the target glyph and glyph weights corresponding to the second region document are generated using the target image generation model based on the target structured prompts.

[0007] Optionally, the step of using the target large language model to perform semantic reconstruction of the second region document based on the target structured prompts to obtain the target reconstructed document includes: Using the target large language model based on the target structured prompts, semantic reconstruction is performed on the second region document to determine all reconstructed documents and their corresponding document weights corresponding to the second region document; The target reconstructed document is determined from all the reconstructed documents based on the document weights corresponding to the reconstructed documents.

[0008] Optionally, generating the target reconstructed text corresponding to the target document image based on the text recognition result, the first region document, and the target reconstructed document includes: The target reconstructed text is determined based on the document weight corresponding to the reconstructed document, the target glyph image and glyph image weight corresponding to the second region document, and the recognition weight corresponding to the text recognition result. The confidence level of the reconstruction result corresponding to the target reconstructed text is also determined. Accordingly, after generating the target reconstructed text corresponding to the target document image based on the text recognition result, the first region document, and the target reconstructed document, the method further includes: Based on the confidence level of the reconstruction result, it is determined whether the target reconstructed text meets the review conditions, so that when the target reconstructed text meets the review conditions, the corresponding text review operation is performed.

[0009] Optionally, before generating the target structured prompt based on the text recognition result corresponding to the first region document, the first page position, and the target geometric features, the method further includes: Optical character recognition is performed on the target document image to obtain the text recognition result corresponding to the first region document, and the position of the first region document on the first page of the target document is determined.

[0010] Optionally, the target image segmentation model is a weight-frozen Segment Anything model.

[0011] Secondly, this application discloses a document reconstruction apparatus, comprising: The document segmentation module is used to acquire the target document image corresponding to the document to be reconstructed, and to perform image segmentation on the target document image using a target image segmentation model to determine the first region document and the second region document of the target document image; the first region document is the document in the area where the text is not occluded by the occlusion pattern, and the second region document is the document in the area where the text is occluded by the occlusion pattern; the occlusion pattern includes a stamp pattern, a watermark pattern, and a fingerprint pattern. The prompt generation module is used to generate a target structured prompt based on the text recognition result corresponding to the first region document, the first page position, and the target geometric features; the target geometric features are the geometric features corresponding to the occlusion pattern; The document reconstruction module is used to perform semantic reconstruction of the second region document based on the target structured prompts using the target large language model to obtain the target reconstructed document, and to generate the target reconstructed text corresponding to the target document image based on the text recognition result, the first region document and the target reconstructed document.

[0012] Thirdly, this application discloses an electronic device, including: Memory, used to store computer programs; A processor is used to execute the computer program to implement the aforementioned document reconstruction method.

[0013] Fourthly, this application discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned document reconstruction method.

[0014] In this application, during document reconstruction, a target document image corresponding to the document to be reconstructed is obtained, and the target document image is segmented using a target image segmentation model to determine a first region document and a second region document. The first region document is the document in the area where the text is not occluded by an occlusion pattern, and the second region document is the document in the area where the text is occluded by an occlusion pattern. The occlusion pattern includes a stamp pattern, a watermark pattern, and a fingerprint pattern. A target structured prompt is generated based on the text recognition result corresponding to the first region document, the first page position, and the target geometric features. The target geometric features are the geometric features corresponding to the occlusion pattern. A target large language model is used to perform semantic reconstruction on the second region document based on the target structured prompt to obtain the target reconstructed document, and the target reconstructed text corresponding to the target document image is generated based on the text recognition result, the first region document, and the target reconstructed document. As can be seen, this application first utilizes a target image segmentation model to accurately segment the areas occluded by seals or fingerprints in the target document image corresponding to the paper document to be reconstructed. Then, the text recognition results, layout position, and geometric features of the seal / fingerprint in the unoccluded area (i.e., the first document region) are encoded into target structured prompts, which are then input into a target large language model to perform semantic reconstruction of the occluded text in the second document region. Finally, by combining the original text recognition results and the target reconstructed document, a highly reliable target reconstructed text is output. The entire process significantly improves the recognition accuracy of text in occluded areas while preserving the integrity of the seal / fingerprint, and can be widely applied in high-requirement scenarios such as finance, judiciary, and government affairs. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0016] Figure 1 This is a flowchart of a document reconstruction method disclosed in this application; Figure 2 This is a flowchart illustrating a specific document reconstruction method disclosed in this application; Figure 3 This is a schematic diagram of the structure of a document reconstruction device disclosed in this application; Figure 4 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Existing OCR engines typically discard the obscured area or display garbled text when encountering large-area occlusions such as official seals, fingerprints, or handwritten signatures, leading to the loss of crucial fields. Some methods attempt to use GANs or diffusion models for image-level restoration of the obscured area, but lack semantic priors, easily producing "illusory characters" inconsistent with the context. Furthermore, seals / fingerprints possess legal credential attributes and cannot be easily erased; their integrity must be preserved, making pure image restoration insufficient for compliance requirements. Simultaneously, traditional document reconstruction schemes do not utilize the semantic reasoning capabilities of large language models, failing to jointly infer the obscured text based on context and seal shape priors. To address these technical issues, this application discloses a document reconstruction method capable of achieving high-precision restoration of obscured text while preserving the integrity of the visual occlusion pattern (such as seals / fingerprints).

[0019] See Figure 1 As shown, this embodiment of the invention discloses a document reconstruction method, including: Step S11: Obtain the target document image corresponding to the document to be reconstructed, and use the target image segmentation model to perform image segmentation on the target document image to determine the first region document and the second region document of the target document image; the first region document is the document in the region where the text is not occluded by the occlusion pattern, and the second region document is the document in the region where the text is occluded by the occlusion pattern; the occlusion pattern includes a stamp pattern, a watermark pattern, and a fingerprint pattern.

[0020] In this embodiment, as Figure 2As shown, firstly, the target document image corresponding to the document to be reconstructed is acquired, such as a scanned or photographed contract image (containing occlusion patterns such as seals, watermarks, or fingerprints). Then, the target document image is segmented using a target image segmentation model. Specifically, the target document image is segmented using the target image segmentation model to determine the first and second regions of the target document image. This includes: performing image segmentation and edge alignment on the target document image using the target image segmentation model to obtain a target binary mask corresponding to the target document image; determining the connected component contours based on the pixel values ​​of the target binary mask, and determining the first and second regions of the target document image based on the connected component contours. The first region is the document where the text is not occluded by the occlusion pattern, and the second region is the document where the text is occluded by the occlusion pattern; the occlusion patterns include seal patterns, watermark patterns, and fingerprint patterns. In one specific implementation, the target image segmentation model is a weight-frozen Segment Anything model. In the output binary mask M, a pixel value of 1 indicates an occluded region. The segmentation result only marks the occluded region without modifying the pixel value, and the connected component contours of the retained binary mask M can be used for subsequent "stamp / fingerprint integrity verification". This ensures that the visual form and legal validity of occluded patterns such as stamps and fingerprints are fully preserved, meeting the requirements of high compliance scenarios such as judicial and financial institutions.

[0021] Step S12: Generate a target structured prompt based on the text recognition result corresponding to the first region document, the first page position, and the target geometric features; the target geometric features are the geometric features corresponding to the occlusion pattern.

[0022] In this embodiment, before generating the target structured prompt based on the text recognition result corresponding to the first region document, the first page position, and the target geometric features, the method further includes: performing optical character recognition on the target document image to obtain the text recognition result corresponding to the first region document, and determining the first region document's position on the first page corresponding to the target document. When determining the text recognition result T_ocr, the corresponding recognition weight C_ocr can also be determined simultaneously.

[0023] In one specific implementation, generating a target structured prompt based on the text recognition result corresponding to the first region of the document, the first page position, and the target geometric features may include: performing OCR recognition on the unoccluded region, for example, using PaddleOCR (Paddle Optical Character Recognition, an OCR tool library) to obtain a text list [(text, bbox, conf)]; then inputting the OCR result of the unoccluded text region into an LLM (Large Language Model), converting it into a JSON Schema, with the following JSON structure: Contract field = { "Party A": ..., "Party B": ..., "Amount": ..., "date": ..., "Seal Type": "Round Red Seal / Oval Seal / Fingerprint" "Seal Color": ... }; Finally, the JSON is converted into a natural language prompt (i.e., a target structured prompt), with the following structure: "This contract is between Party A (XX Company) and Party B (YY Company), amount..." ,date The seal is a red circular seal, approximately 3.5 cm in diameter, which may obscure the amount and date fields. Based on the above, we can infer the "obscured fields". The target geometric feature is the geometric feature corresponding to the obscuring pattern, i.e., the seal type, seal color, etc.

[0024] Step S13: Using the target large language model, semantic reconstruction is performed on the second region document based on the target structured prompts to obtain the target reconstructed document, and the target reconstructed text corresponding to the target document image is generated based on the text recognition result, the first region document and the target reconstructed document.

[0025] In this embodiment, the target reconstructed document is obtained by semantically reconstructing the second region document using a target large language model based on target structured prompts. This includes: using the target large language model based on target structured prompts to perform semantic reconstruction on the second region document to determine all reconstructed documents corresponding to the second region document and their corresponding document weights; and determining the target reconstructed document from all reconstructed documents based on the document weights corresponding to the reconstructed documents. In other words, this embodiment uses a target large language model (e.g., a 7B large model fine-tuned for the legal domain) corresponding to the document to be reconstructed. Based on the target structured prompts obtained in the aforementioned process, semantic reconstruction is performed on the second region document (i.e., the document in the occluded region), thereby obtaining all reconstructed documents corresponding to the occluded region and their corresponding document weights. Then, based on the document weights, the target reconstructed document T_llm and its corresponding document weight C_llm are determined from all reconstructed documents. Here, the LLM output includes Chain-of-Thought reasoning text, which can be used for subsequent manual review or judicial auditing.

[0026] In this embodiment, before generating the target reconstructed text corresponding to the target document image based on the text recognition results, the first region document, and the target reconstructed document, the method further includes: determining the occlusion ratio of the target document image based on the connected component contours; determining whether the second region document meets the glyph generation conditions based on the occlusion ratio; if the second region document meets the glyph generation conditions, then using the target image generation model, generating the target glyph image T_img and glyph weight C_img corresponding to the second region document based on the target structured prompts. For example, if the occlusion area is less than 20%, ControlNet is used to generate the glyph image G with prompt P as the condition; if the occlusion area is greater than or equal to 20%, this step is skipped to prevent illusions. By fusing the target glyph image generated here with the LLM output for glyph similarity, the confidence level can be further improved.

[0027] Furthermore, to improve the credibility of the final reconstructed text, in this embodiment, as follows: Figure 2As shown, the process of generating target reconstructed text corresponding to the target document image based on the text recognition result, the first region document, and the target reconstructed document includes: generating target reconstructed text corresponding to the target document image based on the document weight corresponding to the target reconstructed document, the target glyph image and glyph image weight corresponding to the second region document, and the recognition weight corresponding to the text recognition result, and determining the confidence level of the reconstruction result corresponding to the target reconstructed text; correspondingly, after obtaining the target reconstructed text, the process also includes: judging whether the target reconstructed text meets the review conditions based on the confidence level of the reconstruction result, so as to perform the corresponding text review operation when the target reconstructed text meets the review conditions. For example, based on the text recognition results, the target glyphs (if any) corresponding to the first and second region documents, and the target reconstructed text corresponding to the target document image generated from the target reconstructed document, for the same occluded region, candidate characters obtained in the aforementioned process and their corresponding weights, including T_ocr and C_ocr (OCR of the unoccluded region), T_llm and C_llm (output of the target large model), and T_img and C_img (optional ControlNet-generated glyphs), are used in a multimodal fusion process with weighted voting to achieve multimodal fusion, thereby obtaining the final target reconstructed text and the corresponding reconstruction result confidence C_final=softmax([C_ocr, C_llm, C_img])·[T_ocr, T_llm, T_img]. If C_final is less than 0.7, the target reconstructed text is considered unsatisfactory and meets the review criteria, marked as "manual review" for the corresponding text review operation. Based on the above document reconstruction process, the occluded text is completely restored, and key information is no longer lost. Multi-level confidence filtering significantly reduces the workload of manual review, making business processes simpler and more reliable.

[0028] It is understandable that the document reconstruction system implemented based on this embodiment runs smoothly and can process documents in real time on ordinary hardware, with a low deployment threshold. In addition, an electronic contract processing system can also be implemented based on this embodiment. Besides being used for document reconstruction, it can also be integrated into a scanner or mobile terminal SDK (Software Development Kit) to achieve high-precision recovery and compliance auditing of text obscured by stamps / fingerprints in real time.

[0029] As can be seen, this application first utilizes a target image segmentation model to accurately segment the areas occluded by seals or fingerprints in the target document image corresponding to the paper document to be reconstructed. Then, the text recognition results, layout position, and geometric features of the seal / fingerprint in the unoccluded area (i.e., the first document region) are encoded into target structured prompts, which are then input into a target large language model to perform semantic reconstruction of the occluded text in the second document region. Finally, by combining the original text recognition results and the target reconstructed document, a highly reliable target reconstructed text is output. The entire process significantly improves the recognition accuracy of text in occluded areas while preserving the integrity of the seal / fingerprint, and can be widely applied in high-requirement scenarios such as finance, judiciary, and government affairs.

[0030] See Figure 3 As shown, this application discloses a document reconstruction apparatus, comprising: The document segmentation module 11 is used to acquire the target document image corresponding to the document to be reconstructed, and to perform image segmentation on the target document image using a target image segmentation model to determine the first region document and the second region document of the target document image; the first region document is the document in the region where the text is not occluded by the occlusion pattern, and the second region document is the document in the region where the text is occluded by the occlusion pattern; the occlusion pattern includes a stamp pattern, a watermark pattern, and a fingerprint pattern. The prompt generation module 12 is used to generate a target structured prompt based on the text recognition result corresponding to the first region document, the first page position, and the target geometric features; the target geometric features are the geometric features corresponding to the occlusion pattern. The document reconstruction module 13 is used to perform semantic reconstruction of the second region document based on the target structured prompts using the target large language model to obtain the target reconstructed document, and to generate the target reconstructed text corresponding to the target document image based on the text recognition result, the first region document and the target reconstructed document.

[0031] As can be seen, this application first utilizes a target image segmentation model to accurately segment the areas occluded by seals or fingerprints in the target document image corresponding to the paper document to be reconstructed. Then, the text recognition results, layout position, and geometric features of the seal / fingerprint in the unoccluded area (i.e., the first document region) are encoded into target structured prompts, which are then input into a target large language model to perform semantic reconstruction of the occluded text in the second document region. Finally, by combining the original text recognition results and the target reconstructed document, a highly reliable target reconstructed text is output. The entire process significantly improves the recognition accuracy of text in occluded areas while preserving the integrity of the seal / fingerprint, and can be widely applied in high-requirement scenarios such as finance, judiciary, and government affairs.

[0032] In one specific embodiment, the document segmentation module 11 may include: The mask acquisition unit is used to perform image segmentation and edge alignment on the target document image using a target image segmentation model to obtain a target binary mask corresponding to the target document image; A region determination unit is used to determine the contour of a connected component based on the pixel values ​​of the target binary mask, so as to determine a first region document and a second region document of the target document image based on the contour of the connected component.

[0033] In one specific embodiment, the device may further include: A ratio determination module is used to determine the occlusion ratio of the target document image based on the connected component contour; The condition judgment module is used to determine whether the document in the second region meets the conditions for generating glyphs based on the proportion of the occluded image; The glyph generation module is used to generate a target glyph and glyph weight corresponding to the second region document based on the target structured prompts if the second region document meets the glyph generation conditions.

[0034] In one specific embodiment, the document reconstruction module 13 may include: The semantic reconstruction unit is used to perform semantic reconstruction on the second region document based on the target structured prompts using the target large language model, so as to determine all reconstructed documents and corresponding document weights corresponding to the second region document; The document determination unit is used to determine the target reconstructed document from all the reconstructed documents based on the document weight corresponding to the reconstructed document.

[0035] In one specific embodiment, the document reconstruction module 13 may include: The confidence determination unit is used to determine the target reconstructed text based on the document weight corresponding to the reconstructed document, the target glyph image and glyph image weight corresponding to the second region document, and the recognition weight corresponding to the text recognition result, and to determine the confidence of the reconstruction result corresponding to the target reconstructed text. Accordingly, the device may further include: The review module is used to determine whether the target reconstructed text meets the review conditions based on the confidence level of the reconstruction result, so as to perform the corresponding text review operation when the target reconstructed text meets the review conditions.

[0036] In one specific embodiment, the device may further include: An optical character recognition module is used to perform optical character recognition on the target document image to obtain the text recognition result corresponding to the first region document and determine the position of the first region document on the first page of the target document.

[0037] Furthermore, embodiments of this application also disclose an electronic device, Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0038] Figure 4 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the document reconstruction method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0039] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0040] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored thereon can include an operating system 221, computer programs 222, etc., and the storage method can be temporary storage or permanent storage.

[0041] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the document reconstruction method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0042] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned document reconstruction method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0043] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0044] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0045] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0046] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0047] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A document reconstruction method, characterized in that, include: Obtain the target document image corresponding to the document to be reconstructed, and use the target image segmentation model to perform image segmentation on the target document image to determine the first region document and the second region document of the target document image; the first region document is the document in the region where the text is not occluded by the occlusion pattern, and the second region document is the document in the region where the text is occluded by the occlusion pattern. The obscuring patterns include stamp patterns, watermark patterns, and fingerprint patterns; Based on the text recognition results corresponding to the first region document, the first page position, and the target geometric features, a target structure prompt is generated; The target geometric feature is the geometric feature corresponding to the occlusion pattern; Using a target large language model, semantic reconstruction is performed on the second region document based on the target structured prompts to obtain the target reconstructed document. Based on the text recognition results, the first region document, and the target reconstructed document, the target reconstructed text corresponding to the target document image is generated.

2. The document reconstruction method according to claim 1, characterized in that, The step of segmenting the target document image using a target image segmentation model to determine a first region document and a second region document of the target document image includes: The target document image is segmented and edge-aligned using a target image segmentation model to obtain a target binary mask corresponding to the target document image; Based on the pixel values ​​of the target binary mask, the contours of connected components are determined, and the first region document and the second region document of the target document image are determined based on the contours of the connected components.

3. The document reconstruction method according to claim 2, characterized in that, Before generating the target reconstructed text corresponding to the target document image based on the text recognition result, the first region document, and the target reconstructed document, the method further includes: The occlusion ratio of the target document image is determined based on the connected component contour; Based on the proportion of the occluded image, determine whether the document in the second region meets the conditions for generating glyphs; If the second region document meets the glyph generation conditions, then the target glyph and glyph weights corresponding to the second region document are generated using the target image generation model based on the target structured prompts.

4. The document reconstruction method according to claim 1, characterized in that, The step of using a target large language model to perform semantic reconstruction of the second region document based on the target structured prompts to obtain the target reconstructed document includes: Using the target large language model based on the target structured prompts, semantic reconstruction is performed on the second region document to determine all reconstructed documents and their corresponding document weights corresponding to the second region document; The target reconstructed document is determined from all the reconstructed documents based on the document weights corresponding to the reconstructed documents.

5. The document reconstruction method according to claim 4, characterized in that, The step of generating the target reconstructed text corresponding to the target document image based on the text recognition result, the first region document, and the target reconstructed document includes: The target reconstructed text is determined based on the document weight corresponding to the reconstructed document, the target glyph image and glyph image weight corresponding to the second region document, and the recognition weight corresponding to the text recognition result. The confidence level of the reconstruction result corresponding to the target reconstructed text is also determined. Accordingly, after generating the target reconstructed text corresponding to the target document image based on the text recognition result, the first region document, and the target reconstructed document, the method further includes: Based on the confidence level of the reconstruction result, it is determined whether the target reconstructed text meets the review conditions, so that when the target reconstructed text meets the review conditions, the corresponding text review operation is performed.

6. The document reconstruction method according to claim 1, characterized in that, Before generating the target structured prompt based on the text recognition result corresponding to the first region document, the first page position, and the target geometric features, the method further includes: Optical character recognition is performed on the target document image to obtain the text recognition result corresponding to the first region document, and the position of the first region document on the first page of the target document is determined.

7. The document reconstruction method according to any one of claims 1 to 6, characterized in that, The target image segmentation model is a weight-frozen Segment Anything model.

8. A document reconstruction apparatus, characterized in that, include: The document segmentation module is used to acquire the target document image corresponding to the document to be reconstructed, and to perform image segmentation on the target document image using a target image segmentation model to determine the first region document and the second region document of the target document image; the first region document is the document in the region where the text is not occluded by the occlusion pattern, and the second region document is the document in the region where the text is occluded by the occlusion pattern. The obscuring patterns include stamp patterns, watermark patterns, and fingerprint patterns; The prompt generation module is used to generate a target structured prompt based on the text recognition result corresponding to the document in the first region, the position of the first page, and the target geometric features; The target geometric feature is the geometric feature corresponding to the occlusion pattern; The document reconstruction module is used to perform semantic reconstruction of the second region document based on the target structured prompts using the target large language model to obtain the target reconstructed document, and to generate the target reconstructed text corresponding to the target document image based on the text recognition result, the first region document and the target reconstructed document.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the document reconstruction method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the document reconstruction method as described in any one of claims 1 to 7.

Images