System for automatic detection and inpainting of stamp and signature areas in documents

Abstract

The system for automatically searching for seal and signature areas within a document and processing in-painting according to the embodiment enables the automatic search for suitable spaces within a document where a seal or signature can be inserted. Furthermore, the embodiment allows for the preservation of the document's text and layout while removing existing seals or signatures. Additionally, the embodiment enables the provision of a result consistent with the original document by accurately restoring damaged text and font information after in-painting through in-painting technology. Moreover, the embodiment enables the improvement of model performance through efficient dataset construction and labeling methods.

Description

Technology Field

[0001] The technical concept of the present disclosure relates to a system for automatically searching for and processing seal and signature areas within a document, and more specifically, to a system for automatically searching for a suitable space within a document where a seal or signature can be inserted. Background Technology

[0002] Unless otherwise indicated in this specification, the contents described in this section are not prior art for the claims of this application, and are not to be recognized as prior art simply because they are included in this section.

[0003] With the advancement of digital document processing, there is a growing need to automate the tasks of inserting or removing seals or signatures from various official documents, such as trade and government papers. However, the application of conventional digital document processing is limited in actual work environments due to several issues. First, traditional methods often make it difficult to automatically identify suitable spaces for inserting seals or signatures, frequently requiring reliance on manual work. This problem stems from the diversity and complexity of document layouts and is one of the primary causes of reduced work efficiency.

[0004] Furthermore, there is a high likelihood that important text and layout within a document will be damaged during the process of inserting or removing seals or signatures. Particularly in situations where document consistency is required, such damage can significantly impair the credibility and readability of the document.

[0005] Furthermore, the lack of specialized datasets to perform the task leads to reduced accuracy and reliability of the trained model. This acts as a major obstacle to the development of document processing technology and hinders technological advancement.

[0006] Furthermore, commonly used inpainting techniques fail to adequately reflect the text and layout characteristics of a document. In particular, when a seal or signature overlaps the text, it is difficult to accurately restore the text, making it hard to achieve a natural result. This issue acts as a significant limitation in tasks aimed at preserving the original state of a document. The aforementioned problems serve as major obstacles to the advancement of automation technology in digital document processing, necessitating new approaches and technical improvements to address them.

[0007] Meanwhile, inpainting is a technique that restores damaged areas in images or videos, or removes specific parts and naturally fills the space. General inpainting techniques focus on removing specific areas from an image and restoring them to seamlessly blend with the surroundings. However, in documents, text and layout are critical; particularly when a seal or signature overlaps the text, conventional inpainting techniques have limitations in that they cannot accurately restore the text and may even damage it. Prior art literature

[0008] 1. Korean Patent Publication No. 10-2024-0026060 (Feb. 27, 2024) 2. Korean Patent Publication No. 10-2009-0036918 (April 15, 2009) The problem to be solved

[0009] The automatic search and in-painting processing system for seal and signature areas within a document according to the embodiment enables the automatic search for a suitable space within a document where a seal or signature can be inserted.

[0010] In addition, the embodiment allows the text and layout of the document to be preserved while removing existing seals or signatures.

[0011] In addition, the embodiment enables the accurate restoration of damaged text and font information after inpainting through inpainting technology, thereby providing a result that matches the original document.

[0012] In addition, the embodiments enable the improvement of model performance through efficient dataset construction and labeling methods.

[0013] However, the problem to be solved according to one embodiment is not limited to that mentioned above. means of solving the problem

[0014] An automatic search and in-painting device for seal and signature areas within a document according to an embodiment includes a memory that stores at least one instruction; and a processor that performs an operation according to the instruction, wherein the processor collects a data set for seal areas and signature areas, constructs an in-painting model through learning of the collected data set, and the in-painting model reflects the text and layout structure of the input document so as to preserve information of the original document when the seal or signature is removed.

[0015] In addition, the processor can perform labeling on text and font information below the seal area and signature area, and collect the labeled data as a data set.

[0016] In addition, the inpainting model can search for coordinates for inserting or removing a paint area and a signature area, and can identify the paint area and the signature area using the searched coordinates.

[0017] In addition, the inpainting model can extract text located below the seal area or signature area through OCR, and reconstruct the extracted text according to the font and size of the original document to insert it into the inpainted area.

[0018] In addition, the inpainting model can protect the document's text and layout elements during the inpainting process.

[0019] Additionally, the layout element may include at least one of a table, a line, or a paragraph.

[0020] In addition, the processor can evaluate the performance of the learned inpainting model and update the inpainting model based on the evaluation results.

[0021] In addition, the inpainting model can perform a verification process to evaluate the accuracy of the restored document.

[0022] In addition, the inpainting model can set a protection area for layout protection and perform protection processing when the seal and signature are removed.

[0023] In addition, to evaluate the performance of the inpainting model, the processor can measure the structural similarity of the restored image and verify whether the text restored under the seal and signature is accurately extracted by OCR. Effects of the invention

[0024] The automatic search and in-painting processing system for seal and signature areas within a document according to the embodiment automates the search and removal of seal and signature areas, which were previously performed manually, thereby saving time in document processing and improving work efficiency.

[0025] In addition, the embodiment preserves the text and layout of the document and enables the provision of a result of the same quality as the original even after inpainting.

[0026] In addition, the embodiment reduces human resource costs through the introduction of an automated system and expands the scope of application by making it applicable to various types of documents, such as contracts and administrative documents, in addition to trade documents.

[0027] In addition, through the embodiments, the processing of import and export documents is automated, and the electronic processing of contracts and official documents is enabled.

[0028] In addition, through the embodiments, the digitization and automation of administrative documents are enabled, and added value can be created through integration with an electronic document management system.

[0029] In addition, the embodiments allow for the use of inpainting technology to remove unnecessary or existing seals and signatures from documents.

[0030] In addition, the embodiment develops an inpainting model specialized for documents to accurately restore the text and layout beneath a seal or signature. This prevents the loss of important information in the document.

[0031] In addition, the embodiment extracts text obscured by a seal or signature using OCR technology, reconstructs it to match the font and size used in the document, and inserts it into the inpainted area, thereby enabling the provision of a result that matches the original document even after inpainting. Since documents generally contain complex structural elements such as tables, lines, and paragraphs, the embodiment processes the elements included in the document to prevent damage during the inpainting process, thereby preserving the text and layout of the document and accurately restoring obscured information when the seal and signature are removed.

[0032] In addition, through the embodiments, the limitations of general inpainting are overcome and a document-specific inpainting model is developed, thereby enabling automated processing while maintaining the integrity and consistency of the document.

[0033] The effects obtainable from the exemplary embodiments of the present disclosure are not limited to those mentioned above, and other unmentioned effects can be clearly derived and understood by those skilled in the art to which the exemplary embodiments of the present disclosure belong from the description below. That is, unintended effects resulting from the implementation of the exemplary embodiments of the present disclosure can also be derived by those skilled in the art from the exemplary embodiments of the present disclosure. Brief explanation of the drawing

[0034] FIG. 1 is a drawing showing an automatic detection and in-painting processing system for seal and signature areas within a document according to an embodiment. FIG. 2 is a drawing showing the configuration of an automatic detection and in-painting device for seal and signature areas within a document according to an embodiment. FIG. 3 is a drawing illustrating a method for automatically detecting and in-painting seal and signature areas within a document according to an embodiment. Specific details for implementing the invention

[0035] Hereinafter, various embodiments of the present disclosure are described in conjunction with the accompanying drawings. As various embodiments of the present disclosure may be subject to various modifications and may have various forms, specific embodiments are illustrated in the drawings and described in detail. However, this is not intended to limit the various embodiments of the present disclosure to specific forms, and it should be understood that they include all modifications and / or equivalents and substitutions that fall within the spirit and scope of the various embodiments of the present disclosure. In relation to the description of the drawings, similar reference numerals have been used for similar components.

[0036] In various embodiments of the present disclosure, terms such as “comprising” or “having” are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0037] In various embodiments of the present disclosure, expressions such as “or” include any and all combinations of the words listed together. For example, “A or B” may include A, may include B, or may include both A and B.

[0038] Expressions such as "first," "second," "first," or "second" used in various embodiments of the present disclosure may modify various components of the various embodiments, but do not limit such components. For example, such expressions do not limit the order and / or importance of such components and may be used to distinguish one component from another.

[0039] When it is mentioned that a component is "connected" or "joined" to another component, it should be understood that the component may be directly connected or joined to the other component, but that a new component may also exist between the component and the other component.

[0040] In the embodiments of the present disclosure, terms such as "module," "unit," "part," etc. are used to refer to a component that performs at least one function or operation, and such component may be implemented in hardware or software, or in a combination of hardware and software. Additionally, a plurality of "modules," "units," "parts," etc. may be integrated into at least one module or chip and implemented as at least one processor, except where each needs to be implemented in specific individual hardware.

[0041] Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with their meanings in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in the various embodiments of the present disclosure.

[0042] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0043] FIG. 1 is a diagram showing an automatic search and in-painting processing system for seal and signature areas within a document according to an embodiment.

[0044] Referring to FIG. 1, the automatic search and in-painting processing system for seal and signature areas within a document according to an embodiment may be configured to include an in-painting device (100) and a user terminal (200). In the embodiment, the in-painting device (100) constructs a trade document dataset containing various seals and signatures. Additionally, it performs labeling not only on the seal and signature areas but also on the text and font information below those areas, includes the labeled data in the dataset, and builds an in-painting model by training the dataset. In the embodiment, the in-painting model utilizes a deep learning-based object detection model to insert or remove seals and signatures. To this end, it automatically searches for coordinates for inserting and removing signatures and enables accurate identification of existing seal and signature areas. Furthermore, the in-painting device (100) constructs a document-specific in-painting model to overcome the limitations of existing in-painting models. The in-painting model provided in the embodiment reflects the text and layout structure of the document to ensure that the information of the original document is preserved even when the seal or signature is removed. In addition, the inpainting model according to the embodiment utilizes OCR technology to extract text located beneath a seal or signature, and reconstructs the extracted text to match the font and size of the original document, inserting it into the inpainted area. The embodiment improves accuracy by leveraging the fact that the types of fonts used in the document are limited. Furthermore, it protects important text or layout elements of the document from being damaged during the inpainting process. In the embodiment, the inpainting model ensures the consistency of the document by maintaining structural elements such as tables, lines, and paragraphs. Additionally, in the embodiment, the inpainting device (100) quantitatively and qualitatively evaluates the performance of the developed model. Furthermore, it enables continuous improvement to ensure stable performance even with complex layouts or various fonts.

[0045] The user terminal (200) is a user terminal that searches for a stamp area and a signature area for applying inpainting to a document and processes the inpainting. In an embodiment, the user terminal (200) may receive an inpainting model built by the inpainting device (100), install it, and use it, or communicate with the inpainting device (100) and use the inpainting model. Here, at least one user terminal (200) may be implemented as a computer capable of connecting to a remote server or terminal via a network. Here, the computer may include, for example, a laptop, desktop, or laptop equipped with a navigation system or a web browser. At this time, at least one user terminal (200) may be implemented as a terminal capable of connecting to a remote server or terminal via a network. At least one user terminal (200) may include all kinds of handheld-based wireless communication devices, such as navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminal, smartphone, smartpad, tablet PC, etc.

[0046] The system for automatically searching for seal and signature areas within a document and processing in-painting according to the embodiment enables the automatic search for suitable spaces within a document where a seal or signature can be inserted. Furthermore, the embodiment allows for the preservation of the document's text and layout while removing existing seals or signatures. Additionally, the embodiment enables the provision of a result consistent with the original document by accurately restoring damaged text and font information after in-painting through in-painting technology. Moreover, the embodiment enables the improvement of model performance through efficient dataset construction and labeling methods.

[0047] FIG. 2 is a block diagram of an in-painting device according to an embodiment.

[0048] In the embodiment, the inpainting device may be implemented as a server. In the embodiment, the server is a computing system that provides services to other computers or devices in a computer network or stores and manages data. The server accepts requests from other computers or devices called clients and provides responses or data in response to those requests. The configuration of the inpainting device illustrated in FIG. 2 is merely a simplified example.

[0049] The communication module (110) can be configured regardless of the mode of communication, such as wired or wireless, and can be configured with various communication networks, such as a Personal Area Network (PAN) or a Wide Area Network (WAN). Additionally, the communication module (110) can operate based on the known World Wide Web (WWW) and may utilize wireless transmission technologies used for short-range communication, such as Infrared Data Association (IrDA) or Bluetooth. For example, the communication module (110) may be responsible for transmitting and receiving data necessary to perform a technique according to one embodiment of the present disclosure.

[0050] Memory (120) may refer to any type of storage medium. For example, memory (120) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk. Such memory (120) may also constitute the database shown in FIG. 1.

[0051] The memory (120) can store at least one instruction that can be executed by the processor (130). Additionally, the memory (120) can store any form of information generated or determined by the processor (130) and any form of information received by the server (200). Additionally, the memory (120) stores various types of modules, instruction sets, or models.

[0052] The processor (130) can perform technical features according to embodiments of the present disclosure described below by executing at least one instruction stored in memory (120). In one embodiment, the processor (130) may be composed of at least one core and may include a processor for data analysis and / or processing, such as a central processing unit (CPU) of a computer device, a general purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU). Such a processor (130) can train a neural network or model designed in a machine learning or deep learning manner. To this end, the processor (130) can perform calculations for training the neural network, such as processing input data for training, extracting features from input data, calculating errors, and updating the weights of the neural network using backpropagation. Additionally, the processor (130) may perform inference for a specific purpose using a model implemented in an artificial neural network manner.

[0053] In an embodiment, the processor (130) collects a data set for a paint area and a signature area, and builds an inpainting model through learning the collected data set.

[0054] To this end, the processor (130) collects a dataset of seal areas and signature areas. The dataset consists of document images containing seals and signatures of various shapes and sizes, and includes various conditions including the text of the document, layout, transparency of the seal, and signature characteristics. In this process, the locations of the seals and signatures are accurately labeled to increase the reliability and precision of the dataset. In the embodiment, the collected dataset is used to learn the features of the seal areas and signature areas. To this end, the processor (130) trains an inpainting model including a deep learning-based artificial neural network (e.g., GAN, U-Net, etc.). During the training process, the model learns how to detect and remove the corresponding area in a document containing a seal or signature, and then naturally restore the damaged area based on the surrounding text and layout.

[0055] In the embodiments, the inpainting model uses an architecture optimized for restoring the text and layout of a document. It is designed to restore the original text and layout without damage, even when a seal or signature area overlaps with the text. To achieve this, the model simultaneously considers the document's layout information, such as the shape, size, and spacing of the text, to generate natural and consistent results.

[0056] In addition, the embodiments apply reinforcement learning and transfer learning techniques to enhance accuracy in the process of detecting seal and signature areas. Through this, it demonstrates high adaptability to new forms of seals or signatures that have not been previously learned, and provides consistent performance across various document formats.

[0057] The inpainting model constructed in the aforementioned manner can overcome the difficulties of spatial exploration in existing technologies, text and layout corruption, dataset scarcity, and the limitations of general inpainting techniques. This enables the automation of digital document processing to be performed more efficiently and reliably.

[0058] Additionally, the processor (130) performs labeling on text and font information under the seal area and signature area, and collects the labeled data as a data set. To do this, the processor (130) receives digital document images of various formats as input. These documents consist of PDFs containing seal or signature areas, image files (JPG, PNG, etc.), or scanned documents. Subsequently, the processor (130) detects the seal area and signature area in the document image using a deep learning-based object detection model (e.g., Faster R-CNN, YOLO, etc.). The detected area is displayed in the form of a bounding box, and the location and size of this area are stored. Subsequently, to extract text and font information under the detected seal and signature area, the processor (130) applies Optical Character Recognition (OCR) technology. The OCR engine identifies text hidden under the seal and signature area and extracts information such as text content, font type, font size and color, text alignment, and spacing. In the embodiment, the processor (130) labels the extracted text and font information by associating it with the detected seal and signature areas. Each data point is stored in a structure that includes detailed attributes such as the location of the seal / signature area (Bounding Box coordinates), the text content below the area, and the font type, size, and color of the text. Subsequently, the processor (130) systematically stores the labeled data to form a dataset. The dataset includes the original image of the document, the location of the seal and signature areas, and the text and font attribute information below the areas. This dataset can be stored in a format such as JSON or CSV and used for future model training. To increase the reliability of the collected dataset, the processor (130) performs an automated verification process. For example, it evaluates the accuracy of the OCR or reviews the degree of agreement between the detected seal and signature areas and the text information.If necessary, correct errors manually or through correction algorithms.

[0059] The dataset generated in the example precisely includes text and font information beneath the seal and signature areas and serves as important foundational data for training the inpainting model. This enhances the quality of the dataset and enables the derivation of highly accurate inpainting results across various document formats.

[0060] In the embodiment, the inpainting model reflects the text and layout structure of the input document to preserve information of the original document when a stamp or signature is removed. When the input document is received by the processor (130), the model extracts the text and layout structure of the document using an OCR (Optical Character Recognition) and document layout analysis algorithm. In the embodiment, the text information includes character content, font type, size, color, spacing, etc., and the layout information may include the overall arrangement and format of the document, such as text block locations, paragraph structure, tables, and images.

[0061] Subsequently, the inpainting model automatically detects seal and signature areas within a document using an artificial neural network-based object detection model (e.g., Faster R-CNN, YOLO, etc.). The detected areas are displayed as bounding boxes, and their location information is associated with the document's text and layout structure. Then, to ensure that the text within the detected seal and signature areas remains intact, the model applies text restoration and layout preservation processes.

[0062] To restore text, the inpainting model pre-extracts text existing beneath a seal or signature using OCR and integrates it into an inpainting algorithm to restore the information without loss. Subsequently, to maintain the layout, the model restores the document so that the spacing and alignment with surrounding text blocks are identical to the original document after the seal and signature have been removed. In the embodiment, the inpainting model can correct text and spatial placement based on the document's layout information. Then, the inpainting model utilizes a Generative Adversarial Network (GAN) or U-Net-based algorithm to remove the seal and signature areas and restore them naturally by reflecting the surrounding text and layout structure. Furthermore, the model, having learned the document's text patterns and styles, restores the text in the removed areas using the same font, size, and spacing as the original. Additionally, it ensures that non-text elements, such as backgrounds, tables, and images damaged during the seal and signature removal process, are naturally filled in.

[0063] Additionally, to increase the accuracy of the restored document, the model performs an automated quality check. If the restored text and layout do not match the information of the original document, the processor (130) performs additional correction work to ensure the reliability of the final result.

[0064] The inpainting model according to the embodiment thoroughly reflects the text and layout structure of the input document, thereby minimizing information loss of the original document when removing seals and signatures and generating natural and reliable results. This maintains document consistency and significantly improves the efficiency of automated document processing.

[0065] Additionally, the inpainting model searches for coordinates for inserting or removing the seal area and the signature area, and identifies the seal area and the signature area using the searched coordinates. To this end, the processor (130) receives digital document images of various formats. In the embodiment, the input document image is converted into a form that is easy to analyze through preprocessing steps such as resolution adjustment, color correction, and noise removal. Subsequently, the inpainting model searches for coordinates for inserting or removing the seal and signature area. At this time, the inpainting model utilizes a deep learning-based object detection model (e.g., Faster R-CNN, YOLO, etc.) to detect areas in the document where the seal and signature need to be inserted or removed. Additionally, the model searches for text blocks, empty spaces, and locations where the seal or signature exists within the document, and generates coordinates based on this. Subsequently, the inpainting model analyzes the text and layout structure of the document during the detection process. It calculates the optimal coordinates where the seal and signature can be inserted or removed by considering text spacing, alignment, font size, etc. Subsequently, coordinate optimization and candidate areas are selected. In the embodiments, the inpainting model optimizes coordinates based on the size and location of surrounding text blocks and blank areas so that the document layout is not damaged when inserting or removing a seal and signature. Additionally, it determines priorities by applying weights to multiple detected candidate areas. In the embodiments, weight calculation may be performed based on spacing with text to minimize the possibility of collision, consistency of the document layout, compatibility with the size of the seal or signature, and identification elements of the seal and signature area.

[0066] Subsequently, the inpainting model identifies the seal and signature areas based on the discovered coordinates. During the identification process, a bounding box of the area is generated, and information regarding the size, shape, and location of the seal and signature is stored. In the embodiment, the processor (130) continuously trains the model by adding the identified seal and signature area data to the dataset. Additionally, the accuracy of the discovery and identification process is progressively improved through further training. In the embodiment, the detection and identification results can be verified by visualizing them as an overlay on the document image. The processor (130) automatically evaluates whether the seal and signature areas were accurately detected and whether insertion or removal did not affect the text and layout of the document.

[0067] In the embodiments, the inpainting model can significantly improve the accuracy and efficiency of document processing by automatically searching for coordinates for the insertion or removal of seal and signature areas and utilizing those coordinates to precisely identify the areas. This reduces the reliance on manual work, which was a limitation of existing technologies, and enhances the reliability of digital document automation.

[0068] Additionally, the inpainting model extracts text located below the seal area or signature area through OCR, reconstructs the extracted text according to the font and size of the original document, and inserts it into the inpainted area. The input document image is preprocessed by a processor (130). The preprocessing process includes image resolution improvement, noise removal, color correction, etc.

[0069] In the embodiment, a deep learning-based object detection model (e.g., Faster R-CNN, YOLO, etc.) is used to detect seal and signature areas in a document. The detected areas are stored in the form of bounding boxes, and the location information is used for subsequent operations.

[0070] Subsequently, the inpainting model applies OCR technology to the detected seal and signature areas to extract the text hidden beneath them. The OCR engine identifies the document's text from pixel data and extracts the content of the text obscured by the seal or signature. In the embodiment, the text data is stored as a string, and additional attribute information, such as the text's location, size, and style, is also extracted. The inpainting model then analyzes the font type, size, color, and alignment information used in the entire text of the document. It estimates the font attributes of the original document based on the text surrounding the detected seal and signature areas. The analyzed attributes are utilized to maintain text consistency and document readability. Furthermore, the inpainting model reconstructs the extracted text according to the original document's font type and size. For example, it selects the same font by comparing the OCR results with the original document's font analysis results. It then adjusts the text to match the attributes of the text line where the seal or signature area is located. Finally, it accurately positions the text based on alignment and spacing with surrounding text blocks. In the embodiment, the reconstructed text is inserted into the inpainted area after the seal and signature areas are removed.

[0071] Subsequently, the inpainting model utilizes a Generative Adversarial Network (GAN) or U-Net-based inpainting algorithm to restore the background of the corresponding area after removing the seal and signature. In the embodiment, reconstructed text is inserted over the restored background to maintain the consistency of the document. Blending techniques may be applied during the insertion process to ensure natural integration between the text and the document background. In the embodiment, the inpainting model performs an automated verification process to evaluate the accuracy of the restored document. For instance, it compares whether the text extracted by OCR matches the inserted text with the original document. At this stage, the inpainting model examines whether the text alignment, size, color, spacing, etc., harmonize with other text in the document. If necessary, a correction algorithm is applied to modify the final result.

[0072] In the embodiment, OCR and in-painting technologies are combined to accurately restore text beneath the seal and signature areas, and by faithfully reflecting the font and layout attributes of the original document, the consistency and reliability of the document can be maintained. This allows for a significant improvement in the processing efficiency of digital documents.

[0073] Additionally, the inpainting model protects the text and layout elements of the document during the inpainting process. In the embodiment, the layout elements may include at least one of a table, a line, or a paragraph. To this end, the processor (130) preprocesses the input document to perform resolution adjustment, noise removal, color correction, etc. Subsequently, a deep learning-based document layout analysis model (e.g., LayoutLM, PubLayNet) is used to detect layout elements such as text, tables, lines, and paragraphs in the document. In the embodiment, the detected elements are stored as bounding boxes and attribute information (position, size, style, etc.). Additionally, the document structure is defined by analyzing the interrelationships of the layout elements (spacing between text, alignment of paragraphs and tables, etc.).

[0074] In the embodiment, the inpainting model utilizes object detection models such as Faster R-CNN and YOLO to detect seal and signature areas in a document. Subsequently, it analyzes whether the detected seal and signature areas overlap with text, tables, lines, or paragraphs in the document. If they overlap, a processing step is performed to protect the corresponding layout elements during the inpainting process.

[0075] In the embodiment, the inpainting model may set a protection area to protect the layout. For example, the inpainting model sets a protection area based on the bounding box of a layout element that overlaps with the seal and signature area. Pixel data of the area is stored separately so as not to damage the attributes of text, tables, lines, and paragraphs within the protection area.

[0076] In addition, the inpainting model performs protection processing when the seal and signature are removed. In the embodiment, a Generative Adversarial Network (GAN) or U-Net-based inpainting algorithm is applied, but data within the protected area is excluded from the restoration process to maintain the original state. Furthermore, the inpainting model naturally restores only the background surrounding the protected area to harmonize with the layout elements.

[0077] Subsequently, the inpainting model extracts text obscured beneath the seal and signature via OCR and restores it based on the original document's font, size, and alignment attributes. Additionally, detected tables and lines are processed separately to prevent damage during the seal and signature removal process. If necessary, the table borders and line thickness and position are reconstructed to restore the document to be identical to the original. Afterward, the inpainting model ensures a natural connection between the restored text and the original text based on paragraph spacing and alignment information. Furthermore, the embodiment automatically verifies whether the layout elements of the restored document maintain the same structure as the original document. It also analyzes the relationships and spacing between text, tables, lines, and paragraphs to confirm consistency.

[0078] In the embodiment, if the verification results show that the structure of the inpainting model differs from the original document by a certain level or more, a correction algorithm is applied to improve the precision of the layout. In the embodiment, by protecting key layout elements such as text, tables, lines, and paragraphs during the inpainting process, the structural consistency of the original document is maintained, and the reliability and accuracy of digital document processing can be significantly improved.

[0079] Additionally, the processor (130) evaluates the performance of the learned inpainting model and updates the inpainting model based on the evaluation results. To this end, the processor (130) prepares various document images containing seal and signature areas, including new unlearned document data, as an evaluation dataset. In an embodiment, the dataset may include various layout structures (simple text, including tables, complex paragraphs, etc.), various seal and signature locations, sizes, shapes, and various attributes of obscured text and backgrounds.

[0080] In the embodiment, the processor (130) evaluates the performance of the learned inpainting model using the following quantitative indicators. For example, it calculates the Peak Signal-to-Noise Ratio (PSNR) to numerically express the quality of the restored area and evaluates the difference between the original image and the restored image. Additionally, it measures the structural similarity of the restored image using the Structural Similarity Index (SSIM) and verifies whether the text restored under the seal and signature is accurately extracted by OCR. Furthermore, the processor (130) evaluates how well the layout elements of the document, such as text, tables, and paragraphs, match the original. Subsequently, based on the evaluation results, the processor (130) quantifies the performance difference with the existing model to analyze the accuracy of the model.

[0081] Additionally, the processor (130) performs a restoration quality analysis for qualitative evaluation. In the embodiment, the processor (130) evaluates whether a visually natural result was obtained after restoring the seal and signature areas, and verifies whether layout elements such as text, tables, lines, and paragraphs of the document were not damaged during the seal and signature removal process. Additionally, the restored document is provided to experts or users to receive a quality evaluation. In the embodiment, subjective evaluations (e.g., readability, naturalness, consistency) may be collected. Subsequently, the processor (130) integrates the quantitative and qualitative evaluation results to identify the strengths and weaknesses of the model. Subsequently, it analyzes specific types of documents or seal and signature cases with degraded performance to identify the cause. Additionally, the processor (130) reinforces the dataset focusing on cases with poor performance and retrains the inpainting model using the reinforced dataset.

[0082] In the embodiment, a transfer learning technique is utilized to adapt to newly added data while maintaining the performance of the existing model. Additionally, the processor (130) optimizes the architecture of the inpainting model to improve performance. For example, specialized modules for text-centered restoration may be added, or the loss function may be adjusted to enhance layout preservation. Subsequently, the processor (130) performs the same quantitative and qualitative evaluations on the updated model to verify whether performance has improved. In the embodiment, improvements are identified through a performance comparison between the existing model and the updated model, and the processor (130) automates a continuous quality improvement process based on the evaluation results, thereby enabling the performance improvement process to be performed repeatedly.

[0083] In the embodiment, the performance of the inpainting model is comprehensively evaluated quantitatively and qualitatively, and by continuously updating the model based on this, the accuracy and efficiency of digital document processing can be enhanced, and high reliability can be provided for various document types.

[0084] Below, we will look at FIG. 3. The automatic search and in-painting method for the seal area and signature area within the contract document shown in FIG. 3 can be performed by a contract automatic generation device (100) including a processor (130).

[0085] Meanwhile, FIG. 3 is merely illustrative, and the concept of the present invention is not to be interpreted as being limited to that shown in FIG. 3. For example, each step may be configured in a different order than that shown in FIG. 3, at least one of the steps shown in FIG. 3 may not be performed, or one or more steps not shown in FIG. 3 may be additionally performed.

[0086] Below, the automatic detection and in-painting method for the seal area and signature area within a document will be described in turn. Since the operation (function) of the method according to the embodiment is essentially the same as the function of the automatic detection and in-painting system for the seal area and signature area within a document, descriptions that overlap with FIGS. 1 and FIGS. 2 will be omitted.

[0087] FIG. 3 is a diagram illustrating a method for automatically searching for and in-painting a seal area and a signature area within a document according to an embodiment.

[0088] Referring to FIG. 3, in step S110, a dataset for the seal area and the signature area is collected, and in step S120, an inpainting model is built through learning the collected dataset. In step S130, the text and layout structure of the input document are reflected through the inpainting model, and in step S140, when the seal or signature is removed, the information of the original document is preserved.

[0089] The automatic search and in-painting processing system for seal and signature areas within a document according to the embodiment automates the search and removal of seal and signature areas, which were previously performed manually, thereby saving time in document processing and improving work efficiency.

[0090] In addition, the embodiment preserves the text and layout of the document and enables the provision of a result of the same quality as the original even after inpainting.

[0091] In addition, the embodiment reduces human resource costs through the introduction of an automated system and expands the scope of application by making it applicable to various types of documents, such as contracts and administrative documents, in addition to trade documents.

[0092] In addition, through the embodiments, the processing of import and export documents is automated, and the electronic processing of contracts and official documents is enabled.

[0093] In addition, through the embodiments, the digitization and automation of administrative documents are enabled, and added value can be created through integration with an electronic document management system.

[0094] In addition, the embodiments allow for the use of inpainting technology to remove unnecessary or existing seals and signatures from documents.

[0095] In addition, the embodiment develops an inpainting model specialized for documents to accurately restore the text and layout beneath a seal or signature. This prevents the loss of important information in the document.

[0096] In addition, the embodiment extracts text obscured by a seal or signature using OCR technology, reconstructs it to match the font and size used in the document, and inserts it into the inpainted area, thereby enabling the provision of a result that matches the original document even after inpainting. Since documents generally contain complex structural elements such as tables, lines, and paragraphs, the embodiment processes the elements included in the document to prevent damage during the inpainting process, thereby preserving the text and layout of the document and accurately restoring obscured information when the seal and signature are removed.

[0097] In addition, through the embodiments, the limitations of general inpainting are overcome and a document-specific inpainting model is developed, thereby enabling automated processing while maintaining the integrity and consistency of the document.

[0098] Meanwhile, the methods according to the various embodiments of the present invention described above can be implemented in the form of an application or software program that can be installed on an existing electronic device.

[0099] In addition, the whole or part of the method may be composed of multiple software function modules and implemented on an operating system (OS). Alternatively, each step may be composed of a single software function module, or each step may be combined to form a single software function module and implemented on an operating system. Therefore, even if all of the embodiments of the present disclosure are not implemented as a single software function module, if multiple software function modules implement each step of the present disclosure and multiple software function modules are implemented on a single operating system, it can be understood that the method of the present disclosure has been implemented.

[0100] In addition, the methods according to the various embodiments of the present invention described above can be implemented solely through software upgrades or hardware upgrades of existing electronic devices. Furthermore, the various embodiments of the present invention described above can also be performed through an embedded server equipped in an electronic device or an external server of the electronic device.

[0101] Meanwhile, according to one embodiment of the present invention, the various embodiments described above may be implemented as software comprising instructions stored on a computer-readable recording medium using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented as the processor itself. According to the software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

[0102] Meanwhile, a computer or a similar device may include a device according to the disclosed embodiments, which is capable of calling instructions stored from a storage medium and operating according to the called instructions. When said instructions are executed by a processor, the processor may perform a function corresponding to said instructions directly or by using other components under the control of said processor. The instructions may include code generated or executed by a compiler or an interpreter.

[0103] A computer-readable recording medium may be provided in the form of a non-transitory computer-readable recording medium. Here, "non-transitory" simply means that the storage medium does not contain a signal and is tangible, without distinguishing whether data is stored semi-permanently or temporarily on the storage medium. In this context, a non-transitory computer-readable medium refers to a medium that stores data semi-permanently and is readable by a device, rather than a medium that stores data for a short moment, such as registers, caches, or memory. Specific examples of non-transitory computer-readable media may include CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, and ROMs.

[0104] As described above, exemplary embodiments have been disclosed in the drawings and specification. Although specific terms have been used to describe the embodiments in this specification, they are used only for the purpose of explaining the technical concept of this disclosure and are not intended to limit the meaning or the scope of this disclosure as defined in the claims. Therefore, those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of this disclosure should be determined by the technical concept of the appended claims.

Claims

Claim 1 A document inpainting device comprising: a memory storing at least one instruction for automatic detection of a seal area and a signature area within a document and for inpainting processing; and a processor that performs an operation according to the instruction, wherein the processor collects a data set for a seal area and a signature area, constructs an inpainting model through learning of the collected data set, and the inpainting model reflects the text and layout structure of an input document to preserve information of the original document when a seal or signature is removed. Claim 2 A document inpainting device according to claim 1, wherein the processor performs labeling on text and font information below a seal area and a signature area, and collects the labeled data into a data set. Claim 3 A document inpainting device according to claim 1, wherein the inpainting model searches for coordinates for inserting or removing a seal area and a signature area, and identifies the seal area and the signature area using the searched coordinates. Claim 4 In paragraph 3, the above-described in-painting model extracts text located below a seal area or signature area through OCR, and reconstructs the extracted text according to the font and size of the original document and inserts it into an in-painted area, thereby forming a document in-painting device. Claim 5 In paragraph 4, the above-mentioned inpainting model is a document inpainting device that protects the text and layout elements of a document during the inpainting process. Claim 6 In paragraph 5, the above layout element is a document inpainting device comprising at least one of a table, a line, and a paragraph. Claim 7 In claim 1, the processor evaluates the performance of a learned inpainting model and updates the inpainting model according to the evaluation result, a document inpainting device. Claim 8 In paragraph 4, the above-mentioned inpainting model is a document inpainting device that performs a verification process to evaluate the accuracy of the restored document. Claim 9 In paragraph 5, the above-mentioned inpainting model is a document inpainting device that sets a protection area for layout protection and performs protection processing when removing a seal and signature. Claim 10 In claim 7, the processor is a document inpainting device that measures the structural similarity of the restored image and verifies whether the text restored under the seal and signature is accurately extracted by OCR for the performance evaluation of the inpainting model. Claim 11 A document inpainting method using an inpainting device for automatic detection of seal and signature areas within a document, comprising: a step of collecting a data set for seal areas and signature areas and constructing an inpainting model through learning of the collected data set; and a step of preserving information of the original document when the seal or signature is removed by reflecting the text and layout structure of the input document in the inpainting model. Claim 12 In claim 11, the step of constructing the above model is to perform labeling on text and font information under the seal area and signature area, and to collect the labeled data as a data set, a document inpainting method. Claim 13 In claim 11, the above-mentioned inpainting model searches for coordinates for inserting or removing a seal area and a signature area, and identifies the seal area and the signature area using the searched coordinates, a document inpainting method. Claim 14 In claim 13, the above-mentioned inpainting model extracts text located below a seal area or signature area through OCR, and reconstructs the extracted text according to the font and size of the original document and inserts it into an inpainted area, thereby a document inpainting method. Claim 15 In paragraph 14, the above-mentioned inpainting model is a document inpainting method that protects the text and layout elements of the document during the inpainting process. Claim 16 In claim 11, the step of building the above model is to evaluate the performance of the learned inpainting model and update the inpainting model according to the evaluation result, a document inpainting method. Claim 17 In paragraph 14, the above-mentioned inpainting model is a document inpainting method that performs a verification process to evaluate the accuracy of the restored document. Claim 18 In paragraph 15, the above-mentioned inpainting model sets a protection area for layout protection and performs protection processing when a seal and signature are removed, a document inpainting method. Claim 19 In paragraph 16, the step of constructing the above model is a document inpainting method that measures the structural similarity of the restored image and verifies whether the text restored under the seal and signature is accurately extracted by OCR for the performance evaluation of the inpainting model. Claim 20 A step of collecting a data set for a seal area and a signature area included in a document inpainting method by an automatic search and inpainting device for seal areas and signature areas within a document, and constructing an inpainting model through learning the collected data set; and a computer-readable recording medium storing a computer program in which the inpainting model is programmed to perform a step of preserving information of the original document when a seal or signature is removed by reflecting the text and layout structure of the input document.

Images