Information processing system, information processing method, and program

Abstract

To accurately extract information from a non-standard form without dependence on the format of the form.SOLUTION: A CPU 11 of a management server 1 constituting an information processing system includes: an OCR unit 33 which executes OCR processing on form data of a form read by a user terminal; an item extraction unit 34 which extracts a target item from a result of the OCR processing; and an output control unit 35 which outputs the extracted target item in a predetermined format.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] BACKGROUND ART OCR technology is known that analyzes the arrangement and structure of characters from a scanned image of a document read by a scanner or the like, and converts the analyzed characters into character codes that can be used by a computer (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2024-025219 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional OCR technology requires the setting of coordinates for the reading location for each form format, which means that individual setting work is required to handle non-standard forms, which is cumbersome and time-consuming.

[0005] The present invention has been made in view of the above circumstances, and has as its object to make it possible to extract information with high accuracy even from non-standard forms, without depending on the form format. [Means for solving the problem]

[0006] In order to achieve the above object, one aspect of the present invention is to This information processing system is characterized by having a document reading means for reading a document, an OCR processing means for performing OCR processing on the document reading results, an item extraction means for extracting target items from the results of the OCR processing, and an output control means for outputting the extracted target items in a predetermined format.

[0007] An information processing device, an information processing method, and a program corresponding to the information processing system according to one aspect of the present invention are also provided as an information processing device, an information processing method, and a program corresponding to the information processing system according to one aspect of the present invention. [Effects of the Invention]

[0008] According to the present invention, it is possible to extract information with high accuracy even from non-standard forms, without depending on the form format. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of the overall configuration of an information processing system according to an embodiment of the present invention. [Figure 2] 2 is a block diagram showing an example of a hardware configuration of a management server that constitutes the information processing system of FIG. 1. FIG. [Figure 3] FIG. 2 is a functional block diagram illustrating an example of a functional configuration of a management server. [Figure 4] 1A is a flowchart showing an example of the flow of a form reading process among the processes of the information processing system of Fig. 1. FIG. 1B is a flowchart showing an example of the flow of a form learning process among the processes of the information processing system of Fig. 1. [Figure 5] 1A is a flowchart showing an example of the flow of a generation AI process among the processes of the information processing system of FIG. 1. FIG. 1B is a flowchart showing an example of the flow of a user interface process among the processes of the information processing system of FIG. 1. [Figure 6] FIG. 10 is a diagram showing a specific example of a screen displayed on a user terminal by user interface processing. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, this embodiment will be described with reference to the drawings. <Information Processing System S> FIG. 1 is a diagram showing an example of the overall configuration of an information processing system S according to one embodiment of the present invention. The information processing system S is an information processing system configured to include a management server 1, an administrator terminal 2, and user terminals 3-1 to 3-n (n is an integer value of 1 or greater) connected via a network N. In the following, when there is no need to explain each of the user terminals 3-1 to 3-n individually, they will be collectively referred to as the "user terminal 3."

[0011] Network N is constructed using various communication networks, such as the Internet and intranets, to provide an environment in which users can remotely access services provided by information processing system S. Network N can be configured using a variety of technologies, including wired local area networks (LANs), wireless LANs, mobile communication networks such as 5G, and virtual private networks (VPNs). Communications over network N are encrypted using secure protocols such as HTTPS to protect data confidentiality and integrity. This enables the secure transfer of confidential and personal information contained in reports. Network N has sufficient bandwidth and reliability to provide stable communications even when multiple users access the system simultaneously. It is also possible to implement redundant configurations and failover functions as a countermeasure in the event of a network failure. Various information, such as the transmission of report data, the reception of processing results, and notifications of processing status, is exchanged between management server 1, administrator terminal 2, and user terminal 3 via network N. This enables efficient use of the report processing system in a distributed environment. Furthermore, network N is designed for scalability, allowing it to easily scale up as the system expands or the number of users increases.

[0012] The information processing system S is a system that enables users to receive a flexible information extraction service (hereinafter referred to as "this service") that is independent of the form format. By accessing the management server 1 from a user terminal 3 via a network N, users can upload data (hereinafter referred to as "form data") that is the result of reading forms in various formats. Target items are extracted from the uploaded form data by a generation AI. "Target items" include items specified in advance by the user and items selected by the system. The extraction results of the target items are output in a predetermined format. In this embodiment, the extraction results of the target items are output in a tabular format. This allows users to utilize the extraction results of the target items output in a tabular format in their own work.

[0013] One of the features of this service is that it eliminates the need to pre-set the reading position for forms. Conventional technology required setting the coordinates of the reading position (hereinafter referred to as "reading coordinates") for each form, but with this service, target items can be extracted simply by specifying the items in natural language. For example, a user can simply specify item names such as "client name," "product name," and "amount" in natural language, and the information corresponding to these items will be identified and extracted from the entire form.

[0014] Furthermore, this service allows the administrator of the information processing system S to easily configure and monitor the entire system. For example, it is easy to expand and adjust the system by registering new report types, adding correct answer examples, and defining extraction items. This service can provide customized report processing services that suit a variety of industries and business processes.

[0015] Furthermore, by using this service, users can, for example, refer to past information or compare multiple reports using stored report data and processing results. The service also ensures data consistency by absorbing variations in notation and normalizing report data, which, for example, makes it easier to link with subsequent systems.

[0016] As will be explained in more detail later, Information Processing System S is able to provide this service that solves the problems of conventional technologies, such as batch processing of multiple forms with different formats, highly accurate recognition of handwritten characters, and form classification with a small amount of training data, thereby achieving a significant improvement in the efficiency of users' work.

[0017] [Management Server 1] The management server 1 constituting the information processing system S is an information processing device that serves as a server for managing the entire information processing system S. The management server 1 may be constructed in a cloud environment. The management server 1 can be implemented as a single physical server, or it may be distributed across multiple virtual machines in a cloud environment to ensure scalability and availability. The management server 1 is managed, for example, by an administrator of the information processing system S. The management server 1 is capable of executing a predetermined application program that makes the information processing system S available. The management server 1 transmits various types of information to the administrator terminal 2, the user terminal 3, and external devices, and enables various processes to be executed. The management server 1 also acquires various types of information transmitted from the administrator terminal 2, the user terminal 3, and external devices, and enables various processes to be executed.

[0018] The external device includes, for example, a server that provides the generation AI. The generation AI is an artificial intelligence system that supports various processes. The generation AI operates in cooperation with the management server 1 and has the function of supporting the processing of a high-performance database that supports 1 billion records and 500 columns. The generation AI is an AI system that combines advanced natural language processing capabilities based on a large-scale language model (LLM) with image recognition and image generation capabilities, and plays a central role in various processes in this embodiment.

[0019] For example, the management server 1 executes various processes such as form reading processing, form learning processing, AI generation processing, and user interface processing, thereby enabling users to extract desired information from unstructured forms. The management server 1 also stores various data such as form reading results, OCR results, and target item extraction results. The management server 1 is designed with scalability in mind, allowing for the addition of new form formats and functions tailored to needs. The configuration and processing of the management server 1 will be described in detail below.

[0020] [Administrator terminal 2] The administrator terminal 2 is an information processing device operated by an administrator of the information processing system S. The administrator terminal 2 is configured, for example, as a personal computer, a tablet terminal, a smartphone, or the like. The administrator terminal 2 is capable of executing a predetermined application program that enables use of the information processing system S. Unlike the user terminal 3 described below, the administrator terminal 2 has the authority to access advanced management functions for the entire system. The administrator terminal 2 is capable of executing various processes based on various information transmitted from the management server 1, the user terminal 3, and external sources, as well as various information input by the administrator. The administrator terminal 2 is also capable of transmitting various information to the management server 1, the user terminal 3, and external sources.

[0021] For example, the administrator terminal 2 can access the management server 1 to perform various management operations. Specifically, the administrator terminal 2 can access the management server 1 to perform user account management, report format management, system parameter setting, processing log monitoring, performance analysis, and the like. Of these, user account management includes adding and deleting users who use the service, setting permissions, and the like. Report format management includes registering new report formats, adding learning data, and setting correct answer examples. System parameter setting includes setting the accuracy of OCR processing, adjusting extraction processing parameters, and allocating system resources. Processing log monitoring includes checking the operation log and error log of the entire information processing system S, discovering problems, and responding to them. Performance analysis includes improving the information processing system S by analyzing indicators such as processing speed and recognition accuracy.

[0022] By performing the various management tasks described above, the administrator terminal 2 supports stable operation and continuous improvement of the entire information processing system S. In addition, by performing backup management and update management for the information processing system S, the administrator terminal 2 ensures the safety of the various data handled by the information processing system S and provides the latest information. The user interface displayed on the administrator terminal 2 can aggregate various information required for advanced settings. This enables efficient system management.

[0023] [User terminal 3] The user terminal 3 is an information processing device operated by a user who receives the service. The user terminal 3 may be, for example, a personal computer, a tablet terminal, or a smartphone. The user terminal 3 is capable of executing a predetermined application program that enables use of the information processing system S. The user terminal 3 provides an intuitive user interface by running various web browsers and application programs. As a result, easy operation of a complex system is realized. A predetermined identifier (e.g., a user ID) is assigned to each of the user terminals 3-1 to 3-n. The user terminal 3 is capable of executing various processes based on various information transmitted from the management server 1, the administrator terminal 2, and external devices, as well as various information input by the user. The user terminal 3 is also capable of transmitting various information to the management server 1, the administrator terminal 2, and external devices.

[0024] For example, by accessing the management server 1, the user terminal 3 can perform processes such as uploading form data, setting processing parameters, checking the processing status, displaying and modifying the processing results, and downloading the processing results. Uploading form data is performed by dragging and dropping form data scanned by a scanner or by pressing a browse button to select the desired data. Setting processing parameters is performed by specifying the items the user wants to extract or selecting an output format. Checking the processing status is performed by performing operations that allow the user to check the progress of the processing in real time. Displaying and modifying the processing results is performed by displaying the form data in an editable format on the user interface. Downloading the processing results is performed by downloading the extraction results of the target items in a specified file format for use in a subsequent business system. By utilizing the processing power of the management server 1, the user terminal 3 can perform advanced form processing without relying on its own performance.

[0025] The above-described processing by each of the management server 1, the administrator terminal 2, and the user terminal 3 that constitute the information processing system S is merely an example. In other words, it is sufficient that the information processing system S as a whole has the function of realizing the above-described processing, and therefore some or all of the functions for realizing the above-described processing may be shared or cooperated within the information processing system S.

[0026] For example, some or all of the functions of the management server 1 may be functions of other devices in the information processing system S. Also, some or all of the functions of other devices in the information processing system S may be functions of the management server 1. Furthermore, some or all of the functions of the management server 1 may be transferred to other servers (not shown). This promotes processing in the information processing system S as a whole and also makes it possible for processes to complement each other.

[0027] <Hardware configuration> [Hardware configuration of Management Server 1] FIG. 2 is a block diagram showing an example of the hardware configuration of the management server 1 that constitutes the information processing system S of FIG. The management server 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input / output interface 15, an output unit 16, an input unit 17, a memory unit 18, a communication unit 19, and a drive 20.

[0028] The CPU 11 executes various processes in accordance with programs recorded in the ROM 12 or programs loaded from the storage unit 18 into the RAM 13. The RAM 13 also stores data and the like required for the CPU 11 to execute various processes. The CPU 11, the ROM 12, and the RAM 13 are interconnected via a bus 14. An input / output interface 15 is also connected to this bus 14.

[0029] The input / output interface 15 is connected to an output unit 16, an input unit 17, a storage unit 18, a communication unit 19, and a drive 20. The output unit 16 is composed of a display, a speaker, etc., and outputs various types of information as images, sounds, etc. The input unit 17 is composed of a keyboard, a mouse, a touch panel, etc., and accepts input of various types of information. The storage unit 18 is composed of a hard disk, a DRAM (Dynamic Random Access Memory), etc., and stores various types of data. The communication unit 19 communicates with other devices via the above-mentioned network N, which is composed of the Internet, etc.

[0030] Removable media 21, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, is appropriately attached to the drive 20. Programs read from the removable media 21 by the drive 20 are installed in the storage unit 18 as needed. The removable media 21 can also store various data stored in the storage unit 18 in the same way as the storage unit 18.

[0031] [Hardware configuration of administrator terminal 2 and user terminal 3] Each of the administrator terminal 2 and the user terminal 3 has the same hardware configuration as the management server 1 shown in Fig. 2. That is, each of the administrator terminal 2 and the user terminal 3 has a CPU, ROM, RAM, bus, input / output interface, output unit, input unit, memory unit, communication unit, drive, and removable media (not shown), which correspond to the CPU 11, ROM 12, RAM 13, bus 14, input / output interface 15, output unit 16, input unit 17, memory unit 18, communication unit 19, drive 20, and removable media 21 shown in Fig. 2, respectively.

[0032] In addition to the above hardware configuration, the user terminal 3 also has a reading unit as a form reading means that enables reading of forms. The reading unit is configured, for example, with a CCD type scanner that reduces the reflected light of light irradiated onto a document from a light source using a lens and receives the light with a CCD (Charge Coupled Device), or a CIS type scanner that receives the reflected light of light sequentially irradiated onto a document from an LED light source using a CIS (Contact Image Sensor).

[0033] <Functional configuration of management server 1> FIG. 3 is a functional block diagram showing an example of the functional configuration of the management server 1. As shown in FIG. In the CPU 11 of the management server 1, an information acquisition unit 31 as a captured image acquisition means, an information management unit 32 as a form storage means, an OCR result storage means, an extraction result storage means, and a learning data storage means function during operation. In addition, in the CPU 11, an OCR unit 33 as an OCR processing means, an item extraction unit 34 as an item extraction means, an output control unit 35 as an output control means, and a grouping unit 36 ​​as a grouping means function during operation. In addition, in the CPU 11, a numbering unit 37 as a numbering means, a matching unit 38 as a matching means, a conversion unit 39 as a conversion means, a normalization unit 40 as a normalization means, and a comparison unit 41 as a comparison means function during operation. In addition, in the CPU 11, a configuration extraction unit 42 as a configuration extraction means, a prompt generation unit 43 as a prompt generation means, and a transmission control unit 44 function during operation.

[0034] Furthermore, various databases are provided in the storage unit 18 of the management server 1. For example, databases such as a form DB 51 storing form data, a processing result DB 52 storing processing results, an extraction result DB 53 storing extraction results, and a learning DB 54 storing learning data are provided.

[0035] The form data stored in the form DB 51 includes, for example, image data of paper forms scanned by a scanner or the like, and original forms as electronic data such as electronically provided PDFs. In the form DB 51, a predetermined identifier (for example, a form ID) is assigned to each piece of form data, facilitating association with subsequent processing results. This makes it possible to track the processing results for a specific form. The form data is used as the original when reprocessing, verification, improvement of the processing algorithm, etc., becomes necessary.

[0036] The processing results stored in the processing result DB 52 include, for example, a combination of character information extracted by OCR processing and its coordinate information (hereinafter, sometimes referred to as "OCR results"). The extraction results stored in the extraction result DB 53 include, for example, structured data of pairs of items and values ​​extracted by the generation AI.

[0037] The learning data stored in the learning DB 54 also includes, for example, two pieces of learning data (image data) associated with each form format. Conventional machine learning-based systems require a large amount of learning data to achieve highly accurate classification and extraction. In contrast, the present embodiment enables the use of a generative AI, making it possible to perform efficient processing even when there is a small amount of learning data. For example, two pieces of learning data (image data) associated with each form format are selected as representative of the characteristics of each form format. These pieces of learning data (image data) are used, for example, in the extraction process of the configuration extraction unit 42 and the generation process of the prompt generation unit 43, which will be described later.

[0038] The learning data is not limited to image data, but may also include definitions of items to be extracted from forms and samples of correct extraction results. Such information is provided to the generation AI as information necessary for the process of showing correct examples. The learning data is also designed so that it can be easily updated by the administrator operating the administrator terminal 2 when a new form format is added or an existing format is changed. The learning data is also subject to evaluation to determine whether it adequately expresses the characteristics of the format.

[0039] The information acquisition unit 31 acquires various types of information. For example, the information acquisition unit 31 acquires various types of information transmitted from the administrator terminal 2, the user terminal 3, and external devices to the management server 1. For example, the information acquisition unit 31 acquires form data transmitted from the user terminal 3.

[0040] The information management unit 32 stores and manages various types of information in the database of the storage unit 18. For example, the information management unit 32 stores and manages the form data acquired by the information acquisition unit 31 in a form DB 51. The information management unit 32 also stores and manages various processing results in a processing result DB 52. The information management unit 32 also stores and manages various extraction results in an extraction result DB 53. The information management unit 32 also stores and manages learning data (image data) in a learning DB 54. In this case, the information management unit 32 stores and manages two pieces of learning data (image data) for each form format.

[0041] The OCR unit 33 applies OCR processing to the entire form, extracting a combination of character information (text data) and coordinate information, which is information about the position of the character information on the form. The combination of character information extracted by OCR processing and coordinate information, which is information about the position of the character information on the form, is stored as the processing result in the processing result DB 52. The processing result is used as input information for the grouping process (described later) and the generation AI process, as well as for evaluating and improving the accuracy of the OCR processing. For example, character patterns that are frequently misrecognized can be analyzed to adjust or strengthen the OCR engine. The data structure of the processing result stored in the processing result DB 52 includes character information, coordinate information (e.g., X coordinate, Y coordinate, width, height, etc.), recognition accuracy, font information, etc.

[0042] The item extraction unit 34 extracts target items from the processing results of the OCR unit 33 based on the acquired items specified in natural language, without specifying the reading coordinates. The extraction results of the item extraction unit 34 include pairs of item names, such as "client name," "product name," and "amount," and their corresponding extracted values. The extracted pairs of item names and extracted values ​​are stored in the extraction result DB 53 as structured data of the item names and extracted values. Furthermore, the extraction results stored in the extraction result DB 53 may include, in addition to the pairs of item names and extracted values, information indicating, for example, the extraction confidence level and the relationship with the original character string position information.

[0043] The extraction results of the item extraction unit 34 are not only used as the basis for output in table format, but also for comparing and checking multiple reports. Furthermore, analyzing the history of the extraction results of the item extraction unit 34 can also be used to evaluate and improve the performance of the generation AI. The extraction results of the item extraction unit 34 include the extraction results after the code conversion process and notation normalization process described below. This makes it possible to provide data in a format required for linking with subsequent systems. Furthermore, past extraction results by the item extraction unit 34 are also utilized. For example, past extraction results are used as learning data when adding new correct examples. This allows for continuous improvement of the system.

[0044] The output control unit 35 controls the output of various information to the administrator terminal 2 and the user terminal 3. For example, the output control unit 35 controls the output of target items extracted by the item extraction unit 34 in a table format to the user terminal 3. The output control unit 35 also controls the output of a GUI (Graphical User Interface) to the user terminal 3, which enables viewing and editing of the OCR results of the OCR unit 33.

[0045] The grouping unit 36 ​​groups character information having adjacent coordinate information based on the character information and coordinate information extracted by the item extraction unit 34. The grouping unit 36 ​​also determines the form type based on the grouped character information, and groups the data by form type.

[0046] The numbering unit 37 numbers the text information so that it is easy for the generation AI to understand. Through the processing of the numbering unit 37, consecutive numbers are assigned to each text element and the format is converted into one that is easy for the generation AI to refer to. The numbering is performed based on the layout on the form; for example, numbers are generally assigned in order from the top left to the bottom right. However, if a special structure such as a column layout or table layout is detected, the numbering is performed based on the logical reading order.

[0047] The matching unit 38 performs matching processing that absorbs spelling variations by taking into account the context and shape characteristics of the characters. For example, when characters that are likely to be misrecognized in OCR processing are included, the matching unit 38 performs matching based on the context and shape characteristics. Furthermore, for example, the matching unit 38 performs matching processing that absorbs spelling variations such as a mixture of full-width characters and half-width characters, a mixture of hiragana and katakana, and different language spellings. The matching unit 38 achieves appropriate matching according to the context by using, for example, techniques such as edit distance (e.g., Levenshtein distance), phonetic similarity of characters, and N-gram matching, as well as machine learning models.

[0048] The conversion unit 39 converts the target items extracted by the item extraction unit 34 into pre-specified identification information. For example, the conversion unit 39 converts the read result of a company name or the like into a pre-specified code (for example, an internal company code or a standard code). This allows for smooth integration with subsequent business systems.

[0049] The normalization unit 40 normalizes spelling variations in the OCR results of the OCR unit 33. For example, the normalization unit 40 performs processing to absorb and normalize spelling variations in product names and company names written on forms. Through the processing of the normalization unit 40, different spellings of the same entity are unified into a standard format, ensuring data consistency.

[0050] The comparison unit 41 compares the target items of the plurality of forms extracted by the item extraction unit 34. The comparison unit 41 also highlights the selected items from among the target items to be compared.

[0051] The configuration extraction unit 42 extracts configurations common to the two training data images stored and managed in the training DB 54 by the information management unit 32. For example, the configuration extraction unit 42 extracts text information and coordinate information from the results of OCR processing on the two training data (image data). The configuration extraction unit 42 then compares and analyzes this information to distinguish between elements that appear in common in both images (e.g., fixed text, logos, layout structures, etc.) and different elements (e.g., dates, amounts, product names, etc.). Common elements provide important clues for identifying the format of the form, with particular emphasis placed on company names, form titles, fixed header items, etc.

[0052] The prompt generation unit 43 generates prompts for classifying forms. The processing of the prompt generation unit 43 creates instructions for classifying forms in a format suitable for the generation AI, eliminating the need for complex manual prompt design.

[0053] The transmission control unit 44 controls the communication unit 19 (see FIG. 2) to transmit various types of information to the administrator terminal 2, the user terminal 3, and the outside. For example, the transmission control unit 44 controls the transmission of the extraction results of the target items to the user terminal 3.

[0054] <Processing flow of information processing system S> [Document reading process] FIG. 4A is a flowchart showing an example of the flow of a form reading process among the processes of the information processing system S of FIG. The form reading process is a series of steps that involves importing the target form, performing OCR processing, and storing the results as the reading results. The form reading process is positioned as preprocessing for generating information to be input into the generation AI, and plays an important role in improving the accuracy of item extraction. Furthermore, the coordinate information obtained during the form reading process is used to highlight relevant areas in the GUI. Unlike conventional technology, the form reading process is characterized by the fact that it does not require the reading position to be specified in advance, and OCR processing is performed on the entire form. This allows for flexible handling of non-standard forms with unknown formats.

[0055] 4A, the form reading process includes step S1 of reading the form, step S2 of storing the read form, and step S3 of performing OCR processing. The form reading process further includes step S4 of grouping, step S5 of numbering the character information, and step S6 of storing the OCR results.

[0056] Specifically, step S1 involves digitizing paper forms using a scanner or importing existing electronic forms (e.g., PDF files, image files, etc.). Step S1 is a step for preparing basic data that will serve as input information for subsequent processing. In step S1, appropriate settings such as scanning resolution and image format are made to generate image data of optimal quality for subsequent OCR processing. When scanning paper forms, preprocessing such as tilt correction and brightness adjustment is also performed to improve recognition accuracy. In addition, for forms consisting of multiple pages, it is possible to process them page by page or to treat related pages as a single form. For electronic forms, an appropriate scanning method is selected depending on the file format. For example, for PDF files, the system checks whether a text layer is present and, if present, directly uses the text information to improve accuracy.

[0057] Step S1 can be performed by a user manually uploading a form, or by the system monitoring a specified folder or mailbox and automatically importing stored forms. This can automate business processes. Furthermore, an image of the form captured with a digital camera or smartphone can also be uploaded.

[0058] In step S2, the form data read in step S1 is stored. This process stores the original form, allowing for reprocessing and verification as needed, ensuring system reliability. In step S2, a unique identifier (e.g., form ID) is assigned to each form and associated with subsequent processing results. This makes it possible to track all processing results related to a specific form. The form data is saved in an appropriate data format (e.g., TIFF, JPEG, PNG, PDF, etc.) according to its characteristics, and is compressed as needed. The compression method is selected taking into account the characteristics of the data and its impact on subsequent processing.

[0059] In step S2, information such as the capture date and time, file name, file size, image resolution, color mode, and number of pages is stored in association with the form as metadata. This metadata is used for form management and search. The information processing system S takes data redundancy and availability into consideration. For this reason, important data can be stored in multiple storage areas. Furthermore, the information processing system S can automatically archive or delete forms that have exceeded a predetermined retention period based on a data retention period policy.

[0060] In step S3, OCR processing is performed on the entire target form scanned in step S1, and character information and coordinate information are extracted. Conventional OCR technology requires the position of the items to be scanned to be specified in advance, but in this invention, OCR processing is performed on the entire form. This allows for flexible handling of unconventional forms with unknown formats.

[0061] In step S3, the form image is first preprocessed (for example, by correcting tilt, removing noise, adjusting brightness, etc.) to improve the accuracy of character recognition. Next, a character area detection process is performed to identify areas on the form where characters exist. A character recognition process is applied to each detected character area, and text data is generated. At this time, the position information of each character or character string on the form (for example, X coordinate, Y coordinate, width, height, etc.) is also recorded at the same time. The latest machine learning-based OCR technology is used for character recognition, which is capable of recognizing not only printed characters but also handwritten characters.

[0062] The OCR process in step S3 supports multiple languages, including Japanese, English, and Chinese, making it possible to process forms containing a mixture of different languages. Furthermore, the confidence level of character recognition may be calculated and used as a basis for subsequent processing. As output information from step S3, pairs of character information (i.e., recognized text) and its coordinate information are generated, and these become input information for the subsequent grouping process.

[0063] In step S4, character information with adjacent coordinate information is grouped based on the character information and coordinate information extracted in step S3. This performs preprocessing to facilitate identification of meaningful information units (e.g., pairs of item names and values). In step S4, the spatial arrangement of the character information is first analyzed, and adjacent characters are combined to form words or phrases. Next, the distance and positional relationship between these words and phrases are analyzed, and words and phrases that are thought to have semantic unity are grouped together.

[0064] For example, suppose the value "A12345" is displayed near the location where the field name "Invoice Number" is displayed. In this case, the field name "Invoice Number" and the value "A12345" are grouped as related information. The criteria for grouping are taken into account, such as the horizontal and vertical distance across the entire form, whether they are located in the same row or column, and similarities in font size and format. In addition, if a table is detected, grouping may be performed on a per-cell basis.

[0065] In step S4, information about the grouping results (hereinafter referred to as "grouping information"), such as the group ID, character information contained in the group, and coordinate information (bounding box) of the group, is generated. The generated grouping information serves as an important clue for understanding the context in the subsequent target item extraction process. Furthermore, in step S4, the hierarchical relationships between groups (for example, headings and content, table headers and values, etc.) are also analyzed and recorded as information for understanding the structure of the form. In other words, in step S4, the logical structure of the form is grasped, rather than simply combining adjacent characters.

[0066] In step S5, the character information is numbered to make it easier for the generation AI to understand. This process assigns consecutive numbers to each text element, converting it into a format that is easy for the generation AI to reference. In step S5, a unique identification number is assigned to each text element based on the character information extracted in step S3 and the grouping information in step S4.

[0067] For each text element, a pair of an identification number and text content is formed, such as "#1: Invoice," "#2: Issue Date," and "#3: 20XX / X / X." Hereinafter, the pair of identification number and text content formed will be referred to as "numbering information." The numbering information is used to clearly refer to specific text elements in the subsequent item extraction process. For example, the relationship between item name and value can be explicitly expressed in the format "The value of item #2 is #3."

[0068] The numbering information is also used for display and operation in the GUI. When a user selects or modifies a specific item, the corresponding identification number can be used to identify the target. In other words, in step S5, the OCR results are converted into a format that is easy for the generating AI to process, improving processing efficiency. Furthermore, the generated numbering information is also useful for debugging and verification, enabling system operation to be tracked.

[0069] In step S6, the results of steps S3 to S5 are stored. This process saves intermediate data for processing by the generation AI and intermediate data for verification work. In step S6, the OCR results are saved as structured data, associated with a unique identifier (form ID) for each form. The saved data includes character information (recognized text), coordinate information (X coordinate, Y coordinate, width, height), recognition confidence, group ID, numbering information, etc. This data is efficiently stored using appropriate database technology, such as a relational database or NoSQL database. The data storage format is optimized for fast access in subsequent processing, and indexes are created as needed.

[0070] In step S6, data consistency is also ensured. For example, a check is made to see if the grouping information and numbering information are consistently associated. Quality evaluation information for the OCR process (e.g., recognition rate, proportion of questionable characters, etc.) is also saved and used to monitor the performance of the entire system. The saved OCR results are not only used as information for extracting items in step S3, but are also output to a GUI for viewing and editing the read results. Any corrections made by the user via the GUI are managed in history along with the original OCR results. This allows for continuous improvement of the system.

[0071] [Report learning processing] FIG. 4B is a flowchart showing an example of the flow of the form learning process among the processes of the information processing system S of FIG. The form learning process is a series of steps from classifying forms based on learning data to extracting items. The form learning process is executed when an administrator registers a new form format. The form learning process plays an important role as a preparatory step for efficiently processing forms with limited learning data. Furthermore, it enables flexible response to the addition of new form formats and changes to existing formats, enhancing the scalability and adaptability of the entire system. Unlike conventional technologies, the form learning process achieves efficient processing with only two pieces of learning data image data for each form format. While conventional machine learning-based systems require large amounts of learning data to achieve highly accurate classification and extraction, the form learning process of this embodiment utilizes generative AI to enable efficient processing even with limited learning data.

[0072] As shown in FIG. 4(B), the form learning process includes step S11 of storing learning data, step S12 of extracting common configurations, and step S13 of generating a prompt.

[0073] Specifically, in step S11, two pieces of training data (image data) are stored for each form format. This prepares basic data to enable efficient classification and extraction with minimal training data. In step S11, an administrator or user selects two sample images that represent a specific form format and registers them in the system. It is important that the two sample images selected show the essential characteristics of that format; ideally, they should include both consistent elements such as layout, logo, and header structure, as well as variable elements such as written content and date.

[0074] The training data (image data) registered in the system is saved in a high-quality digital format (e.g., TIFF, PNG, etc.) and preprocessed as needed (e.g., deskew, noise removal, etc.). Each training data (image data) is associated with metadata such as the form type (e.g., invoice, purchase order, delivery note, etc.), issuer information, and format version. The fields to be extracted from the form type (e.g., supplier name, date, product name, amount, etc.) and their extraction rules (e.g., regular expression patterns, context, etc.) are also defined. This information is used in the subsequent processes of steps S12 and S13. Step S11 can also accommodate the addition of new form formats and the updating of existing formats. This allows for flexible adaptation to changing business requirements. Step S11 allows efficient training using a small number of samples, which can significantly reduce the initial setup costs, for example, when introducing the system.

[0075] In step S12, important common features are extracted from the two stored training data images. This process identifies essential features of the form format and improves the accuracy of form classification. In step S12, OCR processing is first applied to the two training data (image data) to extract text information and coordinate information. This information is then compared and analyzed to distinguish between elements that appear in common in both images and elements that differ. In addition to text information, layout information (e.g., table arrangement, column structure, logo position, etc.) is also analyzed to extract visual features. The relative positional relationships between multiple elements are also recorded as important features.

[0076] In step S12, structured data that represents the characteristics of the form format is generated. The generated structured data includes a list of fixed text elements, important visual features, layout structure, etc. The information extracted in step S12 is used to automatically generate prompts in the subsequent step S13. In this way, in step S12, common features among multiple samples (training data) are identified, so the essential structure of the form is understood and classification accuracy is improved. In addition, the processing in step S12 enables efficient form classification even with a small amount of training data, thereby reducing system implementation and operation costs.

[0077] In step S13, a prompt for classifying the form is automatically generated. This process creates instructions for classifying the form in a format suitable for the generation AI, eliminating the need for complex manual prompt design. In step S13, a prompt template is automatically generated based on the information extracted in step S12, allowing the generation AI to effectively classify forms.

[0078] Prompts are generated by incorporating specific information such as extracted fixed text elements, visual features, and layout structure into a basic structure such as, for example, "Given a given form image, please determine its form type based on the following characteristics." The prompts include, for example, "The title contains the word 'invoice,'" "The logo of Company A is in the upper left corner," and "The product details are listed in a table format." The prompts also explicitly describe how to identify each form type and how to distinguish between similar formats. This helps to reduce misclassification.

[0079] The generated prompts are optimized to suit the characteristics of the generation AI and adjusted to a format that enables efficient processing. When multiple generation AIs are used, multiple prompt versions may be generated to suit the characteristics of each. In step S13, prompt templates are generated for each form type and are used for form classification by the generation AI. The prompts generated in step S13 are automatically updated when a new form format is added or when a form format is changed. This ensures the adaptability of the system. In step S13, instructions for form classification are created in a format suitable for the generation AI based on the information extracted in step S12, eliminating the need for complex manual prompt design.

[0080] [Generative AI processing] FIG. 5A is a flowchart showing an example of the flow of the generation AI process among the processes of the information processing system S of FIG. Generative AI processing is a series of processes that uses generative AI to extract target items from OCR-processed information. Generative AI processing utilizes the generative AI's advanced language understanding capabilities to achieve flexible and highly accurate information extraction that exceeds the limits of conventional technology. Furthermore, by linking multiple generative AIs, it is possible to utilize the strengths of each, making it possible to handle a variety of report types and extraction items. A distinctive feature of generative AI processing is that, unlike conventional technology, it does not require specifying reading coordinates, and can extract the necessary information simply by specifying items in natural language, such as "client name," "product name," and "amount."

[0081] 5(A), the generation AI process includes step S21 of processing the forms in bulk, step S22 of showing the generation AI examples of correct answers, and step S23 of extracting items specified in natural language from the OCR results. The generation AI process also includes step S24 of storing the item extraction results, step S25 of matching by absorbing spelling variations, step S26 of code conversion, step S27 of spelling normalization, and step S28 of outputting the extraction results in table format.

[0082] Specifically, in step S21, multiple forms with different formats are processed in a batch. Unlike conventional methods that require individual settings for each format, this process enables efficient processing of different types of forms using a unified processing flow. In step S21, the multiple input forms are grouped by form type using the prompt generated in step S13 of FIG. 4(B) above. Each form is grouped into a category, such as "invoice," "purchase order," or "delivery note," and processing parameters are set according to the characteristics of each form. The grouped forms are either batch-processed by type or processed sequentially according to a unified processing flow.

[0083] In step S21, common extraction items (for example, supplier name, date, amount, etc.) are extracted with a single instruction, even for forms in different formats. This eliminates the need for the user to configure settings individually for each type of form, even when dealing with multiple types of forms. Also, in step S21, multiple related forms (for example, a set of purchase order, invoice, and delivery note) are processed together, making it possible to check their consistency.

[0084] Step S21 utilizes parallel processing technology to improve processing efficiency. Specifically, by using multiple CPU cores and GPUs, multiple forms are processed in parallel at the same time. This allows a large number of forms to be processed in a short time. Furthermore, in a cloud environment, resources are automatically scaled according to the processing load. In other words, in step S21, unlike the conventional method in which individual settings were made for each format, different form types are processed efficiently using a unified processing flow. As a result, business efficiency can be significantly improved.

[0085] In step 22, the generation AI is shown correct examples, instructing it on how to extract items from unstructured forms. This process makes it possible to extract information with high accuracy from unstructured forms in various formats. In step 22, a pair of a sample form and its correct extraction result, which is a pre-prepared correct example, is presented to the generation AI for each form type. The correct example is shown in the form, for example, "When the 'client name', 'date', and 'amount' are extracted from this form, they become 'Corporation A', 'April 1, 2023', and '100,000 yen', respectively." By referring to such examples, the generation AI can extract appropriate information from unknown forms with similar patterns.

[0086] In step 22, information is provided in a format tailored to the characteristics of multiple generation AIs. For example, in step 22, correct examples are presented in natural language format to AIs with strong language comprehension capabilities, while correct examples are presented in a structured format to AIs with strong pattern recognition capabilities. This allows information provision that takes advantage of the strengths of each AI. This maximizes the capabilities of multiple AIs. Correct examples are selected to capture the essential characteristics of the format so that high generalization performance can be achieved even with a small number of samples (e.g., one to three examples). In particular, the relationship between item names and values, as well as the relative positions of multiple items, are clearly indicated. Correct examples are regularly updated and improved based on the evaluation results of extraction accuracy, thereby ensuring continuous performance improvement of the system. In this way, in step 22, correct examples are presented to the generation AI, effectively instructing it on how to extract items from unstructured forms, and information is provided in a format tailored to the characteristics of multiple generation AIs. As a result, processing is achieved that maximizes the capabilities of each AI.

[0087] In step S23, the generation AI extracts the necessary items based on the acquired items specified in natural language without specifying the reading coordinates from the OCR results. This process realizes flexible information extraction that exceeds the limits of conventional technology. In step S23, the extraction items specified by the user (e.g., "client name," "product name," "amount," etc.), the OCR results (e.g., character information, coordinate information, grouping information, numbering information, etc.), and the correct answer example presented in step S22 are first input into the generation AI.

[0088] Based on the input information, the generation AI identifies appropriate values ​​corresponding to the specified items from the form. The generation AI extracts appropriate information by utilizing typical relationship patterns between item names and values ​​(e.g., "item name: value" or "item name value"), the relative positions between items, and general business knowledge (e.g., invoice dates are usually close to the present, amounts are expressed in numbers, etc.).

[0089] Furthermore, if a table is detected, the generation AI understands its structure, grasps the relationship between the header row and values, and extracts information. In step S23, the reading position is not specified in advance; appropriate information is extracted simply by specifying items in natural language. This allows for flexible handling of unconventional forms with unknown formats. Furthermore, by using multiple generation AIs in combination, mutually complementary processing is possible, leveraging each AI's strengths. For example, combining the extraction results of an AI with excellent language comprehension with those of an AI with excellent visual pattern recognition can achieve more accurate extraction. A confidence score is assigned to the extraction results, and a flag is set to prompt human confirmation of unreliable results. In this way, in step S23, the generation AI's advanced language comprehension and contextual understanding capabilities allow it to identify appropriate information simply by specifying natural language.

[0090] In step S24, the extraction results from step S23 are stored in the extraction result DB 53. This process stores the extraction results, which can then be used, for example, in subsequent processing or verification. In step S24, the extraction results from the generation AI are stored as structured data, associated with a unique identifier for each form. The stored data includes the extracted item name (e.g., "client name," "product name," "amount," etc.), extracted value, extraction confidence level, and location information of the original string. This data is efficiently stored using appropriate database technology, such as a relational database or NoSQL database. The data storage format is optimized for high-speed access in subsequent processing, and indexes are created as needed.

[0091] In step S24, data consistency is ensured. For example, if a discrepancy within the same report, such as when different values ​​are extracted for the same item, or if an impossible value is detected in business operations, such as when a past date is set as the invoice issue date, a warning flag is raised. The stored extraction results are used not only as input information for the processing described below, but also for displaying and modifying the processing results. User modifications are managed in history along with the original processing results, allowing for continuous improvement of the system. In other words, since the extraction results are recorded in step S24, they can be used in subsequent processing or for verification. As a result, the efficiency and reliability of the entire system are improved.

[0092] In step S25, a process is performed to absorb spelling variations. This process determines that similar characters or different spelling methods refer to the same entity, eliminating discrepancies due to spelling differences. In step S25, the generation AI performs a process to absorb spelling variations for the item values ​​extracted in step S23. For example, the same company name can be written in various ways, such as "ABC Co., Ltd.", "ABC Co., Ltd.", "ABC Co., Ltd.", and "ABC Corp.", but these are recognized as the same entity.

[0093] In step S25, appropriate matching is performed based on context and shape characteristics, even if the text contains characters that are likely to be misrecognized in OCR processing (for example, "O" and "0," "l" and "1," "S" and "5," etc.). In addition, step S25 also absorbs variations in spelling, such as the mixture of full-width and half-width characters, the mixture of hiragana and katakana, and different language spellings (for example, "Tokyo" and "Tokyo").

[0094] As a result of the processing in step S25, conversion information into a standard notation format, a matching confidence level, and other information are generated. This information is utilized in subsequent processing. In this way, in step S25, processing is performed to absorb orthographic variations by taking into account the shape characteristics of the characters. As a result, even similar characters or characters written in different ways are determined to refer to the same entity. This eliminates discrepancies due to differences in notation. As a result, consistency is ensured in subsequent processing and data integration, improving the reliability of the entire system.

[0095] In step S26, the results of the reading, such as the company name, are converted into a pre-specified code (for example, an internal company code or a standard code). In step S26, conversion processing based on a pre-defined code system is performed on items that require code conversion (for example, company name, product name, department name, etc.) from among the item values ​​extracted in step S23. For example, if the company name "ABC Co., Ltd." is extracted, "ABC Co., Ltd." is converted into the customer code "T12345."

[0096] The correspondence table used for code conversion (hereinafter referred to as the "mapping table") may be managed as master data within the system, or may be obtained in conjunction with an external system (ERP, CRM, accounting system, etc.). In step S26, the results of step S25 are utilized to convert to the appropriate code even when there is variation in spelling. For example, even if the name is written as "ABC Co., Ltd.", it is determined to be the same company as "ABC Co., Ltd." and converted to the same code "T12345."

[0097] Furthermore, in step S26, even if there is no perfectly matching master data, candidates are presented based on the degree of similarity, and registration as a new entry is encouraged. The results of the code conversion are recorded as a correspondence between the original character string and the code, and are included in the tabular output of the processing described below. The degree of certainty of the conversion is also recorded, and a confirmation flag is set for conversion results with low reliability. In this way, in step S26, the read results such as a company name are converted into a specified code, so that it can be converted into, for example, an internal code or a standard code. This facilitates integration with subsequent business systems, thereby improving the efficiency of the entire business process.

[0098] In step S27, the spelling variations of product names and company names written on the form are absorbed and normalized. In step S27, the item values ​​extracted in step S23 are converted to a standard spelling format. This process is performed based on the results of step S25, which absorbs spelling variations.

[0099] The expressions that are normalized in step S27 are mainly named entities such as company names, product names, addresses, and job titles, and appropriate normalization rules are applied to each of them. For example, in the case of company names, the position of "Kaisho Kaisha" is standardized (pre- or post-placed), and the abbreviation "(Kaisho Kaisha)" is expanded to the official name "Kaisho Kaisha." In the case of product names, the notation of model numbers (such as "Type A," "Type-A," and "Type A") and brand names are standardized. In the case of addresses, variations in the notation of prefecture names, city / ward / town / village names (such as "Tokyo" and "Tokyo," or "Chiyoda-ku" and "Chiyoda-ku-nai") are standardized.

[0100] In step S27, normalization rules based on industry standards and internal company regulations are applied, ensuring consistent notation throughout the organization. Normalization rules are managed as master data and updated as necessary. Furthermore, if a new notation pattern is detected, the administrator is notified and prompted to expand the normalization rules. The results of the notation normalization process are recorded as a correspondence between the original string and the normalized string. In this way, in step S27, variations in the notation of product names and company names written on forms are absorbed and normalized. This unifies different notations of the same entity into a standard format, ensuring data consistency and improving the accuracy of analysis and processing. As a result, data quality improves and errors and rework in subsequent business processes are reduced.

[0101] In step S28, the target items extracted by the generation AI are output in a tabular format. This process provides the extraction results in a structured format that is easy for humans to understand, making them easier to use in subsequent business processing and analysis. In step S28, the item-value pairs extracted in step S23 and the processing results of the subsequent steps S25, S26, and S27 are integrated. The data is then output as structured tabular data.

[0102] The basic structure of tabular data may be, for example, rows for each report or product item, and columns for extracted items (e.g., "client name," "date," "amount," etc.). Each cell in the table stores extracted values, normalized values, converted codes, confidence scores, etc. In step S28, various output formats are supported according to user requirements. In addition, it is possible to display the data on a web interface, download it in various file formats such as Excel, CSV, and JSON, and link the data to other systems via API.

[0103] In step S28, the original form image and the extracted results can be displayed side by side to facilitate verification and correction of the extracted results. The extracted portions can also be highlighted. Furthermore, when multiple forms are processed at once, summary information (e.g., total amount, average value, number of processed items, etc.) can be displayed.

[0104] Step S28 enables quality control of the extraction results. For example, a warning mark is attached to extraction results with low confidence or items for which suspicious points have been detected in a business consistency check, prompting the user to confirm. In this way, step S28 outputs the target items extracted by the generation AI in a table format, providing the extraction results in a structured format that is easy for humans to understand and is output in a format that is easy to use for subsequent business processing and analysis. As a result, productivity of the entire business is improved.

[0105] [User Interface Processing] FIG. 5B is a flowchart showing an example of the flow of user interface processing among the processes of the information processing system S of FIG. User interface processing is a series of processes that support interaction between the system and the user. The user interface processing provides a GUI for checking and correcting the reading results. The GUI makes it easy to check and correct as needed, improving the usability and accuracy of the entire system. The purpose of user interface processing is to present the results of form processing to the user in an easy-to-understand manner, and to improve the quality of the extraction results through a GUI that allows for intuitive results checking and correction. The user interface processing makes it possible to compare the reading results of multiple forms. This improves the efficiency of tasks such as reconciling purchase orders and invoices. In this case, when an item to be compared is selected, the corresponding section can be highlighted. This allows the user to easily grasp the differences visually.

[0106] As shown in Figure 5(B), the user interface process includes step S31, which provides a GUI to the user, and step S32, which compares the reports. The user interface process is often implemented as a responsive interface that runs on a web browser and can be accessed not only from desktop PCs but also from tablets and smartphones. Authentication and authorization processes apply functional restrictions according to user privileges, ensuring information security. The user interface process also records an operation log, leaving an audit trail of who made what modifications and when. In this way, the user interface process provides the system's technical processing results in a user-friendly format, and incorporating user judgment improves overall accuracy.

[0107] In step S31, a GUI is provided for checking and correcting the reading results. This process provides the user with a screen that allows them to check and correct the results intuitively. As a result, the user's work efficiency and accuracy are improved. In step S31, a screen is constructed for visually checking and correcting the extraction results of the generation AI process. A standard GUI configuration, for example, is a two-pane format in which the original form image is displayed on the left side of the screen and the extraction results are displayed in tabular format on the right side. A specific example of a two-pane GUI will be described later with reference to Figure 6. The user can check each item in the extraction results of the generation AI process and directly edit any inaccurate results. For user editing, appropriate input methods are provided depending on the nature of the data, such as text input, drop-down selection, and check box operation.

[0108] In step S31, the system enables the display of the extracted items in relation to the corresponding locations on the original ledger form. In this case, when the user clicks on an item in the table display, the source of that information is highlighted on the original ledger form image. This makes it easier to visually check the validity of the extracted results. In addition, a warning mark is displayed for extracted results with low confidence or results that violate business rules, and items that should be checked first are clearly indicated.

[0109] The GUI allows for a dashboard showing the progress and result summary of report processing, a list of batch processing results, and search and filtering. Furthermore, a function for recording and displaying the revision history makes it possible to track who made what revisions and when. Thus, in step S31, the provision of a GUI enables intuitive operation to check and correct results, improving the efficiency and accuracy of user work. As a result, the usability of the entire system improves, and user acceptance and satisfaction increase.

[0110] Step S32 allows the comparison of the scan results of multiple forms. When an item to be compared is selected, the corresponding section is highlighted. This process streamlines the process of checking consistency between multiple forms, such as matching a purchase order with an invoice, thereby reducing the occurrence of errors. Step S32 allows the scan results of multiple related forms (e.g., a set of purchase orders, delivery notes, and invoices) to be displayed side by side. This allows for easy comparison of corresponding items. A typical screen layout in this case would be to display multiple form images and extraction results side by side, with corresponding items on the same line. The main items that are subject to comparison checks include the customer name, date, product name, quantity, unit price, and amount, and it is confirmed whether the values ​​of these items match between the forms.

[0111] In step S32, when an item to be compared is selected, the corresponding portion is highlighted. Also, when the user selects a specific item (for example, product name), the portion of each form in which that item is written is highlighted in color. This makes it easier for the user to make visual comparisons. Also, items with mismatched values ​​are automatically highlighted, drawing the user's attention. For example, if the quantity on the purchase order and the quantity on the invoice differ, the item is displayed in red, providing visual feedback.

[0112] Step S32 not only checks for simple value matches / mismatches, but also performs advanced consistency checks based on business rules. For example, it checks conditions such as "the date on the delivery note should be later than the date on the purchase order" and "the invoice amount, including consumption tax, should match the order amount." Step S32 makes it possible to output the comparison results as a report. In this case, a summary and detailed list of mismatched items are provided. In this way, Step S32 makes it easy to compare and confirm the results of reading multiple forms. As a result, the reconciliation work, which was previously done manually, is significantly more efficient, improving the productivity of the entire business process.

[0113] FIG. 6 is a diagram showing a specific example of a screen displayed on the user terminal 3 by the user interface processing. Figure 6 shows a two-pane GUI that displays the image data of the scanned document (invoice) (display area F1) and the extracted results of the target items (display area F2) side by side on a single screen. By operating the GUI shown in Figure 6, users can compare and edit multiple document data (e.g., OCR results, CSV files, data stored in the Hub, etc.) all at once. Previously, users had to open individual files and manually compare them, but the GUI shown in Figure 6 makes it possible, for example, to simultaneously check the contents of multiple documents and quickly correct any discrepancies. Corrections are reflected in the document data, significantly reducing the need for rework in later processes.

[0114] Specifically, as shown in display area F2 on the right side of the screen in Figure 6, multiple forms and items are displayed in a matrix on one screen. In the example of Figure 6, the original invoice, purchase order, and delivery note are displayed horizontally as multiple forms. Additionally, the invoice number, invoice date, client name, contact person name, product name, unit price, quantity, subtotal, consumption tax, total amount, payment deadline, payment method, and transfer account are displayed vertically as multiple items. This allows the user to visually identify discrepancies.

[0115] When the user selects one of the multiple items displayed in display area F2 on the right side of the drawing, the corresponding portion of the image data displayed in display area F1 on the left side of the drawing is highlighted. In the example of Figure 6, the "Total Amount" content (¥1,229,800) is selected from the multiple items displayed in display area F2. Therefore, the "Billed Amount" content (¥1,229,800-), which corresponds to the image data displayed in display area F1, is highlighted. This allows the user to modify the content of the item while referring to an image of the actual original invoice.

[0116] The GUI in FIG. 6 also compares the character strings in each cell displayed in the display area F2 to determine whether they match or not. The results of the determination are notified to the user by an alert or highlighting. The GUI in FIG. 6 also allows the user to directly correct any mismatches. The corrections are immediately reflected in the form data in the form DB 51 (see FIG. 3). The GUI in FIG. 6 also manages items that have already been compared by displaying them as grayed out, etc. This improves the user's work efficiency.

[0117] <Advantageous Effects of the Present Embodiment> According to the above-described embodiment, it is possible to extract information with high accuracy even from non-standard forms, without depending on the form format.

[0118] <Other> Although one embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and modifications, improvements, etc. within the scope of achieving the object of the present invention are included in the present invention.

[0119] For example, the above-described series of processes can be executed by hardware or software. In other words, the above-described functional configuration is merely an example and is not particularly limited. In other words, it is sufficient for the information processing system to be provided with a function that can execute the above-described series of processes as a whole, and the type of functional block used to realize this function is not particularly limited to the above-described example.

[0120] The location of the functional blocks is not particularly limited and may be arbitrary. For example, the functional blocks of the management server 1 may be transferred to another device, or the functional blocks of another device may be transferred to a server. Furthermore, one functional block may be configured as a single piece of hardware, a single piece of software, or a combination of both.

[0121] When a series of processes is executed by software, the programs constituting the software are installed onto a computer or the like from a network or a recording medium. The computer may be a computer incorporated into dedicated hardware. The computer may also be a computer capable of executing various functions by installing various programs, such as a server, a general-purpose smartphone, or a personal computer.

[0122] The recording medium containing such a program may be configured as a removable medium (not shown) that is distributed separately from the device main body in order to provide the program to users, etc., or may be configured as a recording medium that is pre-installed in the device main body and provided to users, etc. Since the program can be distributed via a network, the recording medium may be installed in or accessible from a computer that is connected or connectable to the network.

[0123] In this specification, the steps describing the program recorded on the recording medium include not only processes that are performed in chronological order, but also processes that are not necessarily performed in chronological order but are performed in parallel or individually. Also, in this specification, the term "system" means an overall device composed of multiple devices or multiple means, etc.

[0124] The following modifications are also included in the present invention. Modifications of the present invention will be described below.

[0125] [Details of the correct answer] One possible configuration is to present the generation AI with pairs of form images and their extraction results as correct examples for extracting specific items from unstructured forms. This variant allows for generalized support for different forms of similar format types by presenting only one correct example per format type. The correct example explicitly shows the correspondence between the input information (character information and coordinate information from the OCR results) and the output information (structured extracted items). Using grouped character information and numbered data makes it easier for the generation AI to understand patterns. Furthermore, providing correct examples optimized for different form types (e.g., invoices, purchase orders, delivery notes, etc.) enables support for a variety of business scenarios. In this way, learning and applying correct examples is automatic, allowing users to benefit from unstructured form processing without specialized knowledge. This enables high extraction accuracy even with limited training data, significantly reducing system initial setup and operational costs.

[0126] [Custom Prompt] A possible configuration is to provide a custom prompt function that allows users to precisely specify the output format and conditions for the scanned results. In this variant, individual custom prompts can be set for each extracted item, allowing users to specify instructions such as "do not include units" or "omit the "Incorporated" part of company names." Custom prompts can be configured using an intuitive GUI, eliminating the need for specialized programming or AI expertise. Users can also select from recommended prompt templates pre-installed in the system, allowing even beginners to optimize their settings. Custom prompt settings can be saved as profiles and reused for similar report processing tasks. Behind the scenes, the system combines user-defined custom prompts with system prompts to issue instructions to the generation AI, achieving highly accurate extraction results that match the user's intentions. This allows for customization to suit specific business or company-specific notation rules, providing flexible functionality that facilitates smooth integration with business systems.

[0127] [Multimodal Processing] One possible configuration is to provide multimodal processing capabilities that integrate image and text information. In this variant, not only text information extracted through OCR processing but also visual features of forms (e.g., logos, layout, ruled lines, font type, etc.) are utilized as analysis targets. Using a multimodal generative AI model enables highly accurate item extraction that takes into account both text and visual information. For example, even when text information is unclear, the content can be inferred from its placement and surrounding visual context. It also facilitates table structure recognition and seal impression detection, which can be used to evaluate the reliability of forms. Furthermore, it also enables information extraction from graphs and charts, allowing necessary information to be obtained from visualized data. This multimodal processing enables highly accurate processing of forms with complex layouts and forms containing many image elements, which were difficult to achieve with conventional text-based processing.

[0128] [Incremental Learning] One possible configuration is to provide an incremental learning function that utilizes correction data accumulated during the system's operation. In this variant, correction information on extraction results made by users through the GUI is collected as learning data to periodically improve the system's extraction accuracy. Specifically, by identifying items and report types with high levels of misrecognition and conducting focused learning on those, accuracy is efficiently improved. Incremental learning employs a method of additional learning based on existing learning results rather than complete re-learning, achieving continuous improvement while ensuring overall system stability. Furthermore, by aggregating correction data from the entire organization and using it for learning, a mechanism is established in which individual user experience contributes to overall accuracy improvement. This function allows the system to self-improve over time and gradually adapt to organization-specific reports and special notations. This is particularly effective when considering not only initial setup but also long-term operation, and can lead to significant improvements in accuracy from the initial installation.

[0129] [Multi-language support] A possible configuration is to provide multilingual processing capabilities for forms containing multiple languages. In this variant, even when different languages, such as Japanese, English, Chinese, and Korean, are mixed in the same form, appropriate OCR processing and semantic understanding are performed for each language, enabling integrated information extraction. A multi-stage process is employed, using OCR engines specialized for each language and integrating the recognition results. Furthermore, by utilizing a multilingual generative AI model, semantic information extraction that transcends language differences is possible. Furthermore, a dictionary function for specific technical terms and industry terminology is implemented, improving translation accuracy. Extraction results can be output in a unified language of the user's choice, or the original language can be retained. This multilingual support streamlines the processing of multilingual forms generated in global transactions and enables seamless business processing across language barriers. This functionality is particularly effective in import / export operations and accounting processes for multinational corporations.

[0130] [Enhanced security features] One possible configuration provides enhanced security features for securely processing forms containing confidential information. This variant automatically detects and protects personal and confidential information to ensure data security. Specifically, it uses generative AI to detect personal information (e.g., name, address, phone number, etc.) and confidential information (e.g., bank account and credit card numbers), and then applies masking or encryption according to access permissions. It also records detailed access logs to form data, enabling tracking of who accessed which data and when. Furthermore, control is implemented according to the confidentiality level of the information when outputting processing results, minimizing the display of certain fields to unauthorized users. Encryption and access control are also applied to data storage, minimizing the risk of unauthorized access and data leaks. This enhanced security feature allows systems to be used safely even in areas with high security requirements, such as medical institutions and financial institutions. This is particularly important in response to recent tightening data protection regulations and contributing to reducing compliance risks.

[0131] In other words, the information processing system to which the present invention is applied can take various forms having the following configurations. (1) That is, the information processing system S to which the present invention is applied is an information processing system characterized by having a document reading means (e.g., a reading unit of a user terminal 3) that reads a document, an OCR processing means (e.g., an OCR unit 33 in FIG. 3) that performs OCR processing on the document reading results, an item extraction means (e.g., an item extraction unit 34 in FIG. 3) that extracts target items from the processing results of the OCR processing, and an output control means (e.g., an output control unit 35 in FIG. 3) that outputs the extracted target items in a predetermined format. This makes it possible to extract information with high accuracy even from non-standard forms, regardless of the form format.

[0132] (2) It can also be characterized by further having a document storage means (e.g., information management unit 32 in FIG. 3) for storing the reading results, an OCR result storage means (e.g., information management unit 32 in FIG. 3) for storing the processing results of the OCR processing, and an extraction result storage means (e.g., information management unit 32 in FIG. 3) for storing the extraction results of the target items.

[0133] (3) The OCR processing means may extract character information and coordinate information by OCR processing of the entire document, and the item extraction means may extract the target items based on acquired items specified in natural language without specifying reading coordinates.

[0134] (4) The present invention can further include a grouping means (for example, the grouping unit 36 ​​in FIG. 3) for grouping character information having adjacent coordinate information based on the extracted character information and coordinate information.

[0135] (5) The information processing device may further comprise a numbering means (for example, the numbering unit 37 in FIG. 3) for numbering the character information.

[0136] (6) The item extraction means may be characterized in that it shows correct examples to the generation AI to cause the generation AI to extract the target items.

[0137] (7) The item extraction means may be characterized in that it displays the correct answer examples in a format suited to the characteristics of each of the plurality of generation AIs.

[0138] (8) The present invention can further include a matching means (for example, the matching unit 38 in FIG. 3) that takes into account the shape characteristics of the characters and absorbs variations in spelling.

[0139] (9) The information processing device may further include a conversion means (for example, the conversion unit 39 in FIG. 3) for converting the extracted target items into pre-specified identification information.

[0140] (10) Furthermore, the document reading means reads a plurality of documents having different formats in a batch, the OCR processing means performs the OCR processing on the plurality of documents read in a batch, the item extraction means extracts the target items from the processing results of the OCR processing in a batch, and the output control means outputs the extracted target items in a batch in the specified format.

[0141] (11) The present invention can further include a normalization means (for example, the normalization unit 40 in FIG. 3) for normalizing spelling variations in the results of the OCR processing.

[0142] (12) The output control means may further output a predetermined GUI that enables viewing and editing of the processing results of the OCR processing.

[0143] (13) The system may further include a comparison means (e.g., comparison unit 41 in FIG. 3) for comparing the target items of the extracted multiple documents, and the comparison means may highlight selected items from the target items that are the subject of comparison.

[0144] (14) The system may further include a learning data storage means (e.g., information management unit 32 in FIG. 3) for storing two learning data images for each format of the form, a configuration extraction means (e.g., configuration extraction unit 42 in FIG. 3) for extracting a configuration common to the two stored learning data images, and a prompt generation means (e.g., prompt generation unit 43 in FIG. 3) for generating a prompt for classifying the form into a generation AI.

[0145] Furthermore, the information processing method to which the present invention is applied can take various forms having the following configurations. (15) In other words, the information processing method to which the present invention is applied is an information processing method characterized by including the steps of reading a document, performing OCR processing on the results of reading the document, extracting target items from the results of the OCR processing, and outputting the extracted target items in a predetermined format.

[0146] Furthermore, the program to which the present invention is applied can take various forms having the following configurations. (16) In other words, the program to which the present invention is applied is a program for causing a computer to execute control processing including the steps of reading a document, performing OCR processing on the results of reading the document, extracting target items from the results of the OCR processing, and outputting the extracted target items in a predetermined format. [Explanation of symbols]

[0147] 1: Management server, 2: Administrator terminal, 3: User terminal, 11: CPU, 16: Output unit, 17: Input unit, 18: Memory unit, 19: Communication unit, 31: Information acquisition unit, 32: Information management unit, 33: OCR unit, 34: Item extraction unit, 35: Output control unit, 36: Grouping unit, 37: Numbering unit, 38: Matching unit, 39: Conversion unit, 40: Normalization unit, 41: Comparison unit, 42: Configuration extraction unit, 43: Transmission control unit, 44: Transmission control unit, S: Information processing system, N: Network

Claims

1. a form reading means for reading a form; an OCR processing means for executing OCR processing on the reading result of the document; an item extraction means for extracting target items from the processing result of the OCR processing; an output control means for outputting the extracted target items in a predetermined format; An information processing system comprising:

2. a form storage means for storing the reading results; an OCR result storage means for storing a processing result of the OCR processing; and an extraction result storage means for storing the extraction results of the target items. The information processing system according to claim 1 .

3. the OCR processing means extracts character information and coordinate information by OCR processing of the entire form; The item extraction means extracts the target items based on acquired items specified in natural language without specifying reading coordinates. The information processing system according to claim 1 .

4. The method further comprises a grouping means for grouping character information having adjacent coordinate information based on the extracted character information and coordinate information. The information processing system according to claim 3 .

5. The method further comprises numbering means for numbering the character information. The information processing system according to claim 3 .

6. The item extraction means is characterized in that it shows correct examples to the generation AI and causes the generation AI to extract the target items. The information processing system according to claim 1 .

7. The item extraction means is characterized in that it shows the correct answer example in a format suited to the characteristics of each of the plurality of generation AIs, The information processing system according to claim 6.

8. The system further comprises a matching means for absorbing variations in spelling by taking into account the shape characteristics of the characters. The information processing system according to claim 1 .

9. The method further comprises converting means for converting the extracted target items into pre-specified identification information. The information processing system according to claim 1 .

10. The form reading means reads a plurality of forms having different formats at once, The OCR processing means executes the OCR processing on the plurality of read forms collectively, the item extraction means extracts the target items all at once from the processing result of the OCR processing, The output control means outputs the extracted target items in the predetermined format all at once. The information processing system according to claim 1 .

11. The method further comprises normalizing means for normalizing spelling variations in the processing result of the OCR processing. The information processing system according to claim 1 .

12. The output control means further outputs a predetermined GUI that enables viewing and editing of the processing result of the OCR processing. The information processing system according to claim 1 .

13. The method further includes a comparison means for comparing the target items of the extracted plurality of forms, The comparison means highlights a selected item from among the target items that are to be compared. The information processing system according to claim 1 .

14. a learning data storage means for storing two learning data images for each format of the form; A feature extraction means for extracting a feature common to the two stored learning data images; The system further comprises a prompt generating means for generating a prompt for causing the generation AI to classify the form. The information processing system according to claim 1 .

15. An information processing method executed by an information processing system, reading the document; a step of performing OCR processing on the reading result of the form; extracting target items from the processing results of the OCR processing; outputting the extracted target items in a predetermined format; An information processing method comprising:

16. On the computer, reading the document; a step of performing OCR processing on the reading result of the form; extracting target items from the processing results of the OCR processing; outputting the extracted target items in a predetermined format; A program for executing control processing including:

Images