Information processing apparatus, information processing method, and program

Abstract

To provide an information processing apparatus, an information processing method, and a program for reducing the load in obtaining character strings corresponding to predetermined items from a document image.SOLUTION: In an information processing system 100, an information processing server 103 includes: a character image analysis unit 154 configured to perform character recognition processing on an image in a document to be processed; an instruction message generation unit 155 configured to generate an instruction message for causing a large language model to reply a character string corresponding to a predetermined item included in the document, based on a result of the character recognition processing, and transmit the instruction message to an external information processing server 105 in order to obtain a reply to the instruction message from the large language model; and a reply receiving unit 156 configured to receive the reply to the instruction message from the large language model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to the process of extracting information from document images. [Background technology]

[0002] There is a technology that performs OCR processing on an input document image and extracts item values, which are character strings corresponding to items such as dates, amounts, and company names, from a group of character strings recognized as a result of the OCR processing.

[0003] Patent Document 1 discloses a method for extracting character strings (values) corresponding to specific items (keys) by inputting the character recognition results of a form image into a learning model for extracting character strings (values) corresponding to specific items (keys). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-077332 Summary of the Invention [Problem to be solved by the invention]

[0005] Patent Document 1 also discloses that if an appropriate character string is not extracted for a specific item, the user checks the character string for the specific item and corrects it to an appropriate character string, and generates learning data for the learning model based on the correction result. However, in such a method, the user needs to carefully check and correct the item values ​​until the collection of data for re-learning the learning model for key-value extraction is completed, which tends to increase the burden on the user. In addition, the method of updating the extraction rules for the learning model for key-value extraction cannot handle cases where the character recognition result of the document image contains misrecognized characters, and it becomes necessary for the user to carefully check and correct the item values. [Means for solving the problem]

[0006] The information processing device disclosed herein is characterized by having a character recognition means for performing character recognition processing on an image of a document to be processed, a generation means for generating an instruction sentence for causing a large-scale language model to respond with a character string corresponding to a specified item contained in the document based on a result of the character recognition processing, a transmission means for transmitting the instruction sentence to obtain an answer to the instruction sentence from the large-scale language model, and a receiving means for receiving the answer to the instruction sentence from the large-scale language model. Effect of the Invention

[0007] According to the present disclosure, it is possible to reduce the burden involved in obtaining a character string corresponding to a predetermined item from a document image. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an information processing system. [Diagram 2] FIG. 2 is a diagram showing an example of the hardware configuration of each device constituting the information processing system. [Diagram 3] FIG. 1 is a sequence diagram of an information processing system. [Figure 4] 1 is a flowchart illustrating details of a process for generating a machine learning model. [Diagram 5] 11 is a flowchart illustrating details of a process for displaying item values ​​to a user. [Figure 6] 11 is a flowchart illustrating details of an item value extraction process using a machine learning model. [Figure 7] 4A and 4B are diagrams showing an example of a document image to be processed and a processing result. [Figure 8] 11 is a flowchart illustrating details of a process for obtaining an answer from a large-scale language model. [Figure 9] FIG. [Figure 10] FIG. 13 is a diagram showing an example of an item value confirmation screen. [Figure 11] 11 is a flowchart illustrating details of a process for displaying item values ​​to a user. [Figure 12]11 is a flowchart illustrating details of a process for obtaining an answer from a large-scale language model. [Figure 13] 4A and 4B are diagrams showing an example of a document image to be processed and a processing result. [Figure 14] FIG. [Figure 15] FIG. 13 is a diagram showing an example of an item value confirmation screen. [Figure 16] FIG. 1 is a sequence diagram of an information processing system. [Figure 17] FIG. 13 is a diagram showing a large-scale language model setting screen. [Figure 18] FIG. 13 is a diagram showing a setting screen for character string correction options. [Figure 19] 11 is a flowchart illustrating details of a process for obtaining an answer from a large-scale language model. [Figure 20] FIG. 13 is a diagram showing an example of an item value confirmation screen. [Figure 21] FIG. 1 is a diagram showing an example of the configuration of an information processing system. [Figure 22] FIG. 1 is a sequence diagram of an information processing system. [Diagram 23] 11 is a flowchart illustrating details of a process for displaying item values ​​to a user. [Figure 24] FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the technology of the present disclosure will be described with reference to the drawings. Note that the components described in the following embodiments are merely examples and are not intended to limit the scope of the technology of the present disclosure.

[0010] <Embodiment 1> [About the configuration of information processing systems] 1 is a diagram showing an example of the configuration of an information processing system 100 according to the present embodiment. The information processing system 100 includes, for example, an image forming device 101, a learning device 102, an information processing server 103, and an external information processing server 105, and each device is connected to each other via a network 104. In the information processing system 100, the image forming device 101, the learning device 102, and the information processing server 103 may be configured to be connected to the network 104 in a multi-connection configuration rather than a single connection. For example, the information processing server 103 may be configured to include a first server device having high-speed computing resources and a second server device having large-capacity storage, and the first server device and the second server device may be configured to be connected to each other via the network 104.

[0011] The image forming apparatus 101 is realized by, for example, an MFP (Multi-Function Peripheral) having multiple functions such as printing, scanning, and faxing. The image forming apparatus 101 has at least an image acquisition unit 151 and a display control unit 159 as functional units.

[0012] The image forming apparatus 101 has a scanner device 206 (see FIG. 2). The scanner device 206 optically reads an original 111, which is a document with a character string printed on a recording medium such as paper, and the image acquisition unit 151 acquires a document image 113, which is a scanned image, by performing predetermined image processing on the resulting data. Alternatively, for example, the image acquisition unit 151 receives FAX data 112 transmitted from a FAX transmitter (not shown), and acquires the document image 113 by performing predetermined FAX image processing. The image acquisition unit 151 transmits the acquired document image 113 to the information processing server 103.

[0013] The display control unit 159 displays the information received from the information processing server 103 on the display device 210. (see FIG. 2) on the display. The display control unit 159 displays, for example, an item value confirmation screen 1000 (see FIG. 10) which will be described later.

[0014] The image forming apparatus 101 may be realized by a PC (Personal Computer) or the like, other than an MFP equipped with scan and FAX functions. In this case, for example, a document image 113 such as PDF or JPEG generated by using a document creation application running on the PC may be transmitted to the information processing server 103 as a processing target.

[0015] The learning device 102 has the functional units of a learning data generation unit 152 and a learning unit 153 .

[0016] The learning data generator 152 generates learning data based on a plurality of document image samples 114 .

[0017] The learning unit 153 generates a trained model (machine learning model) by training the learning model based on the learning data generated by the learning data generation unit 152. In this embodiment, an item value extractor 115 is generated as the machine learning model, which outputs information indicating character strings (item values) corresponding to items included in a document image to be processed as a result. The learning device 102 transmits the generated machine learning model to the information processing server 103 via the network 104. Details of the processing in the learning device 102 will be described later. The item value extractor 115 in this embodiment will be described as a trained model generated by machine learning, but it may also be a model that makes a judgment using a rule-based algorithm and outputs a result.

[0018] The information processing server 103 is a device that processes the input document image 113 to be processed and transmits each processing result to the image forming device 101. The information processing server 103 has each of the functional units, a document image analysis unit 154, an instruction statement generation unit 155, a response receiving unit 156, a data management unit 157, and a display control unit 158.

[0019] The document image analysis unit 154 receives the document image 113 sent from the image forming apparatus 101, executes OCR processing on the document image 113, and acquires a group of character strings recognized from the document image 113. The document image analysis unit 154 uses the item value extractor 115 to extract character strings (item values) corresponding to items such as date, company name, total amount, etc., from the group of character strings recognized from the document image 113. The names of the items are referred to as item names. Also, the character strings corresponding to the items are referred to as item values.

[0020] The instruction statement generation unit 155 generates an instruction statement by applying the item name, etc. to an instruction statement template prepared in advance. The instruction statement generation unit 155 transmits the instruction statement via the network 104 so that the instruction statement is input to the external information processing server 105. The details of the instruction statement will be described later.

[0021] The answer receiving unit 156 receives an answer to a directive sentence output by the large-scale language model 116 .

[0022] The data management unit 157 stores and manages the response to the instruction sentence generated by the large-scale language model 116 in the storage unit. In addition, the data management unit 157 stores and manages the item values ​​of the document image 113 confirmed by the user via the item value confirmation screen 1000 in the storage unit.

[0023] The display control unit 158 ​​performs control for displaying to the user the item values ​​extracted by the document image analysis unit 154 and the answers to the instruction sentences obtained from the large-scale language model 116. For example, the display control unit 158 ​​generates information for displaying an item value confirmation screen 1000 (see FIG. 10 ) described later, and displays it on the display unit. When the user instructs to correct the displayed item value, the display control unit 158 ​​accepts the correction and generates the item value confirmation screen 1000 according to the user's correction.

[0024] The external information processing server 105 is a device that operates a large-scale language model 116. The large-scale language model 116 is a model that can generate sentences in an interactive format called an LLM (Large Language Model), and generates responses to input instructions (prompts). For example, Chat GPT (registered trademark) and Bard (pending trademark) are known as LLMs.

[0025] The large-scale language model 116 is accessed by an API (Application Programming Interface) via the network 104. The large-scale language model 116 outputs a response to a command input from the information processing server 103 as an output result. The external information processing server 105 may be configured to exist on a different system of the same vendor, or may be configured to exist on a system of an external vendor. The large-scale language model 116 may be configured to exist within the information processing server 103, or may be configured such that some functions or devices exist within the information processing server 103.

[0026] The network 104 is realized by a LAN, WAN, etc., and is a communication unit that connects the image forming device 101, the learning device 102, the information processing server 103, and the external information processing server 105 to each other, and transmits and receives data between the devices.

[0027] [Hardware configuration of image forming device] FIG. 2 is a diagram showing an example of the hardware configuration of the image forming apparatus 101, the learning apparatus 102, and the information processing server 103 included in the information processing system 100. As shown in FIG.

[0028] Fig. 2(a) is a diagram showing the hardware configuration of the image forming apparatus 101. As shown in Fig. 2(a), the image forming apparatus 101 has a CPU 201, a ROM 202, a RAM 204, a printer device 205, a scanner device 206, a document transport device 207, and a storage 208. It also has an input device 209, a display device 210, and an external interface 211. Each unit is connected to each other via a data bus 203.

[0029] The CPU 201 is a control unit that controls the overall operation of the image forming apparatus 101. The CPU 201 executes a startup program stored in a ROM 202 to start up the system of the image forming apparatus 101, and executes a control program stored in a storage 208 to realize functions of the image forming apparatus 101, such as printing, scanning, and faxing.

[0030] ROM 202 is a storage unit realized by a non-volatile memory, and stores a startup program for starting up image forming apparatus 101. Data bus 203 is a communication unit for transmitting and receiving data between devices constituting image forming apparatus 101. RAM 204 is a storage unit realized by a volatile memory, and is used as a work memory when CPU 201 executes the control program.

[0031] The printer device 205 is an image output device that prints and outputs a document image on a storage medium such as paper. The scanner device 206 is an image input device that optically reads a storage medium such as paper on which characters, diagrams, etc. are printed. Data obtained by the scanner device 206 reading is acquired as a document image. The document transport device 207 is realized by an ADF (Auto Document Feeder) or the like, detects a document placed on a document platen, and transports the detected document one sheet at a time to the scanner device 206. The storage 208 is a storage unit realized by an HDD (Hard Disk Drive) or the like, and stores the above-mentioned control programs and document images.

[0032] The input device 209 is an operation unit realized by a touch panel, hard keys, or the like, and accepts operation input from a user who uses the image forming apparatus 101. The display device 210 is a display unit realized by a liquid crystal display or the like, and displays and outputs a setting screen, etc. of the image forming apparatus 101 to the user. The external interface 211 is an interface that connects the image forming apparatus 101 and the network 104, and receives FAX data from a FAX transmitter (not shown) and transmits document images to the information processing server 103.

[0033] [Hardware configuration of the learning device] Fig. 2(b) is a diagram showing the hardware configuration of the learning device 102. As shown in Fig. 2(b), the learning device 102 has a CPU 231, a ROM 232, a RAM 234, a storage 235, an input device 236, a display device 237, an external interface 238, and a GPU 239. Each part is connected to each other via a data bus 233.

[0034] The CPU 231 is a control unit that controls the overall operation of the learning device 102. The CPU 231 starts the system of the learning device 102 by executing a boot program stored in the ROM 232, and generates a machine learning model for extracting item values ​​by executing a learning program stored in the storage 235. The ROM 232 is a storage unit realized by a non-volatile memory, and stores a boot program that starts the learning device 102. The data bus 233 is a communication unit for transmitting and receiving data between devices that configure the learning device 102. The RAM 234 is a storage unit realized by a volatile memory, and is used as a work memory when the CPU 231 executes the learning program.

[0035] Storage 235 is a memory unit realized by a HDD (Hard Disk Drive) or the like, and stores the learning program and document image samples described above. Input device 236 is an operation unit realized by a mouse, keyboard, or the like, and accepts operation input from an engineer who controls learning device 102. Display device 237 is a display unit realized by a liquid crystal display or the like, and displays the setting screen of learning device 102, etc., to the engineer.

[0036] The external interface 238 is an interface that connects the learning device 102 and the network 104, and receives the document image sample 114 from the outside and transmits the machine learning model to the information processing server 103. The GPU 239 is a calculation unit configured with an image processing processor. The GPU 239 executes calculations for generating a machine learning model based on a group of character strings included in a given document image, for example, in accordance with a control command given from the CPU 231.

[0037] Each functional unit included in the learning device 102 shown in Fig. 1 is realized by the CPU 231 executing a predetermined program, but is not limited to this. Other hardware such as a GPU 239 for accelerating calculations or an FPGA (Field Programmable Gate Array) (not shown) may be used. Each functional unit may be realized by cooperation between software and hardware such as a dedicated IC, or some or all of the functions may be realized only by hardware.

[0038] [Hardware configuration of information processing server] Fig. 2(c) is a diagram showing a hardware configuration of the information processing server 103. As shown in Fig. 2(c), the information processing server 103 has a CPU 261, a ROM 262, a RAM 264, a storage 265, an input device 266, a display device 267, and an external interface 268, and each part is connected to each other via a data bus 263.

[0039] The CPU 261 is a control unit that controls the overall operation of the information processing server 103. The CPU 261 executes a boot program stored in a ROM 262 to start up the system of the information processing server 103, and executes an information processing program stored in a storage 265 to perform information processing such as character recognition (OCR) and information extraction.

[0040] The ROM 262 is a storage unit realized by a non-volatile memory, and stores a boot program that starts up the information processing server 103. The data bus 263 is a communication unit for transmitting and receiving data between devices constituting the information processing server 103. The RAM 264 is a storage unit realized by a volatile memory, and is used as a work memory when the CPU 261 executes the information processing program. The storage 265 is a storage unit realized by an HDD (Hard Disk Drive) or the like, and stores the above-mentioned information processing program, machine learning model, document image, extracted item value, etc.

[0041] The input device 266 is an operation unit realized by a mouse, a keyboard, or the like, and accepts operation input to the information processing server 103 from a user or an engineer who uses the information processing server 103. The display device 267 is a display unit realized by a liquid crystal display or the like, and displays and outputs a setting screen of the information processing server 103 to a user or an engineer who uses the information processing server 103.

[0042] The external interface 268 is an interface that connects the information processing server 103 and the network 104, and receives a machine learning model from the learning device 102 and a document image from the image forming device 101.

[0043] Each functional unit included in the information processing server 103 in Fig. 1 is realized by the CPU 261 executing a predetermined program, but is not limited to this. Other hardware such as a GPU (Graphics Processing Unit) or an FPGA (Field Programmable Gate Array) for accelerating calculations may also be used. Each functional unit may be realized by cooperation between software and hardware such as a dedicated IC, or some or all of the functions may be realized only by hardware.

[0044] [Sequence of trained model generation] 3 is a diagram showing a sequence of the information processing system 100. The symbol "S" in the description of each process means a step in the sequence, and the same applies to the following flowcharts. For convenience of explanation, the operation of a user or engineer is also explained using steps.

[0045] FIG. 3(a) shows a flow of development by an engineer of the item value extractor 115, which is a machine learning model.

[0046] In S301, an engineer related to the information processing system 100 inputs a plurality of document image samples 114, which are samples of images representing documents, to the learning device 102. The document image samples 114 are document images such as invoices, estimates, purchase orders, and delivery notes.

[0047] In S302, the learning data generation unit 152 of the learning device 102 generates learning data based on the document image sample 114, and the learning unit 153 performs machine learning using the learning data to generate an item value extractor 115, which is a machine learning model.

[0048] In S303, the learning device 102 transmits the generated item value extractor 115 to the information processing server 103. The information processing server 103 stores the item value extractor 115 in the storage 265. Details of S302 to S303 in FIG. 3(a) will be described later with reference to FIG.

[0049] [Item value extraction process sequence] 3B is a diagram for explaining the flow of processing for extracting item values ​​from a document image to be processed. Details of S313 to S316 and S318, which are the processing of the information processing server 103 in FIG. 3B, will be described later with reference to FIG.

[0050] In S311, the user places a paper document (original) on the image forming apparatus 101 and instructs the image forming apparatus 101 to scan the document.

[0051] In S312, the scanner device 206 of the image forming apparatus 101 reads the placed paper document, and the image acquisition unit 151 generates a document image that is an image representing the scanned document. Then, the image acquisition unit 151 transmits the generated document image to the information processing server 103 as a document image to be processed.

[0052] In S313, the document image analysis unit 154 of the information processing server 103 executes character recognition processing (OCR processing) on ​​the document image to be processed transmitted in S312, and acquires a group of character strings recognized from the document image.

[0053] In S314, the document image analysis unit 154 inputs data of a group of character strings recognized from the document image to be processed to the item value extractor 115, and extracts character strings corresponding to the respective items from the group of character strings as item values.

[0054] In S315, the instruction statement generator 155 generates an instruction statement using the item name and the item value extracted in S314. Details of the instruction statement will be described later.

[0055] In S316, the information processing server 103 transmits the instruction generated in S315 to the external information processing server 105.

[0056] In S317, the external information processing server 105 receives the instruction sent in S316, and causes the large-scale language model 116 to generate a response to the received instruction. The response to the instruction is returned to the information processing server 103.

[0057] In S318, the display control unit 158 ​​of the information processing server 103 converts the item values ​​extracted in S314 based on the output result of the item value extractor 115 and the result of the response to the instruction statement transmitted in S317 into information for presenting to the user. The display control unit 158 ​​transmits the converted information to the image forming apparatus 101. The display control unit 159 of the image forming apparatus 101 presents the processing result to the user by displaying an item value confirmation screen 1000 (see FIG. 10).

[0058] [Machine learning model generation process] Fig. 4 is a flowchart explaining the details of the generation process of item value extractor 115 in S302-303 in Fig. 3(a) executed by learning device 102. The series of processes shown in the flowchart in Fig. 4 will be explained as being performed by CPU 231 of learning device 102 expanding program code stored in ROM 232 or storage 235 into RAM 234 and executing it. Some or all of the functions of the steps in Fig. 4 may be executed by GPU 239, or may be realized by hardware such as ASIC or electronic circuitry.

[0059] In S401, the CPU 231 acquires a plurality of document image samples input by an engineer in S301 of Fig. 3. For example, document image samples are acquired that are generally called semi-standard forms such as invoices, estimates, and purchase orders, and are created with different layouts for each issuing company.

[0060] In S402, the CPU 231 executes block selection (BS) processing and character recognition processing (OCR processing) on ​​the document image sample acquired in S401, and acquires a group of character strings recognized from the document image sample.

[0061] Block selection (BS) processing is a process that selects block areas so that a document image is divided into object units that make up the document image, and determines the attributes of each block area. Specifically, it is a process that determines attributes such as text, photos, and diagrams, and divides the document image into block areas with different attributes. Block selection (BS) processing can be realized using a known area determination technique.

[0062] The data of the character string group obtained as a result of the OCR process may be, for example, word-delimited character strings arranged in the document image with spaces or lines, which constitute the document image, read out continuously in a predetermined read order based on the arrangement information.Furthermore, the data of the character string group obtained may be, for example, word-delimited character strings divided into sentences constituting the document image using a morphological analysis technique, which are read out continuously in a predetermined read order based on the arrangement information.

[0063] In S403, the CPU 231 acquires a correct answer label indicating which item corresponds to the character string (item value) to be extracted, which is included in the character string group acquired in S402. The items are, for example, "date", "company name", and "total amount". The correct answer label may be assigned manually by an engineer, or may be assigned automatically by inputting a document image sample into a model that extracts generated item values. Then, the CPU 231 generates learning data which is a combination of the character string group recognized from the document image sample and data in which the correct answer label is assigned to the character string that is the item value of the character string group. Learning data corresponding to each of the multiple document samples is generated.

[0064] In S404, the CPU 231 generates the item value extractor 115, which is a machine learning model, by machine learning using the learning data. The item value extractor 115 is a trained model trained to output information on a character string (item value) corresponding to an item to be extracted from the character string group when feature amount data of a character string group included in a document image to be processed is input. The item value extractor 115 of this embodiment is a trained model trained to be able to output, for example, a label corresponding to a correct label. The item value extractor 115 is generated by training a prepared learning model so that, when feature amounts of a character string group are input, the item value extractor 115 outputs a label indicating the name of the corresponding item for the character string to be extracted and does not output a label for character strings not to be extracted.

[0065] A known method may be used to generate the item value extractor 115. For example, a feature vector indicating the feature amount of a character string converted using Word2Vec, fastText, BERT, XLNet, ALBERT, or the like, and the position coordinates of the character string arranged in a document image may be used. Specifically, for example, a BERT language model that has been trained on a general text (e.g., the entire article on Wikipedia) may be used to convert a single character string data into a feature vector expressed by a 768-dimensional numerical value. The learning model may be a logistic regression, a decision tree, a random forest, a support vector machine, a neural network, or the like, which are generally known as machine learning algorithms. Specifically, for example, an inference result of a label of any predetermined item information may be output according to the output value of a fully connected layer of a neural network to which a feature vector output by the BERT language model is input.

[0066] In S405, the CPU 231 transmits the generated item value extractor 115 to the information processing server 103. Then, the item value extractor 115 is stored in the storage 265 in the information processing server 103.

[0067] [Item value display processing] Fig. 5 is a flowchart for explaining details of a process in which the information processing server 103 extracts item values ​​from a group of character strings included in a document image to be processed and displays the extracted item values ​​to a user. That is, it is a flowchart for explaining details of S313 to S318 in Fig. 3. A series of processes shown in the flowchart in Fig. 5 is performed by the CPU 261 of the information processing server 103 expanding program codes stored in the ROM 262 or the storage 265 into the RAM 264 and executing them. In addition, some or all of the functions of the steps in Fig. 5 may be realized by hardware such as an ASIC or an electronic circuit. First, the flowchart in Fig. 5(a) will be explained.

[0068] In S501, the CPU 261 extracts item values ​​from a group of character strings included in a document image to be processed, using the item value extractor 115, which is a machine learning model. The processing in this step is called an item value extraction process.

[0069] FIG. 6 is a flowchart illustrating the item value extraction process in S501 in detail.

[0070] In S601, the CPU 261 acquires the item value extractor 115, which is a machine learning model, transmitted from the learning device .

[0071] In S602, the CPU 261 acquires the document image transmitted from the image forming apparatus 101. The document image acquired in S602 becomes the document image to be processed.

[0072] 7A is a diagram showing a document image 700, which is an example of a document image to be processed acquired in S602. The document image 700 is an example of a document image of a "purchase order sheet."

[0073] In S603, the CPU 261 executes the above-mentioned block selection (BS) processing and OCR processing on the document image to be processed, and acquires a group of character strings recognized from the document image to be processed.

[0074] Character string areas 701-704 enclosed in dotted lines in Fig. 7(a) indicate some of the character string areas included in the document image 700. In S603, a character string recognized from each character string area in the document image to be processed is obtained. A unique value for identifying each character string area is assigned to each character string area as an area ID.

[0075] In S604, the CPU 261 inputs the data of the character string group acquired in S603 to the item value extractor 115 acquired in S601. Then, based on the output result of the item value extractor 115, the CPU 261 extracts character strings (item values) corresponding to each item from the character string group recognized from the document image to be processed.

[0076] In this embodiment, the items to be extracted are described as "date," "company name," and "total amount." "Date" refers to the document issue date. "Company name" refers to the name of the company to which the document was issued. "Total amount" refers to the total amount in the document.

[0077] Table 710 in FIG. 7(b) summarizes in table form the results of the OCR process in S603 and the results of the item value extraction in S604. Column 711 in table 710 holds area IDs indicating character string areas in the document image. Column 712 holds character strings obtained as a result of OCR process from the character string areas indicated by the area IDs. Column 713 holds item names indicating items corresponding to the character strings held in column 712. In this way, item values ​​of character strings recognized as a result of OCR process are associated with item names indicating the corresponding items. FIG. 7(b) displays character strings extracted as item values ​​from the group of character strings obtained as a result of S603.

[0078] For example, in column 712 of a row (record) in which "703" and "704" are stored as area IDs in column 711 of table 710, the character string "27,500 yen" is stored. Column 713 of the same record stores the item name "Total Amount." This shows that the character string "27,500 yen" was extracted from document image 700 as the item value corresponding to the "Total Amount" item.

[0079] Column 712 of the record in which the area ID "701" is stored holds the character string "2023 / 2 / 7" obtained as a result of OCR processing from the character string area 701 included in the document image 700. As shown in FIG. 7(a), the character string actually written in the character string area 701 of the document image 700 is "2023 / 11 / 7". Therefore, as a result of the OCR processing in S603, "2023 / 11 / 7" included in the character string area 701 was erroneously recognized as "2023 / 2 / 7", and the erroneously recognized character string was obtained. As a result, it is shown that "2023 / 2 / 7" was also extracted as the item value corresponding to the "date" item in S604.

[0080] Similarly, the character string "XY2 Co., Ltd." is stored in column 712 of the record that holds the area ID "702." This indicates that as a result of the OCR process in S603, "XYZ Co., Ltd." contained in the character string area 702 was misrecognized as "XY2 Co., Ltd." As a result, "XY2 Co., Ltd." was also extracted in S604 as the field value corresponding to the "Company Name" field.

[0081] This is the end of the flow chart of Fig. 6, and therefore the process of S501 ends. Then, the process proceeds to S502 in the flow chart of Fig. 5.

[0082] In S502 of FIG. 5, a process of generating a prompt and acquiring an answer to the prompt from the large-scale language model 116 is performed.

[0083] FIG. 8 is a flowchart illustrating the details of the process of S502.

[0084] In S801, the CPU 261 acquires the item names of the items to be extracted in the item value extraction process in S501 and the item values ​​extracted in the item value extraction process in S501. In this embodiment, the item names of the items to be extracted, "Date", "Company Name", and "Total Amount", are acquired. In addition, as shown in table 710, "2023 / 02 / 07", "XY2 Corporation", and "￥27,500" are acquired as the item values ​​of the items "Date", "Company Name", and "Total Amount".

[0085] In S802, the CPU 261 obtains an instruction statement template from the storage 265. The instruction statement template prepared in advance may be one prepared by an engineer as a preset, one prepared by a user, or one modified or added to by the system or the user.

[0086] In S803, the CPU 261 generates an instruction statement for each item by applying the item name and item value acquired in S801 to the instruction statement template acquired in S802.

[0087] Fig. 9A is a diagram showing an example of the instruction statement template acquired in S802. An instruction statement template 900 in Fig. 9A includes item name regions 901 and 902 and an item value region 903.

[0088] In S803, the CPU 261 selects an item for which a command statement is to be generated, and assigns an item name indicating the name of the item for which a command statement is to be generated to item name areas 901 and 902 of the command statement template. In addition, the CPU 261 assigns an item value corresponding to the item for which a command statement is to be generated, among the item values ​​acquired in S801, to an item value area 903. In this manner, a command statement for the item for which a command statement is to be generated is generated.

[0089] Instructions 911 to 913 in FIGS. 9(b) to (d) are examples of instruction statements generated in S804, and indicate instruction statements generated when document image 700 in FIG. 7(a) is acquired as the document image to be processed.

[0090] In the item value extraction process of S501, it is assumed that "XY2 Corporation" is extracted from the document image 700 as an item value corresponding to the "company name" item. In this case, by applying "company name" to the item name areas 901 and 902 of the instruction statement template 900 and applying "XY2 Corporation" to the item value area 903, an instruction statement 911 for the "company name" item is generated as shown in FIG. 9(b). The instruction statement 912 in FIG. 9(c) is an instruction statement for "date" generated by applying "date" to the item name areas 901 and 902 of the instruction statement template 900 and applying "2023 / 02 / 07", which is the item value of "date" extracted, to the item value area 903. The instruction statement 913 in FIG. 9(d) is an instruction statement for “Total Amount” generated by applying “Total Amount” to the item name fields 901 and 902 of the instruction statement template 900 and applying the extracted item value of “Total Amount”, “￥27,500”, to the item value field 903.

[0091] The contents of the instruction statement template may be switched for each item. In this case, in S802, an instruction statement template in which item names are written in advance in item name areas 901 and 902 may be acquired. In this case, in S803, CPU 261 may apply only the item values ​​acquired in S801 to the instruction statement template.

[0092] Furthermore, the CPU 261 may switch the content of the instruction template for each language or destination of the document image to be processed. Furthermore, the instruction template may be one prepared in advance by an engineer or a user, or may be one prepared in advance and edited by the user.

[0093] In S804, the CPU 261 performs processing to input the instruction sentence generated in S803 to the large-scale language model 116. The CPU 261 transmits the instruction sentence to the external information processing server 105 so that the instruction sentence is input to the large-scale language model 116.

[0094] In S805, the CPU 261 receives from the large-scale language model 116 a response to the instruction sentence input in S804.

[0095] The instruction generated in S803 includes an instruction to have the large-scale language model 116 return an item value with the typographical error corrected if there is an error in the item value extracted in the item value extraction process in S501. In other words, if there is an error in the item value, the large-scale language model 116 returns an item value with the typographical error corrected.

[0096] For example, instruction 911 in FIG. 9(b) is an instruction that, if there is a typographical error in "XY2 Corporation," causes the large-scale language model 116 to respond with a string corresponding to the "company name" field with the typographical error corrected. Response 921 from the large-scale language model 116 includes "XYZ Corporation" with the typographical error corrected for "XY2 Corporation." Similarly, response 922 from the large-scale language model 116 corresponding to instruction 912 also includes the field value with the typographical error corrected. Instruction 913 for "total amount" does not have a typographical error in the field value, so response 923 from the large-scale language model 116 to instruction 913 does not include the corrected field value.

[0097] In this way, OCR processing can erroneously recognize character strings in document images. In this case, when field value extraction is performed, the erroneously recognized character string is extracted as the field value. This requires the user to visually check whether the field values ​​extracted by the field value extraction process are correct, and to correct any errors that may be present.

[0098] Characters that are erroneously recognized often have similar shapes, such as 0 (zero) and O (oh), 1 (ichi), l (el) and I (eye). This may make it difficult for the user to visually check and correct the characters. For the sake of convenience, the present embodiment is described assuming that there are three items to be extracted, but the burden on the user increases as the number of items increases. Therefore, in the present embodiment, the large-scale language model 116 is made to respond as to whether the item values ​​extracted in the item value extraction process are correct, and a warning is given to the user if the extracted item values ​​are not appropriate. Details of warning the user will be described later.

[0099] Note that the instruction statement template 900 in FIG. 9(a) is just an example, and the content of the instruction in the instruction statement format is not limited as long as the instruction statement causes a typo included in an item value to be corrected.

[0100] 8 ends, and the process of S502 ends. Then, the process proceeds to S503 in the flowchart of FIG.

[0101] In S503, the CPU 261 performs a process of notifying the user of the item values ​​contained in the document image to be processed that are extracted in the item value extraction process in S501.

[0102] 10 is a diagram showing an item value confirmation screen 1000, which is an example of a screen for presenting item values ​​extracted from a document image to be processed to a user. In this embodiment, the item value confirmation screen 1000 is described as being displayed on the display device 210 of the image forming apparatus 101. For this reason, in S503, the CPU 261 transmits information for displaying the item value confirmation screen 1000 to the image forming apparatus 101. The item value confirmation screen 1000 may be displayed on the display device 267 of the information processing server 103. A preview display area 1010 of the item value confirmation screen 1000 displays a preview of the document image to be processed.

[0103] An item value confirmation screen 1000 in FIG. 10 shows an example of a confirmation screen that is displayed when the document image 700 in FIG. 7(a) is the processing target and the information shown in the table 710 is obtained as a result of S501.

[0104] The item value confirmation screen 1000 includes item value display areas 1001 to 1003 corresponding to the items to be extracted in this embodiment, namely, "date", "company name", and "total amount". The item value display area 1001 is an item value display area corresponding to the "date" item, and displays the item value of "date" extracted by the item value extraction process of S501 by default. As shown in the table 710, the item value of "date" extracted by the item value extraction process is "February 7, 2023". Therefore, the CPU 261 performs display control so that "February 7, 2023" is displayed in the item value display area 1001. Similarly, the item value display area 1002 corresponding to the "company name" item displays the extracted item value "XY2 Corporation" by default. The item value display area 1003 corresponding to the "total amount" item displays the extracted item value "27,500 yen" by default.

[0105] When a corrected item value is obtained as a response to the instruction from the large-scale language model 116 in S502, the item value described in the instruction may contain an error, so the CPU 261 notifies the user that the item value displayed in the item value display area of ​​the corresponding item may be incorrect.

[0106] For example, the answer 922 of the large-scale language model 116 to the instruction 912 of the "Date" item in FIG. 9(c) includes the corrected item value. Also, the answer 921 of the large-scale language model 116 to the instruction 911 of the "Company Name" item in FIG. 9(b) also includes the corrected item value. Therefore, the item values ​​corresponding to the "Date" and "Company Name" items obtained by the item value extraction process in S501 may contain typographical errors. Therefore, the user is notified that the item values ​​corresponding to the "Date" and "Company Name" items may contain typographical errors.

[0107] For example, the CPU 261 displays an alert below the item value display area of ​​an item that may contain a clerical error. Alert 1017 is an alert that notifies the user that the item value of the "date" item may be incorrect. Alert 1018 is an alert that notifies the user that the item value of the "company name" item may be incorrect. The corrected item value answered by the large-scale language model 116 is incorporated into the alerts 1017 and 1018, and is displayed as a candidate when the user corrects the item value.

[0108] By displaying alerts 1017 and 1018 in this way, it is possible to notify the user that the item values ​​displayed in item value display areas 1001 and 1002 may contain typographical errors. In addition, it is possible to notify the user of the character string answered by the large-scale language model 116 as a candidate for the item value after correction. As there is no typographical error in the item value of "Total Amount", the character string answered by the large-scale language model 116 is not displayed.

[0109] In this way, the item value confirmation screen 1000 displays the difference between the character string obtained by the item value extraction process and the character string returned by the large-scale language model 116 so that the difference can be seen. Also, to emphasize the difference, characters with differences, such as the character "I" and the character "1" in the date, may be highlighted or displayed in bold.

[0110] In addition, if there is a clerical error in the item value obtained by the item value extraction process, the large-scale language model 116 may respond with multiple candidates as a character string after correction. For example, in S803, the CPU 261 generates an instruction statement including an instruction such as "Please list all the corrected character strings that are considered appropriate" as an instruction statement for the "date" item. Then, in S805, as an answer of the large-scale language model 116 to the instruction statement, two character strings, "November 7, 2023" and "February 7, 2023", are obtained as candidates for the item value of "date" after the clerical error is corrected. In this case, the CPU 261 may perform control so that "November 7, 2023" and "February 7, 2023" are displayed on the item value confirmation screen 1000 as candidates for the item value corresponding to the "date" item.

[0111] For example, as shown in the item value confirmation screen 1000 of FIG. 10(b), a drop-down list is displayed below the item value display area 1001 of "date". "November 7, 2023" and "February 7, 2023", which are candidates for the corrected item value answered by the large-scale language model 116, are displayed in cells 1042 and 1043 included in the drop-down list. Alternatively, as shown in FIG. 10(c), buttons 1052 and 1053 corresponding to the candidates for the item value answered by the large-scale language model 116 may be displayed side by side below the item value display area 1001. Even if multiple candidates are answered as the corrected item value from the large-scale language model 116, an alert 1016 is displayed indicating that the item value obtained as a result of the item value extraction process may contain a typo.

[0112] S502 is a process for correcting a typographical error in a character string in the large-scale language model 116. Therefore, the process of S502 may be performed when there is a possibility that an item value obtained as a result of the item value extraction process of S501 includes a typographical error.

[0113] Fig. 5(b) is a flowchart in which the step of S511 is added after S501 in the flowchart in Fig. 5(a). In S511, the CPU 261 determines whether the item value extracted by the item value extraction process in S501 satisfies a predetermined condition. If the CPU 261 determines that the item value satisfies the predetermined condition (YES in S511), the process proceeds to S502. On the other hand, if the CPU 261 determines that the item value does not satisfy the predetermined condition (NO in S511), S502 is skipped and the process proceeds to S503.

[0114] The predetermined condition is a condition indicating that the item value obtained as a result of the item value extraction process is a character string that has been erroneously recognized in the OCR process. For example, in S511, an evaluation value that indicates the accuracy of character recognition, such as the likelihood of the character string of the item value extracted in S501, is obtained, and if there is a character string whose evaluation value is equal to or less than a threshold, it is determined that the predetermined condition is satisfied. For example, the item value of an item such as a date or amount may be determined to satisfy the predetermined condition if it does not match a predetermined data type. Or, if an item value such as a company name does not match a character string in a client database, it may be determined to satisfy the predetermined condition. Or, if the character string of an item value has no regularity in the arrangement of character types, such as a mixture of English letters among numbers, it may be determined to satisfy the predetermined condition. In S502, an instruction sentence may be generated only for an item corresponding to an item value determined to satisfy the predetermined condition.

[0115] [About modifying item values] When a user modifies an item value extracted from a document image to be processed, the user may manually input the modified item value. Alternatively, the user may select the modified item value from the candidates of item values ​​answered by the large-scale language model 116 displayed on an item value confirmation screen 1000 in Fig. 10. A method for a user to modify an item value will be described with reference to Fig. 10. The item value confirmation screen 1000 will be described as being displayed on the display device 210 of the image forming apparatus 101 under the control of the CPU 261 of the information processing server 103, and a user operation will be acquired.

[0116] When the user manually modifies the item value, the user presses edit button 1012, which changes the item value display area 1001 to an editable state such as a text box. The user can manually modify the text indicating the item value displayed in the item value display area 1001 using the input cursor.

[0117] In addition, when the user's press of the Yes button 1037 is detected on the item value confirmation screen 1000 of FIG. 10(a), the display of the item value display area 1001 is changed to "November 7, 2023" answered by the large-scale language model 116. In addition, "II 7, 2023" displayed in the item value display area 1001 is saved in the storage unit of the information processing server 103. When the user's press of the No button 1038 is detected, the previously displayed "II 7, 2023" is acquired from the storage unit of the information processing server 103, and "II 7, 2023" is displayed in the item value display area 1001. Note that if the No button 1038 is pressed before the Yes button 1037 is pressed, no operation may be performed.

[0118] Next, a method for correcting an item value when multiple item value candidates are returned by the large-scale language model 116 will be described with reference to FIG. 10(b) and FIG. 10(c).

[0119] When it is detected that the list display button 1015 on the item value confirmation screen 1000 in FIG. 10(b) has been pressed by the user, a drop-down list containing item value candidates proposed by the large-scale language model 116 is displayed. For example, assume that the user selects cell 1042 in which the item value candidate "November 7, 2023" is displayed. In this case, the selection of "November 7, 2023" is detected, and the display in the item value display area 1001 is changed to "November 7, 2023", and "February 7, 2023" displayed in the item value display area 1001 is overwritten so that it is displayed as a candidate item value.

[0120] An item value confirmation screen 1000 in Fig. 10(c) is a screen in which the drop-down list for displaying item value candidates in Fig. 10(b) has been changed to buttons 1052 and 1053. In the case of Fig. 10(c), the user can view item value candidates proposed by the large-scale language model 116 without pressing the list display button 1015. The operation in Fig. 10(c) is the same as the operation in Fig. 10(b) except that the cells of the drop-down list are replaced with buttons.

[0121] 10(a)-(c), the character strings displayed in the item value display areas 1001-1003 are confirmed as item values ​​included in the document image to be processed. The confirmed item values ​​are managed by, for example, the data management unit 157 of the information processing server 103.

[0122] As described above, in this embodiment, even if a character string erroneously recognized in OCR processing is extracted as an item value, it is possible to have the large-scale language model return an appropriate item value. According to this embodiment, since candidates for corrected item values ​​can be suggested to the user based on the response of the large-scale language model, the user's confirmation effort is reduced, and the user's effort to manually input the correct item value is suppressed. In addition, in this embodiment, since an instruction sentence corresponding to each item to be extracted is generated, it is possible to suppress the large-scale language model from returning an inappropriate character string as a candidate for the corrected item value.

[0123] The instruction may include all character strings obtained as a result of OCR processing of the document image to be processed, and may be an instruction to correct errors in the item values ​​for each item while taking into account the context of the character strings.

[0124] <Embodiment 2> In the first embodiment, a method for making the large-scale language model 116 return a corrected character string for a character string that was erroneously recognized in OCR processing was described. In this embodiment, a method for making the large-scale language model 116 return an item value for an item that was erroneously extracted or not extracted in the item value extraction processing will be described. This embodiment will be described focusing on the differences from the first embodiment. Portions that are not specifically stated have the same configuration and processing as the first embodiment.

[0125] [Item value display processing] FIG. 11 is a flowchart for explaining a process for extracting and displaying item values ​​from a document image, and is a flowchart of this embodiment corresponding to the flowchart of FIG. 5 in the first embodiment.

[0126] S1101 is an item value extraction process similar to S501, in which item values ​​are extracted from the document image to be processed based on the output result of the item value extractor 115, which is a machine learning model. In this embodiment, the items to be extracted are also described as "date", "company name", and "total amount". A detailed description will be omitted.

[0127] In S1102, the CPU 261 determines whether there are any unextracted items whose item values ​​could not be extracted or any items whose item values ​​were erroneously extracted among the items to be extracted in the item value extraction process of S501. If the CPU 261 determines that there are any unextracted or erroneously extracted items (YES in S1102), the process proceeds to S1103. If the CPU 261 determines that there are no unextracted or erroneously extracted items (NO in S1102), the process skips S1103 and proceeds to S1104.

[0128] As a method of determining whether an item has been erroneously extracted, for example, for items such as date or amount, if the extracted item value does not match a predetermined data type, it is determined that an item has been erroneously extracted. Also, for items such as company name, if the extracted item value does not match any of the character strings representing company names stored in the client database, it is determined that an item has been erroneously extracted. In S1102, if even one item out of multiple items has not been extracted or has been erroneously extracted, it is determined as YES.

[0129] S1103 is a step corresponding to S502 in FIG. 5, where the CPU 261 generates an instruction sentence and obtains a response to the instruction sentence from the large-scale language model 116.

[0130] FIG. 12 is a flowchart for explaining the details of the process of S1103, and is a flowchart of this embodiment that corresponds to the flowchart of FIG. 8 of the first embodiment.

[0131] In S1201, the CPU 261 acquires the item names of the items that were determined to have not been extracted or were erroneously extracted in S1102. If the item names of the items to be extracted are "date", "company name", and "total amount", at least one of these item names is acquired.

[0132] In S1202, a group of character strings obtained as a result of OCR processing is obtained from the document image to be processed.

[0133] 13(a) is a diagram showing a document image 1300, which is an example of an image to be processed. Dotted frames indicate character string areas 1301-1316 included in the document image 1300. As a result of OCR processing on the document image 1300, recognized character strings are obtained from each of the character string areas 1301-1316. A unique value for identifying each character string area is assigned to each character string area as an area ID.

[0134] 13(b) summarizes in table form the results of OCR processing and the results of item value extraction processing for document image 1300. Column 1321 in table 1320 holds area IDs indicating character string areas in document image 1300. Column 1322 holds character strings obtained as a result of OCR processing from the character string areas indicated by the area IDs. Column 1323 holds item names indicating the items corresponding to the character strings held in column 1322.

[0135] 13(b) does not include an item name indicating the "total amount" item. For example, the document called "donation receipt" shown in document image 1300 is a document that item value extractor 115 has not learned, and therefore it is considered that item value extractor 115 was unable to determine that the character string written next to the donation amount is the total amount. For this reason, it is shown that the item value of the "total amount" item has not been extracted.

[0136] In addition, the character string associated with the item name of "Date" is "2023", which indicates that "2023" was extracted as the item value of "Date" in the item value extraction process. The data type of "Date" is defined as a type consisting of year, month, and day, and the extracted item value does not correspond to that data type. Therefore, the item value of "Date" is an item that is determined to be erroneously extracted. In the document image 1300, the space between the character string "2003" representing the year and the character string "June" representing the month is wider than usual. For this reason, the character string that should be recognized as one character string is recognized as multiple character strings, and it is considered that the item value extractor 115 was unable to properly extract the item value of "Date".

[0137] 13(b) is obtained as a result of the item value extraction process in S1101, "total amount" and "date" are obtained as the item names that have been determined to have not been extracted or have been erroneously extracted in S1201. Also, in S1202, character strings stored in column 1322 of table 1320 are obtained as a group of character strings obtained as a result of OCR processing.

[0138] In S1203, the CPU 261 acquires the instruction statement format of this embodiment from the storage 265.

[0139] In S1204, the CPU 261 generates an instruction statement by applying the item name acquired in S1201 and the group of character strings acquired in S1202 to the instruction statement template acquired in S1203.

[0140] Fig. 14A is a diagram showing an example of a command statement template of this embodiment acquired in S1203. A command statement template 1400 in Fig. 14A includes an item name region 1401 and a character string group region 1402.

[0141] The CPU 261 applies the item names that have been determined to have not been extracted or have been erroneously extracted in S1204 to an item name area 1401 of the instruction statement template 1400. In addition, the CPU 261 applies a group of character strings recognized as a result of OCR processing of the document image to a character string group area 1402. For example, the character strings are applied in the order in which they were recognized. In this manner, the instruction statement 1410 in FIG. 14(b) is generated.

[0142] In S1205, the CPU 261 performs a process of inputting the instruction sentence generated in S1204 into the large-scale language model .

[0143] In S1206, the CPU 261 receives from the large-scale language model 116 a response to the instruction sentence input in S1205.

[0144] For example, the instruction 1410 in Fig. 14(b) includes an instruction to answer an item value corresponding to an item determined to have not been extracted or to have been erroneously extracted from a group of character strings recognized from a document image. The answer 1411 of the large-scale language model 116 to the instruction 1410 indicates that "June 2, 2023" has been answered as the character string for the "Date" item, and "¥13,000" has been answered as the character string for the "Total Amount" item.

[0145] The CPU 261 may generate an instruction sentence by switching the contents depending on the language or destination of the input document image. The instruction sentence template may be one prepared in advance by an engineer or a user, or may be one prepared in advance and edited by the user.

[0146] 12 ends, and the process of S1103 ends. Then, the process proceeds to S1104 in the flowchart of FIG.

[0147] In S1104, the CPU 261 performs a process of notifying the user of the item values ​​contained in the document image to be processed, which are extracted by the item value extraction process in S1101.

[0148] Fig. 15 is a diagram showing an item value confirmation screen 1000 of this embodiment, which is a screen for presenting the extracted item values ​​to the user. The configuration of the item value confirmation screen 1000 of this embodiment is similar to that of the item value confirmation screen 1000 of Fig. 10. In S1104, the CPU 261 transmits information for displaying the item value confirmation screen 1000 to the image forming apparatus 101.

[0149] In the item value display area 1001 of FIG. 15, the item value of "Date" extracted by the item value extraction process is displayed by default. "Date" is an item determined to have been erroneously extracted. For this reason, an alert 1517 is displayed indicating that the item value displayed in the item value display area 1001 of "Date" may be incorrect. The alert 1517 displays an alert so that the character string answered by the large-scale language model 116 is incorporated as a correction candidate. When the press of the Yes button 1537 is detected, the display in the item value display area 1001 is changed to "June 2, 2023" answered by the large-scale language model 116.

[0150] Since the "Total Amount" item is an item that has been determined to have not been extracted, the item value is not displayed on the item value confirmation screen 1000 in Fig. 15. Therefore, for the "Total Amount" item, an alert 1518 is displayed in which the character string answered by the large-scale language model 116 is incorporated as a correction candidate. When pressing of the Yes button 1538 is detected, an item value display area is displayed, and the answer "¥13,000" from the large-scale language model 116 is displayed in the item value display area.

[0151] As described above, in this embodiment, even if an item value is not extracted or is erroneously extracted in an item value extraction process using a trained model, etc., it is possible to have the large-scale language model return a corrected item value. Therefore, according to this embodiment, since item value candidates can be suggested to the user based on the response of the large-scale language model, the user's confirmation effort is reduced and the user is prevented from manually inputting the correct item value.

[0152] <Embodiment 3> In the above-described embodiment, a method of using one large-scale language model has been described. In this embodiment, a method of using one or more large-scale language models set by an engineer or a user will be described. This embodiment will be described focusing on the differences from embodiment 1 or embodiment 2. Parts that are not specifically stated have the same configuration and processing as embodiment 1 or embodiment 2.

[0153] [Item value extraction process sequence] Fig. 16 is a sequence diagram of this embodiment corresponding to the sequence of Fig. 3(b). The sequence during development of the machine learning model is omitted because it is similar to Fig. 3(a).

[0154] In S1611, the user selects a large-scale language model into which a command is to be input, and sets the display method on the item value confirmation screen 1000. The information processing server 103 acquires the contents set by the user in S1611.

[0155] S1612 to S1619 are the same processes as S311 to S318, and the description will be omitted. However, the processes of S1617 and S1618 are different from S316 and S317. In S1617, a command statement is input to the large-scale language model selected by the user in S1611, and in S1618, a response to the command statement is received from the selected large-scale language model. In addition, in S1619, an item value confirmation screen 1000 is displayed based on the user's settings in S1611.

[0156] [About the large-scale language model setting screen] 17 is a diagram showing a large-scale language model setting screen 1700, which is an example of a screen for selecting a large-scale language model to which an instruction statement is to be input in S1611. The large-scale language model setting screen 1700 will be described as being displayed on the display device 210 of the image forming apparatus 101 under the control of the CPU 261 of the information processing server 103. The CPU 261 also functions as a setting unit that sets a large-scale language model to which an instruction statement is to be input. In this embodiment, an instruction statement is input to each large-scale language model selected by the user, and a response to the instruction statement is received from each large-scale language model to which the instruction statement was input.

[0157] A model display area 1731 on a large-scale language model setting screen 1700 in Fig. 17 displays a large-scale language model selected as an input destination of an instruction. Fig. 17 shows that "large-scale language model A" displayed in the model display area 1731 is selected as an input destination of an instruction. When a user wants to change the large-scale language model to which an instruction is to be input, the user presses a list display button 1733 included in the model display area 1731. When the list display button 1733 is pressed, a drop-down list including large-scale language models that the user can select is displayed. When the user selects a large-scale language model from the drop-down list, the name of the selected large-scale language model is displayed in the model display area 1731.

[0158] When inputting a command to multiple large-scale language models and obtaining answers from multiple large-scale language models, pressing the Add button 1735 adds a new model display area 1732, as shown in Fig. 17. When, for example, "large-scale language model B" is selected from the drop-down list 1734 displayed by pressing the list display button 1733 included in the model display area 1732, the selected "large-scale language model B" is displayed in the model display area 1732. Similarly, pressing the Add button 1735 can add an input field for a third large-scale language model.

[0159] In this state, when the user presses the save button 1720, the "large-scale language model A" and the "large-scale language model B" displayed in the model display areas 1731 and 1732 are set as the large-scale language models to which the instruction sentence is to be input. Information indicating the selected large-scale language model is transmitted to the information processing server 103.

[0160] 16 shows a flow in the case where the user selects multiple large-scale language models as input destinations of an instruction message in S1611. The external information processing server 105a indicates a server that operates "large-scale language model A", and the external information processing server 105b indicates a server that operates "large-scale language model B". When the user selects "large-scale language model A" and "large-scale language model B" as input destinations of an instruction message in S1611, an instruction message is sent to each large-scale language model in order to input the instruction message to the respective large-scale language models in S1617. Then, an answer is received from each large-scale language model in S1618.

[0161] [About the Text Editing Options Setting Screen] FIG. 18 is a diagram showing a setting screen 1800 for setting a character string correction option, which is a screen for the user to set the display method on the item value confirmation screen 1000 in S1611.

[0162] In the above-described embodiment, the character string obtained as an answer from the large-scale language model is output and displayed as a candidate character string when the user corrects the item value. This display is called a candidate display. In this embodiment, the user can set, via the setting screen 1800 for each item, which large-scale language model among the large-scale language models selected on the large-scale language model setting screen 1700 to display as an answer candidate.

[0163] In addition, in this embodiment, the item value obtained by the item value extraction process is automatically corrected to the character string answered by the large-scale language model, and the character string answered by the large-scale language model can be output as the item value. That is, the character string answered by the large-scale language model can be displayed by default in the item value display area of ​​the item value confirmation screen 1000. This process is called automatic correction, and it is possible to set for each item via the setting screen 1800 which large-scale language model's answer the automatic correction will be based on. The CPU 261 also functions as a setting unit that sets whether or not to automatically correct the item value obtained by the item value extraction process.

[0164] The setting screen 1800 includes a table for setting automatic correction and candidate display. A column 1801 displays a correction means for outputting a candidate character string for correcting an item value obtained by an item value extraction process. For example, the large-scale language model selected on the large-scale language model setting screen 1700 is displayed as the correction means. The correction means may include a correction rule for outputting a candidate character string by making a predetermined judgment. Therefore, if there is a correction rule, it is also displayed as a correction means in the column 1801.

[0165] A column 1802 holds the names of items to be automatically corrected or candidates to be displayed.

[0166] A column 1803 holds "enabled" for the item held in the column 1802 when the above-mentioned automatic correction is enabled.

[0167] A column 1804 holds "enabled" for the item held in the column 1802 when the candidate display is enabled.

[0168] For example, there are rows 1813 and 1814 in which "company name" is stored in column 1802 of the table in the setting screen 1800. Row 1813 is a row for setting how to output the answer of the large-scale language model A, and automatic correction is enabled in column 1803 of row 1813. Row 1814 is a row for setting how to output the answer of the large-scale language model B, and candidate display is enabled in column 1804 of row 1814. Therefore, in FIG. 18, the character string answered by the large-scale language model A is set to be output so as to be displayed by default in the item value display area 1002 of the item value confirmation screen 1000 as the item value of the "company name" item. Also, the answer of the large-scale language model B is set to be output so as to be displayed as a character string of a correction candidate.

[0169] [Processing to obtain answers from large-scale language models] Fig. 19 is a flowchart for explaining the details of the process of acquiring an answer from a large-scale language model. Fig. 19 is a flowchart of this embodiment corresponding to the flowchart of Fig. 8 of the first embodiment. Or, Fig. 19 is a flowchart of this embodiment corresponding to the flowchart of Fig. 12 of the second embodiment.

[0170] In S1901, the CPU 261 acquires setting information indicating the large-scale language model selected by the user in S1611. For example, if the user selects "large-scale language model A" and "large-scale language model B" as input destinations in S1611, information indicating "large-scale language model A" and "large-scale language model B" is acquired.

[0171] In S1902, a command statement generation process is performed. For example, when the command statement 911 of the first embodiment is generated, the same processes as S801 to S803 in Fig. 8 are performed. When the command statement 1410 of the second embodiment is generated, the same processes as S1201 to S1204 in Fig. 12 are performed.

[0172] The next steps S1903 to S1906 are loop processes. In S1903, the CPU 261 selects a large-scale language model to be processed from among the large-scale language models indicated by the setting information acquired in S1901. In S1906, the CPU 261 determines whether processing has been performed for all large-scale language models indicated by the setting information. If processing has not been performed for all large-scale language models, the process returns to S1903 and selects the next processing target from the unprocessed large-scale language models.

[0173] In S1904, the CPU 261 inputs the instruction statement generated in S1902 to the large-scale language model to be processed selected in S1903.

[0174] In S1905, the CPU 261 receives a response to the instruction sentence from the large-scale language model to be processed.

[0175] In S1906, if the processing has been completed for all large-scale language models set by the user, the flowchart in Fig. 19 ends. For example, if the user selects "large-scale language model A" and "large-scale language model B" in S1611, the CPU 261 executes the processing of S1904 to S1905 for each of "large-scale language model A" and "large-scale language model B." That is, the CPU 261 executes the processing of S1904 to S1905 twice.

[0176] Then, when the instruction statements 911 to 913 of the first embodiment are generated, the process proceeds to S503 in Fig. 5. When the instruction statement 1410 of the second embodiment is generated, the process proceeds to S1104 in Fig. 11. Then, in each step, a process of displaying the item value confirmation screen 1000 is performed.

[0177] [Item value display] Fig. 20 is a diagram showing an item value confirmation screen 1000 of this embodiment, and the configuration of the screen is the same as that of the item value confirmation screen 1000 of Fig. 10. The CPU 261 transmits information for displaying the item value confirmation screen 1000 of Fig. 20 to the image forming apparatus 101, whereby the item value confirmation screen 1000 of Fig. 20 is displayed.

[0178] For example, it is assumed that the "company name" item is set to be automatically corrected based on the response from the large-scale language model A. As described in the first embodiment, it is assumed that the item value of "company name" obtained as a result of the item value extraction process is "XY2 Co., Ltd." Furthermore, it is assumed that the item value of "company name" responded by the large-scale language model A is "XYZ Co., Ltd." In this case, as shown in the item value confirmation screen 1000 of FIG. 20(a), "XYZ Co., Ltd." responded by the large-scale language model A is displayed by default in the item value display area 1002 of the item value confirmation screen 1000.

[0179] In addition, when automatic correction is enabled in multiple correction means, if the character strings output by the multiple correction means match, automatic correction may be performed with the matching character string, and the matching character string may be displayed by default in the item value display area 1002. For example, assume that automatic correction is also enabled for the large-scale language model B, and the item value of "company name" answered by the large-scale language model B is also "XYZ Corporation." In this case, "XYZ Corporation" answered by the large-scale language model A and the large-scale language model B may be displayed by default in the item value display area 1002. If there is no match, for example, the character strings output by each correction means may be displayed as candidates.

[0180] The item value confirmation screen 1000 in FIG. 20(b) is a diagram showing the display of the item value of the "Date" item. For example, assume that automatic correction is not enabled for the "Date" item, and that settings are made to display candidates based on the output of large-scale language model A, large-scale language model B, and correction rule 1. Therefore, the item value display area 1001 displays, by default, "2023," which is the item value obtained as a result of the item value extraction process.

[0181] For example, assume that "June 2, 2023" is obtained as a response to the instruction sentence of the large-scale language model A and the large-scale language model B. Also assume that "June 2" is obtained by correction rule 1. If the character strings output by the multiple correction means with the candidate display enabled are different, a list of the output character strings may be displayed, and correction may be performed by the character string selected by the user.

[0182] As a method of displaying candidates in a list, for example, a drop-down list showing candidates for item values ​​is displayed below the item value display area 1001, and the character string output by the correction means is displayed in the drop-down list. The drop-down list may also include information indicating the large-scale language model or correction rule that output the character string.

[0183] In text 2042 for displaying "June 2, 2023" in the drop-down list, "Large-scale language model A / B," indicating the large-scale language model A and large-scale language model B that responded, are displayed in a format attached to "June 2, 2023."

[0184] Also, a string with a large number of correction means that have answered the same string may be displayed preferentially. For example, "June 2, 2023" is output by two correction means, large-scale language model A and large-scale language model B. "June 2" is output by only one correction means, correction rule 1. For this reason, "June 2, 2023" is displayed at the top of the drop-down list so that it is displayed preferentially.

[0185] As described above, according to this embodiment, it is possible to have item values ​​answered from multiple large-scale language models. Therefore, it is possible to increase the possibility that an appropriate item value is answered. Furthermore, according to this embodiment, it is possible to automatically correct the item value obtained by the item value extraction process based on the item value answered by the large-scale language model. Therefore, it is possible to reduce the burden on the user of correcting the item value.

[0186] <Embodiment 4> In the second embodiment, a method for obtaining answers from the large-scale language model 116 for items that have been erroneously extracted or not extracted in the item value extraction process using the item value extractor 115 has been described. In the present embodiment, a method for obtaining answers for item values ​​only from the large-scale language model 116 will be described. This embodiment will be described focusing on the differences from the second embodiment. Portions that are not specifically stated have the same configuration and processing as the first embodiment.

[0187] [About the configuration of information processing systems] Fig. 20 is a diagram showing an example of the configuration of an information processing system according to this embodiment, which corresponds to Fig. 1. As shown in Fig. 20, in this embodiment, an item value extractor 115 is not generated. Therefore, the information processing system 100 does not include a learning device 102.

[0188] [Item value extraction process sequence] Fig. 22 is a sequence diagram of this embodiment corresponding to the sequence of Fig. 3(b). Since a machine learning model is not used, there is no diagram of this embodiment corresponding to Fig. 3(a). S2211 to S2213 are similar processes to S311 to S313, and S2214 to S2217 are similar processes to S315 to S318. In this embodiment, since the item value extractor 115 is not generated, there is no step of extracting item values ​​based on the output result of the item value extractor 115.

[0189] [Item value display processing] FIG. 23 is a flowchart for explaining the process of extracting and displaying item values ​​contained in a document image, and is a flowchart of this embodiment corresponding to the flowchart of FIG. 11 in the second embodiment.

[0190] In S2301, the CPU 261 acquires the document image transmitted from the image forming apparatus 101. The document image acquired in S2301 becomes the document image to be processed.

[0191] In S2302, the CPU 261 executes block selection (BS) processing and OCR processing on the document image to be processed, and acquires a group of character strings recognized from the document image to be processed.

[0192] S2303 is a step similar to S1103 in Fig. 11, where the CPU 261 generates an instruction sentence and obtains a response to the instruction sentence as an output result from the large-scale language model 116. In S2303, the same process as in the flowchart in Fig. 12 is performed.

[0193] 12, the CPU 261 acquires all item names of items to be extracted. Then, in S1204, the CPU 261 assigns the item names of the items to be extracted acquired in S1201 to item name areas 1401 of the instruction statement template 1400. For example, if the items to be extracted are "date", "company name", and "total amount", the CPU 261 assigns "date", "company name", and "total amount" to item name areas 1401 of the instruction statement template 1400.

[0194] FIG. 24 is a diagram showing an example of an instruction statement of this embodiment. Unlike the instruction statement 1410 of the second embodiment, the instruction statement 2410 of FIG. 24 includes an instruction to answer item values ​​corresponding to all items to be extracted. The answer 2411 of the large-scale language model 116 to the instruction statement 2410 includes "June 2, 2023" as a character string in the "Date" field and "¥13,000" as a character string in the "Total Amount" field. Furthermore, it is shown that "XYZ Corporation" has been answered as a character string in the "Company Name" field.

[0195] In S2304, the CPU 261 performs processing to notify the user of the item values ​​included in the document image to be processed and answered by the large-scale language model in S2303. In this embodiment, the item values ​​answered by the large-scale language model 116 may be displayed by default in the item value display areas 1001 to 1003 of the item value confirmation screen 1000.

[0196] As described above, in this embodiment, the large-scale language model 116 can be made to respond with item values ​​contained in a document image to be processed. Therefore, according to this embodiment, the burden of generating a machine learning model is reduced.

[0197] <Other embodiments> Note that a document type indicated by a document image to be processed may be determined, and an instruction for answering an item value taking the determined document type into consideration may be generated.

[0198] The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0199] The disclosure of the above-described embodiment includes the following configurations.

[0200] (Configuration 1) character recognition means for performing character recognition processing on an image of a document to be processed; a generating means for generating an instruction sentence for causing a large-scale language model to respond with a character string corresponding to a predetermined item included in the document based on a result of the character recognition process; a sending means for sending the instruction to obtain a response to the instruction from the large-scale language model; a receiving means for receiving a response to the instruction sentence by the large-scale language model; 13. An information processing device comprising:

[0201] (Configuration 2) The character recognition processing further includes an extraction unit for extracting a character string corresponding to the predetermined item from a group of character strings obtained as a result of the character recognition processing. 2. The information processing device according to configuration 1.

[0202] (Configuration 3) The extraction means extracts a character string corresponding to the predetermined item from the character string group based on a result obtained by inputting the character string group obtained as a result of the character recognition processing into a trained model. 3. The information processing device according to configuration 2.

[0203] (Configuration 4) The generating means generates the instruction sentence including the predetermined item and a character string extracted by the extracting means as a character string corresponding to the predetermined item. 4. The information processing device according to configuration 2 or 3.

[0204] (Configuration 5) The generating means generates an instruction sentence for causing the large-scale language model to respond as to whether or not there is a typographical error in the character string extracted by the extracting means as the character string corresponding to the predetermined item. 5. The information processing device according to configuration 4.

[0205] (Configuration 6) The generating means generates the instruction sentence corresponding to the predetermined item when the character string extracted by the extracting means as the character string corresponding to the predetermined item is a character misrecognized in the character recognition process. 6. The information processing device according to configuration 4 or 5.

[0206] (Configuration 7) The extraction means extracts, from the group of character strings, an item value that is a character string corresponding to an item to be extracted, including the predetermined item; The predetermined item is an item from among the items to be extracted, for which the extraction means was unable to extract an item value, or an item for which the extraction means erroneously extracted an item value. 3. The information processing device according to configuration 2.

[0207] (Configuration 8) The generating means includes: When the predetermined item is present, the instruction sentence is generated to cause the large-scale language model to respond with a character string corresponding to the predetermined item from a group of character strings obtained as a result of the character recognition process. 8. The information processing device according to configuration 7.

[0208] (Configuration 9) The generating means generates the instruction sentence including the predetermined items and a group of character strings obtained as a result of the character recognition processing. 2. The information processing device according to configuration 1.

[0209] (Configuration 10) The generating means generates the instruction sentence for causing the large-scale language model to return a character string corresponding to the predetermined item from a group of character strings obtained as a result of the character recognition process. 10. The information processing device according to configuration 9.

[0210] (Configuration 11) The present invention further includes a display control means for displaying, on a display unit, at least one of the character string extracted by the extraction means as a character string corresponding to the predetermined item and the character string corresponding to the predetermined item obtained by the large-scale language model providing a response to the instruction sentence. 9. The information processing device according to any one of configurations 2 to 8.

[0211] (Configuration 12) The display control means A difference between a character string extracted by the extraction means as a character string corresponding to the predetermined item and a character string corresponding to the predetermined item obtained by the large-scale language model as a response to the instruction sentence is highlighted and displayed. 12. The information processing device according to configuration 11.

[0212] (Configuration 13) The system further includes a setting unit for setting a plurality of large-scale language models that respond to the instruction sentence, The transmitting means transmits the instruction sentence to the set plurality of large-scale language models. 13. The information processing device according to any one of configurations 1 to 12.

[0213] (Configuration 14) The method further includes a plurality of correction means for outputting candidate character strings for correcting the character string extracted by the extraction means as the character string corresponding to the predetermined item, At least one of the plurality of correction means outputs, as the candidate character string, a character string corresponding to the predetermined item answered by the large-scale language model in response to the instruction sentence. 9. The information processing device according to any one of configurations 2 to 8.

[0214] (Configuration 15) The method further includes display control means for displaying, on a display unit, the matched character string as a character string corresponding to the predetermined item when the character strings of the candidates output by the plurality of correction means match. 15. The information processing device according to configuration 14.

[0215] (Configuration 16) The method further includes a display control unit that displays a list of the candidate character strings output by the plurality of correction units on a display unit. 15. The information processing device according to configuration 14.

[0216] (Configuration 17) The method further includes a display control means for displaying on a display unit a matching character string among the candidate character strings output by the plurality of correction means. 15. The information processing device according to configuration 14.

[0217] (Configuration 18) The apparatus further includes a setting means for setting whether or not to replace the character string extracted by the extraction means as a character string corresponding to the predetermined item with a character string obtained by the large-scale language model providing a response to the instruction sentence and outputting the replaced character string. 9. The information processing device according to any one of configurations 2 to 8.

[0218] (Configuration 19) a character recognition step of performing character recognition processing on an image of a document to be processed; a generating step of generating an instruction sentence for causing a large-scale language model to respond with a character string corresponding to a predetermined item included in the document based on a result of the character recognition process; a sending step of sending the instruction to obtain an answer to the instruction from the large-scale language model; receiving a response to the instruction from the large-scale language model; 13. An information processing method comprising:

[0219] (Configuration 20) 19. A program for causing a computer to execute each of the means of the information processing device according to any one of configurations 1 to 18. [Explanation of symbols]

[0220] 103 Information Processing Server 154 Document Image Analysis Unit 155 Directive sentence generation section 156 Reply Receiving Department

Claims

1. character recognition means for performing character recognition processing on an image of a document to be processed; a generating means for generating an instruction sentence for causing a large-scale language model to respond with a character string corresponding to a predetermined item included in the document based on a result of the character recognition process; a sending means for sending the instruction to obtain a response to the instruction from the large-scale language model; a receiving means for receiving a response to the instruction sentence by the large-scale language model; 13. An information processing device comprising:

2. The character recognition processing further includes an extraction unit for extracting a character string corresponding to the predetermined item from a group of character strings obtained as a result of the character recognition processing.

2. The information processing apparatus according to claim 1,

3. The extraction means extracts a character string corresponding to the predetermined item from the character string group based on a result obtained by inputting the character string group obtained as a result of the character recognition processing into a trained model.

3. The information processing apparatus according to claim 2.

4. The generating means generates the instruction sentence including the predetermined item and a character string extracted by the extracting means as a character string corresponding to the predetermined item.

3. The information processing apparatus according to claim 2.

5. The generating means generates an instruction sentence for causing the large-scale language model to respond as to whether or not there is a typographical error in the character string extracted by the extracting means as the character string corresponding to the predetermined item.

5. The information processing apparatus according to claim 4.

6. The generating means generates the instruction sentence corresponding to the predetermined item when the character string extracted by the extracting means as the character string corresponding to the predetermined item is a character misrecognized in the character recognition process.

5. The information processing apparatus according to claim 4.

7. The extraction means extracts, from the group of character strings, an item value that is a character string corresponding to an item to be extracted, including the predetermined item; The predetermined item is an item from among the items to be extracted, for which the extraction means was unable to extract an item value, or an item for which the extraction means erroneously extracted an item value.

3. The information processing apparatus according to claim 2.

8. The generating means includes: When the predetermined item is present, the instruction sentence is generated to cause the large-scale language model to respond with a character string corresponding to the predetermined item from a group of character strings obtained as a result of the character recognition process.

8. The information processing apparatus according to claim 7,

9. The generating means generates the instruction sentence including the predetermined items and a group of character strings obtained as a result of the character recognition processing.

2. The information processing apparatus according to claim 1,

10. The generating means generates the instruction sentence for causing the large-scale language model to return a character string corresponding to the predetermined item from a group of character strings obtained as a result of the character recognition process.

10. The information processing apparatus according to claim 9,

11. The present invention further includes a display control means for displaying, on a display unit, at least one of the character string extracted by the extraction means as a character string corresponding to the predetermined item and the character string corresponding to the predetermined item obtained by the large-scale language model providing a response to the instruction sentence.

3. The information processing apparatus according to claim 2.

12. The display control means A difference between a character string extracted by the extraction means as a character string corresponding to the predetermined item and a character string corresponding to the predetermined item obtained by the large-scale language model as a response to the instruction sentence is highlighted and displayed.

12. The information processing apparatus according to claim 11 .

13. The system further includes a setting unit for setting a plurality of large-scale language models that respond to the instruction sentence, The transmitting means transmits the instruction sentence to the set plurality of large-scale language models.

2. The information processing apparatus according to claim 1,

14. The method further includes a plurality of correction means for outputting candidate character strings for correcting the character string extracted by the extraction means as the character string corresponding to the predetermined item, At least one of the plurality of correction means outputs, as the candidate character string, a character string corresponding to the predetermined item answered by the large-scale language model in response to the instruction sentence.

3. The information processing apparatus according to claim 2.

15. The method further includes display control means for displaying, on a display unit, the matched character string as a character string corresponding to the predetermined item when the character strings of the candidates output by the plurality of correction means match.

15. The information processing apparatus according to claim 14,

16. The method further includes a display control unit that displays a list of the candidate character strings output by the plurality of correction units on a display unit.

15. The information processing apparatus according to claim 14,

17. The method further includes a display control means for displaying on a display unit a matching character string among the candidate character strings output by the plurality of correction means.

15. The information processing apparatus according to claim 14,

18. The apparatus further includes a setting means for setting whether or not to replace the character string extracted by the extraction means as a character string corresponding to the predetermined item with a character string obtained by the large-scale language model providing a response to the instruction sentence and outputting the replaced character string.

3. The information processing apparatus according to claim 2.

19. a character recognition step of performing character recognition processing on an image of a document to be processed; a generating step of generating an instruction sentence for causing a large-scale language model to respond with a character string corresponding to a predetermined item included in the document based on a result of the character recognition process; a sending step of sending the instruction to obtain an answer to the instruction from the large-scale language model; receiving a response to the instruction from the large-scale language model; 13. An information processing method comprising:

20. A program for causing a computer to execute each of the means of the information processing device according to any one of claims 1 to 18.

Images