Information processing apparatus, method for controlling information processing apparatus, program, and information processing system

Abstract

To present character strings in a document image used for acquisition of item values not directly described in the document image.SOLUTION: An information processing apparatus is to extract character strings corresponding to predetermined items from a document image, and comprises: generation means that generates, through inference, a first instruction sentence for presenting a first character string according to the format of item values corresponding to the predetermined items and a second instruction sentence for presenting a second character strings to be the basis for acquiring the first character string; acquisition means that inputs the first instruction sentence and the second instruction sentence generated by the generation means to a large-scale language model to acquire the first character string and the second character string; and output means that outputs both the first character string and the second character string acquired by the acquisition means.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, a control method for an information processing device, a program, and an information processing system for extracting information from a document image. [Background technology]

[0002] There is a technology for extracting character strings (hereinafter referred to as "item values") corresponding to predetermined items (e.g., title, document number, issue date, company name, amount, etc., hereinafter referred to as "item names") written in an input document image. For example, Patent Document 1 discloses a technology for extracting item values ​​corresponding to each item name from a document image and displaying on a UI screen the positions of each combination of the extracted item name and item value on the document image so that the user can see them. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2021-131593 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, when the item name is "Estimated Due Date," there are cases where a specific date is written, such as "Estimated Due Date: October 12, 2023," and cases where no specific date is written, such as "Estimated Due Date: 30 days." Here, when a specific date is not written, the user must calculate the actual estimate due date using information other than "Estimated Due Date: 30 days" in the document image (e.g., notation such as "Issuance Date: October 12, 2023"). When both the item name (key) and its corresponding item value (value) are written in the document image, it is possible to present character strings corresponding to the item name and item value based on the results of key-value extraction. However, when the item value corresponding to the item name is not written in the document image, it is not possible to present the character string in the document image used to calculate the character string corresponding to the item value based on the results of key-value extraction. The present disclosure has been made in light of this issue. [Means for solving the problem]

[0005] The information processing device according to the present disclosure is an information processing device for extracting a character string corresponding to a predetermined item from a document image, and is characterized by comprising: a generation means for generating, by inference, a first instruction sentence for presenting a first character string according to a format of an item value corresponding to the predetermined item, and a second instruction sentence for presenting a second character string that serves as a basis for obtaining the first character string; an acquisition means for acquiring the first character string and the second character string by inputting the first instruction sentence and the second instruction sentence generated by the generation means into a large-scale language model; and an output means for outputting the first character string acquired by the acquisition means together with the second character string. [Effects of the Invention]

[0006] According to the technology of the present disclosure, even when inferring an item value that is not written in a document image, it is possible to present the character string in the document image used for inference. [Brief explanation of the drawings]

[0007] [Figure 1]FIG. 1 is a diagram showing a system configuration of an information processing system. [Figure 2] FIG. 2 is a diagram showing the hardware configuration of an information processing device and an information processing server. [Figure 3] FIG. 2 is a diagram showing the software configuration of an information processing apparatus and an information processing server. [Figure 4] 10 is a flowchart showing a process of presenting a character string used in inference in a document image. [Figure 5] 10A and 10B are diagrams showing specific examples of an input document image and a displayed document image. [Figure 6] FIG. 10 is a diagram showing the OCR results obtained by OCR processing a document image. [Figure 7] 10A and 10B are diagrams showing specific examples of instructions and responses to be input to a large-scale language model. [Figure 8] 10 is a flowchart showing a process for obtaining item values ​​and character strings used in inference using a large-scale language model. [Figure 9] FIG. 10 is a diagram showing an example of a screen for confirming item values ​​and character strings used in inference. [Figure 10] 10 is a flowchart showing a process of presenting item values ​​in a document image and character strings used for inference. [Figure 11] 10A and 10B are diagrams showing specific examples of instructions and responses to be input to a large-scale language model. [Figure 12] FIG. 10 is a diagram showing a specific example presenting item values ​​in a document image and character strings used for inference. [Figure 13] 10 is a flowchart showing a process for obtaining item values ​​and character strings used in inference using a large-scale language model. [Figure 14] FIG. 10 is a diagram showing an example of a screen for confirming item values ​​and character strings used in inference. [Figure 15] 10 is a flowchart showing a process of presenting item values ​​in a document image and character strings used for inference. [Figure 16] 10A and 10B are diagrams showing specific examples of instructions and responses to be input to a large-scale language model. [Figure 17] 10 is a flowchart showing a process for obtaining item values ​​and character strings used in inference using a large-scale language model. [Figure 18]FIG. 10 is a diagram showing a specific example of a list of item names that are not to be inferred. [Figure 19] FIG. 10 is a diagram showing a specific example of an instruction sentence to be input to a large-scale language model. [Figure 20] FIG. 10 is a diagram showing a specific example of an instruction sentence to be input to a large-scale language model. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Note that the following embodiments do not limit the present disclosure, and not all combinations of features described in the embodiments are necessarily essential to the solutions of the present disclosure. Note that identical components will be described with the same reference numerals. Furthermore, each process (step) in a flowchart will be indicated with a reference numeral beginning with "S."

[0009] [First embodiment] <Information Processing System> FIG. 1 is a diagram showing an example of the configuration of an information processing system according to this embodiment. As shown in FIG. 1, the information processing system 100 includes an information processing device 101, an information processing server 102, and an external information processing server 103. The devices are connected to each other via a network 104. In the information processing system 100, the information processing device 101, the information processing server 102, and the external information processing server 103 may be configured to be connected to the network 104 in multiple rather than single connections. For example, the information processing server 102 may include a first server device having high-speed computing resources and a second server device having large-capacity storage, and the server devices may be connected to each other via the network 104. Furthermore, in this embodiment, the information processing device 101 and the information processing server 102 are described as being separate entities, but the information processing device 101 and the information processing server 102 may be integrated.

[0010] <Hardware configuration of each device> 2 is a diagram showing an example of the hardware configuration of the information processing device 101 and the information processing server 102 of the information processing system 100 according to this embodiment. FIG. 2(a) is a diagram showing the hardware configuration of the information processing device 101. As shown in FIG. 2(a), the information processing device 101 includes a CPU 201, a ROM 202, a RAM 204, a printer device 205, a scanner device 206, a storage 208, and an external interface 211. The devices are connected to each other via a data bus 203.

[0011] The CPU 201 is a control unit for controlling the overall operation of the information processing device 101. The CPU 201 starts up the system of the information processing device 101 by executing a startup program stored in the ROM 202. The CPU 201 also executes a control program stored in the storage 208 to realize functions of the information processing device 101, such as printing, scanning, or faxing.

[0012] The ROM 202 is a non-volatile memory and is a storage unit for storing a startup program that starts up the information processing device 101. The data bus 203 is a communication unit for transmitting and receiving data between devices that make up the information processing device 101. The RAM 204 is a volatile memory and is a storage unit used as a work memory when the CPU 201 executes a control program.

[0013] The printer device 205 is an image output device and a processing unit for printing and outputting a document image inside the information processing device 101 on a sheet of paper or the like. The scanner device 206 is an image input device and a processing unit for optically reading a sheet of paper or the like on which characters, diagrams, etc. are printed and acquiring the document image. The document transport device 207 is realized by an ADF (Auto Document Feeder) or the like and is a processing unit for detecting documents placed on a platen and transporting the detected documents one by one to the scanner device 206.

[0014] The storage 208 is realized by a hard disk drive (HDD) or the like, and is a storage unit for storing the control programs and document images described above. The storage 208 is not limited to an HDD, but may be a solid state drive (SSD) or a flash memory.

[0015] The input device 209 is realized by a touch panel, hard keys, or the like, and is a processing unit for accepting operation input from a user who uses the information processing device 101. The display device 210 is realized by a liquid crystal display, or the like, and is a display unit for displaying output results of a setting screen of the information processing device 101 to the user. By using a touch panel, the input device 209 and the display device 210 may be integrated. The external interface 211 is an interface unit that connects the information processing device 101 and the network 104, receives FAX data transmitted from a FAX transmitter (not shown), and transmits a document image to the information processing server 102.

[0016] Fig. 2(b) is a diagram showing the configuration of the information processing server 102. As shown in Fig. 2(b), the information processing server 102 includes a CPU 261, a ROM 262, a RAM 264, a storage 265, an input device 266, a display device 267, and an external interface 268. The devices are connected to each other via a data bus 263.

[0017] The CPU 261 is a control unit for controlling the overall operation of the information processing server 102. The CPU 261 starts up the system of the information processing server 102 by executing a boot program stored in the ROM 262. The CPU 261 also executes an information processing program stored in the storage 265 to perform information processing such as character recognition and information extraction using OCR (Optical Character Recognition) or the like.

[0018] The ROM 262 is a non-volatile memory and is a storage unit for storing a boot program that starts up the information processing server 102. The data bus 263 is a communication unit for transmitting and receiving data between devices that make up the information processing server 102. The RAM 264 is a volatile memory and is a storage unit used as a work memory when the CPU 261 executes the information processing program. The storage 265 is realized by an HDD or the like and is a storage unit for storing the information processing program, document images, item extraction results, extraction results of information used for inference, etc. The storage 265 is not limited to an HDD and may be an SSD or flash memory.

[0019] The input device 266 is realized by a mouse, a keyboard, or the like, and is a processing unit for accepting input operations to the information processing server 102 from a user who uses the information processing server 102 or an engineer who controls the information processing server 102. The display device 267 is realized by a liquid crystal display or the like, and is a display unit for displaying a setting screen of the information processing server 102 and output results to a user who uses the information processing server 102 or an engineer who controls the information processing server 102. The input device 266 and the display device 267 may be integrated by using a touch panel. The external interface 268 is an interface unit that connects the information processing server 102 and the network 104 and receives document images from the information processing device 101.

[0020] <Software configuration of each device> 3A and 3B are diagrams showing an example of software functional blocks of the information processing device 101 and the information processing server 102 of the information processing system 100 according to this embodiment. Fig. 3A is a diagram showing software functional blocks of the information processing device 101, and Fig. 3B is a diagram showing software functional blocks of the external information processing server 103.

[0021] The information processing device 101 is realized by an MFP (Multi Function Peripheral) or the like having multiple functions such as printing, scanning, and faxing, and functions as an image acquisition unit 301, a first data transmission / reception unit 302, and a display control unit 303.

[0022] The image acquisition unit 301, for example, optically reads an original document printed on a sheet such as paper and performs predetermined scan image processing to generate a document image. The image acquisition unit 301 also receives, for example, fax data transmitted from a fax transmitter (not shown) and generates a document image by performing predetermined fax image processing. The first data transmission / reception unit 302 transmits the generated document image to the information processing server 102. The first data transmission / reception unit 302 also transmits and receives other data to and from the information processing server 102. The display control unit 303, for example, controls the display of image data of the document image and other information received from the information processing server 102 on the display device 210. The display control unit 303, for example, displays a confirmation screen or the like shown in FIG. 9 (described later) on the display device 210.

[0023] The information processing device 101 may be configured to be realized by a PC (Personal Computer), a tablet terminal, a smartphone, etc., other than the MFP having the above-mentioned scan and FAX functions. Specifically, for example, a document image such as PDF or JPEG generated using a document creation application running on a PC serving as the information processing device 101 may be transmitted to the information processing server 102 via the first data transmission / reception unit 302.

[0024] The information processing server 102 functions as a second data transmission / reception unit 351, a character string extraction unit 352, an instruction generation unit 353, a character string acquisition unit 354, and an output unit 355. Specifically, the information processing server 102 has a function of receiving a document image as input, processing the image via the external information processing server 103, and transmitting the processed result to the information processing device 101.

[0025] The second data transmission / reception unit 351 transmits and receives data such as a document image generated by the information processing device 101 to and from the information processing device 101. The character string extraction unit 352 performs OCR processing on the document image received from the information processing device 101, and acquires multiple character strings in the document image.

[0026] The instruction generation unit 353 generates an instruction including an instruction statement for inferring a character string that conforms to the format of an item value corresponding to a predetermined item. Specifically, the instruction generation unit 353 applies multiple character strings included in a document image and an item to be extracted to a pre-prepared instruction template, and generates an instruction to input the character strings to the large-scale language model 151. Here, the pre-prepared instruction template may be one prepared by an engineer as a preset, one prepared by a user, or one that has been modified or added to by the system or the user. Thereafter, the information processing server 102 transmits the generated instruction to the external information processing server 103 via the network 104. The instruction generation unit 353 also generates an instruction including an instruction statement for extracting a character string used to infer the item value using the instruction template.

[0027] The character string acquisition unit 354 receives a response result to the instruction from the external information processing server 103, and acquires a character string according to the above format or a character string according to an instruction statement for extracting a character string used in inferring an item value, etc. The output unit 355 displays the character string according to the format acquired by the character string acquisition unit 354 on the display device 267. The output unit 355 also transmits the character string according to the format, the character string used in inference, and other data to the information processing device 101 via the second data transmission / reception unit 351. The display control unit 303 of the information processing device 101 displays the character string according to the format received via the first data transmission / reception unit 302, the character string used in inference, and other data on the display device 210.

[0028] The network 104 is realized by a LAN, a WAN, or the like, and is a communication unit that connects the information processing device 101, the information processing server 102, and the external information processing server 103 to one another and transmits and receives data between the devices.

[0029] The external information processing server 103 functions as a large-scale language model 151. The large-scale language model 151 is accessed by an API (Application Programming Interface) via the network 104, and outputs a response result to a command received from the information processing server 102 to the information processing server 102. The external information processing server 103 may be configured to exist on a different system of the same vendor, or may be configured to exist on a system of an external vendor. The external information processing server 103 including the large-scale language model 151 may be configured to exist within the information processing server 102. Alternatively, the information processing server 102 may be configured so that some of the functions or some of the devices of the external information processing server 103 exist.

[0030] <Output processing of strings used in inference> 4 is a flowchart showing a processing flow in which the information processing server 102 performs document analysis on a document image received from the information processing device 101 and outputs the position of the character string used for inference as the analysis result to the display device 267. Note that an execution program for each step shown in FIG. 4 is stored in the ROM 262, RAM 264, or storage 265 of the information processing server 102 and executed by the CPU 261 of the information processing server 102.

[0031] In S401, the character string extraction unit 352 performs character recognition processing on the document image to acquire one or more character strings. Specifically, for example, OCR processing is performed on the document image to acquire one or more character strings included in the document image and area information for the character strings. A method for acquiring one or more character strings and area information will be described with reference to FIGS. 5(a) and 6. FIG. 5(a) is an example of a document image, and illustrates a document image 500 in this embodiment. FIG. 6 illustrates an OCR result 600 obtained by performing OCR processing on the document image 500 of FIG. 5(a). The OCR result 600 includes one or more pieces of character information (each line in FIG. 6). The one or more pieces of character information further include a character string 601 of the OCR result and area information 602 indicating the position of the character string acquired by performing OCR processing. The character string 601 is a character string extracted from the document image 500, and the area information 602 is information on the position of the character string 601 on the document image 500. Note that the above-described method for acquiring character information is merely an example and is not limited thereto.

[0032] In S402, the instruction generation unit 353 generates an instruction including an instruction sentence using the character string in the document image 500 acquired in S401. The character string acquisition unit 354 inputs the instruction to the large-scale language model 151 and acquires the extracted result. This will be specifically described with reference to FIGS. 7(b) and 7(c). FIG. 7(b) shows an example of an instruction for the document image 500 in FIG. 5(a). The instruction 710 includes the "character string acquired in S401" (text 713), an instruction sentence 721 (first instruction sentence), and an instruction sentence 722 (second instruction sentence). The instruction sentence 721 (first instruction sentence) is an instruction sentence for "presenting an item value (first character string) for the input item name 701." The instruction sentence 722 (second instruction sentence) is an instruction sentence for "presenting all character strings (second character strings) that serve as the basis for acquiring the above item value." FIG. 7(c) shows an example of an output of the extracted result when the instruction 710 is input to the large-scale language model 151. That is, when the character string acquisition unit 354 inputs the instruction 710 to the large-scale language model 151, it acquires as output an output result 740. The output result 740 includes an output item value 741 and a character string 742 used in the inference. The processing of S402 will be described in detail later.

[0033] In S403, the output unit 355 presents the character string used for the inference acquired in S402 in the document image 500. An example of the character string used for the inference acquired in S402 is "Publication date: October 12, 2023." The output unit 355 compares the character string used for the inference with the character string 601 of the OCR result 600 acquired in S401, and confirms that the character string used for the inference is included in the character string 601.

[0034] A method for presenting the character string used for inference in the document image 500 will be described with reference to FIG. 5(b). FIG. 5(b) is a diagram showing the document image 500 after the area of ​​the character string used for inference has been presented. The output unit 355 presents an area 501 in the document image 500 based on area information 602 of the character string used for inference. The character string used for inference may be presented directly in the document image 500, or may be presented on a document image screen 901 in FIG. 9, which will be described later.

[0035] In S404, the output unit 355 determines whether all character strings used in the inferences acquired in S401 have been presented in the document image 500. If the output unit 355 determines that all character strings used in the inferences have been presented, the process proceeds to S405. If the output unit 355 determines that all character strings used in the inferences have not been presented, the process returns to S403. In S405, the output unit 355 outputs the document image in which the positions of the character strings used in the inferences are presented and the output item value 741 acquired in S402 to the display device 267 and the display device 210 of the information processing device 101. Details of the process of S405 will be described later.

[0036] <Inference using large-scale language models> 8 is a flowchart showing a processing flow in which the information processing server 102 infers item values ​​for a document image using the large-scale language model 151 and acquires the inferred item values ​​and character strings used for the inference. Note that an execution program for each step shown in FIG. 8 is stored in the ROM 262, RAM 264, or storage 265 of the information processing server 102 and executed by the CPU 261 of the information processing server 102.

[0037] In S801, the character string extraction unit 352 acquires the character string of the document image extracted in S401 in FIG. 4 (hereinafter referred to as "document character string"). That is, the character string 601 in the OCR result 600 in FIG. 6 is acquired as the document character string. In S802, the instruction generation unit 353 acquires the name of the item to be inferred (the name of the item to be extracted) as a setting input by the user. A method for acquiring the name of the item to be inferred will be described with reference to FIG. 7(a). FIG. 7(a) is a diagram showing an input screen for the name of the item to be inferred (the name of the item to be extracted). The input item name 701 is the name of the item to be inferred that has been set in advance by the user. Furthermore, the output format 702 is the output format for the input item name 701. For example, if the input item name 701 is set to "Deadline", the output format 702 is set to "Year - Month - Day". If the input item name 701 is set to "Amount including tax", the output format 702 is set to "~ Yen". Other examples include the output format of the item value for the document name or document number. The input item name and output format may be ones for which options are prepared in advance, or may be ones that are set by the user by direct input.

[0038] In S803, the instruction generation unit 353 infers an item value and obtains, from the storage 265, an instruction template that presents the inferred item value and the character string used for the inference. The instruction template will be described with reference to FIGS. 5(a), 7(a), and 7(b). FIG. 7(b) shows an example of an instruction statement regarding the document string 703 of the document image 500 in FIG. 5(a) and the input item name 701 and output format 702 in FIG. 7(a). The document string 703 is the character string 601 in FIG. 6 arranged in order. The instruction template refers to the entire text of the instruction 710 excluding the input item name 701, the output format 702, and the document string 703. In other words, the instruction template for the instruction 710 includes text 711 excluding the input item name 701, text 712 excluding the input item name 701 and the output format 702, and text 713 excluding the document string 703.

[0039] Text 711 contains a directive to instruct large-scale language model 151. Text 711 includes directive 721 and directive 722. Directive 721 contains a directive to infer an item value from input item name 701 and present the item value. Directive 722 contains a directive to present information used during inference from document image 500 as a character string used for the inference when inference is performed by directive 721.

[0040] Text 712 describes the output format output by large-scale language model 151. Text 712 includes output item value format 731 and output clue format 732, which is the output format of the character string used for inference. Output item value format 731 describes the output format of the item value for directive statement 721. Output clue format 732 describes the output format of the character string used for inference for directive statement 722.

[0041] The text 713 is text in which a sentence including a document character string is written. The text 713 is composed of the document character string 703. The instruction generation unit 353 may switch the content of the instruction 710 depending on the language or destination of the input document image. The instruction template may be one prepared in advance by an engineer or a user, or may be one that has been prepared in advance and edited by the user.

[0042] In S804, the instruction generation unit 353 generates an instruction by applying the item name acquired in S802 and the document string acquired in S801 to the instruction template acquired in S803. The method of generating an instruction will be described with reference to Fig. 7(b). As shown in Fig. 7(b), the instruction generation unit 353 generates an instruction 710 by applying, for example, an input item name 701, an output format 702, and a document string 703 to the instruction template acquired in S803.

[0043] In S805, the character string acquisition unit 354 inputs the instruction generated in S804 to the large-scale language model 151. Specifically, the character string acquisition unit 354 inputs the instruction 710 shown in FIG. 7(b) to the large-scale language model 151. In S806, the character string acquisition unit 354 acquires an output result in response to the instruction generated in S804 from the large-scale language model 151. Specifically, the character string acquisition unit 354 acquires an output result 740 (see FIG. 7(c)) in response to the instruction 710 shown in FIG. 7(b) from the large-scale language model 151.

[0044] <Check and correct item values> 9 is a diagram showing a confirmation screen output in S405 to the display device 267 and the display device 210 of the information processing device 101. The confirmation screen 900 presents a document image 500 in which a character string 742 used in inference output from the large-scale language model 151 is presented, and an output item value 741. The confirmation screen 900 includes a document image screen 901, an area 902 for the character string used in inference, an item name 910, text 911 which is the output item value, an edit button 912, and an end button 914.

[0045] The document image screen 901 is a screen that displays the document image 500. An area 902 of the character string used in inference is presented on the document image screen 901. The area 902 of the character string used in inference is the area presented in S403 and is equivalent to the area 501. The item name 910 is the item name acquired from the user in S802 and is equivalent to the input item name 701. The text 911 is the output item value acquired from the large-scale language model 151 in S806 and is equivalent to the output item value 741. The edit button 912 is a button for editing the text 911. The operation when the edit button 912 is pressed will be described later. The exit button 914 is a button for closing the confirmation screen 900. After completing confirmation or correction of the result of the text 911, which is the output item value, on the confirmation screen 900, the user presses the exit button 914 to finish checking the output item value.

[0046] The user checks on the confirmation screen 900 whether the output item value 741 inferred by the large-scale language model 151 and the character string 742 used for the inference are correct. If necessary, the user corrects the output item value 741. The specific procedure is described below.

[0047] A case where the user manually corrects the text will be described with reference to FIG. 9. Text 911 is "November 11, 2023," the output item value 741 in FIG. 7 acquired in S806. When the output unit 355 or the display control unit 303 of the information processing device 101 detects that the user has pressed the edit button 912, the output unit 355 or the display control unit 303 of the information processing device 101 changes the text 911 to an editable state such as a text box, and displays an input cursor 913. The user manually corrects the text 911 using the input cursor 913. Thereafter, the user can confirm the content of the manually corrected text 911 and correct the output item value 741 by pressing an end button 914. However, the display format of the confirmation screen is not limited to this. The display format of the confirmation screen may be a display format such as that shown in FIG. 7(c).

[0048] As described above, according to this embodiment, the information processing server 102 uses the large-scale language model 151 to infer item values ​​and present character strings used for the inference. Therefore, it is possible to display the character strings used for the inference in the document image. This makes it possible to present the character strings in the document image used for the inference even when inferring an item value that is not written in the document image.

[0049] [Second embodiment] In the first embodiment, when inferring an item not described in a document image, the large-scale language model 151 is used to infer an item value and present the character string used for the inference, and the character string used for the inference is displayed on the document image. Meanwhile, a user may wish to acquire an item value described in a document image and an item value not described in the document image. When an item value described in a document image and an item value not described in the document image are simultaneously acquired, the item value described in the document image and the character string used to infer the item value not described in the document image are displayed on the document image. However, when both are presented to the user in the same format on the document image, the user cannot distinguish between the item value described in the document image and the character string used to infer the item value not described in the document image.

[0050] Therefore, in this embodiment, it is determined whether or not an item value extracted using a large-scale language model is described in a document image. If an item value is not described in the document image, the area of ​​the character string used for inference is displayed on the document image. If an item value is described in the document image, the area of ​​the item value described in the document image is displayed on the document image in a display format different from the display of the area of ​​the character string used for inferring an item value not described in the document image. Note that the configuration of the information processing system and device configuration according to this embodiment are the same as those of the first embodiment, and therefore their description will be omitted.

[0051] <Displaying the strings used in inference> 10 shows a flowchart of a process for presenting, on a document image, a character string used for inference and an item value written in the document image in this embodiment. The process flow will be described in which the information processing server 102 performs document analysis on a document image received from the information processing device 101 and outputs the position of the character string used for inference and the position of the item value written in the document image to the display device 267 and the display device 210. The execution program for each step shown in FIG. 10 is stored in the ROM 262, RAM 264, or storage 265 of the information processing server 102, and is executed by the CPU 261 of the information processing server 102.

[0052] The processing of S401 in FIG. 10 is the same as the processing of S401 in the flowchart of FIG. 4. In S1001, the instruction generation unit 353 generates an instruction using the document string acquired in S401. The string acquisition unit 354 inputs the generated instruction into the large-scale language model 151 and acquires the result. The processing of S1001 will be described with reference to FIGS. 5(a), 11(a), and 11(b). FIG. 11(b) shows an example of an instruction for the document image 500 shown in FIG. 5(a). The instruction 1110 includes the "document string acquired in S401" (text 713) and an "instruction statement for displaying item values ​​for the input item names 701 and 1101" (instruction statement 1121). Furthermore, the instruction 1110 includes an "instruction statement for displaying all strings that serve as the basis for acquiring item values" (instruction statement 722). 11(c) is an example of an output result obtained by inputting an instruction 1110 to the large-scale language model 151. That is, when the character string acquisition unit 354 inputs the instruction 1110 to the large-scale language model 151, it acquires an output result 1140 as an output. The output result 1140 includes output item values ​​741 and 1141 and character strings 742 and 1142 used in the inference. The processing of S1001 will be described in detail later.

[0053] In S1002, the character string acquisition unit 354 determines whether the item value acquired in S901 exists in the document character string acquired in S401. The output item value in FIG. 11(c) is compared to determine whether it exists in the character string 601 in the OCR result 600 in FIG. 6. If the character string acquisition unit 354 determines that the output item value exists in the character string 601, the process proceeds to S1003. If the character string acquisition unit 354 determines that the output item value does not exist in the character string 601, the process proceeds to S1004.

[0054] In S1003, the output unit 355 displays the item value obtained in S1001 that is described in the document image 500 in a first format on the document image 500. Of the item value results obtained in S1001, the item value described in the document image 500 is "XYZ Corporation." Therefore, the output unit 355 displays the item value "XYZ Corporation" on the document image 500 in a first format. The display method will be described with reference to FIG. 12.

[0055] FIG. 12 is a diagram showing a document image 500 after the character string area and the item value area used in the inference are presented. The output unit 355 displays an area 1201 in the document image 500 in a first manner using character string area information 602 corresponding to the item value described in the document image 500 by referring to the table shown in FIG. 6 . As an example, in FIG. 12 , the area 1201 is displayed using a solid line. Note that the solid line need not be used as long as it differs from the display method in S1004 described below. For example, the display color of the area 1201 may be different from the display color of the area 1202 displayed in S1004. Alternatively, the combination of the line type and the display color may be different between the area 1201 and the area 1202. The item values ​​described in the document image 500 may be displayed directly on the document image 500 or may be displayed on the screen shown in FIG. 14 described below.

[0056] In S1004, the output unit 355 displays, on the document image 500, the character strings used in the inference acquired in S1001 that were used to infer the item values ​​not listed in the document image 500. Since the character strings used in the inference of the item values ​​not listed in the document image 500 acquired in S1001 are "Publication Date: October 12, 2023" and "Estimated Period: 30 days," the output unit 355 displays the character strings used in the inference in the second format on the document image 500. The display method will be described with reference to FIG. 12. The output unit 355 refers to the table shown in FIG. 6 and uses the area information 602 corresponding to the character strings used in the inference to display the area 1202 on the document image 500 in the second format. As an example, in FIG. 12, the area 1202 is displayed with a dotted line. Note that the dotted line need not be used as long as it is different from the display method of S1003 described above. For example, the display color of area 1202 may be different from the display color of area 1201 displayed in S1003. Alternatively, the combination of line type and display color may be different between area 1201 and area 1202. Furthermore, character strings used to infer item values ​​not described in document image 500 may be displayed directly on document image 500, or may be displayed on the screen of FIG. 14, which will be described later.

[0057] In S1005, the character string acquisition unit 354 determines whether the character strings and item values ​​used in all inferences acquired in S1001 have been displayed in the document image 500. If the character string acquisition unit 354 determines that the character strings and item values ​​used in all inferences have been displayed in the document image 500, the process proceeds to S1006. If the character string acquisition unit 354 determines that the character strings and item values ​​used in all inferences have not been displayed in the document image 500, the process returns to S1002. In S1006, the output unit 355 outputs a document image displaying the positions of the character strings and item values ​​used in the inference, the output item values, and the character strings used in the inference to the display device 267 and the display device 210 of the information processing device 101. Details of the process of S1006 will be described using FIG. 14.

[0058] <Inference using large-scale language models> 13 is a flowchart showing a processing flow in which the information processing server 102 infers item values ​​for a document image using the large-scale language model 151 and acquires the inferred item values ​​and character strings used for the inference. Note that an execution program for each step shown in FIG. 13 is stored in the ROM 262, RAM 264, or storage 265 of the information processing server 102 and executed by the CPU 261 of the information processing server 102.

[0059] The processing of S801 in FIG. 13 is the same as the processing of S801 in the flowchart of FIG. 8. In S1301, the instruction generation unit 353 acquires the name of the item to be inferred as a setting input by the user. The processing of S1301 will be described in detail with reference to FIG. 11(a). FIG. 11(a) is a diagram showing an input screen for the name of the item to be inferred. Input item names 701 and 1101 are the names of the items to be inferred that have been set in advance by the user. Furthermore, output formats 702 and 1102 are output formats for the input item names 701 and 1101, respectively. Note that the input item names and output formats may be ones for which candidate options are prepared in advance, or may be ones that are set by the user by direct input. Furthermore, there may be three or more input item names.

[0060] In S1302, the instruction generation unit 353 determines whether the item names acquired in S1301 are multiple. If the instruction generation unit 353 determines that the item names acquired are multiple, the process proceeds to S1303. If the instruction generation unit 353 determines that the item names acquired are not multiple, the process proceeds to S803. The process of S803 in Fig. 13 is the same as the process of S803 in Fig. 8, and therefore description thereof will be omitted.

[0061] In S1303, the instruction generation unit 353 infers an item value when there are multiple item names, and obtains, from the storage 265, an instruction statement template that presents the inferred item value and the character string used for the inference. Details of the instruction statement template will be described with reference to FIGS. 5(a), 11(a), and 11(b). FIG. 11(b) shows an example of instructions for the document string 703 of the document image 500 in FIG. 5(a), the input item names 701 and 1101, and the output formats 702 and 1102 in FIG. 11(a). The document string 703 is the same as the document string 703 described in S803. The instruction template refers to the entire text of the instruction 1110, excluding the input item names 701 and 1101, the output formats 702 and 1102, and the document string 703. The instruction template for instruction 1110 includes text 1111 excluding input item names 701 and 1101 , text 1112 excluding input item names 701 and 1101 and output formats 702 and 1102 , and text 713 excluding document string 703 .

[0062] Text 1111 contains a directive to instruct large-scale language model 151. Text 1111 includes directive 1121 and directive 722. Directive 1121 contains an instruction to infer an item value from input item names 701 and 1101 and present the item value. Directive 722 contains an instruction to present information used during inference from document image 500 as a character string used for the inference when inference is performed by directive 1121.

[0063] Text 1112 describes the output format output by large-scale language model 151. Text 1112 includes multiple output item value formats 731 and multiple output clue formats 732, which are formats of character strings used in inference. Output item value formats 731 describe the output format of item values ​​for directive statement 1121. Output clue formats 732 describe the output format of character strings used in inference for directive statement 722.

[0064] The instruction generation unit 353 may switch the content of the instruction 1110 depending on the language or destination of the input document image. The instruction template may be one prepared in advance by an engineer or a user, or may be one that has been prepared in advance and edited by the user.

[0065] In S1304, the instruction generation unit 353 generates an instruction by applying the item name acquired in S1301 and the document string acquired in S801 to the instruction template acquired in S803 or S1303. A method for generating an instruction using the instruction template acquired in S1303 will be described with reference to FIG. 11(b). The instruction generation unit 353 generates an instruction 1110 by applying, for example, the input item names 701 and 1101, the output formats 702 and 1102, and the document string 703 to the instruction template acquired in S1303. Once the instruction 1110 has been generated, the process proceeds to S805. The processes from S805 to S806 in FIG. 13 are the same as the processes from S805 to S806 in FIG. 8, and therefore description of the processes from S805 to S806 will be omitted.

[0066] <Check and correct item values> 14 is a diagram showing a confirmation screen 1400 output in S805 to the display device 267 and the display device 210 of the information processing device 101. The confirmation screen 1400 presents the document image 500 in which an area 1402 of the character string used in inference and an item value area 1403 are presented, as well as text 911 and 1411 which are output item values ​​output from the large-scale language model 151. The confirmation screen 1400 includes a document image screen 1401, an area 1402 of the character string used in inference, an item value area 1403, item names 910 and 1410, text 911 and 1411, an edit button 912, and an end button 914.

[0067] The document image screen 1401 is a screen that displays the document image 500. An area 1402 of the character string used in inference and an item value area 1403 are presented on the document image screen 1401. The area 1402 of the character string used in inference is the area presented in S1004 and is equivalent to the area 1202. The item value area 1403 is the area presented in S1003 and is equivalent to the area 1201. The item names 910 and 1410 are item names acquired from the user in S1001 and are equivalent to the input item names 701 and 1101. The text 911 and 1411 are output item values ​​acquired from the large-scale language model 151 in S806 and are equivalent to the output item values ​​741 and 1141. The edit button 912 and the end button 914 are the same as those described in the first embodiment.

[0068] The user checks on the confirmation screen 1400 whether the texts 911 and 1411 inferred by the large-scale language model 151 are correct values. If necessary, the user corrects the texts 911 and 1411. The correction method has been explained in the first embodiment, so explanation will be omitted.

[0069] According to this embodiment, if the extracted item value is not described in the document image, the character string used for inference is presented on the document image. Furthermore, if the extracted item value is described in the document image, the item value is presented on the document image in a manner different from the presentation of the character string used for inference. This allows the user to distinguish between the item value described in the document image and the character string used for inferring the item value not described in the document image.

[0070] [Third embodiment] In the second embodiment, the method of determining whether an inferred item value is described in a document image is performed using a document string. In this embodiment, an instruction statement for outputting "whether an inferred item value is described" is added to the instruction input to the large-scale language model 151. Based on the output result of the instruction statement, the CPU 261 of the information processing server 102 determines whether the inferred item value is described in the document image. The configuration of the information processing system, the device configuration, and the confirmation and correction of item values ​​according to this embodiment are the same as those of the second embodiment described above, and therefore description thereof will be omitted.

[0071] <Displaying the strings used in inference> 15 shows a flowchart of a process in which the information processing server 102 in this embodiment performs document analysis on a document image received from the information processing device 101 and presents, as the analysis result, the positions of the character strings and item values ​​used in the inference in the document image. Note that an execution program for each step shown in FIG. 15 is stored in the ROM 262, RAM 264, or storage 265 of the information processing server 102 and executed by the CPU 261 of the information processing server 102.

[0072] The processing of S401 in FIG. 15 is the same as the processing of S401 in the flowchart of FIG. 4. In S1501, the instruction generation unit 353 generates an instruction using the document string acquired in S401. The string acquisition unit 354 inputs the generated instruction into the large-scale language model 151 and acquires the result. The processing of S1501 will be described with reference to FIGS. 16(a), 16(b), and 16(c). FIG. 16(b) shows an example of an instruction for the document image 500 shown in FIG. 5(a). The instruction 1610 includes "the document string acquired in S401" (text 713) and "an instruction to present the item values ​​for the input item name 701 and the text 1111" (instruction statement 1121). The instruction 1610 also includes "an instruction to present all the strings that serve as the basis for acquiring the item values" (instruction statement 722). Furthermore, the instruction 1610 also includes "an instruction to display whether or not the item value is described in the document image" (instruction 1623 (third instruction)). Also, FIG. 16(c) is an example of an output result obtained by inputting the instruction 1610 to the large-scale language model 151. That is, when the character string acquisition unit 354 inputs the instruction 1610 to the large-scale language model 151, it acquires an output result 1640 as an output. The output result 1640 includes the output item values ​​741 and 1141, the character strings 742 and 1142 used in the inference, and the item value description result 1643. Also, details of the processing of S1501 will be described later with reference to FIG. 17.

[0073] In S1502, the character string acquisition unit 354 determines whether the item value description result acquired in S1501 for each item name was output as "described." If the character string acquisition unit 354 determines that the item value description result was output as "described," the process proceeds to S1003. If the character string acquisition unit 354 determines that the item value description result was output as "not described," the process proceeds to S1004. The processes from S1003 to S1005 in FIG. 15 are the same as the processes from S1003 to S1005 in the flowchart in FIG. 10.

[0074] In S1503, the output unit 355 outputs a document image displaying the position of the string used in the inference and the position of the item value, the output item value, the string used in the inference, and the item value entry results to the display device 267 and the display device 210 of the information processing device 101.

[0075] <Inference using large-scale language models> 17 is a flowchart showing a processing flow in which the information processing server 102 infers an item value for a document image using the large-scale language model 151, and acquires the inferred item value, the character string used for the inference, and a result of whether the item value is described in the document image. Note that an execution program for each step shown in FIG. 17 is stored in the ROM 262, RAM 264, or storage 265 of the information processing server 102, and is executed by the CPU 261 of the information processing server 102.

[0076] The processing of S801 and S1301 to S1302 in FIG. 17 is the same as the processing of S801 in FIG. 8 and the processing of S1301 to S1302 in FIG. 13. In S1701, the instruction generation unit 353 infers an item value when there are multiple item names, and obtains from the storage 265 an instruction template that presents the inferred item value, the character string used for the inference, and a result of whether the item value is described in the document image. Details of the instruction template will be described using FIGS. 5(a), 16(a), and 16(b). FIG. 16(b) shows an example of instructions for the document string 703 of the document image 500 in FIG. 5(a), the input item names 701 and 1101, and the output formats 702 and 1102 in FIG. 16(a). The document string 703 is the same as the document string described in S803. The instruction template refers to the entire text of instruction 1610 excluding input item names 701 and 1101, output formats 702 and 1102, and document string 703. The instruction template of instruction 1610 includes text 1611 excluding input item names 701 and 1101, text 1612 excluding input item names 701 and 1101 and output formats 702 and 1102, and text 713 excluding document string 703.

[0077] Text 1611 contains instructions to instruct large-scale language model 151. Text 1611 includes instructions 1121, 722, and 1623. Instruction 1121 contains an instruction to infer an item value from input item names 701 and 1101 and present the item value. Instruction 722 contains an instruction to present information used during inference from document image 500 when inferring with instruction 1121 as a character string used for the inference. Instruction 1623 contains an instruction to infer whether input item names 701 and 1101 are described in the document image and present the inference result.

[0078] Text 1612 describes the output format output by large-scale language model 151. Text 1612 includes multiple output item value formats 731, multiple output clue formats 732 which are formats of character strings used in inference, and multiple item value entry result formats 1633. Output item value format 731 describes the output format of item values ​​for instruction statement 1121. Output clue format 732 describes the output format of character strings used in inference for instruction statement 722. Item value entry result format 1633 describes the output format of item value entry results for instruction statement 1623.

[0079] The instruction generating unit 353 may switch the content of the instruction 1610 depending on the language or destination of the input document image. The instruction statement template may be one prepared in advance by an engineer or a user, or may be one that has been prepared in advance and edited by the user.

[0080] In S1702, the instruction generation unit 353 infers an item value and obtains an instruction template that presents the inferred item value, the character string used for the inference, and the result of whether the item value is described in the document image from the storage 265. Note that the processing of S1702 is the same as S803 in Fig. 13 except that the content of the obtained template is different.

[0081] In S1304, the instruction generation unit 353 generates an instruction by applying the item name acquired in S1301 and the document character string acquired in S801 to the instruction template acquired in S1701 or S1702. Note that the processing of S1304 in Fig. 17 is the same as S1304 in Fig. 13 except that the content of the template is different.

[0082] In S1703, the instruction generation unit 353 obtains an item name list from the storage 265. The item name list will be described with reference to FIG. 18. FIG. 18 shows an example of an item name list 1800. For example, there is basically an item value written in the document image 500, such as the item name of the destination company name. There are also cases where an item value is not written in the document image 500, and the item value cannot be inferred even using other information in the document image 500. Therefore, there are cases where it is not necessary to ask whether the item name is written in the document image 500.

[0083] For this reason, an item name list 1800 is created, and item names for which it is not necessary to infer whether they are written in the document image 500 are listed in the item name list 1800. The item name list 1800 is blank or includes one or more item names 1801. The item names 1801 are item names for which it is not necessary to infer whether they are written in the document image 500. The item name list 1800 may be one prepared in advance by an engineer or a user, or may be one that has been edited by the user from a prepared list.

[0084] In S1704, the instruction generation unit 353 determines whether the item name to be processed acquired in S1301 is present in the item name list 1800 acquired in S1703. If the instruction generation unit 353 determines that the item name to be processed is present in the item name list 1800, the process proceeds to S1705. If the instruction generation unit 353 determines that the item name to be processed is not present in the item name list 1800, the process proceeds to S1706.

[0085] In S1705, the instruction generation unit 353 deletes the item value entry result format for the item name to be processed from the instruction, and processing proceeds to S1706. The processing of S1705 will be described with reference to Figures 16(b) and 19. Figure 19 is an example of instruction 1950 in which the item value entry result format 1633 has been deleted from instruction 1610 in Figure 16(b). The instruction generation unit 353 deletes the item value entry result format 1633 for the item name listed in item name list 1800 from instruction 1610, and generates instruction 1950. The item name list 1800 includes "destination company name." Therefore, in instruction 1950 in Figure 19, the item value entry result format for "destination company name" has been deleted.

[0086] In S1706, the instruction generation unit 353 determines whether the process of determining whether all item names acquired in S1301 exist in the item name list 1800 has been completed. If the instruction generation unit 353 determines that the process of S1704 has been completed for all item names, the process proceeds to S1707. If the instruction generation unit 353 determines that the process of S1704 has not been completed for all item names, the process returns to S1704.

[0087] In S1707, the instruction generation unit 353 determines whether all item value entry result formats have been deleted in S1705. If the instruction generation unit 353 determines that all item value entry result formats have been deleted, the process proceeds to S1708. If the instruction generation unit 353 determines that all item value entry result formats have not been deleted, the process proceeds to S805.

[0088] In S1708, the instruction generation unit 353 deletes the instruction to present whether the item is described in the document image 500 from the instruction statement. The processing of S1708 will be described with reference to FIGS. 20(a) and 20(b). In the examples of FIGS. 20(a) and 20(b), the only item value to be inferred is the "recipient company name." FIG. 20(a) is an example of instruction 2060 in which all item value description result formats have been deleted. FIG. 20(b) is an example of instruction 2070 in which instruction statement 2023, which contains a command to infer and present whether the item is described in the document image from instruction 2060 in FIG. 20(a), has been deleted. That is, the instruction generation unit 353 deletes instruction statement 2023 from instruction 2060 and generates instruction 2070. The processing from S805 to S806 in FIG. 17 is the same as the processing from S805 to S806 in FIG. 8, and therefore a description of the processing from S805 to S806 will be omitted.

[0089] As described above, according to this embodiment, by adding an instruction statement to be input to a large-scale language model that also outputs whether or not an item value is written in a document image, it becomes possible to infer whether or not an item value is written in a document image.

[0090] The present disclosure can also be realized by providing a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0091] The disclosure of the above-described embodiment includes the following configurations.

[0092] (Configuration 1) An information processing device for extracting a character string corresponding to a specified item from a document image, comprising: a generation means for generating, by inference, a first instruction statement that causes a first character string conforming to the format of an item value corresponding to the specified item to be presented, and a second instruction statement that causes a second character string that serves as a basis for obtaining the first character string; an acquisition means for acquiring the first character string and the second character string by inputting the first instruction statement and the second instruction statement generated by the generation means into a large-scale language model; and an output means for outputting the first character string acquired by the acquisition means together with the second character string.

[0093] (Configuration 2) The information processing device according to Configuration 1, wherein the output means displays the first character string present in the document image on the display device in a manner different from the manner in which the second character string present in the document image is displayed when the first character string is not present in the document image.

[0094] (Configuration 3) The information processing device according to Configuration 2, wherein the output means displays, on the display device, a display color of the first character string present in the document image in a color different from a display color of the second character string not present in the document image.

[0095] (Configuration 4) The information processing device according to configuration 2 or 3, wherein the output means displays on the display device a frame surrounding the first character string present in the document image that is different from a frame surrounding the second character string where the first character string is not present in the document image.

[0096] (Configuration 5) The information processing device according to Configuration 4, wherein the output means surrounds the first character string present in the document image with a solid-line frame, and surrounds the second character string not present in the document image with a dotted-line frame.

[0097] (Configuration 6) The information processing device according to any one of configurations 1 to 5, wherein the generating means further generates a third instruction statement indicating whether the first character string exists in the document image.

[0098] (Configuration 7) A control method for an information processing device for extracting a character string corresponding to a specified item from a document image, comprising: a step of generating, by inference, a first instruction statement that causes a first character string conforming to the format of an item value corresponding to the specified item to be presented, and a second instruction statement that causes a second character string that serves as a basis for obtaining the first character string; a step of acquiring the first character string and the second character string by inputting the first instruction statement and the second instruction statement generated in the generating step into a large-scale language model; and a step of outputting the first character string acquired in the acquiring step together with the second character string.

[0099] (Configuration 8) A program for causing a computer to function as the information processing device according to any one of configurations 1 to 6.

[0100] (Configuration 9) An information processing system including a first information processing device that transmits a document image and predetermined items about the document image and receives information corresponding to the predetermined items, and a second information processing device that receives the document image and the predetermined items about the document image and transmits the information corresponding to the predetermined items, wherein the second information processing device is equipped with a generation means that generates, by inference, a first instruction statement that causes the first information processing device to present a first string that conforms to a format of an item value corresponding to the predetermined items, and a second instruction statement that causes the second information processing device to present a second string that serves as a basis for obtaining the first string, an acquisition means that acquires the first string and the second string by inputting the first instruction statement and the second instruction statement generated by the generation means into a large-scale language model, and an output means that outputs the first string acquired by the acquisition means together with the second string.

[0101] (Configuration 10) The information processing system described in Configuration 9, wherein the first information processing device includes a first display device, and the output means displays the first character string present in the document image on the first display device in a manner that differs from the display manner of the second character string in which the first character string does not exist in the document image.

[0102] (Configuration 11) The information processing system described in Configuration 9 or 10, wherein the second information processing device further includes a second display device, and the output means displays the first character string present in the document image on the second display device in a manner that is different from the display manner of the second character string in which the first character string is not present in the document image.

[0103] (Configuration 12) The information processing system according to any one of configurations 9 to 11, wherein the generating means further generates a third instruction statement indicating whether the first character string exists in the document image.

Claims

1. An information processing device for extracting character strings corresponding to predetermined items from a document image, a generating means for generating, by inference, a first instruction statement for causing a first character string conforming to a format of an item value corresponding to the predetermined item to be presented, and a second instruction statement for causing a second character string to be presented as a basis for obtaining the first character string; an acquisition means for acquiring the first character string and the second character string by inputting the first instruction sentence and the second instruction sentence generated by the generation means into a large-scale language model; an output means for outputting the first character string acquired by the acquisition means together with the second character string; Equipped with 1. An information processing device comprising:

2. 2. The information processing device according to claim 1, wherein the output means displays the document image on the display device in such a manner that the display mode of the first character string present in the document image is different from the display mode of the second character string not present in the document image.

3. 3. The information processing device according to claim 2, wherein the output means displays, on the display device, a display color of the first character string present in the document image that is different from a display color of the second character string not present in the document image.

4. 3. The information processing device according to claim 2, wherein the output means displays on the display device a frame surrounding the first character string present in the document image that is different from a frame surrounding the second character string in which the first character string is not present in the document image.

5. 5. The information processing apparatus according to claim 4, wherein the output means surrounds the first character string present in the document image with a solid-line frame, and surrounds the second character string not present in the document image with a dotted-line frame.

6. 2. The information processing apparatus according to claim 1, wherein said generating means further generates a third instruction statement indicating whether said first character string exists in said document image.

7. A method for controlling an information processing device for extracting character strings corresponding to predetermined items from a document image, comprising: generating, by inference, a first instruction statement for presenting a first character string conforming to a format of an item value corresponding to the predetermined item, and a second instruction statement for presenting a second character string that is a basis for obtaining the first character string; acquiring the first character string and the second character string by inputting the first instruction sentence and the second instruction sentence generated in the generating step into a large-scale language model; outputting the first character string acquired in the acquiring step together with the second character string; Equipped with 2. A method for controlling an information processing apparatus comprising:

8. A program for causing a computer to function as the information processing device according to any one of claims 1 to 6.

9. An information processing system including: a first information processing device that transmits a document image and predetermined items about the document image and receives information corresponding to the predetermined items; and a second information processing device that receives the document image and the predetermined items about the document image and transmits the information corresponding to the predetermined items, The second information processing device a generating means for generating, by inference, a first instruction statement for causing the device to present a first character string conforming to a format of an item value corresponding to the predetermined item, and a second instruction statement for causing the device to present a second character string that is a basis for obtaining the first character string; an acquisition means for acquiring the first character string and the second character string by inputting the first instruction sentence and the second instruction sentence generated by the generation means into a large-scale language model; an output means for outputting the first character string acquired by the acquisition means together with the second character string; Equipped with An information processing system comprising:

10. the first information processing device includes a first display device; 10. The information processing system according to claim 9, wherein the output means displays the first character string present in the document image on the first display device in a manner different from a manner in which the second character string present in the document image is displayed when the first character string is not present in the document image.

11. the second information processing device further includes a second display device; 10. The information processing system according to claim 9, wherein the output means displays the first character string present in the document image on the second display device in a manner different from a manner in which the second character string present in the document image is displayed when the first character string is not present in the document image.

12. 10. The information processing system according to claim 9, wherein the generating means further generates a third instruction statement indicating whether the first character string exists in the document image.

Images