Information processing apparatus, information processing method, and program

Abstract

To provide an information processing apparatus, an information processing method, and a program for reducing the load in identifying the document type of a document image.SOLUTION: In an information processing system 100, an information processing server 103 includes: a character image analysis unit 154 configured to perform character recognition processing on an image in a document to be processed; an instruction message generation unit 155 configured to generate an instruction message for causing a large language model to reply a document type indicated by the document to be processed, and transmit the instruction message in order to obtain a reply to the instruction message from the large language model; and a reply receiving unit 156 configured to receive the reply to the instruction message from the large language model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to the process of extracting information from document images. [Background technology]

[0002] There is a technology to determine the document type of an input document image. One method for determining the document type (such as an invoice, estimate, or delivery note) is to use a trained model obtained by machine learning multiple document images.

[0003] Patent Document 1 discloses a method for determining the document type of an input document image using a classifier (trained model) capable of classifying into trained document types. In this Patent Document 1, the reliability of each of a plurality of document types is calculated for an input document image, and if there is a document type whose reliability exceeds a first threshold, that document type is designated as a classification destination. In addition, if there are a plurality of document types whose reliability is equal to or less than the first threshold and exceeds a second threshold, those document types are presented as classification destination candidates, and an instruction of the correct document type is accepted from the user. Furthermore, it is also disclosed that if there is no document type whose reliability exceeds the second threshold, it is possible to accept the designation of a new document type from the user. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-107455 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the method of Patent Document 1, if there is no document type whose reliability exceeds the second threshold, the user must input a new document type, which is time-consuming for the user to input a string indicating the new document type.

[0006] In addition, additional learning generally requires the collection of a large amount of similar data and a long learning time. Therefore, until the trained model can classify new document types, the user must input a new document type or select an appropriate one from multiple options and correct the document type each time classification fails, which places a heavy burden on the user. [Means for solving the problem]

[0007] The information processing device disclosed herein is characterized by having a character recognition means for performing character recognition processing on an image of a document to be processed, a generation means for generating an instruction statement for causing a large-scale language model to respond with a document type indicated by the document to be processed based on a result of the character recognition processing, a transmission means for transmitting the instruction statement to obtain an answer to the instruction statement from the large-scale language model, and a receiving means for receiving the answer to the instruction statement from the large-scale language model. Effect of the Invention

[0008] According to the present disclosure, it is possible to reduce the burden involved in identifying the document type of a document image. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an information processing system. [Diagram 2] FIG. 2 is a diagram showing an example of the hardware configuration of each device constituting the information processing system. [Diagram 3] FIG. 1 is a sequence diagram of an information processing system. [Figure 4] 1 is a flowchart illustrating details of a process for generating a machine learning model. [Diagram 5] 10 is a flowchart illustrating details of a process for displaying a document type to a user. [Figure 6] 11 is a flowchart illustrating details of a document analysis process using a machine learning model. [Figure 7] FIG. 4 is a diagram showing an example of a document image to be processed. [Figure 8]11 is a flowchart illustrating details of a process for obtaining an answer from a large-scale language model. [Figure 9] FIG. [Figure 10] FIG. 13 is a diagram showing an example of a document type confirmation screen. [Figure 11] FIG. 1 is a sequence diagram of an information processing system. [Figure 12] 11A and 11B are diagrams showing an option setting screen and a document type confirmation screen. [Figure 13] 10 is a flowchart illustrating details of a process for displaying a document type to a user. [Figure 14] 11 is a flowchart illustrating details of a process for obtaining an answer from a large-scale language model. [Figure 15] FIG. [Figure 16] FIG. 1 is a sequence diagram of an information processing system. [Figure 17] 6A and 6B are diagrams showing a setting screen and a document type confirmation screen. [Figure 18] 10 is a flowchart illustrating details of a process for displaying a document type to a user. [Figure 19] 11 is a flowchart illustrating details of a process for obtaining an answer from a large-scale language model. [Figure 20] FIG. 1 is a diagram showing an example of the configuration of an information processing system. [Figure 21] FIG. 1 is a sequence diagram of an information processing system. [Figure 22] 6A and 6B are diagrams showing a setting screen and a document type confirmation screen. [Figure 23] 10 is a flowchart illustrating details of a process for displaying a document type to a user. [Figure 24] 11 is a flowchart illustrating details of a document analysis process using a machine learning model. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the technology of the present disclosure will be described with reference to the drawings. Note that the components described in the following embodiments are merely examples and are not intended to limit the scope of the technology of the present disclosure.

[0011] <Embodiment 1> [About the configuration of information processing systems] 1 is a diagram showing an example of the configuration of an information processing system 100 according to the present embodiment. The information processing system 100 includes, for example, an image forming device 101, a learning device 102, an information processing server 103, and an external information processing server 105, and each device is connected to each other via a network 104. In the information processing system 100, the image forming device 101, the learning device 102, and the information processing server 103 may be configured to be connected to the network 104 in a multi-connection configuration rather than a single connection. For example, the information processing server 103 may be configured to include a first server device having high-speed computing resources and a second server device having large-capacity storage, and the first server device and the second server device may be configured to be connected to each other via the network 104.

[0012] The image forming apparatus 101 is realized by, for example, an MFP (Multi-Function Peripheral) having multiple functions such as printing, scanning, and faxing. The image forming apparatus 101 has at least an image acquisition unit 151 and a display control unit 159 as functional units.

[0013] The image forming apparatus 101 has a scanner device 206 (see FIG. 2). The scanner device 206 optically reads an original 111, which is a document with a character string printed on a recording medium such as paper, and the image acquisition unit 151 acquires a document image 113, which is a scanned image, by performing predetermined image processing on the resulting data. Alternatively, for example, the image acquisition unit 151 receives FAX data 112 transmitted from a FAX transmitter (not shown), and acquires the document image 113 by performing predetermined FAX image processing. The image acquisition unit 151 transmits the acquired document image 113 to the information processing server 103.

[0014] The display control unit 159 displays the information received from the information processing server 103 on the display device 210. (see FIG. 2) on the display. The display control unit 159 displays, for example, a document type confirmation screen 1000 (see FIG. 10) described later.

[0015] The image forming apparatus 101 may be realized by a PC (Personal Computer) or the like, other than an MFP equipped with scan and FAX functions. In this case, for example, a document image 113 such as PDF or JPEG generated by using a document creation application running on the PC may be transmitted to the information processing server 103 as a processing target.

[0016] The learning device 102 has the functional units of a learning data generation unit 152 and a learning unit 153 .

[0017] The learning data generator 152 generates learning data based on a plurality of document image samples 114 .

[0018] The learning unit 153 generates a trained model (machine learning model) by having the learning model learn based on the learning data generated by the learning data generation unit 152. In this embodiment, a document type determiner 115a is generated, which is a machine learning model that outputs, as a result, information indicating the type of document (document type) indicated by the document image to be processed. In addition, an item value extractor 115b is generated, which is a machine learning model that outputs, as a result, information indicating a character string (item value) corresponding to an item included in the document image to be processed.

[0019] The learning device 102 transmits the generated machine learning model to the information processing server 103 via the network 104. Details of the processing in the learning device 102 will be described later. The document type determiner 115a and the item value extractor 115b in this embodiment will be described as trained models generated by machine learning, but they may also perform determination using a rule-based algorithm and output the result.

[0020] The information processing server 103 is a device that processes the input document image 113 to be processed and transmits each processing result to the image forming device 101. The information processing server 103 has each of the functional units, a document image analysis unit 154, an instruction statement generation unit 155, a response receiving unit 156, a data management unit 157, and a display control unit 158.

[0021] The document image analysis unit 154 receives the document image 113 sent from the image forming apparatus 101 , executes OCR processing on the document image 113 , and acquires a group of character strings recognized from the document image 113 .

[0022] The document image analysis unit 154 uses the document type determiner 115a to determine which of preset document types such as an invoice, estimate, order form, or delivery note the document type indicated by the document image 113 is. The document image analysis unit 154 also uses the item value extractor 115b to extract character strings (item values) corresponding to items such as the title, document number, issue date, company name, and amount from the group of character strings recognized from the document image 113. The names of the items are referred to as item names. Also, the character strings corresponding to the items are referred to as item values.

[0023] The instruction statement generation unit 155 acquires item values ​​from the document image analysis unit 154, and generates an instruction statement by applying a predetermined item value to an instruction statement template prepared in advance. The instruction statement generation unit 155 transmits the instruction statement via the network 104 so that the instruction statement is input to the external information processing server 105. The instruction statement generation unit 155 may acquire, from the document image analysis unit 154, a group of character strings recognized from the document image 113 instead of or together with the item value as information to be applied to the instruction statement template. Details of the instruction statements will be described later.

[0024] The answer receiving unit 156 receives an answer to a directive sentence output by the large-scale language model 116 .

[0025] The data management unit 157 stores and manages in a storage unit the response to the instruction sentence generated by the large-scale language model 116. In addition, the data management unit 157 stores and manages in a storage unit the document type of the document image 113 confirmed by the user via the document type confirmation screen 1000.

[0026] The display control unit 158 ​​performs control for displaying to the user the document type determined by the document image analysis unit 154 and the answer to the instruction sentence obtained from the large-scale language model 116. For example, the display control unit 158 ​​generates information for displaying a document type confirmation screen 1000 (see FIG. 10) described later, and displays it on the display unit. When the user instructs to correct the document type determined by the document image analysis unit 154, the display control unit 158 ​​accepts the correction and generates the document type confirmation screen 1000 according to the user's correction.

[0027] The external information processing server 105 is a device that operates a large-scale language model 116. The large-scale language model 116 is a model that can generate sentences in an interactive format called an LLM (Large Language Model), and generates responses to input instructions (prompts). For example, Chat GPT (registered trademark) and Bard (pending trademark) are known as LLMs.

[0028] The large-scale language model 116 is accessed by an API (Application Programming Interface) via the network 104. The large-scale language model 116 outputs a response to a command input from the information processing server 103 as an output result. The external information processing server 105 may be configured to exist on a different system of the same vendor, or may be configured to exist on a system of an external vendor. The large-scale language model 116 may be configured to exist within the information processing server 103, or may be configured such that some functions or devices exist within the information processing server 103.

[0029] The network 104 is realized by a LAN, WAN, etc., and is a communication unit that connects the image forming device 101, the learning device 102, the information processing server 103, and the external information processing server 105 to each other, and transmits and receives data between the devices.

[0030] [Hardware configuration of image forming device] FIG. 2 is a diagram showing an example of the hardware configuration of the image forming apparatus 101, the learning apparatus 102, and the information processing server 103 included in the information processing system 100. As shown in FIG.

[0031] Fig. 2(a) is a diagram showing the hardware configuration of the image forming apparatus 101. As shown in Fig. 2(a), the image forming apparatus 101 has a CPU 201, a ROM 202, a RAM 204, a printer device 205, a scanner device 206, a document transport device 207, and a storage 208. It also has an input device 209, a display device 210, and an external interface 211. Each unit is connected to each other via a data bus 203.

[0032] The CPU 201 is a control unit that controls the overall operation of the image forming apparatus 101. The CPU 201 executes a startup program stored in a ROM 202 to start up the system of the image forming apparatus 101, and executes a control program stored in a storage 208 to realize functions of the image forming apparatus 101, such as printing, scanning, and faxing.

[0033] ROM 202 is a storage unit realized by a non-volatile memory, and stores a startup program for starting up image forming apparatus 101. Data bus 203 is a communication unit for transmitting and receiving data between devices constituting image forming apparatus 101. RAM 204 is a storage unit realized by a volatile memory, and is used as a work memory when CPU 201 executes the control program.

[0034] The printer device 205 is an image output device that prints and outputs a document image on a storage medium such as paper. The scanner device 206 is an image input device that optically reads a storage medium such as paper on which characters, diagrams, etc. are printed. Data obtained by the scanner device 206 reading is acquired as a document image. The document transport device 207 is realized by an ADF (Auto Document Feeder) or the like, detects a document placed on a document platen, and transports the detected document one sheet at a time to the scanner device 206. The storage 208 is a storage unit realized by an HDD (Hard Disk Drive) or the like, and stores the above-mentioned control programs and document images.

[0035] The input device 209 is an operation unit realized by a touch panel, hard keys, or the like, and accepts operation input from a user who uses the image forming apparatus 101. The display device 210 is a display unit realized by a liquid crystal display or the like, and displays and outputs a setting screen, etc. of the image forming apparatus 101 to the user. The external interface 211 is an interface that connects the image forming apparatus 101 and the network 104, and receives FAX data from a FAX transmitter (not shown) and transmits document images to the information processing server 103.

[0036] [Hardware configuration of the learning device] Fig. 2(b) is a diagram showing the hardware configuration of the learning device 102. As shown in Fig. 2(b), the learning device 102 has a CPU 231, a ROM 232, a RAM 234, a storage 235, an input device 236, a display device 237, an external interface 238, and a GPU 239. Each part is connected to each other via a data bus 233.

[0037] The CPU 231 is a control unit that controls the overall operation of the learning device 102. The CPU 231 executes a boot program stored in the ROM 232 to start up the system of the learning device 102, and executes a learning program stored in the storage 235 to generate a machine learning model for determining a document type or extracting an item value. The ROM 232 is a storage unit realized by a non-volatile memory, and stores a boot program that starts up the learning device 102. The data bus 233 is a communication unit for transmitting and receiving data between devices that constitute the learning device 102. The RAM 234 is a storage unit realized by a volatile memory, and is used as a work memory when the CPU 231 executes the learning program.

[0038] Storage 235 is a memory unit realized by a HDD (Hard Disk Drive) or the like, and stores the learning program and document image samples described above. Input device 236 is an operation unit realized by a mouse, keyboard, or the like, and accepts operation input from an engineer who controls learning device 102. Display device 237 is a display unit realized by a liquid crystal display or the like, and displays the setting screen of learning device 102, etc., to the engineer.

[0039] The external interface 238 is an interface that connects the learning device 102 and the network 104, and receives the document image sample 114 from the outside and transmits the machine learning model to the information processing server 103. The GPU 239 is a calculation unit configured with an image processing processor. The GPU 239 executes calculations for generating a machine learning model based on a group of character strings included in a given document image, for example, in accordance with a control command given from the CPU 231.

[0040] Each functional unit included in the learning device 102 shown in Fig. 1 is realized by the CPU 231 executing a predetermined program, but is not limited to this. Other hardware such as a GPU 239 for accelerating calculations or an FPGA (Field Programmable Gate Array) (not shown) may be used. Each functional unit may be realized by cooperation between software and hardware such as a dedicated IC, or some or all of the functions may be realized only by hardware.

[0041] [Hardware configuration of information processing server] Fig. 2(c) is a diagram showing a hardware configuration of the information processing server 103. As shown in Fig. 2(c), the information processing server 103 has a CPU 261, a ROM 262, a RAM 264, a storage 265, an input device 266, a display device 267, and an external interface 268, and each part is connected to each other via a data bus 263.

[0042] The CPU 261 is a control unit that controls the overall operation of the information processing server 103. The CPU 261 executes a boot program stored in a ROM 262 to start up the system of the information processing server 103, and executes an information processing program stored in a storage 265 to perform information processing such as character recognition (OCR) and information extraction.

[0043] The ROM 262 is a storage unit realized by a non-volatile memory, and stores a boot program that starts up the information processing server 103. The data bus 263 is a communication unit for transmitting and receiving data between devices constituting the information processing server 103. The RAM 264 is a storage unit realized by a volatile memory, and is used as a work memory when the CPU 261 executes the information processing program. The storage 265 is a storage unit realized by an HDD (Hard Disk Drive) or the like, and stores the above-mentioned information processing program, machine learning model, document image, determined document type, extracted item value, etc.

[0044] The input device 266 is an operation unit realized by a mouse, a keyboard, or the like, and accepts operation input to the information processing server 103 from a user or an engineer who uses the information processing server 103. The display device 267 is a display unit realized by a liquid crystal display or the like, and displays and outputs a setting screen of the information processing server 103 to a user or an engineer who uses the information processing server 103.

[0045] The external interface 268 is an interface that connects the information processing server 103 and the network 104, and receives a machine learning model from the learning device 102 and a document image from the image forming device 101.

[0046] Each functional unit included in the information processing server 103 in Fig. 1 is realized by the CPU 261 executing a predetermined program, but is not limited to this. Other hardware such as a GPU (Graphics Processing Unit) or an FPGA (Field Programmable Gate Array) for accelerating calculations may also be used. Each functional unit may be realized by cooperation between software and hardware such as a dedicated IC, or some or all of the functions may be realized only by hardware.

[0047] [Sequence of trained model generation] 3 is a diagram showing a sequence of the information processing system 100. The symbol "S" in the description of each process means a step in the sequence, and the same applies to the following flowcharts. For convenience of explanation, the operation of a user or engineer is also explained using steps.

[0048] FIG. 3(a) shows a flow of development by an engineer of the document type determiner 115a and the item value extractor 115b, which are machine learning models.

[0049] In S301, an engineer related to the information processing system 100 inputs a plurality of document image samples 114, which are samples of images representing documents, to the learning device 102. The document image samples 114 are document images corresponding to document types such as invoices, estimates, purchase orders, and delivery notes.

[0050] In S302, the learning data generation unit 152 of the learning device 102 generates first learning data based on the document image sample 114, and the learning unit 153 generates a document type classifier 115a, which is a machine learning model, by performing machine learning using the first learning data.

[0051] In S303, the learning device 102 transmits the generated document-type determinator 115a to the information processing server 103. In the information processing server 103, the storage 265 stores the document-type determinator 115a.

[0052] In S304, the learning data generation unit 152 of the learning device 102 generates second learning data based on the document image sample 114, and the learning unit 153 generates an item value extractor 115b, which is a machine learning model, by performing machine learning using the second learning data.

[0053] In S305, the learning device 102 transmits the generated item value extractor 115b to the information processing server 103. In the information processing server 103, the item value extractor 115b is stored in the storage 265. Details of S302 to S305 in FIG. 3(a) will be described later with reference to FIG.

[0054] In S306, the engineer inputs the document type of the document image sample input in S301 as a "document type option" to the information processing server 103. In this embodiment, the document type option set by the engineer is not set again by the user. The document type option will be described later.

[0055] [Document type determination process sequence] Fig. 3(b) is a diagram illustrating the flow of processing for determining the document type indicated by the document image to be processed. Details of S313 to S316 and S318, which are the processing of the information processing server 103 in Fig. 3(b), will be described later with reference to Fig. 5.

[0056] In S311, the user places a paper document (original) on the image forming apparatus 101 and instructs the image forming apparatus 101 to scan the document.

[0057] In S312, the scanner device 206 of the image forming apparatus 101 reads the placed paper document, and the image acquisition unit 151 generates a document image that is an image representing the scanned document. Then, the image acquisition unit 151 transmits the generated document image to the information processing server 103 as a document image to be processed.

[0058] The document image analysis unit 154 of the information processing server 103 executes character recognition processing (OCR processing) on ​​the document image to be processed transmitted in S312, and acquires a group of character strings recognized from the document image.

[0059] In S313, the document image analysis unit 154 inputs data of the character string group of the document image to be processed to the document type determiner 115a, and determines the document type of the document image to be processed using the output result.

[0060] In S314, the document image analysis unit 154 inputs data of a group of character strings recognized from the document image to be processed to the item value extractor 115b, and extracts character strings (item values) corresponding to each item from the group of character strings.

[0061] In S315, the instruction statement generator 155 generates an instruction statement using the item values ​​extracted in S314. Details of the instruction statement will be described later.

[0062] In S316, the information processing server 103 transmits the instruction generated in S315 to the external information processing server 105.

[0063] In S317, the external information processing server 105 receives the instruction sent in S316, and causes the large-scale language model 116 to generate a response to the received instruction. The response to the instruction is returned to the information processing server 103.

[0064] In S318, the display control unit 158 ​​of the information processing server 103 converts the document type determined in S313 based on the output result of the document type determiner 115a and the result of the response to the instruction statement transmitted in S317 into information for presenting to the user. The display control unit 158 ​​transmits the converted information to the image forming apparatus 101. The display control unit 159 of the image forming apparatus 101 presents the processing result to the user by displaying a document type confirmation screen 1000 (see FIG. 10).

[0065] [Machine learning model generation process] Fig. 4 is a flowchart explaining the details of the generation process of the document type determiner 115a and the item value extractor 115b in S302 to S305 in Fig. 3(a) executed by the learning device 102. The series of processes shown in the flowchart in Fig. 4 will be explained as being performed by the CPU 231 of the learning device 102 expanding program code stored in the ROM 232 or storage 235 into the RAM 234 and executing it. Some or all of the functions of the steps in Fig. 4 may be executed by the GPU 239, or may be realized by hardware such as an ASIC or electronic circuit.

[0066] In S401, the CPU 231 acquires a plurality of document image samples input by an engineer in S301 of Fig. 3. For example, document image samples are acquired that are generally called semi-standard forms such as invoices, estimates, and purchase orders, and are created with different layouts for each issuing company.

[0067] In S402, the CPU 231 executes block selection (BS) processing and character recognition processing (OCR processing) on ​​the document image sample acquired in S401, and acquires a group of character strings recognized from the document image sample.

[0068] Block selection (BS) processing is a process that selects block areas so that a document image is divided into object units that make up the document image, and determines the attributes of each block area. Specifically, it is a process that determines attributes such as text, photos, and diagrams, and divides the document image into block areas with different attributes. Block selection (BS) processing can be realized using a known area determination technique.

[0069] The data of the character string group obtained as a result of the OCR process may be, for example, word-delimited character strings arranged in the document image with spaces or lines, which constitute the document image, read out continuously in a predetermined read order based on the arrangement information.Furthermore, the data of the character string group obtained may be, for example, word-delimited character strings divided into sentences constituting the document image using a morphological analysis technique, which are read out continuously in a predetermined read order based on the arrangement information.

[0070] In S403, the CPU 231 acquires a first correct answer label indicating a document type for each of the multiple document image samples acquired in S401. The document types are, for example, an invoice, an estimate, a purchase order, and a delivery note. The first correct answer label may be one that an engineer manually assigns to the document sample, or may be a document type that is automatically obtained by inputting the document image sample into a model that outputs a generated document type. The CPU 231 generates first learning data, which is data for machine learning that is a combination of a group of character strings recognized from the document image sample and the first correct answer label of the document image sample. The first learning data corresponding to each of the multiple document samples is generated.

[0071] In S404, the CPU 231 acquires a second correct answer label indicating which item the character string to be extracted included in the character string group acquired in S402 corresponds to. The items are, for example, a title, a document number, a publication date, a company name, a total amount, etc. The second correct answer label may be assigned manually by an engineer, or may be assigned automatically by inputting a document image sample into a model that extracts generated item values. Then, the CPU 231 generates second learning data that is a combination of a character string group recognized from the document image sample and data in which the second correct answer label is assigned to a character string that is an item value of the character string group. Second learning data corresponding to each of a plurality of document samples is generated.

[0072] In S405, the CPU 231 generates the document type determiner 115a, which is a machine learning model, by machine learning using the first learning data. The document type determiner 115a is a trained model that, when data on the feature amounts of a group of character strings included in a document image to be processed is input, outputs information on the document type of the document image to be processed from among trained document types.

[0073] Assume that an engineer inputs document image samples of five document types, "invoice, estimate, order, delivery note, and contract," into the learning device 102. In this case, in this step, a document type determiner 115a is generated that outputs information on the document type of the document image to be processed from the five document types, "invoice, estimate, order, delivery note, and contract." In this case, in S306 of FIG. 3, the engineer sets the five document types, "invoice, estimate, order, delivery note, and contract," as document type options. In this way, the "document type options" in this embodiment indicate the learned document types that the document type determiner 115a can output.

[0074] The document type determiner 115a of this embodiment is, for example, a trained model trained to output a label corresponding to the first correct label. The document type determiner 115a is generated by training a learning model prepared to output a predetermined document type label for the feature amount of an input string group as an inference result.

[0075] In S406, the CPU 231 generates the item value extractor 115b, which is a machine learning model, by machine learning using the second learning data. The item value extractor 115b is a trained model that is trained to output information on a character string (item value) corresponding to an item from among the character strings when data on the feature amount of a character string group included in a document image to be processed is input.

[0076] The item value extractor 115b of this embodiment is, for example, a trained model trained to output a label corresponding to the second correct label. The item value extractor 115b is generated by training a prepared learning model so that, when the feature quantities of a string group are input, the item value extractor 115b outputs item labels for strings to be extracted and does not output labels for strings that are not to be extracted.

[0077] It should be noted that the document type determiner 115a and the item value extractor 115b may be generated by a known method. For example, a feature vector indicating the feature amount of a character string converted using Word2Vec, fastText, BERT, XLNet, ALBERT, or the like, and the position coordinates of the character string arranged in the document image may be used. Specifically, for example, a BERT language model that has been trained on a general text (e.g., the entire article on Wikipedia) may be used to convert a single character string data into a feature vector expressed by a 768-dimensional numerical value. The learning model may be a logistic regression, decision tree, random forest, support vector machine, neural network, or the like, which are generally known as machine learning algorithms. Specifically, for example, a predefined document type and an inference result of the label of the item information may be output according to the output value of the fully connected layer of a neural network to which a feature vector output by the BERT language model is input.

[0078] In S407, the CPU 231 transmits the generated document-type determining unit 115a to the information processing server 103. Then, the document-type determining unit 115a is stored in the storage 265 in the information processing server 103.

[0079] In S408, the CPU 231 transmits the generated item value extractor 115b to the information processing server 103. Then, the item value extractor 115b is stored in the storage 265 in the information processing server 103.

[0080] In the above description, the document type determiner 115a and the item value extractor 115b are generated as independent trained models. Alternatively, it is possible to train the learning model so as to generate a single trained model that simultaneously performs document type determination and item value extraction.

[0081] [Document type display processing] Fig. 5 is a flowchart for explaining the details of S313 to S316 and S318, which are processes performed by the information processing server 103 to determine the document type indicated by the document image to be processed and display the determined document type to the user. The series of processes shown in the flowchart in Fig. 5 are performed by the CPU 261 of the information processing server 103 by loading program code stored in the ROM 262 or storage 265 into the RAM 264 and executing it. Also, some or all of the functions of the steps in Fig. 5 may be realized by hardware such as an ASIC or an electronic circuit.

[0082] In S501, the CPU 261 performs document analysis on the document image to be processed using the document type determiner 115a and the item value extractor 115b, which are machine learning models.

[0083] FIG. 6 is a flowchart illustrating the details of the process of S501.

[0084] In S601, the CPU 261 acquires the document type determiner 115a, which is a machine learning model, transmitted from the learning device 102.

[0085] In S602, the CPU 261 acquires the item value extractor 115b, which is a machine learning model, transmitted from the learning device 102.

[0086] In S603, the CPU 261 acquires the document image transmitted from the image forming apparatus 101. The document image acquired in S603 becomes the document image to be processed.

[0087] Fig. 7 shows an example of a document image to be processed acquired in S603. Document image 700 in Fig. 7(a) is an example of a document image with document title 701 "Purchase Order". Document image 710 in Fig. 7(b) is an example of a document image with document title 711 "Donation Receipt".

[0088] In S604, the CPU 261 executes block selection (BS) processing and OCR processing, which are the same processing as that performed in S402, on the document image to be processed, and acquires a group of character strings recognized from the document image to be processed.

[0089] In S605, the CPU 261 inputs data of the character string group recognized from the document image to be processed acquired in S604 to the document type determiner 115a acquired in S601. The CPU 261 determines the document type indicated by the document image to be processed based on the output result of the document type determiner 115a.

[0090] As described above, the document type options set in S306 indicate the document types that the document type determiner 115a can output. For example, assume that data of a group of character strings recognized from the document image 700 of FIG. 7(a) whose title is "purchase order" is input to the document type determiner 115a whose document type options are "invoice, estimate, order, delivery note, contract". Although "purchase order" is not included in the document type options, it can be said that "purchase order" is another expression of "order order" included in the document type options. Therefore, the document type determiner 115a has learned to output the document image of "purchase order" as "order order", which is a document type included in the document type options. Therefore, the document type determiner 115a outputs information indicating "order order" from the document type options as the document type of the document image 700. Therefore, in S605, the CPU 261 can determine the document type of the document image 700 as "order order" using the document type determiner 115a. Since the appropriate document type indicated by the document image 700 is included in the document type options, the document image 700 is a document image whose document type can be appropriately determined using the document type determiner 115a.

[0091] 7B is an example of a document image of a new document type, in which the appropriate document type indicated by the document image 710 is not included in the document type options "invoice, estimate, order, delivery note, contract." Even when such a document image 710 becomes a document image to be processed, the document type determiner 115a outputs a document type from the document type options "invoice, estimate, order, delivery note, contract" as the document type corresponding to the document image 710.

[0092] Document type determination can be generalized as "category classification". When performing category classification using a machine learning model, even if data other than the previously assumed category (OOD: Out Of Distribution) is input, the data may be erroneously classified into one of the previously assumed categories. However, it takes time for the user to check whether the document type output by the machine learning model is correct. Therefore, in this embodiment, the large-scale language model 116 is also made to respond with the document type of the document image to be processed, so that a warning is issued to the user when a document image of a new document type is input. Details will be described later.

[0093] In S606, the CPU 261 inputs the data of the character string group acquired in S604 to the item value extractor 115b acquired in S602. Then, based on the output result of the item value extractor 115b, the CPU 261 extracts character strings (item values) corresponding to each item from the character string group recognized from the document image to be processed.

[0094] Next, the CPU 261 acquires a character string corresponding to a predetermined item from the character strings corresponding to the extracted items as a "character string in document image." In this embodiment, the predetermined item is assumed to be a "title." Therefore, as a result of the process of S606, a character string corresponding to the title included in the document image to be processed is acquired.

[0095] For example, when the document image 700 of FIG. 7(a) is acquired in S603, data of the character string group of the document image 700 is input to the item value extractor 115b. The item value extractor 115b outputs information indicating that the character string corresponding to the item "Title" is "Purchase Order". In this case, in S606, the character string "Purchase Order" is acquired as the "character string within the document image". Also, when the document image 710 of FIG. 7(b) is acquired in S603, the item value extractor 115b outputs information indicating that the character string corresponding to the item "Title" is "Donation Receipt". In this case, in S606, the character string "Donation Receipt" is acquired as the "character string within the document image".

[0096] This is the end of the flow chart of Fig. 6, and therefore the process of S501 ends. Then, the process proceeds to S502 in the flow chart of Fig. 5.

[0097] In S502 of FIG. 5, a process of generating a prompt and acquiring an answer to the prompt from the large-scale language model 116 is performed.

[0098] FIG. 8 is a flowchart illustrating the details of the process of S502.

[0099] In S801, the CPU 261 acquires the document type options set by the engineer in S306. As described above, the document type options are document types output by the document type determiner 115a, which is a machine learning model, and are set in advance by the engineer. For example, if five document types, "invoice, estimate, order, delivery note, and contract," are set as document type options, these five document types are acquired. Note that, assuming that the document type to be output changes depending on the supported language, region, job type, business content, etc., multiple sets of document type options may be prepared and switched between.

[0100] In S802, the CPU 261 acquires the "character string in document image" acquired as a result of the processing in S606 in Fig. 6. In this embodiment, as described above, a character string corresponding to a title included in the document image to be processed is acquired.

[0101] In S803, the CPU 261 obtains an instruction statement template from the storage 265. The instruction statement template prepared in advance may be one prepared by an engineer as a preset, one prepared by a user, or one modified or added to by the system or the user.

[0102] In S804, the CPU 261 generates an instruction sentence by applying the document type option acquired in S801 and the "character string in the document image" acquired in S802 to the instruction sentence template acquired in S803.

[0103] Fig. 9 is a diagram for explaining an instruction text. Fig. 9(a) shows document type options 901 consisting of "invoice, estimate, order, delivery note, contract". Fig. 9 will be explained assuming that the document type options 901 in Fig. 9(a) are acquired in S801.

[0104] 9(b) and 9(d) are diagrams showing an example of an instruction generated in S804. An instruction 910 in FIG. 9(b) is an example of an instruction generated when the document image 700 in FIG. 7(a) is acquired as the document image to be processed. The instruction generated in S804 will be explained using the instruction 910. The instruction 910 includes text areas 915, 916, and 917, and a document type option character string group 919. The text area 915 includes a title character string 913 to which a "character string in document image" extracted from the document image to be processed is applied.

[0105] The instruction template acquired in S803 is the instruction 910 with the title string 913 and the document type option string group 919 removed.

[0106] In S804, the CPU 261 applies the "character string in document image" acquired in S802 as the title character string 913 in the text area 915 to the instruction statement template acquired in S803. For example, since the title character string 913 in the instruction statement template is blank, the character string "purchase order", which is the "character string in document image" acquired from the document image 700, is applied to the title character string 913 of the instruction statement template. Furthermore, the instruction statement template does not include the document type option character string group 919. Therefore, the character string group of "invoice, estimate, purchase order, delivery note, contract" constituting the document type option 901 acquired in S801 is applied to the instruction statement template as the document type option character string group 919. In this manner, the instruction statement 910 is generated. In this manner, the document type option character string group 919 included in the instruction statement indicates the document type included in the document type option acquired in S801.

[0107] Note that the content of the instruction template may be switched for each document type option acquired in S801. In this case, in S803, an instruction template may be acquired in which the document type option acquired in S801 is described in advance as the document type option string group 919. In this case, in S804, the CPU 261 may apply only the "character string in document image" acquired in S802.

[0108] Furthermore, the CPU 261 may switch the content of the instruction template for each language or destination of the document image to be processed. Furthermore, the instruction template may be one prepared in advance by an engineer or a user, or may be one prepared in advance and edited by the user.

[0109] The instruction text 910 includes a text area 917 that contains questions 911 and 912 .

[0110] Question 911 is a question for obtaining an answer as to whether or not the document type of the document image to be processed is any of the document types included in document type selection options 901. Specifically, question 911 is a question for getting an answer of the document type if the document including title string 913 corresponds to any of the document types included in document type selection options 901, and for getting an answer of "not applicable" if it does not correspond.

[0111] Question 912 is a question that prompts a respondent to answer an appropriate document type candidate when the document type of the document image to be processed does not correspond to the document types included in document type selection options 901. Specifically, question 912 is a question that prompts a respondent to answer a document type other than the document types included in document type selection options 901 as a document type candidate when a document including a title string 913 does not correspond to any of the document types included in document type selection options 901.

[0112] In S805, the CPU 261 performs a process of inputting the instruction statement generated in S804 to the large-scale language model 116. For example, the CPU 261 transmits the instruction statement 910 of FIG. 9B to the external information processing server 105 so that the instruction statement 910 is input to the large-scale language model 116.

[0113] In S806, the CPU 261 receives a response to the instruction statement input in S805 from the large-scale language model 116. The response from the large-scale language model 116 to the instruction statement includes answers to the questions 911 and 912 included in the instruction statement.

[0114] For example, when an instruction sentence 910 in Fig. 9(b) is input to the large-scale language model 116, an output result 920 in Fig. 9(c) is obtained from the large-scale language model 116. The output result 920 includes an answer 921 to a question 911 and an answer 922 to a question 912. In this manner, in this embodiment, the large-scale language model 116 is also made to answer the document type of the document image to be processed using information on character strings included in the document image to be processed.

[0115] In this embodiment, the "character string in the document image" is described as a character string corresponding to the "title" among the item values ​​extracted from the document image. The "character string in the document image" is a character string that serves as a criterion for the large-scale language model 116 to answer the document type of the document image to be processed. The character string corresponding to the title is a character string that summarizes the contents of the document, and is highly likely to contain information that can answer the document type. Therefore, in this embodiment, the character string corresponding to the title is described as an example of the "character string in the document image" that is applied to the instruction statement template.

[0116] However, the "character string in the document image" is not limited to the character string corresponding to the title. The "character string in the document image" may be any character string that has the information necessary for the large-scale language model 116 to answer the document type. For example, there are cases where a character string corresponding to the title does not exist in the document image. Also, there are cases where the document type cannot be appropriately answered from the character string corresponding to the title depending on the region, the corresponding industry, and the document type. For this reason, the information processing server 103 may switch the character string to be acquired as the "character string in the document image" depending on the region, the corresponding industry, and the document type.

[0117] The "character string in the document image" may be a group of multiple character strings included in the document image to be processed. For example, the "character string in the document image" may be a group of character strings included in the document image to be processed, or a combination of a character string corresponding to a specific item and a group of character strings included in the document image to be processed. For example, a phrase including a character string corresponding to an item that may be a hint for determining the document type and its prefix character string may be acquired as the "character string in the document image". For example, "invoice date: 10 / 10 / 2022" including a character string corresponding to the invoice date and a prefix character string of the invoice date may be acquired as the "character string in the document image". For an item that is included in both the purchase order and the invoice, such as the purchase order number, a character string that combines the invoice number and the purchase order number, and in the case where no value is stated, a character string that expresses that no value is stated may be acquired as the "character string in the document image".

[0118] 8 ends, and the process of S502 ends. Then, the process proceeds to S503 in the flowchart of FIG.

[0119] In S503, the CPU 261 performs a process of notifying the user of the document type indicated by the document image to be processed, which was determined in S501 based on the output result of the document type determiner 115a.

[0120] 10 is a diagram showing a document type confirmation screen 1000, which is an example of a confirmation screen for presenting the document type determined by the information processing server 103 to the user. In this embodiment, the document type confirmation screen 1000 is described as being displayed on the display device 210 of the image forming apparatus 101. For this reason, the CPU 261 transmits information for displaying the document type confirmation screen 1000 to the image forming apparatus 101. The document type confirmation screen 1000 may be displayed on the display device 267 of the information processing server 103.

[0121] As a result of S503, the document type determined in S501 based on the output result of the document type determiner 115a is displayed in a document type display area 1001 of a document type confirmation screen 1000. A preview display area 1010 displays a preview of the document image to be processed.

[0122] The document type confirmation screen 1000 in Fig. 10(a) is a diagram showing an example of a confirmation screen displayed when the document image 700 in Fig. 7(a) is the processing target. When the document image 700 is the processing target, the document type determined in S501 based on the output result of the document type determiner 115a is "purchase order". Therefore, the CPU 261 performs display control so that "purchase order" is displayed by default in the document type display area 1001 of the document type confirmation screen 1000.

[0123] The document type confirmation screen 1000 in FIG. 10(b)-(d) is a diagram showing an example of a confirmation screen displayed when the document image 710 in FIG. 7(b) is the processing target. Even when a document image of a new document type "receipt" such as the document image 710 is the processing target, the document type determiner 115a outputs the document type from the document type options 901. When the character string group of the document image 710 is input, the document type determiner 115a tries to determine one of the document type options 901, and determines the document type of the document image 710 as "delivery note" because of the common point that it is related to the receipt of goods or money. Therefore, in S501, an incorrect document type such as "delivery note" is determined as the document type of the document image 710. Even in this case, in S503, the CPU 261 displays the incorrect document type "delivery note" as the document type of the document image 710 by default in the document type display area 1001 of the document type confirmation screen 1000.

[0124] In S504 , the CPU 261 determines whether the answer to the instruction sentence obtained from the large-scale language model 116 is an answer that the document type of the document image to be processed does not correspond to the document types included in the document type options 901 .

[0125] The question 911 included in the instruction statements 910 and 930 is a question that causes the large-scale language model 116 to answer "not applicable" if the document type of the document image to be processed does not correspond to any of the document types included in the document type options 901. For this reason, in S504, for example, it is determined whether the answer to the question 911 from the large-scale language model 116 is "not applicable."

[0126] An output result 940 in Fig. 9(e) is an output result of the large-scale language model 116 for the instruction sentence 930 in Fig. 9(d). The instruction sentence 930 in Fig. 9(d) is an example of an instruction sentence generated when the document image 710 in Fig. 7(b) is the processing target, and the title of the document image 710, "Donation Receipt", is assigned as the title string 914.

[0127] As described above, the document image 710 is an example of a document image of a new document type whose appropriate document type is not included in the document type options 901. Among the answers included in the output result 940 in FIG. 9(e), the answer 941 to the question 911 of the instruction statement 930 is "not applicable." That is, the large-scale language model 116 has answered that the document type of the document image 710 to be processed is not included in the document type options 901. In this way, when "not applicable" is obtained as the answer to the question 911 from the large-scale language model 116 (YES in S504), the process transitions to S505.

[0128] In S505, the CPU 261 notifies the user that the document type of the document image to be processed may not correspond to the document types included in the document type options 901. In other words, the CPU 261 notifies the user that the document type displayed in the document type display area 1001 as a result of S503 may be incorrect. For example, the CPU 261 outputs information for displaying a warning to the image forming apparatus 101, and causes the warning to be displayed on the display device 210 of the image forming apparatus 101.

[0129] 10(b)-(c), for example, in S505, the CPU 261 outputs information for displaying the alerts 1016 and 1036 to the image forming apparatus 101 so that the alerts 1016 and 1036 are displayed on the document type confirmation screen 1000. By displaying the alerts in this manner, it is possible to notify the user that the document type displayed in the document type display area 1001 may be a type that is not included in the document type selection options 901 shown in Fig. 9(a). Note that the process may proceed to S506 without performing S505.

[0130] As described above, when a document image corresponding to a new document type not included in the document type options 901 is processed, the document type determiner 115a outputs an incorrect document type. On the other hand, the large-scale language model 116 is not a model that generates an answer within a predetermined range of document types. Therefore, even when a document image of a new document type is processed, a response can be obtained from the large-scale language model 116 that the document type does not correspond to the document type included in the document type options 901. Therefore, it is possible to notify the user that the document type determined based on the output result of the document type determiner 115a, which is the document type presented to the user in S503, may be incorrect.

[0131] In S506, the CPU 261 notifies the user of the candidate document types indicated by the document image to be processed returned by the large-scale language model 116. For example, information for displaying the candidate document types is output to the display device 267.

[0132] Answers 942 and 943 included in the output result 940 of the large-scale language model 116 in FIG. 9E are answers to the question 912 in the instruction sentence 930. When the answer to the question 911 is "not applicable," the question 912 is a question that prompts the user to answer a document type other than the document types included in the document type options 901 as a candidate for the document type of the document image to be processed. In the case of the output result 940 in FIG. 9E, the large-scale language model 116 answers two document types, "receipt" and "receipt," as candidates for the document type of the document image 710 whose title is "Donation Receipt." In this case, the CPU 261 outputs information for display to the image forming apparatus 101 so that "receipt" and "receipt" are displayed on the document type confirmation screen 1000 as candidates for the document type of the document image 710.

[0133] For example, the CPU 261 displays a drop-down list below a document type display area 1001 that displays the document type determined based on the output result of the document type determiner 115a, as shown in a document type confirmation screen 1000 in Fig. 10(b). The CPU 261 displays "receipt" and "invoice", which are candidates for the document type answered by the large-scale language model 116, in cells 1042 and 1043 included in the drop-down list.

[0134] 10(c), buttons 1052, 1053 corresponding to the candidate document types answered by the large-scale language model 116 may be displayed side by side below the document type display area 1001. Alternatively, for example, when the large-scale language model 116 answers that there is one candidate document type, the candidate document type may be displayed by incorporating text 1062 indicating the candidate document type in the alert 1036 as shown in FIG.

[0135] On the other hand, when the large-scale language model 116 answers the question 911 with a document type included in the document type options 901 (NO in S504), as in answer 921 to the question 911 in FIG. 9(c), the flowchart in FIG. 5 ends.

[0136] A document type confirmation screen 1000 in Fig. 10(a) is an example of a confirmation screen when the answer 921 to the question 911 in Fig. 9(c) is "purchase order" and the answer to the question 911 is not "not applicable." If the document type of the document image to be processed is appropriately determined in S501, a confirmation screen is displayed in which no alert or candidate document types are displayed, as shown in Fig. 10(a).

[0137] It is assumed that the document type determined based on the output result of the document type determiner 115a in S501 is "purchase order". Also, it is assumed that the document type answered by the large-scale language model to the question 911 in S502 is "contract". In this way, even if the large-scale language model 116 answers a document type included in the document type options 901, the document type answered by the large-scale language model 116 may differ from the document type determined based on the output result of the document type determiner 115a. In such a case, NO may be determined in S504 and S505 to S506 may be performed. In this case, the document type answered by the large-scale language model 116 to the question 911 may be displayed as a candidate document type on the document type confirmation screen 1000.

[0138] [Regarding correction of document type] When the user modifies the character string presented in S503 as the document type indicated by the document image to be processed, the user may manually input the modified document type. Alternatively, the user may select the modified document type from the document type candidates answered by the large-scale language model 116 and displayed on the document type confirmation screen 1000 of Fig. 10. A method for the user to modify the document type will be described with reference to Fig. 10. Also, the document type confirmation screen 1000 will be described as being displayed on the display device 210 of the image forming apparatus 101 under the control of the CPU 261 of the information processing server 103, and the user's operation will be acquired.

[0139] A method in which a user manually corrects the document type will be described with reference to Fig. 10(a). In the document type display area 1001, "order form", which is the document type determined using the output result of the document type determiner 115a, is displayed. When it is detected that the user has pressed an edit button 1002, the document type display area 1001 is changed to an editable state such as a text box, and an input cursor 1003 is displayed. Using the input cursor 1003, the user can manually correct the text indicating the document type displayed in the document type display area 1001.

[0140] Next, a method for selecting a corrected document type from the document type candidates proposed by the large-scale language model 116 will be described with reference to FIGS. 10(b) to (d).

[0141] When it is detected that the user has pressed the list display button 1015 on the document type confirmation screen 1000 in FIG. 10(b), a drop-down list containing candidate document types proposed by the large-scale language model 116 is displayed. Suppose the user uses the cursor 1019 to select, for example, cell 1042 in which a candidate document type, "receipt," is displayed. In this case, the selection of "receipt" is detected, and the "delivery note" displayed in the document type display area 1001 is changed to "receipt," and the "delivery note" displayed in the document type display area 1001 is overwritten so that it is displayed as a candidate document type.

[0142] A document type confirmation screen 1000 in Fig. 10(c) is a screen in which the drop-down list for displaying candidate document types in Fig. 10(b) has been changed to buttons 1052 and 1053. In the case of Fig. 10(c), the user can view candidate document types proposed by the large-scale language model 116 without pressing the list display button 1015. The operation in Fig. 10(c) is the same as the operation in Fig. 10(b) except that the cells of the drop-down list are replaced with buttons.

[0143] The document type confirmation screen 1000 in FIG. 10(d) is a confirmation screen when there is only one document type candidate answered by the large-scale language model 116. When the user's press of the Yes button 1037 is detected, the display in the document type display area 1001 is changed to "receipt", which is a document type candidate. Furthermore, the "delivery note" displayed in the document type display area 1001 is saved in the memory of the information processing server 103. When the user's press of the No button 1038 is detected, the previously displayed "delivery note" is retrieved from the memory of the information processing server 103, and the "delivery note" is displayed in the document type display area 1001. Note that if the No button 1038 is pressed before the Yes button 1037 is pressed, no action need be taken.

[0144] 10(a) to 10(d), the user can confirm the document type for the document image to be processed as the document type displayed in the document type display area 1001. The confirmed document type is managed by, for example, the data management unit 157 of the information processing server 103.

[0145] As described above, in this embodiment, even if a document image of a new document type other than the document type that the document type determiner 115a can output is input as a processing target, the large-scale language model can be made to respond with an appropriate document type. Therefore, even if a document image of a new document type becomes a processing target, a new document type candidate can be presented to the user before the document type determiner 115a performs additional learning. For example, consider a case where the document types that can be output are "invoice, estimate, order, delivery note, contract" and a document image of "receipt" which is none of these is input by the user. In this case, the document type determiner 115a erroneously outputs "delivery note" as the document type of the document image, but can suggest to the user that it may be "receipt" based on the response of the large-scale language model. Similarly, when the document type determiner 115a performs rule-based determination and outputs the document type, when a document image of a new document type becomes a processing target, an incorrect document type may be determined. In this case, according to this embodiment, a document type candidate can be suggested to the user based on the response of the large-scale language model.

[0146] In addition, in this embodiment, since the instruction sentence is generated using the character string included in the document image to be processed, it is possible to prevent the large-scale language model from answering a document type that is unlikely to be the document type indicated by the document image to be processed. Therefore, even if the document type is erroneously determined to be "delivery note" based on the output result of the document type determiner 115a and the erroneously determined "delivery note" is presented to the user, the document type "receipt", which is likely to be appropriate, can be presented as a candidate document type. Therefore, according to this embodiment, even if a document image of a new document type is input as a processing target, it is possible to prevent the user from having to manually input the correct document type or to search for the correct document type from among many options. In other words, according to this embodiment, it is possible to reduce the effort required for the user to correct the document type.

[0147] Note that the document type answered by the large-scale language model 116 may be displayed in the document type display area 1001 by default, rather than being displayed as a candidate. In this case, the document type determined based on the output result of the document type determiner 115a may be displayed as a candidate, or only the answer of the large-scale language model 116 may be displayed. For example, it may be possible to set which output result of the large-scale language model 116 or the document type determiner 115a is to be preferentially displayed. Then, when the large-scale language model 116 is set to be preferentially displayed, the document type answered by the large-scale language model 116 may be displayed in the document type display area 1001 by default.

[0148] In addition, the "character string in document image" written in the instruction template may be a group of all character strings obtained as a result of OCR processing of the document image to be processed. In this case, item value extraction using the item value extractor 115b is not necessary. In this case, the step of generating the item value extractor 115b and the process of extracting item values ​​using the item value extractor 115b may not be performed.

[0149] <Embodiment 2> In the first embodiment, a method for making the large-scale language model 116 respond with a document type has been described. In this embodiment, a method for making the large-scale language model 116 respond with a reason why the large-scale language model 116 responded with a document type will be described. The present embodiment will be described focusing on the differences from the first embodiment. The configuration and processing are the same as those of the first embodiment unless otherwise specified.

[0150] [Document type determination process sequence] Fig. 11 is a sequence diagram of this embodiment corresponding to the sequence of Fig. 3(b). The sequence during development of the machine learning model is omitted because it is similar to Fig. 3(a).

[0151] In S1111, the user sets an option for obtaining a reason for determining the document type, and the information processing server 103 obtains the content set by the user in S1111. When the option for obtaining a reason is enabled, the reason why the large-scale language model 116 responded with the document type can also be obtained from the large-scale language model 116.

[0152] 12A is a diagram showing an option setting screen 1200, which is an example of a screen on which the user sets whether to enable or disable the reason acquisition option in S1111. The option setting screen 1200 will be described as being displayed on the display device 210 of the image forming apparatus 101 under the control of the CPU 261 of the information processing server 103.

[0153] A radio button 1201 included in the option setting screen 1200 is a radio button for enabling a reason acquisition option. A radio button 1202 is a radio button for disabling the reason acquisition option. When a save button 1220 is pressed, the setting contents corresponding to the selected radio button of the radio buttons 1201 and 1202 are transmitted to the information processing server 103.

[0154] S1112 to S1119 are similar to S311 to S318, and the description thereof will be omitted.

[0155] However, the process of S1116 when the reason acquisition option is set to be valid is different from S315. When the reason acquisition option is set to be valid, in S1116, an instruction sentence is generated so that the large-scale language model 116 also answers the reason for answering the document type. When the reason acquisition option is set to be invalid, an instruction sentence is generated so that the reason for answering the document type is not acquired from the large-scale language model 116, as in the first embodiment.

[0156] [Document type display processing] FIG. 13 is a flowchart for explaining a process for determining and displaying the document type of a document image, and is a flowchart of this embodiment corresponding to the flowchart of FIG. 5 in the first embodiment.

[0157] S1301 is a process similar to S501, in which the document type of the document image to be processed is determined based on the output result of the document type determiner 115a, which is a machine learning model. A detailed description will be omitted.

[0158] In S1302, the CPU 261 generates an instruction sentence and obtains a response to the instruction sentence from the large-scale language model 116.

[0159] FIG. 14 is a flowchart for explaining the details of the process of S1302, and is a flowchart of this embodiment that corresponds to the flowchart of FIG. 8 of the first embodiment.

[0160] In S1401, the CPU 261 acquires the setting contents of the reason acquisition option set by the user in S1111. As the setting contents, information indicating whether the reason acquisition option is enabled or disabled is acquired.

[0161] Since S1402 to S1403 are the same processes as S801 to S802 in FIG. 8, the description thereof will be omitted.

[0162] In S1404, the CPU 261 acquires a command format corresponding to the setting content of the reason acquisition option acquired in S1401.

[0163] In S1405, the CPU 261 generates an instruction statement by applying the document type option acquired in S1402 and the character string in the document image acquired in S1403 to the instruction statement template acquired in S1404.

[0164] When the reason acquisition option is disabled, that is, when radio button 1202 is selected on option setting screen 1200, the instruction statement template acquired is the same as the instruction statement template corresponding to instruction statement 910 in Fig. 9(b). When the reason acquisition option is enabled, that is, when radio button 1201 is selected on option setting screen 1200, the instruction statement template acquired is an instruction statement template that includes a question for prompting the user to answer the reason for determining the document type.

[0165] FIG. 15 is a diagram for explaining an instruction and its answer. An instruction 1510 in FIG. 15(a) is an example of an instruction generated based on an instruction template acquired when the reason acquisition option is set to be valid. In the instruction template acquired when the reason acquisition option is enabled, a question 1513 is added to the instruction template described in the first embodiment. The question 1513 is a question for asking the user to answer the reason for answering the question 911, which is an instruction to answer a document type from the document type options 901, and the question 912, which is an instruction to answer a document type other than the document type options. In other words, the question 1513 is a question for asking the large-scale language model 116 to answer the reason why the document type answered by the large-scale language model 116 is the document type of the document image to be processed.

[0166] An output result 1520 in FIG. 15(b) is an example of an output result obtained by inputting the instruction statement 1510 in FIG. 15(a) into the large-scale language model 116. The answer included in the output result 1520 includes an answer 1523 to a question 1513 added in the instruction statement 1510 as a difference from the output result 920 of the first embodiment shown in FIG. 9(c). The instruction statement 1510 in FIG. 15(a) is an instruction statement generated when the document image 700 in FIG. 7(a) is processed, and the large-scale language model 116 answers that the document type of the document image 700 is "order form" as an answer to the question 911 in the instruction statement 1510. Therefore, the large-scale language model 116 answers the reason for selecting "order form" in the answer 1523 to the question 1513 as shown in FIG. 15(b).

[0167] In addition, the content of the instruction template when the reason acquisition option is enabled may also be switched for each document type option. In this case, the instruction template may include a group of document type option character strings in advance. Furthermore, the CPU 261 may generate an instruction by switching the content for each language or destination of the input document image. Furthermore, the instruction template may be one prepared in advance by an engineer or a user, or may be one that has been prepared in advance and edited by the user.

[0168] S1406 to S1407 are similar to S805 to S806 in Fig. 8, and therefore description thereof will be omitted. For example, in S1406, an output result 1520 as shown in Fig. 14(b) is acquired from the large-scale language model 116 as an output result for the instruction statement 1510 in Fig. 15(a). Returning to Fig. 13, the description of the flowchart in Fig. 13 will be continued.

[0169] 13 are similar to the processes of S503 to S506 in FIG. 5, and therefore the description thereof will be omitted.

[0170] S1307 is a process performed when the reason acquisition option is enabled, and the CPU 261 outputs to the display device 267 the reason why the large-scale language model 116 responded with the document type, which was acquired in S1303.

[0171] The document type confirmation screen 1000 in Fig. 12(b) is a diagram showing an example of a document confirmation screen when the reason acquisition option is enabled. In this embodiment as well, the document type confirmation screen 1000 is described as being displayed under the control of the CPU 261 of the information processing server 103. As shown in Fig. 12(b), the document type confirmation screen 1000 in this embodiment includes an area 1252 that displays an answer to a question 1513 so that the user can recognize the reason why the large-scale language model 116 answered the document type. Note that if the user selects the radio button 1202, the reason is not acquired, and therefore the area 1252 is not displayed.

[0172] As described above, according to this embodiment, the reason for determining the document type of the document image to be processed can be presented. This can reduce the effort required for the user to confirm the document type. For example, if a user scans a "purchase order" document and the document type of the scanned image is determined to be "purchase order," the user can refer to the presented reason to determine whether the document type "purchase order" is correct.

[0173] <Embodiment 3> In the above-described embodiment, a method of using one large-scale language model has been described. In this embodiment, a method of using one or more large-scale language models set by an engineer or a user will be described. This embodiment will be described focusing on the differences from the first embodiment. Parts that are not specifically mentioned have the same configuration and processing as the first embodiment.

[0174] [Document type determination process sequence] Fig. 16 is a sequence diagram of this embodiment corresponding to the sequence in Fig. 3(b). The sequence during development of the machine learning model is omitted because it is similar to Fig. 3(a).

[0175] In S1611, the user selects a large-scale language model to which the instruction sentence is to be input via a setting screen. The information processing server 103 acquires setting information indicating the content selected by the user in S1611.

[0176] 17A is a diagram showing a large-scale language model setting screen 1700, which is an example of a screen for the user to select a large-scale language model in S1611. The large-scale language model setting screen 1700 will be described as being displayed on the display device 210 of the image forming apparatus 101 under the control of the CPU 261 of the information processing server 103. The CPU 261 also functions as a setting unit that sets a large-scale language model to which an instruction statement is input. In this embodiment, an instruction statement is input to each large-scale language model selected by the user, and a response to the instruction statement is received from each large-scale language model to which the instruction statement was input.

[0177] A model display area 1731 on a large-scale language model setting screen 1700 in FIG. 17(a) displays a large-scale language model selected as an input destination of an instruction sentence. FIG. 17(a) shows that "large-scale language model A" displayed in the model display area 1731 is selected as an input destination of an instruction sentence. When a user wants to change the large-scale language model to which an instruction sentence is to be input, the user presses a list display button 1733 included in the model display area 1731. When the list display button 1733 is pressed, a drop-down list including large-scale language models that the user can select is displayed. When the user selects a large-scale language model from the drop-down list, the name of the selected large-scale language model is displayed in the model display area 1731.

[0178] When inputting a command to multiple large-scale language models and obtaining answers from multiple large-scale language models, pressing the Add button 1735 adds a new model display area 1732, as shown in Fig. 17(a). When, for example, "large-scale language model B" is selected from the drop-down list 1734 displayed by pressing the list display button 1733 included in the model display area 1732, the selected "large-scale language model B" is displayed in the model display area 1732. Similarly, pressing the Add button 1735 can add an input field for a third large-scale language model.

[0179] In this state, when the user presses the save button 1720, the "large-scale language model A" and the "large-scale language model B" displayed in the model display areas 1731 and 1732 are set as the large-scale language models to which the instruction sentence is to be input. Information indicating the selected large-scale language model is transmitted to the information processing server 103.

[0180] S1612 to S1619 are similar to S311 to S318, and the description will be omitted. However, the processes of S1617 and S1618 are different from S316 and S317. In S1617, a command statement is input to the large-scale language model selected in S1611, and in S1618, a response to the command statement is received from the selected large-scale language model.

[0181] 16 shows a flow in the case where the user sets multiple large-scale language models as input destinations of an instruction message in S1611. The external information processing server 105a indicates a server that operates "large-scale language model A", and the external information processing server 105b indicates a server that operates "large-scale language model B". When the user sets "large-scale language model A" and "large-scale language model B" as input destinations of an instruction message in S1611, an instruction message is transmitted to each large-scale language model in order to input the instruction message to the respective large-scale language models in S1617. Then, a response is received from each large-scale language model in S1618.

[0182] [Document type display processing] FIG. 18 is a flowchart for explaining the process of determining and displaying the document type of a document image, and is a flowchart of this embodiment corresponding to the flowchart of FIG. 5 in the first embodiment.

[0183] S1801 is the same process as S501 in FIG. 5, and the document type of the document image to be processed is determined based on the output result of the document type determiner 115a, which is a machine learning model.

[0184] S1802 corresponds to S502 in FIG. 5, and is a step in which an instruction sentence is input to a large-scale language model selected by the user, and a response to the instruction sentence is obtained as an output result from each large-scale language model.

[0185] FIG. 19 is a flowchart for explaining the details of the process of S1802, and is a flowchart of this embodiment that corresponds to the flowchart of FIG. 8 of the first embodiment.

[0186] In S1901, the CPU 261 acquires setting information indicating the large-scale language model selected by the user in S1611. For example, if the user selects "large-scale language model A" and "large-scale language model B" as input destinations in S1611, information indicating "large-scale language model A" and "large-scale language model B" is acquired.

[0187] The steps S1902 to S1905 are the same as the steps S801 to S804 in Fig. 8, and therefore the description thereof will be omitted. That is, the steps are carried out up to the step of generating a directive.

[0188] The next steps S1906 to S1909 are loop processes. In S1906, the CPU 261 selects a large-scale language model to be processed from among the large-scale language models indicated by the setting information acquired in S1901. In S1909, the CPU 261 determines whether processing has been performed for all large-scale language models. If processing has not been performed for all large-scale language models, the process returns to S1906 and selects the next processing target from the unprocessed large-scale language models.

[0189] In S1907, the CPU 261 inputs the instruction generated in S1905 to the large-scale language model to be processed.

[0190] In S1908, the CPU 261 receives a response to the instruction sentence from the large-scale language model to be processed.

[0191] In S1909, if the processing has been completed for all large-scale language models selected by the user, the flowchart in Fig. 19 ends, and the process proceeds to S1803 in Fig. 18. For example, if the user selects "large-scale language model A" and "large-scale language model B" in S1611, the CPU 261 executes the processing of S1907 to S1908 for each of "large-scale language model A" and "large-scale language model B." That is, the CPU 261 executes the processing of S1907 to S1908 twice.

[0192] S1803 is the same process as S503 in FIG. 5, and displays the document type determined based on the output result of the document type determining unit 115a on the document type confirmation screen 1000.

[0193] In S1804, the CPU 261 determines whether or not it has received a response from one or more large-scale language models that the document type of the document image to be processed does not correspond to the document types included in the document type selection options 901. If it has received a response from all large-scale language models selected by the user that the document type of the document image to be processed is a document type included in the document type selection options 901 (NO in S1804), the flowchart in FIG.

[0194] When the CPU 261 obtains an answer from at least one large-scale language model that the document type of the document image to be processed does not correspond to the document types included in the document type options 901 (YES in S1804), the process transitions to S1805. For example, assume that the answer to the question 911 sent from "large-scale language model A" is an answer that the document type corresponds to the document type included in the document type options. On the other hand, assume that the answer to the question 911 sent from "large-scale language model B" is an answer that the document type does not correspond to the document type included in the document type options. In this case, YES is determined in S1804 and the process transitions to S1805.

[0195] Steps S1805 to S1806 in Fig. 13 are similar to steps S505 to S506 in Fig. 5. The CPU 261 outputs information required for display to the display device 267 so that the alert 1016 and the document type candidates answered by the large-scale language model selected by the user are displayed on the document type confirmation screen 1000.

[0196] The document type confirmation screen 1000 in FIG. 17(b) is an example of a document type confirmation screen displayed by the CPU 261 when candidate document types are acquired as output results from multiple large-scale language models. For example, assume that the instruction statement 930 shown in FIG. 9(d) is input to at least the large-scale language model A and the large-scale language model B, and the answers to the question 911 from the large-scale language model A and the large-scale language model B are “not applicable”. Then, assume that answers to the question 912 from the large-scale language model A and the large-scale language model B are answers to the document type of the document image to be processed. For example, assume that “receipt” and “receipt” are acquired as answers to the question 912 from the large-scale language model A. Assume that “receipt” and “donation certificate” are acquired as answers to the question 912 from the large-scale language model B. That is, “receipt”, “receipt”, and “donation certificate” are acquired as document type candidates from multiple large-scale language models selected by the user.

[0197] In this case, a drop-down list displaying candidate document types is displayed below the document type display area 1001. The drop-down list displays text 1752, 1753, and 1754 indicating that the candidate document types are "receipt," "invoice," and "donation certificate," respectively. The text 1752, 1753, and 1754 may also include information indicating the large-scale language model that answered the document type as a candidate. For example, "receipt" is an answer obtained from large-scale language model A and large-scale language model B. For this reason, the text 1752 for displaying "receipt" is written in a format in which "large-scale language model A / B," indicating the large-scale language model A and large-scale language model B that answered "receipt," are attached to "receipt."

[0198] Also, a document type that has many large-scale language models that have answered the same document type may be displayed preferentially. For example, since "receipt" is answered by two large-scale language models, large-scale language model A and large-scale language model B, it is displayed at the top of the drop-down list so that it is displayed preferentially.

[0199] As described above, according to this embodiment, when a user inputs a document other than a predetermined document type, the document type can be answered from multiple large-scale language models. This increases the likelihood that an appropriate document type will be answered. Furthermore, according to this embodiment, since document type candidates can be obtained from multiple large-scale language models, it is possible to increase information that can serve as a hint when the user considers whether or not to correct the document type. This reduces the burden on the user of correcting the document type.

[0200] <Embodiment 4> In the first embodiment, a method for displaying the output results of the document type determiner 115a and the large-scale language model 116, which are machine learning models, was described. In this embodiment, a method for displaying only the document type answered by the large-scale language model 116 will be described. This embodiment will be described focusing on the differences from the first embodiment. Portions not specifically mentioned have the same configuration and processing as the first embodiment.

[0201] [About the configuration of information processing systems] Fig. 20 is a diagram showing an example of the configuration of an information processing system according to this embodiment, which corresponds to Fig. 1. As shown in Fig. 20, in this embodiment, the learning device 102 does not generate a document type classifier 115a. The learning data generation unit 152 acquires a second correct answer label and generates second learning data. The learning unit 153 uses the second learning data to generate an item value extractor 115b, which is a machine learning model.

[0202] [Sequence] Fig. 21(a) is a sequence diagram of this embodiment corresponding to the sequence during development of the machine learning model in Fig. 3(a). S2101 is the same process as S301, and S2102 to S2104 are the same processes as S304 to S306, so their explanations are omitted. In this embodiment, the document type discriminator 115a is not generated, so the sequence in Fig. 21(a) does not include steps corresponding to S302 and S303, which are steps related to the generation of the document type discriminator 115a. Also, as detailed processing of S2102 to S2103, a flowchart in which S403, S405, and S407 are not executed is executed among the steps in the flowchart in Fig. 4.

[0203] FIG. 21(b) is a sequence diagram of this embodiment corresponding to the sequence of FIG. 3(b). In S2111, the user sets a document type option. The information processing server 103 acquires information indicating the document type option set by the user in S2111. In this embodiment, not only engineers but also users can set the document type option. Since the document type determiner 115a is not generated in this embodiment, the document type option does not need to be a document type that the document type determiner 115a can output. Therefore, in this embodiment, the user can set any document type option.

[0204] In this embodiment, the document type option is set even though the document type determiner 115a is not generated. The reason for this is that when there are multiple names for a document type, the large-scale language model 116 is made to respond with only one of the multiple names, thereby eliminating variation in the response of the large-scale language model 116. For example, type names that represent the same document type include "purchase order" and "order form." By setting the document type option to include "purchase order" but not to include "purchase order," it is possible to prevent the response of the large-scale language model 116 from becoming "purchase order" or "purchase form."

[0205] Since S2112 to S2113 are similar to S311 to S312, and S2114 to S2118 are similar to S314 to S318, their explanations will be omitted. In this embodiment, the document type determiner 115a is not generated, and therefore the step of determining the document type based on the output result of the document type determiner 115a is not performed.

[0206] [About user settings for document type options] 22(a) is a diagram showing a document type option setting screen 2200, which is an example of a screen for the user to set a document type option in S2111. The document type option setting screen 2200 will be described as being displayed on the display device 210 of the image forming apparatus 101 under the control of the CPU 261 of the information processing server 103. The CPU 261 also functions as a setting unit that sets document type options. Through the document type option setting screen 2200, the user can generate a new document type option by himself or herself, and can edit the contents of the document type option set by the engineer in S2104.

[0207] The document type option setting screen 2200 displays "sets" corresponding to the document type options, and the user can select one of the sets. The document type option associated as "set 1" in Fig. 22(a) indicates the document type option set by the engineer in S2104.

[0208] Column 2201 includes radio buttons. For example, when radio button 2204 corresponding to "Set 1" is selected, "invoice, estimate, purchase order, delivery note, contract" associated with "Set 1" is acquired by CPU 261 as the document type option set by the user. The user does not need to change the document type option set by the engineer. In this case, the user can simply press radio button 2204 for "Set 1" and then press save button 2220.

[0209] Button 2235 is a button for adding a set corresponding to a new document type option. The document type option setting screen 2200 in FIG. 22(a) shows a state in which button 2235 has been pressed and "Set 2" has been newly added. The user can input a document type to be included in the document type option associated with "Set 2" using input cursor 2231. Furthermore, by enabling the radio button in column 2201 corresponding to "Set 2" and pressing save button 2220, the newly generated set is acquired by CPU 261 as a document type option set by the user.

[0210] For example, assume that the user wishes to classify the document types into one of the three "order form, delivery note, daily work report" instead of the five document types "invoice, estimate, order form, delivery note, contract" set by the engineer. In this case, the user inputs "order form, delivery note, daily work report" into "set 2" and presses the save button 2220 with the radio button for "set 2" selected. There are no limitations on the method for adding a new document type option. For example, a set that has already been set may be copied and generated, or a document type that has been set once may be suggested.

[0211] If the user wishes to change the document type options set by the engineer, the user can place the input cursor 2132 on the area displaying the document types of "Set 1" and perform editing such as changing the displayed document type or adding a document type. For example, the user can add "receipt" to "Set 1" and set the document type options corresponding to "Set 1" to six document types: "invoice, estimate, purchase order, delivery note, contract, receipt."

[0212] Column 2202 is a column for setting an operation when a new document type appears for each set. When a new document type appears, it refers to a case where a document image (new document) of a new document type not included in the document type selection options is input as a processing target, and the document type of the new document is input by the user. For example, when a new document type appears, the cell 1042 of "receipt", which is the document type of the new document proposed by the large-scale language model 116, is selected on the document type confirmation screen 1000 in FIG. 10(b) and the OK button 1004 is pressed.

[0213] In this embodiment, the action to be taken when a new document type appears can be selected from a drop-down list: "Do not add," "Automatically add (same set)," "Automatically add (different set)," or "Confirm each time." The drop-down list is displayed by pressing the list display button 2106. In this embodiment, "Do not add" is set as the default action to be taken when a new document type appears.

[0214] "Do not add" is a setting item for not automatically adding the document type of a new document to the document type options even if the document type of a new document is input.

[0215] "Auto add (same set)" is a setting item that automatically adds the document type of a new document to the document type options of a set for which "Auto add (same set)" is set when the document type of a new document is entered.

[0216] "Auto Add (Another Set)" is a setting item that adds a new set when the document type of a new document is input. The document type of the new document is added to the document type options of the set for which "Auto Add (Another Set)" is set, and this setting item is set as the document type option of the added new set.

[0217] "Confirm each time" is a setting item for notifying the user and asking for instructions when the document type of a new document is input.

[0218] For example, assume that "receipt", a document type that is not included in the document type options for "set 1", is input as the document type of a new document. In this case, CPU 261 causes "receipt" to be included in the document type options for "set 1" for which "automatic addition (same set)" is set. In this way, the user can set whether to automatically add to the document type options for each registered set.

[0219] Column 2203 is a column for setting the application destination for each set when "Auto Add" is set in column 2202. Automatic addition means "Auto Add (Same Set)" or "Auto Add (Different Set)". In this embodiment, the application destination of automatic addition can be selected from "Individual", "In Department", and "Company". "Individual" is the application destination of the individual user, "In Department" is the application destination of the department to which the user belongs, and "Company" is the application destination of the company to which the user belongs. As the application destination, for example, a specific user may be specified. When "Auto Add" is selected in column 2202, for example, "Individual" is set by default. The user can press the list display button 2207 and select the application destination from the displayed drop-down list of "Individual", "In Department", and "Company".

[0220] When the save button 2220 is pressed, the setting contents selected by the user on the document type option setting screen 2200 are transmitted to the information processing server 103.

[0221] [Document type display processing] FIG. 23 is a flowchart for explaining the process of determining and displaying the document type of a document image, and is a flowchart of this embodiment corresponding to the flowchart of FIG. 5 in the first embodiment.

[0222] In S2301, similarly to S501, processing is performed on the document image to be processed using the machine learning model.

[0223] FIG. 24 is a flowchart for explaining the details of the process of S2301, and is a flowchart of this embodiment that corresponds to the flowchart of FIG. 6 of the first embodiment.

[0224] Steps S2401 to S2403 are the same as steps S602 to S604 in Fig. 6, and step S2404 is the same as step S606 in Fig. 6. That is, in this embodiment, among the steps in the flowchart in Fig. 6, steps S601 and S605 for determining the document type based on the output result of the document-type determiner 115a are not performed.

[0225] S2302 is a step corresponding to S502 in Fig. 5, where the CPU 261 generates an instruction sentence and obtains a response to the instruction sentence from the large-scale language model 116 as an output result. In S2302, the same process as in the flowchart in Fig. 8 is performed. However, in S801 in Fig. 8, the CPU 261 obtains the setting of the document type option set by the user in S2111 in Fig. 21. If there is no setting provided by the user in S2111, that is, if the default value is used, the document type option provided by the engineer in S2104 in Fig. 21 is obtained.

[0226] When S2302 ends, the process proceeds to S2303. In the flowchart of FIG. 23, there is no step corresponding to S503.

[0227] In S2303, the CPU 261 determines whether the answer to the instruction obtained from the large-scale language model 116 is an answer that the document type does not correspond to the document type included in the document type option. When the user sets the document type option in S2111, the large-scale language model 116 answers the question 911 by saying whether the document type of the document image to be processed corresponds to the document type included in the document type option set in S2111.

[0228] If the large-scale language model 116 responds that the document type corresponds to one included in the document type options (NO in S2303), the process proceeds to S2306.

[0229] In S2306, the CPU 261 performs a process of notifying the user of the document type answered by the large-scale language model 116 from among the document type options. For example, if the large-scale language model 116 answers the question 911 in Fig. 9 with a document type included in the document type options, a document type confirmation screen 1000 as shown in Fig. 10(a) is generated in S2306. That is, the CPU 261 performs display control so that the document type answered by the large-scale language model 116 from among the document type options is displayed by default in the document type display area 1001.

[0230] On the other hand, if the large-scale language model 116 responds that the document type does not correspond to the document type included in the document type options (YES in S2303), the process proceeds to S2304.

[0231] 23 are similar to the processes in S505 and S506 in FIG.

[0232] In S2304, the CPU 261 notifies the user by displaying, for example, an alert 1016, that the document type of the document image to be processed may not correspond to the document types included in the set document type options.

[0233] In S2305, the CPU 261 outputs information on the document type candidates presented by the large-scale language model 116 to the display device 267 so that the document type candidates are displayed on the document type confirmation screen 1000.

[0234] The document type confirmation screen 1000 in Fig. 22(b) is an example of a document type confirmation screen when "not applicable" is obtained as the answer to question 911 from the large-scale language model 116 as shown in Fig. 9(e) and YES is determined in S2303. In this case, as shown in Fig. 22(b), the document type display area 1001 displays the text "not applicable" indicating that the document type does not match any of the document types included in the document type options. The expression indicating that the document does not match may be determined by an engineer or by a user. For example, a similar expression such as "other" may be used as the expression indicating that the document does not match.

[0235] Furthermore, if the answer to question 911 is "not applicable," a candidate document type other than the document type options can be obtained from the large-scale language model 116 as an answer to question 912. For example, assume that the answer "receipt" or "invoice" is obtained as a candidate document type for the document image. In this case, as shown in FIG. 22(b), a drop-down list is displayed below the document type display area 1001 displaying "not applicable." Then, the drop-down list can be configured to display "receipt" and "invoice," which are the candidate document types answered by the large-scale language model 116 to question 912.

[0236] The document type confirmation screen 1000 in FIG. 22(c) is another example of the document type confirmation screen when S2303 is determined to be YES. FIG. 22(c) is an example of a screen that does not display "Not Applicable" even if the large-scale language model 116 gives an answer that does not correspond to the document type included in the document type options. In the document type confirmation screen 1000 in FIG. 22(c), only the texts 2262 and 2263 of "Receipt" and "Invoice", which are the document type candidates that the large-scale language model 116 answered to the question 912, are displayed. When displaying the texts 2262 and 2263 of the document type candidates, they may be displayed in the order in which the large-scale language model 116 listed them, in alphabetical order, or in order of the number of large-scale language models that gave the same answer as shown in the third embodiment.

[0237] As described above, in this embodiment, the large-scale language model 116 can return the document type of the document image to be processed without generating the document type determiner 115a. Also, according to this embodiment, the large-scale language model 116 can return an appropriate document type from among document type options arbitrarily set by the user. Furthermore, if the document type does not match any of the document type options, a candidate document type can be returned, thereby reducing the effort required for the user to correct the document type name.

[0238] As described above, when the "character strings in a document image" are a group of all character strings recognized from within a document image, the item value extractor 115b, which is a machine learning model, does not need to be generated. In this case, since the machine learning model does not need to be generated in this embodiment, the information processing system 100 may be configured not to include the learning device 102.

[0239] In addition, in this embodiment, the user can arbitrarily select a document type option. For example, the user may cancel the document type option set by the engineer and select not to set a document type option. In this case, the instruction generated in the information processing server 103 may not be an instruction that causes the document type of the document image to be processed to be answered from the document type options. For example, an instruction such as "Do not answer the document type of the document having the title "Purchase Order"" may be generated as an instruction when not setting a document type option when the "character string in the document image" is "purchase order". In this way, in S2302, an instruction that does not define the range of document types to be answered by the large-scale language model 116 may be generated. In this case, the document type answered by the large-scale language model 116 may be displayed by default in the document type display area 1001 of the document type confirmation screen 1000.

[0240] <Other embodiments> The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0241] The disclosure of the above-described embodiment includes the following configurations.

[0242] (Configuration 1) character recognition means for performing character recognition processing on an image of a document to be processed; a generating means for generating an instruction sentence for causing a large-scale language model to respond with a document type indicated by the document to be processed based on a result of the character recognition processing; a sending means for sending the instruction to obtain a response to the instruction from the large-scale language model; a receiving means for receiving a response to the instruction sentence by the large-scale language model; 13. An information processing device comprising:

[0243] (Configuration 2) The generating means generates the instruction sentence for causing the large-scale language model to respond with a document type of a document including a character string obtained as a result of the character recognition processing. 2. The information processing device according to configuration 1.

[0244] (Configuration 3) The method further comprises extracting means for extracting a character string corresponding to at least a predetermined item from a character string obtained as a result of the character recognition process, The generating means generates the instruction sentence for causing the large-scale language model to respond with a document type of a document that includes a character string corresponding to the predetermined item. 3. The information processing device according to configuration 1 or 2.

[0245] (Configuration 4) The predetermined item is a title. 4. The information processing device according to configuration 3.

[0246] (Configuration 5) The predetermined item is composed of a plurality of items. 4. The information processing device according to configuration 3.

[0247] (Configuration 6) The extraction means extracts, based on a result obtained by inputting the group of character strings obtained as a result of the character recognition process into a trained model, item values ​​which are character strings corresponding to items including the predetermined item from the group of character strings; The generating means obtains a character string corresponding to the predetermined item from the item value and generates the instruction sentence. 6. The information processing device according to any one of configurations 3 to 5.

[0248] (Configuration 7) The instruction sentence includes a first question for prompting the user to answer whether the document type indicated by the document to be processed corresponds to any one of a plurality of document types set in advance. 7. The information processing device according to any one of configurations 1 to 6.

[0249] (Configuration 8) The instruction sentence includes a second question for prompting the user to answer a document type other than the plurality of document types as the document type indicated by the document to be processed, if the document type indicated by the document to be processed does not correspond to any of the plurality of document types. 8. The information processing device according to configuration 7.

[0250] (Configuration 9) The instruction sentence includes a third question that asks the subject to explain the reasons for the answers given to the first and second questions. 9. The information processing device according to configuration 8.

[0251] (Configuration 10) The method further includes a determination means for determining a document type indicated by the document to be processed based on a result obtained by inputting a group of character strings obtained as a result of the character recognition process into a trained model, The trained model is a trained model trained to output information indicating any one of the plurality of document types. 10. The information processing device according to configuration 8 or 9.

[0252] (Configuration 11) The method further includes display control means for displaying at least one of the document type determined by the determination means and the document type returned by the large-scale language model on a display unit as the document type indicated by the document to be processed. 11. The information processing device according to configuration 10.

[0253] (Configuration 12) The display control means accepts a correction to the displayed document type when the correction is instructed by a user. 12. The information processing device according to configuration 11.

[0254] (Configuration 13) The display control means The document type determined by the determination means is displayed on a display unit as the document type indicated by the document to be processed, and when the large-scale language model answers the second question with a first document type that is a document type other than the plurality of document types, The first document type is displayed on the display unit as a character string to be corrected. 13. The information processing device according to configuration 12.

[0255] (Configuration 14) The document type setting unit may further include a setting unit for setting a document type to be included in the plurality of document types. 14. The information processing device according to any one of configurations 8 to 13.

[0256] (Configuration 15) The setting means is When the large-scale language model answers the second question with a first document type that is a document type other than the plurality of document types, When a user selects the first document type as a document type indicated by the document to be processed, the first document type is set to be included in the plurality of document types. 15. The information processing device according to configuration 14.

[0257] (Configuration 16) A setting means for setting one or more large-scale language models for answering the instruction sentences, The sending means sends the instruction sentence to the set one or more large-scale language models. 16. The information processing device according to any one of configurations 1 to 15.

[0258] (Configuration 17) a character recognition step of performing character recognition processing on an image of a document to be processed; a generating step of generating an instruction sentence for causing a large-scale language model to respond with a document type indicated by the document to be processed based on a result of the character recognition process; a sending step of sending the instruction to obtain an answer to the instruction from the large-scale language model; receiving a response to the instruction from the large-scale language model; 13. An information processing method comprising:

[0259] (Configuration 18) A program for causing a computer to execute each of the means of the information processing device according to any one of configurations 1 to 16. [Explanation of symbols]

[0260] 103 Information Processing Server 154 Document Image Analysis Unit 155 Directive sentence generation section 156 Reply Receiving Department

Claims

1. character recognition means for performing character recognition processing on an image of a document to be processed; a generating means for generating an instruction sentence for causing a large-scale language model to respond with a document type indicated by the document to be processed based on a result of the character recognition processing; a sending means for sending the instruction to obtain a response to the instruction from the large-scale language model; a receiving means for receiving a response to the instruction sentence by the large-scale language model; 13. An information processing device comprising:

2. The generating means generates the instruction sentence for causing the large-scale language model to respond with a document type of a document including a character string obtained as a result of the character recognition processing.

2. The information processing apparatus according to claim 1,

3. The method further comprises extracting means for extracting a character string corresponding to at least a predetermined item from a character string obtained as a result of the character recognition process, The generating means generates the instruction sentence for causing the large-scale language model to respond with a document type of a document that includes a character string corresponding to the predetermined item.

2. The information processing apparatus according to claim 1,

4. The predetermined item is a title.

4. The information processing apparatus according to claim 3.

5. The predetermined item is composed of a plurality of items.

4. The information processing apparatus according to claim 3.

6. The extraction means extracts, based on a result obtained by inputting the group of character strings obtained as a result of the character recognition process into a trained model, item values ​​which are character strings corresponding to items including the predetermined item from the group of character strings; The generating means obtains a character string corresponding to the predetermined item from the item value and generates the instruction sentence.

4. The information processing apparatus according to claim 3.

7. The instruction sentence includes a first question for prompting the user to answer whether the document type indicated by the document to be processed corresponds to any one of a plurality of document types set in advance.

2. The information processing apparatus according to claim 1,

8. The instruction sentence includes a second question for prompting the user to answer a document type other than the plurality of document types as the document type indicated by the document to be processed, if the document type indicated by the document to be processed does not correspond to any of the plurality of document types.

8. The information processing apparatus according to claim 7,

9. The instruction sentence includes a third question that prompts the respondent to explain the reasons for his / her answers to the first and second questions.

9. The information processing apparatus according to claim 8,

10. The method further includes a determination means for determining a document type indicated by the document to be processed based on a result obtained by inputting a group of character strings obtained as a result of the character recognition process into a trained model, The trained model is a trained model trained to output information indicating any one of the plurality of document types.

9. The information processing apparatus according to claim 8,

11. The method further includes display control means for displaying at least one of the document type determined by the determination means and the document type answered by the large-scale language model on a display unit as the document type indicated by the document to be processed.

11. The information processing apparatus according to claim 10,

12. The display control means accepts a correction to the displayed document type when the correction is instructed by a user.

12. The information processing apparatus according to claim 11 .

13. The display control means The document type determined by the determination means is displayed on a display unit as the document type indicated by the document to be processed, and when the large-scale language model answers the second question with a first document type that is a document type other than the plurality of document types, the first document type is displayed on the display unit as a character string that is a candidate for correction.

13. The information processing apparatus according to claim 12.

14. The document type setting unit may further include a setting unit for setting a document type to be included in the plurality of document types.

9. The information processing apparatus according to claim 8,

15. The setting means is When the large-scale language model answers the second question with a first document type that is a document type other than the plurality of document types, and when a user selects the first document type as a document type indicated by the document to be processed, the plurality of document types are set to include the first document type.

15. The information processing apparatus according to claim 14,

16. A setting means for setting one or more large-scale language models for answering the instruction sentences, The transmitting means transmits the instruction sentence to the set one or more large-scale language models.

2. The information processing apparatus according to claim 1,

17. a character recognition step of performing character recognition processing on an image of a document to be processed; a generating step of generating an instruction sentence for causing a large-scale language model to respond with a document type indicated by the document to be processed based on a result of the character recognition process; a sending step of sending the instruction to obtain an answer to the instruction from the large-scale language model; receiving a response to the instruction from the large-scale language model; 13. An information processing method comprising:

18. A program for causing a computer to execute each of the means of the information processing device according to any one of claims 1 to 16.

Images