Information processing system, information processing method, and program

Abstract

To enable extraction of item data from an image.SOLUTION: An information processing system comprises: a storage unit that stores a first character string read from first image data and a first item value extracted from the first character string; a character string acquisition unit that acquires a second character string read from second image data; a retrieval unit that acquires, from the storage unit, a first character string similar to the second character string and the corresponding first item value; and an item value acquisition unit that provides, to a generator that generates an answer based on a trained language model, the second character string, an instruction to extract a second item value from the second character string, and a prompt including the retrieved first character string and the first item value, so as to cause the generator to generate the second item value.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] Patent Document 1 discloses that character data is obtained from an image by OCR processing. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-073201 Summary of the Invention [Problem to be solved by the invention]

[0004] However, it is difficult to extract specific items from the scanned characters.

[0005] The present invention has been made in view of the above background, and has as its object to make it possible to extract item data from an image. [Means for solving the problem]

[0006] The main invention of the present invention for solving the above problem is an information processing system comprising: a memory unit that stores a first character string read from first image data and a first item value extracted from the first character string; a character string acquisition unit that acquires a second character string read from second image data; a search unit that acquires the first character string similar to the second character string and the corresponding first item value from the memory unit; and an item value acquisition unit that gives a generator that generates an answer based on a trained language model an instruction to extract the second character string and the second item value from the second character string, and a prompt including the searched first character string and the first item value, to generate the second item value.

[0007] Other problems and solutions disclosed in this application will be made clear in the section on preferred embodiments of the invention and the drawings. [Effects of the Invention]

[0008] According to the present invention, item data can be extracted from an image. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of an information processing system. [Figure 2] FIG. 2 illustrates an example of a hardware configuration of a management server 2. [Figure 3] FIG. 2 illustrates an example of the software configuration of a management server 2. [Figure 4] FIG. 2 is a diagram illustrating an example of a record stored in a vector store 231. [Figure 5] FIG. 10 is a diagram illustrating the operation of the management server 2. DETAILED DESCRIPTION OF THE INVENTION

[0010] <System Overview> An information processing system according to one embodiment of the present invention will be described below. The information processing system of this embodiment is intended to extract specific items displayed in an image. The information processing system of this embodiment recognizes character strings from an image (OCR) and extracts specific items from the recognized character strings using a large-scale language model (LLM). The information processing system of this embodiment is intended to improve extraction accuracy by learning previously extracted examples using short-shot learning.

[0011] 1 is a diagram showing an example of the overall configuration of an information processing system. The information processing system of this embodiment is configured to include a management server 2. The management server 2 is communicably connected to each of a user terminal 1 and a generation server 3 via a communication network. The communication network is, for example, the Internet, and is constructed using a public telephone network, a mobile phone network, a wireless communication path, Ethernet (registered trademark), etc.

[0012] The user terminal 1 is a computer operated by a user. The user terminal 1 can be, for example, a smartphone, a tablet computer, or a personal computer. The user can operate the user terminal 1 to upload images to be read, such as form images, to the management server 2.

[0013] The management server 2 may be a general-purpose computer such as a workstation or personal computer, or may be logically implemented using cloud computing. The management server 2 operates as a SaaS that returns the results of OCR processing to the user terminal 1.

[0014] The generation server 3 is a computer that provides a function for generating answers using a large-scale limit model (LLM). The generation server 3 functions as a generator that generates answers by providing externally provided prompts to the large-scale limit model. The generation server 3 provides the generator function using the large-scale limit model as an API (Application Programming Interface).

[0015] <Administration Server> FIG. 2 is a diagram illustrating an example of the hardware configuration of the management server 2. Note that the illustrated configuration is an example, and other configurations may also be used. The management server 2 includes a CPU 201, a memory 202, a storage device 203, a communication interface 204, an input device 205, and an output device 206. The storage device 203 stores various data and programs, and is, for example, a hard disk drive, a solid state drive, or a flash memory. The communication interface 204 is an interface for connecting to a communication network, and is, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone network, a wireless communication device for wireless communication, or a USB (Universal Serial Bus) connector or an RS232C connector for serial communication. The input device 205 is used to input data, and is, for example, a keyboard, a mouse, a touch panel, a button, a microphone, or the like. The output device 206 is used to output data, and is, for example, a display, a printer, a speaker, or the like. Each functional unit of the management server 2 described below is realized by the CPU 201 reading a program stored in the storage device 203 into the memory 202 and executing it, and each storage unit of the management server 2 is realized as part of the storage area provided by the memory 202 and the storage device 203.

[0016] 3 is a diagram illustrating an example of the software configuration of the management server 2. The management server 2 includes a vector store 231, an image acquisition unit 211, a character string acquisition unit 212, a search unit 213, and an item value acquisition unit 214.

[0017] <Storage section> The vector store 231 stores values ​​(vector data) obtained by embedding character strings previously read from image data. The vector store 231 can store character strings previously read from image data, items extracted from the character strings (pairs of item names and item values), and vector data in association with each other. The character strings previously read from image data and / or items extracted from the character strings may be managed in, for example, a file, and the vector store 231 may store the path or URL of the file. The embedding process may be performed by the generation server 3 using an Embeddings API provided by the generation server 3, or the management server 2 may perform the embedding process using a method such as Word2Vec, Doc2Vec, or fastText.

[0018] 4 is a diagram showing an example of a record stored in the vector store 231. As shown in the diagram, each record registered in the vector store 231 may include a character string (text data) read from image data, vector data (which may have any number of dimensions) that is the result of embedding the character string, and items (pairs of item names and item values) extracted from the character string expressed in JSON format.

[0019] <Functional section> The image acquisition unit 211 acquires images such as form images from a user. For example, the image acquisition unit 211 can accept an image uploaded from the user terminal 1. For example, the image acquisition unit 211 may acquire an image sent via email or a chat service. The image acquisition unit 211 may accept a URL to an online storage where the image is stored from the user, and access the URL to acquire the image.

[0020] The character string acquisition unit 212 acquires a character string read from image data. The character string acquisition unit 212 can recognize a character string (hereinafter, target character string) displayed in an image (hereinafter, target image) acquired by the image acquisition unit 211, and acquire the recognized character string. Note that the process of recognizing and acquiring a character string from an image is assumed to be a general OCR process, and a detailed description thereof will be omitted here.

[0021] The search unit 213 acquires a character string similar to the target character string from the vector store 231. The search unit 213 creates vector data in which the target character string has been embedded, and can search for a character string (text data) similar to the target character string based on the distance between the created vector data and the vector data stored in the vector store 231. The embedding process may be performed by the generation server 3 using an Embeddings API provided by the generation server 3, or the management server 2 may perform the embedding process using a method such as Word2Vec, Doc2Vec, or fastText. The search unit 213 acquires similar character strings and items (extracted items) corresponding to the character strings from the vector store 231. Note that the search unit 213 can search for a predetermined number (which can be any number) of pairs of text data and extracted items in order of decreasing distance, for example. In this embodiment, it is assumed that three pairs of text data and extracted items are searched for.

[0022] The item value acquisition unit 214 acquires item values ​​from character strings read from an image. The item value acquisition unit 214 provides a prompt including a character string, an instruction to extract an item value from the character string, and a previously extracted example to a generator that generates an answer based on a trained language model, thereby generating result data by extracting pairs of item names and item values ​​from the character string. The character string provided to the prompt is the target character string acquired by the character string acquisition unit 212. The previously extracted example provided to the prompt is a pair of text data and extracted items acquired from the vector store 231 by the search unit 213. This allows the item value acquisition unit 214 to learn how to extract items from text data using a short-shot learning technique and cause the LLM to extract items from the target character string. In this embodiment, it is assumed that the item value acquisition unit 214 calls an API provided by the generation server 3 to provide a prompt to the LLM included in the generation server 3 and receive output results from the LLM. However, the management server 2 may also include a trained language model and a generator.

[0023] <Operation> FIG. 5 is a diagram illustrating the operation of the management server 2.

[0024] The management server 2 accepts an image (S301), reads a string from the image (S302), searches for past text data and examples of extracted items that are similar to the target string (S303), sends a prompt containing the target string, the text data, and examples of extracted items to the generation server 3 (S304), and receives the results of item extraction using LLM from the generation server 3 (S305).

[0025] As described above, according to the information processing system of this embodiment, it is possible to extract necessary items from character strings read from an image using a large-scale limit model (LLM).

[0026] Although the present embodiment has been described above, the above embodiment is intended to facilitate understanding of the present invention and is not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit thereof, and equivalents thereof are also included in the present invention.

[0027] For example, the processing by each of the functional units of the management server 2 described above may be performed by any of the functional units. Also, a different functional unit that performs part of the processing by each of the functional units described above may be added. Also, the functional units of the management server 2 may be distributed across multiple computers.

[0028] Furthermore, the information stored in each storage unit of the management server may be stored in any of the storage units. That is, the information stored in the above-mentioned multiple storage units may be stored in one storage unit, or part of the information stored in one of the above-mentioned storage units may be stored in another storage unit.

[0029] Furthermore, in the above-described embodiment, only the target character string is embedded, but an image may be embedded in addition to or instead of the target character string. In this case, the search unit 213 obtains vector data (image vector) in which the image acquired by the image acquisition unit 211 is embedded, in addition to or instead of the vector data (character string vector) in which the target character string is embedded, and can search for character strings (text data) similar to the target character string and / or image according to a value (e.g., a sum, an average, etc.) based on the distance between the character string vector and the vector data stored in the vector store 231 and the distance between the image vector and the vector data stored in the vector store 231. For image embedding, a known method such as MatchingNet can be used.

[0030] In addition to the target character string and / or image, a graph representation showing the connection relationships between words (or phrases; the same applies below) extracted from the target character string may be embedded. In this case, for example, words are extracted from the target character string, the extracted positions of the extracted words on the image are identified, a graph is created linking a first word with each second word on the image that is within a predetermined distance from the first word, and the created graph is embedded. The vector store 231 stores, in association with each other, vector data (character vectors) in which character strings read from an image are embedded, and vector data (graph vectors) in which a graph is embedded in which links are set to word pairs based on the positions on the image of words in character strings previously read from image data. The search unit 213 can search for a string (text data) that is similar to the target string and / or image and the graph showing the links between words in the target string, based on a value (e.g., a total value, an average value, etc.) based on the distance between the vector data in which the target string and / or image has been embedded and the character vector in the vector store 231, and the distance between the vector data in which a graph showing the links between words extracted from the target string has been embedded and the graph vector in the vector store 231.

[0031] <Disclosures> The present disclosure also includes the following configurations. [Item 1] a storage unit that stores a first character string read from the first image data and a first item value extracted from the first character string; a character string acquisition unit that acquires a second character string read from the second image data; a search unit that acquires the first character string similar to the second character string and the corresponding first item value from the storage unit; an item value acquisition unit that generates an answer based on a trained language model by providing a generator with an instruction to extract the second character string and a second item value from the second character string, and a prompt including the retrieved first character string and the first item value, thereby generating the second item value; An information processing system comprising: [Item 2] Item 1, an information processing system according to item 1, the storage unit stores the first character string, the first item value, and first vector data in which the first character string has been embedded; the search unit compares second vector data in which a second character string has been embedded with the first vector data to search for the first character string similar to the second character string; An information processing system characterized by: [Item 3] Item 1, an information processing system according to item 1, the search unit searches for a plurality of pairs of the first character string and the first item; the item value acquisition unit includes a plurality of pairs of the first character string and the first item in the prompt; An information processing system characterized by: [Item 4] storing a first character string read from the first image data and a first item value extracted from the first character string; obtaining a second character string read from the second image data; acquiring the first character string similar to the second character string and the corresponding first item value from the storage unit; a step of generating the second item value by providing a prompt including the second string and an instruction to extract a second item value from the second string, and the retrieved first string and the first item value to a generator that generates an answer based on a trained language model; An information processing method characterized by being executed by a computer. [Item 5] storing a first character string read from the first image data and a first item value extracted from the first character string; obtaining a second character string read from the second image data; acquiring the first character string similar to the second character string and the corresponding first item value from the storage unit; a step of generating the second item value by providing a prompt including the second string and an instruction to extract a second item value from the second string, and the retrieved first string and the first item value to a generator that generates an answer based on a trained language model; A program that causes a computer to execute the following. [Explanation of symbols]

[0032] 1. User terminal 2 Management Server

Claims

1. a storage unit that stores a first character string read from the first image data and a first item value extracted from the first character string; a character string acquisition unit that acquires a second character string read from the second image data; a search unit that acquires the first character string similar to the second character string and the corresponding first item value from the storage unit; an item value acquisition unit that generates an answer based on a trained language model by providing a generator with an instruction to extract the second character string and a second item value from the second character string, and a prompt including the retrieved first character string and the first item value, thereby generating the second item value; An information processing system comprising:

2. 2. The information processing system according to claim 1, the storage unit stores the first character string, the first item value, and first vector data in which the first character string has been embedded; the search unit compares second vector data in which a second character string has been embedded with the first vector data to search for the first character string similar to the second character string; An information processing system characterized by:

3. 2. The information processing system according to claim 1, the search unit searches for a plurality of pairs of the first character string and the first item; the item value acquiring unit includes a plurality of pairs of the first character string and the first item in the prompt; An information processing system characterized by:

4. storing a first character string read from the first image data and a first item value extracted from the first character string; acquiring a second character string read from the second image data; acquiring the first character string similar to the second character string and the corresponding first item value from the storage unit; a step of generating the second item value by providing a prompt including the second string, an instruction to extract a second item value from the second string, and the retrieved first string and the first item value to a generator that generates an answer based on a trained language model; An information processing method characterized by being executed by a computer.

5. storing a first character string read from the first image data and a first item value extracted from the first character string; acquiring a second character string read from the second image data; acquiring the first character string similar to the second character string and the corresponding first item value from the storage unit; a step of generating the second item value by providing a prompt including the second string, an instruction to extract a second item value from the second string, and the retrieved first string and the first item value to a generator that generates an answer based on a trained language model; A program that causes a computer to execute the following.

Images