Information processing system, information processing method, and program

Abstract

To enable extraction of item data from an image. [Solution] An information processing system comprising: a memory unit that stores a first string read from first image data and a first item value extracted from the first string; a string acquisition unit that acquires a second string read from second image data; a search unit that acquires a first string similar to the second string and the corresponding first item value from the memory unit; and an item value acquisition unit that gives a generator that generates an answer based on a trained language model an instruction to extract the second string and the second item value from the second string, as well as a prompt including the searched first string and first item value, to generate the second item value.

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing method, and a program. [Background technology]

[0002] Patent Document 1 discloses obtaining character data from an image by OCR processing. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-073201 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, it is difficult to extract specific items from the scanned characters.

[0005] The present invention has been made in view of the above background, and has an object to make it possible to extract item data from an image. [Means for solving the problem]

[0006] The main invention of the present invention for solving the above problem is an information processing system comprising: a memory unit that stores a first string read from first image data and a first item value extracted from the first string; a string acquisition unit that acquires a second string read from second image data; a search unit that acquires the first string similar to the second string and the corresponding first item value from the memory unit; and an item value acquisition unit that generates an answer based on a trained language model by giving an instruction to a generator to extract the second string and the second item value from the second string, and a prompt including the searched first string and the first item value, to generate the second item value.

[0007] Other problems and solutions disclosed in this application will be made clear in the description of the preferred embodiments of the invention and the drawings. Effect of the Invention

[0008] According to the present invention, item data can be extracted from an image. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of an overall configuration of an information processing system. [Diagram 2] 2 is a diagram illustrating an example of a hardware configuration of a management server 2. FIG. [Diagram 3] 2 illustrates an example of the software configuration of a management server 2. FIG. [Figure 4] FIG. 2 is a diagram showing an example of a record stored in a vector store 231. [Diagram 5] FIG. 13 is a diagram illustrating the operation of the management server 2. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] <System Overview> An information processing system according to an embodiment of the present invention will be described below. The information processing system of this embodiment is intended to extract specific items displayed in an image. In the information processing system of this embodiment, character strings are recognized from an image (OCR), and specific items are extracted from the recognized character strings using a large-scale language model (LLM). The information processing system of this embodiment is intended to improve extraction accuracy by having previously extracted examples learned by short-shot learning.

[0011] 1 is a diagram showing an example of the overall configuration of an information processing system. The information processing system of this embodiment is configured to include a management server 2. The management server 2 is communicatively connected to each of a user terminal 1 and a generation server 3 via a communication network. The communication network is, for example, the Internet, and is constructed by a public telephone line network, a mobile phone line network, a wireless communication path, Ethernet (registered trademark), or the like.

[0012] The user terminal 1 is a computer operated by a user. The user terminal 1 can be, for example, a smartphone, a tablet computer, a personal computer, etc. The user can operate the user terminal 1 to upload images to be read, such as form images, to the management server 2.

[0013] The management server 2 may be a general-purpose computer such as a workstation or a personal computer, or may be logically realized by cloud computing. The management server 2 operates as a SaaS that returns the result of the OCR processing to the user terminal 1.

[0014] The generation server 3 is a computer that provides a function for generating answers using a large-scale limit model (LLM). The generation server 3 functions as a generator that provides an externally provided prompt to the large-scale limit model to generate an answer. The generation server 3 provides the function of the generator using the large-scale limit model as an API (Application Programming Interface).

[0015] <Administration Server> FIG. 2 is a diagram showing an example of the hardware configuration of the management server 2. Note that the illustrated configuration is an example, and other configurations may be used. The management server 2 includes a CPU 201, a memory 202, a storage device 203, a communication interface 204, an input device 205, and an output device 206. The storage device 203 is, for example, a hard disk drive, a solid state drive, or a flash memory that stores various data and programs. The communication interface 204 is an interface for connecting to a communication network, for example, an adapter for connecting to Ethernet (registered trademark), a modem for connecting to a public telephone line network, a wireless communication device for wireless communication, a USB (Universal Serial Bus) connector or an RS232C connector for serial communication, etc. The input device 205 is, for example, a keyboard, a mouse, a touch panel, a button, a microphone, etc. that input data. The output device 206 is, for example, a display, a printer, a speaker, etc. that output data. Each functional unit of the management server 2 described below is realized by the CPU 201 reading a program stored in the storage device 203 into the memory 202 and executing it, and each storage unit of the management server 2 is realized as part of the storage area provided by the memory 202 and the storage device 203.

[0016] 3 is a diagram showing an example of the software configuration of the management server 2. The management server 2 includes a vector store 231, an image acquisition unit 211, a character string acquisition unit 212, a search unit 213, and an item value acquisition unit 214.

[0017] <Storage section> The vector store 231 stores values ​​(vector data) obtained by embedding character strings previously read from image data. The vector store 231 can store character strings previously read from image data, items extracted from the character strings (pairs of item names and item values), and vector data in association with each other. The character strings previously read from image data and / or items extracted from the character strings may be managed in a file or the like, and the vector store 231 may store a path or URL of the file. The embedding process may be performed by the generation server 3 using an Embeddings API provided by the generation server 3, or the management server 2 may perform the embedding process using a method such as Word2Vec, Doc2Vec, or fastText.

[0018] 4 is a diagram showing an example of a record stored in the vector store 231. As shown in the diagram, each record registered in the vector store 231 may include a character string (text data) read from image data, vector data (which may have any number of dimensions) that is the result of embedding the character string, and items (pairs of item names and item values) extracted from the character string expressed in JSON format.

[0019] <Functional section> The image acquisition unit 211 acquires images such as form images from a user. For example, the image acquisition unit 211 can accept uploading of images from the user terminal 1. For example, the image acquisition unit 211 may acquire images sent via e-mail or a chat service. The image acquisition unit 211 may accept a URL to an online storage in which images are stored from a user, and access the URL to acquire the images.

[0020] The character string acquisition unit 212 acquires a character string read from image data. The character string acquisition unit 212 can recognize a character string (hereinafter, a target character string) displayed in an image (hereinafter, a target image) acquired by the image acquisition unit 211, and acquire the recognized character string. Note that the process of recognizing and acquiring a character string from an image is assumed to be a general OCR process, and a detailed description thereof will be omitted here.

[0021] The search unit 213 acquires a character string similar to the target character string from the vector store 231. The search unit 213 creates vector data in which the target character string is embedded, and can search for a character string (text data) similar to the target character string according to the distance between the created vector data and the vector data stored in the vector store 231. The embedding process may be performed by the generation server 3 using an Embeddings API provided by the generation server 3, or the management server 2 may perform the embedding process using a method such as Word2Vec, Doc2Vec, or fastText. The search unit 213 acquires a similar character string and an item (extracted item) corresponding to the character string from the vector store 231. The search unit 213 can search for a predetermined number (which can be any number) of pairs of text data and extracted items in order of decreasing distance. In this embodiment, it is assumed that three pairs of text data and extracted items are searched for.

[0022] The item value acquisition unit 214 acquires an item value from a character string read from an image. The item value acquisition unit 214 can generate result data by extracting pairs of an item name and an item value from a character string by providing a prompt including a character string, an instruction to extract an item value from the character string, and a previously extracted example to a generator that generates an answer based on a trained language model. The character string provided to the prompt is a target character string acquired by the character string acquisition unit 212. The previously extracted example provided to the prompt is a set of text data and extracted items acquired by the search unit 213 from the vector store 231. This allows the item value acquisition unit 214 to learn how to extract items from text data using a short-shot learning method, and to have the LLM extract items from the target character string. In this embodiment, it is assumed that the item value acquisition unit 214 calls an API provided by the generation server 3 to provide a prompt to the LLM included in the generation server 3 and receive an output result from the LLM, but the management server 2 may be provided with a trained language model and a generator.

[0023] <Operation> FIG. 5 is a diagram for explaining the operation of the management server 2. As shown in FIG.

[0024] The management server 2 accepts an image (S301), reads a character string from the image (S302), searches for past text data and examples of extracted items that are similar to the target character string (S303), sends a prompt containing the target character string, text data, and examples of extracted items to the generation server 3 (S304), and receives the results of item extraction using LLM from the generation server 3 (S305).

[0025] As described above, according to the information processing system of this embodiment, it is possible to extract necessary items from character strings read from an image by using a large scale limit model (LLM).

[0026] Although the present embodiment has been described above, the above embodiment is intended to facilitate understanding of the present invention, and is not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit of the present invention, and equivalents thereof are also included in the present invention.

[0027] For example, the processes performed by each of the functional units of the management server 2 described above may be executed by any of the functional units. Also, a different functional unit that executes part of the processes performed by each of the functional units described above may be added. Also, the functional units of the management server 2 may be provided in a distributed manner on multiple computers.

[0028] Furthermore, the information stored in each storage unit of the management server may be stored by any of the storage units. That is, the information stored in the above-mentioned multiple storage units may be stored by one storage unit, or part of the information stored in one of the above-mentioned storage units may be stored by another storage unit.

[0029] In the above embodiment, only the target character string is embedded, but an image may be embedded in addition to or instead of the target character string. In this case, the search unit 213 obtains vector data (image vector) in which the image acquired by the image acquisition unit 211 is embedded in addition to or instead of the vector data (character string vector) in which the target character string is embedded, and can search for character strings (text data) similar to the target character string and / or image according to a value (for example, a total value, an average value, etc.) based on the distance between the character string vector and the vector data stored in the vector store 231 and the distance between the image vector and the vector data stored in the vector store 231. For example, a known method such as MatchingNet can be used for the image embedding process.

[0030] In addition to the target character string and / or image, a graph representation showing the connection relationship between words (or phrases; the same applies below) extracted from the target character string may be embedded. In this case, for example, words are extracted from the target character string, the extracted positions of the extracted words on the image are specified, a graph is created linking a first word with each of second words within a predetermined distance from the first word on the image, and the created graph is embedded. The vector store 231 stores, in association with each other, vector data (character vector) in which a character string read from an image is embedded, and vector data (graph vector) in which a graph is embedded in which links are set to word pairs based on the positions of words in a character string previously read from image data on the image data. The search unit 213 can search for a string (text data) similar to the target string and / or image and the graph showing the links between words in the target string, based on a value (e.g., a sum, an average, etc.) based on the distance between the vector data in which the target string and / or image is embedded and the character vector in the vector store 231, and the distance between the vector data in which a graph showing the links between words extracted from the target string is embedded and the graph vector in the vector store 231.

[0031] <Disclosures> The present disclosure also includes the following configurations. [Item 1] a storage unit that stores a first character string read from the first image data and a first item value extracted from the first character string; a character string acquisition unit that acquires a second character string read from the second image data; a search unit that acquires the first character string similar to the second character string and the corresponding first item value from the storage unit; an item value acquisition unit that generates an answer based on a trained language model by providing a generator with an instruction to extract the second character string and a second item value from the second character string, and a prompt including the searched first character string and the first item value, thereby generating the second item value; An information processing system comprising: [Item 2] The information processing system according to item 1, the storage unit stores the first character string, the first item value, and first vector data in which the first character string has been embedded; the search unit compares second vector data in which a second character string has been embedded with the first vector data to search for the first character string similar to the second character string; An information processing system comprising: [Item 3] The information processing system according to item 1, The search unit searches for a plurality of pairs of the first character string and the first item, the item value acquisition unit includes a plurality of pairs of the first character string and the first item in the prompt; An information processing system comprising: [Item 4] storing a first character string read from the first image data and a first item value extracted from the first character string; obtaining a second character string read from the second image data; acquiring the first character string similar to the second character string and the corresponding first item value from the storage unit; providing a prompt including the retrieved first string and the first item value to a generator that generates an answer based on a trained language model, the second string being extracted from the second string, and an instruction to extract a second item value from the second string, and the first string being retrieved to generate the second item value; An information processing method characterized by being executed by a computer. [Item 5] storing a first character string read from the first image data and a first item value extracted from the first character string; obtaining a second character string read from the second image data; acquiring the first character string similar to the second character string and the corresponding first item value from the storage unit; providing a prompt including the retrieved first string and the first item value to a generator that generates an answer based on a trained language model, the second string being extracted from the second string, and an instruction to extract a second item value from the second string, and the first string being retrieved to generate the second item value; A program for causing a computer to execute the following. [Explanation of symbols]

[0032] 1 User terminal 2 Management Server

Claims

1. a storage unit that stores a first character string read from the first image data and a first item value extracted from the first character string; a character string acquisition unit that acquires a second character string read from the second image data; a search unit that acquires the first character string similar to the second character string and the corresponding first item value from the storage unit; an item value acquisition unit that generates an answer based on a trained language model by providing a generator with an instruction to extract the second character string and a second item value from the second character string, and a prompt including the searched first character string and the first item value, thereby generating the second item value; An information processing system comprising:

2. 2. The information processing system according to claim 1, the storage unit stores the first character string, the first item value, and first vector data in which the first character string has been embedded; the search unit compares second vector data in which a second character string has been embedded with the first vector data to search for the first character string similar to the second character string; An information processing system comprising:

3. 2. The information processing system according to claim 1, The search unit searches for a plurality of pairs of the first character string and the first item value; the item value acquiring unit includes a plurality of pairs of the first character string and the first item value in the prompt; An information processing system comprising:

4. storing a first character string read from the first image data and a first item value extracted from the first character string in a storage unit; obtaining a second character string read from the second image data; acquiring the first character string similar to the second character string and the corresponding first item value from the storage unit; providing a prompt including the retrieved first string and the first item value to a generator that generates an answer based on a trained language model, the second string being extracted from the second string, and an instruction to extract a second item value from the second string, and the first string being retrieved to generate the second item value; An information processing method characterized by being executed by a computer.

5. storing a first character string read from the first image data and a first item value extracted from the first character string in a storage unit; obtaining a second character string read from the second image data; acquiring the first character string similar to the second character string and the corresponding first item value from the storage unit; providing a prompt including the retrieved first string and the first item value to a generator that generates an answer based on a trained language model, the second string being extracted from the second string, and an instruction to extract a second item value from the second string, and the first string being retrieved to generate the second item value; A program for causing a computer to execute the following.

Images