Method, program, and information processing device for collecting character information printed on printed material

Abstract

To provide a method, a program, and an information processing device for collecting character information related to predetermined items from printed materials of various formats.SOLUTION: A method performed by an information processing device 1 includes the steps for acquiring a group of character strings read from a printed material by optical character recognition, generating a first prompt that instructs a large-scale language model 6 to extract from the group of character strings one or more pieces of character information relating to one or more predetermined items, and providing the generated first prompt to the large-scale language model 6 to obtain one or more pieces of character information extracted from a group of character strings in response to the first prompt from the large-scale language model 6.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a method, a program, and an information processing device for collecting character information printed on a printed matter. [Background technology]

[0002] Optical character recognition (OCR) is a technology that automatically detects letters and numbers from images such as printed matter or handwritten documents and converts them into text that can be understood by a computer. The following patent document describes a method of OCR processing of non-standard documents such as medical certificates and outputting them as structured data. In this method, multiple boxes are detected from an image of the non-standard document, and character strings are detected from areas recognized in relation to each of the detected multiple boxes. Then, combinations of detected character strings are detected based on the positional relationship between each of the detected multiple boxes, and this combination is output as structured data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-99623 Summary of the Invention [Problem to be solved by the invention]

[0004] In the method of the above patent document, a combination of character strings (a combination of an item name and a numerical value) is detected based on the positional relationship of frames, so this method cannot be used for printed matter that does not include frames. Furthermore, when attempting to obtain character information (a string of numerical values ​​or other characters) relating to a specific item printed on a printed matter, the method of the above patent document presumes that the item name is printed, so this method cannot be used for printed matter on which only character information (a string of characters) is printed, without an item name. In other words, conventional methods cannot collect character information relating to a specific item from a variety of printed matter with no set format, such as advertising leaflets.

[0005] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a method, a program, and an information processing device that are capable of collecting character information relating to a predetermined item from printed matter in various formats. [Means for solving the problem]

[0006] A first aspect of the present invention is a method for an information processing device to collect character information printed on a printed material, the method comprising the steps of: the information processing device acquiring a group of character strings read from an image of the printed material by optical character recognition; the information processing device generating a first prompt that instructs a large-scale language model to extract one or more pieces of character information relating to one or more specified items from the group of character strings; and the information processing device providing the generated first prompt to the large-scale language model, and acquiring from the large-scale language model the one or more pieces of character information extracted from the group of character strings in response to the first prompt.

[0007] A second aspect of the present invention is a program including instructions for causing an information processing device to perform a process of collecting character information printed on a printed material, the process performed by the information processing device in accordance with the instructions comprising: a step of acquiring a group of character strings read from an image of the printed material by optical character recognition; a step of generating a first prompt that instructs a large-scale language model to extract one or more pieces of character information relating to one or more specified items from the group of character strings; and a step of providing the generated first prompt to the large-scale language model and acquiring from the large-scale language model the one or more pieces of character information extracted from the group of character strings in response to the first prompt.

[0008] A third aspect of the present invention is an information processing device that performs processing to collect character information printed on a printed material, the information processing device having a processing unit and a memory unit that stores instructions to be executed by the processing unit, and the processing performed by the processing unit in accordance with the instructions includes a step of acquiring a group of character strings read from an image of the printed material by optical character recognition, a step of generating a first prompt that instructs a large-scale language model to extract one or more pieces of character information relating to one or more specified items from the group of character strings, and a step of providing the generated first prompt to the large-scale language model and acquiring from the large-scale language model the one or more pieces of character information extracted from the group of character strings in response to the first prompt.

[0009] A third aspect of the present invention is an information processing device that performs a process of collecting character information printed on a printed material, the information processing device having: a means for acquiring a group of character strings read from an image of the printed material by optical character recognition; a means for generating a first prompt that instructs a large-scale language model to extract one or more pieces of character information relating to one or more specified items from the group of character strings; and a means for providing the generated first prompt to the large-scale language model and acquiring from the large-scale language model the one or more pieces of character information extracted from the group of character strings in response to the first prompt. Effect of the Invention

[0010] According to the present invention, it is possible to provide a method, a program, and an information processing device that are capable of collecting character information relating to a predetermined item from printed matter in various formats. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a system according to the present embodiment. [Diagram 2] FIG. 2 is a diagram for explaining an example of a process for acquiring a group of character strings from an image of a printed matter by OCR. [Diagram 3] FIG. 3 is a diagram for explaining an example of a process for extracting character information relating to a predetermined item from a group of character strings read by OCR. [Figure 4] 4A and 4B are diagrams showing an example of an image of a printed matter. [Diagram 5] FIG. 5 is a diagram showing an example of a group of character strings obtained by OCR from an image of a printout. [Figure 6] FIG. 6 is a diagram showing an example of a prompt given to a large-scale language model. [Figure 7] FIG. 7 is a diagram showing an example of character information extracted by a large-scale language model. [Figure 8] FIG. 8 is a diagram for explaining a modified example of the process of extracting character information relating to a predetermined item from a group of character strings read by OCR. [Figure 9] FIG. 9 is a diagram for explaining a modified example of the process of extracting character information relating to a predetermined item from a group of character strings read by OCR. [Figure 10] FIG. 10 is a diagram for explaining a modified example of the process of extracting character information relating to a predetermined item from a group of character strings read by OCR. [Figure 11] FIG. 11 is a diagram for explaining a modified example of the process of acquiring a group of character strings from an image of a printed matter by OCR. [Figure 12] FIG. 12 is a diagram showing an example of table data read by OCR from an image of a printed matter. [Figure 13] FIG. 13 is a diagram showing an example of a prompt given to a large-scale language model. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Fig. 1 is a diagram showing an example of the configuration of a system according to this embodiment. The system shown in the example of Fig. 1 includes an information processing device 1, a terminal device 3, an OCR processing server 5, and a large-scale language model 6 that are capable of communicating via a communication network 9 such as the Internet. The information processing device 1 is an example of an information processing device of the present invention. The large scale language model 6 is an example of a large scale language model of the present invention.

[0013] In the system shown in FIG. 1, a group of character strings is read from an image of a printout obtained by the terminal device 3 through OCR processing, and character information relating to a predetermined item is extracted from the group of character strings.

[0014] [Terminal device 3] The terminal device 3 is a device operated by a worker who performs work to collect character information printed on printed matter such as advertising flyers, and is, for example, a device equipped with an information communication function such as a personal computer, a tablet, a smartphone, etc. In the example of the system in Fig. 1, there is one terminal device 3, but in other examples of this embodiment, the system may have multiple terminal devices 3 corresponding to multiple workers.

[0015] The terminal device 3 includes, for example, a communication unit, a storage unit, and a processing unit similar to the communication unit 11, the storage unit 12, and the processing unit 13 of the information processing device 1 described later. The terminal device 3 may also include an input unit (such as a touch panel, a mouse, or a keyboard) for inputting a user's instruction to the processing unit, a display unit (such as a liquid crystal display or an organic EL display) for displaying an image corresponding to the image data generated in the processing unit, and a speaker for outputting sound.

[0016] A scanner 4 for acquiring an image of a printed matter is connected to the terminal device 3. The terminal device 3 provides the image of the printed matter acquired by the scanner 4 to the information processing device 1.

[0017] [OCR processing server 5] The OCR processing server 5 is a server that performs OCR processing, reads a group of character strings (letters, numbers, etc.) from an image of a printed matter in response to a request from the information processing device 1, and provides the read group of character strings to the information processing device 1.

[0018] In the example of the system in FIG. 1, the OCR processing is performed by the OCR processing server 5, but in other examples of this embodiment, the information processing device 1 may perform the OCR processing.

[0019] [Large-scale language model 6] The large-scale language model 6 is a system configured to mimic human language understanding capabilities by having an artificial neural network with a large number of parameters (e.g., billions to trillions) learn a large amount of text data. The large-scale language model 6 can perform a wide range of natural language processing tasks, such as understanding sentence patterns and context, answering questions, generating sentences, and translating, according to given prompts. As the large-scale language model 6, for example, the GPT series (Chat GPT, etc.) operated by OpenAI, Inc. in the United States can be used.

[0020] [Information processing device 1] The information processing device 1 is a device that performs processing to collect character information printed on a printed matter. The information processing device 1 reads a group of character strings from an image of the printed matter acquired by a terminal device 3 using OCR in an OCR processing server 5, and extracts character information related to a predetermined item from the group of character strings using natural language processing in a large-scale language model 6. For example, the information processing device 1 includes one or more computers connected to a communication network 9. The information processing device 1 shown in the example of FIG. 1 has a communication unit 11, a storage unit 12, and a processing unit 13.

[0021] The communication unit 11 communicates with other devices (such as the terminal device 3, the OCR processing server 5, and the large-scale language model 6) via the communication network 9. The communication unit 11 includes a device (such as a network interface card) that communicates in compliance with a predetermined communication standard such as Ethernet (registered trademark) or wireless LAN.

[0022] The storage unit 12 stores one or more programs 121 including instructions executed by the processing unit 13, data temporarily stored during processing by the processing unit 13, data used in processing by the processing unit 13, data obtained as a result of processing by the processing unit 13, and the like. The storage unit 12 may include, for example, a main storage device (RAM, ROM, etc.) and an auxiliary storage device (flash memory, SSD, hard disk, memory card, optical disk, etc.). The storage unit 12 may be composed of one storage device or may be composed of multiple storage devices. When the storage unit 12 is composed of multiple storage devices, each storage device is connected to the processing unit 13 via a computer bus or any other communication means.

[0023] The processing unit 13 generally controls the overall operation of the information processing device 1 and executes predetermined information processing. The processing unit 13 includes, for example, one or more processors (such as a central processing unit (CPU), a micro-processing unit (MPU), or a digital signal processor (DSP)) that perform processing according to instructions of one or more programs 121 stored in the storage unit 12. The processing unit 13 operates as a computer by the one or more processors executing instructions of one or more programs 121 stored in the storage unit 12.

[0024] The processing unit 13 may include one or more dedicated hardware (such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA)) configured to realize a specific function. In this case, the processing unit 13 may execute all of the processes described in this embodiment in a computer, or may execute at least a part of the processes in dedicated hardware.

[0025] The program 121 may be recorded, for example, in a computer-readable recording medium (optical disk, memory card, USB memory, or other non-transitory tangible medium). The processing unit 13 may read at least a part of the one or more programs 121 recorded in such a recording medium by a recording medium reading device (optical disk device, etc.) or an interface device (USB interface, etc.) not shown, and write the program in the storage unit 12. Alternatively, the processing unit 13 may download at least a part of the one or more programs 121 from another device connected to the communication network 9 by the communication unit 11, and write the program in the storage unit 12. The one or more programs 121 include an instruction for causing the processing unit 13 to perform at least a part of the processing according to this embodiment described later.

[0026] [Storage device 2] The storage device 2 stores various information used in the processing of the information processing device 1. The information processing device 1 and the storage device 2 can communicate with each other via any communication path (such as a LAN, a dedicated line network, or the Internet). For example, the storage device 2 may be included in a file server or a database server that accepts access from multiple devices, or may be a dedicated storage device that is accessible only to the information processing device 1. In the example of FIG. 1, the storage device 2 stores a printed matter database 21 and a collected information database 22. In the following description, database may be abbreviated to "DB."

[0027] The printed matter DB 21 includes a plurality of pieces of printed matter information related to a plurality of printed matters (such as advertising flyers) whose images have been acquired by the scanner 4 of the terminal device 3. One piece of printed matter information includes, for example, at least a part of the following information as information related to one printed matter: -Identification information for identifying printed matter (printed matter ID) - Images of printed matter or their storage location -Type of information printed on the printed matter (Example) If the printed material is an advertising flyer, the type of product or service being advertised, etc. Information about the publication date of the printed matter

[0028] The collected information DB 22 includes a plurality of pieces of printed matter collection information each collected based on an image of a printed matter. Each piece of printed matter collection information includes at least a part of the following information as information collected based on an image of a printed matter: Print ID A group of characters read by OCR from an image of a printed matter One or more pieces of text information about one or more given items extracted from a set of character strings -The reliability of the text information extraction results for each item was evaluated using a large-scale language model6

[0029] Here, the operation of the system having the above-mentioned configuration shown in FIG. 1 will be described. FIG. 2 is a diagram for explaining an example of a process for acquiring a group of character strings from an image of a printed matter by OCR.

[0030] The terminal device 3 displays a screen on the display of the display unit or the like, which prompts the worker to input information about the printed matter that is the subject of information collection (such as the type of information printed on the printed matter and the publication date of the printed matter), and acquires the information about the printed matter input by the worker's input operation in response to this screen (ST100). The terminal device 3 also acquires an image of the printed matter (such as a PDF file) captured by the scanner 4 through the worker's operation (ST105). The terminal device 3 provides the information about the printed matter acquired in step ST100 and the image of the printed matter acquired in step ST105 (such as an image file or PDF file) to the information processing device 1 (ST120).

[0031] When the information processing device 1 acquires information about the printed matter and an image of the printed matter from the terminal device 3 (ST125), it registers the acquired information in the printed matter DB 21 (ST130). The information processing device 1 also requests the OCR processing server 5 to perform OCR on the image of the printed matter provided from the terminal device 3 (ST135). For example, the information processing device 1 utilizes an API provided by the OCR processing server 5 to request OCR processing of the image of the printed matter.

[0032] The OCR processing server 5, which has received the request for OCR processing, performs OCR on the image of the printed matter provided by the information processing device 1, and recognizes character strings printed on the printed matter (ST140). The OCR processing server 5 provides a group of character strings recognized from the image of the printed matter as OCR results to the requesting information processing device 1 (ST145).

[0033] The information processing device 1 acquires a group of character strings (OCR results) read from an image of a printed matter by OCR from the OCR processing server 5 (ST170). The information processing device 1 registers the group of character strings acquired as the OCR results in the collected information DB 22 (ST180).

[0034] 4A and 4B are diagrams showing an example of an image of a printed matter from which information is collected. This printed matter is a flyer on which an advertisement for an apartment rental property is printed, and FIGS. 4A and 4B show the front and back of the flyer, respectively. FIG. 5 is a diagram showing an example of a group of character strings read by OCR from the image of the printed matter shown in FIGS. 4A and 4B. As can be seen from the OCR result in FIG. 5, the character strings read from the printed matter tend to be placed relatively close to each other in the text data as they are printed closer to each other on the printed matter. However, if the format of the printed matter is not determined, the context and appearance position of the character strings in the text data of the OCR result, and the pattern of words and phrases representing a specific item are not determined. Therefore, it is difficult for a rule-based algorithm to extract character information related to a specific item from such text data of the OCR result. Therefore, in this embodiment, a large-scale language model 6 is used to perform a process of extracting character information related to a specific item from the OCR result.

[0035] FIG. 3 is a diagram for explaining an example of a process for extracting character information relating to a predetermined item from a group of character strings read by OCR.

[0036] The information processing device 1 generates a prompt (hereinafter referred to as a “first prompt”) that instructs the large-scale language model 6 to extract one or more pieces of character information relating to one or more predetermined items from a group of character strings in the OCR result (ST235).

[0037] 6 is a diagram showing an example of a first prompt to be given to the large-scale language model 6. The first prompt shown in this example includes a description P1 regarding a task to be requested of the large-scale language model 6, a description P2 regarding items to be extracted, a description P3 regarding an output format of the item extraction results, a description P4 regarding supplemental instructions and advice to be given to the large-scale language model 6, and a description P5 regarding a group of character strings (property information) of the OCR results.

[0038] Description P1 regarding the task to be requested of the large-scale language model 6 instructs the model to extract specified items from the property information (OCR results), to evaluate the reliability of the extraction results of character information related to each of the specified items, and to format the extraction results of character information related to the items in accordance with a specified output format.

[0039] Description P2 of items to be extracted specifies items to be extracted from advertisements for rental properties. In the example of Fig. 6, the predetermined items include the address of the rental property, the property name, the number of floors in the building, the layout of the room, the nearest station, the structure of the building, the year of construction of the building, common area charges, security deposit, key money, room orientation, whether there is an elevator, whether there is an auto-lock, whether there is a balcony, free internet, furniture and appliances included, corner room, and other items (features, notes).

[0040] The description P3 regarding the output format of the item extraction results specifies that the OCR results are to be formatted in JSON format. This description P3 also includes a sample of the OCR results formatted in JSON format. By presenting one sample, it is possible to have the large-scale language model 6 format the OCR results in the same way as the sample, without specifying a detailed format for each item.

[0041] The supplementary instruction and advice description P4 instructs the user to respond that the character information of an item is unknown if the character information of the item cannot be extracted from the OCR result (a group of character strings). This effectively prevents the large-scale language model 6 from responding with low-accuracy character information about an item whose character information is unknown.

[0042] Furthermore, entry P4 specifies that the building construction date should be entered in the Gregorian calendar, and that the planned construction date (dates later than the current date) should also be extracted.

[0043] The description P5 relating to a group of character strings (property information) of the OCR result includes the text data (FIG. 5) of the OCR result acquired from the OCR processing server 5 in step ST170.

[0044] The information processing device 1 provides the first prompt (including the text data of the OCR result) generated in step ST235 to the large-scale language model 6 (ST240). The large-scale language model 6 extracts one or more pieces of character information related to one or more predetermined items from the OCR text data (a group of character strings) of the printed matter in accordance with the first prompt acquired from the information processing device 1 (ST245). The large-scale language model 6 also evaluates the reliability of the extraction result of the character information related to each item in accordance with the first prompt, and acquires the evaluation result (e.g., a percentage value). The large-scale language model 6 provides the character information extracted for each item and the evaluation result of the reliability of the extraction result to the information processing device 1 (ST255).

[0045] In response to the first prompt, the information processing device 1 obtains character information of one or more items extracted from the text data (a group of character strings) of the OCR result and an evaluation result of the reliability of the character information from the large-scale language model 6 (ST260), and registers it in the collected information DB22 (ST270).

[0046] Fig. 7 is a diagram showing an example of character information extracted by the large-scale language model 6 from the OCR result shown in Fig. 5 in accordance with the first prompt shown in Fig. 6. In the example of Fig. 7, the character information of each item is formatted in JSON format in accordance with the output format specified in the description P3.

[0047] The information processing device 1 provides the terminal device 3 with character information of one or more items extracted in the large-scale language model 6 (ST275), and the terminal device 3 displays the character information of one or more items acquired from the information processing device 1 on the display of the display unit (ST280). This allows the operator operating the terminal device 3 to check the character information of each item extracted from the OCR result by the large-scale language model 6. If an error in the character information is found by this check, the information processing device 1 may correct the character information registered in the collected information DB 22 in response to an instruction from the terminal device 3 to correct the character information.

[0048] As described above, according to this embodiment, even if the format of the printed matter is not fixed, such as an advertising flyer introducing various rental properties, by issuing a first prompt to the large-scale language model 6 to extract character information for each item from the OCR results, appropriate character information that applies to each item can be extracted from the OCR results. Therefore, character information related to a predetermined item can be collected from printed matter of various formats.

[0049] Furthermore, according to this embodiment, when character information regarding a certain item cannot be extracted from the OCR result (a group of character strings) (for example, when there is no description regarding that certain item), an instruction is given in the first prompt to respond that the character information regarding that certain item is unknown, thereby effectively preventing the large-scale language model 6 from responding with low accuracy character information for an item with unknown character information.

[0050] According to this embodiment, the first prompt instructs the user to evaluate the reliability of the extraction result of the character information related to each of one or more items. This enables users of the character information of printed matter registered in the collected information DB 22 to individually determine whether or not to use the character information by referring to the reliability of the extraction result evaluated by the large-scale language model 6.

[0051] Next, some modified examples of the processing in the information processing device 1 described above will be described.

[0052] <Generating the first prompt according to the type of printed information (1)> In the above-described embodiment, an example is given in which character information is extracted for items related to rental properties such as condominiums, but the worker may be allowed to select a type of information other than rental properties (e.g., properties for sale, etc.) to be extracted from the OCR results (a group of character strings). For example, the terminal device 3 inputs type information related to the type selected by the worker as the type of information to be extracted from the OCR results of the printed matter in step ST100 (FIG. 2). The information processing device 1 acquires this type information from the terminal device 3 (ST125) and registers it in the printed matter DB 21 (ST130). In this case, the information processing device 1 generates a first prompt in step ST235 (FIG. 3) that instructs the device to extract one or more pieces of character information related to one or more items determined for the type indicated by the type information from the OCR results (a group of character strings). For example, when the type information indicates a rental property, the information processing device 1 generates a first prompt for rental properties (FIG. 6), and when the type information indicates a property for sale, the information processing device 1 generates a first prompt for properties for sale. This allows the first prompt generated according to the type indicated by the type information to be provided to the large-scale language model 6, making it possible to extract appropriate information that matches the type indicated by the type information from the OCR results (a group of character strings).

[0053] <Generating the first prompt according to the type of printed information (2)> In the above-mentioned modified example (1), the first prompt is generated according to type information input by the operator in the terminal device 3, but in this modified example, the type (type of information to be extracted from the OCR result) is determined based on the OCR result (a group of character strings). Fig. 8 is a diagram for explaining this modified example, showing an example of a process for extracting character information relating to a predetermined item from a group of character strings read by OCR.

[0054] In the modified example shown in FIG. 8, the information processing device 1 searches for a predetermined keyword that characterizes the type of information in the OCR result (a group of character strings) (ST200). For example, if the type of the printed information is a rental property, the OCR result (a group of character strings) contains keywords such as "rent", "deposit", "common service fee", and "renewal fee", and if the type of the printed information is a property for sale, the OCR result (a group of character strings) contains keywords such as "sales price", "sales contract", and "delivery date". The information processing device 1 searches for such keywords in the OCR result. If a predetermined keyword is found by this search, the information processing device 1 determines the type of information printed on the printed matter (=the type of information to be extracted from the OCR result) based on the found keyword (ST205). The information processing device 1 generates a first prompt that instructs the large-scale language model 6 to extract one or more pieces of character information related to one or more items determined for the type determined in step ST205 from the OCR result (a group of character strings) (ST235A), and provides the first prompt to the large-scale language model 6 (ST240). The process from step ST245 onwards is the same as that described in the flow chart shown in FIG.

[0055] According to this modification, the type of information printed on the printed matter (=the type of information to be extracted from the OCR result) is automatically determined, so that the task of the operator inputting type information in step ST100 can be omitted.

[0056] <Generating the first prompt according to the type of printed information (3)> In the above-mentioned modified example (2), the type of information printed on the printed matter is determined based on a predetermined keyword contained in the OCR result (a group of character strings), but in this modified example, the type is determined by using natural language processing of the large-scale language model 6. Figure 9 is a diagram for explaining this modified example.

[0057] In the modified example shown in Fig. 9, the information processing device 1 generates a prompt (hereinafter referred to as "second prompt") that instructs the large-scale language model 6 to determine one type that corresponds to the OCR result (a group of character strings) from a plurality of predetermined types (such as "rental property" and "sale property") (ST210). This second prompt includes, for example, a description that instructs the large-scale language model 6 to determine one type that corresponds to the OCR result (a group of character strings) from a plurality of predetermined types (such as "rental property" and "sale property"), and also includes a description that indicates the OCR result (a group of character strings). The information processing device 1 provides the generated second prompt to the large-scale language model 6 (ST215).

[0058] The large scale language model 6 determines the type that the OCR result (group of character strings) corresponds to in accordance with the second prompt acquired from the information processing device 1 (ST220), and provides the determination result to the information processing device 1 (ST225).

[0059] When the information processing device 1 acquires the type determination result from the large-scale language model 6 (ST230), it generates a first prompt that instructs the large-scale language model 6 to extract one or more pieces of character information related to one or more items determined for the determined type from the OCR result (a group of character strings) (ST235A), and provides the first prompt to the large-scale language model 6 (ST240). The processing from step ST245 onwards is the same as that described in the flow diagram shown in FIG.

[0060] In this modification, the type of information printed on the printed matter (= the type of information to be extracted from the OCR result) is automatically determined, so that the operator does not need to input the type information in step ST100. Even if the keywords that characterize the printed information are not very clear, the natural language processing function of the large-scale language model 6 can be used to obtain an accurate type determination result.

[0061] <Generating the first prompt based on related information obtained from a web search> In the above-described embodiment and its modified example, character information related to a specific item is extracted only from the OCR result (a group of character strings), but if information related to the specific item is not printed on the printed matter or if the information on the printed matter is not the latest, appropriate character information cannot be extracted only from the OCR result (a group of character strings). Therefore, in this modified example, information related to the specific item is extracted while taking into account related information obtained by a web search. Figure 10 is a diagram for explaining this modified example.

[0062] The information processing device 1 generates a first prompt (ST235B) including an instruction (first instruction) for extracting character information related to at least some of the one or more predetermined items from the OCR result (a group of character strings) and an instruction (second instruction) for collecting related information related to the item by a web search based on keywords related to the item included in the OCR result (a group of character strings). The first instruction is for extracting character information from the OCR result (a group of character strings), and may be the same instruction as in the above-described embodiment and modified example.

[0063] The second instruction causes related information about the item to be collected by a web search, and is set for each item. For example, when related information about the orientation of a building in a rental property is to be collected by a web search, the second instruction may be an instruction to search for web pages that contain a unique keyword (property name, address, etc.) indicating a specific rental property and a keyword indicating the orientation of the building to collect related information.

[0064] The information processing device 1 provides the first prompt generated in step ST235B to the large-scale language model 6 (ST240). The large-scale language model 6 extracts character information of each item from the OCR result (a group of character strings) in response to a first instruction included in the first prompt (ST245). The large-scale language model 6 also collects related information for a specific item by a web search in response to a second instruction included in the first prompt (ST250). The large-scale language model 6 provides the character information of each item extracted in step ST245 and the related information of the specific item collected in step ST250 to the information processing device 1 (ST255B). The information processing device 1 acquires the character information and related information provided from the large-scale language model 6 (ST260B).

[0065] The information processing device 1 corrects the character information acquired from the large-scale language model 6 for one item according to the related information acquired for the one item (ST265). For example, when the large-scale language model 6 responds that the character information for the one item is unknown, and the related information acquired for the one item contains information corresponding to the character information, the information processing device 1 may acquire the information contained in the related information as the character information for the one item.

[0066] Furthermore, when character information is extracted for an item, and the related information acquired for the item also contains information corresponding to the character information, if the publication date of the related information on the web is later than the publication date of the printed matter, the information processing device 1 may correct the extracted character information according to the information contained in the related information (information corresponding to the character information). In other words, if the related information is newer than the information printed on the printed matter, the character information extracted from the OCR result may be corrected according to the related information.

[0067] The information processing device 1 registers the character information of each item acquired in step ST260B (if corrected in step ST265, the corrected character information) in the collected information DB 22 (ST270). The process from step ST275 onwards is the same as that described in the flow diagram shown in FIG.

[0068] According to this variant, based on related information collected by a web search, it is possible to obtain character information about items not included in the OCR results (group of character strings) or character information that is newer than that included in the OCR results (group of character strings).

[0069] In the modified example shown in FIG. 10, the large-scale language model 6 is caused to collect related information in step ST250, but the large-scale language model 6 may be caused to perform a process of extracting character information related to a predetermined item from the collected related information. For example, in step ST235B, the information processing device 1 may generate a first prompt further including an instruction (third instruction) to extract character information about one item from related information collected about the one item in response to the second instruction. In this case, in step ST265, the information processing device 1 may correct the character information extracted from the OCR result (a group of character strings) in response to the first instruction in response to the character information extracted from the related information in response to the third instruction. This allows the information processing device 1 to omit the process of extracting character information from related information.

[0070] Also, the large-scale language model 6 may be caused to correct the character information in step ST265. For example, when character information for an item cannot be extracted from an OCR result (a group of character strings), the information processing device 1 may generate a first prompt in step ST235B to include an instruction (fourth instruction) to extract character information for the item from related information. In addition, the information processing device 1 may generate a first prompt in step ST235B so that, if the publication date on the web of related information collected for a certain item is later than the publication date of the printed material, the first prompt includes an instruction (fifth instruction) to modify the character information extracted from the OCR result (a group of character strings) for the certain item in accordance with the character information extracted from the related information for the certain item. This makes it possible to omit the process of correcting the character information based on the related information in the information processing device 1.

[0071] <Process of collecting character information from printed material with a table> In the case of a printed matter on which a table is printed as shown in Fig. 4B, as shown in the OCR result (near the end) in Fig. 5, the relationship between the character strings included in the table becomes unclear, making it difficult to extract correct character information in the large-scale language model 6. Therefore, in this modified example, in the case of a printed matter on which a table is printed, a process is performed in the OCR process of the OCR processing server 5 to read data (table data) that has been formatted into a table from the table of the printed matter. Fig. 11 is a diagram for explaining this modified example, showing an example of a process of acquiring a group of character strings by OCR from an image of a printed matter.

[0072] The terminal device 3 inputs information about the printed matter (ST100) and acquires an image of the printed matter (ST105) in the same manner as in the flow diagram shown in Fig. 2, and if a front is printed on the printed matter (Yes in ST110), performs a process of acquiring an image of the front printed on the printed matter (ST115). For example, the terminal device 3 displays the image of the printed matter acquired in step ST105 on the display of the display unit, and acquires an image of the range of the front specified by an input operation (e.g., operation of a mouse, etc.) by the worker from the image of the printed matter. The terminal device 3 provides the information about the printed matter acquired in step ST100 and the images of the printed matter acquired in steps ST105 and ST115 to the information processing device 1 (ST120).

[0073] When the information processing device 1 acquires information about the printed matter and an image of the printed matter from the terminal device 3 (ST125), it registers the acquired information in the printed matter DB 21 (ST130). The information processing device 1 also requests the OCR processing server 5 to perform OCR on the entire image of the printed matter acquired in step ST105, among the images of the printed matter provided from the terminal device 3 (ST135).

[0074] The OCR processing server 5, which has received the request for OCR processing, performs OCR on the image of the printed matter provided by the information processing device 1 (the entire image acquired in step ST105) and recognizes character strings printed on the printed matter (ST140). The OCR processing server 5 provides the group of character strings recognized from the image of the printed matter as OCR results to the requesting information processing device 1 (ST145).

[0075] When the information processing device 1 obtains from the OCR processing server 5 a group of character strings (OCR results) read from the entire image of the printed material by OCR (ST150), it then requests the OCR processing server 5 to perform OCR on the image of the table obtained in step ST115 (ST155).

[0076] The OCR processing server 5, which has received the OCR processing request, performs OCR on the table image provided by the information processing device 1 (the table image acquired in step ST115) and recognizes character strings from the table image (ST160). The OCR processing server 5 provides the table data recognized from the table image (data in which character strings included in the table are formatted into a table format) as the OCR result to the requesting information processing device 1 (ST165).

[0077] The information processing device 1 acquires table data read from the image of the table by OCR from the OCR processing server 5 (ST175). The information processing device 1 registers the group of character strings (including character strings other than the table) acquired as the OCR result in step ST150 and the table data acquired as the OCR result in step ST175 in the collected information DB 22 (ST180).

[0078] Fig. 12 is a diagram showing an example of table data read by OCR from the image of the table in the print image shown in Fig. 4B. In the example of Fig. 12, character strings belonging to each row and each column of the table are formatted into table data in JSON format.

[0079] When a group of character strings is read by OCR from the entire image of the printed matter and table data is read by OCR from the front of the printed matter, the information processing device 1 generates a first prompt in step ST235 to instruct the device to extract character information for each item from the group of character strings and the table data. In this case, the information processing device 1 may include the group of character strings read by OCR from the entire image of the printed matter (FIG. 5) and the table data read by OCR from the front of the printed matter (FIG. 12) in the description P5 in the first prompt as shown in FIG.

[0080] Alternatively, the information processing device 1 may separate a statement P5 including a group of character strings (FIG. 5) from a statement P6 including table data (FIG. 12), as in the example of the first prompt shown in FIG. 13. In the example of FIG. 13, a statement P1A of a task to be requested of the large-scale language model 6 instructs the large-scale language model 6 to extract character information related to a predetermined item from the separately written "property information" (FIG. 5) and "table" (FIG. 12). The other statements (P2 to P4) in the example of FIG. 13 are the same as the example of the first prompt shown in FIG. 6.

[0081] According to this modification, when a printed matter contains a table, the table data in which the character strings contained in the table are formatted into a table is read by OCR, and the character information is extracted using this table data. Therefore, it is easier to extract the character information of each item accurately than when only a group of character strings that are not formatted into table data is used.

[0082] The present invention is not limited to the above-described embodiment, but includes various variations. Those skilled in the art can make various modifications and substitutions to the components of the above-described embodiment within the technical scope of the present invention or its equivalent.

[0083] For example, in the above-described embodiment, an image of a printed matter acquired on the terminal device 3 is provided to the information processing device 1 via the communication network 9, but if a scanner 4 is connected to the information processing device 1, the information processing device 1 may directly acquire an image of the printed matter captured by the scanner 4. [Explanation of symbols]

[0084] 1...information processing device, 11...communication unit, 12...storage unit, 121...program, 13...processing unit, 2...storage device, 21...printed material DB, 22...collected information DB, 3...terminal device, 4...scanner, 5...OCR processing server, 6...large-scale language model, 9...communication network

Claims

1. A method for collecting character information printed on a printed matter by an information processing device, comprising: acquiring a group of character strings read from an image of the printed matter by optical character recognition using the information processing device; generating a first prompt by the information processing device to instruct a large-scale language model to extract one or more pieces of character information related to one or more predetermined items from the set of character strings; providing the generated first prompt to the large-scale language model by the information processing device, and acquiring, from the large-scale language model, one or more pieces of character information extracted from the group of character strings in response to the first prompt; The method according to claim 1,

2. the generating step of the first prompt includes generating the first prompt instructing the user to answer that the character information regarding one of the items is unknown when the character information regarding the one of the items cannot be extracted from the group of character strings. The method of claim 1.

3. The information processing device includes a step of acquiring type information related to a type of information to be extracted from the group of character strings, the generating step of the first prompt includes generating the first prompt instructing the user to extract, from the group of character strings, one or more pieces of character information related to one or more items defined for the type indicated by the type information. The method of claim 1.

4. The step of acquiring type information includes: searching said set of strings for predefined keywords characterizing said category; determining the type based on the keywords found by the search. The method according to claim 3.

5. The step of acquiring type information includes: generating a second prompt to instruct the large scale language model to determine a corresponding one of a predetermined number of the types based on the set of character strings; providing the generated second prompt to the large-scale language model; and obtaining from the large-scale language model the determined type in response to the second prompt. The method according to claim 3.

6. The step of generating the first prompt includes generating, for at least some of the items, the first prompt including a first instruction to extract character information related to the item from the group of character strings and a second instruction to collect related information related to the item by performing a web search based on keywords related to the item in the group of character strings; the step of acquiring character information includes acquiring, from the large-scale language model, the character information extracted in response to the first instruction and the related information collected in response to the second instruction; The information processing device includes a step of correcting the character information acquired from the large-scale language model for one of the items in accordance with the related information acquired for the one of the items. The method of claim 1.

7. the step of modifying the character information includes, when a publication date of the related information on the web is later than a publication date of the printed matter, modifying the character information in accordance with the related information; The method according to claim 6.

8. the generating step of the first prompt includes generating, for at least some of the items, a first instruction for extracting character information related to the item from the group of character strings, a second instruction for collecting related information related to the item by a web search based on a keyword related to the item in the group of character strings, and a third instruction for extracting the character information for one of the items from the related information collected for the one of the items in response to the second instruction; the step of acquiring character information includes acquiring, from the large-scale language model, the character information extracted from the group of character strings in response to the first instruction and the character information extracted from the related information in response to the third instruction; the step of correcting the character information in accordance with the related information includes a step of correcting the character information extracted from the group of character strings for one of the items in accordance with the first instruction in accordance with the character information extracted from the related information in accordance with the third instruction for the one of the items. The method of claim 1.

9. The method further comprises a step of acquiring table data read from the table of the printed matter by optical character recognition when the printed matter has a table printed thereon, the generating step of the first prompt includes generating the first prompt instructing the user to extract the character information from the group of character strings and the table data when the table data read from the table of the printed matter is acquired. The method of claim 1.

10. The step of generating the first prompt includes generating, for each of the one or more items, the first prompt instructing the user to evaluate reliability of the extraction result of the character information related to the item; The step of obtaining the text information from the large-scale language model includes obtaining a reliability evaluation result for each of the items. The method of claim 1.

11. A program including instructions for causing an information processing device to perform a process of collecting character information printed on a printed matter, The processing performed by the information processing device in accordance with the command includes each step of the method according to any one of claims 1 to 10. program.

12. An information processing device that performs processing to collect character information printed on a printed matter, A processing section; a memory unit that stores instructions to be executed by the processing unit; The processing performed by the processing unit in accordance with the instructions includes each step of the method according to any one of claims 1 to 10. Information processing device.

Images