Information processing device, information processing method, and information processing program

Abstract

An information processing device, an information processing method, and an information processing program are provided that are capable of generating structured data with high accuracy. [Solution] The system comprises an acquisition unit that acquires a target document, a string recognition unit that generates text information, attribute information, and coordinate information for components in the target document, a first determination unit that determines whether the components contain visual information, an image generation unit that cuts out the components and uses them as an input image, a calculation unit that calculates the similarity between the input image and pre-acquired image data by image recognition using a structured object determination model, a second determination unit that determines whether the image is an input image based on the similarity, an output unit that generates structured data for the input image using a visual language model, and an integration unit that integrates the text information generated by the string recognition unit and the structured data generated by the output unit, and the structured object determination model is constructed using one or more pre-acquired image data that has been tagged with tags related to the image information.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, a technique for generating structured data from a target document has been disclosed (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-104411 Summary of the Invention [Means for solving the problem]

[0004] an information processing device according to one embodiment of the present invention includes an acquisition unit that acquires a target document; a string recognition unit that generates text information, attribute information, and coordinate information for each of one or more components in the target document; a first determination unit that determines, for each of the one or more components, whether the component contains visual information based on the text information, attribute information, and coordinate information; an image generation unit that generates one or more input images based on the coordinate information for the one or more components determined by the first determination unit to be components to be cut out; a calculation unit that calculates a similarity between each of the one or more input images and one or more image data acquired in advance through image recognition using a structured object determination model; a second determination unit that determines, based on the similarity, for each of the one or more input images, whether the one or more input images are input images for which structured data will be generated; an output unit that generates structured data for the input images using a visual language model; an integration unit that integrates the text information generated by the string recognition unit and the structured data generated by the output unit; and a memory unit that stores the structured object determination model, wherein the structured object determination model is constructed using one or more image data acquired in advance to which tags related to image information have been assigned.

[0005] In an information processing device according to one embodiment of the present invention, the similarity may be calculated by comparing the contours of one or more input images with the contour features of one or more image data acquired in advance.

[0006] In the information processing device according to an embodiment of the present invention, a visual language model may be used as a structured object determination model.

[0007] In the information processing device according to one embodiment of the present invention, the one or more pieces of image data acquired in advance may include at least one of a graph, a box diagram, a tree diagram, a table, and a flowchart.

[0008] In an information processing device according to one embodiment of the present invention, the first determination unit may determine that one or more components whose attribute information is either "figure" or "table" are components to be cut out.

[0009] The information processing device according to one embodiment of the present invention may further include an input unit, and the input unit may receive input of coordinate information of a desired component out of one or more components.

[0010] an information processing method according to one embodiment of the present invention, in an information processing device having a memory unit that stores a structured object determination model constructed using one or more image data acquired in advance and tagged with image information, the method comprising the steps of: an acquisition step of acquiring a target document; a string recognition step of generating text information, attribute information, and coordinate information for one or more components in the target document; a first determination step of determining, for each of the one or more components, whether the component contains visual information based on the text information, attribute information, and coordinate information; an image generation step of using the one or more components determined in the first determination step to be components to be cut out as one or more input images based on the coordinate information; a calculation step of calculating a similarity between each of the one or more input images and one or more image data acquired in advance by image recognition using the structured object determination model; a second determination step of determining, based on the similarity, whether each of the one or more input images is an input image for generating structured data for the input image; an output step of generating structured data for the input image using a visual language model; and an integration step of integrating the text information generated in the string recognition step with the structured data generated by the output unit.

[0011] an information processing program according to one embodiment of the present invention, which is provided on a computer having a storage unit that stores a structured object determination model constructed using one or more image data items acquired in advance and tagged with image information; an acquisition function that acquires a target document; a string recognition function that generates text information, attribute information, and coordinate information for one or more components in the target document; a first determination function that determines, for each of the one or more components, whether the component contains visual information based on the text information, the attribute information, and the coordinate information; an image generation function that uses the one or more components determined in the first determination function to be components to be cut out as one or more input images based on the coordinate information; a calculation function that calculates a similarity between each of the one or more input images and one or more image data items acquired in advance through image recognition using the structured object determination model; a second determination function that determines, based on the similarity, for each of the one or more input images, whether the one or more input images are input images for which structured data will be generated; an output function that generates structured data for the input images using a visual language model; and an integration function that integrates the text information generated in the string recognition function with the structured data generated by the output unit. [Effects of the Invention]

[0012] According to the present invention, it is possible to provide an information processing device, an information processing method, and an information processing program that are capable of generating structured data with high accuracy. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration and functional units of an information processing apparatus according to a first embodiment of the present invention. [Figure 2] Figure 2 is an example of a target document. [Figure 3] FIG. 3 shows an example of the text information, attribute information, and coordinate information generated by the first generation unit. [Figure 4] FIG. 4 is an example of the structured data generated by the first generation unit. [Figure 5]FIG. 5(a) is a bar graph, FIG. 5(b) is the text information, attribute information, and coordinate information generated by the first generation unit, and FIG. 5(c) is the structured data generated by the first generation unit. [Figure 6] FIG. 6 is an example of the structured data generated by the second generating unit. [Figure 7] FIG. 7 is a flowchart illustrating the information processing method according to the first embodiment. [Figure 8] FIG. 8(a) is an example of a graph, and FIG. 8(b) is an example of a series of multiple graphs. DETAILED DESCRIPTION OF THE INVENTION

[0014] Next, an embodiment of the present invention will be described with reference to the drawings. In the description of the drawings relating to the embodiment, the same or similar parts are designated by the same or similar reference numerals. Of course, there are also parts whose relationships differ between the drawings.

[0015] Furthermore, the embodiments are merely examples of devices and methods for embodying the technical idea of ​​the present invention, and the technical idea of ​​the present invention does not limit the configuration of each component to those described below. The technical idea of ​​the present invention can be modified in various ways within the technical scope defined by the claims.

[0016] (First embodiment) The information processing device according to this embodiment generates structured data from a target document such as a form using a technique that combines OCR and VLM, first generating structured data from the target document using optical character recognition (OCR), then generating structured data from data with attributes other than character strings, such as diagrams, using a vision language model (VLM), and replacing the structured data generated using OCR with corresponding structured data. When generating structured data from data with attributes other than character strings using LLM, the information processing device according to this embodiment generates structured data by excluding images such as logos and illustrations as objects to be structured.

[0017] Commonly known methods for generating structured data from documents such as forms include using OCR, using LLM, and a combination of these. OCR is a method that uses optical techniques to capture printed or handwritten documents as image data and convert the characters in the document into text data.

[0018] When generating structured data for a document using OCR, character strings within the document are detected, and attribute information and coordinate information for the detected character strings are analyzed. However, when generating structured data for figures and charts within a document using OCR, it is not possible to structure the visual meaning of the figures and charts. For this reason, a method combining OCR and LLM is known, in which structured data is generated for character strings within a document using OCR and for non-character string data within the document using LLM. When generating structured data for documents using a method that combines OCR and LLM, if LLM is used to generate structured data for data that has attributes other than character strings, the accuracy of the generated structured data may be reduced, for example, due to images such as logos and illustrations that are not directly related to the content of the document becoming noise. In response to these problems, the information processing device according to this embodiment makes it possible to provide an information processing device, an information processing method, and an information processing program that are capable of generating structured data with high accuracy.

[0019] <Configuration> An example of the configuration and functional units of an information processing device 10 according to this embodiment is shown in Fig. 1. The information processing device 10 shown in Fig. 1 includes a CPU 101 for executing various calculations, a ROM 102 for storing processing programs, a RAM 103 for storing data and the like, a storage unit 104 for storing various data and calculation results and the like, an I / O (input / output interface) 105, a display unit 106, an input unit 107, and the like.

[0020] The I / O 105 is an interface, buffer, etc. for communication (transmission and reception).

[0021] The information processing device 10 according to this embodiment may also be connected to an input keyboard, mouse, and the like.

[0022] The information processing device 10 is any of various electronic computers (computational resources) such as a mobile terminal, a personal computer (PC), a mainframe, a workstation, and a cloud computing system.

[0023] 1 also shows functional units within the CPU 101. When each functional unit of the CPU 101 is realized by software, the CPU 101 realizes it by executing instructions of a program, which is software that realizes each function. In detail, the CPU 101 includes an acquisition unit 108, a first generation unit 109, a first determination unit 110, an image generation unit 111, a calculation unit 112, a second determination unit 113, a second generation unit 114, an integration unit 115, etc. The storage unit 104 includes a structured object determination model storage unit 116.

[0024] The acquisition unit 108 acquires the target document.

[0025] The first generation unit 109 uses optical character recognition (OCR) to generate text information, attribute information, and coordinate information of at least one or more components in the target document, and then generates structured data based on the generated text information, attribute information, and coordinate information.

[0026] The first determination unit 110 determines whether to generate image data of the component elements as an input image based on the text information, attribute information, and coordinate information.

[0027] The image generating unit 111 generates at least one input image of the determined constituent elements that have been determined by the first determining unit to be constituent elements for which image data is to be generated.

[0028] The calculation unit 112 calculates the similarity between the input image and at least one image data acquired in advance by image recognition using the structured object determination model.

[0029] The second determination unit 113 determines, based on the similarity, whether the input image is an input image for which structured data is to be generated.

[0030] The second generation unit 114 generates structured data of the input image using a visual language model.

[0031] The integrating unit 115 integrates the structured data generated by the first generating unit 109 and the structured data generated by the second generating unit 114.

[0032] The structured object determination model storage unit 116 stores the structured object determination model.

[0033] The structured object determination model is constructed using one or more pieces of pre-acquired image data to which tags relating to the pre-acquired image data have been added.

[0034] The operation of each functional unit in CPU 101 will be described in detail below. The target document acquired by acquisition unit 108 is a document containing character strings, images, numbers, graphs, tables, etc., and may be, for example, a report, a calculation sheet, a formula, a data file, etc. FIG. 2 shows an example of a target document acquired by acquisition unit 108. Target document 20 shown in FIG. 2 is composed of a title 21 written as "Title," multiple character strings 22, a table 23, a graph 24, etc. The multiple character strings 22 are composed of character strings "1. Chapter 1," "This chapter is about...," "2. Purpose," "2.1. Table," and "Here is an example of a table...."

[0035] The acquiring unit 108 acquires the target document from an external device (not shown), for example, via the I / O 105. The external device may be, for example, a server or a user terminal. The user terminal is a terminal used by a user of the information processing device 10, and may be, for example, a desktop, a laptop, a tablet, or a smartphone. Furthermore, for example, when the target document is recorded in an external memory (not shown) and the external memory is connected to an interface of the information processing device 10, the acquiring unit 108 may acquire the target document from the external memory. Furthermore, when text information (text file) containing the target document is available and the target document is selected (when an area in which the target document is written is specified) via the input unit 121 and the user terminal, the acquiring unit 108 may acquire the selected target document (the specified area is the target document).

[0036] The first generation unit 109 structures the target document, i.e., generates text information, attribute information, and coordinate information of at least one or more components in the target document, and generates structured data based on the generated at least one or more pieces of text information, attribute information, and coordinate information.

[0037] In this embodiment, at least one or more components in a target document refers to elements described in the target document, such as character strings, images, numbers, graphs, tables, etc. For example, the document shown in Fig. 2 is composed of components such as a title 21, multiple character strings 22, a table 23, and a graph 24.

[0038] The text information generated by the first generation unit 109 is the character string contained in the component. For example, a component does not necessarily contain a character string, such as when the component is a graph that does not contain a character string, or when the component is a logo or illustration. When a component does not contain a character string, the first generation unit 109 generates attribute information and coordinate information. When a predetermined number of components, among at least one or more components in the target document, do not contain a character string, the first generation unit 109 generates text information, attribute information, and coordinate information of the component for the component that contains a character string, and further generates attribute information and coordinate information of the component for the component that does not contain a character string, and then generates structured data based on the generated text information, attribute information, and coordinate information.

[0039] The attribute information generated by the first generation unit 109 is the type used to classify components, and for example, the attribute of title 21 is title, the attribute of string 22 is "chapter title" for the string "Chapter 1", "main text" for the string "This chapter is about XX...", "chapter title" for the string "2. Purpose", "section title" for the string "2.1. Table", "main text" for the string "Here is an example of a table...", and the attribute of table 23 is "table".

[0040] The coordinate information generated by the first generation unit 109 is information that specifies the position of a component within the target document 20, and may include, for example, information such as coordinates indicating the position within the paper surface of the target document 20, the shape of the component, and the vertical and horizontal lengths of the component.

[0041] Fig. 3 shows the text information, attribute information, and coordinate information generated by the first generation unit 109 by structuring the title 21 written in the target document 20 shown in Fig. 2. Fig. 3 shows "character string: "title"" as the text information generated by the first generation unit 109, "attribute: title" as the attribute information, and "coordinate information: 3.45, 1.23..." as the coordinate information.

[0042] 4 shows an example of structured data generated based on the text information, attribute information, and coordinate information of each component in the target document 20 shown in FIG. 2 generated by the first generation unit 109. The content written from the upper left to the lower right on the page of the target document 20 based on the coordinate information of each component generated by the first generation unit 109 is written in the structured data 40 shown in FIG. 4 in order from top to bottom, with character strings written based on the text information of each component in a format based on the attribute information of each component. For example, character strings with the attributes "title," "main text," "chapter title," and "section title" are displayed as corresponding character strings, and character strings with the attribute "table" are displayed in table format.

[0043] 4 is an example of the structured data in which the first generation unit 109 successfully structured the target document. However, because OCR cannot structure visual information such as graphs, illustrations, and logos, the first generation unit 109 cannot generate accurate structured data for components that include visual information.

[0044] FIG. 5 shows an example of structured data generated by the first generating unit 109 for a component including visual information. FIG. 5(a) shows a bar graph, FIG. 5(b) shows text information, attribute information, and coordinate information generated by the first generating unit 109 using the bar graph shown in FIG. 5(a) as a component of a target document, and FIG. 5(c) shows structured data generated by the first generating unit 109 based on the text information, attribute information, and coordinate information. Because the first generating unit 109 cannot structure visual information, it cannot distinguish between the y-axis scale in the graph and the values ​​of the bar graph. As a result, the y-axis scale values ​​in the graph and the values ​​of the bar graph are displayed intermixed in the structured data shown in FIG. 5(c). The visual information of the bar graph shown in FIG. 5(a) is missing from the structured data shown in FIG. 5(c). Therefore, information obtainable from the bar graph shown in FIG. 5(a), such as the values ​​of the bar graph for each month, cannot be obtained from the structured data shown in FIG. 5(c).

[0045] In contrast, the information processing device 10 according to this embodiment generates structured data for components containing visual information using a visual language model (VLM). In this case, if the attribute information of a component containing visual information is a "figure" or a "table," it is possible to generate structured data from the component with high accuracy using an LLM. However, if the attribute information of a component containing visual information is not a "figure" or a "table" and the component is information unnecessary for generating structured data, such as a logo or illustration, it is not possible to generate structured data with high accuracy. Therefore, as described below, the information processing device 10 according to this embodiment generates structured data for components containing visual information using a VLM after removing information unnecessary for generating structured data.

[0046] The first determination unit 110 determines whether or not a component contains visual information based on the text information, attribute information, and coordinate information generated by the first generation unit 109. Here, visual information refers to information that a component has that is not expressed by a character string but is expressed visually. In this embodiment, the first determination unit 110 may determine that a component contains visual information when the attribute information is "figure" or "table." Note that the attribute information of "figure" is assigned to an image.

[0047] For a determined component that has been determined by the first determination unit 110 to be a component containing visual information, the image generation unit 111 generates image data of the determined component as an input image to be input to the calculation unit 112, based on the coordinate information of the determined component. In the example shown in FIG. 5, the image generation unit 111 uses the entire bar graph in FIG. 5(a) as the input image. The image generation unit 111 transmits the generated input image to the calculation unit 112.

[0048] The calculation unit 112 compares the input image received from the image generation unit 111 with at least one or more pieces of image data acquired in advance by image recognition using a structured object determination model, and calculates the similarity between the input image and at least one or more pieces of image data acquired in advance. The one or more pieces of image data acquired in advance are image data that can be components of the target document 20 and can be the subject of structuring, and may be, for example, at least one of a graph, a box diagram, a tree diagram, a table, and a flowchart.

[0049] The structured object detection model is constructed using one or more previously acquired image data tagged with information about the image data. The information about the image data may include the type of image, such as a graph, box diagram, tree diagram, table, or flowchart, or the details and shape of the image, such as the details and shape of the graph if the image data is a graph. A visual language model (VLM) may also be used as the structured object detection model.

[0050] When comparing an input image with at least one or more pieces of image data acquired in advance by image recognition using a structured object determination model, the calculation unit 112 may remove color information from the input image and the at least one or more pieces of image data acquired in advance, and compare the contour feature amount of the input image with the contour feature amount of the at least one or more pieces of image data acquired in advance. This allows the calculation unit 112 to perform a more accurate comparison. In this case, the similarity is calculated from a comparison between each contour of one or more input images and each contour feature amount of one or more pieces of image data acquired in advance.

[0051] The second determination unit 113 determines whether an input image is an input image for generating structured data based on the similarity calculated by the calculation unit 112. When one or more pieces of image data previously acquired and used to construct the structuring target determination model are image data that can be components of the target document 20 and can be subject to structuring, the second determination unit 113 may determine that an input image with a similarity higher than a predetermined threshold is an input image for generating structured data. This makes it possible to remove images that are not subject to structuring, such as logos and illustrations, from the input images for generating structured data.

[0052] The second determination unit 113 may use the highest similarity calculated by the calculation unit 112 between the input image and at least one image data previously acquired as the similarity to be used when determining whether the input image is an input image for generating structured data.

[0053] The second generation unit 114 generates structured data for an input image that the second determination unit 113 determines to be an input image for which structured data is to be generated, using a visual language model. Fig. 6 shows an example of structured data generated by the second generation unit 114 when the bar graph of Fig. 5(a) is used as the input image. The structured data shown in Fig. 6 shows the values ​​of the bar graph on the x-axis. The visual information included in Fig. 5(a), i.e., the y values ​​represented by the bar graph for each x value shown in Fig. 5(a), is shown in Fig. 6.

[0054] The integration unit 115 integrates the structured data generated by the first generation unit 109 and the structured data generated by the second generation unit 114 by replacing, with the structured data generated by the second generation unit 114, the structured data corresponding to the component that is determined by the first determination unit 110 to contain visual information and then determined by the second determination unit 113 to be the input image for generating the structured data.

[0055] In addition, the integration unit 115 may delete from the structured data generated by the first generation unit 109 structured data corresponding to images that are not to be structured, such as logos or illustrations, that the second determination unit 113 has determined are not input images for generating structured data.

[0056] The information processing method according to this embodiment will be described with reference to the flowchart of FIG.

[0057] In step S701, the acquisition unit 108 acquires a target document (acquisition step).

[0058] In step S702, the first generating unit 109 generates text information, attribute information of the component, and coordinate information of the component for at least one component in the target document (first generating step).

[0059] In step S703, the first determination unit 110 determines whether or not to generate image data of the component elements as an input image, based on the text information, attribute information, and coordinate information (first determination step).

[0060] In step S704, the image generating unit 111 generates at least one input image of the determined components that have been determined by the first determining unit to be components for which image data is to be generated (image generating step).

[0061] In step S705, the calculation unit 112 calculates the similarity between the input image and at least one image data acquired in advance by image recognition using a structured object determination model (calculation step).

[0062] In step S706, the second determination unit 113 determines, based on the similarity, whether the input image is an input image for which structured data is to be generated (second determination step).

[0063] In step S707, the second generating unit 114 generates structured data of the input image using the visual language model (second generating step).

[0064] In step S708, the integrating unit 115 integrates the text information generated by the first generating unit 109 and the structured data generated by the second generating unit 114 (integrating step).

[0065] (Second embodiment) In the information processing device 10 according to the first embodiment, the image generation unit 111 generates image data of a determined component that has been determined by the first determination unit 110 to be a component containing visual information, based on the coordinate information of the determined component, as an input image to be input to the calculation unit 112. At this time, there are cases where the range of the image data when generating the image data of the determined component as an input image cannot be set to a range that is sufficiently correct so that the calculation unit 112, the second determination unit 113, and the second generation unit 114 can perform operations with high precision.

[0066] Specifically, as shown in FIG. 8(a), there is a case where a part of graph 81 is set as range 82 of image data. If an image of the region of range 82 shown in FIG. 8(a) is set as an input image, the information of graph 81 may not be accurately reflected when structured data is generated by second generation unit 114. Also, as shown in FIG. 8(b), if there are multiple consecutive graphs in the target document, and only one of the consecutive graphs displays a legend while the other graphs do not, image generation unit 111 may generate input images for each of the multiple graphs that are independent of each other. When second generation unit 114 generates structured data, it may not be possible to accurately generate structured data for graphs without legends.

[0067] In contrast, the information processing device according to this embodiment may further include an input unit. The input unit may accept input of coordinate information of a desired component among one or more components. A user can input the coordinate information of a desired component to the input unit. In the case shown in FIG. 8(a), coordinate information can be input such that the entire graph 81 is within the range 83 of image data. In the case shown in FIG. 8(b), for example, coordinate information can be input such that all consecutive graphs are treated as a single input image.

[0068] After receiving the input of the coordinate information of the desired component, the input unit transmits the received coordinate information of the desired component to the image generation unit 111. The image generation unit 111 generates an input image of the component based on the received coordinate information of the component.

[0069] As mentioned above, the present invention naturally includes various embodiments not described herein. Therefore, the technical scope of the present invention is defined only by the invention-specifying matters according to the scope of the claims that are appropriate from the above description.

[0070] The program of each embodiment of the present disclosure may be provided in a state stored in a storage medium readable by an information processing device. The storage medium may store the program in a "non-transitory tangible medium." The program includes, for example, a software program or a control program. When each functional unit of the information processing device is realized by software, the information processing device functions as an acquisition unit 108, a first generation unit 109, a first determination unit 110, an image generation unit 111, a calculation unit 112, a second determination unit 113, a second generation unit 114, an integration unit 115, and a structured object determination model storage unit 116 by a processor executing a program loaded on a memory.

[0071] The storage medium may, where appropriate, include one or more semiconductor-based or other integrated circuits (ICs) (e.g., field programmable gate arrays (FPGAs), application specific ICs (ASICs), etc.), hard disk drives (HDDs), hybrid hard drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable storage media, or any suitable combination of two or more of these. The storage medium may, where appropriate, be volatile, non-volatile, or a combination of volatile and non-volatile.

[0072] Furthermore, the program of the present disclosure may be provided to an information processing device via any transmission medium (such as a communication network or broadcast waves) capable of transmitting the program.

[0073] Furthermore, each embodiment of the present disclosure may be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission. Note that the program of the present disclosure may be implemented using, for example, a scripting language such as JavaScript (registered trademark) or Python (registered trademark), C language, Go language, Swift (registered trademark), Koltin (registered trademark), Java (registered trademark), or the like.

[0074] According to each aspect of the present disclosure described above, by providing an information processing device capable of generating structured data with high accuracy, it is possible to contribute to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), which is to "build inclusive and sustainable industrial bases, promote inclusive and sustainable industrialization, and build resilient infrastructure." [Explanation of symbols]

[0075] 10. Information processing equipment 101 CPU 102 ROM 103 RAM 104 Storage section 105 I / O (input / output interface) 106 Display section 107 Input section 108 Acquisition Department 109 1st generation part 110 1st Judgment Section 111 Image generation unit 112 Calculation Unit 113 Second Judgment Section 114 Second generation part 115 Integration Department 116 Structured object determination model memory unit 20 Target Documents 21 titles 22 Multiple Strings 23 table 81 graphs 82, 83 range

Claims

1. an acquisition unit that acquires a target document; a first generation unit that generates text information, attribute information, and coordinate information for each of one or more components in the target document, and then generates structured data based on the generated text information, attribute information, and coordinate information; a first determination unit that determines, for each of the one or more components, whether the component includes visual information based on the text information, the attribute information, and the coordinate information; an image generation unit that cuts out one or more components determined by the first determination unit to be components containing visual information based on the coordinate information and generates one or more input images; a second determination unit that determines, using a visual language model, whether each of the one or more input images is an input image for generating structured data; a second generation unit that generates structured data of the input image using a visual language model; an integration unit that integrates the structured data generated by the first generation unit and the structured data generated by the second generation unit; An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, wherein the one or more pieces of image data acquired in advance include at least one of a graph, a box diagram, a tree diagram, a table, and a flowchart.

3. The information processing device according to claim 1 , wherein the first determination unit determines that the one or more components whose attribute information is any of "figure," "table," or "image" are components to be cut out.

4. 2. The information processing apparatus according to claim 1, further comprising an input unit, wherein the input unit receives input of coordinate information of a desired component among the one or more components.

5. In the information processing device, an acquisition step of acquiring a target document; a first generation step of generating text information, attribute information, and coordinate information for one or more components in the target document, and then generating structured data based on the generated text information, attribute information, and coordinate information; a first determination step of determining, for each of the one or more components, whether the component includes visual information based on the text information, the attribute information, and the coordinate information; an image generation step of generating one or more input images based on the coordinate information for one or more components determined to be components including visual information in the first determination step; a second determination step of determining, using a visual language model, whether each of the one or more input images is an input image for which structured data is to be generated; a second generation step of generating structured data of the input image using a visual language model; an integration step of integrating the structured data generated in the first generation step and the structured data generated in the second generation step; An information processing method comprising:

6. On the computer, an acquisition function for acquiring the target document; a first generation function that generates text information, attribute information, and coordinate information for one or more components in the target document, and then generates structured data based on the generated text information, attribute information, and coordinate information; a first determination function that determines, for each of the one or more components, whether the component includes visual information based on the text information, the attribute information, and the coordinate information; an image generation function that generates one or more input images based on the coordinate information from one or more components determined to be components including visual information in the first determination function; a second determination function that determines, using a visual language model, whether each of the one or more input images is an input image for generating structured data; a second generation function for generating structured data of the input image using a visual language model; an integration function that integrates the structured data generated by the first generation function and the structured data generated by the second generation function; An information processing program characterized by realizing the above.

Images