Information processing device, information processing method, and information processing program

Abstract

This invention provides an information processing device, an information processing method, and an information processing program capable of generating highly accurate structured data. [Solution] The system comprises an acquisition unit that acquires a target document, a string recognition unit that generates text information, attribute information, and coordinate information of components within the target document, a first determination unit that determines whether a component includes visual information, an image generation unit that crops the component to use as an input image, a calculation unit that calculates the similarity between the input image and pre-acquired image data by image recognition using a structured target determination model, a second determination unit that determines whether it is an input image based on the similarity, an output unit that generates structured data of the input image using a visual language model, and an integration unit that integrates the text information generated by the string recognition unit and the structured data generated by the output unit. The structured target determination model is constructed using one or more pre-acquired image data to which tags relating to image information have been assigned.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Conventionally, a technique for generating structured data from a target document has been disclosed (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Means for Solving the Problems

[0004] An information processing apparatus according to an embodiment of the present invention includes an acquisition unit that acquires a target document, a character string recognition unit that generates text information, attribute information, and coordinate information for each of one or more components in the target document, a first determination unit that determines whether each of the one or more components includes visual information based on the text information, attribute information, and coordinate information, an image generation unit that uses the one or more components determined to be components to be cut out by the first determination unit as one or more input images based on the coordinate information, a calculation unit that calculates the similarity between each of the one or more input images and one or more image data acquired in advance by image recognition using a structured target determination model, a second determination unit that determines whether each of the one or more input images is an input image for generating structured data based on the similarity, an output unit that generates structured data of the input image by a vision language model, an integration unit that integrates the text information generated by the character string recognition unit and the structured data generated by the output unit, and a storage unit that stores the structured target determination model, and the structured target determination model is constructed using one or more image data acquired in advance to which tags related to image information are assigned.

[0005] In an information processing device according to one embodiment of the present invention, the similarity may be calculated by comparing the contours of one or more input images with the feature quantities of the contours of one or more previously acquired image data.

[0006] In an information processing device according to one embodiment of the present invention, a visual language model may be used as a structured target determination model.

[0007] In an information processing device according to one embodiment of the present invention, one or more previously acquired image data may include at least one of a graph, a box diagram, a tree diagram, a table, or a flowchart.

[0008] In an information processing device according to one embodiment of the present invention, the first determination unit may determine that one or more components whose attribute information is either "figure" or "table" are components to be cut out.

[0009] An information processing device according to one embodiment of the present invention further comprises an input unit, the input unit may accept input of coordinate information of a desired component from among one or more components.

[0010] An information processing method according to one embodiment of the present invention is an information processing device equipped with a storage unit that stores a structured target determination model constructed using one or more previously acquired image data to which tags relating to image information have been assigned, comprising: an acquisition step of acquiring a target document; a string recognition step of generating text information, attribute information, and coordinate information for one or more components in the target document; a first determination step of determining whether or not each of the one or more components contains visual information based on the text information, attribute information, and coordinate information; an image generation step of setting one or more components determined to be components to be cut out in the first determination step as one or more input images based on the coordinate information; a calculation step of calculating the similarity between each of the one or more input images and one or more previously acquired image data by image recognition using the structured target determination model; a second determination step of determining whether each of the one or more input images is an input image for generating structured data based on the similarity; an output step of generating structured data of the input images using a visual language model; and an integration step of integrating the text information generated in the string recognition step and the structured data generated by the output unit.

[0011] An information processing program according to one embodiment of the present invention provides a computer equipped with a storage unit that stores a structured target determination model constructed using one or more previously acquired image data to which tags relating to image information have been assigned, and enables the following functions: acquisition function for acquiring a target document; string recognition function for generating text information, attribute information, and coordinate information for one or more components within the target document; first determination function for determining whether each of the one or more components contains visual information based on the text information, attribute information, and coordinate information; image generation function for setting one or more components determined to be components to be cut out in the first determination function as one or more input images based on the coordinate information; calculation function for calculating the similarity between each of the one or more input images and one or more previously acquired image data by image recognition using the structured target determination model; second determination function for determining whether each of the one or more input images is an input image for generating structured data based on the similarity; output function for generating structured data of the input image using a visual language model; and integration function for integrating the text information generated in the string recognition function with the structured data generated by the output unit. [Effects of the Invention]

[0012] According to the present invention, it is possible to provide an information processing device, an information processing method, and an information processing program that can generate structured data with high accuracy. [Brief explanation of the drawing]

[0013] [Figure 1] Figure 1 is a block diagram showing an example of the configuration and functional parts of an information processing apparatus according to the first embodiment of the present invention. [Figure 2] Figure 2 shows an example of a target document. [Figure 3] Figure 3 shows an example of text information, attribute information, and coordinate information generated by the first generation unit. [Figure 4] Figure 4 shows an example of structured data generated by the first generation unit. [Figure 5]Figure 5(a) is a bar graph, Figure 5(b) is text information, attribute information, and coordinate information generated by the first generation unit, and Figure 5(c) is structured data generated by the first generation unit. [Figure 6] Figure 6 shows an example of structured data generated by the second generation unit. [Figure 7] Figure 7 is a flowchart illustrating the information processing method according to the first embodiment. [Figure 8] Figure 8(a) is an example of a graph, and Figure 8(b) is an example of multiple consecutive graphs. [Modes for carrying out the invention]

[0014] Next, embodiments of the present invention will be described with reference to the drawings. In the drawings of the embodiments, identical or similar parts are denoted by the same or similar reference numerals. It goes without saying that there are parts in the drawings that have different relationships to each other.

[0015] Furthermore, the embodiments are illustrative of apparatus and methods for realizing the technical concept of the present invention, and the technical concept of the present invention does not limit the configuration of each component to those described below. The technical concept of the present invention can be modified in various ways within the technical scope defined by the claims described in the patent claims.

[0016] (First embodiment) The information processing apparatus according to this embodiment generates structured data for a target document using optical character recognition (OCR), and then uses a vision language model (VLM) to generate structured data for data having an attribute that is not a character string, such as a chart, etc., identified by OCR, and replaces the structured data corresponding to the data having an attribute that is not a character string in the structured data generated using OCR. The information processing apparatus according to this embodiment generates structured data from a target document such as a form by a method combining OCR and VLM. When generating structured data for data having an attribute that is not a character string using an LLM, the information processing apparatus according to this embodiment excludes images such as logos and illustrations from the structuring target and generates structured data.

[0017] As methods for generating structured data from documents such as forms, methods using OCR, methods using LLM, and methods combining these are generally known. OCR is a method of capturing a document with printed or handwritten characters as image data by an optical method and converting the characters in the document into text data.

[0018] In the generation of structured data using OCR for a document, the character strings in the document are detected, and the analysis of the attribute information of the detected character strings and the analysis of the coordinate information of the detected character strings are performed. However, in the generation of structured data using OCR for charts in a document, the visual meaning of the chart cannot be structured. Thus, a method combining OCR and LLM is known, in which structured data is generated using OCR for the character strings in the document and structured data is generated using LLM for data that is not a character string in the document. In the generation of structured data using a method combining OCR and LLM for a document, when generating structured data using LLM for data having an attribute that is not a character string, the accuracy of structured data generation may decrease, such as images such as logos and illustrations that are not directly related to the content of the document becoming noise. In response to these problems, the information processing apparatus according to the present embodiment enables the provision of an information processing apparatus, an information processing method, and an information processing program capable of generating highly accurate structured data.

[0019] <Configuration> FIG. 1 shows an example of the configuration and functional units of the information processing apparatus 10 according to the present embodiment. The information processing apparatus 10 shown in FIG. 1 includes a CPU 101 for executing various operations, a ROM 102 for storing processing programs, a RAM 103 for storing data, etc., a storage unit 104 for storing various data and operation results, etc., an I / O (input / output interface) 105, a display unit 106, an input unit 107, and the like.

[0020] The I / O 105 is an interface for communication (transmission / reception), a buffer, etc.

[0021] In addition, a keyboard, a mouse, etc. for input may be connected to the information processing apparatus 10 according to the present embodiment.

[0022] The information processing apparatus 10 is various electronic computers (computing resources) such as a mobile terminal, a personal computer (PC), a mainframe, a workstation, a cloud computing system, etc.

[0023] Furthermore, the block diagram of FIG. 1 shows the functional units in the CPU 101. When each functional unit of the CPU 101 is realized by software, the CPU 101 is realized by executing the instructions of a program which is software for realizing each function. Specifically, it includes an acquisition unit 108, a first generation unit 109, a first determination unit 110, an image generation unit 111, a calculation unit 112, a second determination unit 113, a second generation unit 114, an integration unit 115, etc. The storage unit 104 includes a structured target determination model storage unit 116.

[0024] The acquisition unit 108 acquires a target document.

[0025] The first generation unit 109 uses optical character recognition (OCR) to generate text information, attribute information, and coordinate information for at least one component in the target document, and then generates structured data based on the generated text information, attribute information, and coordinate information.

[0026] The first determination unit 110 determines whether to generate image data of the constituent elements as input images based on text information, attribute information, and coordinate information.

[0027] The image generation unit 111 generates at least one input image of the determined components that the first determination unit has determined to be components for generating image data.

[0028] The calculation unit 112 calculates the similarity between the input image and at least one previously acquired image data by image recognition using a structured target determination model.

[0029] The second determination unit 113 determines, based on the similarity, whether the input image is an input image for which structured data is to be generated.

[0030] The second generation unit 114 generates structured data of the input image using a visual language model.

[0031] The integration unit 115 integrates the structured data generated by the first generation unit 109 and the structured data generated by the second generation unit 114.

[0032] The structured target determination model storage unit 116 stores the structured target determination model.

[0033] The structured object determination model is constructed using one or more previously acquired image data that have been tagged with pre-acquired image data.

[0034] The operation of each functional unit within the CPU 101 is described in detail below. The target document acquired by the acquisition unit 108 is a document containing strings, images, numbers, graphs, tables, etc., and may be, for example, a report, a calculation sheet, a mathematical formula, a data file, etc. Figure 2 shows an example of a target document acquired by the acquisition unit 108. The target document 20 shown in Figure 2 consists of a title 21 labeled "Title", multiple strings 22, a table 23, a graph 24, etc. The multiple strings 22 consist of strings such as "1. Chapter 1", "This chapter is about...", "2. Purpose", "2.1. Table", and "An example of a table is shown...".

[0035] The acquisition unit 108 acquires the target document from an external device (not shown) via, for example, I / O 105. The external device may be, for example, a server and a user terminal. The user terminal is a terminal used by a user of the information processing device 10, and may be, for example, a desktop, laptop, tablet, and smartphone. The acquisition unit 108 may also acquire the target document from an external memory (not shown) if the target document is recorded in that external memory and that external memory is connected to the interface of the information processing device 10. Furthermore, if there is document information (document file) containing the target document, and the target document is selected via the input unit 121 and the user terminal, etc. (if an area in which the target document is written is specified), the acquisition unit 108 may acquire the selected target document (using the specified area as the target document).

[0036] The first generation unit 109 structures the target document, that is, it generates text information, attribute information, and coordinate information for at least one component within the target document, and generates structured data based on the generated text information, attribute information, and coordinate information.

[0037] In this embodiment, "at least one component within the target document" refers to elements contained within the target document, such as strings of text, images, numbers, graphs, tables, etc. For example, the document shown in Figure 2 consists of components such as a title 21, multiple strings of text 22, a table 23, and a graph 24.

[0038] The text information generated by the first generation unit 109 is the string contained in the constituent elements. For example, constituent elements do not necessarily contain strings, such as graphs that do not contain strings, or logos and illustrations. If a constituent element does not contain a string, the first generation unit 109 generates attribute information and coordinate information. If a predetermined number of constituent elements among at least one or more elements in the target document do not contain strings, the first generation unit 109 generates text information, attribute information, and coordinate information for constituent elements that contain strings, and then generates attribute information and coordinate information for constituent elements that do not contain strings, and then generates structured data based on the generated text information, attribute information, and coordinate information.

[0039] The attribute information generated by the first generation unit 109 is the type used to classify the constituent elements. For example, the attribute of title 21 is title, the attributes of string 22 are "chapter title" for the string "Chapter 1", "text" for the string "This chapter is about...", "chapter title" for the string "2.Purpose", "section title" for the string "2.1.Table", "text" for the string "Examples of tables are shown...", and the attribute of table 23 is "table".

[0040] The coordinate information generated by the first generation unit 109 is information that specifies the position of a component within the target document 20, and may include, for example, coordinates indicating the position of the target document 20 on the page, the shape of the component, and information such as its vertical and horizontal lengths.

[0041] Figure 3 shows the text information, attribute information, and coordinate information generated by the first generation unit 109 by structuring the title 21 described in the target document 20 shown in Figure 2. In Figure 3, the text information generated by the first generation unit 109 is shown as "String: "Title"", the attribute information as "Attribute: Title", and the coordinate information as "Coordinate Information: 3.45, 1.23...".

[0042] Figure 4 shows an example of structured data generated by the first generation unit 109 after it has generated text information, attribute information, and coordinate information for each component in the target document 20 shown in Figure 2. Based on the coordinate information of each component generated by the first generation unit 109, the content written in the target document 20 from the top left to the bottom right of the page is written in order from top to bottom in the structured data 40 shown in Figure 4, with strings written based on the text information of each component in a format based on the attribute information of each component. For example, strings whose attributes are "title," "body text," "chapter title," and "section title" are displayed as corresponding strings, and strings whose attribute is "table" are displayed in table format.

[0043] The structured data shown in Figure 4 is an example in which the first generation unit 109 was able to successfully structure the target document. However, since OCR cannot structure visual information such as graphs, illustrations, and logos, the first generation unit 109 cannot generate accurate structured data of components that include visual information.

[0044] Figure 5 shows an example of how the first generation unit 109 generates structured data for components that include visual information. Figure 5(a) is a bar graph, Figure 5(b) is text information, attribute information, and coordinate information generated by the first generation unit 109 using the bar graph shown in Figure 5(a) as a component of the target document, and Figure 5(c) is structured data generated by the first generation unit 109 based on the text information, attribute information, and coordinate information. Because the first generation unit 109 cannot structure visual information, it cannot distinguish between the y-axis scale in the graph and the bar graph values, and in the structured data shown in Figure 5(c), the values ​​of the y-axis scale in the graph and the bar graph values ​​are displayed together. The visual information contained in the bar graph shown in Figure 5(a) is missing in the structured data shown in Figure 5(c), and information that can be obtained from the bar graph shown in Figure 5(a), such as the values ​​of the bar graph for each month, cannot be obtained from the structured data shown in Figure 5(c).

[0045] In contrast, the information processing device 10 according to this embodiment generates structured data using a Vision Language Model (VLM) for components containing visual information. In this case, if the attribute information of the component containing visual information is "figure" or "table", it is possible to generate structured data accurately from the component using the LLM. However, if the attribute information of the component containing visual information is not "figure" or "table", and the component contains information unnecessary for generating structured data, such as a logo or illustration, it is not possible to generate structured data accurately. Therefore, as described below, the information processing device 10 according to this embodiment removes information unnecessary for generating structured data from the component containing visual information, and then generates structured data using the VLM.

[0046] The first determination unit 110 determines whether a component contains visual information based on the text information, attribute information, and coordinate information generated by the first generation unit 109. Here, visual information refers to information that the component possesses but is not expressed as a string of characters, but is expressed visually. In this embodiment, the first determination unit 110 may determine that a component contains visual information when the attribute information is "figure" or "table". For images, the attribute information "figure" is assigned.

[0047] The image generation unit 111 generates image data of the determined components, which have been determined by the first determination unit 110 to contain visual information, based on the coordinate information of the determined components, as input images to be input to the calculation unit 112. In the example shown in Figure 5, the image generation unit 111 uses the entire bar graph in Figure 5(a) as the input image. The image generation unit 111 transmits the generated input image to the calculation unit 112.

[0048] The calculation unit 112 compares the input image received from the image generation unit 111 with at least one previously acquired image data by image recognition using a structured target determination model, and calculates the similarity between the input image and at least one previously acquired image data. The one or more previously acquired image data are image data that can be components of the target document 20 and can be subject to structuring, and may be, for example, at least one of a graph, box diagram, tree diagram, table, or flowchart.

[0049] The structured object determination model is constructed using one or more pre-acquired image data, each tagged with information about the image data. This information may include the image type (e.g., graph, box diagram, tree diagram, table, or flowchart), and details and shape of the image (e.g., if the image data is a graph, details and shape of the graph). A Visual Language Model (VLM) may also be used as the structured object determination model.

[0050] When the calculation unit 112 compares the input image with at least one or more previously acquired image data using image recognition with a structured target determination model, it may remove the color information from both the input image and the at least one or more previously acquired image data, and compare the contour features of the input image with the contour features of the at least one or more previously acquired image data. This allows the calculation unit 112 to perform a more accurate comparison. In this case, the similarity is calculated from a comparison between the contour features of one or more input images and the contour features of one or more previously acquired image data.

[0051] The second determination unit 113 determines whether an input image is an input image for generating structured data, based on the similarity calculated by the calculation unit 112. If one or more pre-acquired image data used when constructing the structure target determination model are images that can be components of the target document 20 and can be subject to structuring, the second determination unit 113 may determine that an input image with a similarity higher than a predetermined threshold is an input image for generating structured data. This makes it possible to exclude images that are not subject to structuring, such as logos and illustrations, from being used as input images for generating structured data.

[0052] The second determination unit 113 may use the highest similarity among the similarities between the input image and at least one previously acquired image data, calculated by the calculation unit 112, as the similarity to determine whether the input image is an input image for generating structured data.

[0053] The second generation unit 114 generates structured data for an input image that the second determination unit 113 has determined to be an input image for which structured data should be generated, based on a visual language model. Figure 6 shows an example of structured data generated by the second generation unit 114 when the bar graph in Figure 5(a) is used as the input image. The structured data shown in Figure 6 shows the values ​​of the bar graph for the x axis. The visual information contained in Figure 5(a), that is, the y values ​​represented by the bar graph for each x value shown in Figure 5(a), are shown in Figure 6.

[0054] The integration unit 115 integrates the structured data generated by the first generation unit 109 with the structured data generated by the second generation unit 114 by replacing the structured data corresponding to the components that the first determination unit 110 has determined to contain visual information, and which the second determination unit 113 has determined to be input images for which structured data should be generated, with the structured data generated by the second generation unit 114.

[0055] Furthermore, the integration unit 115 may delete structured data corresponding to images that are not subject to structuring, such as logos and illustrations, which the second determination unit 113 has determined are not input images for which structured data is to be generated, from the structured data generated by the first generation unit 109.

[0056] The information processing method according to this embodiment will be explained with reference to the flowchart in Figure 7.

[0057] In step S701, the acquisition unit 108 acquires the target document (acquisition step).

[0058] In step S702, the first generation unit 109 generates text information, attribute information of the component, and coordinate information of the component for at least one component in the target document (first generation step).

[0059] In step S703, the first determination unit 110 determines whether to generate image data of the constituent elements as an input image based on text information, attribute information, and coordinate information (first determination step).

[0060] In step S704, the image generation unit 111 generates at least one input image of the determined components that the first determination unit has determined to be components for generating image data (image generation step).

[0061] In step S705, the calculation unit 112 calculates the similarity between the input image and at least one previously acquired image data by image recognition using a structured target determination model (calculation step).

[0062] In step S706, the second determination unit 113 determines, based on the similarity, whether the input image is an input image for which structured data is to be generated (second determination step).

[0063] In step S707, the second generation unit 114 generates structured data of the input image using a visual language model (second generation step).

[0064] In step S708, the integration unit 115 integrates the text information generated by the first generation unit 109 with the structured data generated by the second generation unit 114 (integration step).

[0065] (Second embodiment) In the information processing device 10 according to the first embodiment, the image generation unit 111 generates image data of a determined component, which has been determined by the first determination unit 110 to contain visual information, as an input image to be input to the calculation unit 112, based on the coordinate information of the determined component. However, in this case, the range of the image data when generating the image data of the determined component as an input image may not be within a range that is sufficiently correct for the calculation unit 112, the second determination unit 113, and the second generation unit 114 to perform accurate operations.

[0066] Specifically, as shown in Figure 8(a), there are cases where a portion of graph 81 is used as the image data range 82. If an image of the range 82 shown in Figure 8(a) is used as the input image, the second generation unit 114 may not be able to accurately reflect the information of graph 81 when generating structured data. Also, as shown in Figure 8(b), if there are multiple consecutive graphs in the target document, and a legend is displayed for only one of the consecutive graphs while the others do not, the image generation unit 111 may generate independent input images for each of the multiple graphs. When the second generation unit 114 generates structured data, it may not be able to accurately generate structured data for graphs without legends.

[0067] In contrast, the information processing device according to this embodiment may further include an input unit. The input unit may accept input of coordinate information of a desired component from among one or more components. The user can input coordinate information of a desired component to the input unit. In the case shown in Figure 8(a), coordinate information can be input such that the entire graph 81 is set as the range 83 of the image data. In the case shown in Figure 8(b), for example, coordinate information can be input such that all consecutive graphs are set as a single input image.

[0068] The input unit receives the coordinate information of the desired component and then transmits the received coordinate information to the image generation unit 111. The image generation unit 111 generates an input image of the component based on the received coordinate information of the component.

[0069] As stated above, the present invention naturally includes various embodiments and the like that are not described herein. Therefore, the technical scope of the present invention is determined solely by the inventive features relating to the claims that are reasonable based on the above description.

[0070] The programs of each embodiment of this disclosure may be provided stored in a storage medium readable by the information processing device. The storage medium is a “non-temporary tangible medium” capable of storing a program. The program includes, for example, a software program and a control program. When each functional unit of the information processing device is implemented by software, the information processing device functions as an acquisition unit 108, a first generation unit 109, a first determination unit 110, an image generation unit 111, a calculation unit 112, a second determination unit 113, a second generation unit 114, an integration unit 115, and a structured target determination model storage unit 116, by the processor executing a program loaded into memory.

[0071] The storage medium may, where appropriate, include one or more semiconductor-based or other integrated circuits (ICs) (e.g., field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), hard disk drives (HDDs), hybrid hard drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tape, solid-state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable storage medium, or two or more suitable combinations thereof. The storage medium may, where appropriate, be volatile, non-volatile, or a combination of volatile and non-volatile.

[0072] Furthermore, the program of this disclosure may be provided to an information processing device via any transmission medium capable of transmitting the program (such as a communication network or broadcast waves).

[0073] Furthermore, each embodiment of this disclosure can also be realized in the form of a data signal embedded in a carrier wave, where the program is embodied by electronic transmission. The programs of this disclosure may be implemented using, for example, scripting languages ​​such as JavaScript® and Python®, or languages ​​such as C, Go, Swift®, Koltin®, and Java®.

[0074] According to each aspect of this disclosure described above, by providing an information processing device capable of generating highly accurate structured data, it is possible to contribute to achieving Sustainable Development Goal (SDG) 9, "Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation." [Explanation of Symbols]

[0075] 10 Information Processing Devices 101 CPU 102 ROM 103 RAM 104 Storage section 105 I / O (Input / Output Interface) 106 Display section 107 Input section 108 Acquisition Department 109 1st generation part 110 1st Judgment Section 111 Image generation unit 112 Calculation Section 113 Second Judgment Section 114 Second generation part 115 Integration Department 116 Structured Target Determination Model Storage Unit 20 Target Documents 21 Titles 22 Multiple strings 23 table 81 Graph Range 82, 83

Claims

1. The acquisition unit acquires the target document, A first generation unit generates text information, attribute information, and coordinate information for each of one or more components within the target document, and then generates structured data based on the generated text information, attribute information, and coordinate information. A first determination unit determines, based on the text information, attribute information, and coordinate information, whether each of the one or more components includes visual information. An image generation unit that crops one or more components determined by the first determination unit to be components containing visual information based on the coordinate information to form one or more input images, A calculation unit that calculates the similarity between each of the one or more input images and one or more previously acquired image data by image recognition using a structured target determination model, A second determination unit determines, based on the similarity, whether each of the one or more input images is an input image for which structured data is to be generated. A second generation unit generates structured data of the input image using a visual language model, An integration unit that integrates the structured data generated by the first generation unit and the structured data generated by the second generation unit, A storage unit that stores the structured target determination model and An information processing device comprising the above, wherein the structured target determination model is constructed using one or more previously acquired image data to which tags relating to image information have been assigned.

2. The information processing apparatus according to claim 1, characterized in that the similarity is calculated by comparing the contours of each of the one or more input images with the feature quantities of the contours of each of the one or more previously acquired image data.

3. The information processing apparatus according to claim 1, characterized in that the visual language model is used as the structured target determination model.

4. The information processing apparatus according to claim 1, characterized in that the one or more image data acquired in advance include at least one of a graph, a box diagram, a tree diagram, a table, or a flowchart.

5. The information processing apparatus according to claim 1, characterized in that the first determination unit determines that the one or more components whose attribute information is one of "figure," "table," or "image" are components to be cut out.

6. The information processing apparatus according to claim 1, further comprising an input unit, wherein the input unit receives coordinate information of a desired component from among the one or more components.

7. An information processing device equipped with a storage unit that stores a structured target determination model constructed using one or more previously acquired image data to which tags relating to image information have been assigned, The acquisition step to obtain the target document, A first generation step involves generating text information, attribute information, and coordinate information for one or more components within the target document, and then generating structured data based on the generated text information, attribute information, and coordinate information. A first determination step, based on the text information, attribute information, and coordinate information, determines whether each of the one or more components includes visual information. An image generation step in which one or more components determined to be components including visual information in the first determination step are used as one or more input images based on the coordinate information, A calculation step in which the similarity between each of the one or more input images and one or more previously acquired image data is calculated by image recognition using the structured target determination model, A second determination step is to determine, based on the similarity, whether each of the one or more input images is an input image for which structured data is to be generated. A second generation step involves generating structured data of the input image using a visual language model, An integration step that integrates the structured data generated in the first generation step and the structured data generated in the second generation step. An information processing method characterized by comprising:

8. A computer equipped with a storage unit that stores a structured target determination model constructed using one or more previously acquired image data to which image information tags have been assigned, The acquisition function retrieves the target document, A first generation function generates text information, attribute information, and coordinate information for one or more components within the target document, and then generates structured data based on the generated text information, attribute information, and coordinate information. A first determination function that determines whether each of the one or more components includes visual information, based on the text information, attribute information, and coordinate information, An image generation function that uses one or more components determined to be components containing visual information in the first determination function as one or more input images based on the coordinate information, A calculation function that uses the structured target determination model to perform image recognition and calculates the similarity between each of the one or more input images and one or more previously acquired image data, A second determination function determines, based on the similarity, whether each of the one or more input images is an input image for which structured data is to be generated. A second generation function generates structured data of the input image using a visual language model, An integration function that integrates the structured data generated by the first generation function and the structured data generated by the second generation function. An information processing program characterized by achieving this.

Images