Information processing device, information processing method, and program

Abstract

This invention provides an information processing device capable of converting documents containing visual elements into text. [Solution] In the information processing device, the partial region acquisition means acquires multiple partial regions from the content. The selection means selects a set of regions from the multiple partial regions. The conversion means converts each of the partial regions included in the set of regions into arbitrary content.

Description

Technical Field

[0001] The present disclosure relates to a technique for textifying a document including visual elements.

Background Art

[0002] Techniques for converting a document image into text are known. For example, Patent Document 1 discloses a character recognition device that recognizes characters included in an image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the method of Patent Document 1 does not consider the textification of the table structure and the figure structure included in the document image.

[0005] One object of the present disclosure is to provide an information processing apparatus capable of textifying a document including visual elements.

Means for Solving the Problems

[0006] In one aspect of the present disclosure, the information processing apparatus Document images containing visual elements includes sub-region acquisition means for acquiring a plurality of sub-regions from the content, selection means for selecting a set of regions from the plurality of sub-regions, and conversion means for converting each sub-region included in the set of regions text into content, and is provided with 、 The selection means selects one candidate from each of the plurality of sub-regions, and if adding the candidate to the region set improves a predetermined index, it adds the candidate to the region set. The aforementioned predetermined indicators are indicators that represent the quality of the selection, and include coverage rate, overlap rate, size of sub-regions, or a combination thereof.

[0007] In another aspect of the present disclosure, an information processing method executed by a computer is Document images containing visual elements To obtain a plurality of sub-regions from the content Perform the acquisition process , To select a set of regions from the plurality of sub-regions Perform the selection process, For each of the sub-regions included in the set of regions text Convert to content Perform the conversion process. , The selection process selects one candidate from each of the multiple sub-regions, and if adding the candidate to the region set improves a predetermined index, the candidate is added to the region set. The aforementioned predetermined indicator is an indicator representing the quality of the selection state, and includes coverage rate, overlap rate, size of sub-regions, or a combination thereof. Information processing method.

[0008] In yet another aspect of the present disclosure, a program is Document images containing visual elements To obtain a plurality of sub-regions from the content Perform the acquisition process , To select a set of regions from the plurality of sub-regions Perform the selection process, For each of the sub-regions included in the set of regions text Convert to content Perform the conversion process. , The selection process selects one candidate from each of the multiple sub-regions, and if adding the candidate to the region set improves a predetermined index, the candidate is added to the region set. The aforementioned predetermined indicator is an indicator representing the quality of the selection state, and includes coverage rate, overlap rate, size of sub-regions, or a combination thereof. Cause the computer to execute the process.

Effect of the Invention

[0009] According to the present disclosure, it becomes possible to textify a document including visual elements. <http: / / www.example.com>

Brief Description of the Drawings

[0010] [Figure 1] An information processing apparatus according to the present disclosure is shown. [Figure 2] It is a block diagram showing the hardware configuration of the information processing apparatus according to the present disclosure. [Figure 3] It is a block diagram showing the functional configuration of the information processing apparatus according to the present disclosure. [Figure 4] An example of region division by a region division unit is shown. [Figure 5]This is a diagram for explaining the outline of the processing by the rectangular selection part. [Figure 6] This is a diagram for explaining the processing by the rectangular selection part. [Figure 7] An example of re - region - division by the region - division part is shown. [Figure 8] This is a flowchart of the processing by the information processing apparatus according to the present disclosure. [Figure 9] This is a block diagram showing the functional configuration of another information processing apparatus according to the present disclosure. [Figure 10] This is a flowchart of the processing by another information processing apparatus according to the present disclosure.

Mode for Carrying Out the Invention

[0011] Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings.

[0012] <First Embodiment> [Overview Explanation] A mechanism called RAG (Retrieval Augmented Generation) that utilizes external knowledge to supplement the internal knowledge of a large - language model (LLM: Large Language Model) is known. The external knowledge of RAG is mainly created based on texts such as sentences and articles.

[0013] In RAG, it is desirable that external knowledge be utilized not only in the form of text and documents, but also in documents that include visual elements such as diagrams and tables. One method for converting document images containing visual elements into text is to use a Vision Language Model (VLM). VLM can describe the structure of diagrams and tables in a text-based manner. However, VLM may not be able to adequately recognize complex or detailed documents, and conversion to text may be incomplete. Thus, simply inputting document images containing visual elements into a VLM may not be sufficient to convert them to text with sufficient accuracy. Therefore, in this embodiment, first, the document image containing visual elements is divided into regions, and then text conversion using VLM is performed on each divided region. This reduces the amount of information that VLM handles at once, thereby improving the comprehensiveness and accuracy of the text conversion.

[0014] [Schematic configuration] Figure 1 shows the information processing device related to this disclosure. Document image data, including figures and tables, is input to the information processing device 10. The information processing device 10 converts the document image data into text and outputs it to a text database (DB). The text DB is used, for example, as external knowledge for RAG.

[0015] Here, VLM is a model capable of integrating and processing visual information (images and videos) and text (language data). VLM receives visual information and text input and outputs a response to the input in the form of an image, video, or text. Examples of VLMs include BLIP-2 (Bootstrapping Language-Image Pre-training 2), Flamingo, LLaVA (Large Language and Vision Assistant), and Qwen-VL.

[0016] [Hardware configuration] Figure 2 is a block diagram showing the hardware configuration of the information processing device 10 according to the first embodiment. As shown in the figure, the information processing device 10 includes an interface (I / F) 11, a processor 12, a memory 13, a recording medium 14, and a database (DB) 15.

[0017] I / F11 communicates with terminal device 5 and external LLMs via a network such as the Internet.

[0018] Processor 12 is a computer such as a CPU (Central Processing Unit) and controls the entire information processing device 10 by executing a pre-prepared program. Processor 12 may be a GPU (Graphics Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof. Processor 12 performs the text conversion process described later.

[0019] Memory 13 consists of ROM (Read Only Memory), RAM (Random Access Memory), and other components. Memory 13 is also used as working memory while the processor 12 is executing various processes.

[0020] The recording medium 14 is a non-volatile, non-temporary recording medium such as a disk-shaped recording medium or semiconductor memory, and is configured to be detachable from the information processing device 10. The recording medium 14 stores various programs that the processor 12 executes. When the information processing device 10 executes various processes, the programs stored on the recording medium 14 are loaded into the memory 13 and executed by the processor 12. The DB 15 stores, for example, prompts and the results of text conversion processing.

[0021] In addition to the above, the information processing device 10 may also be equipped with a display device such as a liquid crystal display, and an input device such as a keyboard or mouse. These display devices and input devices are used, for example, by the administrator of the information processing device 10 to perform necessary management.

[0022] [Functional Configuration] Figure 3 is a block diagram showing the functional configuration of the information processing device 10 of the first embodiment. Functionally, the information processing device 10 comprises a region division unit 101, a rectangular selection unit 102, and a text conversion unit 103.

[0023] The region division unit 101, the rectangular selection unit 102, and the text conversion unit 103 are all composed of the processor 12 shown in Figure 2.

[0024] The region segmentation unit 101 acquires document image data, including figures and tables, through the I / F 11. The region segmentation unit 101 performs region segmentation on the document image data. Here, the region segmentation unit 101 performs region segmentation by focusing on semantic units rather than document elements (e.g., figures, tables, text). A semantic unit refers to a group of elements that are semantically deeply related to multiple elements contained in the document image data. That is, if other elements are essential to understanding the meaning of a certain element, those elements are considered a semantic unit. Examples of semantic units include paragraph titles and paragraphs, and figures and captions.

[0025] Figure 4 shows an example of region division by the region division unit 101. In Figure 4, the region division unit 101 divides the document image data 40 into semantically cohesive rectangular regions 41a to 41f. For example, in rectangular region 41d, the paragraph title, figure, and caption are recognized as a single semantically cohesive unit.

[0026] The region segmentation unit 101 can perform the above-described region segmentation by using a pre-trained object detection model. Examples of object detection models include the YOLO (You Only Look Once) series, DETR (End-to-End Object Detection with Transformers) models, and RT-DETR (Real-Time Detection Transformer) models. For the training dataset, for example, a dataset in which semantic regions within document image data are annotated with rectangles may be used.

[0027] The region division unit 101 outputs the region-divided document image data to the rectangle selection unit 102. Each region of the document image data divided by the region division unit 101 will also be referred to as a "candidate rectangle" below.

[0028] The rectangle selection unit 102 selects a candidate rectangle from the candidate rectangles included in the document image data to be used for subsequent processing.

[0029] Figure 5 is a diagram illustrating the overview of the processing performed by the rectangle selection unit 102. For the sake of explanation, Figure 5 uses document image data consisting only of text. Figure 5(A) shows the result of region division by the region division unit 101. The region division unit 101 divides a document image data into candidate rectangles B01 to B14. As shown in Figure 5(A), the region division by the region division unit 101 may output candidate rectangles with overlapping regions. Figure 5(B) shows the result of candidate rectangle selection by the rectangle selection unit 102. The rectangle selection unit 102 selects a set of rectangles from candidate rectangles B01 to B14, consisting of candidate rectangles B02, B04 to B07, B09 to B11, and B14. As shown in Figure 5(B), the rectangle selection unit 102 selects a set of rectangles that do not overlap and have no omissions, i.e., MECE (Mutually Exclusive Collectively Exhaustive).

[0030] The candidate rectangles selected by the rectangle selection unit 102 will also be referred to as "selected rectangles" below.

[0031] Figure 6 is a diagram illustrating the processing performed by the rectangle selection unit 102. For the sake of explanation, Figure 6 uses document image data consisting only of text. As shown in Figure 6(A), first, the rectangle selection unit 102 initializes the selection rectangle to an empty set (i.e., a set with no elements). Next, as shown in Figure 6(B), the rectangle selection unit 102 selects the candidate rectangle with the highest predicted probability from the object detection model. Then, the rectangle selection unit 102 calculates an index representing the quality of the selection. Examples of indices representing the quality of the selection include coverage rate, overlap rate, non-overlap rate, size of the selection rectangle, or a combination thereof. In this embodiment, a combination of coverage rate (R1) and overlap rate (R2) is used as the index representing the quality of the selection. The index representing the quality of the selection (I) can be expressed, for example, by the following formula. W1 and W2 are pre-set weights. I = (R1 × W1) + {(1 / R2) × W2}

[0032] Coverage is expressed, for example, as the ratio of the area of ​​the union of selected rectangles to the total area of ​​the document image data. Overlap rate is expressed, for example, as the ratio of the area of ​​the overlapping portion of two or more rectangular regions to the total area of ​​the document image data.

[0033] Furthermore, the index (I) is not limited to the above; for example, the average of the coverage rate and the non-overlap rate may also be used as an index. The non-overlap rate is expressed, for example, as the ratio of the area covered by one or fewer selected rectangles to the total area of ​​the image.

[0034] The rectangle selection unit 102 does not add any selected rectangles if the index representing the quality of the selection does not improve. On the other hand, the rectangle selection unit 102 adds a selected rectangle if the index representing the quality of the selection improves. Then, the rectangle selection unit 102 selects the next highest candidate rectangle based on the prediction probability of the object detection model and repeats the above process.

[0035] Figure 6(C) shows the selection process by the rectangle selection unit 102. In Figure 6(C), it is assumed that a candidate rectangle B03 has been added by the rectangle selection unit 102. When candidate rectangle B03 is added, the overlap rate increases, while the coverage rate remains unchanged. In this case, the rectangle selection unit 102 does not add candidate rectangle B03 as a selected rectangle, but deletes it.

[0036] Returning to Figure 3, the rectangle selection unit 102 outputs the selected set of rectangles to the region division unit 101. The region division unit 101 performs region division on each rectangular region included in the set of rectangles. Then, the rectangle selection unit 102 selects a MECE set of rectangles from the candidate rectangles. Figure 7 shows an example of further region division by the region division unit 101. Figure 7(A) shows the set of rectangles selected by the rectangle selection unit 102. The set of rectangles includes rectangular regions 71 to 75. The region division unit 101 performs region division on each of the rectangular regions 71 to 75 in Figure 7(A). Figure 7(B) shows the result of region division by the region division unit 101. In Figure 7(B), rectangular region 75 is further divided into rectangular regions 75a to 75e. Note that for rectangular regions 71 to 74, it is determined that sufficient region division has been performed by the object detection model, and no further region division is performed.

[0037] In this way, the region division by the region division unit 101 and the selection of candidate rectangles by the rectangle selection unit 102 are performed recursively. By performing region division recursively, a set of rectangles with an optimal information density for the VLM processing described later can be obtained.

[0038] The rectangle selection unit 102 outputs the rectangle set to the text conversion unit 103 when a predetermined termination condition is met. A first example of the predetermined termination condition is that sufficient region division has been performed and no further region division is performed by the object detection model. In this case, the rectangle selection unit 102 should repeatedly input each rectangular region included in the rectangle set to the region division unit 101 until it can no longer be divided further. A second example of the predetermined termination condition is that for all rectangular regions included in the rectangle set, the ratio of the area of ​​that rectangular region to the total document image data (hereinafter also referred to as "area ratio") is less than or equal to a predetermined value. In this case, the rectangle selection unit 102 calculates the area ratio for each rectangular region and inputs any rectangular regions with an area ratio greater than the predetermined value back into the region division unit 101. The rectangle selection unit 102 then determines that the termination condition has been met when the area ratio of all rectangular regions is less than or equal to the predetermined value.

[0039] The text conversion unit 103 uses a Visual Memory Unit (VLM) to convert document image data within each rectangular area of ​​the rectangular set into text. Specifically, the text conversion unit 103 selects a prompt corresponding to each rectangular area from among multiple prompts. Prompts are pre-prepared for each type of document element, and the text conversion unit 103 selects a prompt based on the type of document element contained in the rectangular area. For example, if a rectangular area contains a table, the text conversion unit 103 selects a prompt from among multiple prompts that includes instructions for converting the table into text. The text conversion unit 103 then inputs the rectangular area and the corresponding prompt into the VLM and obtains text data from the VLM, which is the document image data within the rectangular area converted into text.

[0040] The following is an example of a prompt. • Example prompt for document elements that are text: Please convert the following document images to Markdown text format completely and accurately. • Example prompt for when a document element contains a table: "Please convert the following... If the table in the document contains merged cells, please split the cells and duplicate the contents into the split cells." • Example prompt for document elements containing images: "Please convert the following: Convert graphs to tabular format whenever possible. Convert diagrams to Mermaid format text. Describe other figures in natural language, ensuring all information within the figure is included."

[0041] The text conversion unit 103 outputs the text data obtained from the VLM to a text database.

[0042] In the above configuration, the region division unit 101 is an example of a partial region acquisition means, the rectangular selection unit 102 is an example of a selection means, and the text conversion unit 103 is an example of a conversion means.

[0043] Furthermore, document image data is an example of content, and each region of the divided document image data is an example of a sub-region. Also, a set of rectangles is an example of a set of regions. Furthermore, the conversion to text by the text conversion unit 103 is an example of conversion to arbitrary content by the conversion means.

[0044] [Processing flow] Next, the text conversion process performed by the information processing device 10 will be described. Figure 8 is a flowchart of the text conversion process performed by the information processing device 10. This process is realized when the processor 12 shown in Figure 2 executes a pre-prepared program and operates as each element shown in Figure 3.

[0045] First, the region division unit 101 acquires document image data, including figures and tables, through the I / F 11 (step S101). The region division unit 101 then divides the document image data into regions, focusing on semantic units (step S102). The region division unit 101 outputs the divided document image data to the rectangular selection unit 102.

[0046] Next, the rectangle selection unit 102 selects a set of MECE (Mutually Exclusive, Collectively Exhaustive) rectangles from the candidate rectangles in the document image data (step S103). Next, the rectangle selection unit 102 determines whether or not to perform region division again for each rectangular region. This determination is made using the predetermined termination conditions mentioned above. If the rectangle selection unit 102 decides to perform region division again (step S104: Yes), it outputs the set of rectangles to the region division unit 101. Then, the process returns to step S102. On the other hand, if the rectangle selection unit 102 decides not to perform region division again (step S104: No), it outputs the set of rectangles to the text conversion unit 103. Then, the process proceeds to step S105.

[0047] The text conversion unit 103 converts the document image data within each rectangular area into text (step S105). The text conversion unit 103 outputs the text data to a text database. Then the process ends.

[0048] [Differentiation] Next, a modified example of the first embodiment will be described.

[0049] The region segmentation unit 101 can perform region segmentation of document image data using one or more object detection models. When using multiple object detection models, the region segmentation unit 101 combines the region segmentation results from the multiple object detection models, that is, it collectively outputs the candidate rectangles output by the multiple object detection models to the rectangle selection unit 102. The rectangle selection unit 102 then selects a desirable set of rectangles from the candidate rectangles. Depending on the type of object detection model, it may output different region segmentation results. By combining the results of multiple object detection models of different types and selecting a set of rectangles, fluctuations in accuracy can be prevented.

[0050] <Second Embodiment> Figure 9 is a block diagram showing the functional configuration of the information processing apparatus in the second embodiment. The information processing apparatus 200 includes a partial area acquisition means 201, a selection means 202, and a conversion means 203.

[0051] Figure 9 is a flowchart of the processing performed by the information processing device of the second embodiment. The partial region acquisition means 201 acquires multiple partial regions from the content (step S201). The selection means 202 selects a set of regions from the multiple partial regions (step S202). The conversion means 203 converts each of the partial regions included in the set of regions into arbitrary content (step S203).

[0052] The partial region acquisition means 201 can be implemented using the region division unit 101 according to the first embodiment. The selection means 202 can be implemented using the rectangular selection unit 102 according to the first embodiment. The conversion means 203 can be implemented using the text conversion unit 103 according to the first embodiment.

[0053] According to the information processing device 200 of the second embodiment, it is possible to convert a document containing visual elements into text.

[0054] Some or all of the above embodiments may also be described as follows, but are not limited to the following:

[0055] (Note 1) A means for obtaining multiple sub-regions from content, A selection means for selecting a set of regions from the aforementioned multiple subregions, A transformation means that transforms each sub-region included in the aforementioned set of regions into arbitrary content, An information processing device equipped with the following features.

[0056] (Note 2) The aforementioned partial region acquisition means performs region division of the content using one or more object detection models. The information processing device described in Appendix 1, wherein the one or more object detection models take the content as input and output multiple semantically coherent subregions from the content.

[0057] (Note 3) The information processing apparatus according to Appendix 1 or 2, wherein the selection means selects one candidate from each of the plurality of sub-regions, and adds the candidate to the region set if adding the candidate to the region set improves a predetermined index.

[0058] (Note 4) The aforementioned predetermined indicator is an indicator representing the quality of the selection state, and includes coverage rate, overlap rate, size of a sub-region, or a combination thereof, as described in Appendix 3 of the information processing apparatus.

[0059] (Note 5) The information processing device described in Appendix 4, wherein the predetermined index is the weighted sum of the reciprocal of the coverage rate and the overlap rate, or the average of the coverage rate and the non-overlap rate.

[0060] (Note 6) The sub-region acquisition means performs region division again on each sub-region included in the region set, The selection means is an information processing apparatus according to any one of the appendices 1 to 3, which selects a set of regions from the subregions that have been re-divided into regions.

[0061] (Note 7) The conversion means inputs each sub-region included in the set of regions and a prompt corresponding to each sub-region into a visual language model, and obtains text data for each sub-region from the visual language model, according to any one of the appendices 1 to 3, the information processing apparatus.

[0062] (Note 8) The conversion means switches prompts based on the elements included in each sub-region. The aforementioned element is an information processing device according to Appendix 7, which includes at least one of text, images, and tables.

[0063] (Note 9) A computer-based information processing method, Extract multiple subregions from the content, From the aforementioned multiple subregions, select a set of regions, An information processing method that converts each sub-region included in the aforementioned set of regions into arbitrary content.

[0064] (Note 10) Extract multiple subregions from the content, From the aforementioned multiple subregions, select a set of regions, A program that causes a computer to perform a process of converting each sub-region included in the aforementioned set of regions into arbitrary content.

[0065] Furthermore, some or all of the configurations described in Appendices 2 to 8, which are subordinate to Appendice 1 above, may also be subordinate to Appendices 9 and 10 in the same way as those described in Appendices 2 to 8. Moreover, not limited to Appendices 1, 9, and 10, some or all of the configurations described as appendices may also be subordinate to various hardware, software, various recording means for recording software, or systems, without departing from the embodiments described above.

[0066] Although the present disclosure has been described above with reference to embodiments and examples, the present disclosure is not limited to the above embodiments and examples. Various modifications to the structure and details of the present disclosure can be understood by those skilled in the art within the scope of the present disclosure. [Explanation of Symbols]

[0067] 10 Information Processing Devices 101 Area division part 102 Rectangular selection area 103 Text Conversion Section

Claims

1. A means for acquiring multiple partial regions from document image content including visual elements, A selection means for selecting a set of regions from the aforementioned plurality of subregions, A conversion means that converts each of the subregions included in the aforementioned set of regions into text content, Equipped with, The selection means selects one candidate from each of the plurality of sub-regions, and if adding the candidate to the region set improves a predetermined index, it adds the candidate to the region set. The aforementioned predetermined indicator is an indicator representing the quality of the selection state, and includes coverage rate, overlap rate, size of a sub-region, or a combination thereof, in the information processing device.

2. The aforementioned partial region acquisition means performs region division of the document image content using one or more object detection models. The information processing apparatus according to claim 1, wherein the one or more object detection models take the document image content as input and output multiple semantically coherent subregions from the document image content.

3. The information processing apparatus according to claim 1, wherein the predetermined index is a weighted sum of the reciprocal of the coverage rate and the overlap rate, or the average of the coverage rate and the non-overlap rate.

4. The sub-region acquisition means performs region division again on each sub-region included in the region set, The information processing apparatus according to claim 1 or 2, wherein the selection means selects a set of regions from the subregions that have been re-divided into regions.

5. The information processing apparatus according to claim 1 or 2, wherein the conversion means inputs each sub-region included in the set of regions and a prompt corresponding to each sub-region to a visual language model, and obtains text data for each sub-region from the visual language model.

6. The conversion means switches prompts based on the elements included in each sub-region. The information processing apparatus according to claim 5, wherein the element includes at least one of text, an image, and a table.

7. A computer-based information processing method, The process involves obtaining multiple sub-regions from document image content that includes visual elements. A selection process is performed to select a set of regions from the aforementioned multiple subregions. A conversion process is performed to convert each sub-region included in the aforementioned set of regions into text content. The selection process selects one candidate from each of the multiple sub-regions, and if adding the candidate to the region set improves a predetermined index, the candidate is added to the region set. The aforementioned predetermined indicator is an indicator representing the quality of the selection state, and includes coverage rate, overlap rate, size of sub-regions, or a combination thereof, in an information processing method.

8. An acquisition process is performed to acquire multiple partial regions from document image content that includes visual elements, A selection process is performed to select a set of regions from the aforementioned multiple subregions. A conversion process is performed to convert each sub-region included in the aforementioned set of regions into text content. The selection process selects one candidate from each of the multiple sub-regions, and if adding the candidate to the region set improves a predetermined index, the candidate is added to the region set. The aforementioned predetermined indicator is an indicator representing the quality of the selection state, and includes coverage rate, overlap rate, size of sub-regions, or a combination thereof, and is a program that causes a computer to perform processing.

Images