Information provision device, information provision method, and program

Abstract

To accurately understand the content of a document. [Solution] The information providing device 10 has a drawing unit 151a and an extraction unit 151d. The drawing unit 151a creates an image by drawing the document output based on the file. The merging unit 151c determines the priority of each layer, which is based on information indicating the structure of the document contained in the file (e.g., PDF operators) and an image layer. The extraction unit 151d uses a method appropriate to each layer to extract the contents of the layers whose priority satisfies the conditions and stores the extracted contents in a database.

Description

Technical Field

[0001] The present invention relates to an information providing apparatus, an information providing method, and a program.

Background Art

[0002] Conventionally, OCR (Optical Character Recognition) is known as a method for reading characters from an image (see, for example, Patent Document 1). For example, AI (Artificial Intelligence) may be used in OCR.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the conventional technology has a problem that the content of a document may not be accurately recognized.

[0005] As a document format, PDF (Portable Document Format) that handles various media types is known. In a PDF document, various media such as text and images are mixed. Also, the layouts in PDF documents are various.

[0006] AI may have difficulty reading a document containing various elements such as PDF. For example, AI may not accurately recognize media and layouts that have not been previously learned.

[0007] [[ID=4,7]]The present invention has been made in view of the above, and an object thereof is to accurately recognize the content of a document.

Means for Solving the Problems

[0008] To solve the above-mentioned problems and achieve the objective, the information providing device of the present invention is characterized by comprising: a drawing unit that creates an image of a document output based on a file; a determination unit that determines the priority of each of the layers, which is based on information indicating the structure of the document contained in the file and the image; and an extraction unit that extracts the contents of the layers whose priority satisfies the conditions from among the layers using a method corresponding to each of the layers, and stores the extracted contents in a database. [Effects of the Invention]

[0009] According to the present invention, the contents of a document can be accurately recognized. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 is a diagram illustrating the information provision process of the information provision system. [Figure 2] Figure 2 is a diagram illustrating the data collection process of the information provision system. [Figure 3] Figure 3 illustrates the process of extracting the contents of a document. [Figure 4] Figure 4 shows an example of the configuration of an information provision device. [Figure 5] Figure 5 is a diagram illustrating the drawing process. [Figure 6] Figure 6 is a diagram illustrating the merge process. [Figure 7] Figure 7 is a flowchart showing the flow of the service provision process. [Figure 8] Figure 8 is a flowchart showing the data collection process. [Figure 9] Figure 9 shows an example of a computer configuration for running a program. [Modes for carrying out the invention]

[0011] Hereinafter, an information providing apparatus, an information providing method, and a program according to the present application will be described in detail based on the drawings. Note that the present invention is not limited to the embodiments described below.

[0012] [First Embodiment] The information providing system of the first embodiment provides an answer based on the information stored in the database in response to a question from a user. For example, information used within an organization such as a company or a local government is stored in the database. A user belonging to the organization can input a question regarding business or the like to the information providing system and obtain an answer.

[0013] For example, a user who is an employee of a company inputs a question regarding the method of expense settlement within the company. The information providing system causes an LLM (Large Language Models) to generate an answer based on the information stored in the database. Then, the information providing system provides information regarding the method of expense settlement within the company to the user.

[0014] At this time, the information providing system can obtain information specific to the organization from the database. Also, the database stores information collected from documents issued by the organization internally or externally, or documents acquired by the organization from outside. The documents include manuals for internal use in the organization, publicity magazines of the organization itself or related organizations, etc. Also, the documents may be created as electronic files, or may be electronic versions obtained by scanning paper on which information is printed.

[0015] The information providing system performs a providing process of providing information to the user. Also, the information providing system performs a collecting process of collecting information and storing the collected information in the database.

[0016] The providing process will be described using FIG. 1. FIG. 1 is a diagram for explaining the providing process of the information providing system. Here, it is assumed that the information providing APP2A provides the functions of the information providing system 1.

[0017] The information-providing APP 2A receives an input of a question from the user U10 (step S11). The question is input, for example, by voice or text.

[0018] Next, the information-providing APP 2A searches the external information DB 3 (step S12) and receives the provision of data based on the search result (step S13). The accumulation of data in the external information DB 3 is performed in the collection process described later.

[0019] Subsequently, the information-providing APP 2A inputs a prompt based on the data obtained from the external information DB 3 to the LLM 4 (step S14). The information-providing APP 2A obtains an answer output from the LLM 4 for the prompt (step S15). The LLM 4 may be provided by a server outside the information-providing system 1 or may be included in the information-providing system 1. Also, the information-providing APP 2A may create a prompt based on the data, or may input the data together with the prompt to the LLM 4. The information-providing APP 2A provides information to the user U10 based on the answer (step S16).

[0020] The collection process will be described using FIG. 2. FIG. 2 is a diagram for explaining the collection process of the information-providing system. Here, it is assumed that the preprocessing APP 2B provides the functions of the information-providing system 1.

[0021] The preprocessing APP 2B receives the input of a plurality of documents in various formats (step S21). Then, the preprocessing APP 2B accumulates the information extracted from the documents in the external information DB 3 (step S22).

[0022] FIG. 3 is a diagram for explaining the process of extracting the content of a document. As shown in FIG. 3, the preprocessing APP 2B extracts the information 60 from the document 50. The document 50 is, for example, a PDF. The document 50 includes not only text but also images. The information 60 is text summarizing the content of the document 50 including images. Also, the information 60 converts the table of the document 50 into an html tag format capable of expressing cell merging.

[0023] Figure 4 shows an example of the configuration of an information provision device. As shown in Figure 4, the information provision device 10 includes a communication unit 11, an input unit 12, an output unit 13, a storage unit 14, and a control unit 15.

[0024] The communication unit 11 is an interface for communicating with other devices. For example, the communication unit 11 is a NIC (Network Interface Card).

[0025] The input unit 12 is an interface for receiving data input. For example, the input unit 12 is connected to input devices such as a keyboard and a mouse.

[0026] The output unit 13 is an interface for outputting data. For example, the output unit 13 is connected to output devices such as a display and a speaker.

[0027] The storage unit 14 is a storage device such as an HDD (Hard Disk Drive), SSD (Solid State Drive), or optical disc. Alternatively, the storage unit 14 may be a rewritable semiconductor memory such as RAM (Random Access Memory), flash memory, or NVSRAM (Non-Volatile Static Random Access Memory). The storage unit 14 stores the OS (Operating System) and various programs executed by the information providing device 10. The storage unit 14 also stores model information 141 and external information 142.

[0028] Model information 141 is information about a model used for image recognition. For example, model information 141 consists of parameters of a machine learning model such as a neural network.

[0029] External information 142 corresponds to the external information DB3 in Figures 1 and 2. In other words, the information providing device 10 stores information in external information 142. The information providing device 10 also searches external information 142 and obtains information for generating a response using LLM.

[0030] The control unit 15 controls the entire information providing device 10. The control unit 15 is, for example, an electronic circuit such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), or GPU (Graphics Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0031] The control unit 15 has internal memory for storing programs and control data that define various processing procedures, and executes each process using the internal memory. In addition, the control unit 15 functions as various processing units when various programs are run. For example, the control unit 15 has a collection unit 151 and a supply unit 152.

[0032] The collection unit 151 performs the collection process. The collection unit 151 functions as a pre-processing APP2B. The provision unit 152 performs the provision process. The provision unit 152 functions as an information provision APP2A.

[0033] The collection unit 151 includes a drawing unit 151a, a recognition unit 151b, a merging unit 151c, and an extraction unit 151d.

[0034] The drawing unit 151a creates an image that depicts the document output based on the file. Figure 5 is a diagram illustrating the drawing process. As shown in Figure 5, the drawing unit 151a draws an image 51 that mimics the document 50 output based on the PDF file. Image 51 may be a screenshot of the document 50.

[0035] The recognition unit 151b recognizes the document content through image analysis. For example, the recognition unit 151b performs image recognition using a pre-trained machine learning model built based on the model information 141. The recognition unit 151b can read layers through image analysis. In addition, the recognition unit 151b can determine whether the text in the document is written vertically or horizontally.

[0036] The recognition unit 151b reads layers 511, 512, and 514 based on image 51. Note that each layer represents a region within the document.

[0037] The merge unit 151c determines the priority of layers based on information indicating the structure of the document contained in the file, and layers based on images. For example, the merge unit 151c merges layers 502, 503, 504, and 505, which are indicated by the operators contained in the PDF file from which document 50 originated, with layers 511, 512, and 514, which are read from image 51. Note that the merge unit 151c is an example of a determination unit.

[0038] Figure 6 illustrates the merging process. As shown in Figure 6, the merging unit 151c ultimately determines that layers 511, 502, 503, 514, and 505 are to be included in the document layout.

[0039] Here, we assume the priority is 1 or 0 (1 is higher). As shown in Figure 5, there is no layer indicated by the operator at the location in document 50 corresponding to layer 511 in image 51. The merge unit 151c determines the priority of layer 511, where there is no layer based on the operator at the corresponding location, to be 1.

[0040] Furthermore, no layer read from the image exists at the location of image 51 corresponding to layer 505 of document 50. The merge unit 151c determines the priority of layer 505, for which no layer read from the image exists at the corresponding location, to be 1.

[0041] Layers that do not have a corresponding layer, such as layer 511 and layer 505, are called independent layers. The merge unit 151c can determine the placement of independent layers (where independent layers are inserted) based on the order of the layers, whether the text is written vertically or horizontally, and the position of the layer.

[0042] Furthermore, at the location of image 51 corresponding to layer 502 of document 50, there exists layer 512, which was read from the image. In other words, layer 502 and layer 512 overlap.

[0043] If the merge unit 151c finds that the first layer identified based on the operators contained in the PDF file and the second layer identified based on the image overlap, it determines that the priority of the first layer is higher than the priority of the second layer. That is, the merge unit 151c determines the priority of layer 502 to be 1 and the priority of layer 512 to be 0.

[0044] Furthermore, the merge unit 151c does not always determine the priority of the first layer to be the highest; it may determine the priority based on the quality of each layer. For example, the merge unit 151c may set the quality value of the layer identified based on the operator to a fixed value, and calculate the quality value of the image-based layer based on the confidence level in image recognition. Then, the merge unit 151c may determine the priority of the layer with the higher quality value to be 1, and the priority of the layer with the lower quality value to be 0.

[0045] The method for determining priority is not limited to the methods described above. For example, the merge unit 151c compares the length of the text in layers identified based on operators and layers based on images, and assigns a higher priority to the layer corresponding to the longer text. Also, for example, if character encoding issues occur when decoding the text of a layer identified based on operators, the merge unit 151c assigns a lower priority to that layer.

[0046] Figure 6 shows the layers with a priority of 1. The extraction unit 151d uses a method (logic) appropriate to each layer to extract the contents of the layers whose priority meets the condition, and stores the extracted contents in the database. In this case, the extraction unit 151d extracts the contents of the layers with a priority of 1, i.e., the layers shown in Figure 6.

[0047] Each layer is classified into one of the following categories: figure (image), table (table), text (text), or title (title). Classification may be performed by image recognition by the recognition unit 151b. The extraction unit 151d extracts the contents of the layer using a method corresponding to the result of classifying the layer as either figure, table, text, or title.

[0048] If a layer classified as text or title is based on a PDF operator (e.g., layer 502), the extraction unit 151d extracts the content of that layer from the PDF operator. If a layer classified as text or title is identified based on an image (e.g., layer 511), the extraction unit 151d extracts the content of that layer from the OCR result.

[0049] Furthermore, the extraction unit 151d takes the image of the layer classified as "figure" (for example, layer 514) and inputs it into a model capable of generating explanatory text from images (for example, VisionLLM) to extract the explanatory text as its content.

[0050] Furthermore, the extraction unit 151d takes the images from the layers classified as tables (for example, layer 505) and inputs them into a model capable of generating HTML from images (for example, img2table HTML), extracting the resulting HTML as content.

[0051] The extraction unit 151d stores the extracted content in the external information 142. For example, if the extracted content is text that can be written both vertically and horizontally, such as Japanese, the extraction unit 151d sorts the extracted content according to the corresponding layer based on the document layout before storing it. The sorted order is considered the reading order of the text as a whole document. For example, if the area of ​​the vertical text layer is larger than the area of ​​the horizontal text layer, the extraction unit 151d combines and stores the extracted content from the rightmost layer first. This is because reading vertical text from left to right would disrupt the document structure.

[0052] The extraction unit 151d may sort (determine the reading order) using the algorithm described above, or it may sort using AI. In either case, the extraction unit 151d can sort based on the information held by the layer, such as whether it is written vertically or horizontally, coordinates indicating the position and range, and the classification result (one of figure, table, text, or title).

[0053] Returning to Figure 4, the provisioning unit 152 includes a creation unit 152a, a generation unit 152b, and an output control unit 152c. The creation unit 152a creates a prompt. The prompt may include data obtained from external information 142. The creation unit 152a searches the external information 142 based on a question (inquiry) from the user and obtains the information necessary to create the prompt.

[0054] The generation unit 152b inputs prompts to the LLM and generates a response. The output control unit 152c provides information to the user based on the generated response.

[0055] The flow of the information provision process will be explained using Figure 7. Figure 7 is a flowchart of the information provision process. As shown in Figure 7, first, the information provision device 10 receives an inquiry input from the user (step S101).

[0056] Next, the information providing device 10 searches the external information database based on the inquiry (step S102). The information providing device 10 obtains the search results (step S103).

[0057] Here, the information providing device 10 creates a prompt based on the inquiry and search results (step S104). The information providing device 10 inputs the prompt into the LLM and obtains a response (step S105). Then, the information providing device 10 provides information to the user based on the response (step S106).

[0058] The flow of the data collection process will be explained using Figure 8. Figure 8 is a flowchart of the data collection process. As shown in Figure 8, the information providing device 10 creates an image by rendering (drawing) the document (step S201).

[0059] The information providing device 10 extracts a layout from the created image by layout analysis (step S202). The information providing device 10 also extracts a layout from the document's operators (step S203). Note that the layout is a collection of layers.

[0060] Here, the information provider 10 merges the layers obtained from the document and the image based on the layout (step S204). Then, the information provider 10 determines a logic (method) for each layer (step S205) and generates data based on the content of each layer using the determined logic (step S206). At this time, the information provider 10 generates data by extracting the content from the layers with the highest priority. The information provider 10 stores information corresponding to the generated data in the DB (step S207).

[0061] As explained above, the drawing unit 151a creates an image by drawing the document output based on the file. The merging unit 151c determines the priority of each layer: the layer based on information indicating the structure of the document contained in the file, and the layer based on the image. The extraction unit 151d uses a method appropriate to each layer to extract the contents of the layers whose priority satisfies the conditions, and stores the extracted contents in the database. In this way, the information providing device 10 can accurately recognize the contents of a document by using not only images but also information indicating the structure of the document (for example, PDF operators).

[0062] According to this embodiment, recognition accuracy is improved, particularly for Japanese text, where OCR recognition accuracy can sometimes be lower compared to English.

[0063] For example, the extraction unit 151d extracts the contents of a layer using a method that corresponds to the result of classifying the layer as either an image, a table, text, or a title. This allows the information providing device 10 to extract the contents in a way that is appropriate to the characteristics of each layer.

[0064] The extraction unit 151d converts the layers classified as tables into HTML. This allows the information providing device 10 to extract the content in a format that can later be reproduced as a table.

[0065] The merging unit 151c determines that if a first layer identified based on operators contained in a PDF file and a second layer identified based on an image overlap, the priority of the first layer is higher than the priority of the second layer. The extraction unit 151d extracts the content of the layer with the highest priority among the multiple overlapping layers. The extraction unit 151d also extracts the embedded text of the layer identified based on operators contained in the PDF file. In this way, the information providing device 10 prioritizes the extraction of text based on operators that are expected to be more accurate, thereby enabling it to collect more accurate information.

[0066] [System configuration, etc.] Each component of the illustrated device is a functional concept and does not necessarily have to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed or integrated in any unit according to various loads and usage conditions. Furthermore, each processing function performed by each device can be implemented, all or any part of it, by a CPU and a program that is analyzed and executed by that CPU, or by hardware using wired logic. Note that the program may be executed not only by the CPU but also by other processors such as a GPU.

[0067] Furthermore, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified.

[0068] [program] In one embodiment, the information providing device 10 can be implemented by installing a program that performs the above processing as packaged software or online software on a desired computer. For example, by having the above program run on an information processing device, the information processing device can be made to function as the information providing device 10. The information processing device referred to here includes desktop or notebook personal computers. In addition, mobile communication terminals such as tablet terminals and smartphones are also included in the category of information processing devices.

[0069] Furthermore, the information providing device 10 may be implemented as a web server, or it may be implemented as a cloud service that provides the above-mentioned processing services through outsourcing.

[0070] Figure 9 shows an example configuration of a computer running a program. Computer 1000 has, for example, memory 1010 and a CPU 1020. Computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0071] Memory 1010 includes ROM (Read Only Memory) 1011 and RAM (Random Access Memory) 1012. ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0072] The hard disk drive 1090 stores, for example, the OS 1091, application programs 1092, program modules 1093, and program data 1094. That is, the programs that define each process of the information providing device 10 are implemented as program modules 1093 in which executable code for a computer is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, a program module 1093 for performing processes similar to the functional configuration of the information providing device 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0073] Furthermore, the configuration data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, memory 1010 or hard disk drive 1090. The CPU 1020 then reads the program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as needed and executes the processing of the above-described embodiment.

[0074] Furthermore, the program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via a network interface 1070. [Explanation of Symbols]

[0075] 1. Information Provision System 2A Information provision APP 2B Preprocessing APP 3. External Information Database 4 LLM 10 Information provision device 11 Communications Department 12 Input section 13 Output section 14 Storage section 15 Control Unit 50 documents 51 images 60 Information 141 Model Information 142 External Information 151 Collection Department 151a Drawing section 151b Recognition part 151c Merge section 151d Extraction part 152 Provision Department 152a Creation Section 152b Generator 152c Output Control Unit Layers 502, 503, 504, 505, 511, 512, and 514

Claims

1. A drawing unit that creates an image by drawing a document based on a file, A determination unit that determines the priority of each of the layers, one based on information indicating the structure of the document contained in the file and the other based on the image. An extraction unit that uses a method appropriate to each of the aforementioned layers to extract the contents of the layers whose priority satisfies the conditions, and stores the extracted contents in a database. An information providing device characterized by having the following features.

2. The extraction unit extracts the contents of the layer using a method that corresponds to the result of classifying the layer as either an image, a table, text, or a title. The information providing device according to feature 1.

3. The extraction unit converts the layers classified into tables into HTML. The information providing device according to feature 2.

4. The determination unit determines that if a first layer identified based on the operators contained in the PDF file and a second layer identified based on the image overlap, the priority of the first layer is higher than the priority of the second layer. The extraction unit extracts the contents of the layer with the highest priority among multiple overlapping layers. The information providing device according to feature 2.

5. The extraction unit extracts embedded text from layers identified based on operators contained in the PDF file. The information providing device according to any one of claims 1 to 3.

6. An information provision method performed by an information provision device, The drawing process involves creating an image by rendering a document based on a file, and A decision step to determine the priority of each of the layers: a layer based on information indicating the structure of the document contained in the file, and a layer based on the image. An extraction step in which, using a method appropriate to each of the aforementioned layers, the contents of the layers whose priority satisfies the conditions are extracted from the aforementioned layers, and the extracted contents are stored in a database, A method of providing information characterized by including the following.

7. A drawing step that creates an image by drawing a document based on a file, A decision step to determine the priority of each of the layers: a layer based on information indicating the structure of the document contained in the file, and a layer based on the image. An extraction step in which, using a method appropriate to each of the aforementioned layers, the contents of the layers whose priority satisfies the conditions are extracted from the aforementioned layers, and the extracted contents are stored in a database, A program characterized by causing a computer to execute something.

Images