Information processing device, information processing method, and information processing program

Abstract

When using OCR to obtain data and prompts from image data containing lines and shapes, and then feeding this data to a trained AI model to acquire information, there is a problem in that the AI ​​model cannot properly extract the lines and shapes contained in the image data. [Solution] The present invention provides an information processing device that receives a prompt to input to a trained AI model for obtaining desired information, which is information that is desired regarding geometric information, from file data containing geometric information; obtains geometric information from the file data, which is information that makes the geometric information identifiable; adds the geometric information obtained from the file data to the file data; and inputs the received prompt and the file data to which the obtained geometric information has been added to the trained AI model to obtain the desired information.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus that extracts desired information from file data.

Background Art

[0002] For example, file data such as documents and drawings used in the construction, design, civil engineering, and plant industries contains various types of information and has various formats. For example, there are architectural drawings, regulations documents, construction management histories, design outlines, system diagrams, elevation views, detailed drawings, etc. For example, the number of data items obtained from one architectural design drawing data may be several hundred or more. These file data are not neatly organized in a format such as "columns and rows" or "JSON format", but are so-called unstructured file data and are not suitable for search, statistical processing, use in AI (artificial intelligence) learning data, etc. Therefore, there is a demand to structure these unstructured file data in a format such as a spreadsheet.

[0003] In order to structure unstructured file data as described above, it is necessary to extract individual information contained in the unstructured file data for each data item. As a technique for extracting such information, for example, there is a technique described in Patent Document 1. Specifically, it extracts predetermined information from image data generated by reading a paper-based voucher with OCR (optical character recognition) based on the position information on the paper surface defined for each item.

[0004] On the other hand, in recent years, the development of techniques for information extraction using pre-trained AI models represented by LLM (Large Language Model) has also advanced. For example, there is also a technique for reading a bill to convert it into text data and automatically extracting information ("billing amount", "biller", etc.) necessary for accounting journal processing (Non-Patent Document 1).

Prior Art Documents

Patent Documents

[0005] [Patent Document 1] Japanese Patent Publication No. 2011-60219 [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] Figure 7 is a conceptual diagram illustrating an example of converting PDF file data into text data using OCR. Figure 7(a) shows a portion of a PDF file containing a floor plan of a house. Figure 7(b) shows an example of converting the text for "Western-style room (6 tatami mats)", "closet", and "Japanese-style room (8 tatami mats)" from the floor plan in Figure 7(a) into JSON format, along with their location information. Similarly, Figure 7(c) shows an example of converting the text into Array format, also containing location information. In this way, text data generated by OCR is represented by the content of the characters and their location information.

[0007] By providing the text data generated by OCR in this way to the trained AI model disclosed in Non-Patent Document 1 along with prompts, it is possible to obtain the information indicated by the prompts.

[0008] However, since the information that can be obtained by OCR is limited to the content and location information of the text, there is a problem in that it cannot answer prompt questions such as, "Please tell me the dimensions of the bathroom," which require information about the shapes shown in the floor plan. [Means for solving the problem]

[0009] Therefore, in order to solve the above problems, the present invention provides the following information processing device, etc. Specifically, an information processing device is provided which includes: a prompt receiving unit that receives a prompt to input to a trained AI model for obtaining desired information, which is information that is desired regarding geometric information, from file data containing geometric information; a geometric information acquisition unit that acquires geometric information from the file data, which is information that makes the geometric information identifiable; a geometric information assignment unit that assigns the geometric information acquired from the file data to the file data; and a desired information acquisition unit that inputs the received prompt and the file data to which the acquired geometric information has been assigned to the trained AI model for obtaining the desired information.

[0010] Furthermore, in addition to the above features, the present information processing apparatus further includes a geometric information acquisition unit that acquires geometric information from each of a plurality of file data containing geometric information, a geometric information assignment unit that assigns the geometric information acquired from the file data to each of the file data, and a file data storage unit that stores each of the file data to which the geometric information has been assigned.

[0011] In addition to the features described above, the desired information acquisition unit provides an information processing device that acquires the desired information from the file data stored in the file data storage unit.

[0012] In addition to the features described above, the geometric information acquisition unit selects or generates an acquisition method for acquiring the geometric information by inputting the received prompt and the file data into the trained AI model, and provides an information processing device that acquires the geometric information using that acquisition method.

[0013] In addition to the features described above, the geometric information acquisition unit inputs sample file data, which serves as the correct answer data, and the correct geometric information into a trained AI model to select or generate an acquisition method for acquiring the geometric information, and provides an information processing device that acquires the geometric information using that acquisition method.

[0014] Furthermore, the present invention provides an information processing device that performs an information processing method, comprising: a prompt receiving step of receiving a prompt to input to a trained AI model for obtaining desired information, which is desired information regarding a figure, from file data containing a figure; a geometric information acquisition step of obtaining geometric information from the file data, which is geometric information corresponding to the shape of the figure contained therein and which makes the figure identifiable; a geometric information assignment step of assigning the geometric information obtained from the file data to the file data; and a desired information acquisition step of inputting the received prompt and the file data to which the acquired geometric information has been assigned to the trained AI model to obtain the desired information.

[0015] The present invention also provides an information processing program that causes an information processing device to execute the following steps: a prompt receiving step which receives a prompt to input to a trained AI model for obtaining desired information, which is information about a figure that is desired regarding a figure, from file data containing the figure; a geometric information acquisition step which obtains geometric information from the file data that is geometric information corresponding to the shape of the figure contained and that makes the figure identifiable; a geometric information assignment step which assigns the geometric information obtained from the file data to the file data; and a desired information acquisition step which inputs the received prompt and the file data to which the acquired geometric information has been assigned to the trained AI model for obtaining the desired information. [Effects of the Invention]

[0016] According to the present invention, it is possible to provide an information processing apparatus that can extract information indicated by a prompt even when a prompt including an instruction regarding information such as a figure included in file data is given.

Brief Description of the Drawings

[0017] [Figure 1] Block diagram showing an example of the functional configuration of the information processing apparatus according to the embodiment [Figure 2] Conceptual diagram showing an example of a mode of attaching acquired morphological information to text data [Figure 3] Conceptual diagram showing a mode of acquiring a wiring system diagram of a socket as geometric information [Figure 4] Conceptual diagram of file data showing a floor plan [Figure 5] Conceptual diagram showing an example of the hardware configuration for realizing the information processing apparatus according to the embodiment [Figure 6] Flow diagram briefly showing an example of the processing flow of the information processing apparatus according to the embodiment [Figure 7] Conceptual diagram showing an example of converting PDF-formatted file data into text data by OCR

Modes for Carrying Out the Invention

[0018] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Note that the present invention should not be limited to these embodiments, and can be implemented in various modes without departing from the gist thereof.

[0019] <Example> <Overview> The present invention attaches geometric information such as lines and figures included in file data to the file data, gives a prompt including an instruction regarding the lines and figures and the file data to a learned AI model, and can also extract in detail information regarding the lines and figures in the file data.

[0020] The following describes the functions and processing flow of the information processing device, as well as the hardware components. The functional blocks of this information processing device described below can be implemented as a combination of hardware and software. Specifically, if it utilizes a computer, these include hardware components such as a CPU (Central Processing Unit), main memory, a bus, or storage (hard disk drives, non-volatile memory, storage media such as CDs and DVDs, and drives for reading those media), input devices used for information input, and other external peripheral devices, as well as interfaces for these external peripheral devices, communication interfaces, driver programs and other application programs for controlling the hardware, and user interface applications. The CPU then processes and stores data input from input devices and other interfaces, and held in memory and on the hard disk, according to the programs deployed in main memory. Instructions for controlling the aforementioned hardware and software are also generated. Alternatively, the functional blocks of this information processing device may be implemented using dedicated hardware.

[0021] Furthermore, this invention can be realized not only as a device or system, but also as a method. Moreover, a part of such an invention can be configured as software. In addition, a program used to execute such software on a computer, and a recording medium on which the program is fixed, are naturally included within the technical scope of this invention.

[0022] <Functional configuration> Figure 1 is a block diagram showing an example of the functional configuration of the information processing device of this embodiment. As shown in Figure 1, the information processing device 100 includes a prompt receiving unit 101, a geometric information acquisition unit 102, a geometric information assignment unit 103, a file data storage unit 104, and a desired information acquisition unit 105.

[0023] In this embodiment, the trained AI model is a machine learning model that has been pre-trained and is capable of handling general-purpose tasks. A well-known example is the Large-Scale Language Model (LLM), but it may also be a small-scale language model or a multimodal language model that can handle images or other formats.

[0024] <Prompt Reception Department> The prompt receiving unit 101 has the function of receiving prompts to input into a trained AI model to obtain desired information, which is information about geometric information, from file data containing geometric information.

[0025] "Desired information" refers to desired values ​​or information, or information obtained by performing some calculation or processing on them (total number, average information, maximum information, spatial relationships such as distance and collision, 3D model information, 3D graphic information, images, text information, etc.). For example, if the desired information is the "total number of elevators" from a set of CSV files related to the facilities of multiple buildings, the number of elevators for each building included in each file is extracted, and the sum of these numbers becomes the desired information. In addition to calculations of numerical information, the results of processing such as 3D spatial processing or 3D graphic processing can also be set as desired information. For example, if the desired information is the "locations of collisions between columns and beams" from 3D model files of multiple buildings, all the column and beam models included in each file are extracted, the presence or absence of collisions is calculated, and the 3D spatial information of the locations where collisions occur becomes the desired information.

[0026] Geometric information refers to information that represents the external appearance and contours of an object represented in file data, and consists of points, lines, circles, shapes, 3D models, and parts thereof, including symbols and signs associated with or related to them.

[0027] A prompt, for example, regarding file data such as the floor plan of a house shown in Figure 7, might be something like, "Please tell me the dimensions of the bathroom and Western-style room from this floor plan." The example prompt includes "dimensions of the Western-style room," which is an instruction related to geometric information. Also, a prompt such as, "Please extract the floor area excluding the porch from this floor plan," includes an instruction related to morphological information, such as "characters with a diamond symbol." It should be noted that the desired information is not simply obtained from the information represented in the target file data, but is also sometimes obtained as a result of processing or calculations performed on the information represented in the file data.

[0028] <Geometric information acquisition part> The geometric information acquisition unit 101 has the function of acquiring geometric information from the file data, which is geometric information that makes that geometric information identifiable. There are many types and formats of file data and they are not limited to any particular type, but examples include Word files, PDF (Portable Document Format) files, Excel files, image files, CSV (Comma Separated Values) files, text files, BIM (Building Information Modeling) files, 2DCAD (Computer Aided Design) files, and 3D model files.

[0029] Geometric information can be acquired using various image analysis and image recognition methods. It can also be acquired using pre-trained AI models that have learned about shapes such as lines and figures. Furthermore, it can be acquired by accepting geometric information input from the user. Geometric information is acquired as numerical data representing the positions of multiple points and the lines and curves connecting them, for example, in vector image format. In the case of 3D models, it is represented in the form of lines and curves in three dimensions.

[0030] Furthermore, the acquisition of geometric information can be configured to select or generate a suitable image analysis process for acquiring geometric information in response to the received prompt. For example, if a prompt such as "Please extract the total length of the building" is received, the system will recognize rectangles in the image drawn in the file data, select a processing method to analyze the drawn image based on the combination of those rectangles to determine the entire building, and acquire geometric information using that processing method.

[0031] Furthermore, by inputting the received prompt and file data into a trained AI model, it is possible to select or generate an acquisition method for obtaining geometric information and configure the model to acquire geometric information using that acquisition method.

[0032] Furthermore, by inputting sample file data that serves as the correct answer data and correct geometric information into a trained AI model, it is possible to select or generate an acquisition method for obtaining geometric information and configure the system to acquire geometric information using that acquisition method. The correct answer data is sample file data used to obtain some kind of geometric information from file data, and it contains correct geometric information that serves as an example. For example, when determining the "total number of electrical outlets in a building," a floor plan of the building showing the outlets or a wiring diagram of the outlets would be the sample file data, and the shapes representing the outlets and the lines and curves representing the wiring within that file data would be the correct geometric information. Alternatively, the system may be configured to allow the user to specify the correct geometric information in the sample file data on the viewer. This can improve usability.

[0033] In this way, by providing a trained AI model with sample file data and ground truth geometric information, it is possible to select or generate suitable acquisition methods for obtaining ground truth geometric information or similar geometric information from the file data. Furthermore, by storing the generated acquisition methods in advance, they can be selected as acquisition methods when acquiring geometric information, eliminating the need to generate acquisition methods each time.

[0034] Figure 2 is a conceptual diagram illustrating one method of acquiring geometric information. Figure 2(a) is a floor plan of the house shown in Figure 7(a). Figure 2(b) is a conceptual diagram illustrating an example of acquiring geometric information from a floor plan of a house. Here, rectangles and concave hexagons drawn on the floor plan are acquired as geometric information. For example, rectangle 201, which is drawn as a Western-style room (6 tatami mats), is acquired as geometric information. Then, by associating this rectangle with the text data "Western-style room (6 tatami mats)" on the floor plan, it can be distinguished from other geometric information. The acquired geometric information is, for example, It is represented in the form of vector information such as {name:“Western-style room (6 tatami mats)”,position:{x:110,y:120,w:130,z:240}}.

[0035] Similarly, the rectangle 202, depicted as a closet; the rectangle 203, depicted as a storage room; the rectangle 204, depicted as a Japanese-style room (8 tatami mats); the rectangle 205, depicted as a bathroom; the rectangle 206, depicted as a washroom; the rectangle 207, depicted as a toilet; the rectangle 208, depicted as a porch; and the concave hexagon 209, depicted as a corridor and entrance, are each obtained as geometric information. Alternatively, the entrance and corridor may be obtained separately as geometric information for rectangles.

[0036] Alternatively, the overall outline, including all the floor plans, may be obtained as geometric information of a single concave hexagon. By obtaining geometric information from various perspectives and angles, it is possible to accommodate various desired information. For example, based on the geometric information obtained for a corridor, the "length and width of the corridor" can be obtained as desired information. Furthermore, based on the geometric information of the overall outline of the floor plan, it becomes possible to obtain the "area of ​​the house" as desired information.

[0037] Figure 3 is a conceptual diagram illustrating how to obtain a wiring diagram of electrical outlets as geometric information. Figure 3(a) shows a wiring diagram of electrical outlets superimposed on a floor plan of a house. Figure 3(b) shows the wiring diagram of electrical outlets obtained as geometric information. As shown in this diagram, diagrams composed of lines and symbols can also be obtained as geometric information.

[0038] <Geometric information adding section> The geometric information assignment unit 103 has the function of assigning geometric information obtained from the file data to the file data. As described above, various geometric information is obtained from the file data. This acquired geometric information is assigned to the file data from which it was obtained. This means that the acquired geometric information is associated with and stored in relation to the file data.

[0039] <File Data Storage Section> The file data storage unit 104 has the function of storing each of the file data to which the geometric information has been attached. The geometric information acquisition unit acquires geometric information from multiple file data, and the geometric information attachment unit attaches the geometric information acquired from that file data to each file data. By storing multiple file data to which geometric information has been attached in this way, desired information regarding the geometric information can be obtained from the stored file data.

[0040] <Desired Information Acquisition Unit> The desired information acquisition unit 106 has the function of acquiring the desired information by inputting the received prompt and the file data to which the acquired geometric information has been added into the trained AI model. For example, as shown in Figures 2 and 3, the length and width of the corridor can be acquired by inputting the file data to which the geometric information has been added and a prompt such as "Please tell me the length and width of the corridor" into the trained AI model.

[0041] Figure 4 is a conceptual diagram of file data showing a floor plan. From this file data, geometric information can be obtained for rectangle 401, which is enclosed by a dashed line and represents a chair. Geometric information for all chairs depicted on this floor plan can also be obtained. This file data, with the added geometric information, is then fed to a trained AI model, and a prompt such as "Count the total number of chairs on this floor" is entered, resulting in the desired information "20," which is the total number of chairs on this floor.

[0042] The file data from which the desired information is to be acquired may be stored in the file data storage unit described above, or it may be stored in external storage connected to this information processing device via a communication line or the like. When extracting from file data stored in external storage, it is preferable that geometric information is acquired from the file data stored in this storage and attached to that file data. For file data from which geometric information has not been acquired and attached, it is preferable to acquire and attach the geometric information as described above before processing to acquire the desired information.

[0043] <Hardware Configuration> Figure 5 is a conceptual diagram showing an example of the hardware configuration for realizing the information processing device according to the embodiment. As shown in the figure, the information processing device 500 includes a CPU 501 that performs various calculations, a RAM 502 which is a volatile recording medium, a storage device 503 such as a flash memory or HDD which is a non-volatile storage medium, a communication interface 504, and an input / output interface 505. The RAM 502 reads programs that perform various calculations for the CPU 501 to execute and provides a work area (work area) for those programs. In addition, multiple addresses are assigned to the RAM 502, and programs executed by the CPU 501 can exchange data with each other and perform processing by identifying and accessing these addresses (this is the same throughout this specification).

[0044] Here, the functions of the prompt receiving unit 101, geometric information acquisition unit 102, geometric information assignment unit 103, and desired information acquisition unit 105 of the information processing device 100 in Figure 1 are mainly realized by the CPU 501 and RAM 502 in Figure 5. The function of the file data storage unit 104 is mainly realized by the storage 403 in Figure 4. Furthermore, when the geometric information acquisition unit 102 or the desired information acquisition unit 105 use an externally existing trained AI model, each function is realized by exchanging signals and information with each other via the communication interface 504 and input / output interface 505.

[0045] <Processing flow> Figure 6 is a simplified flowchart showing an example of the processing flow of the information processing device of Embodiment 1. First, a prompt is received to input to the trained AI model to obtain desired information, which is information that is desired regarding geometric information, from file data containing geometric information (S601: Prompt reception step). Then, geometric information, which is information that contains geometric information and makes that geometric information identifiable, is obtained from the file data (S602: Geometric information acquisition step). Then, the geometric information obtained from the file data is added to the file data (S603: Geometric information addition step). Finally, the received prompt and the file data to which the acquired geometric information has been added are input to the trained AI model to obtain the desired information (S604: Desired information acquisition step).

[0046] <Effects> According to this embodiment, the information processing device can provide a trained AI model with instructions that include information about the form of the text, such as the color and font of the characters, and perform extraction that reflects the form of the text. [Explanation of Symbols]

[0047] 100, 500: Information Processing Devices 101: Prompt Reception Department 102:Geometric information acquisition part 103:Geometric information provision part 104: File data storage unit 105: Requested Information Acquisition Department 501:CPU 502:RAM 403: Storage 504: Communication Interface 505: Input / Output Interface

Claims

1. A prompt receiving unit that receives prompts to input into a trained AI model to obtain desired information, which is the desired information regarding geometric information, from file data containing geometric information, and A geometric information acquisition unit acquires geometric information from the aforementioned file data, which is geometric information contained in the file data and is information that makes it possible to identify that geometric information. A geometric information assignment unit that assigns geometric information obtained from the file data to the aforementioned file data, A desired information acquisition unit inputs the received prompt and the file data to which the acquired geometric information has been added to the trained AI model to acquire the desired information. An information processing device having

2. The geometric information acquisition unit acquires the geometric information from each of a plurality of file data containing geometric information, The geometric information assignment unit assigns geometric information obtained from the file data to each of the file data, The information processing apparatus according to claim 1, further comprising a file data storage unit for storing the respective file data to which the geometric information has been assigned.

3. The information processing apparatus according to claim 2, wherein the desired information acquisition unit acquires the desired information from the file data stored by the file data storage unit.

4. The information processing apparatus according to claim 1, wherein the geometric information acquisition unit inputs the received prompt and the file data to the trained AI model to select or generate an acquisition method for acquiring the geometric information, and acquires the geometric information using the acquisition method.

5. The information processing apparatus according to claim 1, wherein the geometric information acquisition unit inputs sample file data which is the correct answer data and the correct geometric information into a trained AI model to select or generate an acquisition method for acquiring the geometric information, and acquires the geometric information using the acquisition method.

6. An information processing method performed by an information processing device, A prompt reception step that receives prompts to input into a trained AI model to obtain desired information, which is desired information about a shape, from file data containing the shape. A geometric information acquisition step involves obtaining geometric information from the aforementioned file data, which is geometric information corresponding to the shape of the included figure and is information that makes the figure identifiable. A geometric information assignment step of adding geometric information obtained from the file data to the aforementioned file data, A desired information acquisition step involves inputting the received prompt and the file data to which the acquired geometric information has been added into the trained AI model to acquire the desired information. An information processing device having

7. A prompt reception step that receives prompts to input into a trained AI model to obtain desired information, which is desired information about a shape, from file data containing the shape. A geometric information acquisition step involves obtaining geometric information from the aforementioned file data, which is geometric information corresponding to the shape of the included figure and is information that makes the figure identifiable. A geometric information assignment step of adding geometric information obtained from the file data to the aforementioned file data, A desired information acquisition step involves inputting the received prompt and the file data to which the acquired geometric information has been added into the trained AI model to acquire the desired information. An information processing program that causes an information processing device to execute.

Images