Information processing device, information processing method, and program

Abstract

To provide an information processing device, information processing method, and program that can ensure conversion accuracy while minimizing the occurrence of various costs. [Solution] The information processing device 10 includes a recognition unit 103 that inputs a first figure, which is a diagrammatic representation of the object, into a trained recognition model for recognition and generates first text data; a first generation unit 111 that generates a second figure in a correctable form based on the first text data; and a second generation unit 121 that generates second text data representing a third figure, which is the corrected second figure, based on instructions from the user.

Description

Technical Field

[0004] ,

[0006] , , , , ,

[0005] , , , , ,

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] There is known a technique for automatically converting document data such as various documents into structured data described in a markup language such as XML (Extensible Markup Language) using AI (Artificial Intelligence) technology (see, for example, Patent Documents 1 and 2).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

[0007] One aspect of the information processing method of the present disclosure includes a recognition step in which a recognition unit recognizes a first figure representing an object in a diagrammatic manner and generates first text data; a first generation step in which a first generation unit generates a second figure in a proofreadable form based on the first text data; and a second generation step in which a second generation unit generates second text data representing a third figure of the second figure after proofreading based on instructions from a user.

[0008] One aspect of the program of this disclosure causes a computer to perform the following steps: a recognition step of recognizing a first figure that graphically represents an object and generating first text data; a first generation step of generating a second figure in a correctable form based on the first text data; and a second generation step of generating second text data that represents a third figure of the corrected second figure based on instructions from a user. [Effects of the Invention]

[0009] According to this disclosure, it is possible to provide an information processing device, an information processing method, and a program that can ensure conversion accuracy while suppressing the occurrence of various costs. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 is a block diagram showing an example of the hardware configuration of the information processing device according to this embodiment. [Figure 2] Figure 2 is a block diagram showing an example of the functional configuration of the information processing device according to this embodiment. [Figure 3] Figure 3 is a diagram showing an example of Figure 1 included in the document data of this embodiment. [Figure 4] Figure 4 shows an example of the recognition result of Figure 1 in this embodiment. [Figure 5] Figure 5 shows an example of the first structured data according to this embodiment. [Figure 6] Figure 6 shows an example of Figure 2 of this embodiment. [Figure 7] Figure 7 shows an example of the display output shown in Figure 2 of this embodiment. [Figure 8] Figure 8 shows another example of the display output shown in Figure 2 of this embodiment. [Figure 9] Figure 9 shows an example of Figure 3 of this embodiment. [Figure 10] Figure 10 shows an example of the second structured data according to this embodiment. [Figure 11] Figure 11 shows an example of the third text data according to this embodiment. [Figure 12] Figure 12 is a flowchart showing an example of processing performed in the information processing device of this embodiment. [Modes for carrying out the invention]

[0011] Hereinafter, embodiments of the present disclosure (hereinafter simply referred to as "these embodiments") will be described in detail with reference to the drawings. However, this disclosure is not limited to the following embodiments. Furthermore, the following embodiments and their variations can be combined as appropriate.

[0012] First, the configuration of the information processing device of this embodiment will be described.

[0013] The information processing apparatus of the present embodiment converts a diagram that graphically represents an object into structured data. Diagrams that graphically represent an object include, but are not limited to, diagrams that graphically represent the flow of processing or the relationships between elements, such as flowcharts and FTA (Fault Tree Analysis) diagrams. Other examples of diagrams that graphically represent an object include, but are not limited to, UML (Unified Modeling Language) diagrams, roadmaps, organizational charts, or Gantt charts. Structured data is data described in a markup language such as XML (Extensible Markup Language). Examples of the information processing apparatus of the present embodiment include, but are not limited to, server apparatuses, PCs (Personal Computers), tablet terminals, or smartphones.

[0014] FIG. 1 is a block diagram showing an example of the hardware configuration of the information processing apparatus 10 of the present embodiment. As shown in FIG. 1, the information processing apparatus 10 includes a control device 11, a main storage device 12, an auxiliary storage device 13, a communication device 14, an input device 15, a display device 16, and various buses 19. The control device 11, the main storage device 12, the auxiliary storage device 13, the communication device 14, the input device 15, and the display device 16 are connected via the various buses 19. Thus, the information processing apparatus 10 of the present embodiment has a general hardware configuration using a normal computer.

[0015] The control device 11 controls the overall operation of the information processing apparatus 10. Examples of the control device 11 include, but are not limited to, at least one of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The number of CPUs and GPUs may be one or more, and they may be single-core or multi-core.

[0016] Examples of the main memory device 12 include, but are not limited to, a ROM (Read Only Memory) and a RAM (Random Access Memory). The ROM stores various programs such as a program for controlling the information processing device 10 and a program for realizing conversion to structured data. The RAM is used as a work area when the control device 11 performs various controls based on the programs stored in the ROM.

[0017] The auxiliary storage device 13 stores various programs described above and various data used to realize conversion to structured data. Note that the various programs described above only need to be stored in at least one of the main memory device 12 and the auxiliary storage device 13. Examples of the auxiliary storage device 13 include, but are not limited to, at least any of existing storage devices that can store magnetically, electrically, or optically, such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), and a DVD (Digital Versatile Disc). The auxiliary storage device 13 may be built into the information processing device 10 or externally attached to the information processing device 10 via an interface such as a USB (Universal Serial Bus). The auxiliary storage device 13 may also be a NAS (Network Attached Storage) connected via a network such as a LAN or a WAN (Wide Area Network).

[0018] The communication device 14 is used to communicate with an external device or the like via a network (not shown). Examples of the communication device 14 include, but are not limited to, a communication device for a wired LAN and a wireless communication device for a wireless LAN.

[0019] The input device 15 is used by the user for various inputs, selections, and specifications, and serves as a user interface between the user and the system. Examples of input devices 15 include, but are not limited to, keyboards, mice, and touch panels. The input device 15 may be built into the information processing device 10 or connected externally to the information processing device 10 via an interface such as USB.

[0020] The display device 16 displays various screens and serves as a user interface between the user and the device. Examples of the display device 16 include, but are not limited to, liquid crystal displays, organic electro-luminescence (OLED) displays, and touch panel displays. The display device 16 may be an internal display built into the information processing device 10, or an external display connected to the information processing device 10 via a display interface such as HDMI®.

[0021] In addition to the above configuration, the information processing device 10 may also be equipped with hardwired circuits such as ICs (Integrated Circuits), ASICs (Application Specific Integrated Circuits), and FPGAs (Field-Programmable Gate Arrays) that are specific to the information processing device 10.

[0022] Figure 2 is a block diagram showing an example of the functional configuration of the information processing device 10 in this embodiment. As shown in Figure 2, the information processing device 10 includes an acquisition unit 101, a recognition unit 103, a first generation unit 111, a display control unit 113, a calibration unit 115, a second generation unit 121, a conversion unit 123, and an output unit 125.

[0023] The acquisition unit 101, recognition unit 103, first generation unit 111, display control unit 113, calibration unit 115, second generation unit 121, conversion unit 123, and output unit 125 can be realized, for example, by the control device 11, main memory 12, and communication device 14 described in Figure 1. The control device 11 reads a program for realizing the conversion to structured data stored in the main memory 12 (ROM) or auxiliary memory 13 and expands it into the main memory 12 (RAM). The control device 11 realizes each of the above-mentioned functional units by executing various processes according to the expanded program. Here, the case in which each of the above-mentioned functional units is realized as software has been explained as an example, but at least a part of each of the above-mentioned functional units may be realized as hardware. In this case, the functional unit to be realized as hardware can be realized, for example, by the hardwired circuit described above. Alternatively, any of the above-mentioned functional units may be realized through the cooperation of software and hardware.

[0024] It should be noted that the information processing device 10 of this embodiment does not necessarily require all of the above-described functional units to be part of its configuration, and at least some of the functional units can be omitted. For example, the information processing device 10 of this embodiment may omit the conversion unit 123.

[0025] The acquisition unit 101 acquires document data that includes Figure 1, which is a diagrammatic representation of the target. The document data can be data from any document. In this embodiment, it is assumed that Figure 1 is inserted into the document data as image data, but it is not limited to this.

[0026] Figure 3 shows an example of Figure 1201 included in the document data of this embodiment. As shown in Figure 3, Figure 1201 is a flowchart showing the process flow from material processing to inspection and assembly. As mentioned above, Figure 1 is image data, and each graphic element and each text element that makes up the flowchart are not recognized as graphics or text, so recognition processing is performed by the recognition unit 103 described later.

[0027] The recognition unit 103 recognizes Figure 1, which is a diagrammatic representation of the object, and generates Figure 1 text data. Specifically, the recognition unit 103 extracts Figure 1 from the document data acquired by the acquisition unit 101, inputs the extracted Figure 1 into a trained recognition model, recognizes the type of each of the multiple geometric elements that make up Figure 1 and the text elements associated with each of those geometric elements, and recognizes the parent-child relationships between the geometric elements from the relationships between the recognized geometric elements.

[0028] A pre-trained recognition model is a model trained to recognize each element constituting a flowchart as a shape or text. In this embodiment, the pre-trained recognition model is assumed to be prepared on an element-by-element basis, but is not limited to this. For example, a first shape recognition model is provided for recognizing shapes other than arrows indicating processing types in a flowchart; a second shape recognition model is provided for recognizing arrows indicating the flow of processing in a flowchart; and a text recognition model is provided for recognizing text within a flowchart. The first and second shape recognition models are generated, for example, by training them with a dataset of training flowcharts in which correct labels are associated with each shape element constituting the flowchart.

[0029] The recognition unit 103 inputs Figure 1 into the first shape recognition model and obtains the type of recognized shape element, the likelihood indicating the accuracy of the recognition result, and coordinate information as recognition results. The recognition unit 103 also inputs Figure 1 into the second shape recognition model and obtains the recognized shape element (arrow), the likelihood indicating the accuracy of the recognition result, and coordinate information as recognition results. The recognition unit 103 also inputs Figure 1 into the text recognition model and obtains the recognized text element, the likelihood indicating the accuracy of the recognition result, and coordinate information as recognition results. The recognition unit 103 integrates these acquired recognition results based on the coordinate information.

[0030] Figure 4 is a diagram showing an example of the recognition result of Figure 1 201 of this embodiment, and mainly shows the recognition result of graphic elements. In the example shown in Figure 4, each graphic element constituting the recognized flowchart is enclosed in a bounding box based on coordinate information, and the recognized type and its likelihood are shown in parentheses. The type is represented by the first three letters, "sta" for start, "dec" for decision, "pro" for process, and "arr" for arrow. In the example shown in Figure 4, the type of graphic element 211, which should indicate the end of processing, is "pro," but this is because a recognition error has occurred, and it should be "end."

[0031] The recognition unit 103 associates each recognized graphic element with each recognized text element based on the coordinate information of each recognized graphic element and the coordinate information of each recognized text element. It also recognizes the parent-child relationships between graphic elements other than arrows based on the direction of the arrows connecting each graphic element other than arrows. In this parent-child relationship between graphic elements, the graphic element on the starting point side of the arrow becomes the parent, and the graphic element on the ending point side of the arrow becomes the child.

[0032] The first text data generated by the recognition unit 103 is text data that represents Figure 2 of Figure 1 in the recognized form, and more specifically, it is first structured data that structures and represents multiple elements constituting Figure 2. In this embodiment, the first structured data is structured data 301 in which multiple elements constituting Figure 2, which is the recognition result of the recognition unit 103, are described in XML (Extensible Markup Language) format as shown in Figure 5, but it is not limited to this.

[0033] The first generation unit 111 generates a proofreadable second figure based on the first text data generated by the recognition unit 103. Specifically, the first generation unit 111 generates a proofreadable second figure represented by the first text data from the first text data by drawing according to the markup of the structured data 301, which is the first text data.

[0034] FIG. 6 is a diagram showing an example of the second FIG. 401 of the present embodiment. As described above, the second FIG. 401 is generated by reversely converting the structured data 301, which is the recognition result of the first FIG. 201, into a diagram from the recognition result. Note that, unlike the first FIG. 201, each element constituting the second FIG. 401 is recognized as a graphic or text, so it is in a form that can be corrected.

[0035] Since the second FIG. 401 shown in FIG. 6 is generated based on the structured data 301, although the description format is different from that of the first FIG. 201, it has the same content as the first FIG. 201 except for the recognition error part. Specifically, in the second FIG. 401, due to a recognition error, the type (shape) of the graphic element 411 is the shape of pro, which is different from the shape of end in the first FIG. 201. Similarly, in the second FIG. 401, due to a recognition error, the text element 421 is the katakana character "エ", which is different from the Chinese character "工" in the first FIG. 201.

[0036] In the present embodiment, in order to allow the user to correct such a recognition error, the second FIG. 401 is displayed and output.

[0037] The display control unit 113 displays the second FIG. generated by the first generation unit 111 on the display device 16. For example, the display control unit 113 displays the first FIG. and the second FIG. on the display device 16 in a manner that allows them to be compared.

[0038] FIG. 7 is a diagram showing an example of the display output of the second FIG. 401 of the present embodiment, and it is a display screen 501 on which the first FIG. 201 and the second FIG. 401 are displayed side by side so that they can be compared. In the present embodiment, by displaying the graphic elements of both figures in an easy-to-compare manner as described above, it is easier for the user to discover the recognition errors of the graphic element 411 and the text element 421 described above, and it supports the correction work by the user.

[0039] Note that the display control unit 113 may be configured to display the differences between FIG. 1 and FIG. 2 in an identifiable manner. FIG. 8 is a diagram showing another example of the display output of FIG. 401 of the second figure in the present embodiment. Not only are FIG. 201 of the first figure and FIG. 401 of the second figure displayed side by side, but also a display screen 511 is shown in which graphic elements 411 and text elements 421, which are the recognition error locations of FIG. 401, are highlighted. By adopting the display mode as shown in FIG. 8, it is possible to make it easier for the user to discover the recognition error locations, and since computer checking is added, a double-check function with the user's check works. Therefore, a further improvement in the conversion accuracy from the first figure to the second figure can be expected. Note that the highlighting mode may be any mode as long as it is a mode in which the recognition error location is emphasized, such as shading display, highlighting display, color change, or marking.

[0040] The calibration unit 115 calibrates the second figure based on the user operation. Specifically, the user discovers the recognition error location by checking the above-described display screen 501 or 511, and performs an operation input for correcting the discovered recognition error location using the input device 15. The calibration unit 115 calibrates the second figure based on this operation input, and sets the calibrated second figure as the third figure.

[0041] FIG. 9 is a diagram showing an example of the third figure 601 of the present embodiment. In the third figure 601 shown in FIG. 9, the type (shape) of the graphic element 611 is calibrated from the pro shape to the end shape, and the text element 621 is calibrated from the katakana character "エ" to the kanji character "工". As a result, the third figure 601 has the same content as the first figure 201.

[0042] The second generation unit 121 generates second text data that represents Figure 3 of Figure 2 after correction, based on instructions from the user. The second text data is text data that represents Figure 3 of Figure 2 after correction, and more specifically, it is second structured data that structures and represents multiple elements that constitute Figure 3. In this embodiment, the second structured data is structured data 701 in which multiple elements that constitute Figure 3, which is the correction result of the correction unit 115, are described in XML (Extensible Markup Language) format as shown in Figure 10, but it is not limited to this.

[0043] The conversion unit 123 converts the second structured data, which is the second text data generated by the second generation unit 121, into third text data in a different form from the structured data. The format of the third text data can be determined according to its intended use. For example, if the third text data is to be used to build a database such as an RDB (Relational Database), the format of the third text data can be set to CSV (Comma Separated Values) format.

[0044] Figure 11 shows an example of the third text data 801 of this embodiment. In the example shown in Figure 11, the contents of the third text data 801 are shown in a table format for easier understanding. The third text data 801 is a structured representation of multiple elements that make up Figure 3, and includes node ID, node type, text, parent ID, and child ID.

[0045] The node ID is the ID of the graphic element other than the arrows that make up Figure 3. The node type is the type of graphic element indicated by the node ID. The text is the text element associated with the graphic element indicated by the node ID. The parent ID is the node ID of the graphic element that is the parent of the graphic element indicated by the node ID. The child ID is the node ID of the graphic element that is a child of the graphic element indicated by the node ID.

[0046] The output unit 125 outputs the third text data converted by the conversion unit 123.

[0047] Next, the operation of the information processing device of this embodiment will be described.

[0048] Figure 12 is a flowchart showing an example of processing performed by the information processing device 10 in this embodiment.

[0049] First, the acquisition unit 101 acquires document data including Figure 1, which is a diagrammatic representation of the target (step S101).

[0050] Next, the recognition unit 103 extracts Figure 1 from the document data acquired by the acquisition unit 101, recognizes the extracted Figure 1, and generates first text data that represents Figure 2 as Figure 1 in the recognized form (step S103).

[0051] Next, the first generation unit 111 generates a second figure in a rectifiable form based on the first text data generated by the recognition unit 103 (step S105).

[0052] Next, the display control unit 113 displays Figure 1 and Figure 2 side by side on the display device 16 so that they can be compared (step S107).

[0053] Next, the calibration unit 115 calibrates Figure 2 based on user operation (step S109).

[0054] Next, the second generation unit 121 generates second text data that represents Figure 3 of Figure 2 after correction, based on instructions from the user (step S111).

[0055] Next, the conversion unit 123 converts the second structured data, which is the second text data generated by the second generation unit 121, into third text data in a different form from the structured data (step S113).

[0056] Next, the output unit 125 outputs the third text data converted by the conversion unit 123 (step S115).

[0057] As described above, in this embodiment, a diagram representing the subject is automatically converted into structured data, the converted structured data is reverse-converted and the resulting diagram is used for manual proofreading, and the proofread diagram is converted back into structured data. In this embodiment, since the diagram representing the subject is automatically converted into structured data, the occurrence of various costs such as time, labor, and expenses can be reduced. Furthermore, in this embodiment, conversion errors (recognition errors) associated with the automatic conversion are corrected by manual proofreading, so the accuracy of the conversion can be ensured.

[0058] In particular, in this embodiment, manual proofreading is performed using a diagram obtained by reverse-transforming the converted structured data. This allows users to perform proofreading on a visualized diagram, reducing the time, effort, and costs associated with proofreading, and thus streamlining the proofreading process.

[0059] Furthermore, in this embodiment, the original drawing (Figure 1) and the reverse-converted drawing (Figure 2) are displayed side-by-side, making it easier for the user to find conversion errors (recognition errors). This further reduces the various costs associated with proofreading, such as time, effort, and expense, and makes the proofreading process more efficient.

[0060] Furthermore, when constructing an RDB by converting flowcharts, FTA diagrams, etc., into structured data, even a single conversion error (recognition error) can result in the RDB being constructed in the exact opposite way, so the required conversion accuracy is inevitably high. For this reason, according to this embodiment, even when converting to structured data that requires such high accuracy, it is possible to ensure conversion accuracy while suppressing the occurrence of various costs.

[0061] (program) The program executed by the information processing device of the above embodiment is provided in an installable or executable file format stored on a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD, or flexible disk (FD).

[0062] Furthermore, the program executed by the information processing device of the above embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading it via the network. Alternatively, the program executed by the information processing device of the above embodiment may be provided or distributed via a network such as the Internet. Alternatively, the program executed by the information processing device of the above embodiment may be pre-installed in ROM or the like and provided.

[0063] The program executed by the information processing device of the above embodiment has a modular configuration for realizing each of the above-described parts on a computer. In actual hardware, for example, the CPU reads the program from the HDD into RAM and executes it, thereby realizing each of the above-described parts on the computer.

[0064] The embodiments described above are merely examples of how this disclosure may be implemented, and they do not limit the technical scope of this disclosure. Therefore, this disclosure can be implemented in various ways without departing from its essence or its main features. For example, the embodiments described above may be combined as appropriate on a component basis. Also, for example, some components may be removed from the total components in the embodiments described above.

[0065] This disclosure also includes the following aspects:

[0066] (1) A recognition unit that recognizes Figure 1, which represents the object in a diagrammatic form, and generates first text data, A first generation unit generates a second figure in a rectifiable form based on the first text data, A second generation unit generates second text data representing Figure 3 of Figure 2 after correction, based on instructions from the user, An information processing device equipped with the following features.

[0067] (2) Figure 2 above is Figure 1 above in the recognized form, The first text data represents the second figure, A display control unit that displays the above Figure 2, The system further includes a calibration unit that calibrates the aforementioned Figure 2 based on user operations, The information processing device described in (1) above.

[0068] (3) The display control unit displays Figure 1 and Figure 2 in a manner that allows for comparison. The information processing device described in (1) above.

[0069] (4) The display control unit displays the differences between Figure 1 and Figure 2 in an identifiable manner. The information processing device described in (3) above.

[0070] (5) The first text data is first structured data that structures and represents the multiple elements that constitute the second figure, The aforementioned second text data is second structured data that structures and represents multiple elements constituting the aforementioned third figure. The information processing device described in (2) above.

[0071] (6) Further comprising a conversion unit that converts the second structured data into a third text data in a different form from the structured data, The information processing device described in (5) above.

[0072] (7) The recognition unit inputs the first figure into a trained recognition model to recognize the type of each of the multiple geometric elements constituting the first figure and the text elements associated with each of the multiple geometric elements, and recognizes the parent-child relationships between the geometric elements from the relationships between the recognized geometric elements. The information processing device described in (1) above.

[0073] (8) Recognition step in which the recognition unit recognizes Figure 1, which represents the object in a diagrammatic form, and generates first text data, The first generation unit performs a first generation step of generating a second figure in a correctable form based on the first text data, The second generation step involves the second generation unit generating second text data that represents the third figure of the second figure after correction, based on instructions from the user, Information processing methods including

[0074] (9) A recognition step that recognizes Figure 1, which represents the object in a diagrammatic form, and generates first text data, A first generation step of generating a second figure in a correctable form based on the first text data, A second generation step involves generating second text data that represents Figure 3 of Figure 2 after correction, based on instructions from the user, A program that causes a computer to execute something. [Explanation of Symbols]

[0075] 10 Information Processing Devices 101 Acquisition Department 103 Recognition part 111 1st generation part 113 Display Control Unit 115 Calibration Department 121 Second generation part 123 Conversion section 125 Output section

Claims

1. A recognition unit recognizes Figure 1, which represents the subject in a diagrammatic form, and generates first text data. A first generation unit generates a second figure in a calibrable form based on the first text data, A second generation unit generates second text data that represents the third figure of the second figure after correction, based on instructions from the user, An information processing device equipped with the following features.

2. Figure 2 is Figure 1 of the recognized embodiment, The first text data represents the second figure, A display control unit that displays the aforementioned Figure 2, The system further comprises a calibration unit that calibrates the aforementioned Figure 2 based on user operations, The information processing apparatus according to claim 1.

3. The display control unit displays Figure 1 and Figure 2 in a manner that allows for comparison. The information processing apparatus according to claim 2.

4. The display control unit displays the differences between Figure 1 and Figure 2 in a way that makes them identifiable. The information processing apparatus according to claim 3.

5. The first text data is first structured data that structures and represents multiple elements constituting the second figure, The second text data is a second structured data that structures and represents multiple elements constituting the third figure. The information processing apparatus according to claim 2.

6. The system further includes a conversion unit that converts the second structured data into a third text data that is different in nature from the structured data. The information processing apparatus according to claim 5.

7. The recognition unit inputs the first figure into a trained recognition model, recognizes the type of each of the multiple geometric elements constituting the first figure and the text elements associated with each of the multiple geometric elements, and recognizes the parent-child relationships between the geometric elements from the relationships between the recognized geometric elements. The information processing apparatus according to claim 1.

8. The recognition unit recognizes Figure 1, which represents the object in a diagrammatic form, and generates Figure 1, which is a recognition step. The first generation unit generates a second figure in a correctable form based on the first text data in a first generation step, The second generation step involves the second generation unit generating second text data that represents the third figure of the second figure after correction, based on instructions from the user, Information processing methods including

9. A recognition step that recognizes Figure 1, which represents the subject in a diagrammatic form, and generates first text data, A first generation step of generating a second figure in a correctable form based on the first text data, A second generation step involves generating second text data that represents the third figure of the second figure after correction, based on instructions from the user, A program that causes a computer to execute something.

Images