Computer system and graph recognition method

Abstract

To accomplish graph recognition processing which reduces an input burden on a user and outputs a highly precise processing result.SOLUTION: A computer system executes graph recognition processing comprising multiple pieces of block processing. The block processing includes evaluation block processing for evaluating a processing result, and the evaluation block processing can obtain the processing result and a certainty factor thereof as outputs. The computer system that executes the graph recognition processing obtains the processing result and the certainty factor when having executed the evaluation block processing, determines whether or not to continue the graph recognition processing based on the certainty factor, presents an interface to receive a correction on the processing result by the evaluation block processing when determining not to continue the graph recognition processing, and executes the block processing subsequent to the evaluation block processing when receiving an input from the user via the interface.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a technology for recognizing graphs included in documents.

Background Art

[0002] Operations performed by humans, such as the operation of entering the description content of a document into a database, are being automated with the development of character recognition technology.

[0003] In recent years, attempts have also been made to automatically create a database of the description content of graphs included in contents such as documents and images by utilizing character recognition processing. In order to automatically create a database of the description content of graphs included in contents, it is necessary to recognize the character string in the area where the graph is described using character recognition processing, identify the axes and scales, and recognize the plot elements.

[0004] Here, the plot element represents a symbol arranged in the graph, such as a bar in the case of a bar graph, a line in the case of a straight line and curve graph, and symbols such as circles and triangles in the case of a scatter plot. Also, a group of plot elements for the same object is called a data series.

[0005] In the following description, the character string in the area where the graph is included in the content, the axes and scales of the graph, and the plot elements are described as graph elements. Also, the process of recognizing graph elements, reproducing a data group for depicting the graph based on the arrangement relationship of the graph elements, and extracting characteristic numerical values and trends of the graph is described as graph recognition processing.

[0006] ]In order to implement graph recognition processing, a method using character recognition processing and object detection processing has been studied. For example, Patent Document 1 describes "an image input means for inputting graph information as an image, a window sensor means having a grid for extracting the graph information input by this image input means, and using this window sensor means to extract the intersection of the graph information and the grid of the window sensor means as an extraction output. A graph extraction means for extracting, a graph recognition means for recognizing the extraction output extracted by this graph extraction means as a graph, and a storage means for storing the recognition result recognized by this graph recognition means. A data reading device characterized by comprising."

[0007] By using the technology described in Patent Document 1, a data group for depicting a graph from an image can be reproduced.

Prior Art Documents

Patent Documents

[0008]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0009] The technology described in Patent Document 1 can be applied to a graph in which grid lines or the like exist in the plot area and numerical values can be obtained from the intersections with plot elements. Therefore, there are limitations on the applicable graphs.

[0010] For graphs without grid lines, graphs with multiple data series, etc., since the freedom of graph depiction is high, it is difficult to completely recognize various graphs. Therefore, when the graph recognition processing fails, the user needs to manually input the graph information.

[0011] An object of the present disclosure is to provide a system and method for realizing a graph recognition process that reduces the user's input burden and outputs a highly accurate processing result. [Means for solving the problem]

[0012] A representative example of the invention disclosed in this application is as follows: a computer system that accepts an image including a graph as input and performs a graph recognition process consisting of a plurality of block processes, comprising at least one computer, wherein the plurality of block processes are A first block process to determine the type of graph, a second block process to extract a string, a third block process to obtain the graph axes based on the processing result of the second block process, a fourth block process to recognize the plot elements of the graph based on the processing result of the third block process, a fifth block process to obtain the plot values ​​of the graph based on the processing results of the third and fourth block processes, a sixth block process to associate the plot values ​​of the graph with data series based on the result of the fifth block process, a seventh block process to obtain information representing the characteristics of the graph, and any of the first to seventh block processes. Evaluate the processing results. at least one Evaluation block processing and, including The evaluation block processing is, Any of the first to seventh block processes that are subject to the evaluation block process Processing results and The The confidence level of the processing result can be obtained as an output, and at least one computer that performs the graph recognition process executes the evaluation block process, Any of the first to seventh block processes that are subject to the evaluation block process Processing results and The Obtain the confidence level of the processing result, Any of the first to seventh block processes that are subject to the evaluation block process Based on the confidence level of the processing result, it is determined whether or not to continue the graph recognition process, and if it is determined not to continue the graph recognition process, the evaluation block process Any of the first to seventh block processing processes that are subject to this An interface is provided for accepting modifications to the processing results, and when user input is received through the interface, the user input and the evaluation block processing are used. Any of the first to seventh block processing processes that are subject to this The block processing after the evaluation block processing is executed using the processing result of the block processing executed before the evaluation block processing. [Effects of the Invention]

[0013] According to one embodiment of the present invention, it is possible to realize graph recognition processing that reduces the user's input burden and outputs highly accurate processing results. Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments. [Brief explanation of the drawing]

[0014] [Figure 1] This diagram illustrates the hardware and software configuration of the computer in Example 1. [Figure 2] This figure shows an example of label information for Example 1. [Figure 3A] This is a flowchart for explaining an example of the graph recognition process executed by the computer of Example 1. [Figure 3B] This is a flowchart for explaining an example of the graph recognition process executed by the computer of Example 1. [Figure 3C] This is a flowchart for explaining an example of the graph recognition process executed by the computer of Example 1. [Figure 4] This is a flowchart for explaining an example of the evaluation process executed by the computer of Example 1. [Figure 5A] This is a diagram showing an example of the GUI presented by the computer of Example 1 to the user. [Figure 5B] This is a diagram showing an example of the GUI presented by the computer of Example 1 to the user. [Figure 5C] This is a diagram showing an example of the GUI presented by the computer of Example 1 to the user. [Figure 5D] This is a diagram showing an example of the GUI presented by the computer of Example 1 to the user. [Figure 5E] This is a diagram showing an example of the GUI presented by the computer of Example 1 to the user.

Mode for Carrying Out the Invention

[0015] Hereinafter, Example 1 of the present invention will be described by referring to the drawings in order.

[0016] In the drawings for explaining the examples, the same reference numerals are given to portions having the same functions, and repeated explanations thereof are omitted. Note that the examples described below do not limit the invention according to the claims. Also, not all of the elements and their combinations described in the examples are essential for the solution means of the invention.

[0017] Furthermore, in the following explanation, expressions such as "xxx data" may be used as an example of information, but the data structure of the information can be anything. In other words, to show that the information is independent of its data structure, "xxx data" can be called "xxx table". Also, in the following explanation, the structure of each piece of information is just an example, and the information may be stored separately or combined. [Examples]

[0018] Figure 1 illustrates the hardware and software configuration of the computer in Example 1.

[0019] Computer 100 performs graph recognition processing. Computer 100 presents a GUI for receiving user input during graph recognition processing. Graph recognition processing includes multiple block processes. Each block process includes at least one block process in which the processing result is evaluated. In the following description, the block process in which the processing result is evaluated will be referred to as the evaluation block process.

[0020] The graph recognition process in Example 1 includes graph type discrimination, string recognition, axis acquisition, plot element recognition, plot numerical value acquisition, plot numerical value mapping, and graph-specific information acquisition.

[0021] Computer 100 includes a processor 101, an input device 102, an output device 103, a main memory 104, a secondary memory 105, and a network interface 106. Each hardware element is connected to the others via an internal bus or the like. In Figure 1, there is one of each hardware element, but there may be two or more. The type of network to connect is not limited. Data can be sent and received or processing can be shared with other computers and memory devices via a network or direct connection.

[0022] The processor 101 executes a program stored in the main memory 104. The processor 101 implements a specific function by performing processing according to the program. In the following description, when a program or a module implemented by the program is the subject of a processing statement, it indicates that the processor 101 is executing the program.

[0023] The input device 102 is a device for inputting data to the computer 100. For example, the input device 102 includes devices for operating the computer, such as a keyboard, mouse, and touch panel. The input device 102 also includes devices for acquiring images, such as a scanner, digital camera, and smartphone.

[0024] The output device 103 is a device that outputs data input screens and processing results, etc. The output device 103 includes a touch panel and a display, etc.

[0025] The main memory 104 stores the program executed by the processor 101 and the information used by the program. The main memory 104 also includes a work area that the program uses temporarily. The main memory 104 could be, for example, memory.

[0026] In Embodiment 1, the main memory 104 stores a graph recognition program 110, which consists of a graph type discrimination module 111, a string detection / recognition module 112, an axis acquisition module 113, a plot element detection / recognition module 114, a plot numerical value acquisition module 115, a mapping module 116, and a graph-specific information acquisition module 117. The main memory 104 may also store programs on a module-by-module basis. The computer 100 loads the module programs and information into the main memory 104 as needed.

[0027] Furthermore, the main memory 104 stores the label information 120. Details of the label information 120 are explained in Figure 2.

[0028] The secondary storage device 105 permanently stores data. The secondary storage device 105 could be, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive). Note that programs and information stored in the main memory 104 may also be stored in the secondary storage device 105. In this case, the processor 101 reads the programs and information from the secondary storage device 105 and loads them into the main memory 104.

[0029] Figure 2 shows an example of label information 120 from Example 1.

[0030] Label information 120 is information for managing user input for evaluation block processing as label data. Label data can be used in machine learning of the model used for block processing.

[0031] The label data includes graph image ID 201, graph type 202, axis 203, plot element 204, and correspondence 205.

[0032] Graph Image ID 201 is a field that stores identification information to uniquely identify the graph image on which graph recognition processing has been performed. For example, Graph Image ID 201 may store a number, file name, etc.

[0033] Graph type 202 is a field that stores graph type information entered by the user. Axis 203 is a field that stores axis information entered by the user. Plot element 204 is a field that stores plot element information entered by the user. This field stores mapping information for plotted values ​​entered by the user.

[0034] Fields in evaluation block processing that do not require user input will be left blank.

[0035] Computer 100 accepts a document containing graphs as input. The document is input as image data. Computer 100 recognizes the areas where graphs are written by performing known image recognition processing on the pages (images) that make up the document. Computer 100 stores the recognition results as graph images in the work area. Computer 100 performs graph recognition processing for each graph image. If multiple graphs are written on a page, a graph image for each graph will be output.

[0036] Figures 3A, 3B, and 3C are flowcharts illustrating an example of the graph recognition process performed by the computer 100 in Example 1. In Example 1, the graph type discrimination process, axis acquisition process, plot element recognition process, and plot numerical mapping process are assumed to be evaluation block processes.

[0037] The graph recognition program 110 selects the graph image to be processed from the work area (step S301).

[0038] The graph type discrimination module 111 of the graph recognition program 110 performs graph type discrimination processing (step S302). Here, graph type represents the classification of graphs such as bar graphs and scatter plots. Graph type discrimination processing can be implemented using known methods such as object recognition methods using deep learning. The graph type discrimination processing outputs the discrimination result and the confidence level of the discrimination. Here, confidence level is a numerical value such as probability that represents the reliability of the processing result. For example, in the case of multi-class classification using a deep learning-based method, the confidence level is calculated as the probability of belonging to the largest class as the final result of the processing. In the case of block processing where the processing result is output as 0 (no output) or 1 (data), these values ​​are calculated as confidence levels. A model that calculates confidence levels using the processing result as input may also be used.

[0039] The graph recognition program 110 performs an evaluation process for the graph type discrimination process (step S303). Details of the evaluation process will be explained using Figure 4.

[0040] The graph recognition program 110 determines whether to continue the graph recognition process based on the results of the evaluation process (step S304). In other words, it determines whether user input is required.

[0041] If it is determined that the graph recognition process should continue, the graph recognition program 110 proceeds to step S305.

[0042] If it is determined that the graph recognition process should not be continued, the graph recognition program 110 displays a GUI for accepting user input (step S318). The GUI displays a page containing the graph image, along with the processing results and confidence level of the graph type determination process. Details of the GUI are explained in Figure 5A. The user modifies the processing results of the graph type determination process or inputs extracted information via the GUI. The extracted information is information about the graph desired by the user. The extracted information includes all the information necessary for the graph, such as the graph type, axes, plot elements, and the correspondence of plotted values.

[0043] The graph recognition program 110 determines whether or not it has accepted the modification to the graph type determination process (step S319). If it has accepted the input of extracted information, the graph recognition program 110 determines that it has not accepted the modification to the graph type determination process.

[0044] If the graph type determination process has not been modified, the graph recognition program 110 associates the graph image with the extracted information received from the user and saves it in the sub-memory 105, then terminates the graph recognition process.

[0045] If a correction to the graph type determination process is accepted, the graph recognition program 110 updates the label information 120 (step S320), and then proceeds to step S305.

[0046] Specifically, the graph recognition program 110 generates label data in which the identification information of the selected graph image is set to the graph image ID 201. The graph recognition program 110 sets the graph type information entered by the user to the graph type 202 of the label data and registers it in the label information 120. In addition, the graph recognition program 110 sets the maximum possible value as the confidence level of the processing result of the graph type discrimination process.

[0047] In step S305, the string detection / recognition module 112 of the graph recognition program 110 performs string recognition processing (step S305). Specifically, the string detection / recognition module 112 detects strings from the graph image using known methods such as Faster R-CNN, and identifies the strings using known methods such as recognition methods using CNN and RNN. In the string recognition processing, the strings and their arrangement are obtained as processing results.

[0048] The axis acquisition module 113 of the graph recognition program 110 performs axis acquisition processing to acquire the graph axes based on the processing results of the string recognition process (step S306). For example, the axis acquisition module 113 recognizes the axes and tick marks using rules based on the arrangement of strings or an object detection method. In the axis acquisition processing, the processing result and the confidence level of the processing result are obtained as output.

[0049] The graph recognition program 110 performs an evaluation process for the axis acquisition process (step S307). The evaluation process is the same as the process in step S303.

[0050] The graph recognition program 110 determines whether or not to continue the graph recognition process based on the results of the evaluation process (step S308).

[0051] If it is determined that the graph recognition process should continue, the graph recognition program 110 proceeds to step S309.

[0052] If it is determined that the graph recognition process should not be continued, the graph recognition program 110 displays a GUI for accepting user input (step S321). The GUI displays a page containing the graph image, along with the processing results and confidence level of the axis acquisition process. Details of the GUI are explained in Figure 5B. The user modifies the processing results of the axis acquisition process or inputs extracted information via the GUI.

[0053] The graph recognition program 110 determines whether or not it has accepted the modification to the axis acquisition process (step S322). If it has accepted the input of extracted information, the graph recognition program 110 determines that it has not accepted the modification to the axis acquisition process.

[0054] If the axis acquisition process has not been modified, the graph recognition program 110 associates the graph image with the extracted information received from the user and saves it in the sub-memory 105, then terminates the graph recognition process.

[0055] If the graph recognition program 110 accepts a modification to the axis acquisition process, it updates the label information 120 (step S323), and then proceeds to step S309.

[0056] Specifically, the graph recognition program 110 refers to the label information 120 and determines whether label data with the identification information of the selected graph image is registered for the graph image ID 201. If label data is registered, the graph recognition program 110 sets the axis information entered by the user for the axis 203 of the label data. If label data is not registered, the graph recognition program 110 generates label data with the identification information of the selected graph image for the graph image ID 201. The graph recognition program 110 sets the axis information entered by the user for the axis 203 of the label data and registers it in the label information 120. In addition, the graph recognition program 110 sets the maximum possible value as the confidence level of the processing result of the axis acquisition process.

[0057] In step S309, the plot element detection / recognition module 114 of the graph recognition program 110 performs plot element recognition processing (step S309). For example, the plot element detection / recognition module 114 identifies a plot area based on axis information and recognizes the plot elements contained within the plot area. Plot elements can be recognized using rules or object detection methods. In the plot element recognition processing, the processing result and the confidence level of the processing result are obtained as output.

[0058] The graph recognition program 110 performs an evaluation process for the plot element recognition process (step S310). The evaluation process is the same as the process in step S303.

[0059] The graph recognition program 110 determines whether or not to continue the graph recognition process based on the results of the evaluation process (step S311).

[0060] If it is determined that the graph recognition process should continue, the graph recognition program 110 proceeds to step S312.

[0061] If it is determined that the graph recognition process should not be continued, the graph recognition program 110 displays a GUI for accepting user input (step S324). The GUI displays a page containing the graph image, along with the processing results and confidence level of the plot element recognition process. Details of the GUI are explained in Figure 5C. The user can modify the processing results of the plot element recognition process or input extracted information via the GUI.

[0062] The graph recognition program 110 determines whether or not it has accepted a modification to the plot element recognition process (step S325). If it has accepted input of extracted information, the graph recognition program 110 determines that it has not accepted a modification to the plot element recognition process.

[0063] If the graph recognition program 110 does not accept any modifications to the plot element recognition process, it associates the graph image with the extracted information received from the user and saves it in the sub-memory 105, then terminates the graph recognition process.

[0064] If the graph recognition program 110 accepts a modification to the plot element recognition process, it updates the label information 120 (step S326), and then proceeds to step S312.

[0065] Specifically, the graph recognition program 110 refers to the label information 120 and determines whether label data with the identification information of the selected graph image is registered for the graph image ID 201. If label data is registered, the graph recognition program 110 sets the plot element information entered by the user in the plot element 204 of the label data. If label data is not registered, the graph recognition program 110 generates label data with the identification information of the selected graph image set for the graph image ID 201. The graph recognition program 110 sets the plot element information entered by the user in the plot element 204 of the label data and registers it in the label information 120. In addition, the graph recognition program 110 sets the maximum possible value as the confidence level of the processing result of the plot element recognition process.

[0066] In step S312, the plot value acquisition module 115 of the graph recognition program 110 executes the plot value acquisition process (step S312). Specifically, the plot value acquisition module 115 converts the position of the plot elements within the plot area into axis values ​​based on axis information and plot element information. For example, a method of detecting intersections with virtual grid lines can be considered.

[0067] The mapping module 116 of the graph recognition program 110 performs plotted numerical mapping processing (step S313). For example, the mapping module 116 maps data series to plotted numerical values, or maps the names of data series to plotted numerical values. In the plotted numerical mapping processing, the processing result and the confidence level of the processing result are obtained as output.

[0068] The graph recognition program 110 performs an evaluation process for the plotted numerical data mapping process (step S314). The evaluation process is the same as the process in step S303.

[0069] The graph recognition program 110 determines whether or not to continue the graph recognition process based on the results of the evaluation process (step S315).

[0070] If it is determined that the graph recognition process should continue, the graph recognition program 110 proceeds to step S316.

[0071] If it is determined that the graph recognition process should not be continued, the graph recognition program 110 displays a GUI for accepting user input (step S327). The GUI displays a page containing the graph image, along with the processing results and confidence level. Details of the GUI are explained in Figure 5D. The user modifies the processing results of the plot numerical correspondence process or inputs extracted information via the GUI.

[0072] The graph recognition program 110 determines whether or not it has accepted a modification to the plotted numerical mapping process (step S328). If it has accepted the input of extracted information, the graph recognition program 110 determines that it has not accepted a modification to the plotted numerical mapping process.

[0073] If no modification to the plotted numerical data mapping process is accepted, the graph recognition program 110 associates the graph image with the extracted information received from the user and saves it in the sub-memory 105, then terminates the graph recognition process.

[0074] If a modification to the plot numerical mapping process is accepted, the graph recognition program 110 updates the label information (step S329), and then proceeds to step S316.

[0075] Specifically, the graph recognition program 110 refers to the label information 120 and determines whether label data with the identification information of the selected graph image is registered to the graph image ID 201. If label data is registered, the graph recognition program 110 sets the mapping information entered by the user to the mapping 205 of the label data. If label data is not registered, the graph recognition program 110 generates label data with the identification information of the selected graph image to the graph image ID 201. The graph recognition program 110 sets the mapping information entered by the user to the mapping 205 of the label data and registers it in the label information 120. In addition, the graph recognition program 110 sets the maximum possible value as the confidence level of the processing result of the plot numerical mapping process.

[0076] In step S316, the graph-specific information acquisition module 117 of the graph recognition program 110 executes the graph-specific information acquisition process (step S316). For example, the graph-specific information acquisition module 117 acquires information that represents the characteristics of the graph, such as the yield stress in the stress-strain diagram.

[0077] The graph recognition program 110 stores the graph image and a series of recognition results in the sub-memory 105, presents the recognition results to the user (step S317), and then terminates the graph recognition process. The graph recognition program 110 may accept corrections to the recognition results.

[0078] Figure 4 is a flowchart illustrating an example of the evaluation process performed by the computer 100 in Example 1.

[0079] The graph recognition program 110 obtains the confidence level of the evaluation block processing (step S401) and calculates the cumulative confidence level (step S402).

[0080] Specifically, the graph recognition program 110 calculates the cumulative confidence score using the confidence scores of evaluation block processes executed before the evaluation block process to be processed and the confidence score of the evaluation block process to be processed. For example, the cumulative confidence score may be calculated as the sum of the confidence scores, a weighted sum of the confidence scores, a product of the confidence scores, or the average of the confidence scores.

[0081] The confidence level is a value that represents the accuracy of the output of the evaluation block process, and the cumulative confidence level is a value that represents the accuracy of the output of all block processes up to the evaluation block process being evaluated.

[0082] The graph recognition program 110 determines whether the acquired confidence level is greater than or equal to the threshold T1 (step S403). The threshold T1 may be different for each evaluation block process, or it may be a common value for all evaluation block processes.

[0083] If the acquired confidence level is less than the threshold T1, the graph recognition program 110 outputs "Processing interrupted" as the judgment result (step S406) and terminates the evaluation process.

[0084] If the acquired confidence level is equal to or greater than the threshold T1, the graph recognition program 110 determines whether the cumulative confidence level is equal to or greater than the threshold T2 (step S404).

[0085] If the cumulative confidence level is less than the threshold T2, the graph recognition program 110 outputs "Processing interrupted" as the judgment result (step S406) and terminates the evaluation process.

[0086] If the cumulative confidence level is equal to or greater than the threshold T2, the graph recognition program 110 outputs "Continue processing" as the judgment result (step S405) and terminates the evaluation process.

[0087] By setting a low threshold T1 for the confidence level of the evaluation block processing, the number of user inputs due to interruptions in the graph recognition process can be reduced while maintaining accuracy. Furthermore, by making a decision using the cumulative confidence level, the computer 100 can determine whether to continue the graph recognition process, taking into account the overall accuracy of the graph recognition process.

[0088] Figures 5A, 5B, 5C, 5D, and 5E show examples of GUIs presented to the user by the computer 100 of Embodiment 1.

[0089] GUI500 includes an image display area 501 and an editing area 502. The image display area 501 displays a page containing a graph image, the confidence level of the evaluation block processing, and the cumulative confidence level. The confidence levels of evaluation block processing performed before the evaluation block processing being evaluated may also be displayed. Furthermore, when displaying the final processing result, the confidence level of the evaluation block processing does not need to be displayed. The editing area 502 displays the evaluation block processing being evaluated and the recognition results from block processing performed before the evaluation block processing being evaluated. The editing area 502 is editable by the user, allowing for modifications to the recognition results, input of extracted information, etc.

[0090] Figure 5A shows the GUI 500 displayed when the process is determined to be interrupted in step S304. The user can use the GUI 500 to make modifications such as adding or deleting graph types, or to input extracted information.

[0091] Figure 5B shows the GUI 500 displayed when processing is determined to be interrupted in step S308. The user can correct the axes or input extraction information via the GUI 500. Alternatively, instead of inputting values ​​related to the axes, the system may be configured to correct the axes by specifying the axis portion of the image display area 501.

[0092] Figure 5C shows the GUI 500 displayed when processing is determined to be interrupted in step S311. The user can modify plot elements or input extracted information via the GUI 500. Alternatively, instead of inputting values ​​for plot elements, the system may be configured to allow modification of plot elements by specifying plot elements in the image display area 501.

[0093] Figure 5D shows the GUI 500, which is displayed when the process is determined to be interrupted in step S315. The user can use the GUI 500 to correct the mapping of plotted values ​​to data series or to input extracted information.

[0094] Figure 5E shows the GUI500 displayed in step S317.

[0095] Furthermore, GUI500 may display the processing results of block processes that have been executed at the time GUI500 is displayed.

[0096] By displaying the confidence level and cumulative confidence level, it is possible to identify evaluation block processes with poor accuracy, even if the overall accuracy of the graph recognition process is maintained.

[0097] The embodiments described above are detailed explanations of the configuration in order to clearly illustrate the present invention, and are not necessarily limited to those comprising all the configurations described. Furthermore, some of the configurations in each embodiment can be added to, deleted from, or replaced with other configurations.

[0098] Furthermore, each of the above-mentioned configurations, functions, processing units, processing means, etc., may be implemented in hardware, in whole or in part, for example, by designing them as integrated circuits. The present invention can also be implemented by software program code that realizes the functions of the embodiment. In this case, a storage medium on which the program code is recorded is provided to a computer, and the processor of that computer reads the program code stored in the storage medium. In this case, the program code read from the storage medium itself realizes the functions of the embodiment described above, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media used to supply such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs (Solid State Drives), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, and the like.

[0099] Furthermore, the program code that implements the functions described in this embodiment can be implemented in a wide range of programming or scripting languages, such as assembler, C / C++, Perl, Shell, PHP, and Java (registered trademark).

[0100] Furthermore, the program code for the software that implements the functions of the embodiment may be distributed via a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the computer's processor may read and execute the program code stored in the storage means or storage medium.

[0101] In the above-described embodiment, the control lines and information lines shown are those deemed necessary for explanation and do not necessarily represent all control lines and information lines in the actual product. All components may be interconnected. [Explanation of Symbols]

[0102] 100 calculator 101 Processors 102 Input device 103 Output device 104 Main memory device 105 Auxiliary memory device 106 Network interface 110 Graph recognition program 111 Graph type discrimination module 112 String detection / recognition module 113 Axis acquisition module 114 Plot element detection / recognition module 115 Plot numerical value acquisition module 116 Association module 117 Graph specific information acquisition module 120 Label information 500 GUI

Claims

1. A computer system that accepts an image containing a graph as input and performs graph recognition processing consisting of multiple block processing steps, Includes at least one computer, The aforementioned plurality of block processes include: a first block process for determining the type of graph; a second block process for extracting strings; a third block process for obtaining the axes of the graph based on the processing result of the second block process; a fourth block process for recognizing the plot elements of the graph based on the processing result of the third block process; a fifth block process for obtaining plot values ​​of the graph based on the processing results of the third and fourth block processes; a sixth block process for associating the plot values ​​of the graph with data series based on the result of the fifth block process; a seventh block process for obtaining information representing the characteristics of the graph; and at least one evaluation block process for evaluating the processing result of any of the first to seventh block processes. The evaluation block processing can obtain as output the processing result of any of the first to seventh block processing processes that are the target of the evaluation block processing, and the confidence level of said processing result. The at least one computer that performs the graph recognition process is: When the evaluation block processing is executed, the processing result of any of the first to seventh block processing processes that are subject to the evaluation block processing and the confidence level of said processing result are obtained. Based on the confidence level of any of the processing results of the first to seventh block processes that are subject to the evaluation block process, a determination is made as to whether or not to continue the graph recognition process. If it is determined that the graph recognition process should not be continued, an interface is presented to accept modifications to the processing result of any of the first to seventh block processes that are subject to the evaluation block process. A computer system characterized in that, when it receives user input through the interface, it uses the user input and the processing result of the block processing executed before any of the first to seventh block processing that are subject to the evaluation block processing to execute the block processing after the evaluation block processing.

2. A computer system according to claim 1, The computer system is characterized in that at least one computer determines whether or not to continue the graph recognition process based on a comparison result between the confidence level of any of the processing results of the first to seventh block processes that are subject to the evaluation block processing and a threshold.

3. A computer system according to claim 1, The aforementioned plurality of block processes include a plurality of evaluation block processes, each of which the target block process is different. The aforementioned at least one computer is Based on the confidence level of the processing result of any of the first to seventh block processes targeted by the evaluation block process and the confidence level of the processing result of any of the first to seventh block processes targeted by other evaluation block processes executed before the evaluation block process, the cumulative confidence level is calculated. A computer system characterized by determining whether or not to continue the graph recognition process based on the comparison result between the confidence level of any of the processing results of the first to seventh block processes that are subject to the evaluation block processing and a threshold, and the comparison result between the cumulative confidence level and a threshold.

4. A computer system according to claim 3, The aforementioned at least one computer is A computer system characterized by outputting the confidence level of the processing results of a plurality of evaluation block processes along with the processing result of the graph recognition process.

5. A graph recognition method performed by a computer system including at least one computer, The first step involves at least one computer receiving an image including a graph as input, The process includes a second step in which at least one computer performs a graph recognition process that includes a plurality of block processing, The aforementioned plurality of block processes include: a first block process for determining the type of graph; a second block process for extracting strings; a third block process for obtaining the axes of the graph based on the processing result of the second block process; a fourth block process for recognizing the plot elements of the graph based on the processing result of the third block process; a fifth block process for obtaining plot values ​​of the graph based on the processing results of the third and fourth block processes; a sixth block process for associating the plot values ​​of the graph with data series based on the result of the fifth block process; a seventh block process for obtaining information representing the characteristics of the graph; and at least one evaluation block process for evaluating the processing result of any of the first to seventh block processes. The evaluation block processing can obtain as output the processing result of any of the first to seventh block processing processes that are the target of the evaluation block processing, and the confidence level of said processing result. The second step described above is: A third step in which, when at least one computer executes the evaluation block processing, the processing result of one of the first to seventh block processings that are the subject of the evaluation block processing and the confidence level of said processing result are obtained, and the graph recognition processing is determined based on the confidence level of the processing result of one of the first to seventh block processings that are the subject of the evaluation block processing, If it is determined that the graph recognition process should not be continued, the fourth step is that at least one computer presents an interface for accepting modifications to the processing result of any of the first to seventh block processes that are subject to the evaluation block process, A graph recognition method characterized in that, when user input is received through the interface, the at least one computer executes the block processing after the evaluation block processing using the user input and the processing result of the block processing executed before any of the first to seventh block processings that are the target of the evaluation block processing.

6. A graph recognition method according to claim 5, The graph recognition method is characterized in that the third step includes a step in which at least one computer determines whether or not to continue the graph recognition process based on a comparison result between the confidence level of any of the processing results of the first to seventh block processes that are subject to the evaluation block processing and a threshold.

7. A graph recognition method according to claim 5, The aforementioned plurality of block processes include a plurality of evaluation block processes, each of which the target block process is different. Step 3 above is, The steps include: the at least one computer calculating a cumulative confidence score based on the confidence score of the processing result of any of the first to seventh block processes that are subject to the evaluation block process and the confidence score of the processing result of any of the other first to seventh block processes that are subject to the evaluation block process and were executed before the evaluation block process; A graph recognition method characterized in that at least one computer determines whether or not to continue the graph recognition process based on the comparison result between the confidence level of any of the processing results of the first to seventh block processing that are subject to the evaluation block processing and a threshold, and the comparison result between the cumulative confidence level and a threshold.

8. A graph recognition method according to claim 7, A graph recognition method characterized in that at least one computer outputs the confidence level of the processing results of a plurality of evaluation block processes along with the processing result of the graph recognition process.

Images