System, method, and program for evaluating performance of graph anti-rendering model using artificial intelligence

Abstract

Disclosed are a system, a method, and a program for evaluating the performance of an artificial intelligence model that extracts chart information by derendering a chart. The system includes: a memory including: an AI model that decodes an image of a chart stored in a test set and outputs at least one data format including meta information or data information in the image of the chart; and a performance evaluation model that outputs a result of evaluating the performance of the AI model. And a processor that executes or trains the AI model and runs the performance evaluation model. The performance evaluation model can output a performance evaluation result by comparing information included in the data format with a GT (Ground Truth: True) which is stored in the test set and includes information related to the graph, and more specifically, the performance evaluation result can include line accuracy, axis accuracy, meta accuracy, and length accuracy.

Description

Technical Field

[0001] This invention relates to systems, methods, and procedures for evaluating the performance of models that de-render charts to extract chart information; more specifically, this invention relates to systems, methods, and procedures for evaluating the performance of artificial intelligence models that de-render charts to extract chart information. Background Technology

[0002] Typically, charts and graphs in papers, reports, textbooks, etc., are generated through the following process: In this process, data tables set by numbers and groups, code defining the overall layout (e.g., type, direction, color / graphic configuration), etc., are sent to the rendering engine.

[0003] De-rendering, the opposite of rendering, refers to the process of analyzing and grouping the visual patterns or information of a chart to extract core information, thereby extracting information about the data (e.g., numbers, groupings, etc.), information about the chart layout, and so on.

[0004] As a representative prior technique related to the accuracy assessment of chart information extracted as a result of chart de-rendering, Relative Number Set Similarity (RNSS) is disclosed (DePlot: One-shot visual language reasoning by plot-to-table translation, Fangyu Liu et al. 9, 2023, 3page). In RNSS, when there are predicted data points through de-rendering and a ground truth (GT) representing information about the actual data points, the distance D between the GT and the nearest predicted data point is first calculated as follows.

[0005]

[0006] Here, the final accuracy evaluation score is calculated as follows: the sum of all calculated distances D is divided by the number of data points in the predicted data or the data points in the GT, and then the value is subtracted from 1.

[0007]

[0008] RNSS has been shown to be effective in evaluating performance on datasets consisting of a small number of data points, but it only considers the distance between data points and does not take into account data indices, so it cannot accurately represent the characteristics of the graph.

[0009] For example, in Figure 1a The actual ground truth (GT) in the graph is oriented towards the lower right, while the AI ​​model predicted a graph oriented towards the upper right. Figure 1b In reality, the ground truth (GT) is represented by chart a below chart b, but the AI ​​model predicts that the positions of chart a and chart b are reversed. Thus, Figure 1a and 1b The data predicted by the AI ​​model is inconsistent with the ground truth (GT), thus this is an instance of incorrect data prediction. However, because the sum of the distances between the predicted data points and the points included in the graph within the GT can be displayed as 0, RNSS, which does not consider data indices, cannot distinguish this and evaluates the accuracy of the predicted data as 100%. Additionally, as... Figure 1c As shown, the first predicted data (triangle) lies on the ground plane (circle). Therefore, although the first predicted data (triangle) should be evaluated as more accurate than the second predicted data (quadrilateral), according to the RNSS, since the first predicted data (triangle) and the second predicted data (quadrilateral) are separated from the ground plane (circle) by the same distance D, they are evaluated as having the same accuracy. Furthermore, referring to... Figure 1d Because prediction 2 is visually closer to the GT value than prediction 1, although prediction 2 should be evaluated as a more accurate prediction, the result is that A and B have the same accuracy because RNSS only uses numbers (e.g., Y-axis values) to evaluate accuracy.

[0010] Existing technical literature (non-patent literature 1) DePlot: One-shot visual language reasoning by plot-to-table translation, Fangyu Liu and 9 others, 2023, page 3 Summary of the Invention

[0011] Technical issues

[0012] The problem to be solved by this invention is to provide a system, method and program that can more objectively evaluate the performance of a model that extracts chart information by de-rendering the chart.

[0013] The problems to be solved by the present invention are not limited to those mentioned above, and those skilled in the art will clearly understand from the following description other problems not mentioned herein.

[0014] Methods for solving problems

[0015] The system for evaluating the accuracy of a chart anti-rendering model according to the present invention includes: at least one processor; and at least one memory, wherein the at least one memory stores commands or information for the at least one processor to perform operations, the operations performed by the commands including the following steps: inputting an image of a chart stored in a test set into an AI model and outputting a data format predicting information about the chart; and inputting the data format into a performance evaluation model and comparing the information in the data format with ground truth (GT) stored in the test set as information related to the chart, thereby outputting a performance evaluation result for the AI ​​model, wherein the performance evaluation result includes: line accuracy, which represents the degree of similarity between the chart based on the information in the data format and the chart based on the GT; and axis accuracy, which represents the degree of overlap between the distribution range of the chart based on the information in the data format and the distribution range of the chart based on the GT on at least one axis of the chart.

[0016] In the above system, the line accuracy is a factor that fixes the area of ​​the axis formed by the X-axis or Y-axis of the chart based on the information included in the above data format and the chart based on the GT to a specific value, and compares the area between the lines of the chart based on the information included in the above data format and the lines of the chart based on the GT with the axis area.

[0017] In the above system, the degree of overlap of the above axis accuracy is the proportion of the overlap between the distribution range of the chart on the X-axis based on the information included in the above data format and the distribution range of the chart on the X-axis based on the above GT in the entire range consisting of the distribution range of the chart on the X-axis based on the information included in the above data format and the distribution range of the chart on the X-axis based on the above GT.

[0018] In the above system, the performance evaluation results also include meta-accuracy, which is the degree of consistency between the meta-information included in the above data format and the meta-information included in the above GT.

[0019] In the above system, the accuracy of the above meta-accuracy is calculated using the character error rate (CER).

[0020] In the above system, the performance evaluation results also include length accuracy, which is the difference between the length of the chart based on the information included in the above data format and the length of the chart based on the above GT.

[0021] The method for evaluating the accuracy of a chart anti-rendering model according to the present invention includes the following steps: inputting an image of a chart stored in a test set into an image encoder to convert it into a first embedding that can be processed by an AI model; inputting the first embedding into the AI ​​model and outputting at least one data format including meta-information or data information contained in the image of the chart; inputting the data format into a performance evaluation model and comparing the information included in the data format with the ground truth (GT) stored in the test set that includes information related to the chart and outputting a performance evaluation result, wherein the performance evaluation result is determined based on line accuracy and axis accuracy, the line accuracy representing the closeness between the lines of the chart based on the information included in the data format and the lines of the chart based on the GT, and the axis accuracy representing the degree of overlap between the distribution range of the chart based on the information included in the data format and the distribution range of the chart based on the GT on at least one axis of the chart.

[0022] In the above method, the line accuracy is a factor that fixes the area of ​​the axis formed by the X-axis or Y-axis of the chart based on the information included in the above data format and the chart based on the GT to a specific value, and compares the area between the lines of the chart based on the information included in the above data format and the lines of the chart based on the GT with the axis area.

[0023] In the above method, the degree of overlap of the above axis accuracy is the proportion of the overlap between the distribution range of the chart on the X-axis based on the information included in the above data format and the distribution range of the chart on the X-axis based on the above GT in the entire range consisting of the distribution range of the chart on the X-axis based on the information included in the above data format and the distribution range of the chart on the X-axis based on the above GT.

[0024] In the above method, the performance evaluation results also include meta-accuracy, which is the degree of consistency between the meta-information included in the above data format and the meta-information included in the above GT.

[0025] In the above method, the character error rate (CER) is used to calculate the above-mentioned meta-accuracy.

[0026] In the above method, the performance evaluation results also include length accuracy, which is the difference between the length of the chart based on the information included in the above data format and the length of the chart based on the above GT.

[0027] Another aspect of the invention is a program stored in a computer-readable recording medium for use with a computer to implement an embodiment of the invention that evaluates the accuracy of a chart inverse rendering model.

[0028] Invention Effects

[0029] According to the present invention, evaluation factors with orthogonality are separated and evaluated, thereby enabling quantitative and objective evaluation of the characteristics of the model.

[0030] Furthermore, according to the present invention, the accuracy of the lines is used to represent the degree of similarity between the lines of the charts output by the AI ​​models under the same conditions and the lines of the charts included in the GT, thereby enabling an objective evaluation of the accuracy of the AI ​​models in predicting the chart shape.

[0031] Furthermore, according to the present invention, axis accuracy can be used to evaluate the size of the range of the chart predicted by the AI ​​model.

[0032] Furthermore, according to the present invention, meta-accuracy can be used to objectively evaluate the accuracy of AI models in recognizing meta-information.

[0033] Furthermore, according to the present invention, the uniformity of the shape predicted by the AI ​​model can be objectively evaluated using length accuracy.

[0034] The effects of the present invention are not limited to those mentioned above, and those skilled in the art will clearly understand from the following description other effects not mentioned herein. Attached Figure Description

[0035] Figures 1a to 1d This is an example of the inverse rendering result of a graph used as a performance evaluation object.

[0036] Figure 2 This is a schematic diagram of a system for implementing a performance evaluation method for a graph inverse rendering model utilizing artificial intelligence, according to an embodiment of this disclosure.

[0037] Figure 3 This is a block diagram illustrating the structure of an apparatus for performing a performance evaluation method using an artificial intelligence-based graph inverse rendering model, according to an embodiment of this disclosure.

[0038] Figure 4 This is a block diagram illustrating a method for evaluating the performance of an artificial intelligence model that de-renders a chart, according to embodiments of the present invention.

[0039] Figures 5 to 7 are examples comparing the graphs output from the AI ​​model with those included in the GT. Detailed Implementation

[0040] The embodiments described below are intended to fully illustrate the concept of the invention to those skilled in the art by way of example. Therefore, the invention is not limited to the embodiments described below, and may be embodied in other forms.

[0041] Throughout this specification, the same reference numerals denote the same constituent elements. This invention does not describe all elements of the embodiments; common knowledge or repetition between embodiments is omitted. Terms such as "part," "module," "component," and "block" as used in the specification can be implemented in software or hardware. According to embodiments, multiple "parts, modules, components, and blocks" may be implemented by a single constituent element, or a "part, module, component, and block" may include multiple constituent elements.

[0042] Throughout the instruction manual, when referring to a part as being "connected" to other parts, this includes not only cases where the part is directly connected to other parts, but also cases where they are indirectly connected, including cases where they are connected through a wireless communication network.

[0043] Furthermore, when referring to a part as "including" a certain element, it does not exclude other elements unless specifically stated otherwise, and may include other elements.

[0044] Throughout the instruction manual, when referring to a component being "on" another component, this includes not only cases where one component is connected to another component, but also cases where another component exists between the two components.

[0045] The terms "first" and "second" are used to distinguish one constituent element from other constituent elements, and these terms do not have a limiting effect on the constituent elements.

[0046] Unless otherwise defined in the text, the singular can mean multiple.

[0047] The identification symbols for each step are for ease of explanation and do not indicate the order of the steps. If a specific order is not explicitly stated in the text, the steps may be performed in a different order than that stated.

[0048] The system for evaluating the performance of a graph de-rendering model according to the present invention may include apparatus, which may include various means of performing computational processing to provide results to a user. For example, the system for evaluating the performance of a graph de-rendering model according to the present invention may include at least one computer, server device, and portable terminal, or any form that implements the same or similar functions. The present invention does not limit this.

[0049] Here, the aforementioned computers include, for example, laptops, desktops, laptops, tablet PCs, and slate PCs with built-in web browsers.

[0050] The aforementioned server device, as a server that processes information through communication with external devices, may include application servers, computing servers, database servers, file servers, game servers, mail servers, proxy servers, and web servers, etc.

[0051] The aforementioned portable terminals, as wireless communication devices that ensure portability and mobility, may include all types of handheld wireless communication devices such as PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet), and smartphones, as well as wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted devices (HMDs).

[0052] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0053] This invention relates to systems, methods, and procedures for evaluating the performance of models that de-render charts to extract chart information; more specifically, this invention relates to systems, methods, and procedures for evaluating the performance of artificial intelligence models that de-render charts to extract chart information.

[0054] Figure 2 This is a schematic diagram of a system for evaluating the performance of a model that performs de-rendering of a chart, according to an embodiment of the present invention.

[0055] like Figure 2 As shown, system 1000 may include device 100, database 200, AI model 300 and performance evaluation model 400.

[0056] The device 100, database 200, AI model 300, and performance evaluation model 400 included in system 1000 can communicate via network W. Here, network W can include wired networks and wireless networks. For example, the network can include various networks such as local area network (LAN), metropolitan area network (MAN), and wide area network (WAN).

[0057] Additionally, network W may include the well-known World Wide Web (WWW). However, network W in embodiments of the present invention is not limited to the networks listed above, and may also include at least a portion of well-known wireless data networks, well-known telephone networks, and well-known wired and wireless television networks.

[0058] The device 100 can input a chart into the AI ​​model 300 and output information related to the chart, and evaluate the performance of the AI ​​model 300 using the performance evaluation model 400 based on the information related to the output chart.

[0059] The AI ​​model 300 is input with a test set consisting of images of charts including text and lines, and ground truth (GT) information related to the charts. The test set can be, but is not limited to, test sets commonly used for training the AI ​​model or evaluating performance. The charts included in the test set can be in various forms, such as horizontal / vertical bar charts, line charts, pie charts, area charts, scatter plots, radar charts, histograms, and / or waterfall charts. The charts can be in a single form or a combination of multiple forms. However, the charts used in this invention are not limited to these forms; charts can include any form of chart. In addition to visual information such as numerical values ​​(e.g., lines, circles), the charts may also include text information, specifically chart annotations, a legend or title, names of each axis (e.g., X-axis, Y-axis, Z-axis, etc.), and the raw data of the points in the chart (e.g., the X-axis and Y-axis values ​​of the points in the chart).

[0060] The information output from the AI ​​model 300 that relates to the chart serves as a representation of the chart's characteristics and may include metadata and data information. Metadata may include the names of the X and Y axes, the names of entity groups recorded in the legend, etc. Data information may include the numerical values ​​of each entity's X and / or Y axes, etc.

[0061] The graph-related information output from AI model 300 is input into performance evaluation model 400, which evaluates the performance of AI model 300 by comparing the ground truth (GT) included in the dataset with the graph-related information output.

[0062] The performance evaluation results output from performance evaluation model 400 represent the accuracy of the AI ​​model's predictions of the charts. Performance evaluation results can be represented by numerical scores, graphical charts, etc., but are not limited to these. Performance evaluation results can be output separately according to individual factor values, which may include line accuracy, axis accuracy, meta-accuracy, or length accuracy, but are not limited to these.

[0063] According to the present invention, the device 100 can output the performance evaluation results of the AI ​​model that performs de-rendering on the input chart.

[0064] Database 200 may store various data (e.g., test sets) used for training AI model 300 or evaluating its performance. Additionally, database 200 may store charts, graph-related information, information related to performance evaluation methods, and, in various embodiments, computational data output by AI model 300. However, system 1000 may not include database 200 once training of AI model 300 is complete.

[0065] Figure 2 The illustration shows a scenario where the database 200 is implemented externally to the device 100. In this case, the database 200 can be connected to the device 100 via wired / wireless means. However, this is only one embodiment, and the database 200 can also be implemented as a component of the device 100.

[0066] Figure 2 The illustration shows the case where the AI ​​model 300 is implemented outside the device 100 (e.g., cloud-based implementation), but is not limited thereto and can also be implemented as a component of the device 100.

[0067] Figure 3 This is a block diagram illustrating the structure of an apparatus for performing a performance evaluation method of an artificial intelligence model that de-renders a chart to extract chart information, according to an embodiment of the present invention.

[0068] like Figure 3As shown, device 100 may include a memory 110, a communication module 120, a display 130, an input module 140, and a processor 150. However, it is not limited to this; depending on the required operation, software and hardware structures within the scope that are obvious to those skilled in the art may be modified, added to, or omitted from device 100. Furthermore, device 100 may be replaced by a system, and device 100 may include multiple devices, in which case the various components included in device 100 may be included in at least one of the multiple devices.

[0069] The memory 110 may store data supporting various functions of the device 100, programs for operating the processor 150, input / output data, multiple applications or applications driven in the device, data, commands, and AI models required for the operation of the device 100. At least some of such applications may be downloaded from an external server via wireless communication.

[0070] Such a memory 110 may include at least one type of storage recording medium selected from flash memory type, hard disk type, SSD type, SDD type, multimedia card micro type, card type memory (e.g., SD or XD memory), RAM (random access memory), SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic storage, magnetic disk, and optical disk.

[0071] Additionally, the memory 110 may be detached from the device and may include a database connected via wired or wireless means. Figure 2 The database 200 shown can be implemented as a component of the memory 110.

[0072] The communication module 120 may include one or more components for communicating with external devices, such as at least one of a broadcast receiving module, a wired communication module, a wireless communication module, a short-range communication module, and a location information module.

[0073] Wired communication modules can include not only various wired communication modules such as local area network (LAN) modules, wide area network (WAN) modules, or value-added network (VAN) modules, but also various cable communication modules such as USB (Universal serial bus), DHMI (High Definition Multimedia Interface), DVI (Digital Visual Interface), RS-232 (recommended standard 232), power line communication, or POTS (plain old telephone service).

[0074] In addition to Wi-Fi modules and wireless broadband modules, wireless communication modules can also include wireless communication modules that support various wireless communication methods such as GSM (global system for mobile communication), CDMA (code division multiple access), WCDMA (wideband code division multiple access), UMTS (universal mobile telecommunications system), TDMA (time division multiple access), LTE (long term evolution), 4G, 5G, and 6G.

[0075] The display 130 displays (outputs) information or data processed in the device 100, data input or output through the AI ​​model 300, etc. In addition, the display 130 may display execution screen information of applications (e.g., applications) driven in the device 100 or UI (User Interface) or GUI (Graphical User Interface) information generated based on such execution screen information.

[0076] The input module 140 is used to receive information from the user. When the user inputs information through the input section, the processor 150 controls the operation of the device 100 in a manner corresponding to the input information.

[0077] Such an input module 140 may include hardware physical keys (e.g., buttons, dome switches, jog wheels, jog switches, etc. located on at least one of the front, rear, and side surfaces of the device) and software touch keys. As an example, the touch keys may consist of virtual keys, soft keys, or visual keys displayed on a touchscreen-type display 130 through software processing, or touch keys located outside the aforementioned touchscreen. On the other hand, the aforementioned virtual keys or visual keys may be displayed on the touchscreen in various forms, such as graphics, text, icons, videos, or combinations thereof.

[0078] The processor 150 may be implemented by a memory storing data related to an algorithm or a program for reproducing an algorithm used to control the operations of the constituent elements within the device 100 (including training or executing an AI model) and at least one processor (not shown) that performs the aforementioned operations using the data stored in the memory. In this case, the memory and the processor may be implemented by separate chips or by a single chip.

[0079] In one embodiment, the system 1000 or apparatus 100 of the present invention may include at least one processor; in the case of multiple processors, the multiple processors may be included in apparatus 100 that are different from each other.

[0080] In addition, the processor 150 may control one or more combinations of the above-described components in order to implement the various embodiments of the present disclosure described below on the device 100.

[0081] Figure 4 This is a block diagram illustrating a method for evaluating the performance of a model that de-renders a chart to extract chart information according to an embodiment of the present invention.

[0082] Reference Figure 4The test set 410, consisting of chart images including text and lines, and ground truth (GT) information related to the chart, is input to the image encoder 420. Here, the chart image includes a title, axis names, legend information, and numerical values. The chart 410 can be in the form of horizontal / vertical bar charts, line charts, pie charts, area charts, scatter plots, radar charts, histograms, and / or waterfall charts, but is not limited to these. Furthermore, the chart can be in a single form or a combination of multiple forms.

[0083] Image encoder 420, following a commonly used encoder architecture, transforms the input graph 410a into a first embedding 421 that can be processed by AI model 300. Here, the first embedding 421 may include a concatenate embedding.

[0084] Next, the first embedding 421 output by the image encoder 420 is input to the AI ​​model 400, which includes a decoder 430 that performs meta-decoding, data decoding, etc. The decoder 430 decodes the input first embedding 421 to extract the chart's meta-information (e.g., chart title, X-axis name, Y-axis name, legend), data information (e.g., data point values), etc., and outputs this as the second embedding 431.

[0085] In this way, the AI ​​model can output a data format 440 that includes metadata, data information, etc., based on the output second embedding 431.

[0086] Next, the data format 440 output from the AI ​​model is input into the chart de-rendering performance evaluation model 450. The chart de-rendering performance evaluation model 450 compares the chart information included in the data format 440 (e.g., chart title, X-axis name, Y-axis name, legend, data point values, etc.) with the ground truth (GT) included in the dataset 410, and calculates factor values ​​to derive the performance evaluation result 460 according to a predetermined method. The calculated factor values ​​reflect information about data indices, and may specifically include, but are not limited to, line accuracy 451, axis accuracy 452, meta accuracy 453, or length accuracy 454.

[0087] The performance evaluation model 450, using the calculated factor values ​​individually or in combination, outputs a performance evaluation result 460 representing the accuracy of the AI ​​model's predicted charts. The performance evaluation result 460 can be represented by numerical scores, graphical charts, etc., but is not limited to these. The performance evaluation result 460 can be output separately according to individual factor values, which may include, but are not limited to, the line accuracy 451, axis accuracy 452, element accuracy 453, or length accuracy 454 described above. For example, the performance evaluation result 460 can be represented as performance evaluation result 460a.

[0088] Therefore, this invention separates and evaluates orthogonal evaluation factors, thereby enabling a quantitative and objective assessment of the model's characteristics.

[0089] Below, refer to Figure 5 to... Figure 7 The factor values ​​used to derive the performance evaluation result 460 are explained in detail.

[0090] Line accuracy 451 is a factor representing how closely the lines of a chart output by the AI ​​model approximate the lines of charts included in the ground truth (GT). Line accuracy 451 assesses the accuracy with which the AI ​​model predicts chart shapes. Specifically, line accuracy 451 refers to fixing the area formed by the X-axis or Y-axis of the chart (axis area) to a specific value (e.g., fixing the X-axis of the chart...). The area along the Y-axis is fixed at 1. 1), and a factor that compares the area between the lines of the output chart and the lines of the chart included in GT with that area value.

[0091] More specifically, refer to Figure 5a The example shown compares charts output by AI models A and B with those included in the ground truth (GT). It demonstrates that model B, whose lines in the AI-output chart are closer to those in the GT, can be considered to predict chart shape more accurately than model A. However, when calculating the area between the lines of the output chart and the lines of the chart included in the GT (i.e., the shaded area in the chart) using X-axis and Y-axis values ​​respectively in AI models A and B, it shows that these areas are the same in both models. Thus, if the area formed by the X-axis or Y-axis of the charts output by AI models A and B is not fixed, the difference in scale along the X-axis or Y-axis makes it impossible to compare the area between the lines of the output chart and the lines of the chart included in the GT under the same conditions. Consequently, it is difficult to compare which AI model predicts chart shape more accurately.

[0092] To solve such problems, such as Figure 5bAs shown, for the charts output from AI model A and AI model B, the area formed by the X-axis or Y-axis is similarly fixed to a certain area (=1). 1) With a fixed chart area (=1), the line accuracy 451 can be calculated by using the proportion of the area between the lines of the output chart and the lines of the chart included in the GT (the part shown by the shading triangle in the chart). By comparing the line accuracy 451 calculated for each AI model in this way, it is possible to compare, under the same conditions, how close the lines of the charts output by each AI model are to the lines of the charts included in the GT (i.e., how small the proportion of the area separating the charts is in the total area).

[0093] More preferably, the line accuracy 451 can be calculated using the following formula.

[0094]

[0095] Here, P represents the function followed by the output chart, and T represents the function followed by the chart included in GT. Additionally, dom represents the domain followed by each function (P and T). According to the above formula, the area formed by the X-axis or Y-axis of the chart ( Figure 5b In the case of "1"), the value obtained by subtracting the area formed between the lines of the output chart and the lines of the chart included in the GT (the shaded triangle) is used as the line accuracy 451. Models with a larger line accuracy 451 value are evaluated as having more accurate performance in predicting chart shapes. For example, in Figure 5a and 5b In the study, Model B has a smaller proportion of the area between the graphs (the shaded triangles) relative to the entire area of ​​1, and therefore a larger line accuracy value of 451. Thus, Model B is evaluated as predicting the graph shape more accurately than Model A.

[0096] Therefore, the present invention uses line accuracy 451 to represent the degree of similarity between the lines of the charts output by AI models under the same conditions and the lines of the charts included in the GT, thereby enabling an objective evaluation of the accuracy of the AI ​​model in predicting the chart shape.

[0097] On the other hand, axis accuracy 452 is a factor representing the degree of overlap between the distribution range of the chart output by the AI ​​and the distribution range of the chart included in the GT on at least one axis of the chart. Axis accuracy 452 can be used to assess whether the output chart predicts the chart within a sufficient range. Even if the predicted output chart is close to the chart included in the GT, if the range of the output chart is not wide, the AI ​​model may not be able to provide useful values ​​over a wide range. Therefore, axis accuracy 452 is needed to evaluate whether the AI ​​model predicts the chart within a sufficient range.

[0098] More specifically, such as Figure 6 As shown, the proportion of the overlap between the distribution range of the chart output from the AI ​​on the X-axis and the distribution range of the chart included in the GT on the X-axis in the entire range b consisting of the distribution range of the chart output from the AI ​​on the X-axis and the distribution range of the chart included in the GT on the X-axis is used as the axis accuracy 452 value (i.e., the higher the axis accuracy 452 value, the more the AI ​​model can be evaluated as predicting charts over a wider range).

[0099] More preferably, the line accuracy is calculated using the following formula.

[0100]

[0101] Here, P represents the function followed by the output chart, and T represents the function followed by the chart included in the GT. Additionally, dom represents the domain followed by each function (P and T), and m represents the Lebesgue measure. According to the above formula, the proportion of the overlap between the distribution range of the chart output from the AI ​​on the X-axis and the distribution range of the chart included in the GT on the X-axis within the entire range comprised of the distribution ranges of the chart output from the AI ​​on the X-axis and the distribution ranges of the chart included in the GT on the X-axis is calculated as the axis accuracy 452.

[0102] Therefore, the present invention can use axis accuracy 452 to evaluate the range size of the chart predicted by the AI ​​model.

[0103] On the other hand, meta-accuracy 453 is a factor value used to evaluate whether the meta-information of the charts output by the AI ​​model is accurately identified. Meta-information included in the output of the chart re-rendering AI model (e.g., chart title, axis labels, legend, etc.) plays an important role in providing insights to users using the output results, so it is necessary to evaluate the accuracy of the AI ​​model in identifying meta-information.

[0104] Preferably, the character error rate (CER) is used to calculate the metadata accuracy 453. CER can be calculated using the following formula.

[0105]

[0106] Here, S represents the number of substitutions, D represents the number of deletions, I represents the number of insertions, C represents the number of correct characters, and N represents the total number of reference characters (=S+D+C). Therefore, a lower CER value indicates a higher recognition rate; CER represents 0 when metadata is perfectly recognized.

[0107] More specifically, according to the following formula and using the CER operand accuracy of 453.

[0108]

[0109] Therefore, it can be assessed that the higher the meta-accuracy (453), the more accurately the AI ​​model predicts meta-information.

[0110] Therefore, the present invention can objectively evaluate the accuracy of AI models in recognizing metadata using the meta-accuracy 453.

[0111] On the other hand, length accuracy 454 is a factor that indicates whether the chart output from the AI ​​model deviates significantly from the chart included in the GT at a specific point. Length accuracy 454 can be used to evaluate the accuracy of the AI ​​model in predicting the shape of the chart.

[0112] Specifically, refer to Figure 7 The chart output by AI model A deviates significantly from the chart included in the ground truth (GT) at certain points, while in the case of AI model B, it almost perfectly follows the shape of the chart included in the GT. Therefore, AI model B can be evaluated as predicting the chart shape more accurately. To reflect this and more accurately evaluate the performance of the AI ​​models, a length accuracy 454, reflecting the difference in length between the chart output by the AI ​​model and the chart included in the GT, can be additionally considered in the performance evaluation results. That is, by assigning a higher length accuracy 454 to a smaller difference in length between the output chart and the chart included in the GT, the performance of the AI ​​model in predicting the chart shape uniformly can be evaluated. For example, in... Figure 7 In the comparison, AI model B's chart length is shorter than AI model A's chart length, so its accuracy value of 454 is higher. Therefore, it can be evaluated that AI model B predicts the chart shape more uniformly than AI model A.

[0113] Preferably, the length accuracy of 454 can be calculated using the following formula.

[0114]

[0115] Here, P represents the function followed by the output chart, and T represents the function followed by the chart included in the GT. Additionally, dom represents the domain followed by each function (P and T). According to the above formula, the length accuracy 454 is calculated by subtracting the sum of the length Lp of the output chart and the length Lt of the chart included in the GT from 1. Models with a larger length accuracy 454 are evaluated as having more accurate performance in predicting the chart shape uniformly.

[0116] Therefore, the present invention can objectively evaluate the uniformity of the shape predicted by the AI ​​model using a length accuracy of 454.

[0117] On the other hand, it can be referenced Figure 4 The system described herein is a performance evaluation method for a model that performs de-rendering of charts to extract chart information, according to embodiments of the present invention.

[0118] The AI ​​model in embodiments of the present invention can be controlled, operated, learned, and driven by a processor, thereby enabling the AI ​​model to perform at least one task among operation, learning, and driving via at least one processor. Furthermore, the AI ​​model can be stored in a memory, and the feature data of the present invention can also be stored in a memory.

[0119] On the other hand, the disclosed embodiments can be implemented in the form of a recording medium for storing computer-executable commands. Commands can be stored in the form of program code, and when executed by a processor, a program module is generated to perform the operations of the disclosed embodiments. The recording medium can be implemented as a computer-readable recording medium.

[0120] Computer-readable recording media include all kinds of recording media that store computer-readable commands. For example, they can be ROM (Read Only Memory), RAM (Random Access Memory), magnetic tape, magnetic disk, flash memory, optical data storage devices, etc.

[0121] The disclosed embodiments have been described above with reference to the accompanying drawings. Those skilled in the art can implement this disclosure in different forms than the disclosed embodiments without altering the technical concept or essential features. The disclosed embodiments are merely illustrative, and the present invention is not limited thereto.

Claims

1. A system for evaluating the accuracy of a chart inversion model, comprising: At least one processor; and At least one memory, wherein the at least one memory stores commands or information that cause the at least one processor to perform operations. The above command performs the following steps: Input images of charts stored in the test set into the AI ​​model, and output a data format that predicts the information of the aforementioned charts; and The data format described above is input into the performance evaluation model, and the information in the data format is compared with the Ground Truth (GT), which is stored in the test set and is related to the graph, to output the performance evaluation result of the AI ​​model. Here, the GT is the Ground Truth. The above performance evaluation results include: Line accuracy, which indicates how closely a chart based on the above data format resembles a chart based on the above GT; and Axis accuracy, which represents the degree of overlap between the distribution range of a chart based on the above data format and the distribution range of a chart based on the above GT on at least one axis of the above chart.

2. The system according to claim 1, wherein, The aforementioned line accuracy is a factor that fixes the area of ​​the axis formed by the X-axis or Y-axis of the chart based on the information included in the aforementioned data format and the chart based on the aforementioned GT to a specific value, and compares the area between the lines of the chart based on the information included in the aforementioned data format and the lines of the chart based on the aforementioned GT with the aforementioned axis area.

3. The system according to claim 1, wherein, The aforementioned degree of overlap in the accuracy of the aforementioned axis refers to the proportion of the overlap between the distribution range of the chart based on the information included in the aforementioned data format and the distribution range of the chart based on the aforementioned GT on the X-axis within the entire range consisting of the distribution range of the chart based on the information included in the aforementioned data format and the distribution range of the chart based on the aforementioned GT on the X-axis.

4. The system according to claim 1, wherein, The above performance evaluation results also include meta-accuracy. The aforementioned metadata accuracy refers to the degree of consistency between the metadata included in the aforementioned data format and the metadata included in the aforementioned GT.

5. The system according to claim 4, wherein, The accuracy of the above-mentioned meta-analysis is calculated using the character error rate (CER).

6. The system according to claim 1, wherein, The above performance evaluation results also include length accuracy. The aforementioned length accuracy is the difference between the length of the chart based on the information included in the aforementioned data format and the length of the chart based on the aforementioned GT.

7. A method for evaluating the accuracy of a chart inversion model, comprising the following steps: Images of charts stored in the test set are input into an image encoder and transformed into a first embedding that can be processed by an AI model; The first embedding is input into the AI ​​model and the output is at least one data format including the metadata or data information contained in the image of the chart. The above data format is input into the performance evaluation model, and the information included in the above data format is compared with the ground truth (GT) stored in the above test set, which includes information related to the above charts, to output the performance evaluation result. The above GT stands for Ground Truth, i.e., the true value. The above performance evaluation results are determined based on the accuracy of the lines and the accuracy of the axes. The line accuracy mentioned above indicates how closely the lines of a chart based on the information included in the above data format are similar to the lines of a chart based on the above GT. The aforementioned axis accuracy indicates the degree of overlap between the distribution range of a chart based on information included in the aforementioned data format and the distribution range of a chart based on the aforementioned GT on at least one axis of the chart.

8. The method according to claim 7, wherein, The aforementioned line accuracy is a factor that fixes the area of ​​the axis formed by the X-axis or Y-axis of the chart based on the information included in the aforementioned data format and the chart based on the aforementioned GT to a specific value, and compares the area between the lines of the chart based on the information included in the aforementioned data format and the lines of the chart based on the aforementioned GT with the aforementioned axis area.

9. The method according to claim 7, wherein, The aforementioned degree of overlap in the accuracy of the aforementioned axis refers to the proportion of the overlap between the distribution range of the chart based on the information included in the aforementioned data format and the distribution range of the chart based on the aforementioned GT on the X-axis within the entire range consisting of the distribution range of the chart based on the information included in the aforementioned data format and the distribution range of the chart based on the aforementioned GT on the X-axis.

10. The method according to claim 7, wherein, The above performance evaluation results also include meta-accuracy. The aforementioned metadata accuracy refers to the degree of consistency between the metadata included in the aforementioned data format and the metadata included in the aforementioned GT.

11. The method according to claim 10, wherein, The above-mentioned meta-accuracy is calculated using the character error rate (CER).

12. The method according to claim 7, wherein, The above performance evaluation results also include length accuracy. The aforementioned length accuracy is the difference between the length of the chart based on the information included in the aforementioned data format and the length of the chart based on the aforementioned GT.

13. A program stored in a computer-readable recording medium for use with a computer to perform the method according to any one of claims 7 to 12.

Images