System, method, and program for evaluating performance of chart de-rendering model using artificial intelligence

Abstract

Disclosed are a system, a method, and a program for evaluating the performance of an artificial intelligence model capable of extracting information of a chart by de-rendering the chart. The system comprises: a memory including an AI model that outputs at least one data format including meta information or data information included in an image of a chart by decoding the image of the chart stored in a test set and a performance evaluation model that outputs the result of evaluating the performance of the AI model; and a processor that executes or trains the AI model, and executes the performance evaluation model. The performance evaluation model may output a performance evaluation result by comparing information included in the data format with ground truth (GT) including information regarding the chart stored in the test set, and more particularly, the performance evaluation result may include line accuracy, axis accuracy, meta accuracy, and length accuracy.

Description

System, method, and program for evaluating the performance of a chart de-rendering model using artificial intelligence

[0001] The present invention relates to a system, method, and program for evaluating the performance of a model that de-renders a chart to extract information from the chart, and more particularly, to a system, method, and program for evaluating the performance of an artificial intelligence model that can de-render a chart to extract information from the chart.

[0002] Charts included in papers, reports, and textbooks are typically created through a process that sends data tables set up as numbers and groups, and code defining the overall layout (e.g., type, direction, color / shape composition) to a rendering engine.

[0003] Chart de-rendering is the opposite process of chart rendering. It involves analyzing and grouping visual patterns or information in a chart to extract key information, and then extracting information about the data (e.g., numbers, groups, etc.) and information about the chart layout.

[0004] As a representative conventional technology related to the accuracy evaluation of chart information extracted as a result of such chart de-rendering, relative number set similarity (RNSS) is disclosed (DePlot: One-shot visual language reasoning by plot-to-table translation, Fangyu Liu et al., 2023, page 3). RNSS first calculates the distance (D) between the GT and the predicted data point closest to it, as follows, when there is a GT (Ground Truth) representing the information of the predicted data point and the actual data point through de-rendering.

[0005]

[0006] Here, the final accuracy evaluation score is calculated by adding all the distances (D) calculated as shown below, dividing the result by the number of data points of the predicted data or GT, whichever has more data points, and then subtracting this value from 1.

[0007]

[0008] RNSS has proven to be useful for evaluating performance on data sets consisting of a small number of data points, but it has limitations in accurately representing the characteristics of the chart because it only considers the distance between data points and does not consider data indices.

[0009] For example, in Fig. 1a, the actual GT is a chart pointing to the lower right, but the AI ​​model predicted a chart pointing to the upper right, and in Fig. 1b, the actual GT is a chart in which chart a is located below chart b, but the AI ​​model predicted by switching the positions of charts a and b. In this way, even though Figs. 1a and 1b are cases in which the data predicted by the AI ​​model does not match the GT and thus the data is incorrectly predicted, the sum of the distances between the predicted data between points and the points of the chart included in the GT may appear as 0, so RNSS, which does not consider data indices, may not be able to distinguish this and may evaluate the predicted data as having 100% accuracy. Also, since the first prediction data (triangle) is located on the GT (circle) as shown in Fig. 1c, the first prediction data (triangle) should be evaluated as more accurate than the second prediction data (square). However, according to RNSS, the first prediction data (triangle) and the second prediction data (square) are evaluated as having the same accuracy because the distance (D) from the GT (circle) is the same. Furthermore, looking at Fig. 1d, since prediction 2 is visually closer to the GT value than prediction 1, prediction 2 should be evaluated as a more accurate prediction. However, since RNSS evaluates the accuracy using only numbers (e.g., Y-axis values), there is a problem in that the accuracies of (A) and (B) are derived as being the same.

[0010] Prior art literature (non-patent literature 1) DePlot: One-shot visual language reasoning by plot-to-table translation, Fangyu Liu et al., 2023. page 3

[0011] The problem to be solved by the present invention is to provide a system, method and program that can more objectively evaluate the performance of a model that extracts information from a chart by de-rendering the chart.

[0012] The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0013] A system for evaluating the accuracy of a chart de-rendering model of the present invention comprises at least one processor, and at least one memory storing instructions or information for causing the at least one processor to perform an operation, wherein the instructions perform an operation including: inputting an image of a chart stored in a test set into an AI model, and outputting a data format predicting information of the chart; and inputting the data format into a performance evaluation model, and comparing information of the data format with GT (Ground Truth), which is information about the chart stored in the test set, and outputting a performance evaluation result for the AI ​​model; wherein the performance evaluation result may include line accuracy, which indicates the degree to which a chart according to information of the data format is close to a chart according to the GT; and axis accuracy, which indicates the degree to which, in at least one axis of the chart, a range in which a chart according to information of the data format is distributed and a range in which a chart according to the GT is distributed overlap.

[0014] In the above system, the line accuracy may be a factor that fixes the axis area formed by the X-axis or Y-axis of the chart according to the information included in the data format and the chart according to the GT to a specific value, and compares the axis area with the area between the line of the chart according to the information included in the data format and the line of the chart according to the GT.

[0015] In the above system, the degree of overlap of the axis accuracy may be the proportion of the overlapping portion between the range of the X-axis in which the chart is distributed according to the information included in the data format and the range of the X-axis in which the chart is distributed according to the GT in the entire range of the X-axis in which the chart is distributed according to the information included in the data format and the range of the X-axis in which the chart is distributed according to the GT.

[0016] In the above system, the performance evaluation result may further include meta accuracy, and the meta accuracy may be the degree to which the meta information included in the data format matches the meta information included in the GT.

[0017] In the above system, the meta accuracy can be calculated using the character error rate (CER).

[0018] In the above system, the performance evaluation result may further include length accuracy, and the length accuracy may be the difference between the length of the chart according to the information included in the data format and the length of the chart according to the GT.

[0019] In a method for evaluating the accuracy of a chart de-rendering model of the present invention, an image of a chart stored in a test set is input to an image encoder to be converted into a first embedding that can be processed by an AI model, the first embedding is input to the AI ​​model to output at least one data format including meta information or data information included in the image of the chart, the data format is input to a performance evaluation model to compare information included in the data format with a GT (Ground Truth) including information about the chart stored in the test set to output a performance evaluation result; as a method, the performance evaluation result may be determined according to line accuracy, which indicates the degree to which a line of a chart according to information included in the data format is close to a line of a chart according to the GT; and axis accuracy, which indicates the degree to which a range in which a chart according to information included in the data format is distributed and a range in which a chart according to the GT is distributed overlap in at least one axis of the chart.

[0020] In the above method, the line accuracy is a factor that fixes the axis area formed by the X-axis or Y-axis of the chart according to the information included in the data format and the chart according to the GT to a specific value, and compares the axis area with the area between the line of the chart according to the information included in the data format and the line of the chart according to the GT.

[0021] In the above method, the degree of overlap of the axis accuracy may be the proportion of the overlapping portion between the range of the X-axis in which the chart according to the information included in the data format is distributed and the range of the X-axis in which the chart according to the GT is distributed, in the entire range of the X-axis in which the chart according to the information included in the data format is distributed and the range of the X-axis in which the chart according to the GT is distributed.

[0022] In the above method, the performance evaluation result may additionally include meta accuracy, and the meta accuracy may be the degree to which the meta information included in the data format matches the meta information included in the GT.

[0023] In the above method, the meta accuracy can be calculated using the character error rate (CER).

[0024] In the above method, the performance evaluation result may further include length accuracy, and the length accuracy may be the difference between the length of the chart according to the information included in the data format and the length of the chart according to the GT.

[0025] According to another aspect of the present invention, a program may be stored in a computer-readable recording medium to implement a model for evaluating the accuracy of a chart de-rendering model according to embodiments of the present invention, in combination with a computer.

[0026] According to the present invention, the characteristics of a model can be quantitatively and objectively evaluated by separating and evaluating evaluation factors having orthogonality.

[0027] In addition, according to the present invention, by using line accuracy to indicate how closely the lines of a chart output under the same conditions between AI models are to the lines of a chart included in GT, it is possible to objectively evaluate how accurately an AI model predicts the shape of a chart.

[0028] Additionally, according to the present invention, it is possible to evaluate how widely an AI model can predict charts using axis accuracy.

[0029] Additionally, according to the present invention, it is possible to objectively evaluate how accurately an AI model recognizes meta information using meta accuracy.

[0030] Additionally, according to the present invention, it is possible to objectively evaluate how evenly an AI model predicts the shape of a chart using length accuracy.

[0031] The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0032] Figures 1a to 1d are examples of chart de-rendering results that are the subject of performance evaluation.

[0033] FIG. 2 is a schematic diagram of a system for implementing a performance evaluation method of a chart de-rendering model using artificial intelligence according to one embodiment of the present disclosure.

[0034] FIG. 3 is a block diagram illustrating a configuration of a device that performs a performance evaluation method of a chart de-rendering model using artificial intelligence according to one embodiment of the present disclosure.

[0035] FIG. 4 is a block diagram illustrating a method for evaluating the performance of an artificial intelligence model for de-rendering a chart according to embodiments of the present invention.

[0036] Figures 5 to 7 are examples comparing charts output from an AI model with charts included in GT.

[0037] The following examples are provided as examples to ensure that those skilled in the art can fully grasp the spirit of the present invention. Therefore, the present invention is not limited to the embodiments described below and may be embodied in other forms.

[0038] Throughout the present invention, the same reference numerals denote the same components. The present invention does not describe all elements of the embodiments, and any content that is general in the technical field to which the present invention pertains or that overlaps between the embodiments is omitted. The terms 'part, module, element, block' used in the specification may be implemented in software or hardware, and depending on the embodiments, multiple 'parts, modules, elements, blocks' may be implemented as a single component, or a single 'part, module, element, block' may include multiple components.

[0039] Throughout the specification, when a part is said to be "connected" to another part, this includes not only direct connection but also indirect connection, and indirect connection includes connection via a wireless communication network.

[0040] Additionally, when a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

[0041] Throughout the specification, when we say that an element is "on" another element, this includes not only cases where the element is in contact with the other element, but also cases where another element exists between the two elements.

[0042] The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

[0043] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0044] The identification codes for each step are used for convenience of explanation and do not describe the order of each step. Each step may be performed in a different order than specified unless the context clearly indicates a specific order.

[0045] A system for evaluating the performance of a chart de-rendering model according to the present invention may include a device, which may include any of a variety of devices capable of performing computational processing and providing results to a user. For example, the system for evaluating the performance of a chart de-rendering model according to the present invention may include at least one of a computer, a server device, and a portable terminal, or may be any form having the same or similar functions as these. However, the present invention is not limited thereto.

[0046] Here, the computer may include, for example, a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

[0047] The above server device is a server that processes information by communicating with an external device, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, and a web server.

[0048] The above portable terminal may include, for example, a wireless communication device that ensures portability and mobility, and may include all kinds of handheld-based wireless communication devices such as a PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, a smart phone, and a wearable device such as a watch, a ring, a bracelet, an anklet, a necklace, glasses, contact lenses, or a head-mounted device (HMD).

[0049] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0050] The present invention relates to a system, method, and program for evaluating the performance of a model that de-renders a chart to extract information from the chart, and more particularly, to a system, method, and program for evaluating the performance of an artificial intelligence model that can de-render a chart to extract information from the chart.

[0051] FIG. 2 is a schematic diagram of a system that can implement a method for evaluating the performance of a model for de-rendering a chart according to one embodiment of the present invention.

[0052] As illustrated in FIG. 2, the system (1000) may include a device (100), a database (200), an AI model (300), and a performance evaluation model (400).

[0053] The device (100), database (200), AI model (300), and performance evaluation model (400) included in the system (1000) can communicate via a network (W). Here, the network (W) may include a wired network and a wireless network. For example, the network may include various networks such as a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN).

[0054] Additionally, the network (W) may include the well-known World Wide Web (WWW). However, the network (W) according to an embodiment of the present invention is not limited to the networks listed above, and may include at least part of a well-known wireless data network, a well-known telephone network, or a well-known wired / wireless television network.

[0055] The device (100) can input a chart into the AI ​​model (300) and output information about the chart, and can evaluate the performance of the AI ​​model (300) using the performance evaluation model (400) based on the information about the output chart.

[0056] The AI ​​model (300) is input with a test set consisting of an image of a chart containing text, lines, etc., and a ground truth (GT) containing information about the chart. The test set is the same as, but not limited to, those commonly used to train an AI model or evaluate its performance. The charts included in the test set may be in the form of vertical / horizontal bar charts, line charts, pie charts, area charts, scatter charts, radar charts, histograms, and / or waterfall charts. The chart may be in a single form or may be a combination of multiple forms. However, the charts used in the present invention are not limited thereto, and any form of chart may be included. In addition to visual information such as numbers (e.g., line, circle, etc.), charts may include text information, and specifically, may include chart annotation information, chart legend or title, names of each axis (e.g., X-axis, Y-axis, Z-axis, etc.), raw data values ​​of points included in the chart (e.g., numerical values ​​of the X-axis and Y-axis of points included in the chart), etc.

[0057] Information about a chart output from the AI ​​model (300) may include meta information, data information, etc., as information indicating the characteristics of the chart. Meta information may be the names of the X-axis and Y-axis, entity group names recorded in a legend, etc. Data information may be the numerical values ​​of the X-axis and / or Y-axis of each entity, etc.

[0058] Information about the chart output from the AI ​​model (300) is input to the performance evaluation model (400), and the performance evaluation model (400) compares the GT included in the data set with information about the output chart to evaluate the performance of the AI ​​model (300).

[0059] The performance evaluation results output from the performance evaluation model (400) indicate how accurately the AI ​​model predicts the chart. The performance evaluation results may be expressed as numerical scores, graphical charts, or the like, but are not limited thereto. The performance evaluation results may be output separately for each individual factor value, and the individual factor values ​​may include, but are not limited to, line accuracy, axis accuracy, meta accuracy, or length accuracy.

[0060] According to the present invention, the device (100) can output performance evaluation results for an AI model that de-renders an input chart.

[0061] The database (200) may store various data (e.g., a test set) for training the AI ​​model (300) or evaluating its performance. Furthermore, the database (200) may store chart images, information about charts, information about performance evaluation methods, and the like, and in various embodiments, may store output data output by the AI ​​model (300). However, the system (1000) may not include the database (200) if training of the AI ​​model (300) is complete.

[0062] FIG. 2 illustrates a case where a database (200) is implemented outside of a device (100). In this case, the database (200) may be connected to the device (100) via wired or wireless connections. However, this is merely an example, and the database (200) may also be implemented as a component of the device (100).

[0063] FIG. 2 illustrates a case where the AI ​​model (300) is implemented outside the device (100) (e.g., cloud-based), but is not limited thereto, and may be implemented as a component in the device (100).

[0064] FIG. 3 is a block diagram illustrating a configuration of a device for performing a method of evaluating the performance of an artificial intelligence model capable of extracting information from a chart by de-rendering the chart, according to one embodiment of the present invention.

[0065] As illustrated in FIG. 3, the device (100) may include a memory (110), a communication module (120), a display (130), an input module (140), and a processor (150). However, the present invention is not limited thereto, and the device (100) may have its software and hardware configurations modified / added / omitted within a range apparent from a perspective of ordinary skill in the art, depending on the required operation. In addition, the device (100) may be replaced with a system, and the device (100) may include a plurality of devices, in which case each component included in the device (100) may be included in at least one of the plurality of devices.

[0066] The memory (110) can store data supporting various functions of the device (100), programs for the operation of the processor (150), input / output data, and a plurality of application programs or applications run on the device, data for the operation of the device (100), commands, and AI models. At least some of these application programs can be downloaded from an external server via wireless communication.

[0067] The memory (110) may include at least one type of storage medium among a flash memory type, a hard disk type, an SSD (Solid State Disk type), an SDD (Silicon Disk Drive type), a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk.

[0068] Additionally, the memory (110) may be separate from the device and may include a database connected wired or wirelessly. The database (200) illustrated in FIG. 2 may be implemented as a component of the memory (110).

[0069] The communication module (120) may include one or more components that enable communication with an external device, and may include, for example, at least one of a broadcast reception module, a wired communication module, a wireless communication module, a short-range communication module, and a location information module.

[0070] The wired communication module may include various wired communication modules such as a Local Area Network (LAN) module, a Wide Area Network (WAN) module, or a Value Added Network (VAN) module, as well as various cable communication modules such as a Universal Serial Bus (USB), a High Definition Multimedia Interface (HDMI), a Digital Visual Interface (DVI), RS-232 (recommended standard 232), power line communication, or plain old telephone service (POTS).

[0071] The wireless communication module may include a wireless communication module that supports various wireless communication methods such as GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, and 6G, in addition to a WiFi module and a Wireless Broadband module.

[0072] The display (130) displays (outputs) information or data processed in the device (100), data input or output through the AI ​​model (300), etc. In addition, the display (130) can display execution screen information of an application program (e.g., an application) running in the device (100), or UI (User Interface) or GUI (Graphical User Interface) information according to such execution screen information.

[0073] The input module (140) is for receiving information from a user. When a user inputs information through the input unit, the processor (150) can control the operation of the device (100) to correspond to the input information.

[0074] The input module (140) may include hardware physical keys (e.g., buttons located on at least one of the front, rear, and side of the device, dome switches, jog wheels, jog switches, etc.) and software touch keys. For example, the touch keys may be formed as virtual keys, soft keys, or visual keys displayed on a touchscreen type display (130) through software processing, or as touch keys placed on a part other than the touchscreen. Meanwhile, the virtual keys or visual keys may have various forms and be displayed on the touchscreen, and may be formed as, for example, graphics, text, icons, videos, or a combination thereof.

[0075] The processor (150) may be implemented as a memory that stores data on an algorithm for controlling the operation of components within the device (100) (including learning or executing an AI model) or a program that reproduces the algorithm, and at least one processor (not shown) that performs the aforementioned operation using the data stored in the memory. In this case, the memory and the processor may be implemented as separate chips, or may be implemented as a single chip.

[0076] In one embodiment, the system (1000) or device (100) according to the present invention may include at least one processor, and when including multiple processors, the multiple processors may be included in different devices (100).

[0077] In addition, the processor (150) can control any one or a combination of the components described above to implement various embodiments according to the present disclosure described below on the device (100).

[0078] FIG. 4 is a block diagram illustrating a method for evaluating the performance of a model that extracts information from a chart by de-rendering the chart according to an embodiment of the present invention.

[0079] Referring to FIG. 4, a test set (410) consisting of an image of a chart including text, lines, etc., and a ground truth (GT) including information about the chart are input into an image encoder (420). Here, the image of the chart includes a title, axis name, legend information, numbers, etc. The form of the chart (410) may be, but is not limited to, a vertical / horizontal bar chart, a line chart, a pie chart, an area chart, a scatter chart, a radar chart, a histogram, and / or a waterfall chart. In addition, the form of the chart may be a single form or a form in which multiple forms are combined.

[0080] The image encoder (420) follows a commonly used encoder architecture and serves to convert the input chart (410a) into a first embed (421) that can be processed by the AI ​​model (300). Here, the first embed (421) may include a concatenate embed.

[0081] Next, the first embed (421) output from the image encoder (420) is input to an AI model (400) including a decoder (430) that performs meta decoding, data decoding, etc. The decoder (430) decodes the input first embed (421) to extract meta information of the chart (e.g., chart title, X-axis name, Y-axis name, legend), data information (e.g., data point number, etc.), etc., and outputs this as a second embed (431).

[0082] In this way, the AI ​​model can output a data format (440) that includes meta information, data information, etc. from the output second embedding (431).

[0083] Next, the data format (440) output from the AI ​​model is input into the chart de-rendering performance evaluation model (450). The chart de-rendering performance evaluation model (450) compares the chart information (e.g., chart title, X-axis name, Y-axis name, legend, data point number, etc.) included in the data format (440) with the GT (Ground Truth) included in the data set (410), and calculates factor values ​​for deriving performance evaluation results (460) according to a predetermined method. The calculated factor values ​​reflect information about data indices, and may specifically include, but are not limited to, line accuracy (451), axis accuracy (452), meta accuracy (453), or length accuracy (454).

[0084] The performance evaluation model (450) outputs a performance evaluation result (460) indicating how accurately the AI ​​model predicts the chart by individually or comprehensively calculating the factor values. The performance evaluation result (460) may be expressed as a numerical score, a diagram expressed as an image, etc., but is not limited thereto. The performance evaluation result (460) may be output separately for each individual factor value, and the individual factor values ​​may include, but are not limited to, the line accuracy (451), axis accuracy (452), meta accuracy (453), or length accuracy (454) described above. For example, the performance evaluation result (460) may be expressed as the performance evaluation result (460a).

[0085] Through this, the present invention can quantitatively and objectively evaluate the characteristics of a model by separating and evaluating evaluation factors having orthogonality.

[0086] Below, the factor values ​​used to derive the performance evaluation results (460) are described in detail with reference to drawings 5 ​​to 7.

[0087] Line accuracy (451) is a factor indicating the degree to which the line of the chart output from the AI ​​model is close to the line of the chart included in the GT, and through this, it is possible to evaluate how accurately the AI ​​model predicts the shape of the chart. Specifically, line accuracy (451) means a factor that fixes the area (axis area) formed by the X-axis or Y-axis of the chart to a specific value (for example, fixes the X-axis*Y-axis area of ​​the chart to 1*1) and compares the area value with the area between the line of the output chart and the line of the chart included in the GT.

[0088] More specifically, if we look at an example comparing the charts output by AI model (A) and AI model (B) shown in Fig. 5a with the charts included in GT, we can see that model B predicts the chart shape more accurately than model A, because the lines of the chart output by AI are closer to the lines of the chart included in GT. However, if we calculate the area between the lines of the output chart and the lines of the chart included in GT using the X-axis and Y-axis values ​​of AI model (A) and AI model (B), the areas above are the same for the two models (i.e., the area indicated by the triangle shade in the chart). If we do not fix the area formed by the X-axis or Y-axis of the charts output by AI model (A) and AI model (B), the areas between the lines of the output chart and the lines of the chart included in GT cannot be compared under the same conditions due to the difference in the scale of the X-axis or Y-axis. Furthermore, this makes it difficult to compare which AI model predicts the chart shape more accurately.

[0089] In order to solve this problem, as shown in Fig. 5b, the area formed by the X-axis or Y-axis is fixed to a constant value (= 1*1) for the charts output from AI model (A) and AI model (B). In this fixed chart area (= 1), the line accuracy (451) can be calculated using the proportion of the area between the lines of the output chart and the lines of the chart included in the GT (the area indicated by the triangle shade in the chart). By calculating and comparing the line accuracy (451) for each AI model in this way, it is possible to compare under the same conditions how closely the lines of the chart output by each AI model are to the lines of the chart included in the GT (i.e., how small the proportion of the area spaced between the charts is in the total area).

[0090] More preferably, line accuracy (451) can be calculated using the following equation.

[0091]

[0092] Here, P represents the function followed by the output chart, and T represents the function followed by the chart included in GT. In addition, dom represents the area followed by each function (P and T). According to the above equation, the line accuracy (451) is calculated by subtracting the area formed between the line of the output chart and the line of the chart included in GT (the triangle shaded area) from the area formed by the X-axis or Y-axis of the chart ("1" in the case of Fig. 5b), and a model with a larger line accuracy (451) value can be evaluated as having a more accurate performance in predicting the shape of the chart. For example, in Figs. 5a and 5b, since the area between the charts (the triangle shaded area) compared to the total area (1) is smaller in model B than in model A, the line accuracy (451) value is larger, and accordingly, model B can be evaluated as predicting the shape of the chart more accurately than model A.

[0093] Through this, the present invention can objectively evaluate how accurately an AI model predicts the shape of a chart by indicating how closely the line of a chart output under the same conditions between AI models is to the line of a chart included in GT using line accuracy (451).

[0094] Meanwhile, axis accuracy (452) is a factor indicating the degree to which the distribution range of the chart output from AI overlaps with the distribution range of the chart included in GT on at least one axis of the chart, and can be used to evaluate whether the output chart predicts the chart within a sufficient range. Even if the output chart is predicted adjacent to the chart included in GT, if the range followed by the output chart is not wide, the AI ​​model may not be able to provide useful values ​​within a wide range. Therefore, it is necessary to evaluate whether the AI ​​model predicts the chart within a sufficient range using axis accuracy (452).

[0095] More specifically, as shown in Fig. 6, the portion (a) where the range of the X-axis where the charts output from AI are distributed and the range of the X-axis where the charts included in GT are distributed overlaps with the entire range of the X-axis where the charts output from AI are distributed and the range of the X-axis where the charts included in GT are distributed is calculated as the axis accuracy (452) value (i.e., the higher the axis accuracy (452) value, the more likely it is that the AI ​​model predicts charts in a wider range).

[0096] More preferably, line accuracy can be calculated by the following equation:

[0097]

[0098] Here, P represents the function followed by the output chart, and T represents the function followed by the chart included in GT. In addition, dom represents the area followed by each function (P and T), and m represents the Lebesgue measure. According to the above equation, the proportion of the overlapping part between the X-axis range of the chart output from AI and the X-axis range of the chart included in GT can be calculated as the axis accuracy (452).

[0099] Through this, the present invention can evaluate how widely an AI model can predict charts using axis accuracy (452).

[0100] Meanwhile, meta accuracy (453) is a factor that evaluates whether the meta information of the chart output from the AI ​​model has been correctly recognized. Meta information included in the output from the chart de-rendering AI model (e.g., chart title, axis labels, legend, etc.) plays a crucial role in providing users with insights using the output results. Therefore, it is necessary to evaluate how accurately the AI ​​model recognizes the meta information.

[0101] Preferably, meta accuracy (453) can be calculated using the Character Error Rate (CER). CER can be calculated as follows.

[0102]

[0103] Here, S represents the number of substitution characters, D represents the number of deletion characters, I represents the number of insertion characters, C represents the number of correct characters, and N represents the total number of reference characters (=S+D+C). Accordingly, a lower CER value indicates a better recognition rate, and a CER of 0 indicates that the meta information is perfectly recognized.

[0104] More specifically, meta accuracy (453) can be calculated using CER as follows.

[0105]

[0106] Accordingly, the higher the meta accuracy (453), the more accurately the AI ​​model can be evaluated to predict meta information.

[0107] Through this, the present invention can objectively evaluate how accurately an AI model recognizes meta information using meta accuracy (453).

[0108] Meanwhile, length accuracy (454) is a factor indicating whether the chart output from the AI ​​model deviates significantly from the chart included in GT at a specific point, and this can be used to evaluate how accurately the AI ​​model predicts the shape of the chart.

[0109] Specifically, referring to FIG. 7, the chart output from the AI ​​model (A) deviates significantly from the chart included in the GT at a certain point, whereas the AI ​​model (B) almost exactly follows the shape of the chart included in the GT. Accordingly, it can be evaluated that the AI ​​model (B) predicts the chart shape more accurately. Reflecting this point, in order to evaluate the performance of the AI ​​model more accurately, the length accuracy (454), which reflects the length difference between the chart output from the AI ​​model and the chart included in the GT, can be additionally considered in the performance evaluation results. That is, by assigning a higher length accuracy (454) as the length difference between the output chart and the chart included in the GT is smaller, it is possible to evaluate whether the AI ​​model predicts the chart shape evenly. For example, in FIG. 7, the chart length of the AI ​​model (B) is shorter than that of the AI ​​model (A), so the accuracy (454) value is larger, and accordingly, it can be evaluated that the AI ​​model (B) predicts the chart shape more evenly than the AI ​​model (A).

[0110] Preferably, the length accuracy (454) can be calculated using the following equation.

[0111]

[0112] Here, P represents the function followed by the output chart, and T represents the function followed by the chart included in GT. In addition, dom represents the area followed by each function (P and T). According to the above equation, the length accuracy (454) is calculated by subtracting 1 from the sum of the length of the output chart (Lp) and the length of the chart included in GT (Lt), and a model with a larger length accuracy (454) can be evaluated as having a more accurate performance in evenly predicting the shape of the chart.

[0113] Through this, the present invention can objectively evaluate how evenly an AI model predicts the shape of a chart using length accuracy (454).

[0114] Meanwhile, a method for evaluating the performance of a model that extracts information from a chart by de-rendering the chart according to embodiments of the present invention can be implemented by the system described with reference to FIG. 4.

[0115] AI models according to embodiments of the present invention can be controlled, executed, trained, driven, etc. by a processor, and thus, at least one of the tasks of executing, training, and driving the AI ​​models can be performed by at least one processor. Furthermore, the AI ​​models can be stored in memory, and feature data according to the present invention can also be stored in memory.

[0116] Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium storing computer-executable instructions. The instructions may be stored in the form of program code, and when executed by a processor, may generate program modules to perform the operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

[0117] Computer-readable storage media include all types of storage media that store instructions that can be deciphered by a computer. Examples include read-only memory (ROM), random access memory (RAM), magnetic tape, magnetic disks, flash memory, and optical data storage devices.

[0118] The disclosed embodiments have been described with reference to the attached drawings as described above. Those skilled in the art will understand that the present disclosure can be implemented in forms other than the disclosed embodiments without altering the technical spirit or essential features of the present disclosure. The disclosed embodiments are illustrative and should not be construed as limiting.

Claims

1. A system that implements a model for evaluating the accuracy of a chart de-rendering model. at least one processor; and At least one memory storing instructions or information that cause at least one processor to perform an operation, The action performed by the above command is: A step of inputting an image of a chart stored in a test set into an AI model and outputting a data format that predicts information of the chart; and A step of inputting the above data format into a performance evaluation model, comparing the information of the data format with GT (Ground Truth), which is information about the chart stored in the test set, and outputting a performance evaluation result for the AI model; The above performance evaluation results are: Line accuracy, which indicates the degree to which a chart according to the information of the above data format is close to a chart according to the above GT; A system comprising an axis accuracy, which indicates the extent to which the range of distribution of the chart according to the information of the data format and the range of distribution of the chart according to the GT overlap in at least one axis of the chart.

2. In claim 1, The above line accuracy is, A system that fixes the axis area formed by the X-axis or Y-axis of a chart according to the information included in the data format and the chart according to the GT to a specific value, and compares the axis area with the area between the line of the chart according to the information included in the data format and the line of the chart according to the GT.

3. In claim 1, The above degree of overlap of the above axis accuracy is, A system in which the portion where the range of the X-axis in which the chart is distributed according to the information included in the above data format overlaps with the range of the X-axis in which the chart is distributed according to the above GT is a proportion of the entire range of the X-axis in which the chart is distributed according to the information included in the above data format and the range of the X-axis in which the chart is distributed according to the above GT.

4. In claim 1, The above performance evaluation results additionally include meta-accuracy, The above meta accuracy is, A system in which the meta information included in the above data format matches the meta information included in the GT.

5. In claim 4, The above meta accuracy is calculated using the character error rate (CER) of the system.

6. In claim 1, The above performance evaluation results additionally include length accuracy, The above length accuracy is, A system in which the length of the chart according to the information contained in the above data format is the difference between the length of the chart according to the above GT.

7. A method for implementing a model for evaluating the accuracy of a chart de-rendering model, The images of the charts stored in the test set are input to the image encoder to be converted into the first embedding that the AI model can process; By inputting the first embed into the AI model, at least one data format including meta information or data information included in the image of the chart is output; A method for inputting the above data format into a performance evaluation model, comparing the information included in the data format with GT (Ground Truth) containing information about the chart stored in the test set, and outputting a performance evaluation result; The above performance evaluation results are: Line accuracy, which indicates the degree to which a line of a chart according to the information contained in the above data format is close to a line of a chart according to the above GT; A method, wherein the axis accuracy is determined according to the extent to which the range over which the chart is distributed according to the information included in the data format overlaps the range over which the chart is distributed according to the GT, on at least one axis of the chart.

8. In claim 7, The above line accuracy is, A method for fixing an axis area formed by the X-axis or Y-axis of a chart according to the information included in the above data format and a chart according to the above GT to a specific value, and comparing the axis area with an area between a line of a chart according to the information included in the above data format and a line of a chart according to the above GT.

9. In claim 7, The above degree of overlap of the above axis accuracy is, A method in which the portion where the range of the X-axis in which the chart is distributed according to the information included in the above data format overlaps with the range of the X-axis in which the chart is distributed according to the above GT is a proportion of the entire range of the X-axis in which the chart is distributed according to the information included in the above data format and the range of the X-axis in which the chart is distributed according to the above GT.

10. In claim 7, The above performance evaluation results additionally include meta-accuracy, The above meta accuracy is, A method in which the meta information included in the above data format matches the meta information included in the GT.

11. In claim 10, The above meta accuracy is calculated using the character error rate (CER).

12. In claim 7, The above performance evaluation results additionally include length accuracy, The above length accuracy is, A method wherein the length of the chart is the difference between the length of the chart according to the information included in the above data format and the length of the chart according to the above GT.

13. A program stored in a computer-readable recording medium to execute any one of the methods of claims 7 to 12, in combination with a computer.

Images