System, method, and program for training or performance evaluation of chart de-rendering artificial intelligence model by using data set including constructed chart information

Abstract

A system, a method, and a program for evaluating the performance of an artificial intelligence model for de-rendering a chart or training the artificial intelligence model by constructing a data set including chart information are disclosed. The system includes: a memory including a data set generation model for storing line information, which is information on at least one line of a chart, and meta information, which is information on meta data, as a ground truth (GT), storing an image formed by using the GT as a chart image, and outputting the GT and the chart image as a data set, and an AI model for outputting a data format obtained by inputting the chart image stored in the data set and predicting information of the chart; and a processor for executing or training the AI model and executing a performance evaluation model. Factors included in the line information may include an X-axis value of a chart line, a function, a coefficient of the function, a color or shape of a line or point, and the like, and factors included in the meta information may include a chart title, an X-axis name, a Y-axis name, a legend, and the like. Furthermore, the data format output from the AI model may be used to evaluate the performance of the AI model or train the AI model.

Description

A system, method, and program for evaluating or learning the performance of a chart de-rendering artificial intelligence model using a data set containing constructed chart information.

[0001] The present invention relates to a system, method, and program for evaluating the performance of a chart de-rendering model or training a model by constructing a data set including chart information, and more particularly, the present invention relates to a system, method, and program for evaluating the performance of an artificial intelligence model for de-rendering a chart or training an artificial intelligence model by constructing a data set including chart information.

[0002] Chart de-rendering is the opposite process of chart rendering. It involves analyzing and grouping visual patterns or information in a chart to extract key information, and then extracting information about the data (e.g., numbers, groups, etc.) and information about the chart layout.

[0003] For chart de-rendering, a data set consisting of images of charts with text, lines, and other information, as well as ground truth (GT) data containing chart information, is input to the AI ​​model. The data set must be refined enough for the AI ​​model to recognize, and it must include diverse data styles to enable multi-faceted evaluation and training of the AI ​​model.

[0004] A conventional method for building datasets involved collecting data by crawling specific websites (PlotQA: Reasoning over Scientific Plots, Nitesh Methani et al., 2020).

[0005] This method had limitations in the amount of data included in the dataset, and often only collected data written in a specific style, resulting in a lack of diversity in the collected data.

[0006] Furthermore, the chart de-rendering AI model recognizes meta information in addition to the numerical information contained in the data, but there is also a problem that the data collected through crawling often does not fully include meta information (e.g., names of the X-axis and Y-axis, entity group names recorded in the legend, etc.).

[0007] In addition, in order to evaluate or train the AI ​​model more accurately, it is necessary to use a method of dividing chart images drawn in a similar style into an experimental group and a control group and comparing the results. However, there is a problem that it is impossible to use the above comparison method because it is difficult to collect chart images in a similar style according to the conventional data set construction method.

[0008] Prior art literature Non-patent literature PlotQA: Reasoning over Scientific Plots, Nitesh Methani et al., 2020.

[0009] The problem to be solved by the present invention is to provide a system, method and program for evaluating the performance of an artificial intelligence model for de-rendering a chart or training an artificial intelligence model by constructing a large data set containing chart information.

[0010] The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0011] A system for implementing a chart de-rendering model of the present invention comprises at least one processor, and at least one memory for storing instructions or information for causing the at least one processor to perform operations, wherein the instructions perform operations including: storing line information, which is information about at least one line of a chart, and meta information, which is information about meta data, as GT (Ground Truth) through a data set generation model, storing an image formed using the GT as a chart image, and outputting the GT and the chart image as a data set; inputting an image of a chart stored in the data set into an AI model, and outputting a data format that predicts information about the chart; and inputting the data format into a performance evaluation model, and outputting a performance evaluation result for the AI ​​model by comparing information about the data format with the GT stored in the data set; wherein values ​​applied to each of the factors included in the line information and the factors included in the meta information are each selected from among predetermined values.

[0012] In the above system, the step of inputting the data format output from the AI ​​model into a performance evaluation model, comparing information of the data format with the GT stored in the data set, and outputting a performance evaluation result for the AI ​​model may be further included.

[0013] In the above system, the step of training the AI ​​model by comparing the information in the data format output from the AI ​​model with the GT stored in the data set through the AI ​​model and using the comparison result may be further included.

[0014] In the above system, the factors included in the line information may be configured to include an X-axis value of the chart line, a function, and a coefficient of the function.

[0015] In the above system, the data set is composed of a first data set and a second data set, and the coefficient of the function included in the second data set may differ from the coefficient of the function included in the first data set by less than a preset value.

[0016] In the above system, the maximum value of the value of the function may be greater than a preset maximum function value, and the minimum value of the value of the function may be less than a preset minimum function value.

[0017] In the above system, the factors included in the line information may be configured to include the color or shape of the line or point.

[0018] In the above system, the factors included in the meta information can be configured to include a chart title, an X-axis name, a Y-axis name, and a legend.

[0019] In the above system, the value applied to each of the above factors can be selected according to a predetermined probability for each predetermined value.

[0020] A method for implementing a chart de-rendering model according to another aspect of the present invention is a method for storing line information, which is information about at least one line of a chart, and meta information, which is information about meta data, as GT (Ground Truth) through a data set generation model, storing an image formed using the GT as a chart image, outputting the GT and the chart image as a data set, inputting the image of the chart stored in the data set into an AI model, outputting a data format in which information of the chart is predicted, and inputting the data format into a performance evaluation model, comparing information of the data format with the GT stored in the data set, and outputting a performance evaluation result for the AI ​​model, wherein a value applied to each of a factor included in the line information and a factor included in the meta information is selected from among predetermined values.

[0021] In the above method, the method may further include a step of inputting the data format output from the AI ​​model into a performance evaluation model, comparing information of the data format with the GT stored in the data set, and outputting a performance evaluation result for the AI ​​model.

[0022] In the above method, the method may further include a step of training the AI ​​model by comparing the information in the data format output from the AI ​​model with the GT stored in the data set, and using the comparison result.

[0023] In the above method, the factors included in the line information may be configured to include an X-axis value of the chart line, a function, and a coefficient of the function.

[0024] In the above method, the data set is composed of a first data set and a second data set, and the coefficient of the function included in the second data set may differ from the coefficient of the function included in the first data set by less than a preset value.

[0025] In the above method, the maximum value of the value of the function may be greater than a preset maximum function value, and the minimum value of the value of the function may be less than a preset minimum function value.

[0026] In the above method, the factor included in the line information may be configured to include the color or shape of the line or point.

[0027] In the above method, the factors included in the meta information may be configured to include a chart title, an X-axis name, a Y-axis name, and a legend.

[0028] In the above method, the value applied to each of the above factors can be selected according to a predetermined probability for each predetermined value.

[0029]

[0030] According to another aspect of the present invention, a program may be stored in a computer-readable recording medium to be combined with a computer and implement a chart de-rendering model according to embodiments of the present invention.

[0031] According to the present invention, a data set that fully includes information about a chart can be derived, chart images formed in various styles with a high degree of freedom can be generated in large quantities using factors of line information and meta information, and an AI model can be more accurately evaluated or trained by distinguishing an experimental group and a control group from the output results using a method of appropriately selecting factors of line information and meta information.

[0032] Additionally, according to the present invention, a large number of data sets including charts of similar shapes can be generated in order to evaluate whether an AI model can accurately recognize a chart of a specific shape or to train an AI model to accurately recognize a chart of a specific shape.

[0033] Additionally, according to the present invention, the shape of a chart included in a data set can be prevented from being distorted so that an AI model can be accurately evaluated or an AI model can be efficiently trained.

[0034] Additionally, according to the present invention, a data set including charts with various lines or points can be output in large quantities to accurately evaluate an AI model or enable an AI model to learn efficiently.

[0035] In addition, according to the present invention, in order to accurately evaluate whether an AI model accurately recognizes meta information included in a chart image or to train an AI model to accurately recognize meta information included in a chart image, a data set including complete meta information can be output in large quantities.

[0036] The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0037] FIG. 1 is a schematic diagram of a system for implementing a method of evaluating the performance of an artificial intelligence model for de-rendering a chart or training an artificial intelligence model by constructing a data set including chart information according to one embodiment of the present disclosure.

[0038] FIG. 2 is a block diagram illustrating a configuration of a device for evaluating the performance of an artificial intelligence model for de-rendering a chart or training an artificial intelligence model by constructing a data set including chart information according to one embodiment of the present disclosure.

[0039] FIG. 3 is a block diagram illustrating a method for evaluating the performance of an artificial intelligence model for de-rendering a chart or training an artificial intelligence model by constructing a data set including chart information according to embodiments of the present invention.

[0040] Figures 4 to 6 are examples of chart images output from a data set generation model according to embodiments of the present invention.

[0041] The following examples are provided as examples to ensure that those skilled in the art can fully grasp the spirit of the present invention. Therefore, the present invention is not limited to the embodiments described below and may be embodied in other forms.

[0042] Throughout the present invention, the same reference numerals denote the same components. The present invention does not describe all elements of the embodiments, and any content that is general in the technical field to which the present invention pertains or that overlaps between the embodiments is omitted. The terms 'part, module, element, block' used in the specification may be implemented in software or hardware, and depending on the embodiments, multiple 'parts, modules, elements, blocks' may be implemented as a single component, or a single 'part, module, element, block' may include multiple components.

[0043] Throughout the specification, when a part is said to be "connected" to another part, this includes not only direct connection but also indirect connection, and indirect connection includes connection via a wireless communication network.

[0044] Additionally, when a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

[0045] Throughout the specification, when we say that an element is "on" another element, this includes not only cases where the element is in contact with the other element, but also cases where another element exists between the two elements.

[0046] The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

[0047] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0048] The identification codes for each step are used for convenience of explanation and do not describe the order of each step. Each step may be performed in a different order than specified unless the context clearly indicates a specific order.

[0049] A system for evaluating the performance of an artificial intelligence model for de-rendering a chart by constructing a data set including chart information according to the present invention or for training an artificial intelligence model may include a device, and the device may include any of various devices capable of performing computational processing and providing results to a user. For example, a system for evaluating the performance of an artificial intelligence model for de-rendering a chart by constructing a data set including chart information according to the present invention or for training an artificial intelligence model may include at least one of a computer, a server device, and a portable terminal, or may be any form having the same or similar functions as these. However, the present invention is not limited thereto.

[0050] Here, the computer may include, for example, a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

[0051] The above server device is a server that processes information by communicating with an external device, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, and a web server.

[0052] The above portable terminal may include, for example, a wireless communication device that ensures portability and mobility, and may include all kinds of handheld-based wireless communication devices such as a PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, a smart phone, and a wearable device such as a watch, a ring, a bracelet, an anklet, a necklace, glasses, contact lenses, or a head-mounted device (HMD).

[0053] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0054] The present invention relates to a system, method, and program for evaluating the performance of a chart de-rendering model or training a model by constructing a data set including chart information, and more particularly, the present invention relates to a system, method, and program for evaluating the performance of an artificial intelligence model for de-rendering a chart or training an artificial intelligence model by constructing a data set including chart information.

[0055] FIG. 1 is a schematic diagram of a system for evaluating the performance of a chart de-rendering model or training a model by constructing a data set including chart information according to one embodiment of the present invention.

[0056]

[0057] *As illustrated in FIG. 1, the system (1000) may include a device (100), a database (200), a data set generation model (300), an AI model (400), and a performance evaluation model (500).

[0058] The device (100), database (200), data set generation model (300), AI model (400), and performance evaluation model (500) included in the system (1000) can communicate via a network (W). Here, the network (W) can include a wired network and a wireless network. For example, the network can include various networks such as a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN).

[0059] Additionally, the network (W) may include the well-known World Wide Web (WWW). However, the network (W) according to an embodiment of the present invention is not limited to the networks listed above, and may include at least part of a well-known wireless data network, a well-known telephone network, or a well-known wired / wireless television network.

[0060] The device (100) can input a data set generated from a data set generation model (300) into an AI model (400) and output information about a chart. Based on the information about the output chart, the performance evaluation model (500) can evaluate the performance of the AI ​​model (400), and the AI ​​model (400) can perform learning.

[0061] The data set generation model (300) generates a data set (410) to be de-rendered as a chart through AI, and inputs this into the AI ​​model (400). The data set (410) is the same as that generally used to train an AI model or evaluate its performance, but is not limited thereto. The data set generated through the data set generation model (300) is composed of an image of a chart including text, lines, etc., a GT (Ground Truth) including information about the chart, etc. The data set includes a train set used to evaluate the accuracy of the AI ​​model through the result of chart de-rendering using artificial intelligence, and a test set used to train (e.g., deep learning, etc.) an AI model used for chart de-rendering.

[0062] The charts included in the data set may be in the form of vertical / horizontal bar charts, line charts, pie charts, area charts, scatter charts, radar charts, histograms, and / or waterfall charts. The chart may be in a single form or may be a form combining multiple forms. However, the charts used in the present invention are not limited thereto, and the charts may include any form of charts. In addition to information that is an image of numbers, etc. (e.g., line, circle, etc.), the charts may include text information, and specifically, may include annotation information of the chart, a legend or title of the chart, names of each axis (e.g., X-axis, Y-axis, Z-axis, etc.), raw data values ​​of points included in the chart (e.g., numerical values ​​of the X-axis and Y-axis of points included in the chart), etc.

[0063] Information about a chart output from the AI ​​model (400) may include meta information, data information, etc., as information indicating the characteristics of the chart. Meta information may be the names of the X-axis and Y-axis, entity group names recorded in a legend, etc. Data information may be the numeric values ​​of the X-axis and / or Y-axis of each entity, etc.

[0064] Information about the chart output from the AI ​​model (400) can be input into the performance evaluation model (500), and the performance evaluation model (500) evaluates the performance of the AI ​​model (400) by comparing the GT included in the data set with the information about the output chart.

[0065] The performance evaluation results output from the performance evaluation model (500) indicate how accurately the AI ​​model predicts charts. The performance evaluation results may be expressed as numerical scores, graphical diagrams, or other representations, but are not limited thereto.

[0066] Additionally, the device (100) can compare information about a chart output from the AI ​​model (400) with information about a chart stored in a data set, and the AI ​​model (400) can perform learning using the comparison result.

[0067] The database (200) may store various data (e.g., data sets) for training or evaluating the performance of the AI ​​model (400). Furthermore, the database (200) may store chart images, information about charts, information about performance evaluation methods, information about data set creation methods, and the like. In various embodiments, the database may also store output data generated by the AI ​​model (400). However, the system (1000) may not include the database (200) if training of the AI ​​model (400) is complete.

[0068] FIG. 1 illustrates a case where a database (200) is implemented outside of a device (100). In this case, the database (200) may be connected to the device (100) via wired or wireless means. However, this is merely an example, and the database (200) may also be implemented as a component of the device (100).

[0069] FIG. 1 illustrates a case where the AI ​​model (400) is implemented outside the device (100) (e.g., cloud-based), but is not limited thereto, and may be implemented as a component in the device (100).

[0070] FIG. 2 is a block diagram illustrating a configuration of a device for evaluating the performance of an artificial intelligence model for de-rendering a chart or training an artificial intelligence model by constructing a data set including chart information according to one embodiment of the present invention.

[0071] As illustrated in FIG. 2, the device (100) may include a memory (110), a communication module (120), a display (130), an input module (140), and a processor (150). However, the present invention is not limited thereto, and the device (100) may have its software and hardware configurations modified / added / omitted within a range apparent from a perspective of ordinary skill in the art, depending on the required operation. In addition, the device (100) may be replaced with a system, and the device (100) may include a plurality of devices, in which case each component included in the device (100) may be included in at least one of the plurality of devices.

[0072] The memory (110) can store data supporting various functions of the device (100), programs for the operation of the processor (150), input / output data, and a plurality of application programs or applications run on the device, data for the operation of the device (100), commands, and AI models. At least some of these application programs can be downloaded from an external server via wireless communication.

[0073] The memory (110) may include at least one type of storage medium among a flash memory type, a hard disk type, an SSD (Solid State Disk type), an SDD (Silicon Disk Drive type), a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk.

[0074] Additionally, the memory (110) may be separate from the device and may include a database connected wired or wirelessly. The database (200) illustrated in FIG. 1 may be implemented as a component of the memory (110).

[0075] The communication module (120) may include one or more components that enable communication with an external device, and may include, for example, at least one of a broadcast reception module, a wired communication module, a wireless communication module, a short-range communication module, and a location information module.

[0076] The wired communication module may include various wired communication modules such as a Local Area Network (LAN) module, a Wide Area Network (WAN) module, or a Value Added Network (VAN) module, as well as various cable communication modules such as a Universal Serial Bus (USB), a High Definition Multimedia Interface (HDMI), a Digital Visual Interface (DVI), RS-232 (recommended standard 232), power line communication, or plain old telephone service (POTS).

[0077] The wireless communication module may include a wireless communication module that supports various wireless communication methods such as GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, and 6G, in addition to a WiFi module and a Wireless Broadband module.

[0078] The display (130) displays (outputs) information or data processed in the device (100), data input or output through the AI ​​model (400), etc. In addition, the display (130) can display execution screen information of an application program (e.g., an application) running in the device (100), or UI (User Interface) or GUI (Graphical User Interface) information according to such execution screen information.

[0079] The input module (140) is for receiving information from a user. When a user inputs information through the input unit, the processor (150) can control the operation of the device (100) to correspond to the input information.

[0080] The input module (140) may include hardware physical keys (e.g., buttons located on at least one of the front, rear, and side of the device, dome switches, jog wheels, jog switches, etc.) and software touch keys. For example, the touch keys may be formed as virtual keys, soft keys, or visual keys displayed on a touchscreen type display (130) through software processing, or as touch keys placed on a part other than the touchscreen. Meanwhile, the virtual keys or visual keys may have various forms and be displayed on the touchscreen, and may be formed as, for example, graphics, text, icons, videos, or a combination thereof.

[0081] The processor (150) may be implemented as a memory that stores data on an algorithm for controlling the operation of components within the device (100) (including learning or executing an AI model) or a program that reproduces the algorithm, and at least one processor (not shown) that performs the aforementioned operation using the data stored in the memory. In this case, the memory and the processor may be implemented as separate chips, or may be implemented as a single chip.

[0082] In one embodiment, the system (1000) or device (100) according to the present invention may include at least one processor, and when including multiple processors, the multiple processors may be included in different devices (100).

[0083] In addition, the processor (150) can control any one or a combination of the components described above to implement various embodiments according to the present disclosure described below on the device (100).

[0084] FIG. 3 is a block diagram illustrating a method for evaluating the performance of a chart de-rendering model or training a model by constructing a data set including chart information according to an embodiment of the present invention.

[0085] Referring to FIG. 3, a data set (410) is generated through a data set generation model (300) that includes an image of a chart including text, lines, etc., and a ground truth (GT) including information about the chart. Here, the image of the chart includes a title, an axis name, legend information, numbers, etc. The form of the chart image included in the data set (410) may be, but is not limited to, a vertical / horizontal bar chart, a line chart, a pie chart, an area chart, a scatter chart, a radar chart, a histogram, and / or a waterfall chart. In addition, the form of the chart may be a single form or a form in which multiple forms are combined.

[0086] Here, the GT of the data set generated by the data set generation model (300) includes line information, which is information about the line formed in the chart (e.g., X-axis value of the chart line, function, coefficient of the function, maximum / minimum function value, color of the line or point, shape of the line or point, etc.), and meta information, which is information about meta data (e.g., chart title, X-axis name, Y-axis name, legend, etc.).

[0087] At this time, the arguments included in the line information and the arguments included in the meta information are selected from among the values ​​that are predefined for each argument.

[0088] For example, among the arguments included in the line information, the "function" argument can be configured to be selected as the sum of one or more functions from among categories such as a polynomial function, a sinusoidal function, a Gaussian function, an exponential function, a logarithm function, a Pareto function, and a random function. In particular, by using multiple sinusoidal functions, all periodic functions can be expressed through a Fourier series (more specifically, the periodic function shown in Fig. 4a can also be expressed using multiple sinusoidal functions).

[0089] In addition, by setting a category of color or shape of a line or point and allowing selection from among these, a chart can be created in which dotted lines and solid lines appear simultaneously, as shown in Fig. 4b, or a chart in which lines with triangular points and lines with square points appear simultaneously.

[0090] Through this, the present invention can derive a data set that completely includes information about a chart, can mass-produce chart images formed in various styles with a high degree of freedom by using factors of line information and meta information, and can more accurately evaluate or train an AI model by distinguishing an experimental group and a control group from the output results by using a method of appropriately selecting factors of line information and meta information.

[0091] Next, the data set (410) generated through the data set generation model (300) is input to the image encoder (420). The image encoder (420) follows a commonly used encoder architecture and serves to convert the input chart (410a) into a first embedding (421) that can be processed by the AI ​​model (400).

[0092] Next, the first embed (421) output from the image encoder (420) is input to an AI model (400) including a decoder (430) that performs meta decoding, data decoding, etc. The decoder (430) decodes the input first embed (421) to extract meta information of the chart (e.g., chart title, X-axis name, Y-axis name, legend), data information (e.g., data point number, etc.), etc., and outputs this as a second embed (431).

[0093] In this way, the AI ​​model can output a data format (440) that includes meta information, data information, etc., i.e., a data format that predicts information of the chart, from the output second embedding (431).

[0094] Next, the data format (440) output from the AI ​​model can be input into the chart de-rendering performance evaluation model (500). The chart de-rendering performance evaluation model (500) compares the chart information (e.g., chart title, X-axis name, Y-axis name, legend, data point number, etc.) included in the data format (440) with the GT (Ground Truth) included in the data set (410), and derives a performance evaluation result (460) according to a predetermined method.

[0095] Meanwhile, the AI ​​model (400) can perform learning using the result of comparing the output data format (440) with information about the chart stored in the data set. Learning of the AI ​​model (400) can be performed using hardware resources such as a central processing unit (CPU) and / or a graphics processing unit (GPU). The process of training the AI ​​model (400) can be performed through a loss function that measures the difference between the output chart information and the GT stored in the data set. The loss function can vary depending on the type of task performed by the AI ​​model (400) or the characteristics of the AI ​​model (400) itself. Mean Squared Error (MSE), Cross-Entropy Loss, etc. can be used as the loss function, but are not limited thereto. Since the AI ​​model (400) can perform better predictions if learning is performed so as to minimize the calculated loss, the AI ​​model is optimized by adjusting the weights of the model in a direction that minimizes the calculated loss. This optimization process can be accomplished using the backpropagation algorithm.

[0096] Below, with reference to FIGS. 5 and 6, the factors included in the line information and meta information used in the data set generation model (300) are described in detail.

[0097] First, in order to evaluate whether the AI ​​model can accurately recognize a specific type of chart, or to train the AI ​​model to accurately recognize a specific type of chart, it is necessary to repeatedly input chart images of similar types into the AI ​​model. To this end, the line information arguments may be configured to include the X-axis value of the chart line, the function, the coefficient of the function, etc., and the data set generation model (300) may be configured to output multiple data sets (410) such that the "function" argument remains the same and only the "function coefficient" argument differs by less than a preset value. In this way, multiple data sets (410) in which the "function" argument is the same and only the "function coefficient" is partially different represent charts in which the lines of the charts are similar but do not exactly match. For example, as shown in Fig. 5, charts (A) and (B) may represent similar charts in which the "functions" are the same, but represent charts in which the "function coefficients" are different and do not exactly match.

[0098] Through this, the present invention can generate a large number of data sets including charts of similar shapes in order to evaluate whether an AI model can accurately recognize a chart of a specific shape or to train an AI model to accurately recognize a chart of a specific shape.

[0099] On the other hand, if the range of values ​​of the function included in the data set accounts for a small proportion of the Y values ​​of the entire chart, the shape of the line may not be displayed well. For example, referring to Fig. 6, the function values ​​of line (A) are distributed from 230 to 340, and line (B) is distributed from 380 to 420, so the function values ​​of line (B) account for a smaller proportion of the overall Y values ​​than the function values ​​of line (A). As a result, line (B) appears compressed compared to line (A), and its shape is distorted. To solve this problem, the maximum value of the function of the line can be configured to be greater than the preset maximum function value, and the minimum value of the function of the line can be configured to be less than the preset minimum function value. For example, referring to Fig. 6, if the preset minimum function value is 230 and the preset maximum function value is 340, the minimum / maximum function values ​​of lines (A) and (B) are located outside the range of 230 to 340, so the line shape is not displayed as being excessively compressed. In other words, the function values ​​of the lines of the chart can be distributed more widely than the range of a certain Y value, so that the proportion of the function values ​​of the chart in the Y values ​​of the entire chart is small, which can prevent the phenomenon of the lines being distorted into a compressed shape.

[0100] Through this, the present invention can prevent the distortion of the charts included in the data set so that the AI ​​model can be accurately evaluated or the AI ​​model can learn efficiently.

[0101] Meanwhile, lines included in a chart can be formed as dotted or straight lines, and their colors can also vary. Furthermore, the shapes of points expressed in the lines can be expressed in various shapes, such as circles, squares, triangles, etc. To enable a data set (410) containing various charts to be output and input into an AI model, the color or shape of the lines or points can be included as factors in the line information of the data set.

[0102] Through this, the present invention can output a large amount of data sets including charts with various lines or points so that the AI ​​model can be accurately evaluated or the AI ​​model can learn efficiently.

[0103] Meanwhile, the AI ​​model recognizes not only numerical information contained in the data but also meta information from the input chart image. In order to evaluate or train such an AI model, it is necessary to input a large data set that fully includes meta information about the chart (e.g., names of the X-axis and Y-axis, entity group names recorded in the legend, etc.). To this end, the data generation model (300) stores meta information about the chart as GT in addition to line information in the data set (410). Here, the factors included in the meta information may be a chart title, X-axis name, Y-axis name, legend, etc., and the values ​​applied to each meta information factor may be selected from among predefined values. For example, among the meta information factors, the values ​​used for the name of the X-axis are predefined as population, year, blood volume, blood sugar, fiscal year, etc., and specific values ​​are selected and reflected through the data generation model.

[0104] Through this, the present invention can output a large amount of data sets containing complete meta information so as to evaluate whether an AI model accurately recognizes meta information contained in a chart image or train an AI model to accurately recognize meta information contained in a chart image.

[0105] Meanwhile, it may be necessary to intensively evaluate the recognition performance of an AI model for charts with specific factors, or to intensively train the AI ​​model to improve its recognition performance for charts with specific factors. In this case, it is necessary to intensively output charts with specific factors from the data set generation model (300) and input them into the AI ​​model. To this end, the values ​​applied to each factor used in the data set generation model (300) may be selected from predetermined values, and each predetermined value may be selected according to a certain probability. For example, if it is necessary to intensively output a "Gaussian function" in relation to the "function" among the factor values, a higher probability may be assigned to the "Gaussian function" compared to other "function" values ​​(e.g., polynomial function, exponential function, logarithmic function, etc.), so that the data set may be configured to include a large number of charts related to the "Gaussian function."

[0106] Through this, the present invention can generate a large number of chart images of various styles with a high degree of freedom while intensively generating a particularly necessary data set.

[0107] Meanwhile, a method for evaluating the performance of a chart de-rendering model or training a model by constructing a data set including chart information according to embodiments of the present invention can be implemented by the system described with reference to FIG. 1.

[0108] AI models according to embodiments of the present invention can be controlled, executed, trained, driven, etc. by a processor, and thus, at least one of the tasks of executing, training, and driving the AI ​​models can be performed by at least one processor. Furthermore, the AI ​​models can be stored in memory, and feature data according to the present invention can also be stored in memory.

[0109] Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium storing computer-executable instructions. The instructions may be stored in the form of program code, and when executed by a processor, may generate program modules to perform the operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

[0110] Computer-readable storage media include all types of storage media that store instructions that can be deciphered by a computer. Examples include read-only memory (ROM), random access memory (RAM), magnetic tape, magnetic disks, flash memory, and optical data storage devices.

[0111] The disclosed embodiments have been described with reference to the attached drawings as described above. Those skilled in the art will understand that the present disclosure can be implemented in forms other than the disclosed embodiments without altering the technical spirit or essential features of the present disclosure. The disclosed embodiments are illustrative and should not be construed as limiting.

Claims

1. As a system that implements a chart de-rendering model, at least one processor; and At least one memory storing instructions or information that cause at least one processor to perform an operation, The action performed by the above command is: A step of storing line information, which is information about at least one line of a chart, and meta information, which is information about meta data, as GT (Ground Truth) through a data set generation model, storing an image formed using the GT as a chart image, and outputting the GT and the chart image as a data set; A step of inputting an image of a chart stored in the above data set into an AI model and outputting a data format that predicts information of the chart; and A step of inputting the above data format into a performance evaluation model, comparing the information of the above data format with the GT stored in the above data set, and outputting a performance evaluation result for the AI model; A system in which the values applied to each of the factors included in the above line information and the factors included in the above meta information are each selected from among predetermined values.

2. In claim 1, A system further comprising a step of inputting the data format output from the AI model into a performance evaluation model, comparing information of the data format with the GT stored in the data set, and outputting a performance evaluation result for the AI model.

3. In claim 1, A system further comprising a step of training an AI model by comparing information in the data format output from the AI model with the GT stored in the data set, and using the comparison result.

4. A system according to claim 1, wherein the factors included in the line information are configured to include an X-axis value of the chart line, a function, and a coefficient of the function.

5. In claim 4, The above data set consists of a first data set and a second data set, A system in which the coefficients of the function included in the second data set differ from the coefficients of the function included in the first data set by less than a preset value.

6. In claim 4, A system in which the maximum value of the above function is greater than a preset maximum function value, and the minimum value of the above function is less than a preset minimum function value.

7. A system according to claim 1, wherein the factor included in the line information is configured to include the color or shape of the line or point.

8. A system according to claim 1, wherein the factors included in the meta information are configured to include a chart title, an X-axis name, a Y-axis name, and a legend.

9. In claim 2, 7, or 8, A system in which the values applied to each of the above factors are selected according to a predetermined probability for each predetermined value.

10. In implementing a chart de-rendering model, Through a data set generation model, line information, which is information about at least one line of a chart, and meta information, which is information about meta data, are stored as GT (Ground Truth), an image formed using the GT is stored as a chart image, and the GT and the chart image are output as a data set; By inputting an image of a chart stored in the above data set into an AI model, a data format predicting information of the chart is output; A method for inputting the above data format into a performance evaluation model, comparing the information of the above data format with the GT stored in the above data set, and outputting a performance evaluation result for the AI model; A method in which the values applied to each of the factors included in the above line information and the factors included in the above meta information are each selected from among predetermined values.

11. In claim 10, A method further comprising: a step of inputting the data format output from the AI model into a performance evaluation model, comparing information of the data format with the GT stored in the data set, and outputting a performance evaluation result for the AI model.

12. In claim 10, A method further comprising: a step of training an AI model by comparing information in the data format output from the AI model with the GT stored in the data set, and using the comparison result.

13. A method according to claim 10, wherein the factors included in the line information are configured to include an X-axis value of the chart line, a function, and a coefficient of the function.

14. In claim 13, The above data set consists of a first data set and a second data set, A method wherein the coefficient of the function included in the second data set differs from the coefficient of the function included in the first data set by less than a preset value.

15. In claim 13, A method wherein the maximum value of the value of the above function is greater than a preset maximum function value, and the minimum value of the value of the above function is less than a preset minimum function value.

16. A method according to claim 10, wherein the factor included in the line information is configured to include the color or shape of the line or point.

17. A method according to claim 10, wherein the factors included in the meta information are configured to include a chart title, an X-axis name, a Y-axis name, and a legend.

18. In claim 13, 16, or 17, A method in which the values applied to each of the above factors are selected according to a predetermined probability for each predetermined value.

19. A program stored in a computer-readable recording medium for executing the method of any one of claims 11 to 18, in combination with a computer.

Images