A system, method, and program for evaluating the performance or learning a chart de-rendering artificial intelligence model using a dataset containing constructed chart information.

Abstract

A system, method, and program are disclosed for evaluating the performance of an artificial intelligence model that constructs a dataset containing chart information and de-renders the chart, or for training an artificial intelligence model. The system includes a memory containing a dataset generation model that stores line information, which is information about at least one line of a chart, and metadata information, which is information about metadata, as GT (Ground Truth), stores an image formed using the GT as a chart image, and outputs the GT and the chart image as a dataset; and an AI model that takes the chart image stored in the dataset as input and outputs a data format that predicts the information of the chart; and a processor that runs or trains the AI ​​model and runs a performance evaluation model. Factors included in the line information may include the X-axis value of the chart line, the function, and the coefficient of the function, the color or shape of the line or point, and factors included in the metadata may include the chart title, X-axis name, Y-axis name, legend, etc. Furthermore, the data format output from the AI ​​model can be used to evaluate the performance of the AI ​​model or to train the AI ​​model.

Description

Technical Field

[0001] The present invention relates to a system, method, and program for constructing a dataset containing chart information to evaluate the performance of a chart de-rendering model or to train the model, and more specifically, the present invention relates to a system, method, and program for constructing a dataset containing chart information to evaluate the performance of an artificial intelligence model that de-renders charts or to train the artificial intelligence model.

Background Art

[0002] Chart de-rendering is the reverse process of chart rendering, and refers to the task of analyzing and grouping the visual patterns and information of a chart to extract core information, and extracting information related to data (e.g., numbers, groups, etc.), information related to chart layout, and the like therefrom.

[0003] For chart de-rendering, a dataset composed of chart images including texts, lines, etc., GT (Ground Truth) containing information related to charts, and the like is input to an AI model. The dataset needs to be refined to be recognizable by the AI model, and must include data of various styles so that the AI model can be evaluated or trained from multiple perspectives.

[0004] As a conventional method for constructing a dataset, a scheme of collecting data by crawling specific websites and the like has been used (PlotQA: Reasoning over Scientific Plots, Nitesh Methani et al., 3 others, 2020.).

[0005] This approach has limitations in the amount of data that can be included in a dataset, and often only data created in a specific style is collected, resulting in a lack of diversity in the collected data.

[0006] Furthermore, while chart de-rendering AI models recognize metadata in addition to numerical information contained in the data, there is a problem that data collected through crawling often does not fully include metadata (e.g., the names of the X and Y axes, the names of entity groups recorded in the legend, etc.).

[0007] Furthermore, in order to more accurately evaluate or train AI models, it is necessary to use a method that divides chart images drawn in a similar style into experimental and control groups and compares the results. However, conventional dataset construction methods have the problem that it is difficult to collect chart images in a similar style, making it impossible to use the comparison method described above. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] PlotQA: Reasoning over Scientific Plots, Nitesh Methani and 3 others, 2020. [Overview of the Initiative] [Problems that the invention aims to solve]

[0009] The invention aims to provide a system, method, and program for evaluating the performance of an artificial intelligence model that de-renders charts by constructing a large dataset containing chart information, or for training an artificial intelligence model.

[0010] The problems that this invention aims to solve are not limited to those described above, and other problems not mentioned will be clearly understood by an ordinary person from the following description. [Means for solving the problem]

[0011] The system for realizing the chart derendering model of the present invention includes at least one processor and at least one memory for storing instructions or information that cause the at least one processor to perform an operation. The operation performed by the instructions includes the steps of: storing line information, which is information about at least one line of a chart, and metadata, which is information about metadata, as GT (Ground Truth) via a dataset generation model; storing an image formed using the GT as a chart image; outputting the GT and the chart image as a dataset; inputting the chart image stored in the dataset into an AI model and outputting a data format that predicts the information of the chart; inputting the data format into a performance evaluation model and outputting a performance evaluation result for the AI ​​model by comparing the information of the data format with the GT stored in the dataset. The values ​​applied to the factors included in the line information and the factors included in the metadata are selected from predetermined values.

[0012] The above system may further include the step of inputting the data format output from the AI ​​model into a performance evaluation model, comparing the information in the data format with the GT stored in the dataset, and outputting a performance evaluation result for the AI ​​model.

[0013] The above system may further include the step of comparing the information in the above data format output from the AI ​​model with the above GT stored in the above dataset, and using the comparison results to train the above AI model.

[0014] In the above system, the factors included in the line information may be configured to include the X-axis value of the chart line, a function, and the coefficient of the function.

[0015] In the above system, the dataset consists of a first dataset and a second dataset, and the difference between the coefficients of the function included in the second dataset and the coefficients of the function included in the first dataset may be less than a predetermined value.

[0016] In the above system, the maximum value of the above function may be greater than a preset maximum function value, and the minimum value of the above function may be less than a preset minimum function value.

[0017] In the above system, the factors included in the line information may be configured to include the color or shape of the line or point.

[0018] In the above system, the factors included in the above metadata may be configured to include the chart title, X-axis name, Y-axis name, and legend.

[0019] In the above system, the values ​​applied to each of the above factors may be selected according to predetermined probabilities for each predetermined value.

[0020] A method for realizing a chart derendering model according to another aspect of the present invention involves storing line information, which is information about at least one line of a chart, and metadata, which is information about metadata, as GT (Ground Truth) via a dataset generation model; storing an image formed using the GT as a chart image; outputting the GT and the chart image as a dataset; inputting the chart image stored in the dataset into an AI model; outputting a data format that predicts the information of the chart; inputting the data format into a performance evaluation model; and outputting a performance evaluation result for the AI ​​model by comparing the information of the data format with the GT stored in the dataset, wherein the values ​​applied to the factors included in the line information and the factors included in the metadata are each selected from predetermined values.

[0021] The above method may further include the step of inputting the data format output from the AI ​​model into a performance evaluation model, comparing the information in the data format with the GT stored in the dataset, and outputting a performance evaluation result for the AI ​​model.

[0022] The above method may further include the step of comparing the information in the above data format output from the AI ​​model with the above GT stored in the above dataset, and using the comparison results to train the AI ​​model.

[0023] In the above method, the factors included in the line information may be configured to include the X-axis value of the chart line, a function, and the coefficient of the function.

[0024] In the above method, the dataset consists of a first dataset and a second dataset, and the difference between the coefficients of the function included in the second dataset and the coefficients of the function included in the first dataset may be less than a predetermined value.

[0025] In the above method, the maximum value of the function value may be larger than a preset maximum function value, and the minimum value of the function value may be smaller than a preset minimum function value.

[0026] In the above method, the factors included in the line information may be configured to include the color or shape of a line or a point.

[0027] In the above method, the factors included in the meta information may be configured to include a chart title, an X-axis name, a Y-axis name, and a legend.

[0028] In the above method, the value applied to each of the factors may be selected according to a predetermined probability for each predetermined value.

[0029] A program according to still another aspect of the present invention may be coupled to a computer and stored in a computer-readable recording medium to implement the chart derendering model according to the embodiment of the present invention. [Advantageous Effects of Invention]

[0030] According to the present invention, a dataset that completely includes information related to charts can be derived, and by using the factors of line information and meta information, a large number of chart images formed in various styles with a high degree of freedom can be generated. In addition, by dividing output results into an experimental group and a control group using a method of appropriately selecting the factors of line information and meta information, the AI model can be evaluated or trained more accurately.

[0031] Further, according to the present invention, it is possible to generate a large amount of datasets including charts of similar schematic shapes, so as to evaluate whether the AI model can accurately recognize a chart of a specific schematic shape, or to train the AI model to accurately recognize a chart of a specific schematic shape.

[0032] Furthermore, according to the present invention, the general shape of the charts included in the dataset can be kept from being distorted so that the AI ​​model can be accurately evaluated or the AI ​​model can learn efficiently.

[0033] Furthermore, according to the present invention, it is possible to output a large amount of datasets including charts with various lines or points so that the AI ​​model can be accurately evaluated or the AI ​​model can learn efficiently.

[0034] Furthermore, according to the present invention, it is possible to output a large dataset containing complete metadata in order to accurately evaluate whether the AI ​​model accurately recognizes the metadata contained in the chart image, or to train the AI ​​model so that it can accurately recognize the metadata contained in the chart image.

[0035] The effects of the present invention are not limited to those described above, and other effects not mentioned will be clearly understood by an ordinary person of the art from the following description. [Brief explanation of the drawing]

[0036] [Figure 1] This is a schematic diagram of a system for realizing a method for evaluating the performance of an artificial intelligence model that constructs a dataset containing chart information and de-renders the chart, or for training an artificial intelligence model, according to one embodiment of the present invention. [Figure 2] This is a block diagram illustrating the configuration of an apparatus according to one embodiment of the present invention for evaluating the performance of an artificial intelligence model that constructs a dataset containing chart information and de-renders the chart, or for training an artificial intelligence model. [Figure 3] This is a block diagram illustrating an embodiment of the present invention for evaluating the performance of an artificial intelligence model that constructs a dataset containing chart information and de-renders the chart, or for training an artificial intelligence model. [Figure 4a] This figure shows an example of a chart image output from a dataset generation model according to an embodiment of the present invention. [Figure 4b] This figure shows an example of a chart image output from a dataset generation model according to an embodiment of the present invention. [Figure 5] This figure shows an example of a chart image output from a dataset generation model according to an embodiment of the present invention. [Figure 6A] This figure shows an example of a chart image output from a dataset generation model according to an embodiment of the present invention. [Figure 6B] This figure shows an example of a chart image output from a dataset generation model according to an embodiment of the present invention. [Modes for carrying out the invention]

[0037] The embodiments described below are provided as examples to adequately convey the concept of the present invention to those who have ordinary skill in the art to which the present invention pertains. Therefore, the present invention is not limited to the embodiments described below and may be embodied in other forms.

[0038] Throughout this invention, the same reference numerals refer to the same component. This invention does not describe all elements of the embodiments, and general content in the art to which this invention belongs or content that overlaps between embodiments is omitted. The terms “part,” “module,” “component,” and “block” as used in this specification can be implemented in software or hardware, and in some embodiments, multiple “parts,” “modules,” “components,” and “blocks” may be implemented as a single component, and one “part,” “module,” “component,” and “block” may contain multiple components.

[0039] Throughout the specification, the term "connected" to another part includes not only direct connections but also indirect connections, including connections via wireless communication networks.

[0040] Furthermore, when a part "includes" a certain component, unless otherwise stated, it does not mean that it excludes other components, but rather that it may include other components.

[0041] Throughout the specification, the statement that one component is located "on top of" another component includes not only cases where one component is in contact with another component, but also cases where there is yet another component between the two components.

[0042] Terms such as "first," "second," etc., are used to distinguish one component from another, and the components are not limited by the terms used above.

[0043] A singular expression includes plural forms unless there is a clear exception in the context.

[0044] The identification codes used in each step are for illustrative purposes only and do not indicate the order of the steps. The steps may be performed in a different order than specified unless the context clearly indicates otherwise.

[0045] The system for evaluating the performance of an artificial intelligence model that constructs a dataset containing chart information and de-renders the charts, or for training an artificial intelligence model, according to the present invention, may include devices, and such devices may include all kinds of devices that can perform computational processing and provide results to a user. For example, the system for evaluating the performance of an artificial intelligence model that constructs a dataset containing chart information and de-renders the charts, or for training an artificial intelligence model, according to the present invention, may include at least one computer, server device, and portable terminal, or any one of the same or similar functions. However, the present invention is not limited thereto.

[0046] Here, the above-mentioned computer may include, for example, a notebook, desktop, laptop, tablet PC, or slate PC equipped with a web browser.

[0047] The above-mentioned server device is a server that communicates with external devices to process information, and may include application servers, computing servers, database servers, file servers, game servers, mail servers, proxy servers, and web servers.

[0048] The above-mentioned mobile terminals include, for example, wireless communication devices that guarantee portability and mobility, and can include all types of handheld-based wireless communication devices such as PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminals, and smartphones, as well as wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted devices (HMDs).

[0049] Embodiments of the present invention will be described in detail below with reference to the attached drawings.

[0050] The present invention relates to a system, method, and program for constructing a dataset containing chart information to evaluate the performance of a chart de-rendering model or to train a model, and more specifically, to a system, method, and program for constructing a dataset containing chart information to evaluate the performance of an artificial intelligence model that de-renders charts or to train an artificial intelligence model.

[0051] Figure 1 is a schematic diagram of a system according to one embodiment of the present invention that constructs a dataset containing chart information to evaluate the performance of a chart de-rendering model or to train a model.

[0052] As shown in Figure 1, the system 1000 may include a device 100, a database 200, a dataset generation model 300, an AI model 400, and a performance evaluation model 500.

[0053] The devices 100, database 200, dataset generation model 300, AI model 400, and performance evaluation model 500 included in system 1000 can communicate via network W. Here, network W can include wired and wireless networks. For example, the network can include various types of networks such as local area networks (LAN), metropolitan area networks (MAN), and wide area networks (WAN).

[0054] Furthermore, network W may also include the well-known World Wide Web (WWW). However, network W according to embodiments of the present invention is not limited to the networks listed above, and may also include, at least partially, well-known wireless data networks, well-known telephone networks, and well-known wired and wireless television networks.

[0055] Device 100 can input the dataset generated by the dataset generation model 300 into the AI ​​model 400 and output information about the chart. Based on the outputted information about the chart, the performance evaluation model 500 evaluates the performance of the AI ​​model 400, and the AI ​​model 400 can perform training.

[0056] The dataset generation model 300 generates a dataset 410 to be de-rendered via AI and inputs it into the AI ​​model 400. The dataset 410 is identical to, but not limited to, those commonly used to train or evaluate the performance of an AI model. The dataset generated via the dataset generation model 300 consists of chart images containing text, lines, etc., and Ground Truth (GT) containing information about the chart. The dataset includes a train set used to evaluate the accuracy of the AI ​​model through the results of chart de-rendering using artificial intelligence, and a test set used to train (e.g., deep learning) the AI ​​model used for chart de-rendering.

[0057] The charts included in the dataset may be in the form of bar / column graphs, line graphs, pie charts, area graphs, scatter plots, radar charts, histograms, and / or waterfall charts. The chart may be a single form or a combination of multiple forms. However, the charts used in the present invention are not limited to these, and the chart may include any form of chart. In addition to information that visualizes numerical values ​​(e.g., lines, circles, etc.), the chart may also include text information. Specifically, this may include chart annotations, a chart legend or title, the names of each axis (e.g., X-axis, Y-axis, Z-axis, etc.), and the numerical values ​​of the raw data of the points included in the chart (e.g., the numerical values ​​of the X-axis and Y-axis of the points included in the chart).

[0058] The chart information output from AI model 400 is information that describes the characteristics of the chart and may include metadata, data information, etc. Metadata may include the names of the X and Y axes, and the names of entity groups recorded in the legend. Data information may include the numerical values ​​of the X and / or Y axes for each entity.

[0059] Information about the charts output from AI model 400 can be input into performance evaluation model 500. Performance evaluation model 500 evaluates the performance of AI model 400 by comparing the GT included in the dataset with the information about the output charts.

[0060] The performance evaluation results output from Performance Evaluation Model 500 indicate how accurately the AI ​​model predicts the chart. These results can be presented as numerical scores, image-based charts, or other visual representations, but are not limited to these formats.

[0061] Furthermore, the device 100 can compare the chart information output from the AI ​​model 400 with the chart information stored in the dataset, and the AI ​​model 400 can learn using the comparison results.

[0062] The database 200 can store various data (e.g., datasets) for training or evaluating the performance of the AI ​​model 400. The database 200 may also store chart images, information about charts, information about performance evaluation methods, information about dataset generation methods, and, in various embodiments, calculated data output by the AI ​​model 400. However, the system 1000 may not include the database 200 once the training of the AI ​​model 400 is complete.

[0063] Figure 1 shows a case where the database 200 is implemented outside of the device 100. In this case, the database 200 can be connected to the device 100 by wired or wireless connection. However, this is only one embodiment, and the database 200 may also be implemented as a component of the device 100.

[0064] Figure 1 shows a case where the AI ​​model 400 is implemented outside of the device 100 (for example, on a cloud-based basis), but it is not limited to this, and may also be implemented as a component within the device 100.

[0065] Figure 2 is a block diagram illustrating the configuration of an apparatus according to one embodiment of the present invention for evaluating the performance of an artificial intelligence model that constructs a dataset containing chart information and de-renders the chart, or for training an artificial intelligence model.

[0066] As shown in Figure 2, the device 100 may include a memory 110, a communication module 120, a display 130, an input module 140, and a processor 150. However, it is not limited to these, and the software and hardware configuration of the device 100 may be modified, added to, or omitted within the scope that is obvious to an ordinary engineer, depending on the required operation. Furthermore, the device 100 may be replaced by a system, and the device 100 may include multiple devices. In this case, each component included in the device 100 may be included in at least one of the multiple devices.

[0067] Memory 110 can store data that supports various functions of the device 100, programs for the operation of the processor 150, and input / output data. It can also store numerous application programs or applications driven by the device, data for the operation of the device 100, instructions, and AI models. At least some of these application programs can be downloaded from an external server via wireless communication.

[0068] Such memory 110 may include at least one type of storage medium, such as flash memory type, hard disk type, SSD type (Solid State Disk type), SDD type (Silicon Disk Drive type), multimedia card micro type, card type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory, magnetic disk, and optical disk.

[0069] Furthermore, the memory 110 is detachable from the device and may include a database connected via wired or wireless connection. The database 200 shown in Figure 1 can be implemented as a component of the memory 110.

[0070] The communication module 120 may include one or more components that enable communication with external devices, for example, at least one of a broadcast receiving module, a wired communication module, a wireless communication module, a short-range communication module, and a location information module.

[0071] Wired communication modules can include a variety of wired communication modules such as Local Area Network (LAN) modules, Wide Area Network (WAN) modules, or Value Added Network (VAN) modules, as well as various cable communication modules such as USB (Universal Serial Bus), HDMI (High Definition Multimedia Interface), DVI (Digital Visual Interface), RS-232 (recommended standard 232), power line communication, or POTS (plain old telephone service).

[0072] Wireless communication modules can include Wi-Fi modules, wireless broadband modules, and other wireless communication modules that support various wireless communication methods such as GSM (Global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (Universal Mobile Telecommunications System), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, and 6G.

[0073] The display 130 displays (outputs) information or data processed by the device 100, data input or output via the AI ​​model 400, etc. The display 130 can also display execution screen information of an application program (e.g., an application) driven by the device 100, or UI (User Interface) or GUI (Graphic User Interface) information corresponding to such execution screen information.

[0074] The input module 140 is for receiving information from the user, and when the user inputs information via the input unit, the processor 150 can control the operation of the device 100 in accordance with the input information.

[0075] Such an input module 140 may include hardware physical keys (e.g., buttons, dome switches, jog wheels, jog switches, etc., located on at least one of the front, rear, and side surfaces of the device) and software touch keys. For example, the touch keys may consist of virtual keys, soft keys, or visual keys displayed on a touchscreen-type display 130 via software processing, or touch keys located in a part other than the touchscreen. On the other hand, the virtual keys or visual keys can be displayed on the touchscreen in various forms, and may consist of graphics, text, icons, videos, or combinations thereof.

[0076] The processor 150 may be implemented with a memory that stores data relating to an algorithm or a program that reproduces an algorithm for controlling the operation of the components within the device 100 (including learning and executing an AI model), and at least one processor (not shown) that performs the above-mentioned operations using the data stored in the memory. In this case, the memory and the processor may be implemented as separate chips or as a single chip.

[0077] In one embodiment, the system 1000 or apparatus 100 according to the present invention may include at least one processor, and if it includes multiple processors, the multiple processors may be contained within different apparatuses 100.

[0078] Furthermore, the processor 150 can control any one or more of the above-mentioned components in combination in order to realize various embodiments of the present invention, as described below, on the device 100.

[0079] Figure 3 is a block diagram illustrating a method according to an embodiment of the present invention for constructing a dataset containing chart information to evaluate the performance of a chart de-rendering model or to train a model.

[0080] Referring to Figure 3, the dataset generation model 300 generates a dataset 410 consisting of chart images containing text, lines, etc., and Ground Truth (GT) containing information about the charts. Here, the chart images include titles, axis names, legend information, numerical values, etc. The chart images included in the dataset 410 may be, but are not limited to, bar / vertical bar graphs, line graphs, pie charts, area graphs, scatter plots, radar charts, histograms, and / or waterfall charts. Furthermore, the chart may be a single form or a combination of multiple forms.

[0081] Here, the GT of the dataset generated by the dataset generation model 300 is information about the chart, and includes line information, which is information about the lines formed on the chart (e.g., X-axis values ​​of the chart lines, functions, coefficients of the functions, maximum / minimum function values, line or point colors, line or point shapes, etc.), and metadata information, which is information about the metadata (e.g., chart title, X-axis names, Y-axis names, legend, etc.).

[0082] In this case, the values ​​applied to the factors included in the line information and the factors included in the meta information are selected from predetermined values.

[0083] For example, among the factors included in line information, the "function" factor can be comprised of polynomial functions, sinusoidal functions, Gaussian functions, exponential functions, logarithmic functions, Pareto functions, random functions, etc., and can be constructed by selecting one or more functions from among these. In particular, by using multiple sinusoidal functions, all periodic functions can be represented through a Fourier series (more specifically, the periodic function shown in Figure 4a can also be represented using multiple sinusoidal functions).

[0084] Furthermore, by pre-defining a range of line or point colors or shapes and allowing users to select from that range, charts can be created that display both dotted and solid lines simultaneously, as shown in Figure 4b, or charts that display both triangular and rectangular points simultaneously.

[0085] As a result, the present invention can derive a dataset that completely contains information about the chart, and by using line information and metadata factors, it can generate a large number of chart images formed in various styles with a high degree of freedom. Furthermore, by using a method that appropriately selects line information and metadata factors and dividing the output results into an experimental group and a control group, the AI ​​model can be evaluated or trained more accurately.

[0086] Next, the dataset 410 generated by the dataset generation model 300 is input to the image encoder 420. The image encoder 420 follows a commonly used encoder architecture and plays the role of converting the input chart 410a into a first embedding 421 that can be processed by the AI ​​model 400.

[0087] Next, the first embedding 421 output from the image encoder 420 is input to an AI model 400 which includes a decoder 430 that performs meta decoding, data decoding, etc. The decoder 430 decodes the input first embedding 421 to extract meta information of the chart (e.g., chart title, X-axis name, Y-axis name, legend), data information (e.g., numerical values ​​of data points), etc., and outputs this as the second embedding 431.

[0088] In this way, the AI ​​model can ultimately output a data format 440 that predicts chart information, which includes metadata, data information, etc., from the output second embedding 431.

[0089] Next, the data format 440 output from the AI ​​model can be input to the chart de-rendering performance evaluation model 500. The chart de-rendering performance evaluation model 500 compares the chart information contained in the data format 440 (e.g., chart title, X-axis name, Y-axis name, legend, numerical values ​​of data points, etc.) with the GT (Ground Truth) contained in the dataset 410, and derives the performance evaluation result 460 according to a predetermined method.

[0090] On the other hand, the AI ​​model 400 can learn by comparing the output data format 440 with the information about the charts stored in the dataset. The AI ​​model 400 can be trained using hardware resources such as a central processing unit (CPU) and / or a graphics processing unit (GPU). The process of training the AI ​​model 400 can be done through a loss function that measures the difference between the output chart information and the GT stored in the dataset. The loss function may vary depending on the type of task the AI ​​model 400 performs and the characteristics of the AI ​​model 400 itself. Possible but not limited to Mean Squared Error (MSE) and Cross-Entropy Loss may be used as loss functions. By training in a way that minimizes the calculated loss, the AI ​​model 400 can make better predictions, so the AI ​​model is optimized by adjusting the model weights in a direction that minimizes the calculated loss. Such an optimization process can be performed using a backpropagation algorithm.

[0091] Below, with reference to Figures 5 and 6, we will describe in detail the factors included in the line information and metadata used in the dataset generation model 300.

[0092] First, to evaluate whether the AI ​​model can accurately recognize a chart with a specific shape, or to train the AI ​​model to accurately recognize a chart with a specific shape, it is necessary to repeatedly input chart images with similar shapes into the AI ​​model. For this purpose, the line information factors can be configured to include the X-axis values ​​of the chart lines, the function, the coefficient of the function, etc., and the dataset generation model 300 can be configured to output multiple datasets 410 such that the "function" factor remains identical and only the "coefficient of the function" factor differs by a value less than a preset value. In this way, multiple datasets 410 with identical "function" factors and only some differences in the "coefficient of the function" represent chart shapes that are similar but do not exactly match. For example, as shown in Figure 5, charts (A) and (B) have the same "function" and therefore represent similar shapes, but because their "coefficients of the function" are different, they may represent shapes that do not exactly match.

[0093] This allows the present invention to generate a large dataset containing charts with similar outlines in order to evaluate whether an AI model can accurately recognize a chart with a specific outline, or to train an AI model to accurately recognize a chart with a specific outline.

[0094] On the other hand, if the range of function values ​​included in the dataset represents a small proportion of the total Y-values ​​in the chart, the overall shape of the line may not be fully represented. For example, referring to Figure 6, the function values ​​of line (A) are distributed from 230 to 340, and line (B) is distributed from 380 to 420. Therefore, the function values ​​of line (B) represent a smaller proportion of the total Y-values ​​compared to the function values ​​of line (A). As a result, line (B) is represented more compressed than line (A), and its overall shape is distorted. To solve this problem, the maximum value of the function in the line can be configured to be greater than a preset maximum function value, and the minimum value of the function in the line can be configured to be less than a preset minimum function value. For example, referring to Figure 6, if the preset minimum function value is set to 230 and the preset maximum function value is set to 340, the minimum / maximum function values ​​of lines (A) and (B) will be located outside the range of 230 to 340, so the line will not be represented as an excessively compressed shape. In other words, the function values ​​of the chart lines can be distributed more broadly than a certain range of Y values, and the proportion of the chart's function values ​​to the total Y values ​​of the chart is small, preventing the phenomenon where the lines are distorted as a compressed general shape.

[0095] This allows the present invention to ensure that the overall shape of the charts included in the dataset is not distorted, enabling accurate evaluation of the AI ​​model or efficient learning of the AI ​​model.

[0096] On the other hand, the lines included in the chart can be formed as dotted lines or straight lines, and their colors can also vary. Furthermore, the shapes of the points represented by the lines can be diverse, such as circles, squares, and triangles. In order to output a dataset 410 containing such diverse charts and make it usable as input to an AI model, the color or shape of the lines or points can be included as a factor in the line information of the dataset.

[0097] This allows the present invention to output a large amount of datasets containing charts with various lines or points, enabling accurate evaluation of AI models or efficient learning of AI models.

[0098] On the other hand, the AI ​​model recognizes metadata in addition to the numerical information contained in the data from the input chart image. In order to evaluate or train such an AI model, it is necessary to input a large dataset that completely contains metadata about the chart (e.g., axis and Y axis names, entity group names recorded in the legend, etc.). For this purpose, the data generation model 300 stores the metadata about the chart as GT in addition to the line information in the dataset 410. Here, the factors included in the metadata may be the chart title, X axis name, Y axis name, legend, etc., and the values ​​applied to each factor of the metadata can be selected from predetermined values. For example, the values ​​used for the X axis name among the factors of the metadata may be predetermined as population size, year, blood sample volume, blood glucose, fiscal year, etc., and specific values ​​can be selected and reflected via the data generation model.

[0099] This allows the present invention to output a large dataset containing complete metadata in order to evaluate whether an AI model accurately recognizes metadata contained in a chart image, or to train an AI model to accurately recognize metadata contained in a chart image.

[0100] On the other hand, it is necessary to intensively train an AI model to evaluate or improve its recognition performance for charts with specific factors. In this case, it is necessary to intensively output charts with specific factors from the dataset generation model 300 and input them into the AI ​​model. For this purpose, the values ​​applied to each factor used in the dataset generation model 300 can be selected from predetermined values, and each predetermined value can be selected according to a predetermined probability. For example, if it is necessary to intensively output "Gaussian functions" among the factor values, a higher probability can be assigned to "Gaussian functions" than other "function" values ​​(e.g., polynomial functions, exponential functions, logarithmic functions, etc.), thereby configuring the dataset to include a large number of charts related to "Gaussian functions".

[0101] As a result, the present invention can generate a large number of chart images in various styles with a high degree of flexibility, while also being able to concentrate on generating particularly necessary datasets.

[0102] On the other hand, a method for constructing a dataset containing chart information, evaluating the performance of a chart de-rendering model, or training a model, according to an embodiment of the present invention, can be realized by the system described with reference to Figure 1.

[0103] An AI model according to an embodiment of the present invention can be controlled, executed, learned, and driven by a processor, thereby enabling the AI ​​model to perform at least one task of execution, learning, and driving by at least one processor. Furthermore, the AI ​​model can be stored in memory, and the feature data according to the present invention can also be stored in memory.

[0104] On the other hand, the disclosed embodiments may be implemented in the form of a recording medium for storing computer-executable instructions. The instructions may be stored in the form of program code, which, when executed by a processor, can generate a program module that performs the operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

[0105] Computer-readable recording media include all types of recording media that store instructions that can be deciphered by a computer. Examples include ROM (Read Only Memory), RAM (Random Access Memory), magnetic tape, magnetic disks, flash memory, and optical data storage devices.

[0106] Embodiments disclosed with reference to the accompanying drawings have been described above. Those with ordinary skill in the art to which the present invention pertains will understand that the present invention may be carried out in forms different from those disclosed without altering the technical idea or essential features of the invention. The disclosed embodiments are illustrative and should not be construed as limiting.

Claims

1. A system that implements a chart de-rendering model, At least one processor, Includes at least one memory that stores instructions or information causing the at least one processor to perform an operation, The operation performed by the aforementioned instruction is: The process involves storing line information, which is information about at least one line of a chart, and metadata, which is information about metadata, as GT (Ground Truth) via a dataset generation model, storing an image formed using the GT as a chart image, and outputting the GT and the chart image as a dataset. The steps include inputting images of charts stored in the dataset into an AI model and outputting a data format that predicts the information of the charts, The step includes inputting the data format into a performance evaluation model, comparing the information in the data format with the GT stored in the dataset, and outputting a performance evaluation result for the AI ​​model. The system is such that the values ​​applied to the factors included in the line information and the factors included in the metadata are each selected from a set of predetermined values.

2. The system according to claim 1, further comprising the steps of inputting the data format output from the AI ​​model into a performance evaluation model, comparing the information of the data format with the GT stored in the dataset, and outputting a performance evaluation result for the AI ​​model.

3. The system according to claim 1, further comprising the step of comparing the information of the data format output from the AI ​​model with the GT stored in the dataset via the AI ​​model, and training the AI ​​model using the comparison results.

4. The system according to claim 1, wherein the factors included in the line information are configured to include the X-axis value of the chart line, a function, and the coefficient of the function.

5. The aforementioned dataset consists of a first dataset and a second dataset. The system according to claim 4, wherein the difference between the coefficients of the function included in the second dataset and the coefficients of the function included in the first dataset is less than a preset value.

6. The system according to claim 4, wherein the maximum value of the function is greater than a preset maximum function value, and the minimum value of the function is less than a preset minimum function value.

7. The system according to claim 1, wherein the factors included in the line information are configured to include the color or shape of the line or point.

8. The system according to claim 1, wherein the factors included in the metadata are configured to include a chart title, X-axis name, Y-axis name, and legend.

9. The system according to claim 2, 7, or 8, wherein the values ​​applied to each of the factors are selected according to a predetermined probability for each predetermined value.

10. In a method for implementing a chart derendering model, Through a dataset generation model, line information, which is information about at least one line of a chart, and metadata, which is information about metadata, are stored as GT (Ground Truth), the image formed using the GT is stored as a chart image, and the GT and the chart image are output as a dataset. The images of the charts stored in the aforementioned dataset are input to the AI ​​model, and a data format predicting the information of the charts is output. A method comprising inputting the aforementioned data format into a performance evaluation model, comparing the information in the data format with the GT stored in the dataset, and outputting a performance evaluation result for the AI ​​model, A method in which the values ​​applied to the factors included in the line information and the factors included in the meta information are selected from predetermined values.

11. The method according to claim 10, further comprising the steps of inputting the data format output from the AI ​​model into a performance evaluation model, comparing the information of the data format with the GT stored in the dataset, and outputting a performance evaluation result for the AI ​​model.

12. The method according to claim 10, further comprising the step of comparing the information of the data format output from the AI ​​model with the GT stored in the dataset via the AI ​​model, and training the AI ​​model using the comparison results.

13. The method according to claim 10, wherein the factors included in the line information are configured to include the X-axis value of the chart line, a function, and the coefficient of the function.

14. The aforementioned dataset consists of a first dataset and a second dataset. The method according to claim 13, wherein the difference between the coefficients of the function included in the second dataset and the coefficients of the function included in the first dataset is less than a predetermined value.

15. The method according to claim 13, wherein the maximum value of the function is greater than a preset maximum function value, and the minimum value of the function is less than a preset minimum function value.

16. The method according to claim 10, wherein the factors included in the line information are configured to include the color or shape of the line or point.

17. The method according to claim 10, wherein the factors included in the metadata are configured to include a chart title, an X-axis name, a Y-axis name, and a legend.

18. The method according to claim 13, 16, or 17, wherein the value applied to each of the factors is selected according to a predetermined probability for each predetermined value.

19. A program stored on a computer-readable recording medium, which is coupled to a computer and used to perform the method according to any one of claims 11 to 18.

Images