Chart de-rendering system, method, and program for extracting meta-information and data information from chart by using artificial intelligence

Abstract

Disclosed are a graph de-rendering system, method, and program for extracting meta-information and data information from a graph by using artificial intelligence. The system includes a memory having an AI model and a processor, the AI model including: an image encoder that inputs a graph, converts the graph into a first insert processable by the AI model, and outputs the first insert; a meta decoder that outputs a second embedding including meta information from the first embedding; and a data decoder that outputs a fourth insert containing data information from a third insert containing entity-related information contained in the second insert, the processor for extracting or training an AI model. The data information contained in the fourth embedding can be marked into a single mark of each piece of data information, and when the data information is extracted from the single mark, a predefined repeated template can be used.

Description

Technical Field

[0001] The present invention relates to a chart rendering system, method and program for extracting meta-information and data information included in a chart, and more specifically, to a chart rendering system, method and program capable of extracting meta-information and data information included in a chart using artificial intelligence. Background Art

[0002] Figures and graphs included in papers, reports, textbooks, etc. are typically generated by transferring a table of data set consisting of numbers and groups, along with code defining the overall layout (e.g., type, orientation, color / shape configuration, etc.) to a rendering engine.

[0003] Chart de-rendering refers to a process opposite to chart rendering, in which visual patterns or information of a chart are analyzed and grouped to extract key information, and information about data (e.g., numbers, groups, etc.) and information about chart layout are extracted from the key information.

[0004] A rule-based model is used during the initial chart rendering process. A rule-based model extracts chart information using predefined functions (e.g., color detection, chart axis value extraction), and combines or analyzes each piece of information using predefined rules. While rule-based models offer the advantage of high accuracy, they also suffer from a lack of scalability, as each chart type requires a separate model.

[0005] To address this shortcoming, generative model approaches using a single trained artificial intelligence (AI) have been introduced (DePlo: One-shot visual language reasoning by plot-to-table translation, In Findings of the Association for Computational Linguistics, ACL 2023, pp. 10381–10399). Unlike rule-based models, generative models can be easily applied to all types of graphs with a single model, offering greater scalability.

[0006] However, conventional generative models adopt a method of including meta information (e.g., names of the X-axis and Y-axis, names of entity (single data unit) groups recorded in the legend, etc.) and data information (e.g., numerical values ​​of the X-axis and / or Y-axis of each entity, etc.) in a single data format without distinguishing them. Figure 6As shown, the chart title (meta-chart), the name of the X-axis (Epoch), the name of the Y-axis (Experimental Results), the X-axis (Epoch) values ​​of [1, 2, 3, 4, 5], and all Y values ​​corresponding to the X-axis (Epoch) values ​​for each entity group (Model 1 and Model 2) (i.e., [10, nan, 30, nan, 50] for Model 1 and [1, 4, 9, 16, 25] for Model 2) are recorded in a single data format. This data format includes a large amount of representation information, which makes data representation complex, has the limitation of having to represent all Y values ​​that depend on each X value, and has problems such as a high probability of formatting errors due to entity values ​​(e.g., "nan" values) not being recognized.

[0007] Furthermore, conventional generative models typically treat numbers as text. Consequently, they often represent numbers by dividing individual digits into irregular units, splitting each unit into multiple tokens, and inputting these tokens. Such methods use a large number of unnecessary tokens to represent a single digit (for example, when representing the number 13.1, two tokens are required because "1" and "3.1" are input as separate tokens). Consequently, conventional language models use a large number of unnecessary tokens, leading to token waste and reduced inference speed.

[0008] Prior art document non-patent document “DePlot: One-shot visual language reasoning by plot-to-table translation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 10381-10399”. Summary of the Invention

[0009] Technical issues

[0010] An object of the present invention is to provide a system, method, and program capable of efficiently and accurately extracting numerical information while providing a data format that simply represents the content of data included in a graph.

[0011] The objects of the present invention are not limited to the above objects, and those skilled in the art will clearly understand other unmentioned objects through the following description.

[0012] Technical Solution

[0013] According to the present invention, a system for implementing an AI model for extracting meta-information and data information included in a chart includes at least one processor and at least one memory, the at least one memory storing commands or information for causing the at least one processor to perform operations, wherein the operations performed by the commands include: inputting a chart into an image encoder to convert the chart into a first embedding that can be processed by the AI ​​model; inputting the first embedding into the AI ​​model to output a second embedding including meta-information based on the first embedding, and outputting a fourth embedding including data information based on a third embedding including information about entities included in the second embedding; and outputting each of a first data format and a second data format, wherein the meta-information included in the second embedding is recorded in the first data format, and the data information included in the fourth embedding is recorded in the first data format.

[0014] In this system, the data information included in the fourth embedding can be differentiated for each entity group.

[0015] In this system, the meta information may include the title of the chart, the names of the axes, and the names of the entity groups included in the legend.

[0016] In the system, the data information may include numerical information included in a graph.

[0017] In this system, data information can be tokenized into a single token for each piece of data information and included in a fourth embedding, and when the data information included in the fourth embedding is recorded in a second data format, the tokenized data information can be extracted from the single token and recorded in the second data format.

[0018] In this system, a multi-layer perceptron (MLP) can be used to extract data information from a single token.

[0019] In this system, when extracting data information from a single token, multiple pieces of data information can be extracted simultaneously by inputting the single token together with a predefined repetitive template.

[0020] A method for extracting meta information and data information included in a chart according to another aspect of the present invention may include: inputting a chart into an image encoder to convert the chart into a first embedding that can be processed by an AI model; inputting the first embedding into the AI ​​model to output a second embedding including meta information based on the first embedding; outputting a fourth embedding including data information based on a third embedding including information about an entity included in the second embedding; and outputting each of a first data format and a second data format, recording the meta information included in the second embedding in the first data format, and recording the data information included in the fourth embedding in the second data format.

[0021] In this method, the data information included in the fourth embedding may be differentiated for each entity group.

[0022] In the method, the meta information may include a title of the chart, names of axes, and names of entity groups included in the legend.

[0023] In the method, the data information may include numerical information included in the graph.

[0024] In this method, data information can be tokenized into a single token for each piece of data information and included in a fourth embedding, and when the data information included in the fourth embedding is recorded in a second data format, the tokenized data information can be extracted from the single token and recorded in the second data format.

[0025] In this method, a multi-layer perceptron (MLP) can be used to extract data information from a single token.

[0026] In this method, when extracting data information from a single token, multiple pieces of data information can be extracted simultaneously by inputting the single token together with a predefined repetitive template.

[0027] A program according to another aspect of the present invention may be stored on a computer-readable recording medium to implement an AI model for extracting meta information and data information included in a graph through the AI ​​model according to an embodiment of the present invention in conjunction with a computer.

[0028] Technical Effects

[0029] According to the present invention, since the length of each data format is shortened by extracting metadata separately, the data content can be simply represented and the possibility of format errors can be reduced.

[0030] In addition, according to the present invention, since the original data is represented independently by distinguishing each entity group, there is no need to input difficult-to-handle "nan" values ​​for empty data values, thereby minimizing format errors caused by the inability to recognize "nan" values ​​or optimizing the use of tokens.

[0031] In addition, according to the present invention, since each number included in the entity group of the diagram uses a single token ( <num>) output, thus promoting the task separation between text understanding and data extraction, and reducing the number of tokens representing numbers, thereby achieving efficient training and prediction in the model.

[0032] In addition, according to the present invention, since a sufficient length of " <num>" token, thus minimizing the need for autoregressive transfer and significantly improving inference speed.

[0033] The effects of the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those skilled in the art from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of a system for implementing an artificial intelligence (AI)-based chart de-rendering method according to one embodiment of the present disclosure.

[0035] Figure 2 It is a block diagram for explaining the configuration of an apparatus for performing an AI-based chart de-rendering method according to one embodiment of the present disclosure.

[0036] Figure 3 is a block diagram for explaining a method of extracting meta information and data information from a graph according to an embodiment of the present invention.

[0037] Figure 4 is a conceptual diagram for explaining a method of representing the number “-0412920” in embedding through SNE according to an embodiment of the present invention.

[0038] Figure 5a This is a conceptual diagram illustrating a method of recognizing and representing numbers using an autoregressive method in a conventional language model.

[0039] Figure 5b A method of identifying and representing numbers using an autoregressive method through SNE according to an embodiment of the present invention.

[0040] Figure 5c A method for recognizing and representing numbers in a non-autoregressive manner using a repetitive template through SNE according to an embodiment of the present invention.

[0041] Figure 6 is an example of a data format used in general generative models. DETAILED DESCRIPTION

[0042] The following embodiments are provided as examples so that the spirit of the present invention can be fully conveyed to those skilled in the art to which the present invention belongs. Therefore, the present invention is not limited to the embodiments described below and can be specified in other forms.

[0043] In the present invention, the same reference numerals refer to the same components. This invention does not describe all the elements of the embodiments, and will omit common content in the field to which the invention belongs or content that overlaps between embodiments. The terms "unit", "module", "component", and "block" used in this specification can be implemented as software or hardware, and depending on the embodiment, multiple "units", "modules", "components", and "blocks" can be implemented as one component, or one "unit", "module", "component", and "block" can also include multiple components.

[0044] Throughout the specification, when a first component is described as being “connected” to a second component, this includes not only the case where the first component is directly connected to the second component but also the case where the first component is indirectly connected to the second component, and the indirect connection includes connection through a wireless communication network.

[0045] In addition, when a part is described as “including” a certain component, unless specifically stated otherwise, it means further including other components rather than excluding other components.

[0046] Throughout this specification, when a first member is described as being positioned "on" a second member, this includes the case where the first member is in contact with the second member and the case where a third member is present between the two members.

[0047] Terms such as “first” and “second” are used to distinguish one component from another component, and the components are not limited by the above terms.

[0048] A singular expression includes a plural expression unless the context clearly requires otherwise.

[0049] In each operation, identification symbols are used for convenience of description, and the identification symbols do not describe the order of each operation, and each operation may be performed in an order different from the specified order unless a specific order is clearly described in the context.

[0050] The chart de-rendering system according to the present invention may include a device, and the device may include various devices capable of performing computational processing to provide results to a user. For example, the chart de-rendering system according to the present invention may include at least one of a computer, a server device, and a portable terminal, or may be implemented in any form having the same or similar functions as those described above. However, the present invention is not limited thereto.

[0051] Here, the computer may include, for example, a notebook computer, a desktop computer, a laptop computer, a tablet PC, a slate PC, etc. equipped with a web browser.

[0052] The server device is a server that communicates with an external device to process information, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, and a web server.

[0053] A portable terminal is, for example, a wireless communication device that ensures portability and mobility, and may include various handheld device-based wireless communication devices such as a personal communication system (PCS), a global system for mobile communications (GSM), a personal digital cellular phone (PDC), a personal handyphone system (PHS), a personal digital assistant (PDA), International Mobile Telecommunications-2000 (IMT-2000), Code Division Multiple Access-2000 (CDMA-2000), Wideband Code Division Multiple Access (W-CDMA), a wireless broadband Internet (WiBro) terminal, a smart phone, and wearable devices such as a watch, a ring, a bracelet, an anklet, a necklace, glasses, contact lenses, or a head-mounted device (HMD).

[0054] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0055] The present invention relates to a chart rendering system, method and program for extracting meta-information and data information included in a chart, and more specifically, to a chart rendering system, method and program capable of extracting meta-information and data information included in a chart using artificial intelligence.

[0056] Figure 1 is a schematic diagram of a system capable of implementing a method for de-rendering a chart according to an embodiment of the present invention.

[0057] like Figure 1 As shown, the system 1000 may include an apparatus 100 , a database 200 , and an AI model 300 .

[0058] The device 100, the database 200, and the AI ​​model 300 included in the system 1000 can perform communication via a network W. Here, the network W may include a wired network and a wireless network. For example, the network may include various networks such as a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN).

[0059] In addition, the network W may also include the well-known World Wide Web (WWW). However, the network W according to an embodiment of the present invention is not limited to the networks listed above, and may at least partially include a well-known wireless data network, a well-known telephone network, or a well-known wired and wireless television network.

[0060] The device 100 may input a graph and output information about the graph based on the AI ​​model 300 .

[0061] The form of the chart input to the AI ​​model 300 can be a vertical / horizontal bar chart, a line chart, a pie chart, an area chart, a scatter plot, a radar chart, a histogram and / or a waterfall chart. The form of the chart can be a single form or a combination of multiple forms. However, the chart used in the present invention is not limited to the above form, and the chart can include any form of chart. In addition, in addition to including information that visualizes numerical values, etc. (for example, lines, circles, etc.), the chart can also include text information, and specifically can include annotation information of the chart, a legend or title of the chart, the name of each axis (for example, X-axis, Y-axis, Z-axis, etc.), the numerical value of the original data of the points included in the chart (for example, the numerical value of the X-axis and Y-axis of the points included in the chart), etc.

[0062] The information about the chart output from the AI ​​model 300 may be information indicating characteristics of the chart, and may include meta information, data information, etc. The meta information may include the names of the X-axis and Y-axis, the names of the entity groups recorded in the legend, etc. The data information may include the numerical values ​​of the X-axis and / or Y-axis for each entity, etc.

[0063] According to the present invention, when outputting information about a chart based on an acquired chart (eg, an image), the device 100 may separately output meta information and data information included in the information about the chart.

[0064] The database 200 may store various training data used to train the AI ​​model 300. In addition, the database 200 may store chart images, information about the charts, etc., and in various embodiments, may store output data output by the AI ​​model 300. However, when the training of the AI ​​model 300 is completed, the system 1000 may not include the database 200.

[0065] Figure 1 The case where the database 200 is implemented outside the device 100 is shown. In this case, the database 200 can be connected to the device 100 via a wired or wireless communication. However, this is only one embodiment, and the database 200 can also be implemented as a component of the device 100.

[0066] Figure 1 The AI ​​model 300 is shown as being implemented outside the device 100 (for example, in a cloud-based manner), but is not limited thereto and may be implemented as a component of the device 100 .

[0067] Figure 2 is a block diagram for explaining a configuration of an apparatus for performing a method for extracting meta information and data information of a graph using artificial intelligence according to one embodiment of the present invention.

[0068] like Figure 2 As shown, the device 100 may include a memory 110, a communication module 120, a display 130, an input module 140, and a processor 150. However, the present invention is not limited thereto, and the software components and hardware components of the device 100 may be modified / added / omitted within a range apparent to those skilled in the art according to the required operation. In addition, the device 100 may be replaced by a system, and the device 100 may include multiple devices, and in this case, each component included in the device 100 may be included in at least one of the multiple devices.

[0069] The memory 110 may store data supporting various functions of the device 100 and programs for the operation of the processor 150, store input / output data, and store a plurality of application programs or applications driven on the device, data, commands, and AI models for the operation of the device 100. At least some of the application programs may be downloaded from an external server via wireless communication.

[0070] Such a memory 110 may include at least one type of storage medium among a flash memory type, a hard disk type, a solid state drive type (SSD type), a silicon disk drive type (SDD type), a multimedia card micro type, a card type memory (for example, an SD or XD memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), a programmable ROM (PROM), a magnetic memory, a magnetic disk, and an optical disk.

[0071] In addition, the memory 110 may be separated from the present apparatus and may include a database connected via wired or wireless communication. Figure 1 The illustrated database 200 may be implemented as a component of the memory 110 .

[0072] The communication module 120 may include one or more components that enable communication with an external device, and may include, for example, at least one of a broadcast receiving module, a wired communication module, a wireless communication module, a short-range communication module, or a location information module.

[0073] The wired communication module may include not only various wired communication modules such as a local area network (LAN) module, a wide area network (WAN) module, and a value-added network (VAN) module, but also various cable communication modules such as a universal serial bus (USB), a high-definition multimedia interface (HDMI), a digital video interface (DVI), a recommended standard 232 (RS-232), power line communication, and a plain old telephone service (POTS).

[0074] In addition to the WiFi module and the wireless broadband (WiBro) module, the wireless communication module may also include a wireless communication module for supporting various wireless communication methods such as Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Wideband CDMA (WCDMA), Universal Mobile Telecommunications System (UMTS), Time Division Multiple Access (TDMA), Long Term Evolution (LTE), 4G, 5G or 6G.

[0075] The display 130 displays (outputs) information or data processed in the device 100, data input or output through the AI ​​model 300, etc. In addition, the display 130 may display execution screen information of an application program (e.g., an application) driven on the device 100, or user interface (UI) or graphical user interface (GUI) information according to such execution screen information.

[0076] The input module 140 is used to receive information from a user, and when information is received through the user input unit, the processor 150 may control the operation of the device 100 to correspond to the input information.

[0077] Such an input module 140 may include hardware physical keys (e.g., buttons, dome switches, rollers, roller switches, etc. located on at least one of the front surface, rear surface, and side surface of the device) and software touch keys. As an example, the touch keys may be formed as virtual keys, soft keys, or visual keys displayed on the touch screen type display 130 through software processing, or may be formed as touch keys provided on a portion outside the touch screen. In addition, the virtual keys or visual keys may have various forms and may be displayed on the touch screen, and may be formed as, for example, graphics, text, icons, videos, or a combination thereof.

[0078] The processor 150 can be implemented with a memory and at least one processor (not shown), the memory storing data of an algorithm for controlling the operation of components in the device 100 (including learning or execution of an AI model) or a program for reproducing the algorithm, and the at least one processor using the data stored in the memory to perform the above-mentioned operations. In this case, the memory and the processor can each be implemented as a separate chip, or can be implemented as a single chip.

[0079] In one embodiment, the system 1000 or the apparatus 100 according to the present invention may include at least one processor, and when multiple processors are included, the multiple processors may be included in different apparatuses 100 .

[0080] In addition, the processor 150 may control the operations of the components by combining any one or more of the above components so as to implement various embodiments according to the present disclosure on the device 100 , which will be described below.

[0081] Figure 3 is a block diagram for explaining a method of extracting meta information and data information from a graph according to an embodiment of the present invention.

[0082] Reference Figure 3 , a chart 410 including a chart title, axis names, legend information, numerical values, etc. is input to the image encoder 420. As described above, the form of the chart 410 can be a vertical / horizontal bar chart, a line chart, a pie chart, an area chart, a scatter chart, a radar chart, a histogram and / or a waterfall chart, but is not limited thereto. In addition, the form of the chart can be a single form or a combination of multiple forms. For example, Figure 3 The chart 410 a included in FIG. 4 may be used as the chart 410 .

[0083] The image encoder 420 is an encoder that follows a common encoder architecture and can be implemented as an AI model, specifically ViT or ResNet, but not limited thereto. The image encoder 420 is used to convert the input graph 410a into a first embedding 421 that can be processed by the AI ​​model 300.

[0084] Next, the first embedding 421 output from the image encoder 420 is input to the AI ​​model 400, which includes a meta decoder 430 for performing a meta decoding task and a data decoder 450 for performing data decoding.

[0085] First, the first embedding 421 is first decoded by the meta decoder 430, which can operate in an autoregressive method. The meta decoder 430 can identify meta information from the embedding 421, such as the names of the X-axis and Y-axis, the names of the entity groups recorded in the legend, etc., but the identified meta information is not limited to this. The meta decoder 430 outputs a second embedding 431 including the identified meta information. For example, in the case of using the chart 410a, the second embedding 431 can include the chart title ( <title> Meta Icon< / title> )、X-axis name( <xtitle> Epoch< / xtitle> ), Y-axis name( <ytitle> Experimental results< / ytitle> ), and the list of entity groups recorded in the legend (Epoch|Model1|Model2).

[0086] The present invention has the effect of shortening the length of each data format by extracting the meta information separately from the data information as described below, enabling the data content to be simply represented, thereby reducing the possibility of format errors.

[0087] Next, among the information included in the second embedding 431, the information about the entity 431b serves as a hint that guides the data decoder 450 in the AI ​​model 400 that decodes the data to output an embedding including the original data. When the information about the entity 431b is input to the data encoder as the third embedding 441, a fourth embedding 451 including the original data of each entity is output. In this case, the fourth embedding 451 generates the original data in the format of [x1|y1&;x2|y2&;...xn|yn&;] and distinguishes and independently represents the original data of each entity group. For example, when using Figure 3 When graph 410a is shown, the original data of model 1 ([1|10&3|30&5|50&]) is represented as embedded as shown in 451a, and the original data of model 2 ([1|1&2|4&3|9&4|16&5|25&]) is represented independently.

[0088] In the present invention, the data format output when the graph 410a is input is different from the data format of the conventional generation model as follows. In order to indicate that when the value of Epoch in the graph 410a is 2, the value of Model 2 is "4" and Model 1 has no value, the conventional generation model inputs the value "4" for Model 2 and the value "nan" for Model 1 to indicate a null value (see Figure 6 ). In contrast, the present invention represents data in such a way that, since the output embedding 451a represents the original data of each entity group independently, the data associated with Model 1 does not define the case of Epoch value 2 at all. That is, the present invention can avoid outputting "nan" values ​​that often cause errors in system processing.

[0089] The present invention has the following effects by using a data format that independently represents data for each entity group: eliminating the need to enter difficult-to-handle "nan" values ​​for empty data values, thereby minimizing format errors caused by the inability to recognize "nan" values, and optimizing token usage by eliminating the need to assign tokens to "nan" values.

[0090] Here, the present invention may use a singularized number embedding (SNE) method as a method of representing numbers in the fourth embedding 451 (in contrast to conventional generative models that treat numbers as text). SNE refers to a method of tokenizing each number included in an entity into a single token ( <num>), and output the number to the fourth embedding 451 using the token.

[0091] A multilayer perceptron (MLP) can be used as a single token for tokenization ( <num>) method for extracting numbers. Here, the MLP consists of an input layer, one or more hidden layers, and an output layer, and each layer includes weights and an activation function. The weights used in the MLP of the present invention can be pre-optimized through a training algorithm to predict nonlinear patterns and relationships.

[0092] For example, Figure 4 As shown, in the case where the number "-0412920" is represented in the embedding by SNE, each number included in the entity is tokenized into a single token ( <num>)451b, and input the token 451b into the MLP to represent the number on the fourth embedding 451.

[0093] In this way, the present invention can expect additional performance gains by promoting the separation of tasks between text understanding and data extraction, can achieve efficient training and prediction in the model by reducing the number of tokens representing numbers, and can be reused <num>tokens to significantly improve inference speed.

[0094] In addition, the present invention can further improve the inference speed by inputting tokens used in SNE using a predefined repetitive template to output numbers.

[0095] Specifically, refer to Figure 5a Conventional language models treat numbers as text and use an autoregressive method that predicts each token sequentially. This method infers numbers by feeding the model tokens sequentially at time step t and generating output until the end-of-sentence (EOS) token () is reached. As mentioned above, this method leads to token waste and slow inference speed.

[0096] like Figure 5b As shown, using a single token via SNE <num>Representing numbers in an autoregressive manner can partially improve this problem.

[0097] In addition, if Figure 5c As shown, the performance can be further improved by additionally providing a non-autoregressive method, when using tokens <num>When inferring numbers, this non-autoregressive method uses a predefined repetitive pattern to input multiple <num>tokens to infer multiple numbers simultaneously (in this case, when tokens are input by repeating the template <num>When EOS tokens are first output (), outputs after the first output () will be discarded).

[0098] In this way, the present invention provides a sufficient length of " <num>” token, thereby minimizing the need for autoregressive transfer and further significantly improving inference speed.

[0099] In this way, by processing using the AI ​​model, the data format 460a in which the meta information is recorded and the data format 460b in which the data information included in the fourth embedding is recorded can be finally obtained respectively.

[0100] In addition, the method for extracting meta information and data information from a graph according to an embodiment of the present invention can be performed by referring to Figure 3 The system described is implemented.

[0101] The AI ​​model according to an embodiment of the present invention can be controlled, executed, trained, driven, etc. by a processor, and therefore, at least one of the tasks of executing, training, and driving the AI ​​model can be performed by at least one processor. In addition, the AI ​​model can be stored in a memory, and the feature data according to the present invention can also be stored in the memory.

[0102] In addition, the disclosed embodiments may be implemented in the form of a recording medium having computer-executable instructions stored therein. These instructions may be stored in the form of program code, and when executed by a processor, may generate a program module to perform the operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

[0103] Computer-readable recording media include all types of recording media in which computer-decodable commands are stored, such as read-only memory (ROM), random access memory (RAM), magnetic tapes, magnetic disks, flash memory, optical data storage devices, and the like.

[0104] As described above, the disclosed embodiments have been described with reference to the accompanying drawings. Those skilled in the art will appreciate that the present disclosure may be implemented in forms different from the disclosed embodiments without departing from the technical spirit or essential features of the present disclosure. The disclosed embodiments are illustrative and should not be construed as limiting.< / num> < / num> < / num> < / num> < / num> < / num> < / num> < / num> < / num> < / num> < / num>

Claims

1. A system for implementing an artificial intelligence (AI) model for extracting meta-information and data information included in a chart, the system comprising: at least one processor; as well as at least one memory storing commands or information for causing the at least one processor to perform operations, The operations performed by the command include: Inputting the graph into an image encoder to convert the graph into a first embedding capable of being processed by the AI ​​model; inputting the first embedding to the AI ​​model to output a second embedding including the meta information based on the first embedding, and outputting a fourth embedding including the data information based on a third embedding including information about an entity included in the second embedding; and Each of a first data format and a second data format is output, wherein the meta information included in the second embedding is recorded in the first data format, and the data information included in the fourth embedding is recorded in the second data format.

2. The system according to claim 1, wherein: The data information included in the fourth embedding is differentiated for each entity group.

3. The system according to claim 1, wherein: The meta information includes a title of the chart, names of axes, and names of entity groups included in a legend.

4. The system according to claim 1, wherein: The data information includes numerical value information included in the chart.

5. The system according to claim 1, wherein: The data information is tokenized into a single token for each piece of the data information and the data information is included in the fourth embedding, and When recording the data information included in the fourth embedding in the second data format, tokenized data information is extracted from the single token and recorded in the second data format.

6. The system according to claim 4, wherein: The data information is extracted from a single token using a multi-layer perceptron (MLP).

7. The system according to claim 5 or 6, wherein: When extracting the data information from the single token, a plurality of pieces of data information are simultaneously extracted by inputting the single token together with a predefined repetitive template.

8. A method for extracting meta information and data information included in a graph, the method comprising the following steps: Inputting the graph into an image encoder to convert the graph into a first embedding capable of being processed by an AI model; Inputting the first embedding into the AI ​​model to output a second embedding including the meta information based on the first embedding; outputting a fourth embedding including the data information based on the third embedding including information about the entity included in the second embedding; as well as Each of a first data format and a second data format is output, wherein the meta information included in the second embedding is recorded in the first data format, and the data information included in the fourth embedding is recorded in the second data format.

9. The method according to claim 8, wherein The data information included in the fourth embedding is differentiated for each entity group.

10. The method according to claim 8, wherein The meta information includes a title of the chart, names of axes, and names of entity groups included in a legend.

11. The method according to claim 8, wherein The data information includes numerical value information included in the chart.

12. The method according to claim 8, wherein The data information is tokenized into a single token for each piece of the data information and the data information is included in the fourth embedding, and When recording the data information included in the fourth embedding in the second data format, tokenized data information is extracted from the single token and recorded in the second data format.

13. The method according to claim 12, wherein: The data information is extracted from the single token using a multi-layer perceptron (MLP).

14. The method according to claim 12 or 13, wherein: When extracting the data information from the single token, a plurality of pieces of data information are simultaneously extracted by inputting the single token together with a predefined repetitive template.

15. A program stored in a computer-readable recording medium, the program being used in conjunction with a computer to execute the method according to any one of claims 8 to 14.

Images