Method for generating structured data from original data

Abstract

The present invention relates to a method for generating structured data from original data, the method comprising: generating two or more pieces of extracted original data having different resolutions with respect to original data including text and images; segmenting images in different manners according to relative resolution levels; inputting multiple image patches generated by the segmentation into an encoder so as to derive multiple tokens; inputting the multiple tokens into a decoder so as to extract text information and image information; inputting multiple pieces of text information and multiple pieces of image information into a large language model; and generating structured data from the original data.

Description

How to create structured data from raw data

[0001] The present invention relates to a method for generating structured data from original data, comprising: generating two or more extracted original data having different resolutions for original data including text and images; segmenting an image in different ways according to a relative resolution level; inputting a plurality of image patches generated by segmentation into an encoder to derive a plurality of tokens; inputting the plurality of tokens into a decoder to extract text information and image information; and inputting the plurality of text information and the plurality of image information into a large language model, thereby generating structured data from the original data.

[0002] LaTeX is a free markup language system for writing high-quality documents based on the TeX system. It has the advantage of being effective in writing documents with complex structures, neat and consistent mathematical formulas, symbols, special characters, tables, and mathematical symbols. It is used to write books, presentations, reports, and papers that contain a lot of the aforementioned text, mathematical formulas, and symbols. However, the accuracy of the relationship structure between data (text, images, etc.) that make up documents written by automatic document writing systems such as LaTeX is still low, so post-processing work is required, causing inconvenience to users.

[0003] Specifically, generative AI models are trained using large datasets and deep learning algorithms that learn the underlying structures, relationships, and patterns present in the data, and can output new and unique results based on input prompts, including images, videos, code, music, designs, translations, question answers, and text.

[0004] Conventional data structuring methods include a text data structuring method and device, such as Korean Patent Publication No. 10-2024-0082294. This method sets multiple text units for text extracted from an image, adds combined tags to each of the multiple text units based on a learned natural language processing model and a classified image format, connects text units with related tags, and structures each connected text unit by mapping a first text corresponding to an item name and a second text corresponding to an item value. However, this method does not disclose or suggest a technology for distinguishing between text and images and structuring the relationship between text and images.

[0005] Therefore, there is a need for a technology that can generate structured data in which relationships between elements of text and images are derived.

[0006] The present invention provides a method for generating structured data from original data, the method comprising: generating two or more extracted original data having different resolutions for original data including text and images; segmenting an image in different ways according to a relative resolution level; inputting a plurality of image patches generated by segmentation into an encoder to derive a plurality of tokens; inputting the plurality of tokens into a decoder to extract text information and image information; and inputting the plurality of text information and the plurality of image information into a large language model; and providing a method for generating structured data from the original data.

[0007] In order to solve the above problem, in one embodiment of the present invention, a method for generating structured data from original data is provided, which is performed in a computing system including one or more processors and one or more memories, the method comprising: an extraction original data generation step of generating two or more extracted original data having different resolutions from the original data; an extraction original data segmentation step of performing image segmentation in different ways according to relative resolution levels on the two or more extracted original data; a token derivation step of inputting a plurality of image patches generated by segmenting the two or more extracted original data into an encoder to derive a plurality of tokens including feature information in a vector form; an information extraction step of inputting the plurality of tokens into a decoder to extract a plurality of text information for text included in the original data and a plurality of image information for images included in the original data; and a structured data generation step of inputting the plurality of text information and the plurality of image information into a large language model to generate structured data in which correlations between elements of the plurality of text information and the plurality of image information are derived.

[0008] In one embodiment of the present invention, the extraction source data division step divides each of the two or more extraction source data into a plurality of segments of a certain size in a grid shape to generate the plurality of image patches, and by dividing the extraction source data generated with a relatively high resolution into smaller and larger segments than the extraction source data generated with a relatively low resolution, the sizes and numbers of the plurality of image patches can be generated differently for each extraction source data.

[0009] In one embodiment of the present invention, the token derivation step may input a plurality of image patches generated by dividing the two or more extracted original data into a single encoder at the same time to derive the plurality of tokens, and the information extraction step may input tokens derived from each of the plurality of image patches divided from each of the two or more extracted original data into a single decoder at the same time to derive the plurality of text information and the plurality of image information.

[0010] In one embodiment of the present invention, the text information may include text extracted from the image patch by applying optical character recognition (OCR) by the encoder and decoder; and text area information including position coordinates where the text is located in the original data; and the image information may include image tag information in which an image included in the image patch is labeled; and image area information including position coordinates where the image is located in the original data.

[0011] In one embodiment of the present invention, the text information further includes meta information in which one of the constituent elements of the original data is labeled by the decoder with respect to the text included in the text information, the constituent element includes a title, a paragraph, and a footnote, and the meta information can be labeled based on the position, structure, and format of the text.

[0012] In one embodiment of the present invention, the macro language model corresponds to an artificial neural network-based language model that is trained to generate structured data in which the meta information is further reflected, and to receive the text information and determine the constituent elements of the meta information based on the text included in the text information, and the structured data can be generated by reflecting the meta information determined by the macro language model when the constituent elements of the meta information derived by the decoder are different from the constituent elements of the meta information determined by the macro language model.

[0013] In one embodiment of the present invention, the structured data may include: a plurality of text information reflecting the arrangement of text included in the original data; first relationship information including a relationship structure between the text information and the corresponding image information based on a correlation between the information included in the text information and the information included in the image information; and second relationship information including a relationship structure between the corresponding text information based on a correlation between the information included in each of the plurality of text information.

[0014] In one embodiment of the present invention, text data extracted from original data by OCR inference can be more efficiently reconstructed to maintain the structure of paragraphs and improve the accuracy of data structuring while rearranging footnotes.

[0015] In one embodiment of the present invention, by generating two or more images with different resolutions for a single image, dividing them into different numbers according to the resolution level, and inputting them into an encoder, the effect of generating structured data in the form of a tree structure in which the correlation between data is clearly derived can be achieved.

[0016] In one embodiment of the present invention, by inputting a plurality of segmented image patches into an encoder and deriving them as tokens in the form of direction vectors, the information contained in the image patches is maintained without being damaged, thereby efficiently computing the process of recognizing text data.

[0017] In one embodiment of the present invention, by distinguishing text information and image information through labeling and generating structured data that includes a relationship structure between the text information and the image information, it is possible to achieve the effect of generating structured data even from original data that includes an image.

[0018] In one embodiment of the present invention, by extracting information about text and the position coordinates at which the text is located in the original data, it is possible to generate data structured identically to the arrangement of texts constituting the original data.

[0019] FIG. 1 schematically illustrates structured data generated from original data according to one embodiment of the present invention.

[0020] FIG. 2 schematically illustrates the execution steps of a method for generating structured data from original data according to one embodiment of the present invention.

[0021] FIG. 3 schematically illustrates a process for generating multiple image patches according to one embodiment of the present invention.

[0022] Figure 4 schematically illustrates the execution process of a token derivation step according to one embodiment of the present invention.

[0023] Figure 5 schematically illustrates the execution process of an information extraction step according to one embodiment of the present invention.

[0024] Figure 6 schematically illustrates the configuration of text information and the configuration of image information according to one embodiment of the present invention.

[0025] Figure 7 illustrates text area information and image area information according to one embodiment of the present invention.

[0026] Figure 8 schematically illustrates meta information according to one embodiment of the present invention.

[0027] Figure 9 schematically illustrates a process of generating structured data by reflecting meta information according to one embodiment of the present invention.

[0028] Figure 10 schematically illustrates the configuration of structured data according to one embodiment of the present invention.

[0029] FIG. 11 exemplarily illustrates the internal configuration of a computing device according to one embodiment of the present invention.

[0030] Hereinafter, various embodiments and / or aspects are now disclosed with reference to the drawings. In the following description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of one or more aspects. However, it will be apparent to one skilled in the art that such aspects may be practiced without these specific details. The following description and the attached drawings detail specific exemplary aspects of one or more aspects. However, these aspects are exemplary, and it is to be understood that any of the various methods within the principles of the various aspects may be utilized, and the description is intended to encompass all such aspects and their equivalents.

[0031]

[0032] Additionally, various aspects and features will be presented by systems that may include a number of devices, components, and / or modules. It is also to be understood and appreciated that various systems may include additional devices, components, and / or modules, and / or may not include all of the devices, components, and modules discussed in connection with the drawings.

[0033] The terms "embodiment," "example," "aspect," and "example" used herein may not be construed as implying that any aspect or design described is better or advantageous than other aspects or designs. The terms "part," "component," "module," "system," and "interface" used below generally refer to computer-related entities, and may refer to, for example, hardware, a combination of hardware and software, or software.

[0034] Additionally, it should be understood that the terms "comprises" and / or "comprising" imply the presence of the features and / or components, but do not preclude the presence or addition of one or more other features, components and / or groups thereof.

[0035] Additionally, terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. The term and / or includes a combination of a plurality of related described items or any of a plurality of related described items.

[0036] Additionally, in the embodiments of the present invention, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by those of ordinary skill in the art to which the present invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined in the embodiments of the present invention.

[0037]

[0038] FIG. 1 schematically illustrates structured data generated from original data according to one embodiment of the present invention.

[0039]

[0040] Specifically, the computing system of the present invention can efficiently reconstruct elements constituting the original data by generating structured data such that relationships between texts or between texts and images are derived from the original data. That is, the present invention is characterized by reconstructing the original data while taking into account the alignment and structure of the elements constituting the original data so that it can be effectively utilized in a system that automatically creates special format documents such as papers or publications, such as LaTeX.

[0041] Meanwhile, the original data of the present invention is document data in the form of an image or text corresponding to an electronic document (e.g., PDF) having one or more pages. In the embodiment of the present invention described below, for the convenience of explanation, a method of generating structured data from original data corresponding to one page will be covered. More specifically, according to the embodiment of the present invention, the original data may be document data, may be document data including image elements, and may be document data in which the structure of paragraphs is standardized to a certain extent.

[0042]

[0043] FIG. 2 schematically illustrates the execution steps of a method for generating structured data from original data according to one embodiment of the present invention.

[0044]

[0045] As illustrated in FIG. 2, a method for generating structured data from original data, which is performed in a computing system including one or more processors and one or more memories, comprises: an extraction original data generation step (S100) of generating two or more extracted original data having different resolutions from the original data; an extraction original data segmentation step (S200) of performing image segmentation in different ways according to relative resolution levels on the two or more extracted original data; a token derivation step (S300) of inputting a plurality of image patches generated by segmenting the two or more extracted original data into an encoder to derive a plurality of tokens including feature information in vector form; an information extraction step (S400) of inputting the plurality of tokens into a decoder to extract a plurality of text information for texts included in the original data and a plurality of image information for images included in the original data; and a structured data generation step (S500) of inputting the plurality of text information and the plurality of image information into a large language model to generate structured data in which correlations between elements of the plurality of text information and the plurality of image information are derived.

[0046]

[0047] Specifically, the computing system may include an extraction source data generation unit, an extraction source data division unit, a token extraction unit, an information extraction unit, and a structured data generation unit, and the extraction source data generation step (S100) is performed by the extraction source data generation unit, and two or more extraction source data having different resolutions are generated for the original data. It is preferable that the two or more extraction source data have a resolution higher than a level at which analysis is possible when input to an encoder.

[0048] The extraction source data segmentation step (S200) is performed by the extraction source data segmentation unit, and performs image segmentation in different ways depending on the relative resolution levels of each of the two or more extraction source data. As a result of the image segmentation, the extraction source data is converted into multiple image patches, and all of the multiple image patches converted from a single extraction source data can be converted to the same size.

[0049] The token derivation step (S300) is performed by the token derivation unit, and inputs a plurality of image patches generated by dividing the two or more extracted original data into an encoder to derive a plurality of tokens including feature information in vector form. The encoder may include a vision encoder, and the tokens include feature information of the image patches in vector form, and the plurality of tokens may be derived corresponding to the number of the plurality of image patches.

[0050] The information extraction step (S400) is performed by the information extraction unit, and inputs a plurality of tokens into a decoder to extract a plurality of text information about texts included in the original data and a plurality of image information about images included in the original data. That is, the decoder classifies and extracts text information and image information so as to be able to match the correlations between elements of the plurality of text information and the plurality of image information.

[0051] The structured data generation step (S500) is performed by the structured data generation unit, and inputs the plurality of text information and the plurality of image information into a macro language model to generate structured data in which a correlation is derived between text information and text information, or text information and image information. For example, the structured data may derive a correlation between text corresponding to a title and text corresponding to a paragraph, text corresponding to a paragraph and text corresponding to a footnote, or text corresponding to a paragraph and an image corresponding to an image.

[0052]

[0053] FIG. 3 schematically illustrates a process for generating multiple image patches according to one embodiment of the present invention.

[0054]

[0055] As illustrated in FIG. 3, a method for generating structured data from original data includes an extraction original data generation step (S100) of generating two or more extraction original data having different resolutions with respect to the original data; and an extraction original data segmentation step (S200) of performing image segmentation in different ways according to the relative resolution levels of the two or more extraction original data.

[0056] In addition, the above extraction source data division step (S200) divides each of the two or more extraction source data into a plurality of segments of a certain size in a grid shape to generate the plurality of image patches, and divides the extraction source data generated with a relatively high resolution into smaller and larger segments than the extraction source data generated with a relatively low resolution, thereby generating the plurality of image patches with different sizes and numbers for each extraction source data.

[0057]

[0058] Specifically, the present invention generates two or more extracted original data having different resolutions for original data from which structured data is to be generated, and performs image segmentation on the two or more extracted original data in different ways according to different resolution levels.

[0059] More specifically, the extraction source data segmentation step (S200) segments each of the two or more extraction source data into a plurality of segments of a certain size in a grid shape to generate the plurality of image patches, and as an embodiment of the present invention, extraction source data #1 generated with a relatively high resolution is segmented into a smaller number of segments than extraction source data #2 generated with a relatively low resolution, so that the number of image patches segmented from the extraction source data #1 is greater than the number of image patches segmented from the extraction source data #2.

[0060] The present invention is characterized in that, instead of inputting single data into an encoder as in the conventional technology, as shown in FIG. 3, original data is input into an encoder in a segmented form, thereby generating structured data arranged in a tree structure based on detailed information included in a plurality of image patches having a high resolution and overall structural information included in a plurality of image patches having a low resolution.

[0061]

[0062] Figure 4 schematically illustrates the execution process of the token derivation step (S300) according to one embodiment of the present invention.

[0063]

[0064] As illustrated in FIG. 4, a method for generating structured data from original data includes a token derivation step (S300) of inputting multiple image patches generated by dividing two or more extracted original data into an encoder to derive multiple tokens including feature information in vector form.

[0065] In addition, the token derivation step (S300) derives the plurality of tokens by simultaneously inputting multiple image patches generated by dividing the two or more extracted original data into a single encoder.

[0066]

[0067] Specifically, multiple image patches generated by dividing two or more extracted source data through the token derivation step (S300) are simultaneously input to a single encoder regardless of the resolution level of the extracted source data, and the encoder derives multiple tokens including feature information in vector form. As in one embodiment of the present invention illustrated in FIG. 4, the encoder may include a vision encoder, and may derive multiple tokens representing various feature information extracted from the image patches in the form of high-dimensional vectors through a CNN or linear transformation algorithm.

[0068] In one embodiment illustrated in FIG. 4, four image patches generated by dividing the extracted original data into a 2X2 format are input to the encoder to derive four tokens, and in another embodiment, three extracted original data are generated for the original data, one extracted original data is divided into a 2X2 format, another extracted original data is divided into a 3X3 format, and another extracted original data is divided into a 4X4 format, the token derivation unit can input 29 (=4+9+16) image patches into the encoder at the same time through the token derivation step (S300) and derive 26 tokens.

[0069]

[0070] Figure 5 schematically illustrates the execution process of the information extraction step (S400) according to one embodiment of the present invention.

[0071]

[0072] As illustrated in FIG. 5, a method for generating structured data from original data includes an information extraction step (S400) of inputting the plurality of tokens into a decoder to extract a plurality of text information for texts included in the original data and a plurality of image information for images included in the original data.

[0073] In addition, the above information extraction step (S400) simultaneously inputs tokens derived from each of a plurality of image patches divided from each of the two or more extracted original data into a single decoder to derive the plurality of text information and the plurality of image information.

[0074]

[0075] Specifically, the information extraction unit extracts multiple text information from the text constituting the original data and multiple image information from the image constituting the original data by inputting the multiple tokens into a decoder through the information extraction step (S400). More specifically, as illustrated in FIG. 5, multiple tokens are inputted simultaneously into a single decoder to derive multiple text information and multiple image information.

[0076] The decoder extracts multiple text information and multiple image information based on vector-type feature information included in multiple tokens, and can extract one text information or one image information from two or more tokens based on the feature information included in the tokens, and can also extract two or more text information or two or more image information from one token.

[0077]

[0078] Figure 6 schematically illustrates the configuration of text information and image information according to one embodiment of the present invention. Additionally, Figure 7 exemplarily illustrates text area information and image area information according to one embodiment of the present invention.

[0079]

[0080] As illustrated in FIGS. 6 and 7, the text information includes text extracted from the image patch by applying optical character recognition (OCR) by the encoder and decoder; and text area information including position coordinates where the text is located in the original data; and the image information includes image tag information in which an image included in the image patch is labeled; and image area information including position coordinates where the image is located in the original data.

[0081]

[0082] As described above in FIG. 5, the present invention generates text information for texts constituting original data, and generates image information for images constituting the original data. Specifically, in the embodiment illustrated in FIG. 6, when text information for text included in the first paragraph of the original data is generated, the text information may include text area information including the text and the position coordinates where the text is located in the original data.

[0083] More specifically, the text included in the text information can be extracted from the image patch through optical character recognition (OCR) by an encoder and a decoder, and the position coordinates of the corresponding text can be derived as coordinates in the form of (x, y) by setting an xy coordinate system targeting the original data, as illustrated in Fig. 7. That is, the text information includes the text constituting the original data and the position of the corresponding text in the original data.

[0084] In addition, when image information for an image constituting the original data in one embodiment illustrated in FIG. 6 is generated, the image information may include image tag information in which an image included in an image patch is labeled and image region information including the position coordinates at which the image is located in the original data. More specifically, in one embodiment of the present invention, the image tag information is labeled as an image, thereby indicating whether an image exists in the original data, and the position coordinates of the image may correspond to coordinates in the form of (x, y) in the xy coordinate system, similar to text information. That is, the image information includes whether an image exists in the original data and the position of the image.

[0085]

[0086] In addition, when the original data is data consisting of two or more pages, it is preferable that the number of pages be further included in each of the text area information and the image area information so that structured data can be generated in which the relationship between elements of the plurality of text information and the plurality of image information is specifically derived.

[0087]

[0088] Figure 8 schematically illustrates meta information according to one embodiment of the present invention.

[0089]

[0090] As illustrated in FIG. 8, the text information further includes meta information in which one of the constituent elements of the original data is labeled by the decoder for the text included in the text information, the constituent elements include a title, a paragraph, and a footnote, and the meta information is labeled based on the location, structure, and format of the text.

[0091]

[0092] Specifically, the decoder can label each text and image that constitutes the original data, and the text information may further include meta information that labels any one of the constituent elements, including a title, paragraph, and footnote, for the text contained in the text information. Preferably, the meta information may be labeled based on the location, structure, and format of the text.

[0093] Typically, titles are placed at the beginning of a document, can be centered or left-aligned on the page, have a short and concise structure, and may be in large, bold font.

[0094] Additionally, paragraphs can be positioned under a heading, multiple paragraphs can be positioned consecutively, consist of multiple sentences, can be separated by indentation or margins, and can include sentences ending in periods, exclamation marks, question marks, etc.

[0095] Additionally, footnotes are located at the bottom of the page, contain numbers that match the numbers for specific words, phrases, or sentences contained within the paragraph, have a short and concise structure, and may be in small type and a thin font.

[0096]

[0097] As described above, by labeling the texts constituting the original data with their constituent elements, the structure of the footnotes and paragraphs constituting the original data can be efficiently rearranged. In other words, the present invention can effectively generate highly reliable structured data by specifically determining the relationships between texts based on the metadata that label the constituent elements of the text.

[0098]

[0099] Meanwhile, the component elements illustrated in FIG. 8 are one embodiment of the present invention, and the component elements in other embodiments are not limited thereto, and may include more component elements depending on the format of the original data.

[0100]

[0101] Figure 9 schematically illustrates a process of generating structured data by reflecting meta information according to one embodiment of the present invention.

[0102]

[0103] As illustrated in FIG. 9, the macro language model corresponds to an artificial neural network-based language model that is trained to generate structured data in which the meta information is further reflected, and to receive the text information and determine the constituent elements of the meta information based on the text included in the text information, and the structured data is generated by reflecting the meta information determined by the macro language model when the constituent elements of the meta information derived by the decoder are different from the constituent elements of the meta information determined by the macro language model.

[0104]

[0105] As described above in FIG. 8, meta information can be generated by a decoder labeling text as one of the constituent elements. However, if the macro language model corresponds to an artificial neural network-based language model that is trained to determine the constituent elements of meta information based on text included in the text information, the macro language model can also generate meta information by labeling the corresponding text as one of the constituent elements. That is, the present invention can once again confirm meta information labeled by the decoder through meta information labeled by the macro language model.

[0106] Specifically, when the constituent elements of meta information labeled by a decoder ('title' in the case of FIG. 10) and the constituent elements of meta information labeled by a macro language model ('paragraph' in the case of FIG. 10) are different for a single piece of text information, the present invention can generate structured data by inputting text information including further meta information determined by the macro language model into the macro language model.

[0107]

[0108] Figure 10 schematically illustrates the configuration of structured data according to one embodiment of the present invention.

[0109]

[0110] As illustrated in Fig. 10, a method for generating structured data from original data includes a structured data generation step (S500) of inputting the plurality of text information and the plurality of image information into a large language model to generate structured data in which a relationship between elements of the plurality of text information and the plurality of image information is derived.

[0111] The above structured data includes: a plurality of text information reflecting the arrangement of texts included in the original data; first relationship information including a relationship structure between the text information and the image information based on a correlation between the information included in the text information and the information included in the image information; and second relationship information including a relationship structure between the text information based on a correlation between the information included in each of the plurality of text information.

[0112]

[0113] Specifically, structured data includes a plurality of text information, and the plurality of text information included in the structured data is included in a form that reflects the arrangement of the text included in the original data based on a plurality of text area information and a plurality of image area information. As an embodiment of the present invention, FIG. 10 illustrates text information in which the arrangement of the texts of Paragraph #2 and Paragraph #3 is reflected and connected.

[0114] Additionally, structured data further includes first relationship information, which includes a relationship structure between text information and image information based on the correlation between the information contained in the text information and the information contained in the image information. As an example of the present invention, the first relationship information illustrated in FIG. 10 includes a connection structure between text composed of multiple paragraphs and an image described by the text.

[0115] Additionally, the structured data further includes second relationship information, which includes a relationship structure between the text information based on the correlation between the information contained in each of the plurality of text information. As an example of the present invention, the second relationship information illustrated in FIG. 10 includes a connection structure between a text composed of multiple paragraphs and footnotes corresponding to supplementary explanations for some sentences contained in the text.

[0116]

[0117] That is, the present invention can have the effect of enabling a plurality of texts and a plurality of images constituting the original data to be rearranged with high accuracy while maintaining the texture between each other by generating structured data from the original data through the above-described execution processes.

[0118]

[0119] FIG. 11 exemplarily illustrates the internal configuration of a computing device according to one embodiment of the present invention.

[0120]

[0121] The computing system mentioned in the description of FIG. 1 may include components of the computing device (11000) illustrated in FIG. 11 described below.

[0122]

[0123] As illustrated in FIG. 11, a computing device (11000) may include at least one processor (11100), memory (11200), peripheral interface (11300), input / output subsystem (I / O subsystem) (11400), power circuit (11500), and communication circuit (11600).

[0124]

[0125] Specifically, the memory (11200) may include, for example, a high-speed random access memory, a magnetic disk, SRAM, DRAM, ROM, flash memory, or non-volatile memory. The memory (11200) may include a software module, a set of instructions, or other various data required for the operation of the computing device (11000).

[0126] At this time, access to the memory (11200) from other components such as the processor (11100) or the peripheral interface (11300) may be controlled by the processor (11100). The processor (11100) may be configured as a single or multiple processors, and may include processors in the form of GPUs and TPUs to improve the processing speed.

[0127] The peripheral interface (11300) may connect input and / or output peripherals of the computing device (11000) to the processor (11100) and the memory (11200). The processor (11100) may execute software modules or instruction sets stored in the memory (11200) to perform various functions for the computing device (11000) and process data.

[0128] The input / output subsystem (11400) can couple various input / output peripheral devices to the peripheral interface (11300). For example, the input / output subsystem (11400) can include a controller for coupling peripheral devices such as a monitor, a keyboard, a mouse, a printer, or, if necessary, a touch screen or a sensor to the peripheral interface (11300). In another aspect, the input / output peripheral devices can be coupled to the peripheral interface (11300) without going through the input / output subsystem (11400).

[0129] The power circuit (11500) may supply power to all or part of the components of the terminal. For example, the power circuit (11500) may include a power management system, one or more power sources such as a battery or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, or any other components for power generation, management, and distribution.

[0130] The above communication circuit (11600) may enable communication with another computing device using at least one external port. Alternatively, as described above, the communication circuit (11600) may, if necessary, include an RF circuit to transmit and receive RF signals, also known as electromagnetic signals, thereby enabling communication with another computing device.

[0131]

[0132] This embodiment of FIG. 11 is only an example of the computing device (11000), and the computing device (11000) may have some of the components illustrated in FIG. 11 omitted, may further include additional components not illustrated in FIG. 11, or may have a configuration or arrangement that combines two or more components. For example, a computing device for a communication terminal in a mobile environment may further include a touch screen or a sensor, in addition to the components illustrated in FIG. 11, and the communication circuit (1160) may include a circuit for RF communication of various communication methods (Wi-Fi, 3G, LTE, 5G, 6G, Bluetooth, NFC, Zigbee, etc.). Components that can be included in the computing device (11000) may be implemented as hardware including one or more signal processing or application-specific integrated circuits, software, or a combination of both hardware and software.

[0133] Methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed by various computing devices and recorded on a computer-readable medium. In particular, the program according to the present embodiment may be configured as a PC-based program or an application exclusively for mobile terminals. An application to which the present invention is applied may be installed on a user terminal through a file provided by a file distribution system. For example, the file distribution system may include a file transmission unit (not shown) that transmits the file at the request of the user terminal.

[0134]

[0135] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0136] The software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing device to perform a desired operation or, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for interpretation by the processing device or for providing instructions or data to the processing device. The software may be standardized and stored or executed in a standardized manner on a network-connected computing device. The software and data may be stored on one or more computer-readable recording media.

[0137] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0138]

[0139] In one embodiment of the present invention, text data extracted from original data by OCR inference can be more efficiently reconstructed to maintain the structure of paragraphs and improve the accuracy of data structuring while rearranging footnotes.

[0140] In one embodiment of the present invention, by generating two or more images with different resolutions for a single image, dividing them into different numbers according to the resolution level, and inputting them into an encoder, the effect of generating structured data in the form of a tree structure in which the correlation between data is clearly derived can be achieved.

[0141] In one embodiment of the present invention, by inputting a plurality of segmented image patches into an encoder and deriving them as tokens in the form of direction vectors, the information contained in the image patches is maintained without being damaged, thereby efficiently computing the process of recognizing text data.

[0142] In one embodiment of the present invention, by distinguishing text information and image information through labeling and generating structured data that includes a relationship structure between the text information and the image information, it is possible to achieve the effect of generating structured data even from original data that includes an image.

[0143] In one embodiment of the present invention, by extracting information about text and the position coordinates at which the text is located in the original data, it is possible to generate data structured identically to the arrangement of texts constituting the original data.

[0144]

[0145] Although the embodiments have been described with limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents. Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims

A method for generating structured data from original data, which is performed on a computing system including 1.1 or more processors and 1 or more memories, An extraction source data generation step for generating two or more extraction source data with different resolutions from the original data; An extraction source data segmentation step that performs image segmentation in different ways according to the relative resolution level of the above two or more extraction source data; A token derivation step for inputting multiple image patches generated by dividing the above two or more extracted original data into an encoder to derive multiple tokens including feature information in vector form; An information extraction step of inputting the plurality of tokens into a decoder to extract a plurality of text information for text included in the original data and a plurality of image information for images included in the original data; and A method for generating data, comprising: a structured data generation step of inputting the plurality of text information and the plurality of image information into a large language model to generate structured data in which a relationship between elements of the plurality of text information and the plurality of image information is derived; 2. In claim 1, The above extraction source data division step is, Each of the above two or more extracted original data is divided into multiple segments of a certain size in a grid shape to generate the multiple image patches. A method for generating data by dividing extracted original data generated with a relatively high resolution into smaller and larger segments than extracted original data generated with a relatively low resolution, thereby generating multiple image patches with different sizes and numbers for each extracted original data.

3. In claim 1, The above token derivation step is: The above two or more extracted original data are divided and multiple image patches are simultaneously input into a single encoder to derive the above multiple tokens, The above information extraction step is, A method for generating data by simultaneously inputting tokens derived from each of a plurality of image patches divided from each of the two or more extracted original data into a single decoder to derive the plurality of text information and the plurality of image information.

4. In claim 1, The above text information is, Text extracted from the image patch by applying optical character recognition (OCR) by the encoder and decoder; and Includes text area information including the location coordinates where the text is located in the original data; The above image information is, Image tag information labeling the image included in the above image patch; and A method for generating data, comprising image area information including location coordinates where the image is located in the original data.

5. In claim 4, The above text information further includes meta information in which one of the constituent elements of the original data is labeled by the decoder for the text included in the text information, The above components include titles, paragraphs, and footnotes, A method for generating data in which the above metadata is labeled based on the location, structure and format of the text.

6. In claim 5, The above large language model is, Generate structured data that further reflects the above meta information, It corresponds to an artificial neural network-based language model that is trained to input the above text information and determine the constituent elements of the above meta information based on the text included in the above text information. The above structured data is, A method for generating data in which the meta information determined by the macro language model is reflected when the constituent elements of the meta information derived by the decoder and the constituent elements of the meta information determined by the macro language model are different.

7. In claim 1, The above structured data is, A plurality of text information reflecting the arrangement of text included in the above original data; First relationship information including a relationship structure between the text information and the image information based on the correlation between the information included in the text information and the information included in the image information; and A method for generating data, comprising second relationship information including a relationship structure between text information based on the correlation between information included in each of the plurality of text information.

Images