Adaptive Template-Based Document Generation Method and Apparatus

The adaptive template-based document generation method addresses the limitations of existing systems by inferring user intent from multimodal input, dynamically selecting or generating templates, and integrating visual information, resulting in high-quality, user-centric document generation.

KR103044689B1Active Publication Date: 2026-11-11INFOGRAB LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
KR1020250188794
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2025-12-03
Publication Date
2026-11-11
Estimated Expiration
2045-12-03

AI Technical Summary

Technical Problem

Existing document generation systems fail to accurately reflect user intent or context due to reliance on single-modal input, lack of image interpretation, and static template selection, leading to degraded document quality and efficiency.

Method used

An adaptive template-based document generation method that infers user intent from multimodal input, including text and images, using a multimodal encoder and pre-trained language models to determine and generate or select templates dynamically, integrating visual information into structured documents.

Benefits of technology

Enables advanced automatic document generation that adapts to diverse user requirements, improves user convenience by integrating visual elements, and ensures structural and semantic consistency with user intent and context.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112025136077443-PAT00002_ABST
    Figure 112025136077443-PAT00002_ABST
Patent Text Reader

Abstract

The present invention relates to a document generation technology, and more specifically, to a technology for generating user-customized documents by selecting or generating an adaptive template based on multimodal input and inference of document creation intent. An adaptive template-based document generation method according to an embodiment of the present specification for achieving the above-described objective may include: a step of obtaining a multimodal input including at least one of a user's context information and text and an image; a step of inferring a user's intention to create a document based on the multimodal input; a step of determining a template corresponding to the user based on the context information and the intention to create a document based on the context information and the intention to create a document based on the template; and a step of generating a document based on the context information, the intention to create a document, and the template.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present invention relates to a document generation technology, and more specifically, to a technology for generating user-customized documents by selecting or generating an adaptive template based on multimodal input and inference of document creation intent. Background Technology

[0002] With the advancement of information technology, document creation has become an essential element in various industries and work environments. There is a growing demand for the rapid generation of documents of consistent format and quality across numerous fields, including corporate reports, government administrative documents, legal documents, and medical records. Consequently, the development of automated document generation systems is actively underway; in particular, the accuracy and efficiency of automated document generation technology are significantly improving due to the advancement of Natural Language Processing (NLP) technology and the commercialization of AI-based generative models.

[0003] However, most existing document generation systems rely on single-modal input (primarily text) and suffer from the problem of failing to accurately reflect the user's actual intent or context. For example, when a user intends to generate a report for a specific field, existing systems have limitations in that they provide uniform formats or rely solely on simple keyword-based auto-complete functions without considering the field, purpose, tone, or target audience of the document. This approach not only degrades document quality but can also reduce efficiency by requiring manual user intervention.

[0004] Furthermore, in recent business environments, multimodal content containing visual information—such as images, charts, and diagrams—as well as simple text, serves as a crucial element in document organization. However, existing technologies lack the capability to effectively interpret and structure such image-based input to seamlessly integrate it into documents. For instance, even if a user uploads an image containing a chart, existing systems merely insert or ignore it, exhibiting technical limitations in their inability to analyze the image's information and meaningfully reflect it within the document content.

[0005] Furthermore, although various 'contextual information'—such as a user's writing history, preferred writing style or language, and the laws and regulations of the target country—can serve as important factors in document generation, technologies for template selection or document generation that reflect this are still lacking. To automatically construct a document structure suited to a user's background or purpose, input data alone is insufficient; the ability to comprehensively analyze users' past behavioral patterns and preference data is required.

[0006] Most existing template-based document generation technologies are also implemented in a static form, remaining limited to selecting one of predefined templates. This approach has the disadvantage of being unable to accommodate diverse and complex user requirements and failing to adequately respond to new types of documents. Furthermore, when selecting the template that best matches the user's intent from multiple options, reliance on simple keyword matching or rule-based methods is frequently observed; consequently, these methods often fail to accurately reflect the user's intent or provide the optimal template.

[0007] Therefore, technology is required that can dynamically adjust the structure and content of a document by comprehensively considering user input (text, images, etc.), contextual information (user history, language, regulations, etc.), and document creation intent (purpose, audience, tone, etc.), and analyzing similarity with existing templates or automatically generating new templates based on this. Furthermore, when image-based input is included, advanced document generation technology is needed to structure the visual information of the images and automatically insert it into the document, ensuring that the generated document maintains structural and semantic consistency with the template.

[0008] The aforementioned background technology is technical information that the inventor possessed for the derivation of the present invention or acquired during the process of deriving the present invention, and it cannot be considered technology that was disclosed to the general public prior to the filing of the present invention. The problem to be solved

[0009] The various embodiments described in this specification aim to provide an adaptive template determination technology capable of selecting an appropriate existing template or generating a new template based on the user's context information and intent in order to solve the aforementioned problems. means of solving the problem

[0010] An adaptive template-based document generation method according to an embodiment of the present specification for achieving the above-described objective may include: a step of obtaining a multimodal input including at least one of a user's context information and text and an image; a step of inferring a user's intention to create a document based on the multimodal input; a step of determining a template corresponding to the user based on the context information and the intention to create a document based on the context information and the intention to create a document based on the template; and a step of generating a document based on the context information, the intention to create a document, and the template.

[0011] The above context information may include at least one of user identification information, applicable domain or regulation, preferred language, and past document creation history.

[0012] The step of inferring the document creation intent may include: a step of converting the multimodal input into a latent vector representation using a multimodal encoder that encodes the multimodal input; and a step of using a pre-trained language model that classifies at least one of a document type, request format, target audience, tone, and security level based on the latent vector representation.

[0013] The step of determining the above template may include determining the template based on the similarity between a first embedding corresponding to each of a plurality of preset templates and a second embedding of the document creation intention.

[0014] The step of determining the above template may include: a step of generating a template based on the context information and the document creation intention when the maximum similarity among the similarities is less than a threshold value; and a step of selecting at least one template among a plurality of preset templates corresponding to the threshold value among the similarities when the maximum similarity among the similarities is greater than or equal to a threshold value.

[0015] The step of generating the above document may include generating the document by inputting the context information and the document creation intention to the generation model, based on the structure and placeholders defined in the determined template.

[0016] If the above image input includes a chart or diagram, the step of generating the document may further include the step of analyzing the visual elements of the chart to convert them into structured data and inserting the converted data into a placeholder corresponding to the template.

[0017] The above image input is an image of a document containing text content and can be used as an optical compression medium for the text content.

[0018] The multimodal input acquisition step includes the step of processing the image through a vision encoder to generate a sequence of vision tokens, and the sequence of vision tokens may consist of fewer tokens than the number of text tokens when the text content is tokenized into digital text.

[0019] The above-described visual encoder includes a first encoder block based on window attention for extracting local features of an image and a second encoder block based on global attention for integrating the global context of an image, and a convolution-based compression module for reducing the number of output tokens of the first encoder block may be serially connected between the first encoder block and the second encoder block.

[0020] The step of generating the above document may include the step of using a large-scale language model (LLM)-based decoder that receives the sequence of the vision tokens and the template as input, restores text content compressed in the vision tokens, and generates text according to the structure of the template.

[0021] The step of generating the above document may perform guided decoding by using the generated structured data as a constraint when the LLM-based decoder generates text, so that the generated text complies with the hierarchical section structure of the template and the semantic properties of the placeholders.

[0022] A server according to one embodiment of the present specification for achieving the above-described objective comprises: a processor; and a memory for loading a computer program executed by the processor. The computer program may be configured to perform the following operations: acquiring a multimodal input including at least one of a user's context information and text and an image; inferring a user's intention to create a document based on the multimodal input; determining a template corresponding to the user based on the context information and the intention to create a document based on the context information and the intention to create a document based on the template.

[0023] A computer-readable recording medium storing instructions for executing a method according to one embodiment of the present specification for achieving the above-described objectives. Effects of the invention

[0024] According to one embodiment of the present invention, advanced automatic document generation is possible by comprehensively considering user context and various multimodal inputs.

[0025] In addition, it allows for adaptation to various document types by enabling not only the use of existing templates based on similarity but also the creation of real-time templates when necessary.

[0026] In addition, by automatically analyzing and converting visual elements within images and reflecting them in the document content, user convenience and information delivery capabilities can be improved.

[0027] The effects of the present invention are not limited to those mentioned above. Brief explanation of the drawing

[0028] FIG. 1 is a diagram schematically illustrating an adaptive template-based document generation system according to one embodiment of the present invention. FIG. 2 is a flowchart illustrating the overall operation of an adaptive template-based document generation method according to an embodiment of the present invention. FIG. 3 is an exemplary diagram illustrating a user interface that allows a user to provide multimodal input including text and images, according to one embodiment of the present invention. FIG. 4 is a conceptual diagram illustrating a detailed process for inferring a user's intention to create a document according to an embodiment of the present invention. FIG. 5 is a conceptual diagram illustrating the process of calculating similarity between an embedding of an inferred document creation intent and a plurality of preset template embeddings according to an embodiment of the present invention. FIG. 6 is a flowchart illustrating a branching process for selecting an existing template or creating a new template based on a calculated similarity value according to an embodiment of the present invention. FIG. 7 is a conceptual diagram illustrating the operation of utilizing the structure of a template determined to generate a final document and a placeholder according to an embodiment of the present invention. FIG. 8 is a flowchart illustrating the process of converting visual elements into structured data and inserting them into a template according to one embodiment of the present invention. FIG. 9 is a conceptual diagram illustrating the process of utilizing a document image containing text content as an optical compression medium according to an embodiment of the present invention. FIG. 10 is a block diagram illustrating the detailed structure of a visual encoder that converts a document image into a compressed vision token sequence according to one embodiment of the present invention. FIG. 11 is according to an embodiment of the present invention This is a flowchart explaining the detailed operation of the compression module included in the visual encoder. FIG. 12 is a conceptual diagram illustrating the operation of an LLM-based decoder that performs guided decoding using a vision token and template constraints according to an embodiment of the present invention. FIG. 13 is a block diagram illustrating the hardware configuration of a server performing a document generation method according to an embodiment of the present invention. Specific details for implementing the invention

[0029] The terms used in this invention are used merely to describe specific embodiments and are not intended to limit the scope of other embodiments. A singular expression may include a plural expression unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by those skilled in the art as described in this invention. Terms used in this invention that are defined in a general dictionary may be interpreted as having the same or similar meaning as they have in the context of the relevant technology, and are not interpreted in an ideal or overly formal sense unless explicitly defined in this invention. In some cases, even terms defined in this invention may not be interpreted to exclude embodiments of this invention.

[0030] Below, various embodiments are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present invention. However, since the technical concept of the present invention can be modified and implemented in various forms, it is not limited to the embodiments described in this specification. In describing the embodiments disclosed in this specification, if it is determined that specifically describing related known technologies may obscure the essence of the technical concept of the present invention, such specific description of known technologies is omitted. Identical or similar components are assigned the same reference numerals, and redundant descriptions thereof are omitted.

[0031] In this embodiment, the term "~part" refers to a component that performs a specific function performed by software or hardware such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). However, "~part" is not limited to being performed by software or hardware. "~part" may exist in the form of data stored on an addressable storage medium, or it may be implemented by instructions and configured so that one or more processors execute a specific function.

[0032] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave in order to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media. Software may be read into main memory from other computer-readable media, such as data storage devices, or from other devices via communication interfaces. Software instructions stored in main memory may cause the processor to perform processes or steps, which will be described in detail below. Alternatively, processes consistent with the principles of the present invention may be executed using a fixed wiring circuit instead of or in combination with software instructions. Accordingly, embodiments consistent with the principles of the present invention are not limited to any specific combination of hardware circuits and software.

[0033] The terms used in this application are used merely to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, terms such as “comprising” or “having” are intended to indicate the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof. Terms such as “first,” “second,” etc., may be used to describe various components, but the components should not be limited by such terms. Such terms are used solely for the purpose of distinguishing one component from another.

[0035] FIG. 1 is a diagram schematically illustrating an adaptive template-based document generation system according to one embodiment of the present invention.

[0036] Referring to FIG. 1, an adaptive template-based document generation system (100) may be configured to include a user terminal (120), a network (130), and a server (110). The user terminal (120) and the server (110) may be connected to communicate through the network (130).

[0037] A user terminal (120) is an electronic device that allows a user to input information for creating a document and receive and verify the created document. The user terminal (120) provides a user interface for interaction with the user, through which it can obtain multimodal input consisting of text, images, or a combination thereof. For example, the user terminal (120) may be a desktop computer, a laptop computer, a tablet PC, or a smartphone. The user may use the user terminal (120) to input draft text of a document, take or upload an image of a document to be referenced, or provide a chart image to be included in the document. Additionally, the user terminal (120) may perform the role of collecting context information through user account information, device settings, or explicit input from the user and transmitting it to the server (110). Finally, the created document may be received from the server (110) and displayed to the user through the display device of the user terminal (120).

[0038] The network (130) is a communication infrastructure that mediates data exchange between a user terminal (120) and a server (110). The network (130) may include a public network such as the Internet, or a private network such as a local area network (LAN) or a wide area network (WAN), and may support both wired and wireless communication protocols. Multimodal input and context information from the user terminal (120) are securely transmitted to the server (110) through the network (130). Conversely, the final generated document, which is the result processed by the server (110), is transmitted to the user terminal (120) through the network (130).

[0039] The server (110) is a computing system that performs operations of the present invention. The server (110) includes one or more processors and memory, and performs an adaptive template-based document generation method by having the processor execute instructions stored in memory. The operation of the server (110) may include the following step-by-step processing.

[0040] The server (110) receives user context information and multimodal input from the user terminal (120) through an input acquisition module. The context information may include data such as information that uniquely identifies the user, laws or internal regulations applicable to the document, the language preferred by the user, or the user's past document creation history. This context information is used as base data for creating a personalized document. The multimodal input includes text and image data and provides clues about the content and format of the document the user intends to create.

[0041] The server (110) infers the user's specific document creation intent based on multimodal input obtained through an intent inference module. The intent inference module can convert text and image inputs into an integrated latent vector representation using a multimodal encoder. Subsequently, multidimensional attributes of intent, such as document type (e.g., report, contract, email, etc.), required format, target audience, formal or informal tone, and security level based on the sensitivity of the content, can be classified or extracted from the converted latent vector representation using a pre-trained large-scale language model or classification model.

[0042] The server (110) determines the optimal document template by synthesizing the document creation intent and context information inferred through the template determination module. The template determination module can access a database that stores various types of predefined templates. Each template has a unique structure, sections, and placeholders and can be represented in the form of vector embeddings. The template determination module calculates the semantic similarity between a second embedding representing the inferred document creation intent and a first embedding corresponding to each template in the database. The template with the highest calculated similarity value can be selected as the optimal template. Meanwhile, if the similarity with all templates is below a preset threshold, the template determination module determines that none of the existing templates sufficiently meet the user's intent and can dynamically generate a new template structure based on the context information and document creation intent. This serves to enhance the adaptability of the system.

[0043] The server (110) generates a final document based on a template determined through a document generation module. The document generation module may include a generation model, for example, a large-scale language model (LLM). This generation model receives the structure of the determined template, i.e., a hierarchical structure such as a title, table of contents, footnotes, etc., and placeholders that specify the types of content to be filled in, as input constraints. In addition, it receives the user's context information and inferred document creation intent together, and fills in the placeholders by generating content that matches the user's intent and context while satisfying the formal constraints of the template.

[0044] In particular, if the input image contains visual information such as a chart or diagram, the image analysis function of the server (110) recognizes the visual elements of the image, such as bars, lines, labels, etc., and converts them into structured data such as a table or list. The converted structured data can be accurately inserted into the relevant placeholders within the template.

[0045] Additionally, if the input image is a document image containing text content, the server (110) can utilize it as an optical compression medium to increase processing efficiency. In this case, the visual encoder within the server (110) processes the document image and, instead of converting the text into optical character recognition (OCR), generates a sequence of vision tokens that compress the visual features of the image. The number of generated vision tokens can be significantly smaller than the number of text tokens when the original text is tokenized, thereby reducing computational costs during the processing. Subsequently, an LLM-based decoder receives the compressed vision token sequence and a determined template as input, and performs guided decoding to restore the text content embedded in the vision tokens while simultaneously reconstructing the content according to the structure and constraints of the template to generate a final document.

[0046] FIG. 2 is a flowchart illustrating the overall operation of an adaptive template-based document generation method according to an embodiment of the present invention. Each step illustrated in FIG. 2 can be performed by the server (110) of FIG. 1.

[0047] Referring to FIG. 2, the adaptive template-based document generation method may include a multimodal input acquisition step (S210), a document creation intent inference step (S220), a template determination step (S230), and a document generation step (S240).

[0048] First, in step S210, the server (110) may obtain data necessary for document creation from the user terminal (120). The obtained data may include the user's context information and multimodal input provided by the user. The context information may function as metadata to adjust the document creation process to suit a specific user or environment. Specifically, the context information may include at least one of user identification information such as the user's account information, applicable domains or regulations such as laws of a specific country or internal corporate regulations that the document must comply with, a preferred language in which the document will be written, and past document creation history such as the user's past document creation patterns or frequently used template history. Multimodal input is data that constitutes the actual content of the document to be created or provides clues thereto. Multimodal input may include at least one of text directly entered by the user, or images such as scanned images of reference documents or data charts.

[0049] Subsequently, in step S220, the server (110) can infer the user's intention to write a document based on the multimodal input obtained in step S210. The purpose of this step is to identify the attributes and requirements of the document that the user has not explicitly specified. To this end, the server (110) can first convert text input and image input into latent vector representations in a common latent space using a multimodal encoder. These latent vectors imply semantic features of the input data. Next, the server (110) can analyze the generated latent vector representations using a pre-trained large-scale language model or a specialized classification model. Through this analysis, the model classifies or extracts various attributes that constitute the intention to write a document. These attributes may include at least one of the following: the type of document, e.g., report, contract, proposal, email, etc.; the required specific format or table of contents structure; the level of expertise or background of the target audience who will read the document; the tone of the writing style, whether formal or informal; and the security level based on the sensitivity of the content.

[0050] Next, in step S230, the server (110) can determine an optimal document template by comprehensively considering the document creation intent inferred in step S220 and the context information obtained in step S210. The server (110) can select a template from an embedded template database or create a new template if necessary. When selecting a template, the server (110) generates a second embedding vector representing the inferred document creation intent and calculates the similarity with a first embedding vector corresponding to each template pre-set in the template database. The similarity calculation can be performed using a vector similarity measurement technique such as cosine similarity. If the maximum similarity among the calculated similarity values ​​is greater than or equal to a preset threshold, the server (110) can select an existing template corresponding to that maximum similarity as the final template. On the other hand, if the maximum similarity is less than the threshold, this may mean that there is no existing template capable of satisfying the user's complex or special intent. In this case, the server (110) can dynamically generate a new template that meets the user's requirements by combining necessary sections, placeholders, formats, etc. based on the attributes of the inferred intent and context information.

[0051] Finally, in step S240, the server (110) can generate a final document based on the template determined in step S230. In this step, a generative model within the server (110), such as a Large Language Model (LLM), operates. The generative model receives structural constraints, such as the hierarchical structure defined in the determined template, for example, the arrangement of titles, subtitles, body paragraphs, tables, figures, etc., and placeholders that specify the location and type of content to be filled. Along with this, it receives the user's context information and inferred document creation intent to determine the style and content of the content to be generated. The generative model completes the document by generating text that corresponds to the semantic role of each placeholder while adhering to the structural constraints of the template. Through this process, the generated document can have a consistent structure while faithfully reflecting the user's individual intent and context.

[0052] FIG. 3 is an exemplary diagram illustrating a user interface that allows a user to provide multimodal input including text and images, according to one embodiment of the present invention.

[0053] Referring to FIG. 3, a user interface that can be displayed on a user terminal (120) can function as a channel for the user to provide information necessary for document creation to the system. This user interface may include a text input area (310), an image upload area (320), and a creation request button (330).

[0054] The text input area (310) provides a space where the user can directly input the content, purpose, or specific requirements of the document they wish to create in text form. The user may input the subject of the document, key keywords, an outline, part of a draft, or natural language commands such as "writing a business proposal for AI-based document automation" into the text input area (310). The text entered here can be used as one of the primary inputs to infer the user's intention to create the document in step S220.

[0055] The image upload area (320) is an interface element designed to allow a user to upload image files to be included in or referenced in a document. Through the image upload area (320), the user can select one or more images and transmit them to the system. The uploaded images may have various purposes. For example, the images may be charts visualizing data analysis results or diagrams explaining a process. In this case, the system can analyze the visual elements of the images as described later in FIG. 8, convert them into structured data, and insert them into appropriate locations in the document. Additionally, the images may be scanned copies or photographs of documents containing text content. Such images may be used as optical compression media for efficiently processing long texts. In this case, the visual encoder can increase processing efficiency by converting the images into fewer vision tokens than text tokens.

[0056] The creation request button (330) is a control element that the user activates to start the document creation process after the user has finished providing all necessary inputs through the text input area (310) and the image upload area (320). When the user selects the creation request button (330), the user terminal (120) transmits the text data entered in the text input area (310) and the image data uploaded through the image upload area (320) to the server (110).

[0057] FIG. 4 is a conceptual diagram illustrating a detailed process for inferring a user's intention to create a document according to an embodiment of the present invention.

[0058] The process illustrated in FIG. 4 may correspond to the document creation intent inference step (S220) of FIG. 2 and may be performed by a server (110). The server (110) processes the acquired multimodal input (410) to infer the user's document creation intent. The multimodal input (410) may include data of different forms, such as text and images provided by the user. For example, the user may input an overview of a report as text and provide a chart image containing data to be referenced.

[0059] A multimodal encoder (420) can receive these heterogeneous multimodal inputs (410) and perform encoding to convert them into a unified high-dimensional vector space, namely a latent vector representation (430) in a latent space. Text inputs can be processed through a text encoder, for example, a model such as BERT (Bidirectional Encoder Representations from Transformers), and image inputs can be processed through an image encoder, for example, a model based on ViT (Vision Transformer) or CNN (Convolutional Neural Network). The multimodal encoder (420) can generate a single latent vector representation (430) by fusing or concatenating feature vectors output from each encoder. The latent vector representation (430) is a numerical representation that encapsulates both the semantic content of the text and the visual features of the image.

[0060] Subsequently, the generated latent vector representation (430) can be provided as input to a pre-trained language model (440). The pre-trained language model (440) is pre-trained with large-scale text and image data and is capable of understanding and interpreting complex patterns inherent in the input latent vector representation (430). This model (440) performs the role of classifying various attributes of a document or predicting specific values ​​by analyzing the latent vector representation (430). For example, the pre-trained language model (440) can be implemented in a multi-task learning manner that receives the latent vector representation (430) as input and simultaneously infers each intentional attribute through multiple classification heads.

[0061] The output of the pre-trained language model (440) is an inferred document creation intent (450), which may consist of a combination of several specific attributes. For example, the inferred document creation intent (450) may include at least one of a document type, a required format, a target audience, a tone, and a security level. The document type may be classified as a report, a contract, a proposal, an email, etc. The tone may be determined as formal, informal, academic, etc. The target audience may be specified as experts, the general public, internal executives, etc. The security level may be set as general, confidential, secret, etc. These inferred attributes may be used as guidelines in the subsequent steps of template determination and document creation.

[0062] FIG. 5 is a conceptual diagram illustrating the process of calculating similarity between an embedding of an inferred document creation intent and a plurality of preset template embeddings according to an embodiment of the present invention.

[0063] The server (110) can perform the step of expressing the document creation intent inferred through FIG. 4 in a vector space and quantitatively measuring the semantic similarity with pre-set templates to select the optimal template.

[0064] First, the server (110) can generate a second embedding (520) by inputting the document creation intent inferred in FIG. 4 into an embedding model. The second embedding (520) is a high-dimensional vector that encapsulates the comprehensive semantic features of the document creation intent, including multiple attributes such as document type, request format, target audience, tone, and security level.

[0065] In parallel with this, the server (110) may prepare a first embedding (510) that is pre-calculated for each of a plurality of pre-configured templates stored in the internal template database (500). Each first embedding (510) is a vector representation that reflects information such as the structure, purpose, and main keywords of the corresponding template. These first embeddings (510) are generated and stored when the template is registered in the system, thereby enabling rapid retrieval during the template determination phase.

[0066] Next, the server (110) may perform a similarity calculation to calculate the similarity between each generated second embedding (520) and all first embeddings (510) in the template database. As a similarity measurement technique, a cosine similarity algorithm, which is effective for measuring directional similarity between vectors, may be used. Cosine similarity calculates the cosine value of the angle formed by two vectors, and the closer the value is to 1, the more semantically similar the two vectors are.

[0067] As a result of the similarity calculation step (530), a set of similarity values ​​between the second embedding (520) and each first embedding (510) is derived. For example, similarity values ​​for Template A, Template B, and Template C may be calculated as 0.95, 0.65, and 0.32, respectively. These similarity values ​​can be used as a quantitative basis for determining whether to select the most suitable template in a subsequent step, or to create a new template if there is no suitable template.

[0068] FIG. 6 is a flowchart illustrating a branching process for selecting an existing template or creating a new template based on a calculated similarity value according to an embodiment of the present invention.

[0069] Referring to FIG. 6, the server (110) can calculate similarity in step S610. In step S610, the server (110) calculates similarity between a second embedding representing a document creation intention and a first embedding corresponding to each of a plurality of pre-set templates stored in advance. The second embedding may be a vector representation generated as a result of document creation intention inference. The first embedding may be a vector representing the structural and semantic features of each template. Similarity calculation may utilize a cosine similarity technique, such as one, to measure the correlation between the two vectors. Through this process, the server (110) obtains a quantitative similarity score between the document creation intention and each pre-set template.

[0070] Subsequently, the server (110) can compare the maximum similarity with the threshold value in step S620. The server (110) identifies the maximum similarity having the maximum value among the multiple similarity scores calculated in step S610. Next, the server (110) compares the identified maximum similarity value with a preset threshold value to determine whether the preset template is suitable.

[0071] If, as a result of the comparison in step S620, the maximum similarity is below the threshold value, the server (110) may determine that there is no template among the pre-set templates that satisfies the user's intent. In this case, the server (110) proceeds to step S630 to create a new template. In step S630, the server (110) may dynamically create a template based on the inferred document creation intent and acquired context information. For example, the server (110) may create a customized template by designing the overall structure of the template by reflecting intent attributes such as document type, target audience, and tone, and by adding specific sections or placeholders by considering context information such as the scope of application or internal regulations.

[0072] Conversely, if the maximum similarity is greater than or equal to the threshold value as a result of the comparison in step S620, the server (110) may determine that a template matching the user's intention exists among the pre-set templates. In this case, the server (110) proceeds to step S640 and selects an existing template. In step S640, the server (110) may select the template corresponding to the maximum similarity as the final template. Alternatively, the server (110) may identify multiple templates with similarity scores greater than or equal to the threshold value as candidates, select one of them, or provide options to the user.

[0073] FIG. 7 is a conceptual diagram illustrating the operation of utilizing the structure of a template determined to generate a final document and a placeholder according to an embodiment of the present invention.

[0074] Referring to FIG. 7, the document generation step (S240) can be performed through a process including a determined template (710), input information (720), a generation model (730), and a final generated document (740).

[0075] First, the server (110) can prepare a template (710) determined in the previous step. The template (710) serves to define the overall framework of the document. The template (710) may include the logical and hierarchical structure of the document, such as the document title, table of contents, and subtitles for each section. Additionally, the template (710) may include one or more placeholders that specify the locations within the document where specific content should be filled. Each placeholder may be defined as a variable name, a data type, or an identifier indicating the nature of the content. For example, a report template may include placeholders such as {{Report_Title}}, {{Author_Name}}, and {{Key_Summary}}.

[0076] Next, the server (110) may prepare context information and inferred document creation intent as input information (720), which serve as the basis for document creation. Context information may include user identification information, the time of document creation, applicable laws, etc. Document creation intent is information that includes the type of document, target audience, tone, and key content inferred from the user's multimodal input. For example, if the user inputs text such as 'Q3 sales performance, 20% increase in sales, new product effect,' this content may be included in the input information (720) as a key element of the document creation intent.

[0077] Subsequently, the generative model (730) within the server (110) can receive the determined template (710) and input information (720) and perform the task of generating a final document. The generative model (730) may be a Large Language Model (LLM). The generative model (730) uses the structure defined in the template (710) as the overall framework of the document and generates and fills in content corresponding to each placeholder based on the context and intent included in the input information (720). For example, user identification information of the context information can be inserted into the {{Author_Name}} placeholder, and a summary sentence appropriate to the context can be generated and inserted into the {{Key_Summary}} placeholder based on the key content of the document creation intent.

[0078] Consequently, the generation model (730) can fill all placeholders with appropriate content and complete the structure of the template to output a final generated document (740). The document (740) thus generated can be a customized result that faithfully reflects the user's specific context and intention while maintaining a consistent format through the template.

[0079] FIG. 8 is a flowchart illustrating the process of converting visual elements into structured data and inserting them into a template according to one embodiment of the present invention.

[0080] First, the server (110) may obtain an image containing a chart or diagram as input in step S810. The input image may be a data visualization material such as a bar graph, a line graph, or a pie chart, or an information structure diagram such as a flowchart or an organizational chart.

[0081] Next, the server (110) can analyze the visual elements of the image input in step S820. In this step, the server (110) may utilize computer vision technology or a specialized image analysis model. The analysis of visual elements includes the process of identifying individual components within the image and extracting their attributes. For example, in the case of a bar graph image, the server (110) can identify and recognize individual elements such as labels for the x-axis and y-axis, the height or length of each bar, a legend, a chart title, etc. In the case of a flowchart, it can analyze each shape (process, decision, etc.), text within the shape, and the direction of the arrows connecting the shapes.

[0082] Subsequently, the server (110) can identify the relationships between the visual elements analyzed in step S830 and convert them into structured data. Structured data refers to a standardized format that can be easily processed by a machine and can be JSON (JavaScript Object Notation), XML (eXtensible Markup Language), or tabular data. For example, the elements of a bar graph analyzed in step S820 can be converted into a set of key-value pairs, i.e., JSON objects, where each x-axis label is the key and the numerical value of the corresponding bar is the value. In the case of a flowchart, it can be converted into a graph data structure where each shape is represented as a node and the arrows as edges.

[0083] The server (110) can insert the structured data generated in step S840 into a corresponding placeholder within the template. The template may have a dedicated placeholder predefined for displaying chart data in a table or for inserting text that explains the contents of a diagram. For example, if there is a placeholder named {{Quarterly_Sales_Table}} in the template, the server (110) can generate a Markdown table or HTML table code based on the sales data JSON generated in step S830 and insert it into the placeholder. Through this, visual information contained in the image can be naturally integrated into the body of the document.

[0084] FIG. 9 is a conceptual diagram illustrating the process of utilizing a document image containing text content as an optical compression medium according to an embodiment of the present invention.

[0085] When an image of a document containing text content is provided as a multimodal input, the server (110) can use the image as an optical compression medium for the text. The user can input the original text document (910) into the system in an image format rather than digital text. This may be an image file that is scanned or screen-captured, such as a long report, paper, or contract.

[0086] The encoder (920) processes the input document image and converts it into a sequence of vision tokens, which are compressed latent representations. In this process, the pixel information of the image is converted into a vector sequence that implies the semantic and structural features of the text. The vision token sequence generated here may consist of fewer tokens than the number of text tokens generated when the text of the original text document (910) is directly tokenized into digital text. This alleviates the problem of context length limitations in large-scale language models and provides the effect of reducing computational costs when processing long documents.

[0087] The decoder (930) receives a compressed vision token sequence from the encoder (920), restores the embedded text content, and reconstructs it into a new document format that matches the user's intent. The decoder (930) can be implemented based on a large-scale language model (LLM) and utilizes the ability to generate text from visual information.

[0088] FIG. 10 is a block diagram illustrating the detailed structure of a visual encoder that converts a document image into a compressed vision token sequence according to an embodiment of the present invention. The visual encoder illustrated in FIG. 10 may correspond to a specific implementation example of the encoder (920) of FIG. 9.

[0089] The time encoder may have a hierarchical structure including a first encoder block (1010), a compression module (1020), and a second encoder block (1030).

[0090] The first encoder block (1010) operates based on a window attention mechanism to extract local features of the input image. Since document images are often high resolution, calculating attention for all pixels is very computationally expensive. Window attention solves this problem by dividing the image into several non-overlapping windows and calculating attention only within each window. In document images, window attention can effectively capture microscopic visual information such as characters, words, and symbols within a specific area.

[0091] The output of the first encoder block (1010) is transmitted to the compression module (1020). The compression module (1020) operates based on convolution operations and performs the role of reducing the number of tokens output from the first encoder block (1010). By applying a convolution filter with a stride of 2 or more, the spatial dimension of the feature map can be reduced. This process has the effect of lowering the computational complexity of subsequent blocks and downsampling the feature map to summarize information.

[0092] The compressed token sequence that has passed through the compression module (1020) is input into the second encoder block (1030). The second encoder block (1030) operates based on a global attention mechanism. Since the number of tokens has been significantly reduced by the compression module (1020), it is now computationally possible to apply global attention to the entire token sequence. By calculating attention for the entire reduced number of tokens, global attention plays a role in integrating and understanding global contexts such as the paragraph structure, layout, and relationships between sections of the document.

[0093] FIG. 11 is a flowchart for explaining the detailed operation of a compression module included in a visual encoder according to one embodiment of the present invention.

[0094] The server (110) can obtain the output token of the first encoder block (1030) as input at step S1120. The first encoder block (1030) operates based on window attention and outputs a sequence of tokens that extract features for a local region of the image. This token sequence contains microscopic and detailed visual information of the image.

[0095] The server (110) can apply a convolution operation to the token sequence input in step S1130. The convolution operation integrates spatial information of adjacent tokens and reconstructs a feature map. Specifically, the entire token sequence can be scanned using a 2D convolution filter with a stride value greater than 1. For example, a convolution operation with a stride value of 2 has the effect of reducing the horizontal and vertical dimensions of the feature map by half.

[0096] The server (110) can reduce the number of tokens as a result of the convolution operation in step S1110. This corresponds to a downsampling process and reduces the overall length of the token sequence, thereby improving the computational efficiency of subsequent processing steps. Additionally, this process contributes to generating a more abstract representation by summing multiple local features into a single higher-level feature.

[0097] The server (110) can output a compressed token sequence with a reduced number of tokens at step S1140. The output compressed token sequence consists of fewer tokens than the input sequence, but each token has a wider receptive field and contains more contextual information. This compressed token sequence can then be passed as input to a second encoder block that incorporates the global context of the image.

[0098] FIG. 12 is a conceptual diagram illustrating the operation of a large-scale language model (LLM)-based decoder that performs guided decoding according to one embodiment of the present invention.

[0099] The LLM-based decoder (1230) can utilize two types of primary inputs to generate the final document. The first input is a vision token sequence (1210) generated from a visual encoder (1010). The vision token sequence (1210) is a latent vector representation of the text content of the original document image that is optically compressed and contains the core content and semantic information of the document.

[0100] The second input is a template (1220) determined in the template determination step (S230). The template (1220) defines the hierarchical structure of the document to be generated, such as the order and arrangement of titles, subtitles, paragraphs, lists, etc. Additionally, the template (1220) may include multiple placeholders indicating locations where specific types of information should be filled. This structure and placeholder information acts as a constraint on the text generation process of the LLM-based decoder (1230).

[0101] An LLM-based decoder (1230) generates a document by performing a guided decoding process (1240). The guided decoding process (1240) is a mechanism that controls the direction of generation so that the LLM-based decoder (1230) does not freely generate text but strictly adheres to the constraints presented by the template (1220).

[0102] Specifically, the LLM-based decoder (1230) generates tokens sequentially in an autoregressive manner. At each token generation point, the decoder calculates a probability distribution for all possible candidate tokens that may follow. In the guided decoding process (1240), this probability distribution is dynamically modified according to the constraints of the template (1220). For example, if the current generation position corresponds to a 'date' placeholder in the template (1220), the generation probability of tokens that do not conform to the date format can be adjusted to 0 or a very low value. Similarly, if the title of a specific section needs to be generated, the generation probability of tokens related to the title text defined in the template (1220) can be increased. In this process, the LLM-based decoder (1230) determines specific values ​​to be inserted into placeholders by referring to the content encoded in the vision token sequence (1210).

[0103] Through this guided decoding process (1240), the LLM-based decoder (1230) outputs a structured final document (1250). The generated structured final document (1250) can have high quality and consistency, accurately following the format of the template (1220) while faithfully reflecting the content of the original document restored from the vision token sequence (1210).

[0104] FIG. 13 is a block diagram illustrating the hardware configuration of a server performing a document generation method according to an embodiment of the present invention.

[0105] Referring to FIG. 13, the server (1300) may be configured to include a communication unit (1310), a memory (1320), and a processor (1330). The illustrated components may be electrically connected to each other via a data bus to exchange signals and data.

[0106] The communication unit (1310) provides a communication interface for the server (1300) to transmit and receive data with an external device, such as a user terminal (120), via a network. The communication unit (1310) may include one or more hardware modules that support wired or wireless communication protocols. The communication unit (1310) may perform the role of obtaining multimodal input from the user terminal (120), which includes at least one of the user's context information, text, and images. Additionally, it may transmit a final generated document to the user terminal (120) under the control of the processor (1330).

[0107] Memory (1320) is a storage medium that stores program code and data necessary for the operation of the processor (1330). Memory (1320) may include volatile memory such as RAM (Random Access Memory) and non-volatile memory such as ROM (Read Only Memory), flash memory, or solid-state drive (SSD). Memory (1320) may store a computer program, which is a set of instructions for performing each step of an adaptive template-based document generation method according to embodiments of the present invention. Specifically, memory (1320) may store executable code for each of the functions of acquiring multimodal input, inferring document creation intent, determining a template, and generating a document. Additionally, memory (1320) may store a template database that stores a plurality of pre-set template information, user context information, and artificial intelligence models such as a multimodal encoder, a pre-trained language model, and a generative model.

[0108] The processor (1330) is a central control unit of the server (1300) and controls the operation of all hardware components of the server (1300) by decoding and executing instructions of the operating system and application programs stored in memory (1320). In particular, the processor (1330) executes a computer program stored in memory (1320) to perform an adaptive template-based document generation method according to an embodiment of the present invention, and the specific operation is as follows.

[0109] The processor (1330) first executes the step of controlling the communication unit (1310) to obtain user context information and multimodal input from the user terminal (120) in order to process a document creation request. The processor (1330) stores the received data in a designated area of ​​the memory (1320) so that it can be accessed in a subsequent processing step.

[0110] Next, the processor (1330) executes a step of inferring the user's intention to write a document. In this process, the processor (1330) loads a multimodal encoder model from memory (1320). The processor (1330) provides the acquired text and image inputs as inputs to the multimodal encoder and performs neural network operations of the encoder to convert the input data into a high-dimensional latent vector representation. If the image input is a document image containing text content, the processor (1330) may execute a specific visual encoder stored in memory (1320). The processor (1330) performs an operation to extract local features of the image through a window attention-based first encoder block within the visual encoder, and then executes a convolution-based compression module to perform an operation to reduce the number of output tokens. Subsequently, the processor (1330) performs an operation to integrate the global context of the image through a global attention-based second encoder block to finally generate a sequence of compressed vision tokens. Afterward, the processor (1330) loads a pre-trained language model or classification model from memory (1320) and inputs the generated latent vector representation or vision token sequence to perform an inference operation to classify intent attributes such as document type, tone, and security level.

[0111] Next, the processor (1330) executes a step of determining an optimal template. The processor (1330) performs an operation to generate a second embedding vector from the inferred document creation intent. Additionally, the processor (1330) accesses the template database in memory (1320) and sequentially loads first embedding vectors corresponding to each pre-set template. The processor (1330) performs a vector operation to calculate the cosine similarity between the second embedding vector and each first embedding vector. The processor (1330) identifies the maximum similarity among all calculated similarity values ​​and performs an operation to compare this value with a threshold value stored in memory (1320). As a result of the comparison, if the maximum similarity is greater than or equal to the threshold value, the processor (1330) executes a command to select an existing template corresponding to that maximum similarity. On the other hand, when the maximum similarity is less than the threshold, the processor (1330) executes logic to define and create the structure, sections, and placeholders of a new template by combining context information and attributes of the inferred document creation intent.

[0112] The processor (1330) executes the step of generating a final document. The processor (1330) loads a generative model, such as a large-scale language model (LLM), from memory (1320). The processor (1330) configures and transmits the structural information, placeholders, context information, and document creation intent of the determined template as inputs to the generative model. If the input image contains a chart or diagram, the processor (1330) executes an image processing algorithm that analyzes the visual elements and converts them into structured data, and performs an operation to map the converted data to the corresponding placeholders of the template. When generating the document, the processor (1330) controls the text generation process of the LLM-based decoder. In particular, the processor (1330) executes a guided decoding algorithm that uses the hierarchical structure of the template and the semantic properties of the placeholders as constraints. Through this controlled execution, the processor (1330) ensures that the generated text strictly adheres to the format of the template. When all content generation is complete, the processor (1330) combines the generated text and template structure to create a final document file and controls the transmission of this document to a user terminal (120) through the communication unit (1310).

[0113] Meanwhile, a method according to one embodiment of the present invention can be implemented as computer-readable code on a computer-readable recording medium. A computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. For example, a computer-readable recording medium may include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0114] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0115] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below. Explanation of the symbols

[0116] 100: Adaptive Template-Based Document Generation System 110: Server 120: User terminal 130: Network 310: Text input area 320: Image upload area 330: Request creation button 410: Multimodal Input 420: Multimodal Encoder 430: Latent Vector Representation 440: Model 450: Document creation intent 500: Internal Template Database 510: 1st Embedding 520: Second Embedding 530: Similarity Calculation Step 710: Template 720: Input Information 730: Generative Model 740: Document 910: Original text document 920: Encoder 930: Decoder 1010: 1st encoder block 1020: Compression Module 1030: 2nd encoder block 1300: Server 1310: Communications Department 1320: Memory 1330: Processor

Claims

Claim 1 A method for generating an adaptive template-based document performed by a server comprises: a step of obtaining a multimodal input including at least one of a user's context information, text, and an image; a step of inferring a user's intention to create a document based on the multimodal input; a step of identifying a maximum similarity based on the similarity between a first embedding corresponding to each of a plurality of preset templates and a second embedding of the intention to create a document, and determining a template by selecting at least one of the plurality of preset templates when the maximum similarity is greater than or equal to a threshold value. The method is characterized by including the step of: the determined template includes one or more placeholders that specify a hierarchical section structure and a location and type of content to be filled, wherein each of the one or more placeholders is defined by an identifier representing at least one of a variable name, a data type, and the nature of the content; and an LLM-based decoder that receives the context information, the document creation intent, and the determined template as input, sequentially generates tokens in an autoregressive manner, wherein, upon token generation, the placeholder corresponding to the current generation location is identified, and the probability distribution for candidates of the next token is dynamically modified based on the identifier of the identified placeholder, thereby generating a document that complies with the hierarchical section structure of the determined template and the semantic attributes of the placeholders. Claim 2 A method according to claim 1, wherein the context information comprises at least one of user identification information, applicable domain or regulation, preferred language, and past document creation history. Claim 3 The method of claim 1, wherein the step of inferring the document creation intent comprises: a step of converting the multimodal input into a latent vector representation using a multimodal encoder that encodes the multimodal input; and a step of using a pre-trained language model that classifies at least one of a document type, a request format, a target audience, a tone, and a security level based on the latent vector representation. Claim 4 A method according to claim 1, wherein the similarity is calculated by cosine similarity, and the first embedding is calculated in advance by reflecting the hierarchical section structure and placeholder definition information of the template and stored in a template database. Claim 5 A method according to claim 4, wherein the step of determining the template includes the step of generating a template based on the context information and the document creation intention when the maximum similarity among the similarities is less than a threshold value. Claim 6 A method according to claim 1, wherein the step of generating the document comprises the step of generating the document by inputting the context information and the document creation intention into a generation model based on the structure and placeholders defined in the determined template. Claim 7 A method according to claim 6, wherein, if the image included in the multimodal input includes a chart or diagram, the step of generating the document further comprises the step of analyzing the visual elements of the chart or diagram to convert them into structured data and inserting the converted data into a corresponding placeholder of the determined template. Claim 8 A method according to claim 1, wherein the image included in the multimodal input is an image of a document containing text content, and is used as an optical compression medium for the text content. Claim 9 In claim 8, the multimodal input acquisition step comprises the step of processing the image through a vision encoder to generate a sequence of vision tokens, wherein the sequence of vision tokens is composed of fewer tokens than the number of text tokens when the text content is tokenized into digital text. Claim 10 A method according to claim 9, wherein the visual encoder comprises a window attention-based first encoder block for extracting local features of an image and a global attention-based second encoder block for integrating the global context of an image, and wherein a convolution-based compression module for reducing the number of output tokens of the first encoder block is serially connected between the first encoder block and the second encoder block. Claim 11 A method according to claim 9, wherein the step of generating the document comprises the step of using a large-scale language model (LLM) based decoder that receives the sequence of the vision tokens and the determined template as input, restores text content compressed in the vision tokens, and generates text according to the structure of the determined template. Claim 12 A method according to claim 11, wherein dynamically modifying the probability distribution for the candidates of the next token comprises adjusting the generation probability of a candidate token that does not conform to the date format to 0 or a lower value when the current generation position corresponds to a placeholder of a date data type, or increasing the generation probability of a candidate token associated with the title text defined in the determined template when the current generation position corresponds to a section title. Claim 13 A server comprises: a processor; and a memory for storing a computer program executed by the processor; wherein the computer program comprises: a step of acquiring a multimodal input including at least one of a user’s context information and text and an image; a step of inferring a user’s document creation intention based on the multimodal input; a step of identifying a maximum similarity based on the similarity between a first embedding corresponding to each of a plurality of preset templates and a second embedding of the document creation intention, and determining a template by selecting at least one of the plurality of preset templates when the maximum similarity is greater than or equal to a threshold value. A server configured to perform the step of generating a document that complies with the hierarchical section structure of the determined template and the semantic attributes of the placeholders by receiving the context information, the document creation intent, and the determined template as input, and the LLM-based decoder that sequentially generates tokens in an autoregressive manner, wherein at the time of token generation, the placeholder corresponding to the current generation position is identified, and the probability distribution for candidates of the next token is dynamically modified based on the identifier of the identified placeholder. Claim 14 A computer-readable recording medium that stores instructions for executing the method of any one of claims 1 through 12 when executed by a computer.

Citation Information

Patent Citations

  • Apparatus and method for controlling electronic document based on natural language processing

    KR102189568B1

  • User-intent based generation of layout for content data

    KR1020170066374A

  • Template automatic secletion system

    KR1020240135100A

  • Method for generating OCR data for support of multiple languages and scripts and computer-readable recording medium containing instructions for the method to be performed by a processor

    KR102872314B1