Method and system for generating answer

Abstract

The present invention relates to a method and system for generating an answer. More specifically, the present invention relates to a method and system for generating an answer to a query in a multi-modal context including various modalities (images, charts, text, etc.).

Description

Method and system for generating answers

[0001] The present invention relates to a method and system for generating answers. More specifically, the present invention relates to a method and system for generating answers to queries in a multi-modal context that includes various modalities (images, charts, text, etc.).

[0002] The dictionary definition of artificial intelligence is "the technology that embodies human learning, reasoning, perception, and natural language comprehension abilities through computer programs." This type of artificial intelligence has made remarkable progress thanks to deep learning.

[0003] In particular, thanks to the development of artificial intelligence, various language models have been developed, and these language models have reached the level of not only recognizing text and understanding its meaning, but also extracting and classifying information from data containing large amounts of text, such as documents, and even directly generating text.

[0004] These language models are actively used in various fields, such as search engines, document writing (e.g., resume writing, report writing, posting writing, etc.), free conversation on various topics, data parsing from a given text (e.g., data summarization, classification, etc.), providing expert knowledge, programming, and converting a given sentence into sentences in an appropriate style, etc., and they exist in various fields that can perform text-based tasks.

[0005] In this regard, large language models (LLMs) have recently emerged, capable of understanding and generating human language through pre-training on massive amounts of text data. Unlike traditional manually built chatbots, which only offer limited responses, LLMs demonstrate the technological prowess to communicate naturally, almost human-like, and provide fast, accurate information, revolutionizing the artificial intelligence market.

[0006] Meanwhile, research is actively underway to expand large-scale language models to recognize various modalities. In particular, information is conveyed through various modalities, not just text. Many documents include images to aid comprehension or provide visual evidence, and charts are utilized to effectively illustrate data trends or when numerical data cannot be presented in a table format. For example, the Pew Research Center uses images and charts in addition to text when publishing articles related to public opinion polls, demographic studies, and data-driven social science research.

[0007] To answer questions in a multi-modal context that includes various modalities such as images, charts, and text, a model that can understand various modalities and respond to questions effectively within that context is required.

[0008] The present invention provides a method and system for generating answers that can understand various modalities and effectively generate answers to questions in a multi-modal context.

[0009] More specifically, the present invention provides an answer generation method and system that can understand various modalities and generate high-quality answers to queries based on integrated information in a multi-modal context.

[0010] In addition, the present invention provides a method and system for generating answers that can improve the quality of answers for various questions.

[0011] Furthermore, the present invention provides an answer generation method and system capable of generating high-quality answers to various queries in a multi-modal context.

[0012] In order to solve the problem discussed above, a computerized answer generation method according to the present invention may include the steps of: receiving a document including a plurality of modalities having different data characteristics; extracting data from each of the plurality of modalities through different feature extraction methods based on the data characteristics of each of the plurality of modalities; receiving a user query for the document from a user terminal; specifying at least one specific modality associated with the user query among the plurality of modalities constituting the document; and generating an answer to the user query using specific data extracted from the specific modality.

[0013] In an embodiment, the method may further include a step of classifying each of the plurality of modalities according to data characteristics of each of the plurality of modalities in order to extract data from each of the plurality of modalities included in the document.

[0014] In the embodiment, in the classification step, the document including the plurality of modalities is processed as input to a classification module, and the classification module can classify the plurality of modalities according to the data characteristics of each of the plurality of modalities.

[0015] In the embodiment, in the extracting step, data is extracted from each of the classified multiple modalities through the different feature extraction methods based on the data characteristics of each of the classified multiple modalities from the classification module, and the classified multiple modalities may include at least one of the first modality, the second modality, and the third modality.

[0016] In an embodiment, the method further includes a step of processing user queries for data and documents extracted from each of the classified plurality of modalities as inputs of a pre-trained large-scale language model (LLM), wherein the different feature extraction methods may include at least one of a first feature extraction method specialized for the first modality, a second feature extraction method specialized for the second modality, and a third feature extraction method specialized for the third modality.

[0017] The data extracted from each of the classified plurality of modalities may include at least one of first specific data extracted from the first modality through the first feature extraction method, second specific data extracted from the second modality through the first feature extraction method, and third specific data extracted from the third modality through the third feature extraction method.

[0018] In an embodiment, the large-scale language model may specify at least one specific modality among a plurality of modalities constituting the document, which is associated with the user query and is to be used for generating the answer, and may generate an answer to the user query using specific data extracted from the specific modality among data extracted from each of the classified plurality of modalities.

[0019] In an embodiment, the user query may include different types of user queries, and the different types of user queries may include a first type of user query and a second type of user query.

[0020] In an embodiment, when a first type of user query is received among the different types of user queries, the specific modality may include one specific modality associated with the first type of user query among the plurality of modalities, and when a second type of user query is received among the different types of user queries, the specific modality may include a plurality of specific modalities associated with the second type of user query among the plurality of modalities.

[0021] In an embodiment, when a user query of the first type is received, one of the plurality of modalities constituting the document is associated with the user query of the first type and is to be used for generating the answer, and by using specific data extracted from the one of the specific modalities, an answer to the user query can be generated.

[0022] In an embodiment, when a user query of the second type is received, a specific modality among the plurality of modalities constituting the document is specified that is associated with the user query of the second type and is to be used for generating the answer, and an answer to the user query can be generated by performing inference between data extracted from each of the plurality of specific modalities.

[0023] In an embodiment, the large-scale language model may be trained with multiple different training data sets to understand the multiple modalities and generate answers to the user queries from the documents containing the multiple modalities.

[0024] In an embodiment, the plurality of different learning data sets may include a first learning data set and a second learning data set, and the first learning data set may include at least one of a first sub-learning data set and a second sub-learning data set.

[0025] In an embodiment, the first sub-learning data set includes learning instruction data of a plurality of categories including images, charts, and texts, and the second sub-learning data set can be constructed based on a chart image and a table, taking into consideration visual information included in the chart image and numerical values ​​of the table.

[0026] In an embodiment, the second sub-learning data set may include at least one of chart summarization data, open-ended QA data, and reasoning QA data.

[0027] In an embodiment, the large-scale language model can be trained to understand the plurality of modalities using the first learning data set among the plurality of different learning data sets.

[0028] In an embodiment, the document including the plurality of modalities may be further trained using the second learning data set to generate an answer to a first type of user query or an answer to a second type of user query.

[0029] In an embodiment, the second learning data set may be constructed such that a first type of query and a response to the first type of query for a multi-modal context form a pair, and a second type of query and a response to the second type of query for the multi-modal context form a pair.

[0030] An answer generation system according to the present invention, comprising a memory configured to store executable instructions, and one or more processors configured to perform operations by executing one or more instructions, receives a document including a plurality of modalities having different data characteristics, extracts data from each of the plurality of modalities through different feature extraction methods based on the data characteristics of each of the plurality of modalities, receives a user query for the document from a user terminal, specifies at least one specific modality associated with the user query among the plurality of modalities constituting the document, and generates an answer to the user query using specific data extracted from the specific modality.

[0031] A program according to the present invention is a program that is executed by one or more processes in an electronic device and can be stored in a computer-readable recording medium, and may include instructions for performing the steps of: receiving a document including a plurality of modalities having different data characteristics; extracting data from each of the plurality of modalities through different feature extraction methods based on the data characteristics of each of the plurality of modalities; receiving a user query for the document from a user terminal; specifying at least one specific modality associated with the user query among the plurality of modalities constituting the document; and generating an answer to the user query using specific data extracted from the specific modality.

[0032] A computerized answer generation method according to the present invention, comprising the steps of: specifying a document including a plurality of modalities; classifying each of the plurality of modalities included in the document based on a preset classification criterion; extracting data from each of the classified plurality of modalities using different feature extraction methods specialized for each of the classified plurality of modalities; receiving a user query for the document from a user terminal; and generating an answer to the user query using specific data extracted from at least one specific modality associated with the user query among the data extracted from each of the classified plurality of modalities.

[0033] In the embodiment, in the classification step, a document including the plurality of modalities is processed as input to a classification module to extract data from each of the plurality of modalities using the different feature extraction methods, and a plurality of modalities classified according to the preset classification criteria can be obtained from the classification module.

[0034] In an embodiment, the classification module may be configured to classify each of the plurality of modalities constituting the document based on the preset classification criteria.

[0035] In an embodiment, the different feature extraction methods for extracting data from each of the plurality of modalities may be determined based on data characteristics of each of the classified plurality of modalities obtained from the classification module.

[0036] In an embodiment, the extracting step may include a step of determining a different feature extraction method specialized for each of the classified plurality of modalities based on data characteristics of each of the classified plurality of modalities, and a step of extracting data from each of the classified plurality of modalities using the determined different feature extraction methods.

[0037] In an embodiment, at least some of the different feature extraction methods are related to a multi-encoder, and the classified plurality of modalities include at least one of a first modality, a second modality, and a third modality having different data characteristics, and in the determining step, based on a preset criterion, whether each of the classified plurality of modalities is input to the multi-encoder can be determined, and at least one modality to be input to the multi-encoder can be determined based on a result of the determination.

[0038] In an embodiment, the at least one modality to be input to the multi-encoder may include at least one of the first modality and the second modality.

[0039] In an embodiment, the multi-encoder may include at least one of a first encoder specialized for the first modality and a second encoder specialized for the second modality.

[0040] In an embodiment, in order to extract specific data from the first modality, the first modality may be input to the first encoder, and first specific data may be extracted from the first modality through the first encoder.

[0041] In an embodiment, in order to extract specific data from the second modality, the second modality may be input to the second encoder, and second specific data may be extracted from the second modality through the second encoder.

[0042] In an embodiment, the method further comprises a step of configuring input data including data extracted from each of the classified plurality of modalities, wherein the different feature extraction methods may include at least one of a first feature extraction method specialized for the first modality, a second feature extraction method specialized for the second modality, and a third feature extraction method specialized for the third modality.

[0043] In an embodiment, the method may further include a step of processing user queries for the input data and documents as inputs to a pre-trained large-scale language model (LLM).

[0044] In an embodiment, the input data may include at least one of first specific data extracted from the first modality through the first feature extraction method, second specific data extracted from the second modality through the first feature extraction method, and third specific data extracted from the third modality through the third feature extraction method.

[0045] In an embodiment, the large-scale language model can, based on the input data, specify at least one specific modality among the classified plurality of modalities that is associated with the user query and is to be used to generate an answer to the user query, and can generate an answer to the user query using the specific data extracted from the specific modality.

[0046] In an embodiment, when the specific modality includes a plurality of specific modalities, an answer to the user query can be generated by performing inference between specific data extracted from each of the plurality of specific modalities among the classified plurality of modalities.

[0047] In an embodiment, the large-scale language model may be trained to understand multiple modalities contained in the document and to generate an answer to the user query from the document including the multiple modalities.

[0048] An answer generation system according to the present invention, comprising a memory configured to store executable instructions, and one or more processors configured to perform operations by executing one or more instructions, specifies a document including a plurality of modalities, classifies each of the plurality of modalities included in the document based on a preset classification criterion, extracts data from each of the classified plurality of modalities using different feature extraction methods specialized for each of the classified plurality of modalities, receives a user query for the document from a user terminal, and generates an answer to the user query using specific data extracted from at least one specific modality associated with the user query among the data extracted from each of the classified plurality of modalities.

[0049] A program according to the present invention is a program that is executed by one or more processes in an electronic device and can be stored in a computer-readable recording medium, and may include commands that perform the steps of: specifying a document including a plurality of modalities; classifying each of the plurality of modalities included in the document based on a preset classification criterion; extracting data from each of the classified plurality of modalities using different feature extraction methods specialized for each of the classified plurality of modalities; receiving a user query for the document from a user terminal; and generating an answer to the user query using specific data extracted from at least one specific modality associated with the user query among the data extracted from each of the classified plurality of modalities.

[0050] As described above, the method and system for generating answers according to the present invention classifies multiple modalities having different data characteristics, and applies a specialized feature extraction method to each of the classified multiple modalities, thereby enabling precise extraction of data from each of the different modalities, such as images, charts, and text. In other words, the present invention extracts information by reflecting the data characteristics of each of the multiple modalities, thereby enabling precise extraction of the core data required to generate answers to queries without information loss. Through this, the present invention can more precisely grasp the semantic connectivity between various expression forms, thereby effectively utilizing only the core information required to generate answers to user queries while maintaining the semantic connectivity between various expression forms.

[0051] Furthermore, the answer generation method and system according to the present invention utilizes a multi-encoder structure including encoders specialized for each of a plurality of modalities, thereby providing optimized recognition and processing performance tailored to the data characteristics of each modality. Accordingly, the present invention can generate accurate answers to queries by recognizing visual and / or contextual information of each modality without loss. In other words, the present invention can improve the recognition accuracy of each modality while reflecting the characteristics of each modality without loss, and maintain the performance of the overall model.

[0052] Furthermore, according to the method and system for generating an answer according to the present invention, a modality related to a user query is specified among a plurality of modalities included in a document, and an answer is generated based on data extracted from the specified modality, thereby providing the user with a high-quality answer generated based on information related to the user query. In particular, the present invention can save system resources and improve the answer generation speed by selecting and efficiently processing a modality related to the user query, and can generate an appropriate answer through integrated inference even for a user query including various modalities such as images, charts, and text.

[0053] Furthermore, according to the answer generation method and system according to the present invention, a learning data set can be configured to enable integrated understanding and inference of various modalities such as images, charts, and text, and a large-scale language model (LLM) can be trained using this data set. Through this, the large-scale language model can comprehensively understand the information contained in complex multi-modal documents and generate accurate answers to user queries based on the results of the understanding. In other words, the large-scale language model according to the present invention can generate various types of answers such as not only identification-type answers but also summary-type, comparison-type, context-interpretation-type, and purpose-inference-type answers, and in particular, it can derive integrated inference results even for complex queries including images, charts, and text. Through this, users can receive customized answers that meet various query purposes.

[0054] Furthermore, the answer generation method and system according to the present invention can improve the chart interpretation accuracy and contextual understanding of a large-scale language model by training a large-scale language model using a training data set that reflects various visual elements (e.g., axis labels, legends, color information, etc.) contained in a chart image. This allows the large-scale language model to interpret chart images by reflecting not only the numbers contained in the chart image but also its semantic structure, and generate accurate answers to queries based on the interpreted results.

[0055] Figures 1, 2a, 2b and 2c are conceptual diagrams for explaining an answer generation system according to the present invention.

[0056] Figures 3a, 3b, 4a and 4b are conceptual diagrams for explaining a method for generating a learning data set in the present invention.

[0057] Figures 5a, 5b and 5c are conceptual diagrams illustrating an embodiment in which a model learned using a learning method according to the present invention generates an answer.

[0058] FIGS. 6A and 6B are conceptual diagrams illustrating a method for generating answers to various types of user queries in the present invention.

[0059] Figure 7 is a flowchart for explaining a method for generating an answer according to the present invention.

[0060] Figures 8a, 8b and 8c are conceptual diagrams for explaining a method for generating an answer according to the present invention.

[0061] FIG. 9 and FIG. 10 are tables and figures showing one example of learning results of a model learned using a learning method according to the present invention.

[0062] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0063] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0064] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0065] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0066] In this application, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0067] The present invention relates to a method and system for generating an answer. The answer generation system according to the present invention performs answer generation based on generative artificial intelligence (Generative AI) or a foundation model, and may also be referred to as an answer generation platform based on a super-large foundation model. However, the “super-large foundation model” may also be referred to as a generative model, a foundation model, or a large language model (or a large language model (LLM)). The answer generation system according to the present invention may be a system that understands various modalities and effectively generates answers to queries in a multi-modal context. In addition, the answer generation system according to the present invention may be a system that understands various modalities and generates high-quality answers to queries based on integrated information in a multi-modal context.

[0068] Multimodal context (or multimodal context, multi-mode context, multi-modal context, etc.) can refer to a contextual structure in which data belonging to different information expression forms (modalities), such as images, charts, and text, exist integratedly within a document or information (or data) environment, and semantic linkages or complementary utilization between them are possible. Here, modality can refer to expression means (or forms, types, etc.) with different information delivery methods, such as visual information (e.g., photos, pictures, charts, etc.) and linguistic information (e.g., articles, descriptions, sentences, etc.). Multimodal context can provide a foundation for humans or artificial intelligence to perform richer and more precise semantic understanding and inference by organically linking or complementarily utilizing data of various modalities within a single context.

[0069] Meanwhile, the answer generation system according to the present invention includes a large-scale language model (or a multi-modal large language model (MLLM)), and the present invention aims to generate and provide high-quality answers by improving the quality of answers for various queries in a multi-modal context.

[0070] Hereinafter, we will look at it in more detail with the attached drawings. FIG. 1, FIG. 2a, FIG. 2b, and FIG. 2c are conceptual diagrams for explaining an answer generation system according to the present invention. FIG. 3a, FIG. 3b, FIG. 4a, and FIG. 4b are conceptual diagrams for explaining a method for generating a learning data set in the present invention. FIG. 5a, FIG. 5b, and FIG. 5c are conceptual diagrams showing an embodiment of a model trained using a learning method according to the present invention to generate answers. FIG. 6a and FIG. 6b are conceptual diagrams for explaining a method for generating answers to various types of user queries in the present invention. FIG. 7 is a flowchart for explaining an answer generation method according to the present invention, and FIG. 8a, FIG. 8b, and FIG. 8c are conceptual diagrams for explaining an answer generation method according to the present invention. Furthermore, FIG. 9 and FIG. 10 are tables and figures showing an embodiment of a learning result of a model trained using a learning method according to the present invention.

[0071] Meanwhile, as illustrated in FIG. 1, the answer generation system (100) according to the present invention may include at least one of an input unit (110), an output unit (120), a communication unit (130), a storage unit (140), a data collection unit (150), a data generation unit (160), a large-scale language model (170), and a control unit (180).

[0072] The answer generation system (100) according to the present invention may include at least one processor and at least one memory containing computer program code. In this case, the memory may function as the storage unit (140) described above. In the present invention, the memory and the program code may cooperate with the processor to perform a series of processes described below.

[0073] Although not shown, the answer generation system (100) according to the present invention may include one or more processors, which may include one or more general-purpose processors and / or one or more special-purpose processors (e.g., a digital signal processor, a tensor processing unit (TPU), a graphics processing unit (GPU), a neural network processing unit (NPU), an application-specific integrated circuit, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a quantum processing unit (or quantum processor, QPU), etc.). The one or more processors may be configured to execute instructions stored (or included) in the storage unit (140), computer-readable instructions, and / or other instructions described herein. The answer generation method and system according to the present invention may allow the memory and at least one processor to cooperate to perform data processing as described below. The processor may perform a series of operations and data processing using data and information stored in the memory. At this time, the memory may be a component of the storage unit (140).

[0074] Furthermore, the answer generation system (100) according to the present invention can perform data processing and calculation processes utilizing quantum gates, quantum entanglement, and quantum superposition states, taking into account implementation in a quantum computer environment. For example, the present invention can perform parallel operations based on qubits, and such quantum operations can operate complementarily with existing classical computers.

[0075] These quantum computers may include qubit-based parallel computing and high-speed data processing devices leveraging quantum entanglement, enabling hardware-based computational optimization using FPGAs and ASICs. Furthermore, quantum computers can utilize quantum processors capable of qubit-based parallel computing, and data processing efficiency can be improved through hybrid architectures with existing classical computers.

[0076] Meanwhile, the input unit (110) may be configured as a means for data input and may be configured in various types. For example, the input unit (110) may be configured to receive user input. The input unit (110) may be configured to receive user input from a user terminal (10). Here, “receiving input” may mean receiving an input signal (or selection signal) corresponding to the user’s input based on the input being made by the user through the input unit configuration provided in the user terminal (10).

[0077] Here, the user terminal (10) may include at least one of a mobile phone, a smart phone, a notebook computer, a laptop computer, a slate PC, a tablet PC, an ultrabook, a desktop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, and a wearable device (e.g., a smartwatch, a smart glass, a head mounted display (HMD)).

[0078] In addition, in the present invention, the input unit (110) does not necessarily mean a hardware means, and can be understood as a passage for receiving input from a user.

[0079] The input unit (110) may also be referred to as a user interface module. The input unit (110) may include a touch screen, computer mouse, keyboard, keypad, touch pad, trackball, joystick, voice recognition module, or other similar device. However, the present invention does not limit the type of input unit (110).

[0080] Here, the user input may include documents, text, images (or videos), voice, etc. In this case, the answer generation system (100) may further include a module that converts voice into text.

[0081] Next, the output unit (120) can output information through an output unit configuration (e.g., a display unit, a touch screen, a speaker, etc.) provided in a user terminal (10) linked to the answer generation system (100) according to the present invention. For example, the output unit (120) can output a page (or service page, 1000) linked to the answer generation system (100) according to the present invention to the display unit of the user terminal (10). In addition, the output unit (120) does not necessarily mean a hardware means, and can be understood as a passage for outputting results to the user.

[0082] Next, the communication unit (130) may be connected to a user terminal (10), a server (e.g., a central server, an external server, etc.), a device, and at least one network through a wireless or wired network, and may be configured to receive or transmit overall data and information necessary for the operation of the answer generation system (100) according to the present invention.

[0083] The communication unit (130) can support various communication methods depending on the communication standards of the communicating device.

[0084] For example, the communication unit (130) may be configured to communicate with a communication target using at least one of WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G (5th Generation Mobile Telecommunication), Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra-Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies.

[0085] Next, the storage unit (140, or memory) serves to store various data related to the present invention, and may include one or more non-transitory computer-readable storage media that can be read and / or accessed by at least one of the one or more processors.

[0086] The one or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, or other memory or disk storage devices. In some examples, the storage (140) may be implemented using a single physical device (e.g., a single optical, magnetic, organic, or other memory or disk storage device), while in other examples, the storage (140) may be implemented using two or more physical devices.

[0087] The storage unit (140) may include computer-readable instructions and additional data. The storage unit (140) may include storage necessary to perform at least some of the methods, scenarios, and techniques described herein and / or at least some of the functions of the devices and networks.

[0088] Furthermore, at least a portion of the storage unit (140) may be a cloud storage or a cloud server. The storage unit (140) may store at least a portion of data corresponding to user input received from the input unit (110) and a training data set (or training data, 210, 220).

[0089] That is, it can be understood that the storage unit (140) is sufficient as a space in which information necessary for the operation of the answer generation system (100) according to the present invention is stored, and there are no restrictions on the physical space.

[0090] Furthermore, the storage unit (140) may store a computer program including computer program instructions. Furthermore, the storage unit (140) may store a computer program including computer program instructions that control the operation of the system (100) or control the operation of the control unit (180) when loaded onto the processor of the system (100).

[0091] Next, the data collection unit (150) may be configured to perform the role of collecting data required for the answer generation system (100) according to the present invention from various sources (e.g., database (DB), website, API, server linked to the answer generation system (100), central server, external server, cloud storage, user terminal (10), etc.).

[0092] For example, the data collection unit (150) may collect chart and table pairs (or chart-table pairs) in which a chart and a table form a pair, in order to generate training data (or training data set) required for training a large-scale language model (170).

[0093] For another example, the data collection unit (150) can collect multi-modal contexts to create multi-modal contexts based on real-world scenarios. The data collection unit (150) can crawl articles from websites or collect at least one document, paper, or report from various sources. These articles, documents, papers, reports, etc. can be configured to include at least one of text, images, and charts.

[0094] Next, the data generation unit (160) may be configured to perform the role of generating (or constructing) a learning data set required for learning a large-scale language model (170).

[0095] For example, the data generation unit (160) can generate a learning data set using charts (or chart images) and tables or multi-modal contexts collected through the data collection unit (150). More specific details on the method for generating a learning data set will be described later.

[0096] This data generation unit (160) may include at least one of a generative model (or a generative AI model), a large-scale language model, and a multi-modal model) provided by a server (e.g., an artificial intelligence server (not shown)) linked to the answer generation system (100) to generate a learning data set. Alternatively, the data generation unit (160) itself may be understood to correspond to at least one of a generative model, a large-scale language model, and a multi-modal model.

[0097] Next, a large-scale language model (170) may be configured to associate a user query with one of multiple modalities constituting a document, specify a modality to be used for generating an answer, and generate an answer to the user query using data extracted from the specified modality.

[0098] A large-scale language model (170) can be trained with multiple different training data sets (210, 220) to understand different modalities and generate appropriate answers to queries in a multi-modal context (e.g., documents) containing multiple modalities with different data characteristics.

[0099] For example, the trained large-scale language model (170) can perform contextual inference on an input query by considering all tokens extracted from each of multiple modalities. Assuming that the user query is “What does this chart describe?”, the trained large-scale language model (170) can generate an answer by simultaneously utilizing the semantic patterns of chart tokens corresponding to the chart modality and contextual clues in text tokens corresponding to the text modality. The trained large-scale language model (170) can perform information identification, comparison, inference, summary, etc. depending on the purpose of the query. More specific details on the training method of the large-scale language model (170) will be described later.

[0100] Next, the control unit (180) may perform a role of controlling the overall operation of the answer generation system (100) related to the present invention. The control unit (180) may process signals, data, information, etc. input or output through the components of the answer generation system (100) discussed above, or perform a series of data processing to provide or process appropriate information and functions to the user. The control unit (180) may be physically implemented by the processor discussed above.

[0101] The control unit (180) can extract data from each of the plurality of modalities using different feature extraction methods specialized for each of the plurality of modalities. In this case, for example, as illustrated in FIG. 2A, when a document (20) including multiple modalities (21, 22, 23) having different data characteristics is received, the control unit (180) can extract data from each of the plurality of modalities (21, 22, 23) using different feature extraction methods based on the data characteristics of each of the plurality of modalities (21, 22, 23).

[0102] Here, “having different data characteristics” may mean that each modality includes different characteristics (or attributes) in terms of data expression format, information structure, temporal and spatial continuity, semantic interpretation criteria (or methods), data processing methods, etc. These multiple modalities (21, 22, 23) may include at least one of an image modality, a chart modality, and a text modality.

[0103] For example, image modality is composed of pixel-based visual information (e.g., two-dimensional visual information), and the data characteristics of the image modality may include at least one of pixel-level color and brightness information, shape of an object (outer contour and structure of an object), texture (texture or repetitive visual pattern), edge or boundary (part where color or brightness change occurs abruptly), and spatial layout (relative position and arrangement relationship between objects).

[0104] For another example, the chart modality has a structure for conveying information through data visualization, and the data characteristics of the chart modality may include at least one of the following: title, axis labels, legend, etc. for explanatory text; scales, units, and ranges of the x-axis and y-axis; visual arrangement of data points (location on coordinates and visual forms such as bars / lines / circles); color information (such as category distinction or emphasis); and shape or marker form (expression method according to chart type).

[0105] For another example, a text modality has a character-based sequential information structure, and the data characteristics of the text modality may include at least one of lexical information (meaning and selection of words and terms), syntactic structure (grammatical arrangement and relationship of sentence units), semantic structure (semantic interpretation of sentences according to context), entity name (proper noun information such as people, places, and organizations), relational expression (logical connection expression such as cause-effect, condition-effect), and document structure (text composition format such as title, body, paragraph, list, and table).

[0106] However, the types of modalities included in the plurality of modalities in the present invention are not necessarily limited to the examples mentioned. For example, the plurality of modalities may further include auditory-based modalities (e.g., audio (background sound, environmental sound, music, etc.)), sensor-based modalities (e.g., sensor data (accelerometer, gyro, heart rate, temperature, etc.)), behavior-based modalities (e.g., gestures / actions (hand gestures, head movements, poses, etc.)), biosignals (bioelectrical signals such as EEG, ECG, EMG, etc.)), spatial-based modalities (e.g., location / geodata (GPS coordinates, location on a map, etc.)), structured modalities (e.g., code / scripts (computer commands, source code, etc.)), etc.

[0107] The control unit (180) can classify each of the plurality of modalities based on preset classification criteria in order to extract data from each of the plurality of modalities using different feature extraction methods. The control unit (180) can classify each of the plurality of modalities (21, 22, 23) according to the data characteristics of each of the plurality of modalities (21, 22, 23) in order to extract data from each of the plurality of modalities (21, 22, 23) included in the received document (20).

[0108] Specifically, the control unit (180) can process the received document (20) as input to the classification module (171). The classification module (171) can be configured to analyze the data characteristics of each of the plurality of modalities (21, 22, 23) constituting the document (20) and classify the plurality of modalities (21, 22, 23) according to the data characteristics (or types, types, categories, kinds, properties, etc.) of each of the plurality of modalities (21, 22, 23). Alternatively, the classification module (171) can be a model learned using a learning data set including various modalities so as to classify various modalities according to the data characteristics of each of the various modalities (e.g., image modality, chart modality, text modality, chemical structure modality including chemical structural formulas or molecular images, etc.). This classification module (171) may also be referred to as a “classification model” or a “classifier”, and may be included as a component of a large-scale language model (170) or may be included in the answer generation system (100) as a separate component from the large-scale language model (170).

[0109] For example, the classification module (171) may classify the first modality (21) among the multiple modalities (21, 22, 23) included in the document (20) as an image modality, the second modality (22) as a chart modality, and the third modality (23) as a text modality based on classification criteria set in accordance with the data characteristics of each of the multiple modalities.

[0110] In addition, the classification module (171) can classify the plurality of modalities according to the characteristics (or types) of each of the plurality of modalities, even if multiple modalities themselves (e.g., multiple images) rather than the document (20) are input. As an example, the image modality and the chart modality each have the characteristics of image data including visual information (or elements), and the file formats (or formats) of the input modalities may be the same. In this case, the classification module (171) can classify the first modality (21) and the second modality (22) based on the data characteristics of the first modality (21) and the data characteristics of the second modality (22), even if the file formats (e.g., JPG, PNG, JPEG, GIF, TIFF, etc.) of the input first modality (21) and second modality (22) are the same.

[0111] In this way, the classification module (171) can classify each of the multiple modalities according to the data characteristics of each of the multiple modalities. That is, the classification module (171) can be configured to classify each of the multiple modalities constituting the document based on preset classification criteria.

[0112] Furthermore, to extract data from each of the multiple modalities (21, 22, 23) classified through the classification module (171).

[0113] Different feature extraction methods may exist differently depending on the data characteristics of each of the classified multiple modalities (21, 22, 23). Different feature extraction methods for extracting data from each of the multiple modalities (21, 22, 23) may be determined based on the data characteristics of each of the classified multiple modalities through the classification module (171).

[0114] The control unit (180) can extract data from each of the classified plurality of modalities (21, 22, 23) through different feature extraction methods based on the data characteristics of each of the classified plurality of modalities (21, 22, 23). The control unit (180) can determine different feature extraction methods specialized for each of the classified plurality of modalities (21, 22, 23) based on the data characteristics of each of the classified plurality of modalities (21, 22, 23), and can extract data from each of the classified plurality of modalities (21, 22, 23) using the determined different feature extraction methods.

[0115] For example, the different feature extraction methods may include at least one of: i) a first feature extraction method specialized for an image modality for extracting data from the image modality, ii) a second feature extraction method specialized for a chart modality for extracting data from the chart modality, and iii) a third feature extraction method specialized for a text modality for extracting data from the text modality.

[0116] However, examples of different feature extraction methods are not necessarily limited to the cases mentioned. For example, if the multiple modalities include an auditory-based modality (or voice modality) and a sensor-based modality (or sensor modality), the different extraction methods may further include a fourth feature extraction method for extracting data from the auditory-based modality and a fifth feature extraction method for extracting data from the sensor-based modality.

[0117] Meanwhile, different feature extraction methods in the present invention may be related to a multi-encoder structure. The answer generation system (100) according to the present invention may extract features based on an independent encoder module specialized for each of a plurality of modalities (21, 22, 23), i.e., a multi-encoder structure. In the case of a multi-encoder structure, a plurality of encoders may be configured (or made) to be activated in parallel or selectively to extract features optimized for the unique data characteristics of each of the plurality of modalities (images, charts, texts, etc.). Such a multi-encoder may be included as a component of a large-scale language model (170) or may be included in the answer generation system (100) as a separate component from the large-scale language model (170).

[0118] In this regard, the control unit (180) may determine whether each of the classified plurality of modalities is input to the multi-encoder based on preset criteria, and may determine at least one modality to be input to the multi-encoder based on the determination result. In this case, the preset criteria may be set to process the image modality and the chart modality through input of the multi-encoder. Accordingly, the at least one modality to be input to the multi-encoder may include at least one of the image modality and the chart modality. Such a multi-encoder may include an image encoder specialized for the image modality and a chart encoder specialized for the chart modality.

[0119] For example, as illustrated in FIG. 2C, the answer generation system (100) can extract visual embeddings using an image encoder (172) in the case of an image modality among multiple modalities. The image encoder (172) may be an encoder of the CLIP or Vision Transformer series. The image encoder (172) can transform pixel-based visual information included in the image modality into a high-dimensional embedding space. This image encoder (172) may also be referred to as a “first encoder,” a “CLIP encoder,” or a “vision encoder.”

[0120] For another example, the answer generation system (100) can extract visual and semantic information of the chart modality using a chart encoder (173) specialized in chart structure recognition in the case of the chart modality among multiple modalities. The chart encoder (173) optimized to effectively understand chart images may be an encoder based on ChartOCR or PlotQA preprocessing. The chart encoder (173) can extract visual and semantic information specific to the chart modality, such as tick information, legends, and color-based data points. This chart encoder (173) may also be referred to as a “second encoder” or a “UniChart Encoder.”

[0121] As another example, although not shown, the answer generation system (100) can extract an embedding that reflects the grammatical structure and linguistic context contained in the text modality by using a pre-trained language (or text) encoder (e.g., BERT, RoBERTa, etc.) in the case of a text modality among multiple modalities. However, after the text modality is classified through the classification module (171), it can be directly input to and processed by a large-scale language model (170) without an additional encoder processing step.

[0122] Furthermore, the present invention can also consider the possibility of expansion to include domain-specific modalities. For example, chemical structures included in chemistry papers or pharmaceutical reports often have ambiguous boundaries between images and charts. Therefore, an encoder specialized for chemical structure recognition can be applied, in conjunction with the SMILES converter, ChemBERTa, or ChemOCR. This can be classified as a specialized modality encoder for chemical structure recognition.

[0123] For example, the answer generation system (100) according to the present invention can process the chemical structure modality through an independent encoder when a chemical structure or molecular image is included in a document of a specific domain (e.g., chemistry, biology, etc.). The answer generation system (100) can convert a chemical structure image into an embedding in the form of text or graph using an encoder specialized in chemical structure recognition (e.g., a chemical structure encoder, a chemical structure recognition encoder, a chemistry-specific encoder, etc.). Although such chemical structures are visually similar to general images or charts, their semantic interpretations are completely different, and thus they can be regarded and processed as a separate modality.

[0124] Referring back to 2a, the control unit (180) may utilize the first feature extraction method to extract data from the first modality (21) corresponding to the image modality among the classified multiple modalities (21, 22, 23). The first modality (21) may also be referred to as an “image modality.”

[0125] Here, the first feature extraction method may be related to an image encoder (172) specialized in extracting data from an image modality. The control unit (180) may extract first specific data (21a) from the first modality (21) through the first feature extraction method. In this case, the first specific data (21a) may also be understood as output data output from the image encoder (172) for the first modality (21).

[0126] The image encoder (172) generates a high-dimensional image embedding vector corresponding to the image modality through encoding for the image modality (e.g., a general image (RGB image) including people, objects, landscapes, etc.), and converts the image embedding vector into a format suitable for the input of a large-scale language model (170) through a projection layer (172a) (i.e., converts into a format suitable for the embedding dimension of the large-scale language model (170)). The output data of the image encoder (172) (i.e., the image embedding vector) may also be named as “image feature (or image feature)” or “visual feature vector” or “image feature vector” or “image representation” or “image representation vector” or “image modality vector”. In this specification, for the convenience of explanation, it will be uniformly described as “image embedding vector”.

[0127] Image embedding vectors may contain information reflecting visual features, such as visual objects, colors, textures, shapes, and spatial arrangements between them, extracted from image modalities. For example, image embedding vectors summarize the semantic content of an image and may be configured as fixed-dimensional vectors, such as 512 or 768 dimensions, that can be utilized for image classification, similarity calculations, text-to-image alignment, etc. In the present invention, image embedding vectors can be utilized as intermediate representations for converting into a format that can be processed by a large-scale language model (170).

[0128] In addition, the projection layer (172a) of the image encoder (172) can convert the output of the image encoder (172) into a format suitable for the input of the large-scale language model (170). For example, the projection layer (172a) of the image encoder (172) can perform an operation to align (or convert) the image embedding vector output from the image encoder (172) to fit the input dimension (or input embedding dimension, token embedding dimension, etc.) of the large-scale language model (170). That is, the projection layer (172a) of the image encoder (172) can perform a role of linearly converting the image embedding vector to fit the input format of the large-scale language model (170). The output data of the projection layer (172a) of the image encoder (172) may also be named as “image token” or “image modality token” or “image feature token” or “image embedding sequence” or “image token sequence” or “projected (or projected) image token” or “projected image token sequence” or “projected image embedding token” or “image embedding vector sequence” or “image token vector sequence”. In this specification, for the convenience of explanation, the term “image token sequence” will be uniformly used for explanation.

[0129] The output data (i.e., the image token sequence) of the projection layer (172a) of the image encoder (172) is the result of converting the image embedding vector into a multi-vector sequence suitable for the input of a large-scale language model (170), and each vector may include information corresponding to a specific region or object within the image. For example, each token of the image token sequence represents a partial representation of visual information and may include information corresponding to a meaningful region of the image, such as a person, a background, an object, or a specific component of the scene. Each token is normalized to match the embedding dimension of the large-scale language model (170), and the overall length of the sequence may be determined according to a preset number of tokens. In addition, each token includes a positional embedding that encodes positional information within the sequence to preserve the visual order, and this structure may be utilized for image semantic interpretation or text generation within a natural language context. Additionally, these sequences can be designed to enable integrated inference with language information through the attention mechanism of a large-scale language model (170).

[0130] In the present invention, the projection layer may also be referred to as a “projection layer” or “projection layer” or “projection layer” or “projection layer” or “projection layer” or “projection layer”. In this specification, for the convenience of explanation, the projection layer will be uniformly referred to as the projection layer.

[0131] Next, the control unit (180) may use a second feature extraction method to extract data from a second modality (22) corresponding to a chart modality among the classified multiple modalities (21, 22, 23). The second modality (22) may also be referred to as a “chart modality.”

[0132] Here, the second feature extraction method may be related to a chart encoder (173) specialized in extracting data from a chart modality. The control unit (180) may extract second specific data (22a) from the second modality (22) through the second feature extraction method. In this case, the second specific data (22a) may also be understood as output data output from the chart encoder (173) for the second modality (22).

[0133] The chart encoder (173) generates a high-dimensional chart embedding vector corresponding to the chart modality through encoding for the chart modality (e.g., chart images such as bar graphs, line graphs, and line graphs), and converts the chart embedding vector into a format suitable for the input of a large-scale language model (170) through a projection layer (172a) (i.e., converts it into a format suitable for the embedding dimension of the large-scale language model (170)). The output data of the chart encoder (173) (i.e., the chart embedding vector) may also be named as “chart feature (or chart feature)” or “chart feature vector” or “chart representation” or “chart representation vector” or “chart modality vector”. In this specification, for the convenience of explanation, it will be described uniformly as chart embedding vector.

[0134] These chart embedding vectors can include information that reflects visual elements within a chart (e.g., axis labels, data series, legends, tick marks, titles, etc.) extracted from the chart modality, as well as the logical and numerical relationships between them. For example, in the case of a bar graph, the height, position, color, and correspondence with the legend of each bar are reflected, and in the case of a line graph, the change trend between data points (or points), slope, etc. can be included. In other words, the chart embedding vector simultaneously reflects positional information, numerical information, and visual hierarchy of visual elements within the chart, and can be a feature representation learned to efficiently express the structured visual information of the chart. In addition, since the chart modality has a more structured visual structure than the image modality, the chart embedding vector can reflect chart-specific information (e.g., maximum / minimum value locations, proportional relationships between items, etc.) that enables numerical comparison and quantitative analysis.

[0135] Additionally, the projection layer (173a) of the chart encoder (173) can convert the output of the chart encoder (173) into a format suitable for the input of a large-scale language model (170). For example, the projection layer (173a) of the chart encoder (173) can perform an operation to align (or convert) the chart embedding vector output from the chart encoder (173) to fit the input dimension (or input embedding dimension, token embedding dimension, etc.) of the large-scale language model (170). The output data of the projection layer (173a) of the chart encoder (173) may also be named as “chart token” or “chart modality token” or “chart feature token” or “chart embedding sequence” or “chart token sequence” or “projected (or projected) chart token” or “projected chart token sequence” or “projected chart embedding token” or “chart embedding vector sequence” or “chart token vector sequence”. In this specification, for the convenience of explanation, it will be described uniformly as a chart token sequence.

[0136] The output data (i.e., chart token sequence) of the projection layer (173a) of the chart encoder (173) is a sequence of multiple continuous vectors that are converted from chart embedding vectors to be suitable for the input format of a large-scale language model, and each token may correspond to a single chart component. For example, one token may represent an x-axis label, another token may represent the value of a specific data bar, and another token may represent a specific series item in a legend. At this time, each token includes both visual properties (e.g., position, color, size, etc.) and semantic properties (e.g., label text, numerical value, unit, etc.) of the corresponding element, and may have the same shape as the input dimension of the large-scale language model (170). In other words, the chart token sequence maps numerical information and structural relationships (e.g., comparison between items, increase / decrease trends, etc.) within the chart into the input format of the large-scale language model (170), and each token may reflect semantic information of a specific chart element (e.g., a specific bar or line section). As a result, chart token sequences can be utilized as information units that enable high-level question-and-answer performance, such as numerical interpretation, comparative question-and-answer, and objective analysis, through fusion with text.

[0137] Furthermore, the control unit (180) may utilize a third feature extraction method to extract third specific data (23a) from a third modality (23) corresponding to a text modality among the classified multiple modalities (21, 22, 23). The third modality (23) may also be referred to as a “text modality.”

[0138] Here, the third feature extraction method may involve a text encoder (not shown, or tokenizer) specialized in extracting data from a text modality. In this case, the text encoder may tokenize the text modality into a form that can be processed by a large-scale language model (170), and input the tokenized text sequence (or text token sequence) into the large-scale language model (170). Alternatively, the text modality may be input directly into the large-scale language model (170) without separate encoding through an encoder. Alternatively, the control unit (180) itself may perform tokenization on the text modality to generate a text sequence (or text token sequence).

[0139] In this way, the present invention analyzes the data characteristics of each of the multiple modalities contained in a document and utilizes an encoder specialized for each modality, thereby minimizing information loss for each modality and extracting optimized features for each of the multiple modalities with different data characteristics. Furthermore, the present invention can flexibly respond to specialized modalities (e.g., chemical structures) through domain-specific encoder extensions, significantly enhancing the system's versatility and robustness.

[0140] Meanwhile, the control unit (180) can process data (21a, 22a, 23a) extracted from each of the plurality of modalities (21, 22, 23) as input to a large-scale language model (170). More specifically, the control unit (180) can input a user query (or a user query token tokenized through tokenization) and data (21a, 22a, 23a) extracted from each of the plurality of classified modalities (21, 22, 23) into the large-scale language model (170) to generate an answer to a user query for a document (20).

[0141] In this case, the first specific data (21a), the second specific data (22a), and the third specific data (23a) extracted through different feature extraction methods can be combined (or integrated, connected, etc.) into one and input into the large-scale language model (170). The control unit (180) can combine the image token sequence, the chart token sequence, and the text token sequence into one input sequence and input it into the large-scale language model (170). For example, the image token sequence, the chart token sequence, and the text token sequence can be combined into one continuous sequence including positional information, and the combined token sequence (or the entire token sequence) can be input into the large-scale language model (170). However, in the present invention, the method of inputting into the large-scale language model (170) is not necessarily input as a combined token sequence, and a method of processing the image token sequence, the chart token sequence, and the text token sequence as inputs separately may also be possible.

[0142] In addition, the combined token sequence may include special tokens (or distinguishing tokens) that distinguish the token sequences corresponding to each of the multiple modalities (21, 22, 23). For example, in the process of generating the combined token sequence, a special token (e.g.,<image_start> ,<image_end> ,<chart_start> ,<chart_end> ,<text_start> ,<text_end> ) can be assigned (or assigned, inserted, etc.). The control unit (180) can assign a distinction token (e.g., ) before the image token sequence, and include a delimiter token according to the data characteristics of the chart modality (e.g., <chart>) before the chart token sequence, and include a delimiter token according to the data characteristics of the text modality (e.g., <text>) can be included before a sequence of text tokens.

[0143] That is, the control unit (180) can assign a special token to each of a plurality of modalities, so that when performing an attention operation of a large-scale language model (170), it can identify which modality a token originates from. That is, during the attention process, the large-scale language model (170) can recognize which token belongs to which modality through position encoding (or position embedding) and a distinguishing token (for example, it can identify a structural hint such as “this token is an image-based token”).

[0144] A large-scale language model (170) can learn, through an attention mechanism, which tokens correspond to which modality a user query primarily allocates attention to. Furthermore, the large-scale language model (170) can learn to infer associations between query types and modalities.

[0145] For example, suppose a user query is “Tell me the highest value in this chart.” In this case, a large-scale language model (170) can assign a high attention weight to a token corresponding to the chart modality. That is, if the attention weight between the token corresponding to the user query and the modality is high, the modality can be determined to be related to the user query.

[0146] For another example, suppose a user query is "Tell me the lowest value in this chart." In this case, the parameters of the large-scale language model (170) can be adjusted during training to focus on chart tokens corresponding to the chart modality for similar queries.

[0147] As another example, suppose the user query is “Where are the objects in the picture?” In this case, a large-scale language model (170) can be trained to focus attention on image tokens corresponding to the image modality.

[0148] Furthermore, the learned large-scale language model (170) is associated with a user query among multiple modalities (21, 22, 23) constituting a document (20), specifies a specific modality to be used for answer generation, and can generate an answer to the user query using specific data extracted from the specific modality.

[0149] In one embodiment, the large-scale language model (170) can process the combined token sequence through an attention mechanism. The large-scale language model (170) can perform self-attention on the combined token sequence. Each token (or query token) of the user query token calculates a pairwise relationship with all tokens including itself, and can focus on tokens with higher correlation, regardless of whether they are text, images, or charts. In this process, tokens corresponding to a specific modality have high attention weights with the user query token, and as a result, the large-scale language model (170) can focus on information of the modality associated with the user query.

[0150] In another embodiment, the large-scale language model (170) can perform a self-attention mechanism on the combined token sequence and calculate the correlation between each token based on the user query token. The user query token can learn which modality's information is important through the attention weight with respect to the previously input combined token sequence. Assuming the user query is "What trend does this chart show?", the user query token assigns a high attention score to the chart tokens corresponding to the chart modality, which can be determined by the large-scale language model (170) that the chart modality is the main information.

[0151] In another embodiment, cross-modal inference occurs through self-attention on the combined token sequence of the large-scale language model (170), and information related to the query can be inferred through the relationships between multiple modalities (21, 22, 23). If information related to the query is contained only in a specific modality, the large-scale language model (170) can generate an answer to the query by assigning attention to the tokens corresponding to the modality. Furthermore, if the query requires inference between multiple modalities (21, 22, 23), the large-scale language model (170) can generate an answer to the query by integrating the meanings between the multiple modalities (21, 22, 23) through cross-attention.

[0152] In addition, the learned large-scale language model (170) can be configured to perform cross-modal inference rather than focusing on a single modality. For example, suppose a user query is “Why is the value shown in the chart explained in the text?” In this case, the control unit (180) can perform inference by simultaneously learning chart tokens corresponding to the chart modality and text tokens corresponding to the text modality. This can be achieved based on the similarity between tokens corresponding to each of the multiple modalities (21, 22, 23) and the correlation between user query tokens within the cross-attention or self-attention layer of the large-scale language model (170).

[0153] That is, the large-scale language model (170) can process a combined token sequence on a token-by-token basis through an autoregressive approach. In this process, the large-scale language model (170) can learn information correlations between multiple modalities (21, 22, 23) through cross-attention between the multiple modalities (21, 22, 23) and generate appropriate answers to user queries.

[0154] Meanwhile, if a specific modality has multiple data characteristics, the control unit (180) can process the modality having multiple data characteristics through multiple feature extraction methods.

[0155] For example, as illustrated in FIG. 2b, the control unit (180) can process the second modality (22) having the data characteristics of the chart modality and the data characteristics of the text modality among the plurality of modalities (21, 22, 23) classified by the classification module (171) through the second feature extraction method and the third feature extraction method. In this case, it can also be understood that the second modality (22) having the plurality of data characteristics is classified into the chart modality and the text modality in the classification module (171). The control unit (180) can extract the second specific data (22a) and the third specific data (23a) from the second modality (22), respectively, through the second feature extraction method and the third feature extraction method.

[0156] Meanwhile, the answer generation system (100) according to the present invention may include a Mixture of Experts (MoE) structure in which multiple expert models are selectively activated according to modality characteristics. Alternatively, the large-scale language model (170) may include an MoE structure in which multiple expert models (or expert modules, specialized models, expert processing models, etc.) and / or sub-models are dynamically activated according to query characteristics and document context. This structure enables the accuracy and versatility of query interpretation and answer generation to be secured simultaneously by selectively deploying the most appropriate expert model in various types of queries or domain situations.

[0157] As an example, the MoE structure can be configured to include an Expert Pool in which a predefined set of sub-models specialized for each modality (image, chart, text, chemical structure, etc.) exist; a Gating Network that dynamically selects an expert model to activate based on the type of user query, the modality associated with the query, the topic of the document, and contextual information; and an Answer Generation Module that synthesizes intermediate outputs or inference results generated by multiple expert models to generate a final answer.

[0158] That is, the answer generation system (100) has a specialized expert model for a specific modality (or multiple modalities), and the control unit (180) or classification module (171) can operate in a manner that comprehensively analyzes the query content, document context, data characteristics of the modality, etc. to activate an appropriate expert model.

[0159] For example, if a user query involves charts, a specialized model specialized in chart analysis can be activated to simultaneously interpret both visual elements (colors, markers, linear trends, etc.) and numeric information (table values, etc.).

[0160] For example, if a complex user query for a document is entered (e.g., “See the changes shown in the chart and check if they match the text description”), a specialized model specializing in chart-text cross-modal inference can be activated to perform relationship inference across different modalities.

[0161] As another example, if a user query related to chemical structures (e.g., “What are the functional groups in the structural formula of this compound?”) is entered, an expert model specialized in chemical structure recognition can be selected to make a judgment on the structure.

[0162] Furthermore, each specialized model or classification module (171) can be utilized not only for direct output to generate an answer to a query, but also for intermediate-stage judgment (e.g., estimating modality importance, emphasizing relevant information, etc.), and in some cases, a structure that cooperates with an upper module (large-scale language model (170)) for multi-stage inference can also be implemented.

[0163] In this way, the present invention can accurately interpret the meaning between multiple modalities through this MoE structure, and can provide a flexible and scalable processing structure for complex queries that are difficult to handle with a single model. Furthermore, the present invention performs multi-stage inference for complex queries through a collaborative inference process involving multiple expert models, rather than a single expert model, within the MoE structure. This enables precise response to both single-modal and cross-modal queries.

[0164] Meanwhile, the present invention provides a method and system for generating answers that can understand various modalities and effectively generate answers to queries in a multi-modal context. More specifically, the present invention provides a large-scale language model that can understand various modalities and generate high-quality answers to queries based on integrated information in a multi-modal context. Below, a method for generating a training data set and a method for training a large-scale language model using the generated training data set will be examined in more detail.

[0165] In the present invention, training of a large-scale language model (170) can be performed to efficiently answer questions in a multi-modal context containing various modalities. More specifically, in the present invention, in a multi-modal context (e.g., a document) containing multiple modalities with different data characteristics, a large-scale language model (170) can be trained so that the large-scale language model (170) can understand each of the multiple modalities and generate answers to queries.

[0166] To this end, the control unit (180) can generate a learning data set to be used for training a large-scale language model (170) and perform training on the large-scale language model (170) using the generated learning data set. In the present invention, the learning data set used for training the large-scale language model (170) may include a plurality of different learning data sets. For example, the plurality of different learning data sets may include at least one of a first learning data set (210) and a second learning data set (220).

[0167] The control unit (180) can train the large-scale language model (170) using a first learning method among a plurality of preset learning methods (or methods, techniques, etc.) to improve the comprehension ability of the large-scale language model (170) for a plurality of modalities having different data characteristics. Here, the first learning method is a method for improving the comprehension ability of the large-scale language model (170) for each of the plurality of modalities, and may also be referred to as “IC-tuning (Image and Chart Instruction Tuning).”

[0168] For training a large-scale language model (170) using the first learning method, a first learning data set (210) among a plurality of different learning data sets (210, 220) may be utilized (or used, utilized, etc.). This first learning data set (210) may be configured to include at least one of a first sub-learning data set and a second sub-learning data set.

[0169] The first sub-learning data set may be configured to include multiple categories of instruction data for learning, including image modality, chart modality, and text modality. For example, the first sub-learning data set may include at least one of: i) instruction data based on image modality and text modality, ii) instruction data based on chart modality and text modality, iii) instruction data based on image modality, iv) instruction data based on chart modality, and v) instruction data based on text modality. The first sub-learning data set may be composed of a plurality (e.g., 1.2 million) or more pieces of question-and-answer (QA) data or description (or description) data. This first sub-learning data set may be data collected by the data collection unit (150) through various sources.

[0170] The second sub-learning data set may be a set generated (or constructed) based on a chart (or chart image) and a table (or table). In the present invention, the second sub-learning data set may be generated by considering not only numerical information (or numerical values) within the table, but also visual information included in the chart image (e.g., text (title, etc.) surrounding the chart, legend (x-axis and y-axis labels), information such as color, etc.). In other words, the second sub-learning data set may be constructed by considering visual information (or visual information) included in the chart image and numerical values ​​of the table.

[0171] The control unit (180) can generate a second sub-learning data set reflecting visual information using the data generation unit (160) based on the table and chart images. For example, as illustrated in FIGS. 3A and 3B , the control unit (180) can generate a second sub-learning data set based on both the chart image (301) and the table (302) using the data generation unit (160). This second sub-learning data set can include at least one of chart summarization data (or chart description data, 211) generated using input data in which a chart image (301) and a table (302) form a pair, open-ended question-answering (or open-ended question-answering (QA) data (or reasoning QA) data (or reasoning-based question-answering (QA) data, 213) generated based on the chart summary data (211).

[0172] In this case, the storage unit (140, or memory) may store a prompt (or input prompt) for generating a second sub-learning data set. More specifically, the storage unit (140) may store at least one prompt that is input to the data generation unit (160) to generate the second sub-learning data set. For example, the storage unit (140) may store a plurality of prompts for generating chart summary data (211), open-ended question response data (212), and inferential question response data (213), respectively.

[0173] In this regard, the control unit (180) may input a prompt (or a first prompt, or a first input prompt) for generating chart summary data (211) among a plurality of prompts for generating a second sub-learning data set and input data (or chart-table pairs) in which a chart image (301) and a table (302) collected through the data collection unit (150) form a pair, to the data generation unit (160). Then, the control unit (180) may obtain chart summary data (211) for the input data generated from the data generation unit (160). The prompt for generating such chart summary data (211) may be configured as in the following embodiment (see Table 1 below). At this time, if only a chart is used and no table exists, the prompt may be entered with certain parts (e.g., "and its table”, "Your description must not mention "e” and as if it were generated from chart alone”) excluded.

[0174] Chart: {Chart}Table: {Table}Given a chart and its table, your goal is to generate a detailed description for the chart. Your description must not mention "table" and as if it were generated from the chart alone. If the chart contains text, you must copy that text. The description should include all numeric values ​​shown in the chart. If you estimate a numeric value from a chart where the numbers are not displayed, you must use the term "estimated value" in the description. If the numbers are not displayed on the chart and you must estimate them, you must use the term "approximately" to indicate that they are estimates.

[0175] The chart summary data (211) generated through the data generation unit (160) provides a more comprehensive understanding than a summary generated simply based on a table, and enables the generation of various queries utilizing visual elements. In addition, the control unit (180) can generate questions and answers for open question response based on the chart summary data (211) generated through the data generation unit (160). The control unit (180) can input a prompt (or a second prompt, or a second input prompt) for generating open question response data (212) among a plurality of prompts for generating a second sub-learning data set and the chart summary data (211) into the data generation unit (160). In addition, the control unit (180) can obtain the open question response data (212) generated from the data generation unit (160). A prompt for generating such open-ended question response data (212) can be configured as in the following example (see Table 2 below, where “Description of chart” in the prompt indicates chart summary data (211)).

[0176] Chart: {Description of chart}Given a chart, generate 4 questions and answers. (Based on the given chart, generate 4 questions and answers.)You should start with "Question_1: ” and "Answer_1: ”.)Question_1 requires identifying a specific target or describing the characteristics of the specific target. (Question_1 requires a query to identify a specific target or describe the characteristics of the specific target.)Question_2 requires comparisons between specified targets. (Question_2 requires comparisons between specific targets.)Question_3 requires mathematical or logical operations such as average, sum, difference, and so on. (Question_3 requires mathematical or logical operations such as average, sum, difference, and so on.)Question_4 requires describing the trends shown in the chart. (Question_4 requires describing the trends shown in the chart.)Question must be simple enough. (Questions should be simple enough.) For Question_1, Question_2, and Question_3, the answer includes step-by-step thinking and a final answer. (Answers to Question_1, Question_2, and Question_3 should include a step-by-step thinking process and provide a final answer.)For Question_4, the answer must be detailed enough but do not use external knowledge to explain the trends in the chart.

[0177] Furthermore, the inferential question response data (213) may be generated similarly to the open question response data (212), but may be generated to include answers in a Chain of Thought (CoT) format. Here, the Chain of Thought format may refer to a format in which a series of logical and step-by-step thought processes for deriving an answer are explicitly described, rather than immediately presenting the correct answer to a question. For example, it may be a format in which a series of intermediate reasoning processes necessary in the problem-solving process are described step-by-step before directly presenting the correct answer to a complex question. The control unit (180) may input a prompt (or a third prompt, or a third input prompt) for generating the inferential question response data (213) from among a plurality of prompts for generating the second sub-learning data set into the data generation unit (160). In addition, the control unit (180) may obtain the inferential question response data (213) generated from the data generation unit (160). The prompt for generating such inferential question response data (213) can be configured as in the following example (see Table 3 below).

[0178] You are responsible to generate an answer for the given chart and question. Note that the chart is converted to a table. Assume that the given table is a chart, not a table. Let's think step by step. And then generate an answer after "The answer is ". (You are responsible to generate an answer for the given chart and question. Note that the chart is converted to a table. Assume that the given table is a chart, not a table. Let's think step by step. And then generate an answer after "The answer is ".) [Chart] Year | Democrats | Republicans | Independents & 2004 | 68.1% | 45.0% | 53.0% & 2006 | 58.0% | 42.0% | 53.0% & 2007 | 59.0% | 38.0% | 45.0% & 2009 | 72.0% | 49.0% | 60.0% & 2011 | 71.0% | 51.2% | 58.0% & 2012 | 70.0% | 48.0% | 53.0% & 2013 | 72.0% | 41.0% | 60.0% Q: In which year did republicans have the lowest favorability rate? A: Let's find the column of republicans. Then let's extract the favor rates, they are [45.0, 42.0, 38.0, 49.0, 51.2, 48.0, 41.0]. The smallest number is 38.0, that's Row 3. Row 3 is year 2007. The answer is 2007. The answer is 2007. (Find the Republican column and find the approval ratings for each year [45.0, 42.0, 38.0, 49.0, 51.2, 48.0, 41.0]. The lowest value is 38.0, which is in the third row. The third row is 2007. The correct answer is 2007.)Q: What is the sum of Democrats' favor rates of 2004, 2012, and 2013? (What is the sum of Democrats' favor rates in 2004, 2012, and 2013?)A: Let's find the rows of years 2004, 2012, and 2013. We find Rows 1, 6, and 7. The favor dates of Democrats on those three rows are 68.1, 70.0, and 72.0. 68.1+70.0+72=210.1. The answer is 210.1. (Find the rows for 2004, 2012, and 2013. They are rows 1, 6, and 7. The Democratic approval ratings in those rows are 68.1, 70.0, and 72.0, respectively. 68.1+70.0+72.0 = 210.1. The correct answer is 210.1.)Q: By how many points do Independents surpass Republicans in the year of 2011? A: Let's find the row with year = 2011. We find Row 5. We extract Independents and Republicans' numbers. They are 58.0 and 51.2. 58.0-51.2=6.8. The answer is 6.8. (Look at the 2011 row (the 5th row) and compare the Independents' 58.0 to the Republicans' 51.2. 58.0 - 51.2 = 6.8. The correct answer is 6.8.) Q: Which group has the overall worst performance? A: Let's sample a couple of years.In Row 1, year 2004, we find Republicans having the lowest favor rate of 45.0 (since 45.0<68.1, 45.0<53.0). In year 2006, Row 2, we find Republicans having the lowest favor rate of 42.0 (42.0<58.0, 42.0<53.0). The trend continues to other years. The answer is Republicans. (Let's sample a few years. In 2004 (Row 1), Republicans had the lowest favor rate of 45.0 (45.0<68.1, 45.0<53.0). In 2006 (Row 2), Republicans also had the lowest favor rate of 42.0 (42.0<58.0, 42.0<53.0). This trend continues to other years. The answer is Republicans.) Q: Which party has the second highest favor rates in 2007? (Which party had the second highest approval rating in 2007?) A: Let's find the row of year 2007, that's Row 3. Let's extract the numbers on Row 3: [59.0, 38.0, 45.0]. 45.0 is the second highest. 45.0 is the number of Independents. The answer is Independents. (2007 is the third row. The numbers are [59.0, 38.0, 45.0]. The second highest is 45.0, which is the number for Independents. The correct answer is Independents.) [Chart]{Table} Q: {Question}.

[0179] As discussed above, the present invention can generate a second sub-learning data set that includes both the exact numerical information included in the table and the visual information expressed in the chart image. In the present invention, the second sub-learning data set can also be named “visual-aware chart instruction-following data”, “visual recognition (or information)-based chart instruction-following data”, “visual information-based chart instruction data”, or “visual information-reflecting chart instruction-based data”. Meanwhile, the first learning method can be configured as a learning procedure including a plurality of steps (or steps, stages, etc.). First, the first step (or the first step, the first stage, etc.) among the plurality of steps of the first learning method can be understood as a pre-learning step for aligning image and chart features. In the first step, the projection layer (172a) of the image encoder (172) can be trained with training data in which an image and text form a pair, and the projection layer (173a) of the chart encoder (173) can be trained with training data in which a chart and text form a pair and training data in which a table and a chart form a pair. At this time, the parameters of the large-scale language model (170) can be maintained in a frozen state. That is, in the first step, only the projection layers (172a, 173a) of the image encoder (172) and the chart encoder (173) are trained, and modality feature alignment data can be mixed and used.

[0180] Next, the second step (or second step, second stage, etc.) among the multiple steps of the first learning method can be understood as a learning step based on image and chart instruction following data. In the second step, learning (e.g., fine-tuning) can be performed on the projection layers (172a, 173a) of the large-scale language model (170) and the image encoder (172) and the chart encoder (173) respectively, using the first sub-learning data set among the multiple sub-learning data sets included in the first learning data set (210).

[0181] Furthermore, the third step (or third step, third stage, etc.) among the multiple steps of the first learning method can be understood as an additional fine-tuning step using the second sub-data set. In the third step, a large-scale language model (170) can be additionally trained using the second sub-data set generated in the present invention. At this time, in order to prevent the forgetting phenomenon of the large-scale language model (170), in the third step, at least a portion of the first sub-learning data set used in the second step can be sampled together to perform additional training of the large-scale language model (170).

[0182] Meanwhile, a large-scale language model (170) trained with a first learning data set (210) can be further trained using a second learning data set (220) to generate responses to different types of queries in a multi-modal context including multiple modalities with different data characteristics.

[0183] Here, the different types of queries may include a first type of query and a second type of query. The first type of query may include a single-modal query, and the second type of query may include a cross-modal query.

[0184] For example, for unimodal queries, a large-scale language model must accurately identify the evidence supporting the answer within a multimodal context. As another example, for cross-modal queries, a large-scale language model must be able to identify evidence across multiple modalities and generate an answer based on that evidence.

[0185] To this end, the control unit (180) may construct a second learning data set (220) to enable a large-scale language model (170) to generate efficient responses to queries in a multi-modal environment including image, chart, and text modalities. In the present invention, the second learning data set (220) may also be referred to as an “ICT-QA (Question Answering over Image, Chart, and Text) data set.”

[0186] As discussed above, the data collection unit (150) can collect multimodal contexts to create multimodal contexts based on real-world scenarios. The data collection unit (150) can crawl articles from websites or collect at least one document, paper, or report from various sources. These articles, documents, papers, reports, etc. can be configured to include at least one of text, images, and charts.

[0187] For example, as illustrated in FIG. 4a, the control unit (180) can extract one image (401), one chart (402), and the entire text (403) included in a document (or article) collected through the data collection unit (150) to create (or configure) one multi-modal context.

[0188] The control unit (180) may define multiple types of queries to generate first-type and second-type queries for the multi-modal context. For example, the table (450) illustrated in FIG. 4B may illustrate examples of each query type. A query may specify a location where evidence exists based on a specific modality combination (e.g., "Image + Text," "Chart + Text," etc.). These types may each refer to a query requiring inference capabilities between an image (401) and text (403) or a chart (402) and text (403).

[0189] The control unit (180) may utilize the data generation unit (160) to generate predefined question and response type pairs for the multi-modal context. For the image modality and the chart modality, the control unit (180) may first generate a description of the image (401) or the chart (402), and then generate a question and response based thereon. In addition, when generating a query for the image modality and the chart modality, the control unit (180) may also include text (e.g., a paragraph) surrounding the image (401) or the chart (402) in the prompt for generating the second learning data set (220) in order to generate a second type of query.

[0190] The storage unit (140, or memory) may store a prompt (or input prompt) for generating a second learning data set (220). More specifically, the storage unit (140) may store at least one prompt that is input to the data generation unit (160) to generate the second learning data set (220). For example, the storage unit (140) may store prompts for generating image question-answering (Image QA) data, chart question-answering (Chart QA) data, and text question-answering (Text QA) data, respectively.

[0191] The control unit (180) can input the multi-modal context and a prompt for generating the second learning data set (220) to the data generation unit (160) to generate the second learning data set (220).

[0192] Among the prompts for generating the second learning data set (220), the prompts input to the data generation unit (160) for generating image question response data (image QA data) can be configured as in the following example (see Table 4 below).

[0193] {Image}Title: {Article title}Context: {Context}First, generate a detailed description of the given image. (Starting with "Detailed description of the image:”)Second, generate 3 questions and answers about the image in the context of the given text. The generated question and answer should start with "Question_1: " and "Answer_1”: (Next, write 3 questions and 3 answers based on the image in the context of the given text. The generated questions and answers must start with the following format: "Question_1”:, "Answer_1”:)Question should ask about information described in the image in the context of text. Question_1 ask identifying the target or describing the characteristics of the specific target. Question_2 ask how the specific target of image relates to the surrounding text. Question_3 ask why the image is included in the document. (Queries should ask what the information described in the image means in the context of the text. Question_1 should be a query that identifies a specific object in the image or describes a characteristic of that object. Question_2 should be a query that asks how a specific object in the image relates to the surrounding text.Question_3 should ask why the image was included in the document. Answer should include only the information described in the image and text. Answer should be detailed. Answer should explain where the evidence was derived from image or text. (Your answer should only include the information described in the image and text. Your answer should be specific and detailed. Your answer should clearly state whether the evidence was derived from the image or the text.)

[0194] In addition, among the prompts for generating the second learning data set (220), the prompts input to the data generation unit (160) for generating chart question response data (chart QA data) can be configured as in the following example (see Table 5 below).

[0195] {Chart}Title: {Article title}Context: {Context}First, generate a detailed description of the given chart. (Starting with "Detailed description of the chart:”)Second, generate 3 questions and answers about the chart in the context of the given text. The generated question and answer should start with "Question_1”: and "Answer_1”: (Next, write 3 questions and answers based on the chart in the context of the given text. The generated questions and answers must start with the following format: "Question_1”:, "Answer_1”:)Question should ask about information described in the chart. Question should not contain phrases that refer to charts, such as "on this chart." Question_1 requires identifying a specific target or describing the characteristics of the specific target. Question_2 requires comparisons between specified targets. Question_3 why the chart is included in the document. Answer should include only the information described in the chart and text. Answer should be detailed.Answer should explain where the evidence was derived from chart or text. (Questions should be based on the information described in the chart and should not include phrases that directly refer to the chart, such as “in this chart.” Question_1 should identify a specific entity or describe a characteristic of that entity. Question_2 should ask for a comparison between the specified entities. Question_3 should ask why the chart was included in the document. Answers should include only information specified in the chart and text. Answers should be specific and detailed, and should clearly state whether the evidence is derived from the chart or the text.)

[0196] In addition, among the prompts for generating the second learning data set (220), the prompt input to the data generation unit (160) for generating text question response data (text QA data) can be configured as in the following example (see Table 6 below).

[0197] Title: {Article title}Context: {Context}Generate 3 questions and answers. Generated questions and answers should start with "Question_1: " and "Answer_1": Question should ask about information described in the given text. Question_1 requires identifying a specific target or describing the characteristics of the specific target. Question_2 requires reasoning. Question_3 requires summarizing. Answer should include only the information described in the text. Answer should be detailed. Answer should explain where the evidence was derived from the text.

[0198] In this case, {Context} included in the prompt examined above may mean a paragraph close to the image or chart in the case of image QA data and chart QA data (see Tables 4 and 5). In addition, in the case of text QA data, the remaining text excluding the paragraph used for image and chart QA generation may be used as {Context} (see Table 6). Furthermore, the control unit (180) may obtain a second learning data set (220) generated from the data generation unit (160). The second learning data set (220) can be constructed so that a first type of query and a response to the first type of query for the multi-modal context form a pair, and a second type of query and a response to the second type of query for the multi-modal context form a pair (see FIG. 4a). A large-scale language model (170) trained with the first learning data set (210) can be further trained using the second learning data set (220) to generate an answer to a first type of query or a second type of query in a multi-modal context including multiple modalities.

[0199] The control unit (180) can additionally train the large-scale language model (170) trained with the first learning data set (210) using the second learning data set (220). The trained large-scale language model (170) can understand multiple modalities and generate answers to user queries from documents containing multiple modalities. That is, in the present invention, by further fine-tuning the large-scale language model (170) trained with the first learning data set (210) using the second learning data set (220), the ability to search for evidence related to user queries within a multimodal context including images, charts, text, etc. can be improved, and the quality of answers generated for various types of queries (e.g., single-modal queries, cross-modal queries) can be improved.

[0200] Below, we will look at an example embodiment in which a large-scale language model (170) trained with a first learning data set (210) and a second learning data set (220) generates an answer, together with the attached drawings.

[0201]

[0202] In one embodiment, FIG. 5A illustrates image QA data belonging to a target type (see table (450) of FIG. 4B), where both the image modality (601) and the text modality (603) among multiple modalities (601, 602, 603) must be referenced. In this case, the query (600) may be included in a second type of query among different types of queries. The trained large-scale language model (170) effectively understood the text (603) of the document, accurately interpreted the image (601), and derived an answer (610) to the query (600). This result may indicate that the trained large-scale language model (170) possesses multi-modal inference capabilities.

[0203] In another embodiment, FIG. 5b illustrates a case in which chart QA data belonging to a comparison type (see table (450) of FIG. 4b) must refer to the chart modality (622) among multiple modalities (621, 622, 623). In this case, the query (620) may be included in a first type of query among different types of queries. The trained large-scale language model (170) accurately derived an answer (630) to the query (620) based on the information presented in the chart (622).

[0204] In another embodiment, FIG. 5c illustrates a case in which text QA data belonging to an identification type (see table (450) of FIG. 4b) must refer to a text modality (643) among multiple modalities (641, 642, 643). In this case, the query (640) may be included in a first type of query among different types of queries. The trained large-scale language model (170) successfully extracted the necessary information from the text modality (643) and accurately derived an answer (650) to the query (640).

[0205] In this way, it can be confirmed that the large-scale language model (170) learned through the learning method according to the present invention effectively performs QA tasks in a multimodal context and shows high comprehension in all situations requiring single or multiple modalities.

[0206] Meanwhile, as discussed above, a query input for a multi-modal context containing multiple modalities may include different types of queries (e.g., a first type of query and a second type of query).

[0207] Accordingly, user queries received (or input) from a user terminal (10) for a document including multiple modalities may also include different types of user queries. For example, the different types of user queries may include a first type of user query and a second type of user query. The first type of user query may include a single-modal query, and the second type of user query may include a cross-modal query.

[0208] As discussed above, a large-scale language model (170) can be trained to understand multiple modalities and generate answers to queries in a multi-modal context containing multiple modalities with different data characteristics. That is, the large-scale language model (170) can be trained to understand multiple modalities in a document containing multiple modalities and generate answers to a first type of user query or an answer to a second type of user query.

[0209] First, let's examine a method for generating a response to a first type of user query when receiving a first type of user query among different types of user queries in the present invention. Since the process of classifying multiple modalities through the classification module (171) and / or the process of extracting data from each of the multiple modalities through the multi-encoder have been described in greater detail above, they will be briefly described below to avoid duplication of explanation.

[0210] In one embodiment, as illustrated in FIG. 6A, when the control unit (180) receives a plurality of modalities (501, 502, 503) having different data characteristics, the control unit (180) may input the plurality of modalities (501, 502, 503) into the classification module (171). The classification module (171) may classify the first modality (501) as an image modality, the second modality (502) as a chart modality, and the third modality (503) as a text modality based on (or based on) the data characteristics of each of the plurality of modalities (501, 502, 503).

[0211] In addition, the control unit (180) can input the first modality (501) classified through the classification module (171) into the image encoder (172) and input the second modality (502) into the chart encoder (173). In addition, the control unit (180) can perform tokenization on data extracted from the third modality (503) to generate a text token sequence (503a).

[0212] Furthermore, the control unit (180) can obtain an image token sequence (501a), which is data extracted from the first modality (501), from the projection layer (172a) of the image encoder (172), and can obtain a chart token sequence (502a), which is data extracted from the second modality (502), from the projection layer (173a) of the chart encoder (173). The control unit (180) can combine the image token sequence (501a), the chart token sequence (502a), and the text token sequence (503a) into one input sequence to generate a combined token sequence, and process the combined token sequence and a user query (500) for multiple modalities (501, 502, 503) as inputs of a large-scale language model (170).

[0213] A large-scale language model (170) is associated with a first type of user query (500) among a plurality of modalities (501, 502, 503) based on the reception (or input) of a first type of user query (500) among different types of user queries, specifies one specific modality to be used for answer generation, and generates an answer to the user query (500) using specific data extracted from one specific modality.

[0214] For example, a large-scale language model (170) can generate an answer by intensively analyzing parts related to a user query (500) focusing on location information, numerical information, color, etc. within a specific token sequence through an attention mechanism. Based on the user query (500) being “What is the percentage of people in Japan who consider North Korea’s nuclear weapons program a major threat?”, the large-scale language model (170) can extract information (or evidence) related to the user query from chart tokens corresponding to chart modalities among the combined token sequences, and generate an answer (510) using the extracted information.

[0215] Next, let us examine a method for generating an answer to a second type of user query when a second type of user query is received among different types of user queries in the present invention.

[0216] In one embodiment, as illustrated in FIG. 6b, when the control unit (180) receives a plurality of modalities (521, 522, 523) having different data characteristics, the control unit (180) may input the plurality of modalities (521, 522, 523) into the classification module (171). The classification module (171) may classify the first modality (521) as an image modality, the second modality (522) as a chart modality, and the third modality (523) as a text modality based on (or based on) the data characteristics of each of the plurality of modalities (521, 522, 523).

[0217] In addition, the control unit (180) can input the first modality (521) classified through the classification module (171) into the image encoder (172) and input the second modality (522) into the chart encoder (173). In addition, the control unit (180) can perform tokenization on data extracted from the third modality (523) to generate a text token sequence (523a).

[0218] Furthermore, the control unit (180) can obtain an image token sequence (521a), which is data extracted from the first modality (521), from the projection layer (172a) of the image encoder (172), and can obtain a chart token sequence (522a), which is data extracted from the second modality (522), from the projection layer (173a) of the chart encoder (173). The control unit (180) can combine the image token sequence (521a), the chart token sequence (522a), and the text token sequence (523a) into one input sequence to generate a combined token sequence, and process the combined token sequence and a user query (500) for multiple modalities (521, 522, 523) as inputs of the large-scale language model (170).

[0219] A large-scale language model (170) is associated with a second type of user query (520) among multiple modalities (521, 522, 523) based on the second type of user query (520) being received (or input) among different types of user queries, and specifies one specific modality to be used for generating an answer, and can generate an answer to the user query (520) using specific data extracted from one specific modality.

[0220] For example, a large-scale language model (170) can perform semantic connections between multiple modalities (521, 522, 523) through cross-attention. Assume that a user query (520) is “How does this image relate to the concerns of gay or bisexual men regarding the monkeypox vaccine response?” The large-scale language model (170) can infer a connection between an image token sequence (521a) extracted from a first modality (521) and a text token sequence (523a) extracted from a third modality (523) among the combined token sequences (i.e., infer a correspondence between elements included in a scene of an image and sentences included in a text), thereby generating an answer (530) to the user query (520). That is, based on the integrated multi-modal output (combined token sequence) in the attention mechanism-based structure of a large-scale language model (170), information (or evidence) related to the user query (520) can be extracted (or identified), and an answer (530) to the user query (520) can be generated through chain-of-thought reasoning or regression.

[0221] Meanwhile, the answer generation method and system (100) discussed above can be applied and utilized effectively in various artificial intelligence application services. For example, the present invention can be applied and utilized effectively in conversational artificial intelligence (or agent) services (e.g., chatbots).

[0222] In this case, the answer generation system (100) according to the present invention can be effectively applied to a chatbot-type application that enhances the accuracy, suitability, and personalization of responses to user queries through a processing structure including at least one of a planner (or planner module), a selector (or selector module), and a classifier (or classification module).

[0223] In one embodiment, the answer generation system (100), upon receiving user input via a chatbot, may utilize a classifier to analyze the context, intent, and requirements of the query included in the user input. The classifier may provide reference information for subsequent processing by categorizing the user query into predefined query types or intent categories. This classifier may determine whether the query corresponds to an information request, a recommendation request, or a command execution type. Furthermore, the classifier may classify multiple modalities, each comprising at least one document included in the user input, based on the data characteristics of each of the multiple modalities.

[0224] In addition, the classifier can interpret the meaning contained in the user query by classifying the intent, emotion, topic, or query type of the user query, and infer the final response content based on the interpretation result. For example, if the classification result is determined to be an "information request," the classifier can operate by extracting and generating the most appropriate information among the response types corresponding to the request. In this way, the classifier can go beyond a simple preprocessing step and play a key role in determining the type and content of the response to be generated by the system (100), thereby contributing to inferring an accurate answer.

[0225] Furthermore, the planner can configure an appropriate processing flow and plan an appropriate path among multiple candidate responses or processing modules. The planner can analyze the context of user input, conversation history, etc. to infer the purpose of the overall conversation and plan a high-level response strategy or next step task to be performed by the answer generation system (100). For example, based on the classification results of the classifier, the planner can design the subsequent conversation flow and sequentially arrange tasks to be performed at each step (e.g., response generation, information search, follow-up query request, etc.). The planner can be configured to operate dynamically by comprehensively considering conversation history, user information, internal system status, etc.

[0226] Furthermore, the selector can select the optimal response from among multiple response candidates or actions corresponding to a step or task specified by the planner, taking into account the current conversation context and user characteristics. For example, the selector can select the optimal processing module or response method based on the current conversation situation, user profile, and previous conversation history. This allows for different processing paths to be selected for the same recommendation request depending on the user's preferences or circumstances, thereby increasing the system's flexibility and adaptability.

[0227] In this way, the present invention, through its collaborative structure of planner-selector-classifier, can provide sophisticated responses tailored to the context of a user's query, enabling the implementation of customized chatbot systems in various domains. Therefore, the present invention goes beyond simple question-and-answer functionality and provides comprehensive conversation management capabilities that consider various factors, such as the purpose of the conversation, user status, and conversation flow design.

[0228] Meanwhile, the method and system for generating an answer including a large-scale language model (170) learned according to the learning method discussed above can provide a high-quality answer that meets the user's needs (or intentions) through the steps of receiving a document including a plurality of modalities having different data characteristics (S710), as illustrated in FIG. 7, in the inference stage, a step of extracting data from each of the plurality of modalities through different feature extraction methods based on the data characteristics of each of the plurality of modalities (S720), a step of receiving a user query for the document from a user terminal (S730), a step of specifying at least one specific modality related to the user query among the plurality of modalities constituting the document (S740), and a step of generating an answer to the user query using specific data extracted from the specific modality (S750).

[0229] As illustrated in FIG. 8a, the control unit (180) can receive a document (800) including multiple modalities based on a user input received through a service page (1000) output (or provided) on a display unit of a user terminal (10). For example, the control unit (180) can receive a document (800) including multiple modalities and a user query (810) for the document (800) based on selection of a graphic object (1000a) linked to a query and document receiving function included in the service page (1000) from the user terminal (10).

[0230] The control unit (180) can classify each of the multiple modalities included in the document based on preset classification criteria in order to extract data from each of the multiple modalities included in the document using different feature extraction methods. The control unit (180) can classify each of the multiple modalities included in the document (800) according to the data characteristics of each of the multiple modalities included in the document (800) in order to extract data from each of the multiple modalities included in the document (800). For example, as illustrated in FIG. 8B, the control unit (180) can process the document (800) as input to the classification module (171). The classification module (171) can analyze the data characteristics of each of the multiple modalities (801, 802, 803) constituting the document (800) and classify each of the multiple modalities (801, 802, 803) according to the data characteristics of each of the multiple modalities (801, 802, 803). The classification module (171) can classify the first modality (801) as an image modality, the second modality (802) as a chart modality, and the third modality (803) as a text modality based on predetermined classification criteria according to data characteristics of each of the multiple modalities (801, 802, 803) included in the document (800).

[0231] In addition, the control unit (180) can extract data from each of the plurality of modalities (801, 802, 803) using different feature extraction methods specialized for each of the plurality of modalities (801, 802, 803). More specifically, the control unit (180) can extract data from each of the plurality of classified modalities (801, 802, 803) using different feature extraction methods based on data characteristics of each of the plurality of classified modalities (801, 802, 803).

[0232] At this time, the control unit (180) determines different feature extraction methods specialized for each of the classified plurality of modalities (801, 802, 803) based on the data characteristics of each of the classified plurality of modalities (801, 802, 803), and extracts data from each of the classified plurality of modalities (801, 802, 803) using the determined different feature extraction methods. In this case, the control unit (180) determines whether each of the classified plurality of modalities (801, 802, 803) is input to a multi-encoder based on a preset standard, and determines at least one modality to be input to the multi-encoder based on the determination result.

[0233] As discussed above, at least one modality to be input to the multi-encoder may include at least one of an image modality and a chart modality. Accordingly, the control unit (180) may determine the first modality (801) and the second modality (802) as the modalities to be input to the multi-encoder.

[0234] For example, the different feature extraction methods may include at least one of i) a first feature extraction method for extracting first specific data (801a) from a first modality (801), ii) a second feature extraction method for extracting second specific data (802a) from a second modality (802), and iii) a third feature extraction method for extracting third specific data (803a) from a third modality (803).

[0235] In relation to the first feature extraction method for extracting the first specific data (801a), the control unit (180) can input the first modality (801) to the image encoder (172). Here, the image encoder (172) can be an encoder specialized in extracting data from the first modality (801).

[0236] The image encoder (172) can generate an image embedding vector corresponding to the first modality (801) through encoding for the first modality (801). The projection layer (172a) of the image encoder (172) can convert the image embedding vector into an image token sequence suitable for the input dimension of a large-scale language model (170).

[0237] In relation to the second feature extraction method for extracting the second specific data (802a), the control unit (180) can input the second modality (802) to the chart encoder (173). Here, the chart encoder (173) can be an encoder specialized in extracting data from the second modality (802).

[0238] The chart encoder (173) can generate a chart embedding vector corresponding to the second modality (802) through encoding for the second modality (802). The projection layer (173a) of the chart encoder (173) can convert the chart embedding vector into an image token sequence suitable for the input dimension of a large-scale language model (170).

[0239] In relation to the third feature extraction method for extracting the third specific data (803a), the control unit (180) can tokenize the third modality (803) into a form that can be processed by a large-scale language model (170) to generate a tokenized text token sequence.

[0240] Furthermore, the control unit (180) can configure input data including specific data (801a, 802a, 803a) extracted from each of the classified plurality of modalities (801, 802, 803), and process a user query (810) for a document (800) including the input data and the classified plurality of modalities (801, 802, 803) as input to a pre-trained large-scale language model (LLM). The input data can include at least one of first specific data (801a) extracted from the first modality (801) through a first feature extraction method, second specific data (802a) extracted from the second modality (802) through the first feature extraction method, and third specific data (803a) extracted from the third modality (803) through a third feature extraction method.

[0241] The control unit (180) can process input data including data (801a, 802a, 803a) extracted from each of a plurality of modalities (801, 802, 803) as input to a pre-trained large-scale language model (170). More specifically, the control unit (180) can input a user query (810, or a user query token tokenized through tokenization) and data (801a, 802a, 803a) extracted from each of a plurality of classified modalities (801, 802, 803) into the large-scale language model (170) to generate an answer to a user query (810) for a document (800). In the present invention, the input data input to the large-scale language model (170) may also be referred to as a “combined token sequence.”

[0242] In this case, the first specific data (801a), the second specific data (802a), and the third specific data (803a) extracted through different feature extraction methods can be combined (or integrated, connected, etc.) into one and input into a large-scale language model (170). For example, the control unit (180) can generate a combined token sequence by combining an image token sequence, a chart token sequence, and a text token sequence, and input the combined token sequence into the large-scale language model (170).

[0243] A large-scale language model (170) can generate an answer (820) to a user query (810) by specifying at least one specific modality among the multiple modalities (801, 802, 803) associated with a user query (810) based on (or using) data (801a, 802a, 803a) extracted from each of the multiple modalities (801, 802, 803), and using specific data extracted from the specified modality. More specifically, the large-scale language model (170) can generate an answer (820) to a user query (810) by using a combined token sequence. In this case, the large-scale language model (170) can specify the first modality (801), the second modality (802), and the third modality (803) as specific modalities to be used for generating an answer and associated with the user query (810) among the multiple modalities (801, 802, 803) based on the fact that the user query (e.g., "How do white reporters and black reporters use Instagram differently?", 810) belongs to the second type of query. The large-scale language model (170) can generate an answer (820) to the user query (810) by using specific data (801a, 802a, 803a) extracted from the multiple specific modalities (801, 802, 803).

[0244] For example, as illustrated in FIG. 8c, the large-scale language model (170) can generate an answer (820) to the user query (810) by inferring a correlation between an image token sequence extracted from a first modality (801), a chart token sequence extracted from a second modality (802), and a text token sequence extracted from a third modality (803). That is, the large-scale language model (170) can extract information related to the user query (810) from specific data (801a, 802a, 803a) extracted from each of a plurality of modalities (801, 802, 803), and generate an answer (820) to the user query (810) through a chain of reasoning or regression method of thought.

[0245] Furthermore, the control unit (180) can provide a generated response to a user query (810) (e.g., “Instagram is more popular among black reporters than white reporters. In particular, 33% of black reporters cited Instagram as one of their most used sites, which is a higher figure than about 18% of white reporters. This information was derived through demographic analysis of reporters’ use of social media sites, which shows a significant difference in Instagram popularity between the two racial groups.”, 820) to the user terminal (10) on which the service page (1000) is output.

[0246] In another embodiment of the present invention, when a document and a user query regarding the document are received, the answer generation system (100) can specify a modality associated with the user query among a plurality of modalities included in the document. At this time, the answer generation system (100) can analyze the user query to determine whether the user query includes a plurality of specific modalities, and based on the determination result, can specify a plurality of specific modalities included in the user query among the plurality of modalities included in the document. In addition, the answer generation system (100) can extract data from each of the plurality of specific modalities and use the extracted data to generate an answer to the user query.

[0247] That is, the present invention, when a document including multiple modalities and a user query for the document are received, performs a process of specifying a specific modality related to the user query, a process of extracting data from the specific modality, and a process of generating an answer to the user query using the extracted data, thereby reducing the amount of computation by selectively processing only the modalities related to the query, simultaneously improving response speed and accuracy, and preventing unnecessary resource consumption, thereby increasing system efficiency and multimodal scalability.

[0248] Meanwhile, FIGS. 9 and 10 show one example of the learning results of a model learned using a learning method according to the present invention.

[0249] In one embodiment, referring to FIG. 9 (a), it can be confirmed that a large-scale language model (170) trained using a learning method according to the present invention records superior performance compared to conventional models for chart summary and OpenCQA (Open-ended Chart Question Answering) tasks. In particular, the performance of the large-scale language model (170) according to the present invention is improved through additional fine-tuning using the second sub-data set, which may indicate that training using the second sub-data set is effective in improving the chart understanding ability of the large-scale language model (170).

[0250] In another embodiment, referring to (b) of FIG. 9, it can be confirmed that the large-scale language model (170) trained with the second learning data set (220) according to the present invention exhibits excellent performance in various modality-based question-answering tasks. In particular, the large-scale language model (170) according to the present invention is additionally trained through the second learning data set (220), thereby simultaneously considering all input modalities and improving its performance in comprehensively understanding the interconnected information between images, charts, and texts.

[0251] In another embodiment, referring to FIG. 10, while the conventional model used only charts and text as input, the large-scale language model (170) according to the present invention uses all modalities, including images, charts, and text, as input to generate answers. Consequently, the present invention can enhance the question-answering performance in a multimodal context and contribute to generating more accurate and high-quality answers by training the large-scale language model (170) using the first and second learning data sets.

[0252] As described above, the method and system for generating answers according to the present invention classifies multiple modalities having different data characteristics, and applies a specialized feature extraction method to each of the classified multiple modalities, thereby enabling precise extraction of data from each of the different modalities, such as images, charts, and text. In other words, the present invention extracts information by reflecting the data characteristics of each of the multiple modalities, thereby enabling precise extraction of the core data required to generate answers to queries without information loss. Through this, the present invention can more precisely grasp the semantic connectivity between various expression forms, thereby effectively utilizing only the core information required to generate answers to user queries while maintaining the semantic connectivity between various expression forms.

[0253] Furthermore, the answer generation method and system according to the present invention utilizes a multi-encoder structure including encoders specialized for each of a plurality of modalities, thereby providing optimized recognition and processing performance tailored to the data characteristics of each modality. Accordingly, the present invention can generate accurate answers to queries by recognizing visual and / or contextual information of each modality without loss. In other words, the present invention can improve the recognition accuracy of each modality while reflecting the characteristics of each modality without loss, and maintain the performance of the overall model.

[0254] Furthermore, according to the method and system for generating an answer according to the present invention, a modality related to a user query is specified among a plurality of modalities included in a document, and an answer is generated based on data extracted from the specified modality, thereby providing the user with a high-quality answer generated based on information related to the user query. In particular, the present invention can save system resources and improve the answer generation speed by selecting and efficiently processing a modality related to the user query, and can generate an appropriate answer through integrated inference even for a user query including various modalities such as images, charts, and text.

[0255] Furthermore, according to the answer generation method and system according to the present invention, a learning data set can be configured to enable integrated understanding and inference of various modalities such as images, charts, and text, and a large-scale language model (LLM) can be trained using this data set. Through this, the large-scale language model can comprehensively understand the information contained in complex multi-modal documents and generate accurate answers to user queries based on the results of the understanding. In other words, the large-scale language model according to the present invention can generate various types of answers such as not only identification-type answers but also summary-type, comparison-type, context-interpretation-type, and purpose-inference-type answers, and in particular, it can derive integrated inference results even for complex queries including images, charts, and text. Through this, users can receive customized answers that meet various query purposes.

[0256] Furthermore, the answer generation method and system according to the present invention can improve the chart interpretation accuracy and contextual understanding of a large-scale language model by training a large-scale language model using a training data set that reflects various visual elements (e.g., axis labels, legends, color information, etc.) contained in a chart image. This allows the large-scale language model to interpret chart images by reflecting not only the numbers contained in the chart image but also its semantic structure, and generate accurate answers to queries based on the interpreted results.

[0257] Meanwhile, the present invention discussed above can be implemented based on a quantum computer. The present invention implemented based on a quantum computer may include a quantum processor and quantum memory based on qubits, and may include software and hardware interfaces optimized for quantum computing.

[0258] Quantum processors in quantum computers utilize qubits (qubits) to efficiently perform complex calculations, such as parallel computation, quantum entanglement, and quantum superposition, that classical computers' binary bits cannot perform. Quantum processors process data using quantum gates, and can provide exponential speedups for certain problems.

[0259] Meanwhile, the present invention discussed above can be implemented as a program that is executed by one or more processes on a computer and can be stored on a medium (or recording medium) that can be read by the computer.

[0260] Furthermore, the present invention discussed above can be implemented as computer-readable code or instructions on a program-recorded medium. In other words, the present invention can be provided in the form of a program.

[0261] Meanwhile, computer-readable media include all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include hard disk drives (HDDs), solid-state disk drives (SSDs), silicon disk drives (SDDs), ROMs, RAMs, CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices.

[0262] Furthermore, the computer-readable medium may include a storage device and may be a server or cloud storage device accessible via communication. In this case, the computer may download the program according to the present invention from the server or cloud storage device via wired or wireless communication.

[0263] A computer program can reach the system (100) via a variety of suitable transmission mechanisms. The transmission mechanism may be, for example, a computer-readable storage medium, a computer program product, a memory device, a recording medium such as a CD-ROM or DVD, or a product tangibly embodying a computer program. The transmission mechanism may be a signal configured to reliably transmit a computer program over air or through an electrical connection. The system (100) may propagate or transmit a computer program as a computer data signal.

[0264] Furthermore, references to 'computer-readable storage media', 'computer program product', 'a computer program tangibly embodied', or the like, or to 'controller', 'computer', 'processor', etc., should be understood to include computers having various architectures, such as single / multiprocessor architectures and sequential (Von Neumann) / parallel architectures, as well as specialized circuits such as field-programmable gate arrays (FPGAs), application-specific circuits (ASICs), signal processing devices, and other devices. References to computer programs, instructions, code, etc., should be understood to include software for programmable processors or firmware, such as the programmable contents of hardware devices, whether instructions for a processor or configuration settings for fixed-function devices, gate arrays, or programmable logic devices.

[0265] Furthermore, in the present invention, the computer described above is an electronic device equipped with a processor, i.e., a CPU (Central Processing Unit), and there is no particular limitation on its type.

[0266] Meanwhile, the above detailed description should not be construed as limiting in any respect and should be considered illustrative. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are intended to be included within the scope of the present invention.< / text> < / chart>

Claims

1. A computerized method comprising: A step of receiving a document including multiple modalities having different data characteristics; A step of extracting data from each of the plurality of modalities through different feature extraction methods based on data characteristics of each of the plurality of modalities; A step of receiving a user query for the document from a user terminal; A step of specifying at least one specific modality associated with the user query among the plurality of modalities constituting the document; and An answer generation method comprising a step of generating an answer to the user query using specific data extracted from the specific modality.

2. In paragraph 1, A method for generating an answer, characterized in that it further comprises a step of classifying each of the plurality of modalities according to data characteristics of each of the plurality of modalities in order to extract data from each of the plurality of modalities included in the document.

3. In paragraph 2, In the above classification step, Processing the document including the above multiple modalities as input to the classification module, An answer generation method characterized in that, in the above classification module, each of the plurality of modalities is classified according to the data characteristics of each of the plurality of modalities.

4. In paragraph 3, In the above extraction step, Based on the data characteristics of each of the plurality of modalities classified from the above classification module, data is extracted from each of the plurality of classified modalities through the different feature extraction methods, The above classified multiple modalities are: A method for generating an answer, characterized in that it includes at least one of a first modality, a second modality, and a third modality.

5. In paragraph 4, Further comprising a step of processing user queries for data and documents extracted from each of the above-mentioned classified multiple modalities as input to a pre-trained large-scale language model (LLM), The above different feature extraction methods are, A first feature extraction method specialized for the first modality, A second feature extraction method specialized for the second modality and A method for generating an answer, characterized in that it includes at least one of the third feature extraction methods specialized for the third modality.

6. In paragraph 5, Data extracted from each of the above classified multiple modalities are: First specific data extracted from the first modality through the first feature extraction method, Second specific data extracted from the second modality through the first feature extraction method and An answer generation method characterized in that it includes at least one of the third specific data extracted from the third modality through the third feature extraction method.

7. In paragraph 5, The above large-scale language model is, Specifying at least one specific modality among the multiple modalities that constitute the above document and that is associated with the user query and to be used to generate the answer; An answer generation method characterized in that an answer to the user query is generated by using specific data extracted from the specific modality among the data extracted from each of the classified multiple modalities.

8. In paragraph 1, The above user queries include different types of user queries, The above different types of user queries are: A method for generating an answer, characterized in that it includes a first type of user query and a second type of user query.

9. In paragraph 8, When a user query of the first type is received among the different types of user queries, The specific modality includes any one of the plurality of modalities associated with the first type of user query, When a user query of the second type is received among the different types of user queries, A method for generating an answer, wherein the specific modality comprises a plurality of specific modalities associated with the second type of user query among the plurality of modalities.

10. In paragraph 9, When the first type of user query is received, Among the plurality of modalities constituting the document, specifying a specific modality associated with the first type of user query and to be used for generating the answer; An answer generation method characterized in that an answer to the user query is generated using specific data extracted from any one of the above specific modalities.

11. In paragraph 9, When the second type of user query is received, Among the plurality of modalities constituting the above document, the plurality of specific modalities are associated with the second type of user query and are used to generate the answer, An answer generation method characterized in that it generates an answer to the user query by performing inference between data extracted from each of the plurality of specific modalities.

12. In paragraph 7, The above large-scale language model is, An answer generation method characterized in that it is trained with a plurality of different learning data sets to understand the plurality of modalities and generate an answer to the user query from the document including the plurality of modalities.

13. In paragraph 12, The above multiple different learning data sets include a first learning data set and a second learning data set, The above first learning data set is, An answer generating method characterized by including at least one of a first sub-learning data set and a second sub-learning data set.

14. In paragraph 13, The first sub-learning data set includes instruction data for learning of multiple categories including images, charts, and text, The above second sub-learning data set is, A method for generating an answer, characterized in that it is constructed based on a chart image and a table, taking into consideration visual information included in the chart image and numerical values ​​of the table.

15. In paragraph 14, The above second sub-learning data set is, An answer generation method characterized by including at least one of chart summarization data, open-ended QA data, and reasoning QA data.

16. In paragraph 13, A method for generating an answer, characterized in that the large-scale language model is trained to understand the plurality of modalities using the first learning data set among the plurality of different learning data sets.

17. In paragraph 16, The large-scale language model trained with the first learning data set is, An answer generation method characterized in that additional learning is performed using the second learning data set to generate an answer to a first type of user query or an answer to a second type of user query in the document including the plurality of modalities.

18. In paragraph 17, The above second learning data set is, A first type of query and a response to the first type of query for a multi-modal context form a pair, A method for generating an answer, characterized in that a second type of query for the above multi-modal context and a response to the second type of query are constructed to form a pair.

19. A system comprising a memory configured to store executable instructions, and one or more processors configured to perform operations by executing one or more instructions, The above system, Receive a document containing multiple modalities with different data characteristics, Based on the data characteristics of each of the plurality of modalities, data is extracted from each of the plurality of modalities through different feature extraction methods. Receive a user query for the above document from a user terminal, Specifying at least one specific modality associated with the user query among the plurality of modalities constituting the document, An answer generation system characterized in that it generates an answer to the user query using specific data extracted from the specific modality.

20. A program that is executed by one or more processes in an electronic device and stored in a computer-readable recording medium, The above program is, A step of receiving a document including multiple modalities having different data characteristics; A step of extracting data from each of the plurality of modalities through different feature extraction methods based on data characteristics of each of the plurality of modalities; A step of receiving a user query for the document from a user terminal; A step of specifying at least one specific modality associated with the user query among the plurality of modalities constituting the document; and A program stored on a computer-readable recording medium, characterized in that it includes commands for performing a step of generating an answer to the user query using specific data extracted from the specific modality.

Images