Method and System of Multimodal Document-based Retrieval and Response Using Process-unit Indexing

Abstract

The present invention relates to a multimodal document-based search and response method and system using process-unit indexing, and more specifically, to a multimodal document-based search and response method and system using process-unit indexing that utilizes a Vision-Language Model (VLM) to analyze the structural association between text and visual information within a multimodal document and indexes it into process units including specific topics and procedures to construct a Retrieval-Augmented Generation (RAG) database, thereby generating an optimal answer to a user's question that includes visual evidence images of related business procedures.

Description

Technology Field

[0001] The present invention relates to a multimodal document-based search and response method and system using process-unit indexing, and more specifically, to a multimodal document-based search and response method and system using process-unit indexing that utilizes a Vision-Language Model (VLM) to analyze the structural association between text and visual information within a multimodal document and indexes it by process units including specific topics and procedures to construct a Retrieval-Augmented Generation (RAG) database, thereby generating an optimal answer to a user's question that includes visual evidence images of related business procedures. Background Technology

[0003] With the recent rapid advancement of artificial intelligence technology, AI chatbot services utilizing RAG (Retrieval-Augmented Generation) technology are becoming widely adopted across various industries to automate customer interactions and improve work efficiency. RAG technology generates answers by searching for information related to a question within a vast external knowledge base and inputting it into a generative language model. Its strength lies in providing accurate answers based on the latest information or specialized internal data that the model has not yet learned.

[0004] Conventional AI chatbots and RAG systems are primarily designed to be optimized for the search and summarization of text-based documents. These systems divide documents into simple text chunks, embed and store them, and use text fragments with high similarity to the user's question as the basis for the answer. However, the majority of documents, such as corporate work manuals or technical guidelines, take on a multimodal form that combines various visual elements, including images, tables, and graphs, in addition to text.

[0005] Among conventional technologies, there is a method that utilizes text, images, and tables within a PDF document by dividing them through layout parsing and converting them into a Markdown structure for search. Additionally, attempts are being made to improve search performance by assigning weights to keyword-based and semantic-based searches, or to search for reference materials in a database based on sentences interpreting images.

[0006] However, these existing methods perceive documents merely as simple pages or layouts, and have limitations in fully understanding the context of the actual 'business process' that the document intends to describe. For example, in response to a question about the registration process for a specific system, existing systems can find fragmented page information containing relevant keywords, but it is difficult to consistently explain the entire procedure from start to finish with visual screen images.

[0007] Consequently, there is a growing need for a new type of search and response system capable of precisely parsing the structural relationships of information in multimodal documents containing a mixture of text and images, and reorganizing and indexing this information according to 'processes,' which are the actual units of work performed by users. Prior art literature

[0009] Korean Registered Patent KR 10-2909344 B1 “Method and System for Providing RAG-Based AI Question Answer Service” (2026.01.05.) The problem to be solved

[0010] The present invention aims to provide a multimodal document-based search and response method and system utilizing process-unit indexing, and more specifically, to provide a multimodal document-based search and response method and system utilizing process-unit indexing capable of generating an optimal answer including visual evidence images of relevant business procedures to a user's question by utilizing a Vision-Language Model (VLM) to analyze the structural association between text and visual information within a multimodal document and indexing it into process units including specific topics and procedures to construct a Retrieval-Augmented Generation (RAG) database. means of solving the problem

[0012] To solve the above problem, a multimodal document-based search and response method using process-unit indexing is performed on a server system comprising one or more processors and one or more memories, wherein the server system includes a plurality of embedding vectors generated for each document component comprising one or more of text, images, graphs, and tables included in a specific page of a multimodal document; process information including an identifier, a start index, and an end index for the process of the corresponding page determined based on one or more of the metadata and table of contents information of the multimodal document; and a page image capturing the entire corresponding page; wherein the RAG (Retrieval-Augmented Generation) database is stored for each of the plurality of pages, and the multimodal document-based search and response method comprises a question receiving step of receiving a question query from a user and generating a question embedding vector for the question query; A multimodal document-based search and response method is provided, comprising: a search page group generation step of deriving a plurality of embedding vectors from the RAG database that are greater than or equal to a preset similarity threshold with respect to the question embedding vector, and generating a search page group including a plurality of search pages that include document components corresponding to each of the embedding vectors; a page image group generation step of deriving a process to which the search page belongs based on the process information for each of the plurality of search pages included in the search page group, and deriving a page image group including page images for each of the plurality of pages included in the process; and an answer derivation step of inputting the page image group and the question query into an external or internal VLM (Vision-Language Model) to derive an answer to the question query.

[0013] In one embodiment of the present invention, the embedding vector may be stored in the RAG database by performing: a text information extraction step for extracting text information included in each of a plurality of pages included in the multimodal document; a document component derivation step for deriving one or more document components including text, images, graphs, and tables by inputting the page image of the corresponding page and the extracted text information of the corresponding page into the VLM for each of the plurality of pages; a document component information generation step for generating document component information including one or more of the type, content, and location coordinates of the corresponding document component for each of the derived document components; and an embedding vector generation step for generating the embedding vector based on the document component information and storing it in the RAG database.

[0014] In one embodiment of the present invention, the process information may be stored in the RAG database by performing: a process derivation step of deriving one or more processes included in the multimodal document based on one or more of the table of contents information and metadata of the multimodal document; a process identification step of identifying which process a page corresponds to based on the document components included in each page of the multimodal document; and a process information generation step of generating process information including one or more of an identifier, a start index, and an end index for each of the processes and storing it in the RAG database.

[0015] In one embodiment of the present invention, the step of generating a search page group may further include: generating the search page group, and for each of the plurality of search pages included in the search page group, calculating a similarity between an embedding vector corresponding to each of one or more document components included in the search page and a question embedding vector, and calculating a maximum similarity which is the maximum value of the similarity and an average similarity which is the average value; calculating an integrated similarity score for each of the plurality of search pages by applying a preset weight to the maximum similarity and average similarity calculated for each of the plurality of search pages; and reconstructing the search page group by excluding from the search page group any search pages among the plurality of search pages whose integrated similarity score is less than a preset threshold value.

[0016] In one embodiment of the present invention, each of the plurality of search pages included in the search page group belongs to a process including one or more of a cover, a table of contents, a main text, and an appendix, and the search page group creation step may further include a search page group reconstruction step, wherein, after the search page group is created, if the processes for each of the plurality of search pages included in the search page group are not unified into one process, the search page group reconstruction step provides a user with a list of one or more processes belonging to the search page group among the plurality of processes, asks the user to query which process corresponds to the user's intention, and based on the user's response to the query, reconstructs the search page group while excluding the search page corresponding to the process that does not correspond to the user's intention from the search page group.

[0017] In one embodiment of the present invention, the answer derivation step may further include: for each of a plurality of page images included in the page image group, a step of deriving a non-text element among the document components excluding text, wherein the embedding vector corresponding to the corresponding document component has a similarity of greater than or equal to a preset similarity threshold with respect to the question embedding vector; a step of generating a guide image by cropping or highlighting an area corresponding to the non-text element in the page image based on the position coordinates of the non-text element; and an answer provision step of providing the answer derived for the question query and the guide image to a user.

[0018] In one embodiment of the present invention, the interface provided through the answer derivation step comprises: an answer display layer including an answer derived for the question query; and a related image display layer displaying all or part of the page image that served as the basis for the answer among a plurality of page images included in the page image group; wherein the related image display layer can visually display a guide image cropped or highlighted in an area corresponding to a non-text element, which is a document component that serves as the basis for the answer in the page image.

[0019] To solve the above problem, a multimodal document-based search and response system using process-unit indexing comprising one or more processors and one or more memories, wherein a plurality of embedding vectors generated for each document component including one or more of text, images, graphs, and tables included in a specific page of a multimodal document; process information including an identifier, a start index, and an end index for the process of the corresponding page determined based on one or more of the metadata and table of contents information of the multimodal document; and a page image capturing the entire corresponding page; are stored for each of the plurality of pages in a RAG (Retrieval-Augmented Generation) database; a question receiving unit that receives a question query from a user and generates a question embedding vector for the question query; and a search page group generating unit that derives a plurality of embedding vectors from the RAG database that are equal to or greater than a preset similarity threshold with respect to the question embedding vector, and generates a search page group comprising a plurality of search pages including document components corresponding to each of the said embedding vectors. A multimodal document-based search and response system is provided, comprising: a page image group generation unit that, for each of a plurality of search pages included in the search page group, derives a process to which the search page belongs based on the process information and derives a page image group including page images for each of the plurality of pages included in the process; and an answer derivation unit that inputs the page image group and the question query into an external or internal VLM (Vision-Language Model) to derive an answer to the question query. Effects of the invention

[0021] According to one embodiment of the present invention, by indexing and searching documents in process units including specific topics and procedures rather than in simple page units, it is possible to extract information based on the context of the work that matches the user's question intent and provide an answer.

[0022] According to one embodiment of the present invention, by analyzing the positional and structural associations of text and visual elements within a document through VLM and embedding them, it is possible to precisely understand complex table or image information included in multimodal documents and achieve the effect of enhancing search performance.

[0023] According to one embodiment of the present invention, by inputting text information and page images together into a VLM and parsing them, the hierarchical structure and coordinate information between text and visual elements within an unstructured document can be preserved as structured data, thereby maximizing the precision of the database.

[0024] According to one embodiment of the present invention, by calculating an integrated similarity score based on the maximum and average values ​​for the embedding similarity of the searched pages, it is possible to achieve the effect of precisely selecting only the search results that are most closely related to the question.

[0025] According to one embodiment of the present invention, when the searched process is multiple or unclear, by performing an additional query on the user and reflecting the feedback, it is possible to eliminate ambiguity in the answer and ensure consistency in the response.

[0026] According to one embodiment of the present invention, by applying an algorithm that quantitatively calculates the distribution ratio by process included within a searched page group and compares it with a preset standard ratio, the system can objectively identify ambiguous situations requiring user feedback and achieve the effect of automatically controlling the reliability of search results.

[0027] According to one embodiment of the present invention, by generating and providing a guide image in which non-text elements serving as the basis for an answer are cropped or highlighted based on coordinates, it is possible to enhance the user's visual understanding and maximize convenience during actual work performance.

[0028] According to one embodiment of the present invention, by providing a multilayer interface including an answer display layer and a related image display layer, it is possible to achieve the effect of enabling intuitive verification of text answers and visual evidence simultaneously.

[0029] According to one embodiment of the present invention, by combining a data structure at the business process unit level with a VLM-based generation capability, it is possible to achieve the effect of implementing a high-performance AI professional consultation service that is differentiated from existing fragmentary text search chatbots.

[0030] According to one embodiment of the present invention, by guiding the actual system screen image and the step-by-step execution method in conjunction, it is possible to reduce training costs for new users or workers performing complex tasks and improve work accuracy. Brief explanation of the drawing

[0032] FIG. 1 illustrates a server system that performs a multimodal document-based search and response method using process unit indexing according to an embodiment of the present invention. FIG. 2 illustrates detailed steps for generating and storing embedding vectors and process information for constructing a RAG database according to an embodiment of the present invention. FIG. 3 illustrates the process of storing an embedding vector in a RAG database according to one embodiment of the present invention. FIG. 4 illustrates the process of storing process information in a RAG database according to one embodiment of the present invention. FIG. 5 illustrates the process of storing a page image in a RAG database according to one embodiment of the present invention. FIG. 6 illustrates the process of performing a question receiving step, a search page group creation step, a page image group creation step, and an answer derivation step for responding to a user question according to an embodiment of the present invention. FIG. 7 illustrates the process of performing a search page group creation step according to an embodiment of the present invention. FIG. 8 illustrates the process of performing a page image group generation step according to an embodiment of the present invention. FIG. 9 illustrates the process of performing the answer derivation step according to one embodiment of the present invention. FIG. 10 illustrates a step of calculating maximum similarity and average similarity for each document component within a search page in a search page group generation step according to an embodiment of the present invention. FIG. 11 illustrates the step of calculating an integrated similarity score and reorganizing a search page group in the search page group generation step according to an embodiment of the present invention. FIG. 12 illustrates a step of calculating the process ratio included in a search page group according to an embodiment of the present invention. FIG. 13 illustrates the process of reorganizing a search page group based on user feedback according to an embodiment of the present invention. FIG. 14 illustrates the process of generating a guide image according to one embodiment of the present invention. FIG. 15 illustrates an example of an interface provided through an answer-providing step according to an embodiment of the present invention. FIG. 16 schematically illustrates the internal configuration of a computing device according to one embodiment of the present invention. Specific details for implementing the invention

[0033] Hereinafter, various embodiments and / or aspects are disclosed with reference to the drawings. For illustrative purposes, numerous specific details are disclosed in the following description to aid in a general understanding of one or more aspects. However, it will also be recognized by those skilled in the art that these aspects may be practiced without such specific details. The following description and the accompanying drawings describe specific exemplary aspects of one or more aspects in detail. However, these aspects are exemplary, and some of the various methods in the principles of the various aspects may be used, and the description is intended to include all such aspects and their equivalents.

[0035] In addition, various aspects and features will be presented by a system that may include a number of devices, components and / or modules, etc. It should also be understood and recognized that various systems may include additional devices, components and / or modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in relation to the drawings.

[0036] Terms such as "examples," "examples," "aspects," and "examples" as used herein may not be interpreted as implying that any aspect or design described is superior or more advantageous than other aspects or designs. Terms used below, such as "part," "component," "module," "system," and "interface," generally refer to computer-related entities and may refer, for example, to hardware, a combination of hardware and software, or software.

[0037] Additionally, the terms “comprising” and / or “comprising” should be understood to mean that the relevant feature and / or component is present, but not to exclude the presence or addition of one or more other features, components and / or groups thereof.

[0038] Additionally, terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0039] Furthermore, in the embodiments of the present invention, all terms used herein, including technical or scientific terms, unless otherwise defined, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in the embodiments of the present invention.

[0041] FIG. 1 illustrates a server system that performs a multimodal document-based search and response method using process unit indexing according to an embodiment of the present invention.

[0042] A multimodal document-based search and response method using process-unit indexing, performed in a server system comprising one or more processors and one or more memories according to an embodiment of the present invention, wherein the server system comprises a plurality of embedding vectors generated for each document component including one or more of text, images, graphs, and tables included in a specific page of a multimodal document; process information including an identifier, a start index, and an end index for a process of the corresponding page determined based on one or more of metadata and table of contents information of the multimodal document; and a page image capturing the entire corresponding page; wherein the RAG (Retrieval-Augmented Generation) database is stored for each of the plurality of pages, and the multimodal document-based search and response method comprises a question receiving step of receiving a question query from a user and generating a question embedding vector for the question query; A multimodal document-based search and response method can be performed, comprising: a search page group generation step of deriving a plurality of embedding vectors from the RAG database that are greater than or equal to a preset similarity threshold with respect to the question embedding vector, and generating a search page group including a plurality of search pages that include document components corresponding to each of the embedding vectors; a page image group generation step of deriving a process to which the search page belongs based on the process information for each of the plurality of search pages included in the search page group, and deriving a page image group including page images for each of the plurality of pages included in the process; and an answer derivation step of inputting the page image group and the question query into an external or internal VLM (Vision-Language Model) to derive an answer to the question query.

[0044] In the present invention, the multimodal document refers to a document in which one or more visual information, such as images, graphs, diagrams, and images, are mixed and included within a single page, in addition to simple text strings. For example, it may correspond to a document in which each step of a task is described in text, such as a method for registering new employees of a specific system or a facility management operation manual, while simultaneously organically combining control screens (UI screenshots) of actual software where the corresponding step is performed, flowcharts illustrating the task processing flow, and statistical tables showing the status of the managed subject in numerical form.

[0045] In the aforementioned multimodal document, individual document components do not exist independently of one another, but can have meaningful structural relationships, such as text describing the location or content of a visual image or visual elements providing the specific execution basis for the text.

[0047] As illustrated in FIG. 1, the server system (1) of the present invention may be connected to a user terminal (2) and a Vision-Language Model (VLM) (3) to transmit and receive data through a network. The server system (1) may include the VLM (3) itself or be connected to communicate with an externally established VLM (3). Additionally, the server system (1) may correspond to a computing readable recording medium comprising one or more processors and one or more memories.

[0049] The above user terminal (2) is a communication device used by a user to input a question query related to work or to check a final answer, and may include various devices capable of connecting to a wired or wireless network, such as a smartphone, tablet PC, or desktop.

[0050] The internal or external VLM (3) is a multimodal artificial intelligence model capable of processing text information and image information simultaneously, and in the answer generation step described later, it receives a page image group and a question query and generates an answer to the user's question based thereon.

[0052] As illustrated in FIG. 1, the server system (1) may include a question-response control unit (10) for deriving answers to user questions and a RAG database management unit (11) for managing the RAG database (12).

[0053] The above question-and-answer control unit (10) controls the process from receiving a question from a user to deriving a final answer, and may include a question receiving unit (100), a search page group generating unit (101), a page image group generating unit (102), and an answer deriving unit (103).

[0055] The above RAG database (12) stores embedding vectors, process information, and page images for each page of a multimodal document, and the above RAG database management unit (11) can manage the creation, storage, and input / output of the data.

[0056] Specifically, the RAG database (12) can store, for each page of a multimodal document, an embedding vector generated by extracting type, content, and location coordinate information for one or more document components including text, images, graphs, and tables included in the page, process information defining the business procedure to which the page belongs, and a page image which is an image of the entire page.

[0058] The process defined in the present invention may correspond to a logical grouping of information having individual topics or procedures within the overall information structure contained in a multimodal document. This corresponds to a structured category that enables a server system to derive specific information by preserving the overall context and flow of knowledge originally intended by the multimodal document, rather than fragmentary pieces of data, when searching for such information; it is a unit that defines the sequential relationships and cross-reference relationships of information as a logical scope.

[0059] For example, a set of pages with individual purposes and a complete flow, such as a method for creating an account in a specific system, a hardware failure guide, or an approval procedure for a specific task, can constitute a single process unit.

[0061] The data stored in the RAG database (12) does not necessarily have to be combined and stored as a single record or a single storage format, and even information extracted from each page can be stored in the database as independent fields or separate data collections. That is, even if the embedding vector, the process information, and the page image are distributed and stored in different storage formats or tables within the RAG database (12), they can be managed by linking them so that they can be mutually referenced through each page or element identifier.

[0063] The above question receiving unit (100) can perform a question receiving step (S100) of receiving a question query in the form of natural language from a user terminal (2) and generating a question embedding vector quantified through a preset embedding model.

[0064] The search page group generation unit (101) can perform a search page group generation step (S101) of calculating the similarity between the question embedding vector and a plurality of embedding vectors stored in the RAG database (12) and selecting a search page containing an embedding vector having a similarity higher than a reference value. Specifically, among the embedding vectors of individual document components stored in the RAG database (12), one or more embedding vectors that are equal to or greater than a preset similarity reference value with the question embedding vector can be derived, and one or more search pages containing document components corresponding to the embedding vectors can be selected to generate a search page group.

[0065] The above page image group generation unit (102) can identify the process information of each of the one or more pages included in the search page group, and identify the start index to the end index of the process to which the page belongs. Subsequently, by deriving page images of all pages included in each of the identified processes from the RAG database (12), a page image group including multiple page images including processes related to the user's question can be derived.

[0066] The above answer derivation unit (103) can perform an answer derivation step of generating a final answer by inputting the derived page image group and the user's question query into the above VLM (3). The above VLM (3) derives a specific method for performing the question by directly referencing the visual flow within the provided image data, and the above answer derivation unit (103) can transmit the generated answer to the user terminal (2) to simultaneously provide the user with visual evidence and text explanation.

[0068] To summarize, when the question receiving unit (100) receives a question query from the user terminal (2) and generates a question embedding vector, the search page group generating unit (101) can generate a search page group including selected search pages based on the similarity between a plurality of embedding vectors stored in the RAG database (12) and the question embedding vector. Subsequently, the page image group generating unit (102) can identify the process to which the selected search pages belong based on the process information, and form a page image group by calling the page images of all pages included in the corresponding process range (start index to end index). The answer derivation unit (103) inputs the generated page image group and the question query into the VLM (3), and can generate an answer to the question query and provide it to the user.

[0070] The pre-configured embedding model used in the present invention may be one or more of multimodal embedding models capable of projecting semantic features of text and visual information into a vector space, such as BERT (Bidirectional Encoder Representations from Transformers), OpenAI Embedding model, or Cohere Embedding model.

[0072] According to one embodiment of the present invention, by indexing and searching documents in process units including specific topics and procedures rather than in simple page units, it is possible to extract information based on the context of the work that matches the user's question intent and provide an answer.

[0074] FIG. 2 illustrates detailed steps for generating and storing embedding vectors and process information for constructing a RAG database according to an embodiment of the present invention.

[0075] According to one embodiment of the present invention, the embedding vector may be stored in the RAG database by performing: a text information extraction step for extracting text information included in each of a plurality of pages included in the multimodal document; a document component derivation step for deriving one or more document components including text, images, graphs, and tables by inputting the page image of the corresponding page and the extracted text information of the corresponding page into the VLM for each of the plurality of pages; a document component information generation step for generating document component information including one or more of the type, content, and location coordinates of the corresponding document component for each of the derived document components; and an embedding vector generation step for generating the embedding vector based on the document component information and storing it in the RAG database.

[0076] In addition, according to one embodiment of the present invention, the process information may be stored in the RAG database by performing: a process derivation step of deriving one or more processes included in the multimodal document based on one or more of the table of contents information and metadata of the multimodal document; a process identification step of identifying which process a page corresponds to based on the document components included in each page of the multimodal document; and a process information generation step of generating process information including one or more of an identifier, a start index, and an end index for each of the processes and storing it in the RAG database.

[0078] Figure 2(a) illustrates the process of generating an embedding vector based on the document components of a multimodal document through the RAG database management unit (11) and storing it in the RAG database (12).

[0080] First, the RAG database management unit (11) can extract text information containing only text, i.e., strings, from each page of the multimodal document in the text information extraction step (S1100). The text information extraction step (S1100) can be performed by directly extracting text from the text layer of the multimodal document, or by applying an optical character recognition (OCR) algorithm to detect text areas included within an image-shaped document and converting them into strings.

[0082] Subsequently, in the document component derivation step (S1101), the RAG database management unit (11) can input the text information extracted in the text information extraction step (S1100) and the page image of the page from which the text information was extracted into an internal or external VLM. Based on the input text (text information) and image (page image) data, the VLM (3) identifies which text content corresponds to a specific area within the image, and through this, can derive one or more document components including text, tables, images, graphs, and charts within the input page image (preferably the page corresponding to the page image).

[0083] In the subsequent document component information generation step (S1102), the VLM can generate document component information for each of the one or more document components derived in the document component derivation step (S1101), including a type defining what kind of document component the document component is, content describing semantic features or specific text data included in the document component, and location coordinates where the document component exists on the page image.

[0085] Finally, in the embedding vector generation step (S1103), the RAG database management unit (11) inputs the document component information generated in the document component information generation step (S1102) into a preset embedding model to generate an embedding vector corresponding to each document component information, and can store the embedding vector in the RAG database.

[0087] Figure 2(b) illustrates the detailed steps for the RAG database management unit (11) to generate process information in order to manage documents in logical configuration units, such as process units.

[0088] In the present invention, the process refers to a series of logical flows extending from the start to the end of a specific topic or procedure within a multimodal document, or a bundle of information including independent categories.

[0089] For example, in a user manual, a set of pages allocated to perform distinct purposes, such as new user registration, password reset, and system log verification, constitutes a single process unit. In other words, the aforementioned process may correspond to a type of category defined based on the contextual completeness of the subject and procedure that the multimodal document intends to convey.

[0091] The above RAG database management unit (11) can perform a process derivation step (S1110) to derive one or more processes that constitute the entire document based on the table of contents information and metadata of the multimodal document.

[0092] As a specific embodiment, the RAG database management unit (11) can identify a specific page containing table of contents information based on multiple page images or text information included in the multimodal document. The page containing table of contents information can be identified by searching whether a pre-set identification keyword including table of contents, contents, table of contents, or index exists in each of the multiple pages included in the multimodal document, or by detecting a specific layout structure in which text and numbers are repeatedly arranged within the page and the topic and page number are connected. Additionally, the RAG database management unit (11) can identify a page containing table of contents information by inputting each page image into the VLM (3) and checking whether the category of the corresponding page is classified as a table of contents.

[0093] The above RAG database management unit (11) can derive one or more processes by extracting subject names, section divisions, and corresponding page number information written in a page containing identified table of contents information.

[0094] In another embodiment, the RAG database management unit (11) inputs the entire page image of the multimodal document into the VLM (3) to identify changes in the visual layout and semantic flow of text throughout the multimodal document and derive one or more processes divided into logically completed subject units.

[0096] Next, the RAG database management unit (11) can determine which process a page belongs to by comparing the document components included in each page with the derived process through the process identification step (S1111).

[0097] Specifically, the RAG database management unit (11) can identify which process a page belongs to based on document component information, such as a title, body keywords, or images included in the page, which are included in one or more document components of a specific page.

[0098] In this case, as an example, the RAG database management unit (11) inputs the entire content of a specific page of a multimodal document (preferably text information and page image) into the VLM (3), and the VLM (3) can determine the process to which the page belongs among the identified processes based on the contextual meaning and visual composition of the page.

[0099] In another embodiment, if the table of contents information and category of a page are specified in all pages or specific areas of the multimodal document, for example, if the category or procedure name of a page is placed in the form of text or an icon in the upper left area of ​​each page of the multimodal document, the RAG database management unit (11) can identify the process to which the page belongs by first referring to the document component information at that location.

[0101] Finally, the RAG database management unit (11) can perform a process information generation step (S1112) to generate process information for each of the one or more processes, including a unique identifier, a start index including the number of the page where the process starts, and an end index including the number of the page where the process ends, and store the process information in the RAG database.

[0103] According to one embodiment of the present invention, by inputting text information and page images together into a VLM and analyzing the structural relationships between components within a document to extract the type, content, and coordinate information of individual elements, the hierarchical structure and coordinate information between text and visual elements within an unstructured document can be preserved as structured data, thereby maximizing the precision of the database.

[0105] FIG. 3 illustrates the process of storing an embedding vector in a RAG database according to one embodiment of the present invention.

[0106] Referring to Fig. 3(a), the RAG database management unit (11) can perform a text information extraction step (S1100) to obtain text information, which is string data, from individual pages of the multimodal document.

[0107] The above RAG database management unit (11) analyzes the data format of a multimodal document, and in the case of a document with a text layer, it can directly parse and extract text data from the corresponding layer.

[0108] If the original document is a scanned image or a form in which text extraction is restricted, the RAG database management unit (11) can detect character regions within the image using a preset optical character recognition (OCR) algorithm and convert them into computer-recognizable text information to separate them.

[0110] Referring to FIG. 3(b), the RAG database management unit (11) can input text information extracted in the text information extraction step (S1100) and a page image capturing the entire page into an internal or external VLM (3). The VLM (3) can calculate the correlation between the input text information and the page image and, based on this, derive the document components included in the page. The document components may include not only body text but also tables with numerical values ​​and structures, photographs or diagrammatic images, graphs and charts visualizing statistical figures, etc.

[0111] In the example of Fig. 3(b), the VLM (3) derives document components such as text 1, text 2, text 3, text 4, and image 1 based on input page image and text information.

[0112] Next, the above VLM (3) can generate and output document component information, which is structured data for each derived document component. In conclusion, the above VLM (3) can receive text information and page images as input and output document component information.

[0113] The above document component information may include type information defining whether the element is text, a table, or an image, content information including a semantic description of the actual text content or image of the element, and location coordinate information of the area occupied by the element on the page image.

[0115] The document component information output by the above VLM (3) is transmitted to the RAG database management unit (11), and the RAG database management unit (11) can generate an embedding vector by inputting the document component information into a preset embedding model. The generated embedding vector can finally be stored in the RAG database (12).

[0117] FIG. 4 illustrates the process of storing process information in a RAG database according to one embodiment of the present invention.

[0118] Referring to FIG. 4, the left side of FIG. 4 illustrates a process derivation step (S1110) and a process identification step (S1111) in which the flow of the entire multimodal document is identified to distinguish processes, and each of the multiple pages included in the multimodal document is determined to belong to one of the identified processes. The right side of FIG. 4 illustrates a process information generation step (S1112) in which process information including an identifier, a start index, and an end index is generated based on the identified processes and multiple pages, and this is stored in the RAG database (12).

[0120] In the example illustrated in FIG. 4, as a result of performing the process derivation step (S1110) and the process identification step (S1111) on a multimodal document (PDF), page 1 is identified as 'Cover Page', pages 2, 6, and 8 as 'Table of Contents', pages 3, 4, and 5 as '1.1 insite Operation Process', page 7 as '1.2 insite Training', and page 9 as '1.3 User Support'.

[0121] In other words, the processes derived from the example are ‘Cover Page’, ‘Table of Contents’, ‘1.1 insite Operation Process’, ‘1.2 insite Training’, and ‘1.3 User Support’, and the pages (preferably page numbers) corresponding to each process are identified as 1 for ‘Cover Page’, 2, 6, and 8 for ‘Table of Contents’, 3, 4, and 5 for ‘1.1 insite Operation Process’, 7 for ‘1.2 insite Training’, and 9 for ‘1.3 User Support’.

[0122] Accordingly, a process information generation step (S1112) can be performed to generate process information and store it in the RAG database (12). As illustrated in the example in FIG. 4, in the case of a process named '1.1 insite operation Process', the identifier can be set to '1.1 insite operation Process', the start index, which is the point where the process starts, can be set to 3, and the end index, which is the point where the process ends, can be set to 5 to generate process information and store the process information in the RAG database.

[0124] FIG. 5 illustrates the process of storing a page image in a RAG database according to one embodiment of the present invention.

[0125] The above RAG database management unit (11) can perform the process of generating multiple page images as original data to provide answers to questions and visual grounds, along with text-based embedding vectors and logical structure process information extracted from the above multimodal document, and storing them in the above RAG database (12).

[0127] The above RAG database management unit (11) can generate a page image by capturing or converting the entire page into an image form to preserve the visual information of each page. The generated page image may include all visual context conveyed by the page in its original form, such as the arrangement of text included in the document, the location of images, and the structure of graphs and tables.

[0128] Subsequently, the RAG database management unit (11) can store a plurality of generated page images in the RAG database (12). Within the RAG database (12), each page image can be stored in association with an embedding vector extracted from the corresponding page and process information defining the process to which the corresponding page belongs.

[0130] The page image saved in this way can be input into the VLM (3) along with the question query in the subsequent answer derivation step (S103), and can be used as actual supporting data that enables the VLM (3) to generate an accurate answer to the user's question by directly referencing not only the text of the document but also the visual components.

[0132] According to one embodiment of the present invention, by storing and managing page images captured from the entire page in a RAG database, the limitations of simple text search can be overcome, and the multimodal model can fully understand the visual context of the document to derive an answer, thereby achieving an effect that supports the multimodal model.

[0134] FIG. 6 illustrates the process of performing a question receiving step, a search page group creation step, a page image group creation step, and an answer derivation step for responding to a user question according to an embodiment of the present invention.

[0136] The above server system (1) can perform information retrieval and answer derivation based on embedding vectors, process information, and page images stored in the RAG database (12) in response to a user's question.

[0138] In the above question reception step (S100), the question receiving unit (100) of the question-response control unit (10) can receive a question query in natural language form from the user terminal (2). The question receiving unit (100) can input the question query into a preset embedding model to generate a question embedding vector.

[0140] In the above search page group generation step (S101), the search page group generation unit (101) of the question-response control unit (10) can calculate the similarity between the question embedding vector and a plurality of embedding vectors stored in the RAG database (12). The search page generation unit (101) can derive one or more embedding vectors that are greater than or equal to a preset similarity threshold with respect to the question embedding vector by utilizing indicators such as cosine similarity or Euclidean distance. Subsequently, document component information corresponding to each derived embedding vector is identified, and search pages containing the corresponding document components are selected to generate a search page group containing a plurality of search pages.

[0141] Furthermore, when the search page group generation unit (101) generates a search page group, it can generate the search page group by selecting a predetermined number of search pages containing document components corresponding to the embedding vector in order of high similarity between the question embedding vector and the embedding vector.

[0142] At this time, when multiple embedding vectors for multiple document components existing within a single page of a multimodal document are derived at or above the similarity threshold, the search page group generation unit (101) can perform filtering to remove duplicate pages, thereby selecting a preset number of search pages with high similarity based on different unique pages and forming a search page group based thereon.

[0144] In the above page image group generation step (S102), the page image group generation unit (102) of the question and answer control unit (10) does not simply provide page images of multiple search pages included in the search page group, but can derive page images in units of the 'process' to which the corresponding page belongs.

[0145] Specifically, the page image group generation unit (102) can query process information stored in the RAG database (12) for each search page included in the search page group. After deriving the identifier, start index, and end index of the process to which the page belongs through the process information, the page images of all pages included in the range from the start index to the end index are retrieved from the RAG database (12), and a page image group including the retrieved multiple page images can be generated.

[0146] This prevents information fragmentation by preserving the context of the retrieved information (e.g., the entire screenshot flow from the preparation stage to post-processing of a specific task) and enables the VLM receiving the query to grasp the entire flow of the process.

[0148] Finally, in the answer derivation step (S103), the answer derivation unit (103) of the question-response control unit (10) inputs the page image group and the user's question query into the VLM (3) to derive an answer to the question query. The derived answer is composed in the form of natural language and transmitted to the user terminal (2), and the user can visually check the page images that served as the basis for the answer along with the answer.

[0150] According to one embodiment of the present invention, by providing the visual data (page image group) of the entire process to which the page belongs to the VLM, rather than merely finding fragmentary pages similar to the question, it is possible to generate an answer that does not break the context, even for complex business procedures or layout-based information.

[0152] FIG. 7 illustrates the process of performing a search page group creation step according to an embodiment of the present invention.

[0153] In the above search page group generation step (S101), the server system (1) can select a search page containing information that best matches the user's question intent based on the embedding vector of the RAG database (12) and generate a search page group containing the selected search page.

[0155] FIG. 7(a) illustrates a process of calculating similarity between a question embedding vector, which is a converted question query received from a user generated in the question receiving step (S100), and a plurality of embedding vectors included in the RAG database, deriving an embedding vector whose similarity is greater than or equal to a preset similarity threshold, and deriving a page containing a document component corresponding to the embedding vector.

[0156] In the example of FIG. 7 (a), the search page group generation unit (101) can derive an embedding vector 1 and an embedding vector 3 having a similarity greater than or equal to a similarity threshold with the question embedding vector, and derive pages 10 and 13, which are pages containing document components corresponding to each of embedding vector 1 and embedding vector 3.

[0158] FIG. 7(b) illustrates the process of generating a search page group containing a predetermined number of pages in order of high similarity, based on a plurality of pages derived through the same process as FIG. 7(a).

[0159] In the example of FIG. 7(b), the search page group generation unit (101) can generate a search page group including pages 10, 13, 7, 5, and 4 by selecting a predetermined number (5 in the example) in order of high similarity among the derived pages 10, 13, 7, 5, 4, 8, … as shown on the right side of FIG. 7(b).

[0161] According to one embodiment of the present invention, by comparing the similarity between a question embedding vector and an embedding vector for each document component, search pages most suitable for the question intent are selected in order of similarity, thereby enabling the efficient securing of pages containing key information necessary for generating answers.

[0163] FIG. 8 illustrates the process of performing a page image group generation step according to an embodiment of the present invention.

[0164] In the above page image group creation step (S102), the page image group creation unit (102) of the question-and-answer control unit (10) can query process information stored in the RAG database (12) for each of the plurality of search pages included in the search page group. Specifically, the page image group creation unit (102) can determine which logical flow (process) each identified search page belongs to, and can create a page image group including a plurality of page images selected according to a start index and an end index that define the range of the process.

[0166] FIG. 8 illustrates the process of performing the page image group creation step based on the example of FIG. 7.

[0167] As shown in the example of FIG. 8(a), first, the page image group generation unit (102) can determine which process each page corresponds to for each of the search page group (generated in FIG. 7) including page 10, page 13, page 7, page 5, and page 4. It can be confirmed that pages 10 and 13 correspond to process A, and pages 7, 5, and 4 correspond to process B.

[0169] Next, as shown in the example of FIG. 8(b), the page image group generation unit (102) can retrieve page images for all pages corresponding to ‘Process A’ and ‘Process B’ derived in FIG. 8(a). As a result of querying the process information stored in the RAG database, it can be confirmed that the start index of ‘Process A’ is 10 and the end index is 13, and the start index of ‘Process B’ is 4 and the end index is 7. Accordingly, page images corresponding to pages 10, 11, 12, and 13 and page images corresponding to pages 4, 5, 6, and 7 can be retrieved from the RAG database (12).

[0170] Finally, the page image group generation unit (102) can generate a page image group including a plurality of page images (pages 10-13, pages 4-7) derived based on the search page group.

[0172] FIG. 9 illustrates the process of performing the answer derivation step according to one embodiment of the present invention.

[0173] In the above answer derivation step (S103), the answer derivation unit (103) of the above question-response control unit (10) can input the page image group generated in the page image group creation step (S102) and the user's question query into an internal or external VLM (3) to derive an answer to the question query.

[0175] Referring to FIG. 9, the answer derivation unit (103) can input images of pages 10, 11, 12, and 13 corresponding to 'Process A' and images of pages 4, 5, 6, and 7 corresponding to 'Process B' into the VLM (3) along with the user's question query.

[0176] The above VLM (3) can derive an answer to a user question based on the relationship between the visual information of multiple page images within the page image group and the question query.

[0178] The answer derived as a result of FIG. 9 can be composed in the form of natural language. The answer derivation unit (103) can receive the answer generated from the VLM (3) and transmit it to the user terminal (2). At this time, the answer derivation unit (103) can provide not only the text answer but also the page image of the page image group that served as the actual basis for the answer, and this can have the effect of supporting the user to immediately verify the validity of the answer visually.

[0180] FIG. 10 illustrates a step of calculating maximum similarity and average similarity for each document component within a search page in a search page group generation step according to an embodiment of the present invention.

[0181] According to one embodiment of the present invention, the step of generating a search page group may further include: generating the search page group, and for each of the plurality of search pages included in the search page group, calculating a similarity between an embedding vector corresponding to each of one or more document components included in the search page and a question embedding vector, and calculating a maximum similarity which is the maximum value of the similarity and an average similarity which is the average value; calculating an integrated similarity score for each of the plurality of search pages by applying a preset weight to the maximum similarity and average similarity calculated for each of the plurality of search pages; and reconstructing the search page group by excluding from the search page group any search pages among the plurality of search pages whose integrated similarity score is less than a preset threshold value.

[0183] Referring to FIG. 10(a), a search page group generated through the search page group generation unit (101) described above is shown. As in the example of FIG. 7 above, the search page group may include pages 10, 13, 8, 7, 5, and 4. The search page group generation unit (101) may re-measure the similarity for each of one or more document components within the page to evaluate the quality level of each of the plurality of search pages included in the search page group, and reconstruct the search page group through an integrated similarity score calculated based thereon.

[0185] FIG. 10(b) illustrates a process of calculating the similarity between each of the multiple document components included in page 10 and the question embedding vector, using page 10 as an example among the multiple search pages included in the search page group, and finally calculating the integrated similarity score of page 10.

[0186] First, the search page group generation unit (101) can identify all document components (text, images, tables, etc.) included in page 10. Then, the similarity between the embedding vector of each document component and the question embedding vector can be calculated. In the example of FIG. 10 (b), the five document components in page 10 can be calculated to have similarities of 0.6, 0.8, 0.6, 0.9, and 0.7, respectively.

[0188] At this time, the search page group generation unit (101) can calculate the maximum similarity and average similarity of the corresponding page based on the calculated multiple similarity values. In the example of FIG. 10 (b), the maximum similarity is determined to be '0.9', which is the maximum value among the similarity values, and the average similarity can be calculated as '0.72', which is the average value of the similarity values.

[0189] Maximum similarity is an indicator of whether the page contains key correct answer candidates for the question, and average similarity can be an indicator of how well the content of the entire page generally matches the context of the question.

[0191] The search page group generation unit (101) above can calculate an integrated similarity score by applying a preset weight to the derived maximum similarity and average similarity. In the example of FIG. 10 (b), the integrated similarity score is calculated by assigning a weight of 0.7 to the maximum similarity and a weight of 0.3 (1 - 0.7) to the average similarity, and 0.9 × 0.7 + 0.72 × 0.3 = 0.846 can be determined as the final integrated similarity score of page 10.

[0193] FIG. 11 illustrates the step of calculating an integrated similarity score and reorganizing a search page group in the search page group generation step according to an embodiment of the present invention.

[0195] FIG. 11 (a) illustrates the process of comparing the integrated similarity score calculated by the search page group generation unit (101) for each search page with a preset threshold value.

[0196] Through the process shown in Fig. 10(b), the integrated similarity score for page 10 was calculated to be 0.846. The same process can be performed for the remaining search pages to calculate their integrated similarity scores. As shown in Fig. 11(a), the integrated similarity scores for page 10, page 13, page 8, and page 7 were calculated to be 0.846, 0.814, 0.72, and 0.783, respectively, all having values ​​greater than or equal to the threshold of 0.7. On the other hand, the integrated similarity scores for page 5 and page 4 were calculated to be 0.54 and 0.647, respectively, having values ​​less than the threshold of 0.7.

[0197] The search page group generation unit (101) may determine that pages (10, 13, 8, 7) with an integrated similarity score above a threshold are pages highly relevant to the question and maintain them in the search page group, and pages (5, 4) with an integrated similarity score below a threshold are determined to be pages that are far removed from the context of the question and may hinder the generation of answers and classify them as exclusion targets from the search page group.

[0199] FIG. 11(b) illustrates a search page group reconstructed according to the result of FIG. 11(a). As shown on the left side of FIG. 11(b), the initial search page group includes pages 10, 13, 8, 7, 5, and 4, but the search page group generation unit (101) can exclude pages 5 and 4 from the group if they are below a threshold value through filtering based on an integrated similarity score. As a result, as shown on the right side of FIG. 11(b), the search page group can be reconstructed into a refined form including only pages 10, 13, 8, and 7.

[0201] The search page group that has been finally reconstructed above can be passed as basic data to the subsequent page image group generation step (S102). Through this, the server system does not perform unnecessary process extension operations on pages that are not relevant to the question, and as a result, the VLM (3) can receive only page images focused on the question intent, thereby maximizing the accuracy of the answer.

[0203] According to one embodiment of the present invention, by utilizing an integrated similarity score that combines the maximum and average values ​​of individual document components, it is possible to select only search pages that contain core information most closely related to the question and whose overall context matches the intent of the question, thereby achieving the effect of improving the answer accuracy of VLM.

[0205] FIG. 12 illustrates a step of calculating the process ratio included in a search page group according to an embodiment of the present invention.

[0206] According to one embodiment of the present invention, each of the plurality of search pages included in the search page group belongs to a process including one or more of a cover, a table of contents, a main text, and an appendix, and the search page group creation step may further include a search page group reconstruction step, wherein, after the search page group is created, if the processes for each of the plurality of search pages included in the search page group are not unified into one process, the search page group reconstruction step provides a user with a list of one or more processes belonging to the search page group among the plurality of processes, asks the user to query which process corresponds to the user's intention, and based on the user's response to the query, reconstructs the search page group while excluding the search page corresponding to the process that does not correspond to the user's intention from the search page group.

[0207] In addition, according to one embodiment of the present invention, the search page group reconstruction step may further include: a step of identifying a process corresponding to each of a plurality of search pages included in the search page group based on the process information and calculating the ratio of each process included in the search page group; a step of transmitting a query to a user terminal to identify a process that matches the user's intention among the plurality of processes included in the search page group when the maximum ratio among the calculated ratios of each process is less than a preset reference ratio; and a step of receiving feedback on the query from the user and reconstructing the search page group by excluding from the search page group a page that does not correspond to a process corresponding to the feedback among the plurality of search pages.

[0209] FIG. 12 illustrates a process for calculating the ratio of processes included in a search page group to perform a search page group reorganization step.

[0210] The search page group generation unit (101) can identify a process corresponding to each of the plurality of search pages included in the search page group based on the process information of the RAG database (12) and calculate the ratio of each process included in the search page group.

[0212] In the example of FIG. 12(a), the search page group generation unit (101) can match which process each page belongs to by comparing it with the process information stored in the RAG database (12) for each of pages 10, 13, 7, 5, and 4 included in the search page group. As a result, pages 10 and 13 are identified as belonging to 'Process A', and pages 7, 5, and 4 are identified as belonging to 'Process B'.

[0213] At this time, the search page group generation unit (101) calculates the ratio occupied by each process, and as shown in the circular graph on the right side of (a) of FIG. 12, process A can be calculated as 40% and process B as 60%.

[0215] For Fig. 12(b), the process for each search page within the search page group can be identified in the same way, and the ratio of the process can be calculated. As a result of the identification by the search page group generation unit (101), pages 10 and 13 are found to belong to 'Process A', page 7 to 'Process B', page 5 to 'Process C', and page 4 to 'Process D'. In this case, the calculated ratio is 40% for Process A and 20% each for Processes B, C, and D.

[0217] The search page group generation unit (101) compares the maximum ratio among the ratios calculated in this way with a preset standard ratio. For example, if the standard ratio is set to 50%, in the example of FIG. 12 (a), the ratio of process B, which accounts for the maximum ratio, is 60%, which exceeds the standard ratio of 50%, so the search page group generation unit (101) can proceed to the subsequent page image group generation step without a separate user query.

[0219] On the other hand, in the example of FIG. 12 (b), the maximum ratio of Process A, which is 40%, falls short of the standard ratio of 50%. This may mean that there is a lack of dominant context of information among the search results, resulting in a state of high information ambiguity. Therefore, the search page group generation unit (101) can present a list of identified processes (Processes A, B, C, D) to the user and send a query to the user terminal (2) to induce the user to select their actual intention. The specific process will be described later in FIG. 13.

[0221] FIG. 13 illustrates a process of sending a query containing a process list to a user to perform the search page group reorganization step, and reorganizing the search page group based on user feedback.

[0223] FIG. 13(a) illustrates the process of receiving user feedback and the query interface transmitted by the search page group generation unit (101) to the user terminal (2). When the query initially entered by the user (e.g., "I want to register") is ambiguous and multiple processes (processes A, B, C, D) are mixed in the search page group, the server system may provide the user with a list of identified processes along with a query such as "Which of the following fields would you like me to help you 'register' for?"

[0225] At this time, the process list provided to the user by the search page group generation unit (101) may not be a list of all processes stored in the RAG database (12), but may be a list consisting only of specific processes identified as corresponding to each of the multiple search pages included in the search page group.

[0226] That is, the server system can achieve the effect of enabling the user to accurately select their intention from the entire process by selecting and providing only the candidate process group that is determined to have a direct correlation with the current search result from the RAG database (12).

[0228] According to one embodiment of the present invention, the search page group reconstruction step may include a process list configured by selecting one or more processes identified as belonging to the search page included in the search page group among a plurality of processes included in the multimodal document, in order to verify a process that matches the user's intention in a query transmitted to the user.

[0230] Subsequently, when the user provides feedback selecting a specific process (e.g., 'Process A'), the search page group generation unit (101) can finally identify and confirm a specific process that matches the user's question intent based on the received feedback.

[0232] FIG. 13(b) illustrates the process of filtering and reconfiguring a search page group based on user feedback. Following the example in FIG. 12(b), the search page group before reconfiguring based on user feedback includes pages 10, 13 (Process A), page 7 (Process B), page 5 (Process C), and page 4 (Process D).

[0233] The search page group generation unit (101) can identify the remaining pages that do not belong to the corresponding process by referring to the process information of the RAG database (12) as user feedback is received as 'Process A'. As shown in FIG. 13 (b), page 7 belonging to Process B, page 5 belonging to Process C, and page 4 belonging to Process D can be classified as exclusion targets.

[0235] Finally, as illustrated on the right side of FIG. 13 (b), the search page group generation unit (101) can reconfigure the search page group to include only pages 10 and 13 of process A that match the user's intent by removing exclusion targets. The search page group thus refined can then be passed to the page image group generation step (S102).

[0237] According to one embodiment of the present invention, in ambiguous situations where the search system finds it difficult to make a judgment on its own, it can provide the user with explicit options and narrow the search range based on the results, thereby blocking pages unrelated to the question from being input into the VLM (3). Through this, the VLM (3) can generate answers by focusing only on the specific process context that the user actually needs, and as a result, it can have the effect of improving search precision and the accuracy of the answers.

[0239] FIG. 14 illustrates the process of generating a guide image according to an embodiment of the present invention.

[0240] According to one embodiment of the present invention, the answer derivation step may further include: for each of a plurality of page images included in the page image group, a step of deriving a non-text element in which an embedding vector corresponding to a document component excluding text has a similarity of greater than or equal to a preset similarity threshold with respect to the question embedding vector; a step of generating a guide image by cropping or highlighting an area corresponding to the non-text element in the page image based on the position coordinates of the non-text element; and an answer provision step of providing the answer derived for the question query and the guide image to a user.

[0242] Referring to FIG. 14, the process of the answer derivation unit (103) processing a visual object that serves as the core basis of the answer into a separate guide image is illustrated in order to increase the reliability of the answer generated through the VLM (3).

[0244] Specifically, the answer derivation unit (103) can perform similarity verification on elements such as images, tables, and graphs, excluding text, among the document components included in each page image within the page image group. At this time, the answer derivation unit (103) can compare the embedding vector of the document component within the page image stored in the RAG database (12) with the question embedding vector generated in the question receiving step (S100), and derive elements whose similarity is greater than or equal to a preset similarity threshold by defining them as non-text elements.

[0246] Afterward, the answer derivation unit (103) can refer to the location coordinates included in the document component information of the derived non-text element. As shown in the example of FIG. 14, the answer derivation unit (103) can generate a guide image by specifying the area occupied by the non-text element in the original page image based on the location coordinates and performing a cropping process to cut out only that area.

[0248] Finally, in the above answer provision step, the server system (1) can transmit the guide image generated through the above process to the user terminal (2), along with the natural language answer generated by the internal or external VLM (3).

[0249] Accordingly, users can reduce unnecessary search time to locate information by clearly identifying only the key parts of the table or image that served as the basis for the answer through guide images while simultaneously reading the text description provided by the system.

[0251] According to one embodiment of the present invention, by automatically extracting and highlighting non-text elements (tables, images, etc.) highly relevant to the question based on coordinates, it is possible to visually and immediately present key information within a multimodal document and support the user's intuitive verification of the accuracy of the answer.

[0253] FIG. 15 illustrates an example of an interface provided through an answer-providing step according to an embodiment of the present invention.

[0254] According to one embodiment of the present invention, the interface provided through the answer derivation step comprises: an answer display layer including an answer derived for the question query; and a related image display layer displaying all or part of the page image that serves as the basis for the answer among a plurality of page images included in the page image group; wherein the related image display layer can visually display a guide image cropped or highlighted in an area corresponding to a non-text element, which is a document component that serves as the basis for the answer in the page image.

[0256] Referring to FIG. 15, the layout of the result screen that the server system (1) ultimately provides to the user terminal (2) is illustrated. When a user inputs a question query such as "Tell me how to register a new employee," the server system (1) can perform the above-described question reception step (S100), search page group creation step (S101), page image group creation step (S102), and answer derivation step (S103) to create and deliver an interface that combines text and visual evidence data (image) with the derived answer.

[0258] The above answer display layer (L1030) may correspond to a layer that outputs a natural language answer generated by VLM (3) by referencing a page image group. As shown in the example of FIG. 15, the above answer display layer (L1030) may display standardized text guides such as 'access path' and 'actual work procedure'.

[0260] The above-mentioned related image display layer (L1031) may correspond to a layer that visually presents objective grounds for the content displayed in the above-mentioned answer display layer (L1030). As illustrated in FIG. 15, the above-mentioned related image display layer (L1031) may display a list of original page images of the multimodal document that served as the basis for the answer. At this time, the above-mentioned related image display layer (L1031) may include guide images so that the user can easily compare and understand the text-based explanation with the content within the actual images.

[0262] The above guide image may display non-text elements (e.g., buttons in the system UI, settings screens for specific functions, etc.) by highlighting them within the above-mentioned related image display layer (L1031). For example, a button to be clicked or an input window area within a system UI screenshot may be highlighted with a colored border, or a precisely cropped and enlarged image of a specific diagram or table area may be placed as the guide image.

[0263] This allows users to intuitively understand where on the actual software screen the corresponding task is performed while simultaneously reading the text response.

[0265] In addition, when a user selects a specific image displayed on the related image display layer (L1031), the system may switch to a detailed viewer mode that allows the user to view the entire process pages containing the image in their original size. This can be usefully utilized when the user wishes to fully examine the context of the manual containing the task, going beyond the basis of a fragmentary answer.

[0267] According to one embodiment of the present invention, by simultaneously providing a text answer (L1030) and a visual guide (L1031) in logically separated layers, the user can be given confidence in the information and eliminate the hassle of having to find the location of the source within a complex document, thereby achieving the effect of dramatically improving the speed of business processing.

[0269] FIG. 16 schematically illustrates the internal configuration of a computing device according to one embodiment of the present invention.

[0270] The server system illustrated in FIG. 1 described above may include the components of the computing device (11000) illustrated in FIG. 16.

[0271] As illustrated in FIG. 16, the computing device (11000) may include at least one processor (11100), memory (11200), peripheral interface (11300), input / output subsystem (I / O subsystem) (11400), power circuit (11500), and communication circuit (11600). In this case, the computing device (11000) may correspond to the server system illustrated in FIG. 1.

[0272] The memory (11200) may include, for example, high-speed random access memory, a magnetic disk, SRAM, DRAM, ROM, flash memory, or non-volatile memory. The memory (11200) may include software modules, instruction sets, or various other data required for the operation of the computing device (11000).

[0273] At this time, access to memory (11200) from other components, such as the processor (11100) or peripheral device interface (11300), can be controlled by the processor (11100).

[0274] The peripheral device interface (11300) can connect input and / or output peripheral devices of the computing device (11000) to the processor (11100) and memory (11200). The processor (11100) can perform various functions for the computing device (11000) and process data by executing software modules or instruction sets stored in the memory (11200).

[0275] The input / output subsystem can connect various input / output peripherals to the peripheral interface (11300). For example, the input / output subsystem may include a controller for connecting peripherals such as a monitor, keyboard, mouse, printer, or, if necessary, a touchscreen or sensor to the peripheral interface (11300). According to another aspect, input / output peripherals may be connected to the peripheral interface (11300) without passing through the input / output subsystem.

[0276] The power circuit (11500) can supply power to all or part of the components of the terminal. For example, the power circuit (11500) may include one or more power sources such as a power management system, a battery or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, or any other components for power generation, management, and distribution.

[0277] The communication circuit (11600) can enable communication with another computing device using at least one external port.

[0278] Alternatively, as described above, the communication circuit (11600) may enable communication with other computing devices by including an RF circuit and transmitting and receiving an RF signal, also known as an electromagnetic signal.

[0279] The embodiment of FIG. 16 is merely an example of a computing device (11000), and the computing device (11000) may have some components shown in FIG. 16 omitted, additional components not shown in FIG. 16 added, or a configuration or arrangement that combines two or more components. For example, a computing device for a communication terminal in a mobile environment may include a touchscreen or sensors, etc., in addition to the components shown in FIG. 16, and the communication circuit (11600) may include a circuit for RF communication of various communication methods (WiFi, 3G, LTE, Bluetooth, NFC, Zigbee, etc.). Components that can be included in the computing device (11000) may be implemented as hardware, software, or a combination of both hardware and software, including one or more integrated circuits specialized for signal processing or applications.

[0280] Methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computing devices and recorded on a computer-readable medium. In particular, the program according to the present embodiment may be configured as a PC-based program or an application dedicated to a mobile terminal. An application to which the present invention is applied may be installed on a computing device (11000) through a file provided by a file distribution system. For example, the file distribution system may include a file transmission unit (not shown) that transmits the file in response to a request from the computing device (11000).

[0282] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0283] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computing devices and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0284] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0286] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents. Therefore, other implementations, other embodiments, and equivalents to the claims are also included within the scope of the claims set forth below.

Claims

Claim 1 A multimodal document-based search and response method using process-unit indexing, performed on a server system comprising one or more processors and one or more memories, wherein the server system comprises: a plurality of embedding vectors generated for each document component including one or more of text, images, graphs, and tables included in a specific page of a multimodal document; and process information including an identifier, a start index, and an end index for a process of the corresponding page determined based on one or more of metadata and table of contents information of the multimodal document. A multimodal document-based search and response method comprises: a Retrieval-Augmented Generation (RAG) database in which page images capturing the entire page are stored for each of a plurality of pages; and a multimodal document-based search and response method comprising: a question receiving step of receiving a question query from a user and generating a question embedding vector for the question query; a search page group generating step of deriving a plurality of embedding vectors from the RAG database that are equal to or greater than a pre-set similarity threshold with respect to the question embedding vector and generating a search page group including a plurality of search pages that include document components corresponding to each of the embedding vectors; a page image group generating step of deriving a process to which the search page belongs based on the process information for each of the plurality of search pages included in the search page group and generating a page image group including page images for each of the plurality of pages included in the process; and an answer derivation step of inputting the page image group and the question query into an external or internal Vision-Language Model (VLM) to derive an answer to the question query. Claim 2 A multimodal document-based search and response method according to claim 1, wherein the embedding vector is stored in the RAG database by performing: a text information extraction step of extracting text information included in each of a plurality of pages included in the multimodal document; a document component derivation step of deriving one or more document components including text, images, graphs, and tables by inputting a page image of the corresponding page and the extracted text information of the corresponding page into the VLM for each of the plurality of pages; a document component information generation step of generating document component information including one or more of the type, content, and location coordinates of the corresponding document component for each of the derived document components; and an embedding vector generation step of generating the embedding vector based on the document component information and storing it in the RAG database. Claim 3 A multimodal document-based search and response method according to claim 1, wherein the process information is stored in the RAG database by performing: a process derivation step of deriving one or more processes included in the multimodal document based on one or more of the table of contents information and metadata of the multimodal document; a process identification step of identifying which process a page corresponds to based on document components included in each page of the multimodal document; and a process information generation step of generating process information including one or more of an identifier, a start index, and an end index for each of the processes and storing it in the RAG database. Claim 4 A multimodal document-based search and response method according to claim 1, wherein each of the plurality of search pages included in the search page group belongs to a process including one or more of a cover, a table of contents, a main text, and an appendix, and the search page group creation step further comprises: a search page group reconstruction step in which, after the search page group is created, if the processes for each of the plurality of search pages included in the search page group are not unified into a single process, the search page group reconstruction step provides a user with a list of one or more processes belonging to the search page group among the plurality of processes, asks the user to query which process corresponds to the user's intention, and based on the user's response to the query, reconstructs the search page group while excluding the search page corresponding to the process that does not correspond to the user's intention from the search page group. Claim 5 A multimodal document-based search and response method according to claim 1, wherein the answer derivation step further comprises: for each of a plurality of page images included in the page image group, a step of deriving a non-text element in which an embedding vector corresponding to a document component excluding text has a similarity of greater than or equal to a preset similarity threshold with respect to the question embedding vector; a step of generating a guide image by cropping or highlighting an area corresponding to the non-text element in the page image based on the position coordinates of the non-text element; and an answer provision step of providing the answer derived for the question query and the guide image to a user. Claim 6 A multimodal document-based search and response system using process-unit indexing comprising one or more processors and one or more memories, comprising: a plurality of embedding vectors generated for each document component including one or more of text, images, graphs, and tables included in a specific page of a multimodal document; and process information including an identifier, a start index, and an end index for a process of the corresponding page determined based on one or more of metadata and table of contents information of the multimodal document. A multimodal document-based search and response system comprising: a RAG (Retrieval-Augmented Generation) database in which page images capturing the entire page are stored for each of a plurality of pages; a question receiving unit that receives a question query from a user and generates a question embedding vector for the question query; a search page group generating unit that derives a plurality of embedding vectors from the RAG database that are equal to or greater than a preset similarity threshold with respect to the question embedding vector and generates a search page group including a plurality of search pages that include document components corresponding to each of the embedding vectors; a page image group generating unit that, for each of the plurality of search pages included in the search page group, derives a process to which the search page belongs based on the process information and derives a page image group including page images for each of the plurality of pages included in the process; and an answer derivation unit that inputs the page image group and the question query into an external or internal VLM (Vision-Language Model) to derive an answer to the question query. Claim 7 delete Claim 8 delete

Images