Document-based question-answering system

Abstract

The present invention relates to a document-based question-and-answer system, and more specifically, to an artificial intelligence-based document-based question-and-answer system and a method of operation thereof that, in generating answers to a user's natural language query from large-scale document data, evaluates and verifies the reliability of the answer from various angles and dynamically allocates system operational resources in accordance with real-time demand changes to simultaneously improve the quality of the answer and the stability of the service.

Description

Technology Field

[0001] The present invention relates to a document-based question-and-answer system, and more specifically, to an artificial intelligence-based document-based question-and-answer system and a method of operation thereof that, in generating answers to a user's natural language query from large-scale document data, evaluates and verifies the reliability of the answer from various angles and dynamically allocates system operational resources in accordance with real-time demand changes to simultaneously improve the quality of the answer and the stability of the service. Background Technology

[0002] With the recent advancement of Large Language Models, the technological level of question-and-answer systems has significantly improved, enabling them to respond to user questions in natural sentences based on vast document data. These systems are changing the paradigm of information retrieval by rapidly summarizing and providing the information users desire.

[0003] However, conventional document-based question-and-answer systems have several obvious technical limitations.

[0004] First, conventional systems evaluate the reliability of the information referenced when generating answers only superficially. Most systems merely refer to documents from credible sources and lack the capability to conduct in-depth verification to determine whether content conflicts between multiple reference documents, whether they contain expressions biased toward a particular perspective, or whether the generated answer itself contains logical contradictions. Consequently, there is a risk that systems may cause serious misunderstandings by providing users with answers that contain cleverly manipulated information that appears factual, or errors that are difficult for non-experts to identify.

[0005] Second, conventional systems allocate system resources in a static or passive manner. Most systems operate with a fixed amount of allocated computing resources or use a method of increasing resources retrospectively only after the internal load exceeds a threshold. However, when queries on specific topics surge instantaneously due to particular social events or issues, this reactive approach causes a drastic decrease in response speed or service interruptions. Conversely, there was a problem in that unnecessary resources were occupied during periods of low usage, leading to wasted operational costs.

[0006] Therefore, there is an urgent technical need for a new type of question-and-answer system that can fundamentally ensure the reliability of generated responses and provide stable services by actively responding to unpredictable changes in demand. The problem to be solved

[0007] The present invention was devised to solve the problems of the prior art as described above, and aims to solve the following technical problems.

[0008] The first objective is to provide a systematic method capable of evaluating the reliability of generated answers from various perspectives and in depth. To this end, the present invention provides a hierarchical reliability assessment procedure that includes content consistency among reference documents, diversity of sources, bias of the original text, internal logic of the generated answers, and finally, expert verification, thereby aiming to minimize the risk of users being exposed to incorrect or biased information.

[0009] The second objective is to provide a method for simultaneously maximizing service stability and operational efficiency by predictively and dynamically allocating the system's computational resources. To this end, the present invention aims to maintain stable service quality under any circumstances by providing a procedure to predict future query demand through a comprehensive analysis of not only the complexity of individual queries but also real-time social interest and the occurrence of unexpected events, and to preemptively reallocate system resources based on this prediction.

[0010] The problems of the present invention are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0012] According to an embodiment of the present invention for solving the above problem

[0013] A data collection unit that collects document data from the outside;

[0014] A document storage unit that stores the above-mentioned collected document data;

[0015] A user interface module that receives a user's query;

[0016] A document search module that searches for documents related to the above query among the above stored document data;

[0017] A document analysis module that analyzes the content of the above-mentioned searched documents and divides them into semantic units;

[0018] A question-and-answer processing module that generates a draft answer to the above query using the above-described divided semantic units;

[0019] A reliability evaluation module for determining the reliability of the above-mentioned generated answer draft; and

[0020] A response generation module that generates a final response based on the reliability determined above;

[0021] Includes,

[0022] The above reliability evaluation module determines a reliability grade by synthesizing the internal characteristics of the documents referenced in the generation of the above answer draft and the logical characteristics of the above answer draft itself, and

[0023] The above question-and-answer processing module is characterized by dynamically determining the level of computational resource allocation of the system by integrating the characteristics of the query, external data, and the current state of the system.

[0024] The above reliability evaluation module is,

[0025] A first procedure for determining a basic confidence level by combining document mutual integrity consistency, which indicates semantic consistency among multiple referenced documents, and source diversity entropy, which indicates source diversity of the referenced documents;

[0026] A second procedure for analyzing source bias based on the linguistic expression of the referenced document and cognitive consistency based on the internal sentence structure of the generated answer draft, respectively, and determining whether the difference between the two analysis results exceeds a preset first threshold, such as when the source bias is high and the cognitive consistency is high, thereby downgrading the basic confidence level; and

[0027] A third procedure for determining the final confidence level by reflecting the results of 'validity' or 'error' verification by a registered expert on the confidence level adjusted through the second procedure above;

[0028] By performing sequentially,

[0029] It is characterized by improving the reliability of response results by blocking the provision of answers containing sophisticatedly manipulated information or expert-level errors, and

[0030] The above question-and-answer processing module is,

[0031] A first procedure for calculating query complexity by analyzing the length of the user query and the number of included entities, calculating user interest by collecting real-time search volume and social media mention volume from external data sources, and determining the basic resource requirement level by combining the calculated query complexity and user interest;

[0032] A second procedure for monitoring a real-time news feed to detect whether an unexpected event related to the above-mentioned query topic has occurred, and if the unexpected event is detected, forcibly raising the resource requirement level to a preset upper level regardless of the basic resource requirement level determined in the first procedure; and

[0033] A third procedure for determining the system load rate based on the central processing unit (CPU) usage and memory occupancy of the current system, and determining the final resource allocation amount at a level one step lower than the resource requirement level adjusted in the second procedure if the system load rate is in an 'overload' state;

[0034] By performing sequentially,

[0035] It is characterized by securing service stability by proactively responding to a surge in queries and reducing operating costs by allocating resources in accordance with the system's current processing capacity,

[0036] The above reliability evaluation module is,

[0037] The above document mutual integrity consistency is obtained by converting the key sentences of each of the referenced documents into vectors using a natural language processing embedding model, calculating the cosine similarity between all pairs of vectors, and determining the average value;

[0038] The above source diversity entropy is obtained by calculating the amount of information by applying the Shannon entropy formula to the source distribution of the above-mentioned referenced documents;

[0039] The above source bias is characterized by being obtained by applying a pre-trained sentiment analysis model to each of the above-mentioned referenced documents to calculate the average of the absolute values ​​of positive or negative polarity scores, and

[0040] If the final resource allocation amount is determined to be at the 'maximum allocation' level by the above question-and-answer processing module,

[0041] The document search module is controlled to search for a larger number of documents than a preset standard depth, and the question-answer processing module is controlled to activate a high-performance language model with parameters larger than a preset default language model;

[0042] If the above final resource allocation is determined to be at the 'minimum allocation' level,

[0043] The document search module is controlled to search for fewer documents than the standard depth, and the question-answer processing module is controlled to activate a lightweight language model with parameters smaller than the base language model.

[0044] The document search module is controlled to operate in a 'context-based filtering mode' that extracts only the paragraphs most similar to the query within the searched top-level documents when the question-answer processing module analyzes the query and determines it to be a question about a single concept, and to operate in a 'multi-document combining mode' that returns all related top-level documents when the question-answer processing module determines it to be a question asking for a comparison between multiple concepts;

[0045] The above document analysis module is,

[0046] The method may be characterized by dividing the received document text into sentence units, sequentially calculating the similarity between the embedding vectors of consecutive sentences, identifying a point where the similarity value drops sharply below a preset second threshold as a 'semantic boundary', and grouping related sentences into a single group based on the identified boundary to generate a semantic paragraph. Effects of the invention

[0048] The present invention, for solving the above-mentioned problems, provides the following effects.

[0049] The reliability of the response results can be significantly improved. Through a multi-stage reliability determination procedure that includes mutual verification between reference documents, bias detection, and expert verification, the present invention effectively blocks the provision of responses containing sophisticated fake news or expert-level errors that conventional technology could not filter out. As a result, users can utilize the information provided by the system of the present invention with a high level of reliability.

[0050] The stability and availability of the service can be maximized. The present invention detects the occurrence of social events related to a specific topic in real time and predicts a surge in queries resulting therefrom in advance to preemptively increase system resources. This prediction-based dynamic resource allocation method guarantees stable response speeds even with sudden increases in traffic and minimizes the risk of service interruption.

[0051] The cost efficiency of system operation can be increased. The present invention dynamically controls the system to allocate only the minimum necessary resources for time periods with few user queries or topics of low interest. This prevents the waste of unnecessary computing resources, thereby providing an economic benefit of reducing overall system operation costs, such as cloud service fees.

[0052] It provides users with a more in-depth information search experience. This invention analyzes the type of query to vary the search method using 'context-based filtering' or 'multi-document merging' modes, and extracts key information by analyzing documents based on semantic boundaries. Through this, users can receive information-dense answers processed into a form most suitable for their inquiry intent.

[0053] The effects according to the present invention are not limited to those exemplified above, and a wider variety of effects are included within the present invention. Brief explanation of the drawing

[0055] Figure 1 illustrates an overall relationship diagram according to the present invention. Figure 2 illustrates the overall operation flowchart of a document-based question-and-answer system according to the present invention. Figure 3 illustrates a flowchart of the reliability determination procedure according to the present invention. Figure 4 illustrates a flowchart of an adaptive resource allocation procedure according to the present invention. Figure 5 illustrates a flowchart of a contextual time weight determination procedure according to the present invention. Figure 6 illustrates a flowchart of an information contribution-based duplicate determination procedure according to the present invention. Specific details for implementing the invention

[0056] Hereinafter, various embodiments are described in more detail with reference to the attached drawings. The embodiments described in this specification may be modified in various ways. Specific embodiments may be depicted in the drawings and described in detail in the detailed description. However, specific embodiments disclosed in the attached drawings are intended only to facilitate understanding of various embodiments. Accordingly, the technical concept is not limited by specific embodiments disclosed in the attached drawings, and it should be understood that it includes all equivalents or substitutions that fall within the spirit and scope of the invention.

[0057] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but these components are not limited by the aforementioned terms. The aforementioned terms are used solely for the purpose of distinguishing one component from another.

[0058] The functions related to artificial intelligence according to the present disclosure are operated through a processor and memory. The processor may be composed of one or more processors. In this case, the one or more processors may be general-purpose processors such as CPUs, APs, and DSPs (Digital Signal Processors), graphics-dedicated processors such as GPUs and VPUs (Vision Processing Units), or artificial intelligence-dedicated processors such as NPUs. The one or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if the one or more processors are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0059] The predefined rules of operation or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a basic artificial intelligence model is trained using a number of training data by a learning algorithm, thereby creating predefined rules of operation or artificial intelligence models configured to perform desired characteristics (or objectives). Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0060] An artificial intelligence model can be composed of multiple neural network layers. Each of the multiple neural network layers has multiple nodes and weight values, and performs neural network operations through calculations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, multiple weights can be updated so that the loss value or cost value obtained by the artificial intelligence model during the learning process is reduced or minimized. Additionally, to minimize the loss value or cost value, multiple weights can be updated in a direction that minimizes the gradient associated with the loss value or cost value. Artificial neural networks may include deep neural networks (DNNs), such as Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), or Deep Q-Networks, but are not limited to the examples mentioned above.

[0061] A network is a network that serves as a transmission path for web pages; it may be a closed network such as a LAN (Local Area Network) or WAN (Wide Area Network), but it is desirable for it to be an open network such as the Internet. The Internet refers to a global open computer network structure that provides the TCP / IP protocol and various services existing at its upper layers, namely HTTP (HyperText Transfer Protocol), Telnet, FTP (File Transfer Protocol), DNS (Domain Name System), SMTP (Simple Mail Transfer Protocol), SNMP (Simple Network Management Protocol), NFS (Network File Service), and NIS (Network Information Service).

[0062] Terminals can be implemented in various forms. For example, the terminals described in this specification may include mobile terminals such as smartphones, tablet PCs, PDAs, portable multimedia players, and MP3 players, as well as fixed terminals such as smart TVs and desktop computers.

[0063] In this specification, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof. When a component is described as being “connected” or “connected” to another component, it should be understood that it may be directly connected to or connected to that other component, or that there may be other components in between. On the other hand, when a component is described as being “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

[0064] Meanwhile, a "module" or "part" for a component as used in this specification performs at least one function or operation. Furthermore, a "module" or "part" may perform a function or operation by hardware, software, or a combination of hardware and software. Additionally, a plurality of "modules" or a plurality of "parts," excluding a "module" or "part" that must be performed on specific hardware or on at least one processor, may be integrated into at least one module. A singular expression includes a plural expression unless the context clearly indicates otherwise.

[0065] In addition, power, power transmission, and control therefor for the following assembly configurations and embodiments, including "by control," follow conventional technology including terminals, applications, hardware control modules, etc., so they are omitted to avoid redundancy.

[0066] In addition, the operation embodiments and configurations described in a general manner without being explained in detail below follow the prior art and are omitted in order to focus on describing the purpose of the present invention and the resulting effects.

[0067] Furthermore, in describing the present invention, if it is determined that a detailed description of related known functions or configurations may unnecessarily obscure the essence of the invention, such detailed description is abbreviated or omitted.

[0068] A document-based question-and-answer system according to the present invention performs operations by including a data collection unit (100) that collects document data from an external source, a document storage unit (200) that stores and indexes the collected data, a user interface module (300) that handles interaction with a user, a document search module (400) that searches for documents suitable for a query, a document analysis module (500) that processes the searched documents in semantic units, a question-and-answer processing module (600) that generates a draft answer and coordinates the entire process, a reliability evaluation module (700) that evaluates the reliability of the generated answer from various angles, and a response generation module (800) that combines and provides a final response customized to the user.

[0069] The document-based question-and-answer system of the present invention configured as described above may be implemented in one embodiment based on a Retrieval-Augmented Generation (RAG) architecture that combines document search and a large-scale language model. In a more specific embodiment, the system operates on an Ubuntu operating system, and each component may be executed in isolation within a Docker container environment. External requests are received via an NGINX reverse proxy, the backend API including the question-and-answer processing module (600) is implemented using a Python-based FastAPI framework, and the semantic search function of the document storage unit (200) may be implemented using a vector database such as Qdrant. The unique processing procedures of the present invention, described below, operate on this specific technology stack environment to provide improved effects that go beyond conventional simple RAG systems.

[0070] The data collection unit (100) performs the operation of acquiring document data from external data sources, such as web pages, academic databases, or internal file systems, and collecting it into the system. In one embodiment, the data collection unit (100) may be characterized by using the PyMuPDF library for PDF documents, the tesseract-ocr engine-based ocrmypdf library for scanned images or PDFs, and the docx2txt library for Microsoft Word documents to extract text from documents of various formats. The data collection unit (100) operates periodically according to a schedule pre-set by an administrator to collect the latest information from a designated data source, and also transmits a command to ensure real-time operation by non-periodically triggering the collection of data sources related to a specific topic when the user interest index for a specific topic exceeds a preset threshold. The above data collection unit (100) receives document data including original text and metadata from an external source via an HTTP / HTTPS protocol, FTP, or database connector, packages the collected document data into a defined data format, and transmits it to a document storage unit (200) to be described later.

[0071] The document storage unit (200) permanently stores document data received from the data collection unit (100) and performs the operation of structuring and indexing the data to enable rapid searching. The document storage unit (200) stores the original text and metadata of the received document data in a database and, simultaneously with storage, creates an inverse index for keyword-based searching of the text. In addition, to support semantic similarity searching of documents, it converts each document into a high-dimensional vector through a pre-trained language model (as an example, OpenAI’s Embedding API) and performs the operation of creating and managing a vector index (as an example, stored and managed in a Qdrant vector database) that can be rapidly searched. In one embodiment, communication with the vector index may be performed through the qdrant-client library. The document storage unit (200) receives document data from the data collection unit (100), receives a search query from the document search module (400) to be described later, and returns the identifier or original data of the document corresponding to the query to the document search module (400).

[0072] The user interface module (300) serves as a channel for a user to input a query into the system in text or voice and to visually check the final response generated by the system. The user interface module (300) detects the user's text input or converts voice input into text to generate a query, receives the query text from the user, and transmits it to the question-response processing module (600) described later. Additionally, it receives the final response object from the response generation module (800) described later and executes a command to render UI elements such as text, a reliability badge, a warning message, and a button included therein onto the screen.

[0073] The document search module (400) performs the operation of quickly finding the document most relevant to the user query from the document storage unit (200). The document search module (400) first filters a group of candidate documents from the inverse index of the document storage unit (200) using the core keywords of the query, secondarily calculates the cosine similarity between the embedding vector of the entire query and the embedding vectors of the candidate documents through the vector index, and finally calculates a final relevance score by weighted summing the keyword score and the similarity score, and sorts the documents in order of highest score. The document search module (400) switches the operation mode according to a control signal from the question-answer processing module (600), and controls the system to operate in a 'context-based filtering mode' when the query is determined to be an in-depth question about a single concept, and to operate in a 'multiple document combination mode' when the query is determined to be a question asking about the relationship or comparison between multiple concepts. The document search module (400) receives a processed query and operation mode command from the question-and-answer processing module (600), transmits the search query and results to and from the document storage unit (200), and finally transmits the selected document list or original document data to the document analysis module (500) to be described later.

[0074] The document analysis module (500) performs the operation of dividing and structuring the long text of the searched document into small, semantically related units so that it is easy for subsequent modules to process. The document analysis module (500) first divides the received document text into individual sentence units based on sentence termination symbols, and then sequentially calculates the similarity between the embedding vectors of the divided consecutive sentences to identify a point where the similarity value drops sharply below a preset threshold as a 'semantic boundary', and transmits a data processing command to generate semantic paragraphs based on this. The document analysis module (500) receives document data from the document search module (400) and transmits the set of semantic paragraphs, which have been analyzed and divided, to the question-and-answer processing module (600) to be described later.

[0075] The above question-and-answer processing module (600) performs a central processing operation of the system that identifies the intent of the user query, coordinates the operation of each component, and generates a draft of the answer based on the analyzed document content. The above question-and-answer processing module (600) analyzes the received user query to extract key keywords and query types, determines the operation mode to be transmitted to the document search module (400) based on this, and generates a draft of the answer by utilizing semantic paragraphs received from the document analysis module (500) as contextual information and transmitting them to a pre-trained artificial intelligence language model. In one embodiment, interaction with the artificial intelligence language model may be performed through an OpenAI API client library, and the above question-and-answer processing module (600) may be implemented as a Python FastAPI and a Uvicorn-based web API. Additionally, the above question-and-answer processing module (600) performs adaptive resource allocation logic to dynamically determine the grade or search depth of the language model to be used according to the system load and external factors, and transmits commands to control related modules. The above question-and-answer processing module (600) receives an initial query from the above user interface module (300), receives semantic paragraphs from the above document analysis module (500), and transmits the generated answer draft and referenced document information to the reliability evaluation module (700) and response generation module (800) to be described later.

[0076] The reliability evaluation module (700) performs the operation of determining the final reliability grade by verifying from various angles whether the generated draft answer is based on facts, unbiased, and logically valid. The reliability evaluation module (700) performs a three-stage hierarchical and logical processing procedure using five criteria: document mutual integrity, source diversity, source bias, cognitive consistency, and expert verification, and determines the final reliability grade. The reliability evaluation module (700) receives the draft answer and reference document information from the question-answer processing module (600), and when the evaluation is completed, transmits the final reliability grade and the corresponding UI display information to the response generation module (800) to be described later.

[0077] The response generation module (800) performs the operation of finally combining the results processed from each module and generating a completed response by reflecting user custom settings. The response generation module (800) performs a duplicate determination based on information contribution for the received response draft, and if it is determined to be a 'substantial duplicate,' transmits a control signal to the question-and-answer processing module (600) to perform response reconstruction. Additionally, the response generation module (800) performs a user-customized response control operation by querying the 'response detail level' and 'tone' values ​​pre-set in the user account, and accordingly, transmitting a control signal to the question-and-answer processing module (600) to dynamically change a part of the prompt to be used when generating the response. The response generation module (800) receives the response body and a list of reference documents from the question-and-answer processing module (600), receives trust rating information from the trust evaluation module (700), and transmits the final response object, in which all information is combined, to the user interface module (300).

[0078] The present invention performs the following organic operations in response to a scenario in which a query is input, "Tell me about the latest research trends and safety of COVID-19 vaccines." First, a user interface module (300) transmits the query text to a question-and-answer processing module (600). Subsequently, the question-and-answer processing module (600) analyzes the type of query and transmits a command to a document search module (400) to operate in 'multi-document combination mode'. The document search module (400) searches for the latest academic papers and institutional reports through a document storage unit (200) and transmits them to a document analysis module (500). The document analysis module (500) divides the document content into semantic paragraphs and transmits them to the question-and-answer processing module (600). The question-and-answer processing module (600) generates a draft answer based on the received paragraphs and transmits it to a reliability evaluation module (700) and a response generation module (800). The reliability evaluation module (700) determines the reliability of the draft answer as 'A (High)' grade and transmits the grade information to the response generation module (800). The response generation module (800) analyzes the information contribution of the draft answer, determines it as 'not duplicate', and checks the user settings. Finally, the response generation module (800) combines the final answer body, 'High Reliability' badge information, and the list of reference documents into a single response object and transmits it to the user interface module (300), and the user interface module (300) displays the received response object on the screen to provide the user with a detailed, highly reliable answer.

[0079] The following describes in detail the four major processing steps constituting the core operating principles of the present invention. The present invention goes beyond simply generating answers to user queries and includes processing procedures that verify the quality of the generated answers from various angles, efficiently manage system resources, and comprehensively evaluate the value of information.

[0080] The four main processing processes of the present invention to be described below are characterized by organically intervening in each stage of the data flow of a specific embodiment based on the previously described RAG architecture—namely, 'query, embedding, vector search (Qdrant), LLM prompt configuration (OpenAI), and answer generation'—to enhance the performance of the system.

[0081] First, the hierarchical reliability determination procedure of the present invention operates by intervening in the stage immediately after a large-scale language model (e.g., the OpenAI Chat model) generates a draft response within a standard RAG flow, and immediately before the response is finally provided to the user. This functions as a safeguard that verifies quality internally by the system itself, rather than uncritically accepting the results generated by the language model. Specifically, it determines basic reliability by synthesizing content consistency and source diversity among reference documents, identifies potential manipulation by detecting discrepancies between the bias of the original text and the logical consistency of the response, and finally performs a hierarchical reliability determination procedure that incorporates expert verification.

[0082] Next, the adaptive resource allocation procedure of the present invention operates throughout the entire process, from the stage where a query is first received from a user until the completion of answer generation. In particular, when an external event is detected, it intervenes by preemptively expanding the memory cache of an index to be used for vector search (e.g., Qdrant) or by dynamically controlling the use of a larger model when calling a large-scale language model (e.g., OpenAI). Specifically, it determines the basic resource requirement level by analyzing the complexity and social interest of individual queries, preemptively raises the requirement level according to external sudden events detected in real time, and finally performs an adaptive resource allocation procedure that allocates optimal resources by considering the current system load status.

[0083] Furthermore, the contextual time weight determination procedure of the present invention operates by intervening in the stage after a group of candidate documents is retrieved through vector search (e.g., Qdrant) in a standard RAG flow, and before the selection of reference documents to be finally passed to a large-scale language model. This goes beyond evaluating documents solely based on semantic similarity; it readjusts the ranking of search results by reflecting the temporal value of information. Specifically, it calculates the base value of information based on the unique temporal characteristics of a topic, assigns an added value based on current social interest or relevance to specific events, and performs a contextual time weight determination procedure that asymmetrically combines these two values ​​to correct for the omission of information that is old but has high current value.

[0084] Finally, the information contribution-based duplicate determination procedure of the present invention operates on a draft answer generated by a large-scale language model almost simultaneously with the reliability evaluation procedure. It serves to verify whether the generated answer is qualitatively poor or repeats unnecessary content, and, if necessary, provides feedback to reconstruct the answer focusing only on higher-value information. Specifically, it evaluates the actual information contribution by synthesizing the scarcity and expert reliability of each piece of information constituting the answer, determines the answer as actual duplication if the proportion of information with low contribution exceeds a certain level, and performs an information contribution-based duplicate determination procedure that controls the reconstruction of the answer focusing only on information with high contribution.

[0085] The above reliability evaluation module (700) performs a reliability evaluation operation by applying multiple mutually independent and complementary judgment criteria to determine the reliability of the answer.

[0086] The reliability evaluation module (700) performs a judgment operation by applying a document mutual integrity consistency criterion that indicates the degree to which multiple source documents referenced in the generation of the answer provide logically consistent information regarding the same topic. It transmits a control signal to prevent the system from generating an answer that appears plausible on the surface but actually contains internal contradictions by arbitrarily combining sentences from documents containing conflicting information, and to ensure the robustness of the factual basis of the answer. Additionally, the reliability evaluation module (700) conducts an evaluation by applying a source diversity entropy criterion that measures how diversely the sources of the documents serving as the basis for the answer are distributed. It controls the system operation to identify and suppress potential biases that may arise from the answer excessively relying on a small number of sources holding a specific perspective or position, and to systematically implement the effect of multiple independent sources cross-verifying the same facts. Furthermore, the reliability evaluation module (700) performs an analysis according to a source bias estimation index criterion that measures the tendency of the document to induce specific emotions or positions beyond the transmission of objective facts by analyzing the vocabulary, sentence structure, and sentiment expressions used by individual reference documents. When generating an answer from a document containing a mixture of facts and opinions, a processing command is transmitted to prevent hidden biases in the original text from being transferred directly to the answer and to ensure the neutrality of the answer. Additionally, the reliability evaluation module (700) analyzes the answer by applying a cognitive consistency index criterion to evaluate the completeness of the internal logical structure of the answer sentences themselves that the system ultimately generates. Even if there are no issues with the reference document, it detects logical leaps or contradictions that occur when sentences are unnaturally combined during the generation process, and generates output data to ensure the readability and persuasiveness of the answer itself.Finally, the reliability evaluation module (700) performs a final judgment by applying expert verification and reliability criteria that indicate the reliability of the expert themselves and the results of an evaluation, such as 'valid', 'error', or 'judgment withheld', made by a pre-registered and verified expert in a specific subject area regarding the content of the generated answer. For fields requiring a high level of expertise that are difficult to verify solely through the automated judgment of the system, the system controls the system to minimize the possibility of fatal errors in the answer by utilizing the knowledge of human experts as a final safeguard.

[0087] The reliability evaluation module (700) avoids the method of simply summing the five judgment criteria and performs the operation of determining reliability through a three-stage hierarchical and logical processing procedure. First, it executes a basic reliability grade determination procedure that first evaluates the quality of the source data that serves as the 'material' for the answer. It assigns a basic reliability grade by combining the consistency of the content, which is the document mutual integrity consistency, and the diversity of sources, which is the source diversity entropy. This operation is performed based on technical grounds for initially identifying the creation of an answer using poor materials. Second, it executes a grade adjustment procedure based on potential manipulation possibility that examines the possibility that the answer may appear well-processed but has been intentionally manipulated. It determines whether a significant discrepancy, or 'discrepancy,' occurs between the source bias estimation index, which is the bias of the source text, and the cognitive consistency index, which is the logic of the generated answer. When there is a high probability that the information is sophisticatedly manipulated, such as when a very logical answer is generated based on highly biased data, it transmits an adjustment command based on valid technical grounds for downgrading the reliability. Third, a procedure for determining the final grade is implemented through expert verification, which complements the limitations of the system's automated judgment with the knowledge of human experts. Since errors may exist from a professional perspective even if the system has passed the aforementioned mechanical analysis, the expert evaluation is used as the decisive basis for a final grade upgrade or a 'non-reliable' judgment to determine the final grade.

[0088] The reliability evaluation module (700) performs the operation of acquiring data for each judgment criterion in a specific manner and determining a grade. For the document mutual integrity consistency, it transmits a data processing command to convert the core sentences of each referenced document into vectors using a natural language processing embedding model, calculate the cosine similarity between all pairs of vectors, and calculate the average value. For example, if the average similarity is 0.9 or higher, it generates judgment data as 'High'; if it is 0.7 or higher but less than 0.9, it generates data as 'Medium'; and if it is less than 0.7, it generates data as 'Low'. For the source diversity entropy, it transmits a command to calculate the amount of information by applying the Shannon entropy formula to the source distribution of the documents referenced by the answer. For example, if the entropy value is 1.5 or higher, it generates data as 'High'; if it is 0.7 or higher but less than 1.5, it generates data as 'Medium'; and if it is less than 0.7, it generates data as 'Low'. Regarding the above source bias estimation index, an analysis command is transmitted to apply a pre-trained sentiment analysis model to each reference document to calculate positive / negative polarity scores and to calculate the average of their absolute values. For example, judgment data is generated as 'Low' if the average absolute value is less than 0.2, 'Medium' if it is between 0.2 and less than 0.6, and 'High' if it is 0.6 or higher. Regarding the above cognitive consistency index, an analysis command is transmitted to apply a natural language inference model to all sentence pairs within the generated answers to determine 'implication,' 'contradiction,' and 'neutral' relationships. For example, judgment data is generated as 'Low' if even one instance of 'contradiction' is found, 'Medium' if there is no 'contradiction' but 'neutral' relationships account for 50% or more, and 'High' in all other cases. Regarding the above expert reliability, a calculation command is transmitted to calculate the information submitted upon expert registration by converting it into a score according to pre-defined weights.For example, if the total score is 90 points or higher, judgment data is generated as 'High', if it is 70 points or higher but less than 90 points, as 'Medium', and if it is less than 70 points, as 'Low'.

[0089] The final confidence level determined through the above three-step procedure transmits a control signal to be used to directly control the output and operation of the system. If the final confidence level is determined to be 'Highest', the response generation module (800) transmits an output command to display the content along with a 'Expert Verified' badge at the top of the answer. If the final confidence level is determined to be 'High', the response generation module (800) transmits an output command to display a 'High Confidence' badge at the top of the answer. If the final confidence level is determined to be 'Medium', the response generation module (800) transmits an output command to provide a warning message along with the answer, stating, "This answer may contain multiple perspectives or lack consistency from some sources." If the final confidence level is determined to be 'Low', the response generation module (800) does not display the answer by default and controls the user interface so that the user can view the content only by clicking the "View Low Confidence Answer" button. If the final reliability rating is determined to be 'unreliable', the response generation module (800) blocks the generation of a response and outputs a message saying "The probability of error is high, so we cannot provide a response," and automatically reports the query and the result of the determination to an internal review system.

[0090] The reliability evaluation module (700) performs a step-by-step processing procedure for a scenario in which a user asks, "What are the long-term side effects of the new drug 'Solaris'?" First, the system receives an input that generates a draft answer by referring to three related papers and two news articles. Then, it enters a basic reliability grade determination procedure, calculates the average cosine similarity between all documents of the papers and articles as 0.82 and determines the agreement as a 'medium' grade, and calculates the entropy value of the source as 0.91 and determines the diversity as a 'medium' grade. Based on the above determination results and a predefined judgment rule table, it generates data that determines the basic reliability grade as 'B (average)'. Subsequently, it enters a grade adjustment procedure based on potential manipulation possibility, and analyzes the bias of all referenced documents as 'low' and the cognitive consistency of the generated answer as 'high'. Since it is determined that the discrepancy between the bias and the logicality is not large and does not meet the grade adjustment conditions, it generates data that maintains the reliability grade as 'B (average)'. Finally, the process for determining the final grade through expert verification is initiated, requesting a review of the response from an expert in the field of 'oncology,' and receiving a 'valid' evaluation result from an expert with a 'high' level of reliability. Based on the expert verification result and a judgment rule table, the reliability level is raised by one step, and data is generated to finally confirm the grade as 'A (High)'. Finally, the response generation module (800) transmits an output command to provide the response to the user along with a 'high reliability' badge.

[0091] The above reliability evaluation module (700) performs operations to ensure the reliability of the logic and handle exceptions. If it is impossible to obtain data for a specific judgment criterion, the criterion is treated as 'unable to judge' and the evaluation is conducted using only the remaining criteria, but output data is generated by adding a notice to the final grade stating "Some verification was omitted due to insufficient information." Additionally, if expert verification is requested but there is no response within a certain period of time, the grade up to the previous procedure is adopted as the final grade and output data is generated to display the status "This is a response awaiting expert verification."

[0092] Regarding intentionally manipulated fake news scenarios, unlike conventional technology which generates answers containing contradictory content, the present invention controls system operation to prevent damage by determining the 'document mutual integrity consistency' as 'low' in the first procedure and finally displaying a 'low reliability' warning to the user. Furthermore, regarding biased public opinion article scenarios, unlike conventional technology which conveys biased content as if it were objective fact, the invention generates output data to assist the user in critical acceptance by determining the 'source bias estimation index' as 'high' in the second procedure and warning of the possibility of bias in the final result. In particular, regarding scenarios containing minute errors in professional medical information, unlike conventional technology which fails to recognize dangerous errors and provides them as is, the invention controls system operation to protect the user by confirming the final rating as 'unreliable' through an expert's 'error' judgment in the third procedure and blocking the provision of answers.

[0093] The reliability evaluation module (700) operates according to a predefined judgment rule table. In the first procedure, it operates by storing and querying rule data, such as assigning a basic grade 'A' when the conditions of 'high' consistency and 'high' diversity are combined by AND, and assigning a basic grade 'C' when the conditions of 'low' consistency or 'low' diversity occur. In the second procedure, it operates according to a rule that lowers the grade by one level when a discrepancy condition occurs, such as when the conditions of 'high' bias and 'high' cognitive consistency are combined by AND. In the third procedure, it operates according to a rule that raises the grade by one level when the conditions of 'high' expert reliability and 'valid' expert evaluation are combined by AND, and determines the final grade as 'F (unreliable)' when the expert evaluation is 'error'. Furthermore, the present invention demonstrates its effectiveness through reliability judgment simulations for various query types. For example, regarding the 'manipulated statistics' query, even if agreement and diversity are high, a result is output where the base grade 'A' is downgraded to 'B' due to bias and a discrepancy in cognitive consistency; and regarding the 'medical information error' query, even if all other conditions are satisfactory, the response is blocked by ultimately determining it as 'unreliable' based on an expert's 'error' judgment.

[0094] [Table 1] Judgment Rule Table

[0095]

[0096] [Table 1-1]

[0097] The judgment rules presented in this invention are configured to ensure objectivity and necessity. The system is configured so that the data for each judgment criterion is generated by technical means that are academically widely recognized and reproducible in the fields of information retrieval and natural language processing, such as cosine similarity, Shannon entropy, and sentiment analysis models. Furthermore, in order to respond to sophisticated manipulation or biased information threats in the modern information environment, operations are performed according to the technical necessity of adopting a multi-stage logical procedure that includes 'discrepancy judgment' and 'expert verification'.

[0098] By adopting the aforementioned judgment criteria and logical processing procedures, the present invention hierarchically combines multiple independent criteria to effectively filter out false positives that may occur due to the limitations of specific criteria, and ultimately produces the effect of significantly improving the reliability of the provided answers compared to conventional technology. Furthermore, by actively blocking or warning against the provision of answers rated 'unreliable' or 'low,' it prevents users from being defenseless against incorrect or biased information and functions as a safety mechanism to protect users. In particular, the third procedure functions as a key safety mechanism in which the system independently cross-verifies the reliability of answer drafts generated by external large-scale language models, such as OpenAI's Chat model, in one embodiment. That is, the present invention relies on the superior sentence generation capabilities of external models, but is characterized by ultimately guaranteeing the factual accuracy and logical validity of the results through the unique multi-stage evaluation procedure of the present invention.

[0099] In this invention, the term 'semantic cluster' refers to a unit formed by grouping semantically similar sentences or paragraphs using natural language processing technology, and the term 'expert confidence' is defined to refer to the level of confidence assigned to an expert according to objective criteria established by this system. The examples of numerical thresholds presented in this specification are provided so that a person skilled in the art can easily understand and reproduce them; in actual embodiments, these thresholds are not fixed values, but rather the system periodically learns from large-scale document datasets and user feedback logs to identify statistically significant distributions and performs dynamic updates based on these findings. The dynamic update logic controls the system to adapt to new types of information or changing language patterns, thereby ensuring that it always maintains consistent judgment performance.

[0100] [Table 1-2]

[0101]

[0102] Table 1-2 above shows the simulation results for setting the threshold for determining a 'High' grade of 'Document Integrity Consistency'. For this simulation, the average cosine similarity between the documents serving as the basis for the answers was calculated for 10 sample data sets that were previously evaluated as 'Very Reliable' by a third-party expert group. When the 10 calculated similarity values ​​(0.94, 0.88, 0.96, 0.91, 0.89, 0.98, 0.87, 0.93, 0.90, 0.95) are sorted in ascending order, they are (0.87, 0.88, 0.89, 0.90, 0.91, 0.93, 0.94, 0.95, 0.96, 0.98). The present invention is characterized by setting the third quartile (75th percentile) point as the standard for a 'high' grade for statistical stability. The third quartile value of the data is located between the 8th value, 0.95, and the 7th value, 0.94, and can be calculated by the specific formula Q3 = 0.94 + 0.75 * (0.95 - 0.94) = 0.9475. Therefore, the present invention demonstrates that it is statistically valid to set the threshold value of a 'high' grade to 0.9 or higher based on experimental data.

[0103] [Table 1-3]

[0104]

[0105] Table 1-3 above shows the simulation results for verifying the effectiveness of the 'discrepancy judgment' logic, which is a core component of the present invention. In particular, attention should be paid to the case of 'sophisticated fake news' in Simulation 3. Since the prior art evaluates bias and consistency only independently, it may misjudge reliability as 'high' even if bias is high, provided that the logical completeness of the sentence itself is high. However, the present invention applies a rule that determines 'discrepancy occurs' when the bias grade ('high') and the consistency grade ('high') form a specific combination. This rule is implemented with computational logic such as IF (Bias_Score_Grade == 'High' AND Consistency_Score_Grade == 'High') THEN Downgrade_Reliability(). Based on this logic, the present invention achieves a technical effect of improving the judgment accuracy of the system by effectively identifying sophisticated manipulation information missed by the prior art and downgrading the reliability grade.

[0107] The above question-and-answer processing module (600) or a separate resource management module performs the operation of dynamically determining the resource allocation level using multiple judgment criteria in order to efficiently allocate the system's computational resources and maintain stable service quality.

[0108] The resource management module performs a determination operation by applying a query complexity criterion, which indicates the depth of analysis and reasoning required to generate an answer to a query entered by a user. By comprehensively measuring the query length, the number of named entities included, the expertise of the vocabulary used, and the level of sentence ambiguity, it distinguishes between simple fact-checking queries and complex queries requiring in-depth analysis, and transmits a command to predict the minimum necessary resources corresponding to the difficulty of the query. Additionally, the resource management module proceeds with the determination by applying a user interest index criterion, which predicts how many users will show interest in a specific query topic in the present or near future. By collecting and measuring real-time search term rankings, mentions on social media, and views of related news articles from the external data source, it performs operations to recognize in advance the potential for a surge in queries regarding a specific topic, predict system load that may occur due to repetitive queries on the same topic, and prepare for it. Furthermore, the resource management module performs analysis based on an event impact coefficient criterion, which indicates the occurrence of socially significant external events directly related to the query topic and their ripple effects. It continuously monitors the aforementioned real-time breaking news, government agency notices, and announcements from specific companies to transmit control signals for ensuring stability, responding to sudden changes in system demand caused by unpredictable emergencies, and preventing service interruptions. Finally, the resource management module performs a final determination by applying system load criteria that indicate the status of available computational resources in the current system. By comprehensively measuring the CPU utilization, memory occupancy, network bandwidth usage, and the length of waiting task queues, it considers the system's current processing capability to ultimately determine the optimal resource allocation level, and controls the system to function as a protection mechanism to prevent a system-wide crash.

[0109] The resource management module performs the operation of determining the final resource allocation level through a three-stage sequential, conditional judgment procedure based on the four criteria mentioned above. First, it executes a basic resource requirement level determination procedure that calculates a baseline resource requirement by combining the characteristics of individual queries with potential demand. It determines the level of resources the system must fundamentally prepare by synthesizing the complexity of the query and user interest, performing this operation based on technical grounds for efficient resource distribution in everyday situations. Second, it executes an event-based requirement level upward adjustment procedure that forcibly raises the response level by disregarding the predicted basic requirement level in the event of an unexpected situation. If an external event related to the query topic is detected, it interprets this as a strong signal of actual large-scale traffic occurrence, immediately raises the resource requirement level to the highest grade, and performs preemptive preparation. Third, it executes a system state-based final allocation control procedure that confirms the final execution plan by comparing the required resource amount with the system's current supply capacity. Using the current system load rate as the final filter, it allocates resources within the range the system can handle and transmits control commands to adjust the quality of service if necessary.

[0110] The resource management module described above performs the operation of acquiring data for each judgment criterion and determining a grade in a specific manner. Regarding the query complexity, it transmits a data processing command to calculate a complexity score by extracting the number of tokens, the number of named entities, and the number of words included in a predefined specialized terminology dictionary for the input query, respectively, and summing them by multiplying each item by a weight assigned to it. For example, judgment data is generated as 'High' if the complexity score is 30 or higher, 'Medium' if it is 10 or higher but less than 30, and 'Low' if it is less than 10. Regarding the user interest index, it transmits a command to receive the search volume index for the core keywords of the query over the past hour via an external search engine API, and to collect and sum the mention volume within the same time period via a social media API. For example, judgment data is generated as 'High' if the normalized summed value exceeds 50, 'Medium' if it is between 10 and 50 or lower, and 'Low' if it is 10 or lower. Regarding the above-mentioned event impact coefficient, an analysis command is transmitted to parse a real-time news feed and determine whether the core keyword of the query appears together with event-related keywords such as 'breaking news', 'announcement', or 'alert'. For example, if it appears three or more times within 10 minutes along with event-related keywords, judgment data is generated as 'occurred', and otherwise as 'not occurred'. Regarding the above-mentioned system load rate, a command is transmitted to receive CPU average usage and memory occupancy data from an internal system monitoring unit at 1-minute intervals and calculate the average value. For example, if the average value is 80% or higher, judgment data is generated as 'overload'; if it is 50% or higher but less than 80%, as 'warning'; and if it is less than 50%, as 'free'.

[0111] The final resource allocation level determined through the above three-step procedure transmits a control signal to be used to directly control the specific operation of the system. If the final resource allocation level is determined to be 'maximum allocation', the document search module (400) is controlled to perform a maximum depth search (500 documents), and the question-and-answer processing module (600) is controlled to activate a high-performance large-scale language model (e.g., OpenAI's GPT-4 model is set as the API call target). In addition, to increase the search speed for related topics, in one embodiment, a command is automatically transmitted to dynamically increase the ratio of indexes of the Qdrant vector database to be in-memory, or to temporarily increase the number of replicas of the Docker container running the FastAPI backend. If the final resource allocation level is determined to be 'standard allocation', the document search module (400) is controlled to perform a standard depth search, and the question-and-answer processing module (600) is controlled to activate a basic language model. When the final resource allocation level is determined to be 'minimum allocation', the document search module (400) performs only minimum depth search and the question-answer processing module (600) activates a lightweight language model and controls the output of a notice to the user stating, "Simplified answers are provided due to system load."

[0112] The resource management module described above performs a step-by-step processing procedure for a scenario in which a query, "Compile all information regarding the release date and expected specifications of the next-generation smartphone," is entered immediately after a new product announcement event. First, it receives the query, real-time search volume data, breaking news data for "smartphone announcement," and "Borderline" state data indicating a current system load rate of 60%. Subsequently, it enters the basic resource requirement level determination procedure; the complexity score of the query is calculated as 35 to determine it as "High," and the search volume index of related keywords is measured at 95, determining user interest as "High." Based on the above determination results and a predefined judgment rule table, it generates data determining the basic resource requirement level as "High." Next, it enters the event-based requirement level upward adjustment procedure; a related event is detected in the real-time news feed, determining the event impact coefficient as "Occurred." Based on the above determination results and the priority rule of the judgment rule table, it generates data forcibly upgrading the resource requirement level to "Critical." Finally, it enters the system state-based final allocation control procedure, confirms that the current system load is in a 'boundary' state at 60%, and generates data to determine the final resource allocation level as 'maximum allocation' since the resource request level is 'maximum' and the system load is not in an 'overload' state. Finally, the system automatically sends control commands to relevant modules to perform operations such as maximum depth search, activation of the highest performance model, and cache expansion.

[0113] The resource management module described above performs operations to ensure the reliability of the logic and handle exceptions. If data reception from an external API temporarily fails, it treats the criterion as 'undeterminable' and sends a control command to switch to a safe mode that operates based on the most recently successfully received data or solely on query complexity and system load. Additionally, if multiple 'maximum allocation' requests occur simultaneously and there is a concern about a system deadlock, the system controls operations to process them sequentially by applying a priority queue based on the order of query reception and the importance of predefined query types.

[0114] In scenarios where queries surge, such as immediately after a new product launch, the present invention controls system operation to stably handle large-scale traffic by detecting the situation through an event impact coefficient and immediately switching to a 'maximum allocation' level, unlike conventional technologies that cause response times to drastically slow down or server crashes. Furthermore, regarding scenarios during late-night hours with few users, the invention controls system operation to minimize energy consumption by allocating only optimal resources by considering system load and user interest, unlike conventional technologies that waste unnecessary energy. In particular, regarding scenarios where general queries and viral issues occur simultaneously, the invention controls system operation to recognize issues through user interest indices and allocate more resources, while dynamically redistributing resources to ensure minimum processing performance for general queries by considering system load, unlike conventional technologies that delay responses to general queries.

[0115] The resource management module described above operates according to a predefined judgment rule table. If the event impact coefficient is determined to be 'occurring' according to the priority rule of the table, data is generated to determine the requirement level as 'highest'. In the first procedure, rule data is stored, such as assigning a requirement level of 'high' when the complexity 'high' and interest 'high' conditions are combined via AND, and the operation is performed by querying this data. In the third procedure, the operation is performed according to a rule that determines the final allocation as 'maximum allocation' when the requirement level is 'highest' and the system load is not 'overloaded', and determines the final allocation as 'minimum allocation' for all requirement levels when the system load is 'overloaded'.

[0116] [Table 2] Table of Rules for Determining Resource Allocation Levels

[0117]

[0118] The judgment rules used in the present invention are constructed to ensure objectivity and necessity so that there is no room for the inventor's arbitrary judgment to intervene. 'Query complexity,' 'search volume,' 'CPU utilization,' etc., are all objective data that can be quantitatively measured, and the judgment rule table configures the system to perform clear conditional logic based on this objective data. Furthermore, to ensure cost efficiency and service stability in a cloud environment, the dynamic adjustment of resource allocation amounts in accordance with real-time changing supply and demand is performed based on the technical necessity that is absolutely required.

[0119] By adopting the aforementioned judgment criteria and logical processing procedures, the present invention controls the system to avoid allocating excessive resources to unnecessary queries and to use only minimal resources during periods of low system load, thereby producing the effect of significantly reducing the total cost required for system operation. Furthermore, by detecting social events that trigger a surge in queries in advance and preemptively increasing system resources, it produces the effect of maximizing the overall availability of the service by maintaining stable response speeds and preventing server downtime even when a large volume of user requests flood in.

[0120] In this invention, the term "named entity" refers to a specific entity within the text, such as a person, organization, place, or product, that has a unique name, and the term "cache memory" is defined to refer to a high-speed memory that temporarily stores frequently used data to enable rapid access. The threshold values ​​presented as examples in this specification are not fixed values; instead, the system is configured to automatically optimize by continuously machine learning from past query data and system response logs. For example, if response delays frequently occur at a specific threshold, the system learns to automatically lower that threshold to respond more sensitively, and this self-optimization logic operates as a core technical component of this invention.

[0121] [Table 2-1]

[0122]

[0123] Table 2-1 above shows the simulation results for setting a news appearance frequency threshold that activates the 'Event Impact Factor' to 'Occurrence'. For 10 past events, the correlation between the frequency of appearance of related breaking news within 10 minutes immediately following the event and the resulting increase in the actual system query volume was analyzed. The analysis results showed that 'Major' or 'Critical' grade events (No. 1, 2, 3, 4, 8, 10), which saw a surge in query volume of over 500%, commonly had a news appearance frequency of 8 or more. On the other hand, 'Minor' grade events, which showed a negligible increase in query volume, all had an appearance frequency of 4 or fewer. Based on this data distribution, a calculation formula for early detection of events that have a substantial impact on system load can be defined as IF (News_Frequency_in_10min >= Threshold_N) THEN Activate_Event_Factor(). The results of this simulation demonstrate that setting the threshold (Threshold_N) between 5 and 8 times, for example to 5 times, is a reasonable criterion for predicting the actual surge in query volume.

[0124] [Table 2-2]

[0125]

[0126] Table 2-2 above presents the simulation results verifying the effectiveness of the 'preemptive resource allocation' logic of the present invention over time. The prior art employs a reactive approach that increases resources only after waiting for the system load to actually exceed a threshold. In this case, as shown in Simulation 4, the response time drops sharply to 5.8 seconds the moment the load surpasses the threshold, significantly compromising the user experience. In contrast, the present invention preemptively switches resource allocation to the 'maximum' level the moment the news appearance frequency exceeds a set threshold of 5 in Simulation 3, even though the system load has not yet reached the threshold. This preemptive measure is implemented using computational logic such as Resource_Level = MAX IF Event_Activated == TRUE ELSE Current_Resource_Level. As a result, as shown in Simulations 4 and 5, it demonstrates a technical effect of stably maintaining the response time at less than 1 second, even during periods of explosive increase in actual query volume.

[0128] The document search module (400) performs an operation of determining a contextual time weight using multiple judgment criteria to evaluate the temporal value of information in three dimensions when adjusting the ranking of the searched documents or when the response generation module (800) constructs an answer.

[0129] The document search module (400) performs a judgment operation by applying a topic-specific time sensitivity criterion, which is a unique attribute indicating how quickly the value of information changes over time. By preventing the error of evaluating all information on the same time scale and applying a differentiated time value evaluation model suited to the characteristics of the topic, it transmits a control signal to evaluate information that is timely in chronological order and information that is not timely based on the depth of content. Additionally, the document search module (400) conducts an evaluation by applying a topic persistence criterion that measures how long a specific topic receives continuous social interest. It controls the system operation to prevent the value of past information from being devalued too hastily by correcting the time sensitivity and adjusting the natural depreciation rate of information to match the life cycle of the topic. Furthermore, the document search module (400) performs an analysis according to a current interest criterion that measures the degree to which the topic of the document is receiving public attention again at the current time, regardless of the time of creation of the document. The value of the aforementioned 'forgotten' past information is re-evaluated, and a processing command is transmitted to temporarily increase the temporal value of documents related to old original works to meet the information needs of current users. Finally, the document search module (400) performs a final determination by applying event relevance criteria to determine whether the document is directly related to a specific major event that recently occurred. By reflecting a factor that increases temporal value more powerfully and directly than the current interest, the system captures a situation where the importance of information changes rapidly starting from a specific event and controls the system to expose the information at the top of the search results.

[0130] The document search module (400) views the temporal value of information as a combination of 'intrinsic value' and 'added value' and performs the operation of calculating a final weight through a three-stage logical processing procedure. First, it executes a basic temporal value determination procedure that evaluates the inherent temporal lifecycle of the information. By comprehensively considering the time sensitivity and subject persistence of the subject to which the document belongs, it calculates the basic value of the information that naturally decreases over time, and performs this operation according to technical grounds refined to reflect the characteristics of the subject. Second, it executes an additional temporal value determination procedure that evaluates the additional value that the current social and cultural context assigns to the information. By analyzing the current public interest or the association with specific events, it calculates the additional temporal value to be assigned to the document, and performs this operation according to technical grounds that reflect the reality that the value of information is constantly re-evaluated within the social context. Third, it executes a final weight determination and correction procedure that determines the final weight by logically combining the calculated basic value and the added value according to a specific calculation formula. In one embodiment, each value grade ('A', 'B', 'C') is converted into a numeric score (e.g., A=1.0, B=0.6, C=0.2), and then an intermediate weight score is calculated through a weighted summation formula such as intermediate weight score = (basic value score * 0.4) + (added value score * 0.6). Subsequently, if the added value is determined to be higher than a certain level, such as grade 'A', a minimum value guarantee logic such as IF (added value grade == 'A') THEN final weight score = MAX(intermediate weight score, 0.8) is applied to correct the final weight so that it does not fall below a certain level (e.g., 0.8) even if the basic value is low.This operates according to inevitable technical procedures designed to prevent information with important current context from being omitted from search results simply because it is outdated.

[0131] The document search module (400) performs the operation of acquiring data for each judgment criterion in a specific manner and determining a grade. Regarding the time sensitivity by topic, for the topic category extracted from the query, a data processing command is transmitted to analyze the average edit cycle of Wikipedia documents belonging to that category. For example, if the average edit cycle is within one month, judgment data is generated as 'High'; if it is between one month and one year, as 'Medium'; and if it is over one year, as 'Low'. Regarding the topic persistence, a command is transmitted to receive monthly search volume data for the past five years of the topic keyword through the Google Trends API and to calculate the coefficient of variation of the data. For example, if the coefficient of variation is less than 0.2, judgment data is generated as 'High'; if it is between 0.2 and less than 0.8, as 'Medium'; and if it is 0.8 or higher, as 'Low'. Regarding the current interest, an analysis command is transmitted to measure the increase rate of search volume within the last 24 hours for the core keyword of the document. For example, judgment data is generated as 'High' if it increases by 200% or more compared to the previous day, 'Medium' if it increases by 50% or more but less than 200%, and 'Low' if it increases by less than 50%. Regarding the above event correlation, an analysis command is sent to determine whether the creation date or content of the relevant document is located within ±3 days of the occurrence time of a major news event that occurred within the last 7 days. For example, judgment data is generated as 'Present' if the condition is satisfied, and 'None' if it is not satisfied.

[0132] The final contextual time weight determined through the above three-step procedure transmits a control signal to be used to directly control the system's search and response generation processes. In one embodiment, the document search module (400) recalculates the final ranking score by applying the final time weight calculated through this process to the similarity score of each document (e.g., weighted multiplication or addition) for the top K candidate documents returned from the Qdrant vector database along with semantic similarity scores. Subsequently, it automatically transmits a command to reorder the search results in order of highest recalculated scores. This is characterized by operating as a 're-ranking' step that reorders the initial vector search results (in order of similarity) according to the temporal context. Additionally, when the response generation module (800) extracts answer candidate sentences from multiple documents, it controls the final time weight assigned to the source document of each sentence to the confidence score of the corresponding sentence to increase the probability that a sentence extracted from a more timely document will be included in the final answer. Finally, when providing answers to the user, the user interface is controlled to intuitively understand the temporal context of the information by assigning tags such as 'information related to the latest event' and 'information with a recent surge in interest' along with the source of the referenced document.

[0133] The document search module (400) performs a step-by-step processing procedure for a scenario in which a query "evaluation of the movie 'Parasite'" is entered immediately after 'Parasite' wins an Academy Award. First, it receives input data for a past movie review document A written two years ago, a recent news article B written 12 hours ago, and the event data of 'Parasite winning an Academy Award'. Then, it enters a basic time value determination procedure for the past review document A, and since the sensitivity of the topic 'movie evaluation' is 'low' and the persistence is 'high', it generates data in which the basic time value is determined to be 'B (average)' despite two years having passed. Subsequently, it enters an additional time value determination procedure for the same document A, and determines the current interest level as 'high' because the search volume of the keyword 'Parasite' has surged by 5000% compared to the previous day, and determines the event correlation as 'present' because a direct correlation with the 'Academy Award winning' event is determined. Based on the above judgment result and a predefined judgment rule table, data is generated in which the added time value is determined as 'A (High)'. Finally, the final weight determination and correction procedure is entered, the base value 'B' and the added value 'A' are combined, and data is generated in which the final time weight is confirmed upward to 'A (High)' according to the correction rule of the judgment rule table, since the added value is 'A'. Finally, the document search module (400) automatically transmits a command to control the search ranking so that both past review document A and the latest news B are given high weights and are displayed together at the top of the search results.

[0134] The document search module (400) performs operations to ensure the reliability of the logic and handle exceptions. If the subject of the query does not exist in a predefined category, the system sends an exception handling command that inherits the time sensitivity of the most similar parent category or uses the average sensitivity value of all subjects as the default value. In addition, to prevent the current interest from increasing abnormally due to temporary abuse, the system sends a data processing command that calculates a moving average value together to correct short-term outliers.

[0135] Regarding scenarios involving the re-examination of past technologies, the present invention controls system operation to ensure the depth of information by determining current interest as 'high,' thereby assigning high added value to past original papers and consequently exposing key original documents at the top, unlike conventional technologies that relegate core original documents to lower priority. Furthermore, regarding query scenarios concerning outdated topics, the invention controls system operation by determining the time sensitivity of the topic as 'low,' thereby allowing other quality indicators, such as content fidelity, to have a greater influence on ranking than the time of document creation, unlike conventional technologies that reduce the reliability of information. In particular, regarding query scenarios concerning recurring annual events, unlike conventional technologies that expose low-quality recent content at the top, the invention controls system operation by learning topic persistence and periodic patterns of rising current interest to assign high weight to high-quality content that remains valid even if it was written last year.

[0136] The document search module (400) operates according to a predefined judgment rule table. In the first procedure, it operates by querying rule data that determines the depreciation rate of the base value as 'very low', 'medium', or 'very high' based on a combination of sensitivity and persistence. In the second procedure, it operates according to a rule that determines the additional time value as 'A (high)' when the current interest level is 'high' or the event correlation is 'present'. In the third procedure, it operates according to a correction rule that determines the final weight for all base values ​​as 'A (high)' when the additional time value is 'A (high)'.

[0137] [Table 3] Contextual Time Weighting Judgment Rule Table

[0138]

[0139] The judgment rules used in this invention are constructed to ensure objectivity and necessity, leaving no room for the inventor's subjectivity to intervene. 'Average editing cycle,' 'coefficient of variation,' 'search volume growth rate,' etc., are all objective indicators that can be directly calculated from data, and the system is configured to determine grades based on these indicators. Furthermore, in a modern society where information is exploding, it performs operations in accordance with the technical necessity of overcoming the limitations of the perspective that only the latest information is valuable, and of providing users with truly meaningful and contextually relevant information.

[0140] By adopting the aforementioned judgment criteria and logical processing procedures, the present invention goes beyond merely listing superficial, up-to-date information to provide key source information or in-depth analysis data that is important in the current context, thereby enhancing the overall depth of search results and user satisfaction. Furthermore, it provides high-quality content that was once popular but has been forgotten with the opportunity to reappear at the top of search results when it connects with new events or interests, thereby increasing the utilization of the entire content held by the system and contributing to the establishment of a sustainable information ecosystem.

[0141] As used in this invention, the term 'edit cycle' refers to the average time from when a specific online document is edited once until the next edit is made, and the term 'coefficient of variation' is defined to refer to a statistical measure representing the relative variability of data. The thresholds exemplified in this specification are not fixed constants but are set based on the statistical distribution of data across all subject categories, and the system performs the operation of periodically readjusting each threshold by learning information consumption patterns that change over time. The dynamic readjustment function transmits commands that ensure the system can continuously adapt to new trends or changing user behavior.

[0142] [Table 3-1]

[0143]

[0144] Table 3-1 above presents the simulation results for setting the threshold for the search volume increase rate to determine the 'Current Interest' grade. For 10 past cases, the relationship between the search volume increase rate within 24 hours and the change in the actual user click-through rate (CTR) of documents related to the topic was analyzed. The analysis results showed that cases where the CTR increased significantly by more than 5% (No. 1, 2, 3, 4, 8, 9) commonly exhibited a pattern where the search volume increase rate exceeded 200%. This implies that an increase in search volume has a high positive correlation with actual user interest. Therefore, the formula for determining the 'Current Interest' grade can be defined as Interest_Grade = 'High' IF (Search_Volume_Increase_Rate >= 2.0). The results of this simulation demonstrate that setting the threshold for the 'High' grade to 200% is an objective criterion that reflects changes in users' actual interest.

[0145] [Table 3-2]

[0146]

[0147] Table 3-2 above is a search result ranking simulation assuming the situation immediately after the movie 'Parasite' wins the Academy Award. Since the prior art simply places a large weight on the time of creation, a problem arises where a core review (DOC-02) with depth of content is pushed down to 15th place simply because it is a document from two years ago. However, the present invention assigns an added value grade of 'A' to DOC-02 based on 'high current interest'. Subsequently, a basic ranking calculation such as Final_Rank_Score = Base_Value_Score * w1 + Bonus_Value_Score * w2 is performed, while applying a minimum value guarantee logic as IF (Bonus_Value_Grade == 'A') THEN Final_Rank_Score = MAX(Final_Rank_Score, Minimum_Guarantee_Score). Thanks to this correction logic, even if the base value is somewhat low, the final ranking of DOC-02, which has a high added value, surges to 2nd place and is exposed to users along with the latest news. This demonstrates that the present invention has a technical effect of preserving the contextual importance of information.

[0149] The above response generation module (800) performs an operation to determine whether there is duplication based on information contribution using multiple judgment criteria in order to verify the quality of the generated candidate answers and evaluate the actual value of the information.

[0150] The response generation module (800) performs a judgment operation by applying a semantic cluster criterion, which is a unit that groups various information segments referenced in the response generation according to semantic similarity. It groups segments that convey virtually identical information, even though their expressions differ, into a single cluster to identify the number of actual types of information and transmits a control signal to prevent the response from repeating the same content by merely changing the expression. Additionally, the response generation module (800) conducts an evaluation by applying an information scarcity criterion that measures how rarely the information conveyed by a specific semantic cluster appears within the entire document database held by the system. By evaluating the rare information that is not easily found as more valuable, it controls the system operation so that the response does not merely list widely known facts but includes information that can provide new insights to the user. Furthermore, the response generation module (800) performs an analysis according to an expert credibility criterion that evaluates the expertise of the source or author of the original document supporting the content of a specific semantic cluster. By assigning high value to verified information from the aforementioned rare yet reliable sources, a processing command is transmitted to prevent the indiscriminate inclusion of unverified and rare claims in the answer and to ensure the accuracy of the information.

[0151] The above response generation module (800) performs a three-stage value evaluation and judgment procedure to determine duplication based on the quality rather than the quantity of information. First, it executes a semantic clustering and contribution evaluation procedure to first evaluate the actual value of individual pieces of information constituting the answer. After grouping all referenced information segments into semantic clusters, it assigns a 'final contribution grade' to each cluster by combining information scarcity and expert trustworthiness through a specific calculation formula. In one embodiment, after the scarcity and trustworthiness grades are each converted into numerical scores (e.g., High = 1.0, Medium = 0.5, Low = 0.1), the final contribution grade is calculated through a non-linear combination calculation formula such as Final Contribution Grade = (Scarcity Score) * (Trustworthiness Score ^ 1.5). The above calculation formula drastically reduces the contribution of information with low expert trustworthiness even if it has high scarcity, thereby preventing unverified information from being overvalued, and this performs an operation as a preliminary step for individually quantifying the value of each component. Second, a procedure for determining the level of substantial duplication is executed to analyze the distribution of informational value across the entire response and determine whether duplication exists. The proportion of information with the aforementioned 'low contribution' within the entire response is calculated; if this proportion exceeds a specific threshold, the response is determined to be qualitatively poor and substantially duplicated. This operation is performed based on technical grounds for evaluating the overall quality of the response from a macroscopic perspective. Third, a procedure for reconstructing the response is executed to discard responses determined to be duplicates and to control the system to regenerate responses centered on high-quality information. Utilizing the aforementioned determination results as feedback, semantic clusters with 'high contribution' are prioritized to trigger the process of reconstructing the response. This functions as a cyclical quality improvement mechanism in which the system critically evaluates and improves its own generated results.

[0152] The above response generation module (800) performs the operation of acquiring data for each judgment criterion in a specific manner and determining a grade. For the semantic cluster, it transmits a data processing command to convert all information segments referenced in the answer generation into vectors through a sentence embedding model, and to group vectors of similar meanings into the same cluster by applying a density-based clustering algorithm such as DBSCAN. For the information sparsity, for the core keyword of a specific semantic cluster, it transmits a command to calculate an inverse document frequency value representing the ratio of documents containing the keyword in the entire document database. For example, if the inverse document frequency value is within the top 10%, it generates judgment data as 'High'; if it is between the top 10% and 50%, it generates data as 'Medium'; and if it is above the top 50%, it generates data as 'Low'. For the expert reliability, it transmits an analysis command to identify the source of the original document of each semantic cluster and to query the reliability score of the source in a pre-established 'source reliability database'. For example, if the confidence score is 90 points or higher, it is judged as 'High', if it is 60 points or higher but less than 90 points, it is judged as 'Medium', and if it is less than 60 points, it is judged as 'Low'.

[0153] The determination result derived through the above three-step procedure transmits a control signal to be used to directly control the operation of the response generation module (800). If it is determined that there is no duplicate, a command is automatically transmitted to confirm the generated candidate answer as the final answer and provide it to the user. If it is determined that there is a duplicate, a command is transmitted to immediately discard the currently generated candidate answer, and only semantic clusters with a contribution of 'best' or 'high' grade are set as required inclusion elements, and the response generation module (800) is controlled to perform answer reconstruction based on the required inclusion elements. More specifically, the response generation module (800) is characterized by removing reference texts corresponding to semantic clusters with a contribution of 'low' grade from the prompt to be delivered to a large-scale language model (e.g., OpenAI API), modifying the prompt to include only the content of clusters with a contribution of 'best' or 'high' grade, and then controlling the question-answer processing module (600) to call the language model again with the modified prompt. The system controls the reconstructed answer to perform a duplicate determination procedure again, and to repeat this process until it is determined not to be a duplicate or the maximum number of reconstructions is reached.

[0154] The above response generation module (800) performs a step-by-step processing procedure for a scenario in which candidate answers are generated by referencing 10 information segments in response to the query "Tell me about the latest commercialization trends of quantum computing." First, it receives 10 information segments and a related database. Then, it enters a semantic clustering and contribution evaluation procedure to group the 10 segments into 4 semantic clusters, and evaluates information sparsity and expert reliability for each cluster, thereby generating judgment data in which Cluster A has a contribution grade of 'Low', Cluster B has a 'High', Cluster C has a 'High', and Cluster D has a 'Low' grade. Subsequently, it enters a substantial redundancy level determination procedure, and since there are 2 clusters with a contribution grade of 'Low' among the total 4 clusters, the ratio of low contribution clusters is calculated as 50%. Since the calculated ratio exceeds a predefined threshold of 30%, it generates data in which the current candidate answer is determined to be 'substantial redundant'. Finally, it enters the answer reconstruction control procedure, sends a command to discard the currently generated answer, and automatically sends a control command to reconstruct a more concise and essential answer focusing only on the contents of clusters B and C, which have a contribution rating of 'High'.

[0155] The above response generation module (800) performs operations to ensure the reliability of the logic and handle exceptions. If the contribution grade of all semantic clusters is determined to be 'low', the system sends a control command to prevent premature information provision by outputting a message saying "Cannot find sufficient reliable information for the current query" instead of attempting to reconstruct the answer. Additionally, if calculating information sparsity for the entire database places a heavy load on the system, the system sends a command to increase calculation efficiency by utilizing sampled indexes of the entire database or pre-calculated tables by topic.

[0156] Regarding scenarios involving the repetition of common sense content, the present invention controls system operation to reconstruct the response by retaining only the most concise and essential explanations, unlike conventional technology which generates long, verbose answers lacking new information, by determining the information scarcity of the content as 'low' and classifying it as 'duplicate.' Furthermore, regarding scenarios containing unverified information, unlike conventional technology which includes dangerous information as is, the invention controls system operation to protect the user by determining the expert reliability of the information as 'low,' assigning a contribution level of 'low,' and consequently excluding it from the response reconstruction process. In particular, regarding scenarios where key information is omitted, unlike conventional technology which dilutes the core message, the invention controls system operation to ensure that key information is not omitted by evaluating the contribution level of each piece of information and, upon a 'duplicate' determination, mandating the reconstruction of the response centered on information with high contribution.

[0157] The above response generation module (800) operates according to a predefined judgment rule table. In the first procedure, it operates by querying rule data such as assigning a contribution grade of 'highest' when the scarcity 'high' and expert reliability 'high' conditions are combined by AND, and assigning a contribution grade of 'low' when the scarcity 'low' or expert reliability 'low' conditions occur. In the second procedure, it operates according to a rule that determines the judgment result as 'substantial duplication' when the ratio of low contribution grade clusters exceeds 30%.

[0158] The judgment rules used in this invention are configured to ensure objectivity and necessity so as to exclude the inventor's arbitrary judgment. 'Reverse document frequency' is a standard statistical indicator in the field of information retrieval, and the 'source reliability database' is configured to utilize data built according to objective criteria. Furthermore, it performs operations in accordance with the indispensable technical necessity of overcoming technical limitations of generative AI, such as 'plausible lies' or content repetition, and securing reliability.

[0159] [Table 4]

[0160]

[0161] By adopting the aforementioned judgment criteria and logical processing procedures, the present invention eliminates unnecessarily repetitive or common-sense content and compresses answers around rare and highly reliable information, thereby producing the effect of improving the information density of answers and user satisfaction so that users can obtain more valuable information by investing the same amount of time. In addition, by actively filtering unverified rare information or information from low-reliability sources during the answer reconstruction process, it produces the effect of solving the chronic reliability issues of generative AI and enabling safe AI service operation.

[0162] In this invention, the term 'information segment' is defined to refer to the minimum semantic unit constituting an answer, and 'inverse document frequency' is defined to refer to a measure indicating the sparsity of words. The threshold values ​​exemplified in this specification are not fixed constants but are set by analyzing the correlation with user satisfaction evaluation data, and the system performs the operation of automatically tuning the threshold values ​​by continuously learning from user feedback data.

[0163] [Table 4-1]

[0164]

[0165] Table 4-1 above presents the simulation results for setting a threshold for the proportion of low-contribution clusters to determine 'substantial overlap'. For 10 generated answers, the proportion of semantic clusters with low contribution was calculated, and the correlation with actual user satisfaction rating scores for those answers was analyzed. The analysis revealed that answers with satisfaction scores of 3 points or less (average or dissatisfied) (Nos. 2, 4, 6, 7, 8, 10) all had a low-contribution cluster proportion of 33% or more. In contrast, for answers with a satisfaction score of 4 points or more, the proportion was 25% or less. This data suggests the existence of an inflection point where user satisfaction drops sharply as the amount of low-contribution information increases. Therefore, the formula for determining 'actual redundancy' can be defined as IF (Low_Contribution_Cluster_Ratio >= Threshold_R) THEN Judgment = 'Redundant', and the results of this simulation prove that setting the threshold (Threshold_R) to 30% (0.3) is a reasonable standard that reflects the point at which users perceive actual quality degradation.

[0166] [Table 4-2]

[0167]

[0168] Table 4-2 above presents the simulation results verifying the effectiveness of the 'answer reconstruction' logic of the present invention. As seen in the cases of ANS-A and ANS-C, the initially generated answers contained a large amount of information with low contribution (common sense, unverified claims, etc.), causing the ratio of low contribution clusters to exceed a threshold (30%). The prior art failed to detect this qualitative degradation and provided answers as is, resulting in low user satisfaction. However, the present invention identifies these as 'substantial duplicates' and triggers reconstruction operation logic such as Reconstruct_Answer(answer) WHERE Contribution_Grade IN ('Critical', 'High'). As a result, low contribution clusters are removed, and high-quality answers compressed around core information are generated, ultimately leading to high satisfaction. This demonstrates that the present invention has a technical effect of substantially improving the quality of answers through an active reconstruction process, going beyond merely determining duplicates.

[0170] The present invention sequentially performs the following organic operations in response to a scenario in which a user inputs a query saying, "Tell me about the latest research trends and safety of COVID-19 vaccines."

[0171] First, the user interface module (300) receives a query text input by the user and immediately transmits it to the question-response processing module (600) described later. Subsequently, the question-response processing module (600) analyzes that the received query includes multiple concepts such as 'latest trends' and 'stability', and based on this, transmits the analyzed query to the document search module (400) described later, along with a control command to operate in 'multiple document combination mode'. Upon receiving the control command, the document search module (400) utilizes the vector index stored in the document storage unit (200) to search for three recent academic papers and two reports from credible institutions that are semantically most related to the query, and transmits the searched document data to the document analysis module (500) described later.

[0172] The document analysis module (500) that receives the above document data performs an analysis operation to divide and structure the entire content of five documents into semantically related paragraphs, and transmits the set of semantic paragraphs generated as a result to the question-answer processing module (600). The question-answer processing module (600) utilizes the received semantic paragraphs as rich contextual information for generating answers, generates a draft of an answer through a pre-trained artificial intelligence language model, and simultaneously transmits the generated draft of the answer and the reference document information that served as the basis for the answer to the reliability evaluation module (700) and the response generation module (800) described later.

[0173] The reliability evaluation module (700) that receives the above-mentioned draft response performs the reliability determination procedure described in detail in [Process 1] of the present invention. It confirms that the sources of all the referenced documents are highly reliable academic institutions and public institutions, and that the content is also mutually consistent, and determines the final reliability grade as 'A (High)', and transmits the determined grade information to the response generation module (800) described later. Almost simultaneously, the response generation module (800) performs an information contribution analysis described in detail in [Process 4] of the present invention on the received draft response and determines that the response is not substantially duplicated. In addition, the response generation module (800) confirms that the default response settings for the user's account are 'Detailed' and 'Expert' modes, and performs a final review to determine whether the currently generated response conforms to those settings.

[0174] When all reviews and evaluations are completed, the response generation module (800) combines the finally confirmed answer body, the 'high reliability' badge UI information received from the reliability evaluation module (700), and the list of sources of the referenced documents into a single completed response object, and transmits the combined final response object to the user interface module (300). Finally, the user interface module (300) visually displays all information included in the received response object on the user screen to complete the operation of providing a highly reliable and detailed answer to the user.

[0175] Although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. Various modifications are possible by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention. Explanation of the symbols

[0176] Data collection unit (100) Document storage unit (200) User interface module (300) Document search module (400) Document Analysis Module (500) Question and answer processing module (600) Reliability evaluation module (700) Response generation module (800)

Claims

Claim 1 A data collection unit for collecting document data from an external source; a document storage unit for storing the collected document data; a user interface module for receiving a user's query; a document search module for searching for documents related to the query among the stored document data; a document analysis module for analyzing the content of the searched documents and dividing them into semantic units; a question-and-answer processing module for generating a draft answer to the query using the divided semantic units; and a reliability evaluation module for determining the reliability of the generated draft answer. The system includes a response generation module that generates a final response based on the reliability determined above; wherein the reliability evaluation module determines a reliability grade by synthesizing the internal characteristics of the documents referenced in generating the answer draft and the logical characteristics of the answer draft itself, and the question-answer processing module dynamically determines the level of computational resource allocation of the system by synthesizing the characteristics of the query, external data, and the current state of the system; and wherein the reliability evaluation module comprises: a first procedure for determining a basic reliability grade by combining document mutual integrity consistency, which indicates the semantic consistency between multiple referenced documents, and source diversity entropy, which indicates the source diversity of the referenced documents; and a second procedure for downgrading the basic reliability grade by analyzing source bias based on the linguistic expression of the referenced documents and cognitive consistency based on the internal sentence structure of the generated answer draft, respectively, and determining whether the difference between the two analysis results exceeds a preset first threshold, such as when the source bias is high and the cognitive consistency is also high. and a third procedure for determining the final confidence level by reflecting the results of the 'validity' or 'error' verification by a registered expert on the confidence level adjusted through the second procedure above;The method is characterized by improving the reliability of response results by blocking the provision of answers containing sophisticatedly manipulated information or expert-level errors by sequentially executing the following steps: the question-answer processing module comprises: a first procedure for calculating query complexity by analyzing the length of the user query and the number of included entities, calculating user interest by collecting real-time search volume and social media mention volume from external data sources, and determining a basic resource requirement level by combining the calculated query complexity and user interest; and a second procedure for monitoring a real-time news feed to detect whether an unexpected event related to the topic of the query has occurred, and, if the unexpected event is detected, forcibly raising the resource requirement level to a preset higher level regardless of the basic resource requirement level determined in the first procedure. A document-based question-answering system characterized by sequentially performing a third procedure to check the system load rate based on the central processing unit (CPU) usage and memory occupancy of the current system, and to determine the final resource allocation amount at a level one step lower than the resource requirement level adjusted in the second procedure if the system load rate is in an 'overload' state, thereby preemptively responding to a surge in queries to secure service stability and allocating resources according to the system's current processing capacity to reduce operating costs; wherein the reliability evaluation module acquires the document mutual integrity agreement by converting the core sentences of each referenced document into vectors using a natural language processing embedding model, calculating the cosine similarity between all pairs of vectors, and calculating the average value; acquires the source diversity entropy by calculating the amount of information by applying the Shannon entropy formula to the source distribution of the referenced documents; and acquires the source bias by applying a pre-trained sentiment analysis model to each referenced document to calculate the average of the absolute values ​​of positive or negative polarity scores. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 A document-based question-and-answer system according to claim 1, wherein when the final resource allocation is determined by the question-and-answer processing module to be at the 'maximum allocation' level, the document search module is controlled to search for a number of documents greater than the pre-set standard depth, and the question-and-answer processing module is controlled to activate a high-performance language model having parameters larger than the pre-set basic language model; and when the final resource allocation is determined by being at the 'minimum allocation' level, the document search module is controlled to search for a number of documents less than the standard depth, and the question-and-answer processing module is controlled to activate a lightweight language model having parameters smaller than the basic language model. Claim 6 In claim 1, the document search module is controlled to operate in a 'context-based filtering mode' that extracts only the paragraphs most similar to the query within the top-level documents found when the question-answer processing module analyzes the query and determines it to be a question about a single concept, and to operate in a 'multi-document combination mode' that returns all related top-level documents when the question-answer processing module determines it to be a question asking for a comparison between multiple concepts; and the document analysis module is characterized by dividing the received document text into sentence units, sequentially calculating the similarity between the embedding vectors of consecutive sentences, identifying a point where the similarity value drops sharply below a preset second threshold as a 'semantic boundary,' and grouping related sentences into one group based on the identified boundary to generate a semantic paragraph.

Images