AI Request Execution Path Control Based on Net Utility Score Device and Method Thereof

The execution path control device and method address inefficiencies in LLMs by quantifying and evaluating AI request utility, cost, and risk, optimizing resource usage and security through dynamic path control, enhancing efficiency and reducing costs.

KR103044589B1Active Publication Date: 2026-11-06BPMG CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
KR1020250197791
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Filing Date
2025-12-12
Publication Date
2026-11-06
Estimated Expiration
2045-12-12

AI Technical Summary

Technical Problem

Current AI execution engines, such as Large-Scale Language Models (LLMs), face issues with excessive computational resource usage, increased execution costs, and security risks due to meaningless or duplicate requests, spam queries, and the processing of sensitive information without adequate control, leading to reduced service responsiveness and wasted resources.

Method used

An execution path control device and method that quantifies and evaluates the utility, cost, and risk of AI requests using a Net Utility Score (NUS) to dynamically control the processing path, processing complex requests via high-performance paths, simple requests via low-cost paths, and sensitive information via blocking or human review paths, while suppressing unnecessary model calls.

Benefits of technology

This approach improves LLM processing efficiency and security by optimizing resource usage and reducing unnecessary computations and costs, ensuring secure and efficient handling of diverse request types.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 112025140883544-PAT00001_ABST
    Figure 112025140883544-PAT00001_ABST
Patent Text Reader

Abstract

The present invention relates to an execution path control device and method based on the net utility score of an AI request, which prevents excessive calls to an LLM by evaluating the utility, cost, and risk of an input request to calculate a net utility score and dynamically controlling the execution path where the AI ​​request is processed according to the result. First, the execution path control device for an AI request based on net utility score of the present invention, upon receiving LLM question data, preprocesses it using natural language processing, decomposes the sentence into phrase units to generate a dependency phrase tree, extracts a first intent candidate through a pre-configured dependency relationship model, vectorizes the LLM question data into an embedding model, performs a proximity vector search on an intent embedding database to extract a second intent candidate, and combines the first intent candidate and the second intent candidate to generate an intent candidate set; an intent analysis module that receives the intent candidate set from the intent analysis module, calculates inclusion relationship scores between each intent candidate through a pre-configured inclusion relationship score calculation model, assigns weights according to the calculated inclusion relationship scores, determines the intent candidate whose sum of these scores is maximized as the core intent of the LLM question data, and determines at least one of the target word, constraint phrase, and domain within the sentence of the LLM question data based on the determined core intent to generate a first signature; and in the signature generation module A duplicate request blocking module that calculates similarity between vectors by comparing the embedding vector of the generated first signature with the embedding vectors of existing signatures generated prior to the first signature, and reuses an existing response corresponding to an existing signature if the calculated similarity is greater than or equal to a preset reuse threshold; an intent evaluation module that, after receiving the first signature generated by the signature generation module, calculates a utility score of the LLM question data based on a preset utility score calculation standard, calculates a cost score based on a preset cost score calculation standard, or calculates a risk score based on a preset risk score calculation standard if the similarity calculated by the duplicate request blocking module is less than the preset reuse threshold; a net utility calculation module that receives one of the utility score, cost score, or risk score from the intent evaluation module, inputs it into a preset net utility score formula, assigns weights to the cost score or risk score, and calculates the net utility score; and the net utility score calculated by the net utility calculation module is If it is above the preset utility threshold, select the first model path, andIt includes a path determination module that selects a second model path when the net utility score is less than the utility threshold. In addition, the method for controlling execution paths based on net utility scores of AI requests comprises: (a) when an intent analysis module receives LLM question data, preprocesses it using natural language processing, decomposes the sentence into phrase units to generate a dependency phrase tree, extracts a first intent candidate through a pre-configured dependency relationship model, vectorizes the LLM question data into an embedding model, performs a proximity vector search on an intent embedding database to extract a second intent candidate, and combines the first intent candidate and the second intent candidate into one to generate a set of intent candidates; (b) after a signature generation module receives the set of intent candidates from the intent analysis module, calculates inclusion relationship scores between each intent candidate through a pre-configured inclusion relationship score calculation model, assigns weights according to the calculated inclusion relationship scores, determines the intent candidate with the maximum sum of these scores as the core intent of the LLM question data, and determines at least one of the target word, constraint phrase, and domain within the sentence of the LLM question data based on the determined core intent to generate a first signature; (c) (d) a step in which the duplicate request blocking module compares the embedding vector of the first signature generated by the signature generation module with the embedding vectors of existing signatures generated prior to the first signature to calculate the similarity between the vectors, and if the calculated similarity is greater than or equal to a preset reuse threshold, reuses the existing response corresponding to the existing signature; (d) a step in which, after the intent evaluation module receives the first signature generated by the signature generation module, if the similarity calculated by the duplicate request blocking module is less than the preset reuse threshold, calculates the utility score of the LLM question data based on a preset utility score calculation standard, calculates the cost score based on a preset cost score calculation standard, or calculates the risk score based on a preset risk score calculation standard; (e) a step in which the net utility calculation module receives one of the utility score, cost score, or risk score from the intent evaluation module, inputs it into a preset net utility score formula, andThe method includes the step of calculating a net utility score by assigning weights to cost scores or risk scores, and (f) the step of the path determination module selecting a first model path if the net utility score calculated by the net utility calculation module is greater than or equal to a preset utility threshold, and selecting a second model path if the net utility score is less than the utility threshold.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present invention relates to an execution path control device and method based on a net utility score of an AI request for operational optimization of a large-scale language model (LLM) execution engine. More specifically, the present invention relates to an execution path control device and method based on a net utility score of an AI request that prevents excessive calls to the LLM by evaluating the utility, cost, and risk of an input request to calculate a net utility score and dynamically controlling the execution path where the AI ​​request is processed according to the result. Background Technology

[0002] Current AI execution engines, such as Large-Scale Language Models (LLMs) or Multimodal AI models, require massive computational resources and token usage, resulting in significantly increased execution costs and response latency.

[0003] In particular, meaningless or duplicate requests and spam queries are frequently incoming, and some requests include personally identifiable information such as names, resident registration numbers, and bank account numbers, leading to skyrocketing costs and serious security risks. Furthermore, there is a problem where unnecessary computations and delays accumulate because the model continues to output the final token even after sufficient meaningful responses have already been generated during execution. These issues go beyond mere cost concerns and directly result in reduced service responsiveness and wasted system resources.

[0004] Currently, static blocking methods at the pre-input stage are primarily used to address these issues. Examples include rate limits, forbidden word filters, basic PII filters, and prompt length limits.

[0005] However, this method fails to simultaneously quantify the utility, cost, and risk of a request, and is inadequate for controlling interruptions during execution or re-identification risks at the output stage. As a result, simple requests and complex analysis requests are processed under the same cost structure, and requests containing sensitive information are not blocked during execution, increasing the likelihood of security incidents. Prior art literature

[0006] Republic of Korea Registered Patent No. 10-2655929 'System and Method for Generating Conversational Text Data Using Generative AI Model' (Publication Date: April 11, 2024) The problem to be solved

[0007] The objective of the present invention, devised to solve the aforementioned problems, is to provide an execution path control device and method based on the Net Utility Score of an AI request that automatically controls the execution path of an AI request by quantifying and evaluating the utility, cost, and risk of an input request and calculating a Net Utility Score (NUS) based on the correlation thereof.

[0008] In addition, the objective of the present invention is to provide an execution path control device and method based on the net utility score of AI requests, which processes complex and high-value requests via a high-performance path, processes simple or repetitive requests via a low-cost path, and processes requests containing policy violations or sensitive information via a blocking or human review path.

[0009] In addition, the objective of the present invention is to provide an execution path control device and method based on the net utility score of an AI request that can suppress unnecessary model calls and simultaneously improve LLM processing efficiency and security by automatically branching the processing path of a response request.

[0010] The problems to be solved by the present invention are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0011] The execution path control device based on net utility score of an AI request according to the present invention, for achieving the above-mentioned problem to be solved, comprises: an intent analysis module that, upon receiving LLM question data, preprocesses it using natural language processing, decomposes a sentence into phrase units to generate a dependency phrase tree, extracts a first intent candidate based on the phrase tree derived from the dependency phrase tree, vectorizes the LLM question data into an embedding model, performs a proximity vector search on an intent embedding database to extract a second intent candidate, and combines the first intent candidate and the second intent candidate into one to generate an intent candidate set; a signature generation module that, after receiving the intent candidate set from the intent analysis module, calculates inclusion relationship scores between each intent candidate, assigns weights according to the calculated inclusion relationship scores, determines the intent candidate whose sum of these scores is maximized as the core intent of the LLM question data, and determines at least one of a target word, a constraint phrase, and a domain within the sentence of the LLM question data based on the determined core intent to generate a first signature; and the signature generated in the signature generation module A duplicate request blocking module that calculates similarity between vectors by comparing the embedding vector of the first signature with the embedding vectors of existing signatures generated prior to the first signature, and reuses an existing response corresponding to an existing signature if the calculated similarity is greater than or equal to a preset reuse threshold; an intent evaluation module that, after receiving the first signature generated by the signature generation module, calculates the utility score of the LLM question data based on a preset utility score calculation standard, calculates the cost score based on a preset cost score calculation standard, or calculates the risk score based on a preset risk score calculation standard, if the similarity calculated by the duplicate request blocking module is less than the preset reuse threshold;

[0012] It includes a net utility calculation module that receives one of a utility score, a cost score, or a risk score from an intention evaluation module, inputs it into a preset net utility score formula, assigns weights to the cost score or the risk score, and calculates a net utility score, and a path determination module that selects a first model path if the net utility score calculated by the net utility calculation module is greater than or equal to a preset utility threshold, and selects a second model path if the net utility score is less than the utility threshold.

[0013] The intent analysis module can preprocess LLM question data using natural language processing, perform syntactic analysis to generate a syntactic tree, identify the sentence structure of the LLM question data, generate a syntactic tree-based first intent candidate based on the dependency syntactic tree, calculate the semantic embedding vector of the question data, extract a second intent candidate, integrate the first intent candidate and the second intent candidate to generate an integrated intent candidate, calculate the semantic embedding-based similarity between the question data and the integrated intent candidate, and generate an intent candidate set of integrated intent candidates whose calculated similarity is greater than or equal to a preset intent group classification threshold.

[0014] The signature generation module can generate an intent signature by normalizing the selected core intent, target word, constraint phrase, and domain among the intent candidates using pre-configured rules, serializing them into a single string with a fixed item order, and then applying a hash function.

[0015] The duplicate request blocking module calculates a semantic embedding vector of LLM question data corresponding to a newly generated signature and calculates semantic embedding vectors of existing signatures stored in the signature storage unit, calculates cosine similarity for each of the calculated semantic embedding vectors, and if the maximum similarity value among the calculated similarities is greater than or equal to a preset reuse threshold, the existing signature is determined to be the same request and the previously stored response data is reused, and if the maximum similarity value is less than the reuse threshold, a new response can be generated in the LLM for the signature.

[0016] The intention evaluation module independently calculates the utility score, cost score, and risk score of the LLM question data by normalizing them to a range of 0 to 1, and the net utility calculation module can standardize the evaluation values ​​so that the net utility score increases as the utility score increases and as the cost score and risk score decrease.

[0017] The net utility calculation module calculates based on one or more indicators among the intent relevance, task difficulty, information recency, and substitutability of the LLM question data, and can produce a higher net utility score as the degree of alignment between intent and purpose increases.

[0018] The path determination module has a high-cost artificial intelligence model connected to the first model path and a low-cost artificial intelligence model connected to the second model path, and can branch into a blocking path or a human review path depending on preset conditions.

[0019] When the second model path is selected because the net utility score is evaluated as being below the utility threshold, the path determination module inputs the LLM question data input into the artificial intelligence model connected to the second model path into the shadow model, generates a reference response in the shadow model, and can calculate a quality comparison value using at least one of the semantic similarity, format matching rate, and key information inclusion rate between the first type response generated by the low-cost artificial intelligence model and the reference response generated by the shadow model.

[0020] If the quality comparison value is calculated to be less than a preset reference comparison threshold, the path determination module can activate the high-cost artificial intelligence model connected to the first model path to output a correction signal to regenerate a third-type response for the same LLM question data, and control the regeneration to be repeated until the quality comparison value of the third-type response is greater than or equal to the reference comparison threshold.

[0021] Additionally, the execution path control device based on net utility score of an AI request may further include a context management module that collects multiple context candidates semantically associated with the requested LLM question data among external search results, search augmentation generated data, conversation history, output results of a tool, or policy metadata, calculates semantic embedding-based similarity between each context candidate and the LLM question data, determines candidates whose similarity is greater than or equal to a reference context threshold as associated contexts, calculates the utility contribution per token of each associated context candidate, attaches context candidates whose calculated utility contribution is greater than or equal to a preset utility threshold to a prompt, removes context candidates whose utility contribution is less than the preset utility threshold, and adjusts the length of the prompt for the LLM question data.

[0022] In addition, it may further include an output risk control module that evaluates the risk of re-identification of pseudonym tokens or sensitive information within the response during the output stage, and if the risk level exceeds a set threshold, converts the output into a partially restored or alternative representation and outputs it.

[0023] The output risk control module performs a re-identification risk assessment based on context similarity, token patterns, and the frequency of co-occurrence of named entities, and may allow restoration within the policy scope only when the risk level is below the threshold.

[0024] It may further include a monitoring module that records the utility score, cost score, risk score, net utility score, selected path, response information, and judgment basis for each LLM question data in an immutable form and utilizes them for threshold adjustment, policy verification, and performance analysis.

[0025] In addition, the execution path control device based on net utility score for AI requests of the present invention may further include a monitoring module that records the utility score, cost score, risk score, net utility score, selected path, response information, and judgment basis for each LLM question data in an immutable form, while ensuring log integrity through hash verification or electronic signature methods, and utilizing it for threshold adjustment, policy verification, and performance analysis.

[0026] Another method for controlling execution paths based on net utility scores of AI requests to achieve the aforementioned problem comprises: (a) when an intent analysis module receives LLM question data, preprocesses it using natural language processing, decomposes the sentence into phrase units to generate a dependency phrase tree, extracts a first intent candidate based on the phrase tree derived from the dependency phrase tree, vectorizes the LLM question data into an embedding model, performs a proximity vector search on an intent embedding database to extract a second intent candidate, and combines the first intent candidate and the second intent candidate into one to generate a set of intent candidates; (b) after a signature generation module receives the set of intent candidates from the intent analysis module, calculates inclusion relationship scores between each intent candidate, assigns weights according to the calculated inclusion relationship scores, determines the intent candidate whose sum of these scores is maximized as the core intent of the LLM question data, and determines at least one of the target word, constraint phrase, and domain within the sentence of the LLM question data based on the determined core intent to generate a first signature. (c) A step in which the duplicate request blocking module compares the embedding vector of the first signature generated by the signature generation module with the embedding vectors of existing signatures generated prior to the first signature to calculate the similarity between the vectors, and if the calculated similarity is greater than or equal to a preset reuse threshold, reuses the existing response corresponding to the existing signature; (d) A step in which, after the intent evaluation module receives the first signature generated by the signature generation module, if the similarity calculated by the duplicate request blocking module is less than the preset reuse threshold, calculates the utility score of the LLM question data based on a preset utility score calculation standard, calculates the cost score based on a preset cost score calculation standard, or calculates the risk score based on a preset risk score calculation standard; (e) A step in which the net utility calculation module receives one of the utility score, cost score, or risk score from the intent evaluation module, inputs it into a preset net utility score formula, andThe method includes the step of calculating a net utility score by assigning weights to cost scores or risk scores, and (f) the step of the path determination module selecting a first model path if the net utility score calculated by the net utility calculation module is greater than or equal to a preset utility threshold, and selecting a second model path if the net utility score is less than the utility threshold.

[0027] And (g) a step in which a context management module collects multiple context candidates semantically associated with the requested LLM question data among external search results, search augmentation generated data, conversation history, tool output results, or policy metadata, and calculates semantic embedding-based similarity between each context candidate and the LLM question data, determines candidates whose similarity is greater than or equal to a reference context threshold as associated contexts, calculates the utility contribution per token of each associated context candidate, attaches context candidates whose calculated utility contribution is greater than or equal to a preset utility threshold to the prompt, removes context candidates whose utility contribution is less than the preset utility threshold, and adjusts the length of the prompt for the LLM question data; and (h) a step in which an output risk control module performs a re-identification risk assessment based on context similarity, token patterns, and named entity co-occurrence frequency, and allows restoration within the policy scope only when the risk is below a threshold; and (i) a step in which a monitoring module records the utility score, cost score, risk score, net utility score, selected path, response information, and judgment basis for each LLM question data in an immutable form, and the threshold It may include steps utilized for coordination, policy verification, and performance analysis. Effects of the invention

[0028] The present invention provides a system and method for controlling an execution path based on a net utility score of an AI request, which quantifies and evaluates the utility, cost, and risk of an input request and calculates a net utility score based on the correlation between them to automatically control the execution path of an AI request.

[0029] In addition, the present invention provides an execution path control system and method based on net utility scores of AI requests, which processes complex and high-value requests via a high-performance path, processes simple or repetitive requests via a low-cost path, and processes requests containing policy violations or sensitive information via a blocking or human review path.

[0030] In addition, the present invention suppresses unnecessary model calls by automatically branching the processing path of response requests, and can simultaneously improve LLM processing efficiency and security. Brief explanation of the drawing

[0031] FIG. 1 is a block diagram of an execution path control device based on the net utility score of an AI request according to one embodiment of the present invention. FIG. 2 is a flowchart of an execution path control method based on net utility score of an AI request according to an embodiment of the present invention. Specific details for implementing the invention

[0032] The advantages and features of the present invention and the apparatus for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments described below but can be implemented in various different forms. The embodiments described below are provided merely to convey the description of the present invention and to fully inform those skilled in the art of the scope of the invention. The scope of the claims of the present invention may be defined solely by the claims.

[0033] First, prior to the specific description of the present invention, the terms used in this specification will be briefly explained, and the described embodiments will be described in detail.

[0034] The terms used in this specification are generally accepted terms, taking into account their functions in this description; however, they may vary depending on the intent of those skilled in the relevant field, case law, the emergence of new technologies, etc. Additionally, for terms arbitrarily selected by the applicant, their meanings are described in detail in the relevant description of the invention.

[0035] Therefore, the terms used in this description are defined not merely by their names, but based on their meanings and the overall content of this description.

[0036] Additionally, singular expressions in this specification include plural expressions unless the context clearly indicates that they are singular. Furthermore, when a part of the specification is described as including a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Moreover, the term "module" as used in this specification refers to a software or hardware component, and the "module" performs the roles defined for each. Such a "module" may be configured to reside in an addressable storage medium or configured to run one or more processors. As an example, a "module" may include components such as software components, object-oriented software components, class components, and task components, and at least one of processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables.

[0037] The functions provided within the components and 'modules' may be combined into a smaller number of components and 'modules' or further separated into additional components and 'modules'. Furthermore, the Large Language Models (LLMs) described herein are defined as language models composed of artificial neural networks having numerous parameters. Such large language models enable AI technology that analyzes user queries and provides appropriate responses.

[0038] FIG. 1 is a block diagram of an execution path control device based on the net utility score of an AI request according to one embodiment of the present invention.

[0039] The execution path control device (100) based on the net utility score of the AI ​​request of the present invention quantifies and evaluates the utility, cost, and risk of input LLM data and calculates a net utility score (NUS) based on the correlation thereof to control the execution path of the AI ​​request. At the same time, complex and high-value requests are processed by a high-performance model, while simple or repetitive requests are processed by a low-cost model. Additionally, requests containing policy violations or sensitive information are processed by blocking or human review.

[0040] The execution path control device (100) based on the net utility score of such AI requests is composed of an intent analysis module (110), a signature generation module (120), a duplicate request blocking module (130), an intent evaluation module (140), a net utility calculation module (150), and a path determination module (160). Additionally, if necessary, it may include a context management module (170), an output risk control module (180), and a monitoring module (190), and by automatically branching the processing path of response requests, it suppresses unnecessary model calls and can maximize LLM processing efficiency, security, transparency, reliability, and resource efficiency.

[0041] Hereinafter, the intention analysis module (110), signature generation module (120), duplicate request blocking module (130), intention evaluation module (140), net utility calculation module (150), and path determination module (160) constituting the present invention will be described in more detail.

[0042] The intent analysis module (110) preprocesses the received LLM question data through natural language processing, performs vocabulary refinement, tokenization, and stop word removal, and performs morphological analysis and syntactic analysis through word unit splitting. At the same time, it identifies the sentence structure of the LLM question data by generating a dependency syntax tree and extracts key named entities and attributes within the LLM question data using Named Entity Recognition (NER). Based on the dependency syntax tree and the results of morphological analysis, the intent analysis module (110) identifies words in subject relationships as agents and words in object relationships as attributes by referencing dependency labels and morphological part-of-speech tags between each word in the sentence through a pre-established dependency relationship model. Then, it extracts first intent candidates based on pre-established rules regarding the identified agent, object, and attribute relationships and converts the LLM question data into an embedding vector form using a pre-trained embedding model. Subsequently, second intent candidates are extracted by searching for proximity vectors in the intent embedding database using the vector as a query (KNN).

[0043] The intention analysis module (110) integrates the first intention candidate and the second intention candidate to cluster vectors whose similarity is greater than or equal to a preset intention group classification threshold into the same group, thereby forming an integrated intention candidate set. For example, when the intention analysis module (110) receives LLM question data such as “Will it rain in Seoul tomorrow?”, it preprocesses the data to tokenize and remove stop words into ‘Seoul’, ‘tomorrow’, ‘rain’, and ‘will it come’, classifies ‘Seoul’, ‘tomorrow’, and ‘rain’ as ‘nouns’, classifies ‘oh’ as a verb stem, classifies ‘lkka’ as a terminal ending, and classifies ‘?’ as a punctuation mark, and performs morphological analysis of the LLM question data.

[0044] In addition, based on the results of morphological analysis and syntactic analysis, 'Seoul' can be identified as a noun or adverbial particle combination representing location information, 'tomorrow' as a noun or adverbial functional element representing time information, 'come' as a verb, and 'rain' as a precipitation-related noun. Furthermore, the intent analysis module (110) extracts 'Seoul' as a city entity name and 'tomorrow' as a time information entity name through Named Entity Recognition (NER).

[0045] Additionally, the intent analysis module (110) generates a candidate entity named 'weather_query' (intent_type=weather_query), i.e., a first intent candidate (C1), based on the results of the dependency syntax tree and morphological analysis, vectorizes the structure and meaning of the LLM question data using a pre-trained embedding model, and extracts candidate entities named 'precipitation probability inquiry' and 'weather information query', i.e., a second intent candidate (C2), by performing vector proximity search (KNN). Subsequently, the first intent candidate (C1) and the second intent candidate (C2) can be integrated to form a set of integrated intent candidates such as {'weather_query', 'precipitation probability inquiry', 'weather information query'}.

[0046] The intent analysis module (110) generates a set of intent candidates by combining the first intent candidate and the second intent candidate into one. It calculates the semantic embedding-based cosine similarity between the LLM question data and each candidate, and if the cosine similarity of 'weather inquiry' is greater than or equal to a threshold, it confirms this as the final set of intent candidates. That is, when the intent analysis module (110) receives LLM question data, it preprocesses it using natural language processing, breaks down the sentence into phrase units, and extracts named entities to identify the structure and meaning of the LLM question data. It also extracts the action intent (intent_type), target words (slots), constraint phrases, and domain implied by the question, and minimizes misjudgment of the question intent by comprehensively interpreting the structure, meaning, and named entities within the sentence. Furthermore, various expressions with the same meaning, such as “Is it raining?” and “Is there precipitation?” It recognizes the back and enables subsequent signature generation modules and duplicate request blocking modules to receive a consistent input structure, thereby reducing the accumulation of errors between modules.

[0047] The signature generation module (120) calculates relationship scores between each candidate using a pre-configured inclusion relationship score calculation model for multiple intent candidates included in the set of intent candidates output from the LLM question data and the intent analysis module (110). Then, by applying weights according to the calculated inclusion relationship scores, the candidate entity name that maximizes the weighted sum is determined as the core intent of the LLM question data. Based on the determined core intent, the signature generation module (120) identifies target words (slots), constraint phrases (constraints), and domains within the sentences in the LLM question data, and normalizes the identified elements by unifying them according to rules, such as synonyms, formats, case sensitivity, and date units. Then, after fixing the order of the items and serializing them into a single string, a hash function is applied to generate an intent signature (sig), i.e., H(intent_type, slots, constraints, domain)). For example, when LLM question data such as “Please tell me the temperature in Seoul tomorrow in a simple table” is input, the sentence is simplified into components such as time=tomorrow, region=Seoul, item=temperature, expression constraint=table, etc., and when a set of intent candidates is generated such as {C1: weather inquiry, C2: precipitation probability inquiry, C3: temperature statistics summary}, the signature generation module (120) calculates vector similarity, keyword inclusion rate, regular expression matching score (isIn(q, c)) based on text inclusion, NER matching score based on dependency syntax (depNER(q, c)), and domain correction value based on policy (policy(q, c)), and calculates the final inclusion relationship score using the formula W1*isIn + W2*depNER + W3*policy. At this time, W1+W2+W3=1, and W1=0.5, W2=0.4, and W3=0.1. For example, the signature generation module (120) has a regular expression matching score of 0.9, a NER matching score according to dependency syntax of 0.88, and a domain correction value according to policy of 0.It is calculated as 0.5 and the final inclusion relationship score for “weather inquiry” is calculated as 0.807. Then, the signature generation module (120) calculates the regular expression matching score for the “precipitation probability inquiry” candidate as 0.4, the NER matching score according to the dependency syntax as 0.55, and the domain correction value according to the policy as 0.05, and the final inclusion relationship score for “precipitation probability inquiry” is calculated as 0.425.

[0048] And the signature generation module (120) calculates a regular expression matching score of 0.65, a NER matching score according to dependency syntax of 0.7, and a domain correction value according to policy of 0.05 for “temperature statistics summary”, and calculates a final inclusion relationship score of 0.61 for “temperature statistics summary”.

[0049] The signature generation module (120) determines the candidate “weather inquiry”, which has the maximum final inclusion relationship score, as the core intent of the LLM question data. Then, the signature generation module (120) identifies the target word, constraint phrase, and domain within the LLM question data. At this time, the signature generation module (120) identifies the target word by setting the city to “Seoul,” the date to “tomorrow,” and the item to “temperature.” Then, it identifies the constraint phrase by setting the output format to “table” and the response tone / length to “briefly.” Then, it identifies the domain as “weather.” Afterward, it normalizes and serializes the identified content. For example, it can be normalized as intent_type = weather lookup, slots = {city=Seoul, date=2025-11-04, item=temperature} (applying the rule for alphabetical sorting of key names), constraints = {output format=table, summary map=simple}, and domain = weather. This is serialized. Afterwards, a one-way hash function is applied to generate the final signature, i.e., the first signature, in the form of H(intent_type, slots, constraints, domain), to create “91f3b9a2...ed2a”.

[0050] The signature generation module (120) can achieve an accuracy of over 99% in identifying duplicate requests by always generating the same signature for various question expressions that have the same meaning. In addition, it can improve security by preventing personal information from being exposed in plaintext through hash-based irreversible expressions, and can reduce the search speed of the signature storage database by about 40% by improving storage and comparison efficiency with fixed-length keys. The first signature generated by such a signature generation module (120) can be used for reuse determination in the duplicate request blocking module (130) described later and as an identification key in the monitoring module (190).

[0051] The duplicate request blocking module (130) calculates a similarity by comparing the embedding vector of the first signature generated by the signature generation module (120) with the embedding vectors of existing signatures stored in the signature storage unit, and if the calculated similarity is greater than a preset reuse threshold value, e.g. 0.85, the existing signature is reused. On the other hand, if the maximum similarity is less than the reuse threshold value, the request is determined to be a new request and a new response is generated through LLM.

[0052] For example, the duplicate request blocking module (130) can calculate the cosine similarity between the embedding vector of the current request signature “91f3b9a2...ed2a” and the embedding vector of the existing signature list, i.e., {“91f3b9a2...ed2a”, “b8f1d0...”, “e923ff... ”}, and calculate sim(91f3b9a2, b8f1d0) = 0.72, sim(91f3b9a2, e923ff) = 0.64 and sim(91f3b9a2, 91f3b9a2) = 1.00.

[0053] Among these, if the maximum similarity is greater than or equal to the reuse threshold of 0.85, the duplicate request blocking module (130) determines that the existing signature is the same request and immediately returns the response stored corresponding to the signature.

[0054] On the other hand, if the maximum similarity is less than the reuse threshold, it is determined to be a new request, and a new response is generated in the LLM. Such a duplicate request blocking module (130) does not reprocess the same request, thereby reducing the model call cost to 1 / N, maintaining response consistency, and alleviating server load. In addition, the average response time per request can be reduced from 3.2 seconds to 1.1 seconds.

[0055] The intent evaluation module (140) operates only when the similarity calculated by the duplicate request blocking module (130) is less than a preset reuse threshold after receiving the signature generated by the signature generation module (120). The intent evaluation module (140) performs independent evaluations on LLM question data from three perspectives: utility, cost, and risk. At this time, the intent evaluation module (140) calculates each independently by normalizing them to a range of 0 or more and 1 or less.

[0056] More specifically, the intent evaluation module (140) calculates a utility score (U) of the LLM question data based on at least one of intent accuracy, task difficulty, information recency, or substitutability, calculates a cost score (C) based on model unit cost, estimated number of tokens, processing delay time, or load rate, or calculates a risk score (R) based on whether sensitive information is included, whether there is a policy violation, or the risk level by domain. Here, the utility score (U) is calculated using indicators such as intent suitability, task difficulty, information recency, and substitutability. At this time, the score is calculated to have a higher value as the degree of alignment with the intent and purpose of the question increases, and as it is more difficult to obtain the relevant information from external low-cost channels (such as search) (as it is non-substitutable). The cost score (C) is calculated based on indicators such as the estimated number of input and output tokens, model unit cost, processing delay time, and system load rate, and is calculated to have a higher value as resource consumption increases. The risk score (R) is calculated by considering whether sensitive information is included in the input statement, the probability of a policy violation, and the regulatory risk level by domain, and is calculated to have a higher value as there are more risk factors. At this time, the intention evaluation module (140) calculates the utility score (U), cost score (C), and risk score (R) by normalizing them to a range of 0 or greater and 1 or less, respectively, in order to exclude the influence of different scales. For example, the intention evaluation module (140) can evaluate the intent suitability, information recency, and non-replaceability of a specific LLM question data as 0.9, 0.8, and 0.7, and calculate the utility score (U) as 0.8 by applying the average value thereof. In addition, for the same request, the estimated number of tokens is evaluated as 0.6, the model unit cost as 0.3, and the processing latency as 0.4, and the overall cost score (C) is calculated as 0.45 by applying pre-set weights to each item.

[0057] The intention evaluation module (140) performs the role of quantitatively evaluating the economic feasibility of each request by independently calculating the utility (U), cost (C), and risk (R) in this manner.

[0058] The net utility calculation module (150) receives at least one score among the utility score (U), cost score (C), and risk score (R) from the intention evaluation module (140) and substitutes it into a preset net utility score formula. At this time, the net utility score is calculated by assigning weights to the cost score and the risk score, but assigning different weights to the cost score and the risk score. Here, the preset net utility score formula is defined as a formula formed by the utility score (U) - (first weight (W1) * cost score (C)) - (second weight (W2) * risk score (R)).

[0059] Here, the utility score is a utility score based on the importance and necessity of the request and is a real number between 0 and 1; the cost score is a cost score normalized by the expected token consumption, processing time, model unit cost, etc., to process the request; the risk score is a risk score reflecting whether personal information (PII) is included, the probability of policy violation, domain risk, etc.; and the first weight (W1) and the second weight (W2) are weights reflecting the relative importance of utility, cost, and risk. For example, when the net utility calculation module (150) is set to a risk indicator value of 0 because personal information (PII) is not included in the request and the risk level is evaluated as 0.2 due to domain characteristics and the total risk score (R) is standardized to 0.1, the net utility calculation module (150) can calculate a net utility score (NUS) of 0.545 based on a formula such as 0.8 (0.5×0.45) (0.3×0.1) = 0.8 0.225 0.03, when the utility score (U) is set to 0.8, the cost score (C) to 0.45, the first weight to 0.5, and the second weight to 0.3.

[0060] According to this configuration, the net utility calculation module (150) assigns a high score to requests that have high utility and low cost and risk. Subsequently, the path determination module (160) controls the request to be delivered to a high-cost LLM path only when the calculated net utility score is greater than or equal to a preset utility threshold.

[0061] The net utility calculation module (150) is configured to adjust the weights of indicators according to domain-specific policies. More specifically, the net utility calculation module (150) calculates a net utility score using indicators such as the intent suitability of LLM question data, the recency of information, and the possibility of substitution through external low-cost channels. At this time, the higher the degree of alignment between intent and purpose, and the more difficult it is to obtain the same information through external low-cost channels, the higher the net utility score is calculated. On the other hand, cost increase factors due to increased work difficulty are reflected in the cost item (C) when calculating the net utility score and are applied in a direction that reduces net utility.

[0062] The path determination module (160) receives the net utility score (NUS) calculated by the net utility calculation module (150) and, if the NUS is greater than or equal to a preset utility threshold, selects a first model path connected to a high-cost artificial intelligence model, and if the net utility score is less than the utility threshold, selects a second model path connected to a low-cost artificial intelligence model. At this time, the first model path is connected to a high-cost artificial intelligence model such as GPT-4 to enable high-precision processing, and the second model path is connected to a low-cost artificial intelligence model such as DistilGPT or Rule-engine to enable efficient processing. Additionally, if a preset condition, such as a policy violation, inclusion of sensitive information, or a condition of judgment uncertainty, is detected, the third model path or the fourth model path is configured to be selected, respectively. Here, the third model path is a path that immediately blocks the request to prevent processing of policy violations or risky inputs, and the fourth model path is a path that allows a reviewer to determine approval or rejection.

[0063] When the path determination module (160) selects a first model path connected to a high-cost artificial intelligence model (such as GPT-4) because the net utility score calculated by the net utility calculation module (150) is evaluated to be above the utility threshold, it inputs the same LLM question data that is input to the first model path into the shadow model connected to the first model path. At this time, the shadow model is a high-performance or policy-enhanced artificial intelligence model that serves as a baseline to verify the response quality of the lightweight model, and it receives the same LLM question data to generate a reference response (second type response). Then, it can calculate a comparison value among semantic similarity, format matching rate, and key information inclusion rate between the first type response and the reference response. For example, the path determination module (160) might say, “The summary of Samsung Electronics’ financial statements for the last five years is sales of 320 trillion, operating profit of 50 trillion, and net profit of 43 trillion.” For the first type response, “Samsung Electronics’ average revenue over the last 5 years is approximately 320 trillion won, and operating profit is 50 trillion won,” the second type response, “Samsung Electronics’ average revenue over the last 5 years is approximately 320 trillion won, and operating profit is 50 trillion won,” a semantic similarity of 0.87, a formal match rate of 0.92, and a key information inclusion rate of 0.95 can be calculated, and a weighted average comparison value for the three indicators can be calculated as 0.91.

[0064] The path determination module (160) determines that if the weighted average comparison value is greater than or equal to a preset reference comparison threshold, for example, 0.8 to 0.9, the two responses are sufficiently matched and approves the verification result of the shadow model so that the response of the high-cost artificial intelligence model is adopted as is.

[0065] On the other hand, if the weighted average comparison value is less than a preset reference comparison threshold, the path determination module (160) generates a correction signal to activate the first model path connected to the high-cost artificial intelligence model, thereby causing the high-cost artificial intelligence model to regenerate a third-type response for the same LLM question data. At this time, the correction signal may include causes of inconsistency such as missing key information, distorted representation, and contextual inconsistency found in the response of the lightweight model, thereby inducing quality improvement during the regeneration process of the high-cost model.

[0066] The path determination module (160) calculates a quality comparison value by comparing a first-type response generated through a selected model path, for example, a low-cost model path, with a reference response of the Shadow Model. It then controls the response regeneration process to be performed only when the calculated comparison value is less than a preset reference comparison threshold. When the comparison value reaches the threshold or higher, the regeneration procedure is terminated to ensure consistency and reliability of the response quality. Additionally, the path determination module (160) causes the Shadow Model to monitor the streaming output of the low-cost model in real time or to verify the generated first-type response, and triggers correction or regeneration using the high-cost model only when the quality is judged to be below the threshold as a result of the comparison. Accordingly, since computational resources are selectively consumed only when quality correction is required, rather than processing all requests with the high-cost model, the efficiency of LLM operations can be maximized and the overall response generation cost can be reduced.

[0067] As described above, the present invention suppresses unnecessary high-cost LLM execution, optimizes the distribution of model resources, and achieves the effects of improved response speed and reduced operating costs through the organic combination of an intent analysis module (110), a signature generation module (120), a duplicate request blocking module (130), an intent evaluation module (140), a net utility calculation module (150), and a path determination module (160).

[0068] Furthermore, the present invention may further include a context management module (170) in addition to the basic structure above. The context management module (170) collects a plurality of context candidates that can be semantically associated with LLM question data from external search results, search augmentation generation (RAG) data, conversation history, tool output results, or policy meta-information. Subsequently, it calculates semantic embedding-based similarity between each context candidate and LLM question data to determine the primarily associated context.

[0069] Subsequently, the context management module (170) calculates a Marginal Utility per Token (MU) for each associated candidate, taking into account the token length relative to the amount of information. Then, it selects only candidates whose utility per token is greater than or equal to a preset utility threshold and attaches them to the prompt, and excludes candidates whose utility per token is less than the threshold, thereby optimizing the prompt size and preventing unnecessary token charges. For example, when question data such as “Tell me the sales trend of Samsung Electronics over the last 5 years” is entered, the context management module (170) collects candidates such as “2020~2024 sales table” (A), “operating profit trend” (B), and “related news article” (C), and calculates embedding-based similarity values ​​for these as 0.95 (A), 0.82 (B), and 0.55 (C), respectively. When the context threshold is 0.7, candidate C, which has low similarity and utility contribution per token, is removed, and only candidates A and B, which are above the threshold, are included in the prompt, thereby configuring the prompt with only key information that is directly related to the question, thereby improving answer accuracy and token efficiency.

[0070] The context management module (170) can significantly reduce the number of tokens in an input prompt by removing unnecessary information within the prompt and selecting and attaching only semantically essential context. For example, according to the embodiment, a prompt length reduction effect of about 40% or more can occur, and accordingly, the accuracy and consistency of response quality can be improved along with a reduction in token costs.

[0071] By adding such a context management module (170) to the basic structure, the present invention can maintain or improve response quality while minimizing the number of tokens in the input prompt, and reduces unnecessary computations of high-cost models, thereby realizing processing efficiency and reducing operating costs of the entire system. Additionally, the present invention may further include an output risk control module (180) together with the context management module (170) as needed to control the risk of exposure of sensitive information at the output stage, thereby strengthening the safety and reliability of the entire response pipeline.

[0072] The output risk control module (180) detects pseudonym tokens or expressions of sensitive information included in the response and evaluates the risk of re-identification therefor before the response generated through a high-cost artificial intelligence model or a low-cost artificial intelligence model is actually output. The output risk control module (180) analyzes the contextual meaning and structure of the output sentence and calculates the risk of re-identification based on one or more of contextual similarity, token patterns, and named entity co-occurrence frequency.

[0073] For example, if the LLM question data is “Tell me the sales trend of Samsung Electronics over the last 5 years,” and among the responses to the LLM question data is “Samsung Electronics’ sales between 2020 and 2024 are at an annual average of 320 trillion won and operating profit is at 50 trillion won, based on a summary report written by finance manager Hong Gil-dong,” the output risk control module (180) can detect pseudonym tokens and sensitive expressions and calculate context similarity that identifies the association between a person's name and financial information in the context of the report, token patterns that identify whether names, job titles, etc. are exposed, and the frequency of co-occurrence of entities that reflect whether expressions representing specific people or organizations such as “Samsung Electronics,” “Finance,” and “Hong Gil-dong” appear simultaneously.

[0074] For example, when the output risk control module (180) calculates a context similarity of 0.8, a token pattern-based risk of 0.9, and a named entity co-occurrence frequency of 0.6, the re-identification risk can be calculated as 0.81 through the weighted sum of these values. When the set risk threshold value is 0.7, the output risk control module (180) does not output the response as is, but converts it into an alternative expression that cannot directly identify an individual or a specific organization, such as “summary information based on internal reports of the relevant department,” and outputs it.

[0075] On the other hand, even if the LLM question data is identical, if the generated response does not include sensitive information or personal identification elements, such as “Samsung Electronics’ average sales over the last 5 years are at the level of 320 trillion won,” the output risk control module (180) evaluates the re-identification risk level as, for example, 0.32 and controls the response to be output as is.

[0076] In this way, the output risk control module (180) of the present invention quantitatively evaluates the risk of re-identification of sensitive information and pseudonym tokens during the output stage, and if the risk level is above a standard risk threshold, it automatically applies partial restoration, alternative representation, or information deletion, thereby preventing leakage of sensitive information and ensuring safe output that complies with the policy.

[0077] The present invention may further include a context management module (170), an output risk control module (180), and a monitoring module (190) in the basic structure.

[0078] The monitoring module (190) is configured to ensure data integrity and auditability by recording data generated throughout the processing process in an immutable form. For example, when a request for “Samsung Electronics’ financial statements for the last 5 years” is processed, the monitoring module (190) stores metadata such as the calculated utility score (U), cost score (C), risk score (R), net utility score (NUS), selected model path, output risk assessment result, applied restoration policy, and reason codes in the form of an immutable log. The log is recorded in a hash chain-based integrity verification structure, a blockchain, or WORM (Write Once Read Many) storage so that external intrusion or arbitrary modification and deletion by internal administrators is virtually impossible.

[0079] Additionally, the monitoring module (190) monitors in real time whether the log has been tampered with or forged through an electronic signature or hash value verification method, and forms a feedback loop for threshold adjustment, security policy validation, performance analysis, and response quality improvement based on the accumulated log.

[0080] Therefore, the present invention can significantly improve the security, transparency, and reliability of an AI execution engine by evaluating the risk of re-identification in real time during the output stage to automatically control the restoration range, as well as by recording the basis for all decisions and judgment results in an immutable form.

[0081] Through this, the present invention can be utilized for auditing and forensic analysis of each judgment result, and by dynamically tuning the optimal threshold value based on recorded data, it can maximize resource efficiency of the entire AI execution engine and minimize operating costs.

[0082] In conclusion, the present invention provides a comprehensive execution path control environment that enables the safe and efficient operation of high-cost AI models by integrally managing the entire lifecycle of AI request processing, ranging from intent analysis at the input stage to path optimization at the processing stage, risk control at the output stage, and post-audit and analysis.

[0083] Meanwhile, the technical configuration and operating principle of the execution path control device based on the net utility score of an AI request according to the present invention described above can be equally applied to the execution path control method based on the net utility score of an AI request according to the present invention.

[0084] All descriptions of the execution path control device based on the net utility score of an AI request according to the present invention can be applied as is to the execution path control method based on the net utility score of an AI request according to the present invention.

[0085] FIG. 2 is a flowchart of a method for an execution path control device based on the net utility score of an AI request according to an embodiment of the present invention.

[0086] The method for controlling execution paths based on net utility scores of AI requests proceeds through steps (a) (S110) to (f) (S160), which can suppress unnecessary LLM execution, optimize the distribution of model usage resources, improve response speed, and reduce operating costs.

[0087] Hereinafter, steps (a) (S110) to (f) (S160) of the execution path control method based on net utility score of an AI request will be described in detail.

[0088] The method for controlling the execution path based on the net utility score of an AI request begins with step (a) (S110), in which an intent analysis module (110) receives LLM question data, preprocesses it using natural language processing, decomposes the sentence into phrase units to generate a dependency phrase tree, and extracts entity names using named entity recognition. At the same time, it identifies the structure and meaning of the LLM question data and generates a set of intent candidates corresponding to the LLM question data based on the results. Subsequently, the signature generation module (120) calculates weights based on the inclusion relationship scores between the LLM question data and the set of intent candidates, selects the candidate whose sum of these scores is maximized, determines the entity name of the corresponding candidate as the core intent of the LLM question data, identifies the target word, constraint phrase, and domain within the sentence of the LLM question data based on the determined core intent, generates a first signature by combining these, and stores it in a database, i.e., a signature database. Subsequently, the duplicate request blocking module (130) calculates a similarity by comparing the embedding vector of the first signature generated by the signature generation module (120) with the embedding vectors of existing signatures generated before the first signature and stored in the signature storage unit, and proceeds to step (c) (S130) where the existing response corresponding to the existing signature is reused if the calculated similarity is greater than or equal to a preset reuse threshold.

[0089] Afterwards, the intention evaluation module (140) receives the signature generated by the signature generation module (120), and operates only when the similarity calculated by the duplicate request blocking module (130) is less than a preset reuse threshold, and proceeds to step (d) (S140), which calculates a utility score (U) of the LLM question data based on at least one of the intention accuracy, task difficulty, information recency, or substitutability, calculates a cost score (C) based on the model unit price, estimated number of tokens, processing delay time, or load rate, or calculates a risk score (R) based on whether sensitive information is included, whether there is a policy violation, or the risk level by domain.

[0090] Subsequently, the process proceeds to step (e) (S150), in which the net utility calculation module (150) receives the utility score (U), cost score (C), and risk score (R) from the intention evaluation module (140), inputs them into a preset net utility score formula, assigns different weights to the cost score and risk score, and calculates the net utility score. Then, the process proceeds to step (f) (S160), in which the path determination module (160) selects the first model path if the net utility score calculated by the net utility calculation module (150) is greater than or equal to a preset utility threshold, and selects the second model path if the net utility score is less than the utility threshold.

[0091] The execution path control method based on net utility score of an AI request according to the present invention proceeds through steps (a) (S110) to (f) (S160), suppressing unnecessary LLM execution and optimizing the distribution of model usage resources, thereby improving response speed and reducing LLM operating costs.

[0092] The method for controlling the execution path based on the net utility score of an AI request according to the present invention may proceed to step (g) (S170), step (h) (S180), and step (i) (S190) after steps (a) (S110) to (f) (S160).

[0093] Step (g) (S170) can be performed after Step (f) (S160).

[0094] Here, step (g) (S170) is a step in which the context management module (170) collects multiple context candidates that are semantically related to the requested LLM question data among external search results, Retrieval-Augmented Generation data, conversation history, output results of a tool, or policy metadata, calculates the semantic embedding-based similarity between each context candidate and the LLM question data, determines a candidate whose similarity is greater than or equal to a reference context threshold as an associated context, calculates the utility contribution per token of each associated context candidate, attaches the context candidate whose calculated utility contribution is greater than or equal to a preset utility threshold to the prompt, removes the context candidate whose utility contribution is less than the preset utility threshold, and adjusts the length of the prompt for the LLM question data.

[0095] Through these steps, the execution path control method based on net utility scores for AI requests according to the present invention can suppress unnecessary LLM execution, optimize the distribution of model usage resources, improve response speed, and reduce operating costs.

[0096] After this step (g) (S170), the output risk control module (180) performs a re-identification risk assessment based on context similarity, token patterns, and the frequency of co-occurrence of entity names, and proceeds to step (h) (S180), which allows restoration within the policy range only when the risk level is below a standard risk threshold of 0.6 to 0.8.

[0097] Through such step (h) (S180), the present invention can minimize the number of tokens in the input prompt, maintain accuracy and consistency regarding response quality, and reduce unnecessary computations of high-cost models, thereby doubling the processing efficiency and cost-saving effects of the entire system. Additionally, after this step (h) (S180), the monitoring module (190) proceeds to step (i) (S190), in which it records the utility score (U), cost score (C), risk score (R), net utility score (NUS), selected path, response information, and judgment basis for each LLM question data in an immutable form and utilizes them for threshold adjustment, policy verification, and performance analysis.

[0098] The method for controlling execution paths based on net utility scores of AI requests according to the present invention proceeds through steps (a) (S110) to (i) (S190), allowing each judgment result to be utilized for post-verification and analysis. At the same time, an administrator or device can dynamically adjust threshold values ​​to perform policy tuning and performance improvement, thereby maximizing the transparency, reliability, security, and resource efficiency of the entire AI execution engine.

[0099] Although embodiments of the present invention have been described above with reference to the attached drawings, those skilled in the art will understand that the present invention may be implemented in other specific forms without changing its technical concept or essential features. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. Explanation of the symbols

[0101] 100: Execution path controller based on net utility score of AI requests 110: Intent Analysis Module 120: Signature Generation Module 130: Duplicate Request Blocking Module 140: Intent Evaluation Module 150: Net Utility Calculation Module 160: Path Determination Module 170: Context Management Module 180: Output Risk Control Module 190: Monitoring Module

Claims

Claim 1 An intent analysis module (110) that, upon receiving LLM question data, preprocesses it using natural language processing, decomposes the sentence into phrase units to generate a dependency phrase tree, extracts a first intent candidate based on the phrase tree based on the dependency phrase tree, vectorizes the LLM question data into an embedding model, performs a proximity vector search on the intent embedding database to extract a second intent candidate, and combines the first intent candidate and the second intent candidate into one to generate a set of intent candidates; a signature generation module (120) that, after receiving the set of intent candidates from the intent analysis module (110), calculates inclusion relationship scores between each intent candidate, assigns weights according to the calculated inclusion relationship scores, determines the intent candidate whose sum of these scores is maximized as the core intent of the LLM question data, and determines at least one of the target word, constraint phrase, and domain within the sentence of the LLM question data based on the determined core intent to generate a first signature; and the signature generated in the signature generation module (120). A duplicate request blocking module (130) that calculates similarity between vectors by comparing the embedding vector of the first signature with the embedding vectors of existing signatures generated prior to the first signature, and reuses an existing response corresponding to an existing signature if the calculated similarity is greater than or equal to a preset reuse threshold; an intent evaluation module (140) that, after receiving the first signature generated by the signature generation module (120), calculates a utility score of the LLM question data based on a preset utility score calculation standard, calculates a cost score based on a preset cost score calculation standard, or calculates a risk score based on a preset risk score calculation standard, if the similarity calculated by the duplicate request blocking module (130) is less than a preset reuse threshold;An execution path control device based on the net utility score of an AI request, comprising: a net utility calculation module (150) that receives one of a utility score, a cost score, or a risk score from an intention evaluation module (140), inputs it into a preset net utility score formula, assigns weights to the cost score or the risk score, and calculates the net utility score; and a path determination module (160) that selects a first model path if the net utility score calculated by the net utility calculation module (150) is greater than or equal to a preset utility threshold, and selects a second model path if the net utility score is less than the utility threshold. Claim 2 In claim 1, the intent analysis module (110) preprocesses LLM question data using natural language processing, performs syntactic analysis to generate a syntactic tree, identifies the sentence structure of the LLM question data, generates a syntactic tree-based first intent candidate based on the dependency syntactic tree, calculates a semantic embedding vector of the question data, extracts the second intent candidate, integrates the first intent candidate and the second intent candidate to generate an integrated intent candidate, calculates a semantic embedding-based similarity between the question data and the integrated intent candidate, and generates an integrated intent candidate whose calculated similarity is greater than or equal to a preset intent group classification threshold as an intent candidate set group, thereby controlling the execution path based on the net utility score of an AI request. Claim 3 In claim 1, the signature generation module (120) normalizes selected core intent, target word, constraint phrase, and domain among intent candidates using preset rules, serializes them into a single string by fixing the order of items, and then generates an intent signature by applying a hash function, thereby forming an execution path control device based on the net utility score of an AI request. Claim 4 In claim 1, the duplicate request blocking module (130) calculates a semantic embedding vector of LLM question data corresponding to a newly generated signature and calculates semantic embedding vectors of existing signatures stored in a signature storage unit, calculates cosine similarity for each of the calculated semantic embedding vectors, and if the maximum similarity value among the calculated similarities is greater than or equal to a preset reuse threshold, the existing signature is determined to be the same request and the previously stored response data is reused, and if the maximum similarity value is less than the reuse threshold, the signature is used to generate a new response in the LLM, thereby controlling the execution path based on the net utility score of an AI request. Claim 5 In claim 1, the intention evaluation module (140) independently calculates the utility score, cost score, and risk score of the LLM question data by normalizing them to a range of 0 or more and 1 or less, respectively, and the net utility calculation module (150) standardizes the evaluation values ​​such that the net utility score increases as the utility score increases and as the cost score and risk score decrease, thereby controlling the execution path based on the net utility score of an AI request. Claim 6 delete Claim 7 In paragraph 5, the path determination module (160) is an execution path control device based on the net utility score of an AI request, wherein a high-cost artificial intelligence model is connected to the first model path and a low-cost artificial intelligence model is connected to the second model path, and branches into a blocking path or a human review path according to preset conditions. Claim 8 In claim 7, the path determination module (160) inputs LLM question data input into the AI ​​model connected to the second model path into the shadow model when the net utility score is evaluated as being less than the utility threshold value and the second model path is selected, and generates a reference response in the shadow model, and calculates a quality comparison value using at least one of the semantic similarity, format matching rate, and key information inclusion rate between the first type response generated by the low-cost AI model and the reference response generated by the shadow model. Claim 9 In claim 8, the path determination module (160) outputs a correction signal to activate a high-cost artificial intelligence model connected to the first model path to regenerate a third-type response to the same LLM question data when the quality comparison value is calculated to be less than a preset reference comparison threshold, and controls the regeneration to be repeated until the quality comparison value of the third-type response is greater than or equal to the reference comparison threshold, thereby controlling the execution path control device based on the net utility score of the AI ​​request. Claim 10 An execution path control device based on net utility score of an AI request, further comprising a context management module (170) in claim 1, which collects multiple context candidates semantically associated with the requested LLM question data among external search results, search augmentation generated data, conversation history, output results of a tool, or policy metadata, calculates semantic embedding-based similarity between each context candidate and the LLM question data, determines a candidate whose similarity is greater than or equal to a reference context threshold as an associated context, calculates the utility contribution per token of each associated context candidate, attaches context candidates whose calculated utility contribution is greater than or equal to a preset utility threshold to a prompt, removes context candidates whose utility contribution is less than the preset utility threshold, and adjusts the length of the prompt for the LLM question data. Claim 11 An execution path control device based on net utility score of an AI request, further comprising an output risk control module (180) that evaluates the risk of re-identification of pseudonym tokens or sensitive information within the response at the output stage, and if the risk level is greater than or equal to a set reference risk threshold, converts the output into a partial restoration or alternative representation and outputs it. Claim 12 In claim 11, the output risk control module (180) performs a re-identification risk assessment based on context similarity, token patterns, and the frequency of co-occurrence of named entities, and allows restoration within the policy range only when the risk level is below a reference risk threshold, a net utility score-based execution path control device for AI requests. Claim 13 An execution path control device based on net utility score of an AI request, further comprising a monitoring module (190) that records the utility score (U), cost score (C), risk score (R), net utility score (NUS), selected path, response information, and judgment basis for each LLM question data in an immutable form and utilizes them for threshold adjustment, policy verification, and performance analysis. Claim 14 (a) When the intent analysis module (110) receives LLM question data, it preprocesses it using natural language processing, then breaks down the sentence into phrase units to generate a dependency phrase tree, extracts a first intent candidate based on the phrase tree based on the dependency phrase tree, vectorizes the LLM question data into an embedding model, performs a proximity vector search on the intent embedding database to extract a second intent candidate, and combines the first intent candidate and the second intent candidate into one to generate an intent candidate set (S110); (b) When the signature generation module (120) receives the intent candidate set from the intent analysis module, it calculates inclusion relationship scores between each intent candidate, assigns weights according to the calculated inclusion relationship scores, determines the intent candidate whose sum of these scores is maximized as the core intent of the LLM question data, and determines at least one of the target word, constraint phrase, and domain within the sentence of the LLM question data based on the determined core intent to generate a first signature (S120); (c) (d) A duplicate request blocking module (130) calculates a similarity between vectors by comparing the embedding vector of a first signature generated by the signature generation module (120) with the embedding vectors of existing signatures generated prior to the first signature, and if the calculated similarity is greater than or equal to a preset reuse threshold, reuses an existing response corresponding to an existing signature (S130); (d) After the intention evaluation module (140) receives the first signature generated by the signature generation module (120), if the similarity calculated by the duplicate request blocking module is less than a preset reuse threshold, calculates a utility score of the LLM question data based on a preset utility score calculation standard, calculates a cost score based on a preset cost score calculation standard, or calculates a risk score based on a preset risk score calculation standard (S140);(e) a step (S150) in which a net utility calculation module (150) receives one of a utility score, a cost score, or a risk score from an intention evaluation module (140), inputs it into a preset net utility score formula, assigns weights to the cost score or the risk score, and calculates a net utility score; and (f) a step (S160) in which a path determination module (160) selects a first model path if the net utility score calculated by the net utility calculation module (150) is greater than or equal to a preset utility threshold, and selects a second model path if the net utility score is less than the utility threshold. A method for controlling an execution path based on a net utility score of an AI request.; Claim 15 In claim 14, (g) a context management module (170) collects multiple context candidates semantically associated with the requested LLM question data among external search results, search augmentation generated data, conversation history, tool output results, or policy metadata, and calculates semantic embedding-based similarity between each context candidate and the LLM question data, determines candidates whose similarity is greater than or equal to a reference context threshold as associated contexts, calculates the utility contribution per token of each associated context candidate, attaches context candidates whose calculated utility contribution is greater than or equal to a preset utility threshold to a prompt, removes context candidates whose utility contribution is less than the preset utility threshold, and adjusts the length of the prompt for the LLM question data (S170); (h) an output risk control module (180) performs a re-identification risk assessment based on context similarity, token patterns, and entity name co-occurrence frequency, and allows restoration within the policy scope only when the risk is less than a reference (S180); and (i) a monitoring module (190) each LLM A method for controlling an execution path based on the net utility score of an AI request, comprising the step (S190) of recording the utility score (U), cost score (C), risk score (R), net utility score (NUS), selected path, response information, and judgment basis for each question data in an immutable form and utilizing them for threshold adjustment, policy verification, and performance analysis.

Citation Information

Patent Citations

  • Method and apparatus for providing llm-based customer counseling services

    KR1020250090445A

  • Prompt relay system for optimizing prompting of generative ai based on large language model

    KR1020250132256A

  • System for providing large langauge model service for business

    KR102827595B1

  • Method and system for enabling trustworthy artificial intelligence systems through transparent model analysis

    US20250265545A1

  • Cache replacement for text data using semantic diversity

    US20250328566A1