Method and controller configured to execute agent for adapting LLM response based on user preferences

Abstract

A method for an agent based on a large-language model LLM comprises receiving a text query from a user, transforming to an LLM query and sending to the LLM. Thereafter, receiving an LLM response and evaluating the LLM response by determining one or more features of the LLM response, determining one or more targets of the text query, determining a distance between the one or more features of the LLM response and the one or more targets of the text query, determining that the LLM response does not meet an expected target when the distance falls below a threshold acceptance distance. If the LLM response does not meet an expected target, then generating an improved LLM response by re-parsing the text query to generate an improved LLM query and sending the improved LLM query to the LLM. Finally, the method includes providing the improved response to the user.

Description

[0001] METHOD AND CONTROLLER CONFIGURED TO EXECUTE AGENT FOR ADAPTING LLM RESPONSE BASED ON USER PREFERENCES

[0002] TECHNICAL FIELD

[0003] The present disclosure relates generally to the field of large language models; and more specifically, to a method and a controller configured to manipulate input to meet user’ s comprehension targets and to execute an agent based on a Large Language Model (LLM), such as a context synchronization in continuous LLM sessions.

[0004] BACKGROUND

[0005] Advancements in the field of Natural Language Processing (NLP) have gained popularity over the years due to a plethora of applications, such as machine translation, sentiment analysis, and chatbots. The NLP is a subfield of Artificial Intelligence (Al) that focuses on the interaction between computers and human language. The NLP involves the development of algorithms and models that enable computers to understand, interpret, and generate human language in a way that is meaningful and contextually relevant. The NLP techniques have revolutionized various industries, including healthcare, finance, customer service, and information retrieval, by enabling efficient and accurate processing of large volumes of textual data. In the general domain of NLP, there are several challenges and problems that researchers and developers have been addressing. One such key challenge includes output optimization and personalization of the output. Moreover, the output optimization and personalization are some of the promising vectors of the LLM improvement. However, current, systems and techniques are focused on smart agents able to learn user habits and preferences and manipulate prompts in order to deliver output matching the user’s expectations (readability, volume, used language, information loading, and the like.). Furthermore, typical adapting agent architecture includes a user metering system for learning user preferences and creating expected output characteristics and an input optimizer for modifying the original user’s input to reach optimization targets. After being transformed, the generated response is sent to LLM for further handling and the resulting output is returned to the user. Moreover, the user metering system periodically performs analysis of the user’s inputs, related outputs, sentiments, and feedback to update preferences while the preferences changes may take effect after delay (that may reach hours or days). As a result, the generally optimized prompt may create unexpected and far from optimal output. In addition, the LLM is random by nature (i.e., 30% of outputs might not be in line with prompt guidelines) and the current systems that utilize prompt engineering techniques (e.g., in context learning (ICL), chain of thoughts (COT), and the like.) are complicated and not so generic and robust. Thus, there exists a technical problem of how to deliver an optimal output despite possible optimization failures, LLM confusions, output randomization, and the like.

[0006] Therefore, in light of the foregoing discussion, there exists a need to overcome the aforementioned drawbacks associated with the conventional methods and controllers for executing the agent based on LLM.

[0007] SUMMARY

[0008] The present disclosure provides a method, and a controller configured to Manipulate LLM prompt with goal to achieve output meeting user’s expectations / skills, and the like. Furthermore, the method and the controller are also configured to execute an agent based on a Large Language Model (LLM). The present disclosure provides a solution to the existing problem of how to deliver an optimal output despite possible optimization failures, LLM confusion, output randomization, and the like. An aim of the present disclosure is to provide a solution that overcomes at least partially the problems encountered in prior art, and provide an improved method, and controller configured to execute an agent based on a Large Language Model (LLM). The object of the present disclosure is achieved by the solutions provided in the enclosed independent claims. Advantageous implementations of the present disclosure are further defined in the dependent claims.

[0009] In one aspect, the present disclosure provides a method for an agent based on a large-language model (LLM). The method comprises the agent for receiving a text query from a user, transforming the text query to an LLM query, sending the LLM query to the LLM, receiving an LLM response to the LLM query from the LLM, and providing the LLM response to the user, wherein the method further comprises the agent evaluating the LLM response and if the LLM response does not meet an expected target, generating an improved LLM response and providing the improved response to the user. Moreover, the method is characterized in that the method further comprises the agent evaluating the LLM response by determining one or more features of the LLM response, determining one or more targets of the text query, determining a distance between the one or more features of the LLM response and the one or more targets of the text query, determining that the LLM response does not meet an expected target when the distance falls below a threshold acceptance distance and generating the improved LLM response by re-parsing the text query to generate an improved LLM query and sending the improved LLM query to the LLM wherein the improved LLM response is the response to the improved LLM query.

[0010] Advantageously, the method is used to enhance the quality and relevance of the outputs generated by Large Language Models (LLMs). In an implementation, a software runs on the endpoint device and supports the interaction with the LLM in order to retrieve the LLM response. The method is used to analyze the LLM response against predefined targets derived from user preferences, ensuring that the outputs are accurate and aligned with the user's expectations. Furthermore, the method is used to adjust and refine the user queries based on user-specific targets, such as text readability and language complexity, without requiring manual intervention, thereby enhancing the adaptability of LLM responses for a wide range of applications. Moreover, the output evaluation module (or an output evaluator) is aware about the target set for the prompt generator so that the output evaluation module can understand how the LLM response is far from the targets and how specific prompt affects the overall LLM response. Additionally, by limiting the number of iterations and ensuring that the user receives an optimized response within a reasonable timeframe, the method is further used to maintain a balance between efficiency and output quality. A generative feedback, which is a function of the output targets and recent prompt sent to the LLM is used by the output evaluator to predict the expected output and compare the generated LLM response with the actual to calculate delta. Moreover, such generative feedback is further utilized for prompt manipulation also. As a result, the method provides reliable, optimal outputs despite potential challenges such as optimization failures, LLM confusion, and output randomization.

[0011] In another aspect, the present disclosure provides a controller configured to execute an agent based on a large-language model, and the controller is configured to execute an input optimizer module configured to receive a text query from a user, transform the text query to an LLM query, send the LLM query to the LLM, receive a LLM response to the LLM query from the LLM, and provide the LLM response to the user. Moreover, the agent is further configured to execute an evaluation module configured to evaluate the LLM response and if the LLM response does not meet an expected target, generate an improved LLM response and provide the improved response to the user, wherein the controller is characterized in that the evaluation module is further configured to evaluate the LLM response by determining one or more features of the LLM response, determining one or more targets of the text query, determining a distance between the one or more features of the LLM response and the one or more targets of the text query, determining that the LLM response does not meet an expected target when the distance falls below a threshold acceptance distance and wherein the controller is further characterized in that the controller is further configured to execute a redirector module configured to generate feedback based on the distance between the one or more features of the LLM response and the one or more targets of the text query further comprises determining one distance between each of the one or more features of the LLM response and the corresponding target of the one or more targets of the text query, and send the feedback to the input optimizer module whereby the input optimizer module is further configured to generate the improved LLM response by re-parsing the text query to generate an improved LLM query, and sending the improved LLM query to the LLM wherein the improved LLM response is the response to the improved LLM query. The controller achieves all the advantages and technical effects of the method after execution of the method.

[0012] In yet another aspect, the present disclosure provides a computer program comprising instructions for carrying out all the steps of the method, when said computer program is executed on a computer system.

[0013] The computer program achieves all the advantages and effects of the method after execution of the method.

[0014] It is to be appreciated that all the aforementioned implementation forms can be combined.

[0015] It has to be noted that all devices, elements, circuitry, units and means described in the present application could be implemented in the software or hardware elements or any kind of combination thereof. All steps which are performed by the various entities described in the present application as well as the functionalities described to be performed by the various entities are intended to mean that the respective entity is adapted to or configured to perform the respective steps and functionalities. Even if, in the following description of specific embodiments, a specific functionality or step to be performed by external entities is not reflected in the description of a specific detailed element of that entity which performs that specific step or functionality, it should be clear for a skilled person that these methods and functionalities can be implemented in respective software or hardware elements, or any kind of combination thereof. It will be appreciated that features of the present disclosure are susceptible to being combined in various combinations without departing from the scope of the present disclosure as defined by the appended claims.

[0016] Additional aspects, advantages, features and objects of the present disclosure would be made apparent from the drawings and the detailed description of the illustrative implementations construed in conjunction with the appended claims that follow.

[0017] BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The summary above, as well as the following detailed description of illustrative embodiments, is better understood when read in conjunction with the appended drawings. For the purpose of illustrating the present disclosure, exemplary constructions of the disclosure are shown in the drawings. However, the present disclosure is not limited to specific methods and instrumentalities disclosed herein. Moreover, those skilled in the art will understand that the drawings are not to scale. Wherever possible, like elements have been indicated by identical numbers.

[0019] Embodiments of the present disclosure will now be described, by way of example only, with reference to the following diagrams wherein:

[0020] FIG. 1 is a diagram illustrating a flowchart of a method for an agent based on a large-language model (LLM), in accordance with an embodiment of the present disclosure, in accordance with an embodiment of the present disclosure;

[0021] FIG. 2 is a diagram illustrating a large language model (LLM) system to execute an agent based on a large-language model, in accordance with an embodiment of the present disclosure;

[0022] FIG. 3 illustrates an exemplary LLM system depicting the evaluation of the LLM response, in accordance with an embodiment of the present disclosure; and

[0023] FIG. 4 illustrates a flowchart of the operations of the evaluation of the LLM response, in accordance with an embodiment of the present disclosure. In the accompanying drawings, an underlined number is employed to represent an item over which the underlined number is positioned or an item to which the underlined number is adjacent. A non-underlined number relates to an item identified by a line linking the non-underlined number to the item. When a number is non-underlined and accompanied by an associated arrow, the non-underlined number is used to identify a general item at which the arrow is pointing.

[0024] DETAILED DESCRIPTION OF EMBODIMENTS

[0025] The following detailed description illustrates embodiments of the present disclosure and ways in which they can be implemented. Although some modes of carrying out the present disclosure have been disclosed, those skilled in the art would recognize that other embodiments for carrying out or practicing the present disclosure are also possible.

[0026] FIG. 1 is a flowchart of a method for an agent based on a large-language model (LLM), in accordance with an embodiment of the present disclosure. With reference to FIG. 1, there is shown a method 100 for the agent based on a large-language model (LLM). The method 100 includes steps 102 to 118. A controller is configured to execute the method 100.

[0027] There is provided the method 100 for the agent based on the LLM model. The method 100 is used to evaluate the LLM response before delivering the LLM response to a user in order to provide an accurate, efficient, and reliable output.

[0028] At step 102, the method 100 includes receiving a text query from the user by the agent. In an implementation, the text query is received from the user via the user device through an interface, such as a chat window, voice input converted to text, or another text input mechanism. Moreover, the text query is used to obtain specific information, solve a problem, or request an action. In addition, the receiving of the text query triggers an entire sequence of processing, evaluation, and response generation.

[0029] At step 104, the method 100 includes transforming the text query to an LLM query by the agent. The text query into multiple components, such as identifying keywords, intent, and context, such as through parsing. Thereafter, the structure of the test query is aligned by simplifying language, reordering words for clarity, or adding context for the LLM model. Furthermore, the text query is converted into the LLM query format. Text queries by the users may be ambiguous, complex, or not structured in a way that can be understood by the LLM model. Therefore, by transforming the text query into the LLM query, the likelihood of misinterpretation is reduced and the possibility of getting an accurate, relevant, and aligned LLM response is enhanced.

[0030] At step 106, the method 100 includes sending the LLM query to the LLM by the agent. The LLM is configured to process natural language and generate LLM response that can answer questions, provide information, or perform specific tasks requested by the user, such as through the text query. As a result, the transmission of the LLM query to the LLM reduces the computational overhead on the LLM, leading to a quick response time with reduced resource consumption. Additionally, such transmission also enhances the accuracy and relevance of the LLM response, as the text query is provided to the LLM in the form of LLM query leading to an improved user satisfaction through best possible LLM response that can be further generated by the LLM in response to the LLM query.

[0031] At step 108, the method 100 includes receiving an LLM response to the LLM query from the LLM by the agent. The LLM response is received to further evaluate, refine, or directly provide the same to the user in order to maintain a smooth and responsive user experience. The agent is configured to receive the LLM response to allow real-time data processing and decision-making that enhance the overall performance of an LLM system.

[0032] At step 110, the method 100 includes providing the LLM response to the user by the agent. In other words, after receiving the LLM response, the agent may perform further data processing, such as response evaluation to ensure that the generated LLM response is clear, accurate, and aligned with the user's expectations. Moreover, such LLM response is then delivered to the user through the user device via chat window or text display. As a result, the agent is configured to provide relevant LLM response, which enhances the user satisfaction along with the overall efficiency of the LLM system by ensuring that the LLM response is directly useful to the user and is presented without any unnecessary delays.

[0033] At step 112, the method 100 includes evaluating the LLM response. The evaluation of the LLM response is performed to determine if the generated LLM response meets the desired quality, accuracy, and relevance according to the user's expectations or predefined criteria or not. Moreover, the LLM response is evaluated to ensure if the generated LLM response require further refinement or not in order to maintain high output quality and also reduces the risk of delivering unsatisfactory responses that enhances the overall reliability of the LLM system by ensuring that only well-evaluated content reaches the user.

[0034] At sub-step 112 A, the method 100 includes evaluating the LLM response by determining one or more features of the LLM response. In an implementation, the agent is configured to analyze the one or more features of the LLM response in order to assess how well the LLM response aligns with the desired outcomes. Moreover, examples of the one or more features may include but are not limited to readability, tone, accuracy, length of the response, adherence to user preferences, and the like. Additionally, the evaluation of the LLM response by the determination of the one or more features of the LLM response allows a granular and precise assessment of the generated LLM response, enabling the LLM system to identify specific areas for improvement and further correct the final output for the user before sending the same to the user.

[0035] At sub-step 112B, the method 100 includes evaluating the LLM response by determining one or more targets of the text query. Firstly, the one or more targets, such as readability, context, tone, complexity, and the like of text query are determined by the agent. Thereafter, the determined one or more targets are used to evaluate the LLM response, such as by analyzing the text query to extract key objectives, for example, answering a specific question, providing detailed information, or fulfilling a particular request. In an implementation, the text readability score, (e.g., a score that is measured by the Flesch-Kincaid metric) can be used to assess the readability of the text. Similarly, in another implementation, the language complexity score can be measured by the Dale-Chall metric that can be used to evaluate the sophistication of the language used, where higher scores indicate more complex text. In such implementations, if the LLM response fails to meet the expected targets for these features, such as not achieving a required readability level or exceeding the desired complexity, then, in that case, the LLM response may be further refined, without affecting the scope of the present disclosure. As a result, such determination of the one or more targets of the text query enhances the precision and relevance of the evaluation, ensuring that the LLM response is not only of high quality but also closely aligned with the specific needs and goals of the user, leading to an accurate LLM response.

[0036] In accordance with an embodiment, the method 100 further includes determining the target based on user preferences by the agent. In an implementation, the target refers to a criterion based on which the LLM responses are generated by the LLM, for example, tone (e.g., formal or casual), the level of detail (e.g., concise or comprehensive), readability, accuracy, and the like. As a result, the determination of the target based on the user preferences in order to ensure that the generation of the LLM response is more relevant and satisfying to the user thereby enhancing the overall user experience.

[0037] In accordance with an embodiment, the method 100 further includes determining the user preferences based on monitoring the inputs and outputs to a user, and the user’s reactions in a user metering system by the agent. In an implementation, the agent is configured to track and analyze the user preferences that include text queries, LLM responses, and the user reactions (e.g., clicks, time spent, feedback, and the like) to allow the user to dynamically learn and adapt the user preferences thereby ensuring that the LLM responses are customized based on the user preferences. In an implementation, the metering system is configured to collect data on user behavior, analyze patterns, and then update the same. As a result, the determination of the user preferences based on the monitoring inputs and outputs to the user leads to an intuitive and effective user experience. At sub-step 112C, the method 100 includes evaluating the LLM response by determining a distance between the one or more features of the LLM response and the one or more targets of the text query. In other words, the controller is configured to measure the characteristics of the LLM response that aligns with the predefined targets derived from the user’s query in order to provide a quantifiable way to assessing whether the LLM response meets the user’s expectation. In an implementation, the distance between the one or more features includes differences in readability, tone, content coverage, or other features that are compared against the targets set by the user's query. As a result, the determination of the distance between the one or more features of the LLM response and the one or more targets of the text query along with the recent prompt allows precise and objective evaluation of the response quality, enabling the controller to make informed decisions about whether to deliver the response to the user or to refine the LLM response further.

[0038] In accordance with an embodiment, the method 100 further includes determining the distance between the one or more features of the LLM response and the one or more targets of the text query further comprises determining one distance between each of the one or more features of the LLM response and the corresponding target of the one or more targets of the text query by the agent. Moreover, the determination of the distance between the one or more features of the LLM response and the one or more targets of the text query is used to provide a detailed and nuanced evaluation of the generated LLM response thereby, ensuring that no feature falls short of the desired standard. As a result, by determining the distance between the one or more features of the LLM response is used to identify specific areas (e.g., readability, tone, context, and the like) for improvement and make targeted adjustments, thereby enhancing the accuracy and relevance of the final output (or the LLM response) that is delivered to the user. Additionally, the determination of the distance is also beneficial in the case of several iterations output evaluator (or an output evaluation module) can understand impact of changes introduced in prompt and adjust change aggressiveness.

[0039] In accordance with an embodiment, the method 100 further includes determining the one or more targets of the text query when parsing the text query by the agent. In other words, the agent is configured to process and analyze the text query provided by the user and further identify the one or more targets of the text query while parsing the text query. In an implementation, the one or more targets of the text query may include desired level of detail, tone, or specific content areas the user is interested in. As a result, the method 100 is used to capture and address the user's intent right from the parsing stage, ensuring that the LLM’s response is directly aligned with the specific goals of the text query. By determining the one or more targets during parsing, the agent is configured to generate response effectively, resulting in LLM responses that are more relevant, targeted, and satisfying to the user’s requirement.

[0040] In accordance with an embodiment, the method 100 further includes parsing the text query in an input optimizer module by the agent. In an implementation, the input optimizer module is configured to utilize natural language processing techniques to dissect the query, identifying elements, such as keywords, context, and user preferences. Moreover, by parsing the text query within the input optimizer module, the controller is configured to transform complex or ambiguous queries into a well-defined, structured inputs that the LLM can process more effectively and accurately. As a result, the overall accuracy and quality of the LLM response is enhanced resulting in an efficient and satisfying user experience.

[0041] At sub-step 112A, the method 100 includes evaluating the LLM response by determining that the LLM response does not meet an expected target when the distance falls below a threshold acceptance distance. The agent is configured to assess the LLM response that aligns with the expected targets, such as by comparing the determined distance between the LLM response and the expected target. In an implementation, the LLM response is evaluated by determining the LLM response. Moreover, if the distance falls below the threshold acceptance distance, then, in that case, the LLM response does not meet an expected target. As a result, the method is used to enhance the reliability and accuracy of the LLM response that reduces the likelihood of delivering unsatisfactory results to the user, thereby improving the overall user experience and maintaining a high standard of response quality. In accordance with an embodiment, the expected target relates to Text Readability and / or Language Complexity. In an implementation, the expected target relates to the text readability. In another implementation, the expected target relates to the language complexity. In yet another implementation, the expected target relates to the text readability and language complexity. By focusing on text readability and the language complexity as expected targets, the method 100 is used to generate the LLM responses that match the specific needs and preferences of the users.

[0042] In accordance with an embodiment, the distance indicates an aspect to be improved for the LLM query. The agent then uses the distance to modify the LLM query that can be further utilized to generate the LLM response that aligns with the user’s expectations by allowing a precise, data-driven approach for refining the LLM responses that leads to an efficient query optimization and faster convergence of the LLM response that meets the user’s expectations. As a result, by continuously refining the text query based on distance, the method 100 is used to provide accurate, efficient, and reliable LLM responses.

[0043] At step 114, if the LLM response does not meet an expected target by the agent, then, the method 100 further includes generating an improved LLM response, at step 116. Firstly, the LLM response is generated from the text query, which is received by the user. Thereafter, the generated LLM response is evaluated in order check if the generated response is meeting the user expectations and requirements or not. In an implementation, if the LLM response does not meet an expected target by the agent, then, the controller is configured to generate the improved LLM response. In another implementation, if the LLM response meets an expected target by the agent, then, the controller is configured to share the generated LLM response with the user. As a result, the method 100 is used to generate an improved LLM response in an iterative manner that enhances the adaptability and accuracy of the LLM, leading to better user experiences.

[0044] At sub-step 116 A and 116B, the method 100 includes generating the improved LLM response by re-parsing the text query to generate an improved LLM query and sending the improved LLM query to the LLM. Moreover, the improved LLM response is the response to the improved LLM query. In other words, when an initial LLM response fails to meet the expected targets, then, the controller is configured to re-parse the text query in order to modify or refine the generation of the LLM response. Moreover, by analyzing the shortcomings of the initial LLM response, making necessary adjustments to the query (e.g., clarifying intent, simplifying language, or specifying additional details), and then re-submitting the text query to the LLM allows the LLM to process the improved LLM query and further generate the improved LLM response. As a result, the generation of the improved LLM response by parsing the text query to generate the improved LLM query enhances the possibility of the generation of the LLM response that matches with the expectations of the user.

[0045] At step 118, the method 100 further includes providing the improved response to the user. In an implementation, the agent is configured to evaluate the improved LLM response in order to confirm that the generated improved LLM response meets the user expectations than the initial LLM response. Once confirmed, the improved LLM response is forwarded to the user, replacing the initial LLM response. As a result, the method 100 is used to ensure that the user receives the response that is more aligned with the user expectations and the targets set for the query, thereby increasing the quality and relevance of the interaction.

[0046] In accordance with an embodiment, the method 100 further includes keeping count of the number of improved LLM queries generated for a text query and determine that the improved LLM response is to be provided to the user regardless of whether the distance falls below the threshold acceptance distance or not when the number of LLM responses reaches a threshold number by the agent. In an implementation, the method 100 is used to track the number of times, the LLM has generated an improved LLM response for a particular text query. Moreover, if the LLM response to the improved query does not meet the expected targets after a predetermined number of iterations or within a set time frame, the process may be halted. In such cases, the user is provided with the best available, albeit sub-optimal, response in order to prevent unnecessary delays or excessive computational resources while still delivering a usable LLM response that aligns as closely as possible with the user's expectations.

[0047] Advantageously, the method 100 is used to enhance the quality and relevance of the outputs generated by Large Language Models (LLMs). The method 100 is used to analyze the LLM response against predefined targets derived from user preferences, ensuring that the outputs are accurate and aligned with the user's expectations. Furthermore, the method 100 is used to adjust and refine the user queries based on user-specific targets, such as text readability and language complexity, without requiring manual intervention, thereby enhancing the adaptability of LLM responses for a wide range of applications. By limiting the number of iterations and ensuring that the user receives an optimized response within a reasonable timeframe, the method 100 is further used to maintain a balance between efficiency and output quality. As a result, the method 100 provides reliable, optimal outputs despite potential challenges such as optimization failures, LLM confusion, and output randomization.

[0048] The steps 102 to 118 are only illustrative, and other alternatives can also be provided where one or more steps are added, one or more steps are removed, or one or more steps are provided in a different sequence without departing from the scope of the claims herein.

[0049] There is provided a computer program product comprising instructions that, when executed by a computer, cause the computer to execute the method 100. In an example, the instructions are implemented on the computer-readable media, which include, but are not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Random Access Memory (RAM), Read-Only Memory (ROM), Hard Disk Drive (HDD), Flash memory, a Secure Digital (SD) card, Solid-State Drive (SSD), a computer-readable storage medium, and / or CPU cache memory. In an example, the instructions are generated by a computer program, which is implemented in view of the method 100 and for use in implementing the method 100 in an LLM system.

[0050] FIG. 2 is a diagram illustrating a large language model (LLM) system to execute an agent based on a large-language model, in accordance with an embodiment of the present disclosure. With reference to FIG. 2, there is shown an LLM system 200 to execute the agent based LLM. The LLM system 200 includes an agent 202, an LLM 214, and a user device 204. Moreover, the agent 202 includes a controller 206, an input optimizer module 208, an evaluation module 210, and a redirector module 212.

[0051] The LLM system 200 may be referred to as a system configured to execute an agent based on a large-language model. The LLM system 200 is configured to evaluate LLM response before delivering the LLM response to a user in order to provide an accurate, efficient and reliable output.

[0052] The agent 202 may be referred to as a component or a software service that acts as an intermediary between the user device 204, the agent 202, and the LLM 214. The agent 202 may be configured to execute the input optimizer module 208 and forward the user inputs to the LLM 214, receive responses from the LLM 214 and relay the received responses back to the user through the user device 204.

[0053] The controller 206 may be configured to execute the input optimizer module 208 in order to provide LLM response through the LLM 214. Examples of the controller 206 may include, but are not limited to, a rule-based context controller, machine learning based context controller, time-aware context controller, multi-source context aggregator, adaptive context controller, real-time context controller, and the like.

[0054] The LLM 214 is a type of artificial intelligence model designed to understand, generate, and manipulate human-like text across a wide range of topics and tasks. Examples of the LLM 214 may include but are not limited to, Generative Pre-trained Transformer (GPT) series, Bidirectional Encoder Representations from Transformers (BERT) and their variants, and the like. There is provided the controller 206 configured to execute the agent 202 based on the LLM 214. The agent 202 is used to evaluate LLM response before delivering the LLM response to a user in order to provide an accurate, efficient and reliable output.

[0055] The controller 206 is configured to execute the input optimizer module 208 configured to receive a text query from a user. In an implementation, the text query is received from the user via the user device 204 through an interface, such as a chat window, voice input converted to text, or another text input mechanism. Moreover, the text query is used to obtain specific information, solve a problem, or request an action. In addition, the receiving of the text query triggers an entire sequence of processing, evaluation, and response generation.

[0056] Furthermore, the input optimizer module 208 is configured to transform the text query to an LLM query. By transforming the text query into the LLM query, the likelihood of misinterpretation is reduced and the possibility of getting an accurate, relevant, and aligned LLM response is enhanced.

[0057] Furthermore, the input optimizer module 208 is configured to send the LLM query to the LLM 214. The LLM 214 is configured to process natural language and generate LLM response that can answer questions, provide information, or perform specific tasks requested by the user, such as through the text query. As a result, the transmission of the LLM query to the LLM 214 reduces the computational overhead on the LLM 214, leading to a quick response time with reduced resource consumption. Additionally, such transmission also enhances the accuracy and relevance of the LLM response, as the text query is provided to the LLM 214 in the form of LLM query leading to an improved user satisfaction through best possible LLM response that can be further generated by the LLM 214 in response to the LLM query. Furthermore, the input optimizer module 208 is configured to receive a LLM response to the LLM query from the LLM 214. The LLM response is received to further evaluate, refine, or directly provide the same to the user in order to maintain a smooth and responsive user experience. The agent 202 is configured to receive the LLM response to allow real-time data processing and decision-making that enhances the overall performance of the LLM system 200. Furthermore, the input optimizer module 208 is configured to provide the LLM response to the user. In an implementation, the input optimizer module 208 is configured to provide the LLM response to the user via the user device 204.

[0058] Furthermore, the agent 202 is further configured to execute the evaluation module 210 that is configured to evaluate the LLM response and if the LLM response does not meet an expected target, then generate an improved LLM response and provide the improved response to the user. The evaluation of the LLM response is performed to determine if the generated LLM response meets the desired quality, accuracy, and relevance according to the user's expectations or predefined criteria or not. Moreover, the LLM response is evaluated to ensure if the generated LLM response require further refinement or not in order to maintain high output quality and also reduces the risk of delivering unsatisfactory responses that enhances the overall reliability of the LLM system 200 by ensuring that only well-evaluated content reaches the user. Moreover, the controller 206 is characterized in that the evaluation module 210 is further configured to evaluate the LLM response by determining one or more features of the LLM response, determining one or more targets of the text query, determining a distance between the one or more features of the LLM response and the one or more targets of the text query, determining that the LLM response does not meet an expected target when the distance falls below a threshold acceptance distance. In an implementation, the agent 202 is configured to analyze the one or more features of the LLM response in order to assess how well the LLM response aligns with the desired outcomes. Moreover, examples of the one or more features may include but are not limited to readability, tone, accuracy, length of the response, adherence to user preferences, and the like. Additionally, the evaluation of the LLM response by the determination of the one or more features of the LLM response, one or more targets of the text query, one or more features of the LLM response, the one or more targets of the text query, and determining that the LLM response does not meet an expected target when the distance falls below a threshold acceptance distance. As a result, the precision and relevance of the evaluation is enhanced, thereby ensuring that the LLM response is not only of high quality but also closely aligned with the specific needs and goals of the user, leading to an accurate LLM response.

[0059] In accordance with an embodiment, the evaluation module 210 is further configured to determine the one or more targets of the text query by receiving them from the input optimizer module 208. In an implementation, the evaluation module 210 is configured to determine the one or more targets of the text query by receiving them from the input optimizer module 208 that enables the evaluation module 210 to accurately assess the LLM response against user-defined targets, ensuring alignment with the user's goals. As a result, the evaluation module 210 is configured to provide an accurate and relevant LLM responses by providing clear and direct criteria against which the LLM response can be measured.

[0060] Furthermore, the controller 206 is further configured to execute the redirector module 212 that is configured to generate feedback based on the distance between the one or more features of the LLM response and the one or more targets of the text query further comprises determining one distance between each of the one or more features of the LLM response and the corresponding target of the one or more targets of the text query and send the feedback to the input optimizer module 208. The controller 206 is configured to execute the redirector module 212 to generate and send feedback based on the calculated distance between the LLM response features and the targets of the text query and ensure that the LLM system 200 can dynamically adjust and refine the input to the LLM 214 based on real-time performance metrics, improving the overall accuracy and relevance of the generated responses. Moreover, the redirector module 212 is configured to calculate the distance between each feature of the LLM response and the corresponding target, generates feedback based on such evaluation, which is further provided to the input optimizer module 208 for further adjustments. As a result, by continuously refining the input based on feedback, the LLM system 200 is configured to align the LLM response between the user expectations and the LLM outputs, reducing errors and improving user satisfaction. Additionally, this feedback loop can optimize processing efficiency by identifying and correcting issues earlier in the query processing pipeline. Furthermore, the input optimizer module 208 is configured to generate the improved LLM response by re-parsing the text query to generate an improved LLM query and sending the improved LLM query to the LLM 214. Moreover, the improved LLM response is the response to the improved LLM query. The generation of the improved LLM response by parsing the text query to generate the improved LLM query enhances the possibility of the generation of the LLM response that matches with the expectations of the user.

[0061] Advantageously, the controller 206 is configured to enhance the quality and relevance of the outputs generated by Large Language Models (LLMs). The controller 206 is configured to analyze the LLM response against predefined targets derived from user preferences, ensuring that the outputs are accurate and aligned with the user's expectations. Furthermore, the method 100 is used to adjust and refine the user queries based on user-specific targets, such as text readability and language complexity, without requiring manual intervention, thereby enhancing the adaptability of LLM responses for a wide range of applications. By limiting the number of iterations and ensuring that the user receives an optimized response within a reasonable timeframe, the controller 206 is configured to maintain a balance between efficiency and output quality. As a result, the controller 206 is configured to provide reliable, optimal outputs despite potential challenges such as optimization failures, LLM confusion, and output randomization.

[0062] FIG. 3 illustrates an exemplary LLM system depicting the evaluation of the LLM response, in accordance with an embodiment of the present disclosure. FIG. 3 is described in conjunction with elements from FIGs. 1 and 2. With reference to FIG. 3, there is shown a diagram 300 that depicts the LLM system 200 for evaluating the LLM response before providing the same to the user.

[0063] In an implementation scenario, the LLM system 200 is configured to evaluate LLM response before providing the same to the user via user device 204. Firstly, the user device 204 is configured to send the text query to the agent 202 (or an adaptive agent). Thereafter, the agent 202 is configured to transform the text query to an LLM query and send the LLM query to the LLM 214 through the input optimizer module 208. After that, the LLM 214 generates the LLM response, which is received by the evaluation module 210 (or an output evaluator). The evaluation module 210 is further configured to evaluate the generated LLM response and suspend the LLM response delivery to the user’s in case of significantly low output qualities. Moreover, the results of evaluation (i.e., distance from target) are translated to the generative feedback, which is further forwarded to the LLM input generator for reiteration with the LLM 214. Additionally, the redirector module 212 is configured to redirect the decision and manages operational flow accordingly, such as by generation the feedback based on the distance between the one or more features of the LLM response and the one or more targets of the text query and send the feedback to the input optimizer module 208. For example, the evFunc (Ft-Fr)>threshold output can be redirected to the input optimizer module 208 for repetition. In an implementation, the evaluation module 210 is configured to measure the distance, weighted distances average, and the like, between the one or more targets and features of the recent output, for example, Ft = {FT1 , FT2, ... , FTN} and Fr = {F 1 , F2, ... , FN} . Furthermore, a feedback generator is configured to translate the calculated distances to the output controls. For example, 40% higher than the target ease of read score can be translated to “make it simpler” command forwarded to the LLM 214. In an example, the evaluation module 210 may have setTargets (Ft) called by Input Optimizer to inform about targets (Ft) that are required to be set for recent Output and reinteract (Feedback) API that can be implemented in the input optimization engine to support flow redirection. Additionally, the user metering system 302 is used to learn user’s preferences and provides output characteristics, such as by monitoring the user inputs and outputs along with their reactions in order to gather data on user behavior. As a result, the LLM system 200 is configured to ensure that the LLM responses generated by the LLM 214 are highly tailored to the user's specific needs and preferences, providing a more personalized and satisfactory user experience.

[0064] FIG. 4 illustrates a flowchart of the operations of the evaluation of the LLM response, in accordance with an embodiment of the present disclosure. FIG. 4 is described in conjunction with elements from FIGs. 1, 2, and 3. With reference to FIG. 4, there is shown a diagram of the flowchart 400 that depicts the LLM system 200 for evaluating the LLM response before providing the same to the user. With reference to FIG. 4, there is shown an input optimizer 402 (or the input optimizer module 208), an output evaluator 404 (or the evaluation module 210), a redirector 406, a feedback generator 408, and the LLM 214.

[0065] At operation 410, the input optimizer 402 is configured to receive an input from the user. Furthermore, at operation 412, the input optimizer 402 is configured to adapt prompt in order to meet the output targets. Furthermore, at operation 414, the input optimizer 402 is configured to forward the text query or the input to the LLM 214. The LLM 214 at operation 416, generates the LLM response (or an output) and further provides the same to the redirector 406 at operation 418. Furthermore, at operation 420, the output evaluator 404 is configured to measure the distance between the output and the one or more targets that are further transferred to the redirector 406, at operation 422. The redirector 406, at operation 424, is configured to determine if the generated LLM response is close to the one or more targets. Moreover, if yes, then, in that case, the redirector 406 is configured to forward the LLM output to the user, at operation 426 and if not, then, in that case, the redirector 406 is configured to check the repetitions of the generated LLM responses, such as at operation 428. Moreover, if the number of repetitions is greater than the maximum, then, in that case, the redirector 406 is configured to forward the output to the user, such as at operation 430. Moreover, at operation 432, the redirector 406 is configured to calculate the distances between the one or more targets and at operation 432, the feedback generator 408 is configured to generate the feedback and further adjust the targets. Moreover, the generated feedback is sent to the redirector 406 by the feedback generator 408, such as at operation 436. At operation 438, the redirector 406 is configured to interact with the input optimizer 402 based on the generated feedback and at operation 438, the input optimizer 402 is configured to generate an adaptive prompt, such is further adapted by the input optimizer 402 to meet the one or more targets, such as at operation 440. As a result, the LLM system 200 is configured to provide an improved LLM response to the user.

[0066] Modifications to embodiments of the present disclosure described in the foregoing are possible without departing from the scope of the present disclosure as defined by the accompanying claims. Expressions such as "including", "comprising", "incorporating", "have", "is" used to describe and claim the present disclosure are intended to be construed in a non-exclusive manner, namely allowing for items, components or elements not explicitly described also to be present. Reference to the singular is also to be construed to relate to the plural. The word "exemplary" is used herein to mean "serving as an example, instance or illustration". Any embodiment described as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments and / or to exclude the incorporation of features from other embodiments. The word "optionally" is used herein to mean "is provided in some embodiments and not provided in other embodiments". It is appreciated that certain features of the present disclosure, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the present disclosure, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable combination or as suitable in any other described embodiment of the disclosure.

Claims

CLAIMS1. A method (100) for an agent (202) based on a large-language model, LLM (214), wherein the method (100) comprises the agent (202): receiving a text query from a user, transforming the text query to an LLM query, sending the LLM query to the LLM (214), receiving an LLM response to the LLM query from the LLM (214), and providing the LLM response to the user, wherein the method (100) further comprises the agent (202) evaluating the LLM response and if the LLM response does not meet an expected target, generating an improved LLM response and providing the improved response to the user, wherein the method ( 100) is characterized in that the method (100) further comprises the agent (202) for evaluating the LLM response by determining one or more features of the LLM response, determining one or more targets of the text query, determining a distance between the one or more features of the LLM response and the one or more targets of the text query, determining that the LLM response does not meet an expected target when the distance falls below a threshold acceptance distance and generating the improved LLM response by re-parsing the text query to generate an improved LLM query, and sending the improved LLM query to the LLM wherein the improved LLM response is the response to the improved LLM query.

2. The method (100) according to claim 1, wherein the expected target, relates to Text Readability and / or Language Complexity.

3. The method (100) according to claim 1 or 2, wherein the method (100) further comprises the agent (202) determining the one or more targets of the text query when parsing the text query.

4. The method (100) according to claim 2, wherein the method (100) further comprises the agent (202) parsing the text query in an input optimizer module (208).

5. The method (100) according to any preceding claim, wherein the method (100) further comprises the agent (202) for determining the distance between the one or more features of the LLM response and the one or more targets of the text query further comprises determining one distance between each of the one or more features of the LLM response and the corresponding target of the one or more targets of the text query.

6. The method (100) according to any preceding claim, wherein the distance indicates an aspect to be improved for the LLM query.

7. The method (100) according to any preceding claim, wherein the method (100) further comprises the agent (202) for keeping count of the number of improved LLM queries generated for a text query and determine that the improved LLM response is to be provided to the user regardless of whether the distance falls below the threshold acceptance distance or not when the number of LLM responses reaches a threshold number.

8. The method (100) according to any preceding claim, wherein the method (100) further comprises the determining the target based on user preferences.

9. The method (100) according to claim 7, wherein the method (100) further comprises the agent (202) for determining the user preferences based on monitoring the inputs and outputs to a user, and the user’s reactions in a user metering system (302).

10. A computer program product comprising program instructions for performing the method according to any preceding claim, when executed by one or more processors in an LLM system (200).

11. A controller (206) configured to execute an agent (202) based on a large-language model, LLM, (212) wherein the controller (206) is configured to execute an input optimizer module (208) configured to: receive a text query from a user, transform the text query to an LLM query, send the LLM query to the LLM, receive an LLM response to the LLM query from the LLM, and provide the LLM response to the user, wherein the agent (202) is farther configured to execute an evaluation module (210) configured to evaluate the LLM response and if the LLM response does not meet an expected target, generate an improved LLM response and provide the improved response to the user, wherein the controller (206) is characterized in that the evaluation module (210) is further configured to evaluate the LLM response by determining one or more features of the LLM response, determining one or more targets of the text query, determining a distance between the one or more features of the LLM response and the one or more targets of the text query, determining that the LLM response does not meet an expected target when the distance falls below a threshold acceptance distance and wherein the controller (206) is further characterized in that the controller (206) is further configured to execute a redirector module (212) configured to generate feedback based on the distance between the one or more features of the LLM response and the one or more targets of the text query further comprises determining one distance between each of the one or more features of the LLM response and the corresponding target of the one or more targets of the text query, and send the feedback to the input optimizer module (208) whereby the input optimizer module (208) is further configured to: generate the improved LLM response by re-parsing the text query to generate an improved LLM query, and sending the improved LLM query to the LLM (214) wherein the improved LLM response is the response to the improved LLM query.

12. The controller (206) according to claim 11 , wherein the evaluation module (210) is further configured to determine the one or more targets of the text query by receiving them form the input optimizer module (208).

Images