System and method for incorporating user feedback into a retrieval augmentation tool based on a large language model.

Abstract

A system for incorporating user feedback into large language model (LLM)-based retrieval augmentation (RAG) tools includes a user feedback incorporation (UFA) application. The UFA receives input into the LLM from a user of a host device, generates an LLM output in response to the input, prompts a system user to verify the LLM output and provide user feedback via a human-machine interface (HMI) of the host device, and, in response to the user feedback, deploys a performance tuning optimizer that modifies the LLM output by revising domain knowledge, revising system requests, and performing constrained regeneration of the LLM output.The performance tuning optimizer progressively reduces the use of computing resources over time, progressively increases computing efficiency, and progressively reduces the reliance on human reviewers and user feedback, with the LLM output being a command to one or more systems of the host device.

Description

introduction

[0001] This disclosure relates to systems and methods for generating validation test suites for AI-powered tools, and more specifically, the automatic generation of validation and evaluation test case suites for tools powered by large language models. Artificial intelligence (AI) models that include large language models (LLMs) are increasingly used to perform tasks for end users in a variety of technical and non-technical endeavors. The results or outputs of AI models that include LLMs can be affected by the lack of systematic coverage metrics, non-diversified test sets, and uneven test coverage and coverage measurements. Accordingly, LLMs are trained on massive datasets, often from a variety of sources, and then subjected to retrieval-augmented generation (RAG) processes.Retrieval-augmented generation (RAG) is used to optimize the output of LLMs to ensure accuracy. However, even RAG-supported LLMs can produce output that is inaccurate, inappropriate, unsuitable, or otherwise compromised for a variety of reasons.

[0002] Although current systems and procedures for validating tools powered by generative AI fulfill their intended purpose, there is a need for a new and improved system and procedure that provides a systematic approach to automatically determine the accuracy, relevance, and consistency of LLMs supported by RAG tools. This system should ensure the accuracy, precision, consistency, and reliability of the LLM outputs and provide redundant and consistent checks to maintain these qualities while preserving or reducing system complexity. User feedback should also be provided to further ensure the accuracy, relevance, and consistency of the LLM outputs. Summary

[0003] From several perspectives, a system for incorporating user feedback into large language model (LLM)-based retrieval augmentation (RAG) tools comprises a host device with a controller. The controller includes a processor, memory, and input / output (I / O) ports. The I / O ports are connected to a human-machine interface (HMI) and one or more databases. The processor executes programmatic control logic stored in memory. This programmatic control logic includes an application for incorporating user feedback (UFA) into LLM-based RAG tools. The UFA comprises at least a first, a second, a third, and a fourth control logic. The first control logic receives input into the LLM from a user of the host device. The second control logic generates an LLM output as a response to the input.The third control logic causes a system user to verify the LLM output and provide user feedback via a human-machine interface (HMI) of the host device. In response to this user feedback, the fourth control logic deploys a performance tuning optimizer that modifies the LLM output by revising domain knowledge, revising system prompts or requests, and performing a limited rebuild or regeneration of the LLM output. Over time, the performance tuning optimizer progressively reduces the use of computing resources, progressively increases computing efficiency, and progressively reduces reliance on human reviewers and user feedback, where the LLM output is a command to one or more systems of the host device.

[0004] In another aspect of the present disclosure, the first control logic further comprises a control logic for receiving the input via the human-machine interface (HMI) of the host device.

[0005] In another aspect of the present disclosure, the second control logic further comprises control logic for employing an ensemble retriever. The ensemble retriever determines a similarity between the input and predetermined data in one or more databases stored in memory. The second control logic further comprises control logic for causing the ensemble retriever to generate the LLM output and an output confidence score, control logic for causing the human reviewer to prioritize and review the LLM output according to the output confidence score, and control logic for mapping the output confidence score to the LLM output. The LLM output commands one or more actuators of the host device to adjust the performance of relevant systems of the host device.The second control logic further includes a control logic that causes the human reviewer to prioritize and review the LLM output according to the output confidence score and a classified context, and a control logic that causes the human reviewer to selectively update one or more of a text vector database and a raw text database with data obtained from the input, the LLM output, and the output confidence score. The host device is a vehicle, and LLM output commands to the vehicle's one or more actuators modify the vehicle's functionality according to user-received input.

[0006] In another aspect of the present disclosure, the third control logic further comprises a control logic that prompts the system user for feedback via a confirmation request that the LLM output is a sufficiently accurate response to the input or user command. The confirmation request further comprises an audiovisual, tactile, verbal, numeric, or alphanumeric confirmation request, and the sufficiently accurate response is judged based on the user's personal preference.

[0007] In another aspect of the present disclosure, the fourth control logic further includes a control logic for executing a subroutine for revising domain knowledge (SRDK). The SRDK receives the LLM output and user feedback and determines whether the LLM output has been modified by the user. If it is determined that the LLM output has been modified by the user, it executes a control logic for optimizing a domain knowledge base (DKB).

[0008] In another aspect of the present disclosure, the control logic for optimizing the domain knowledge base (DKB) further includes control logic that retrieves the confidence score of the user-modified LLM output and determines whether the user-modified LLM output is already present in the DKB. If it is determined that the user-modified LLM output is not already present in the DKB, the control logic for optimizing the DKB receives input from human experts to verify that the user-modified LLM output was correctly added to the DKB. If it is determined that the user-modified LLM output is already present in the DKB, the control logic for optimizing the DKB revises the domain knowledge embedded in the DKB, thereby producing an updated DKB that includes the user-modified LLM output.

[0009] In another aspect of the present disclosure, the fourth control logic further includes control logic for executing a subroutine for revising system prompts (SRSP). The SRSP receives a request to verify a prompt revision, the LLM output, and user feedback, and determines whether there is sufficient evidence to implement revisions to system prompts. To determine whether there is sufficient evidence, the SRSP compares a host device prompt with the user feedback and determines whether a threshold of similarity exists between the host device prompt and the user feedback. If sufficient evidence is determined, the fourth control logic executes control logic to rewrite the system prompt, subject to a performance check.

[0010] In another aspect of this disclosure, the control logic for rewriting the system request further includes control logic that performs a regression test on a rewritten system request. The regression test uses test inputs stored in the DKB's memory. The regression test verifies that new information in rewritten system requests allows the system to continue functioning without adversely affecting system responses. If the rewritten system request is determined to function correctly, the system executes control logic that updates the system request in the DKB.If it is determined that the rewritten system request does not function correctly, the system continues to use user feedback to recursively and continuously rewrite the system request, perform regression testing on the rewritten system request, and conduct a system functionality test until the regression test indicates that the new information allows the system to continue functioning without negatively impacting system responses.

[0011] In another aspect of the present disclosure, the fourth control logic further includes control logic for executing a subroutine for constrained regeneration (SCR). The SCR uses user feedback in response to LLM outputs to revise LLM outputs based on user constraints.

[0012] In another aspect of this disclosure, the SCR further includes control logic for checking a context used by the LLM to generate the LLM output and control logic for determining whether a low-quality context is present. Low-quality contexts define low-quality matches between the context used by the LLM to generate the LLM output and a user input context. Low-quality contexts are defined according to user preferences. The SCR further includes control logic that receives user feedback via the HMI indicating the presence of a low-quality context. The low-quality context is removed, and contexts are reprioritized before control logic is executed to regenerate LLM output, subject to a context constrained by the user feedback.

[0013] In another aspect of the present disclosure, a method for incorporating user feedback into large language model (LLM)-based retrieval augmentation (RAG) tools includes executing, by means of a vehicle controller's processor, programmatic control logic stored in the controller's memory. The controller further comprises input / output (I / O) ports, the I / O ports being connected to a human-machine interface (HMI) and one or more databases. The programmatic control logic includes an application for incorporating user feedback (UFA) into LLM-based RAG tools.The UFA comprises control logic for receiving input into the LLM from a vehicle user via the vehicle's HMI, generating an LLM output in response to the input, prompting a user to verify the LLM output and provide user feedback via the vehicle's HMI; and, in response to the user feedback, initiating a performance tuning optimizer that modifies the LLM output by revising domain knowledge, revising system prompts or requests, and performing constrained regeneration of the LLM output. The performance tuning optimizer progressively reduces the use of computing resources over time, progressively increases computing efficiency, and progressively reduces reliance on human reviewers and user feedback, with the LLM output being a command to one or more of the vehicle's systems.

[0014] In another aspect of the present disclosure, the method further comprises employing an ensemble retriever, wherein the ensemble retriever determines a similarity between the input and predetermined data in one or more databases stored in memory, and causing the ensemble retriever to generate the LLM output and an output confidence score. The method further comprises causing a human reviewer to prioritize and verify the LLM output according to the output confidence score, and assigning the output confidence score to the LLM output. The LLM output instructs one or more actuators of the vehicle to adjust the performance of relevant vehicle systems.The procedure further includes instructing the human reviewer to prioritize and review the LLM output according to the output confidence score and a classified context, and instructing the human reviewer to selectively update one or more of a text vector database and a raw text database with data obtained from the input, the LLM output, and the output confidence score. LLM output commands to the vehicle's one or more actuators modify the vehicle's functionality according to the user input.

[0015] In another aspect of the present disclosure, the method further includes prompting the user for feedback via a request for confirmation that the LLM output is a sufficiently accurate response to the input or user command. The confirmation request may be audiovisual, tactile, verbal, numeric, or alphanumeric, and the sufficiently accurate response is judged based on the user's personal preference.

[0016] In another aspect of the present disclosure, the method further includes the deployment of a domain knowledge revision subroutine (SRDK). The SRDK receives the LLM output and user feedback, determines whether the LLM output has been modified by the user, and, if determined to be modified, executes control logic to optimize a domain knowledge base (DKB).

[0017] In another aspect of the present disclosure, optimizing the DKB further includes retrieving the confidence score of the LLM output modified by the user and determining whether the LLM output modified by the user is already present in the DKB. If it is determined that the LLM output modified by the user is not already present in the DKB, the procedure receives input from human experts to verify that the LLM output modified by the user was correctly added to the DKB. If it is determined that the LLM output modified by the user is already present in the DKB, the procedure revises domain knowledge embedded in the DKB, thereby producing an updated DKB that includes the LLM output modified by the user.

[0018] In another aspect of this disclosure, the procedure further includes the deployment of a System Prompt Revision Subroutine (SRSP). The SRSP receives a request to verify a revision of a prompt, the LLM output, and user feedback, and determines whether there is sufficient evidence to implement revisions to system prompts. The SRSP further includes determining whether sufficient evidence exists by comparing a vehicle prompt with the user feedback and determining whether a threshold for similarity exists between the vehicle prompt and the user feedback. If sufficient evidence is determined, the procedure rewrites the system prompt subject to a performance review.

[0019] In another aspect of this disclosure, rewriting the system request further includes performing a regression test on the rewritten system request. The regression test uses test inputs stored in the DKB's memory. The regression test verifies that new information in rewritten system requests allows the vehicle to continue functioning without adversely affecting the vehicle's reactions or responses. If the rewritten system request is determined to function correctly, the regression test executes control logic that updates the system request in the DKB.If it is determined that the rewritten system request does not function correctly, the regression test continues to use user feedback to recursively and continuously rewrite the system request, performs the regression test on the rewritten system request, and conducts a test for vehicle functionality until the regression test indicates that the new information allows the vehicle to continue functioning without negatively affecting system responses.

[0020] In another aspect of the present disclosure, the method further includes the execution of a subroutine for constrained regeneration (SCR). The SCR uses user feedback in response to LLM outputs to revise LLM outputs based on user constraints.

[0021] In another aspect of this disclosure, the SCR further includes control logic for checking the context used by the LLM to generate the LLM output and determining whether a low-quality context is present. Low-quality contexts define low-quality matches between the context used by the LLM to generate the LLM output and a user input context. Low-quality contexts are defined according to user preferences. The SCR further includes control logic for receiving user feedback via the HMI indicating the presence of a low-quality context, removing the low-quality context, and reprioritizing contexts before executing control logic to regenerate LLM output, subject to the context constrained by the user feedback.

[0022] In another aspect of the present disclosure, a method for incorporating user feedback into large language model (LLM)-based retrieval augmentation (RAG) tools includes executing, by means of a vehicle controller's processor, programmatic control logic stored in the controller's memory. The controller further comprises input / output (I / O) ports that communicate with a human-machine interface (HMI) and one or more databases. The programmatic control logic includes an application for incorporating user feedback (UFA) into LLM-based RAG tools, including control logic for receiving input into the LLM from a vehicle user via the vehicle's HMI, generating an LLM output in response to the input, and including the deployment of an ensemble retriever.The ensemble retriever determines a similarity between the input and predetermined data in one or more databases stored in memory. The procedure further includes control logic to cause the ensemble retriever to generate the LLM output and an output confidence score, to cause a human reviewer to prioritize and review the LLM output according to the output confidence score, and to associate the output confidence score with the LLM output. The LLM output instructs one or more vehicle actuators to adjust the performance of relevant vehicle systems.The procedure further includes control logic to cause the human reviewer to prioritize and review the LLM output according to the output confidence score and a classified context, and to cause the human reviewer to selectively update one or more of a text vector database and a raw text database with data obtained from the input, the LLM output, and the output confidence score. The LLM output commands to the vehicle's one or more actuators modify the vehicle's functionality according to the inputs received from the user. The procedure also includes control logic to cause a user to verify the LLM output and to prompt the user for feedback via the vehicle's HMI, including providing a request for confirmation that the LLM output is a sufficiently accurate response to the input or user command.The confirmation request further includes an audiovisual, tactile, verbal, numeric, or alphanumeric confirmation request, and the sufficiently accurate response is judged based on the user's personal preference. In response to the user feedback, a performance tuning optimizer is employed, which modifies the LLM output by revising domain knowledge, including the use of a domain knowledge revision subroutine (SRDK). The SRDK receives the LLM output and the user feedback and determines whether the LLM output has been modified by the user. If determined to be modified, it executes control logic to optimize a domain knowledge base (DKB).The procedure further includes control logic to retrieve the confidence score of the LLM output modified by the user and determines whether the LLM output modified by the user already exists in the DKB. If it is determined that the LLM output modified by the user does not already exist in the DKB, the procedure executes control logic to obtain input from human experts to verify that the LLM output modified by the user was correctly added to the DKB. If it is determined that the LLM output modified by the user already exists in the DKB, the procedure revises the domain knowledge embedded in the DKB, thereby generating an updated DKB that contains the LLM output modified by the user.The procedure further includes control logic for revising system prompts, which involves the deployment of a System Prompt Revision Subroutine (SRSP). The SRSP receives a request to verify a prompt revision, the LLM output, and user feedback, and determines whether there is sufficient evidence to implement revisions to system prompts. The procedure further includes control logic to determine whether sufficient evidence exists by comparing a vehicle prompt with the user feedback using the SRSP, and to determine whether a threshold for the similarity between the vehicle prompt and the user feedback exists.If sufficient evidence is determined, the procedure executes control logic to rewrite the system request, subject to a performance test, and performs a regression test on the rewritten system request. The regression test uses test inputs stored in the DKB memory. The regression test verifies that new information in rewritten system requests allows the vehicle to continue functioning without adversely affecting vehicle responses. If the rewritten system request is determined to function correctly, the procedure executes control logic that updates the system request in the DKB.If the rewritten system request is determined to be malfunctioning, the procedure continues to use user feedback to recursively and continuously rewrite the system request, performs a regression test on the rewritten system request, and tests the vehicle functionality until the regression test indicates that the new information allows the vehicle to continue functioning without adversely affecting system responses. The procedure further includes control logic to perform constrained LLM output regeneration, comprising executing a constrained regeneration (SCR) subroutine. The SCR uses user feedback in response to LLM outputs to revise LLM outputs based on user constraints, checking the context used by the LLM to generate the LLM output and determining if a low-quality context is present.Low-quality contexts define low-quality matches between the context used by the LLM to generate the LLM output and a user input context. Low-quality contexts are defined according to user preferences. The process further includes control logic to receive user feedback via the HMI indicating the presence of a low-quality context, removing the low-quality context, and reprioritizing contexts before the control logic executes to regenerate an LLM output, subject to a context constrained by the user feedback. The performance tuning optimizer progressively reduces the use of computing resources over time, progressively increases computing efficiency, and progressively reduces reliance on human reviewers and user feedback, with the LLM output being a command to one or more vehicle systems.

[0023] Further areas of application will become apparent from the description provided herein. It should be understood that the description and specific examples are intended for illustrative purposes only and are not meant to limit the scope of this disclosure. Brief description of the drawings

[0024] The drawings described herein are for illustrative purposes only and are not intended to limit the scope of the present disclosure in any way. Fig. Figure 1 is a schematic representation of a system for calculating confidence scores for large language model (LLM) based retrieval augmentation (RAG) tools according to an exemplary embodiment; Fig. Figure 2 is a flowchart that illustrates the logical sequence of an application of the system for calculating confidence scores for LLM-based RAG tools. Fig. 1 according to an exemplary embodiment; Fig. Figure 3 is a flowchart that shows the logical sequence of an application for incorporating user feedback (UFA) into the system. Fig. 1 according to an exemplary embodiment; Fig. Figure 4 is a flowchart that shows the logical sequence of a subroutine for revising domain knowledge (SRDK) within the UFA of Fig. 3 represents an exemplary embodiment; Fig. Figure 5 is a flowchart that shows the logical sequence of a System Request Revision Subroutine (SRSP) within the UFA of Fig. 3 according to an exemplary embodiment; and Fig. Figure 6 is a flowchart that shows the logical sequence of a subroutine for restricted regeneration (SCR) within the UFA of Fig. 3 represents an exemplary embodiment. Detailed description

[0025] The following description is merely exemplary and is not intended to limit the present disclosure, application or uses.

[0026] Referring to Fig. Figure 1 schematically illustrates a system 10 for calculating confidence scores for large language model (LLM)-based retrieval augmentation (RAG) tools. The system 10 generally operates in or on a host device 12. The host device 12 can take a variety of forms, including a vehicle 14. However, it should be understood that the system 10 of this disclosure need not be bound to such a vehicle 14. Rather, the vehicle 14 is merely an exemplary, non-limiting embodiment with respect to which the system 10 of this disclosure is described herein.System 10 can operate in any hardware and software configuration where a generative AI-powered tool is used to receive input from a user 16, such as user commands 18, and to produce an output 20 that modifies the function of the hardware and / or software configuration or system in which the generative AI-powered tool is used. Although the vehicle 14 depicted is a car, it should be understood that the vehicle 14 can be any type of vehicle 14 without deviating from the scope or intention of this disclosure. In some non-limiting examples, the vehicle 14 could be: a car, a truck, an off-road vehicle (SUV).(e.g., sport utility vehicle), a semi-trailer truck, a tractor trailer, a tractor, a combine harvester or other such agricultural equipment, a powered or unpowered aircraft such as an airplane, a helicopter, a glider or an autogyro or gyroplane, a powered or unpowered watercraft such as a ship, a sailboat, a motorboat, a pleasure craft, a jet ski, a sailboat or the like. It should be understood that in further, non-limiting embodiments, the system 10 described herein can be adapted to function with host devices 12 such as manned and unmanned spacecraft such as satellites, rockets, space stations and other orbital and extra-orbital satellite-communication capable devices or equipment, without deviating from the scope or intention of the present disclosure.In further, non-restrictive examples, the host devices can include mobile computing platforms such as laptops, mobile phones, tablets, or any other such host device through which a user can work with a tool powered by generative AI.

[0027] System 10 further comprises a controller 22, which is a non-generalized electronic control device comprising a pre-programmed digital computer or processor 24, a non-transitory computer-readable medium or memory 26 used to store data such as control logic, software applications, instructions, computer code, data, lookup tables, etc., and a transceiver or input / output (I / O) ports 28. The computer-readable medium or memory 26 includes any type of medium accessible by a computer, such as read-only memory (ROM), random-access memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory 26. A “non-transitory” computer-readable memory 26 includes wired orWired, wireless, optical, or other communication links carry transitory electrical or other signals. Non-transient computer-readable memory 26 includes media on which data can be stored permanently and media on which data can be stored and later overwritten, such as a rewritable optical disk or disc, or an erasable storage device. Computer code includes any type of program code, including source code, object code, and executable code. The processor 24 is configured to execute the code or instructions.

[0028] When the system 10 operates on a vehicle 14, the controller 22 can include a special or dedicated Wi-Fi controller or an engine control module, a transmission control module, a body control module, an infotainment control module, etc. The transceiver or I / O ports 28 are configured to communicate wirelessly with a back office 30 using cellular protocols such as Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Enhanced Data Rates for GSM Evolution (EDGE), Universal Mobile Telecommunications Services (UMTS), High Speed ​​Packet Access (HSPA), Code-Division Multiple Access (CDMA), Evolution-Data Optimized (EV-DO / EVDO / 1xEV-DO), Short Message Services (SMS), Wi-MAX, Manufacturing Messages Specification (MMS), 2G, 3G, 4G, 5G, and wireless and cellular standards as defined in IEEE 802.1X, IEEE 802 LAN / MAN and IEEE Mobile Communication Networks Standards Committee (MobiNet-SC) define and include the like. The back office 30 can comprise one or more controllers 22 and / or one or more human experts or auditors 32, which are shown in the following figures and described in detail.

[0029] The controller 22 also includes one or more applications 34. An application 34 is a software program configured to perform a specific function or set of functions. The application 34 may contain one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, associated data, or a portion thereof, suitable for implementation in appropriate machine-readable program code. The applications 34 may be stored within the memory 26 or in additional or separate memory. Examples of applications 34 include audio or video streaming services, games, browsers, social media, etc., an algorithm that calculates confidence scores for RAG tools 38 (hereinafter referred to as CLR application 40) based on a large language model (LLM) 36 for a generative AI-powered tool or tool with the LLM 36, and an application 34 for incorporating user feedback (hereinafter referred to as UFA 41). The generative AI-powered tool or LLM 36 defines an application 34 stored either locally on the controller 22 of the vehicle 14 and / or in a remote back office 30 or a cloud-based computing device. The CLR application 40 comprises a variety of subroutines or control logic components.

[0030] In examples where the system 10 operates in or on a vehicle 14, the system 10 and the CLR application 40 can be used by the vehicle operator or a user 16 from vehicle development to validate the tools used to develop and test systems that dynamically adjust the way the vehicle 14 operates and its characteristics. In some examples, the vehicle 14 may be equipped with a navigation system 42, one or more drive motors 44 that provide and modify the torque quantities supplied to the wheels 46 of the vehicle 14 to cause the vehicle 14 to move, stop, or the like, and a steering system 48 that can adjust the direction of travel of the vehicle 14.In additional examples, the vehicle 14 is equipped with a braking system 50 which, when activated by the controller 22, causes the vehicle 14 to decelerate. The vehicle 14 may be equipped with a variety of other systems for controlling body movements, which may be used to modify or otherwise control the dynamic performance of the vehicle 14, including, but not limited to, aerodynamic control surfaces and actuators, active and / or semi-active suspension systems and actuators, and the like, without deviating from the scope or intent of this disclosure.

[0031] LLMs 36 are trained on massive datasets in one or more online and offline modes. The training data provides the LLMs 36 with a means to effectively generate outputs for tasks such as answering questions, performing mathematical calculations, performing language translation, and / or summarizing. Retrieval Augmentation Generation (RAG) is an architectural approach to improve the quality of responses generated by an LLM 36 by augmenting the LLM 36 model with external knowledge sources to enhance its internal representation of information.Implementing RAG in a system that answers questions based on the LLM 36 offers at least two main advantages: it ensures that the LLM 36 has access to the most up-to-date, reliable data and that users have access to the LLM 36's sources; it also ensures that the LLM 36's responses to user input can be trusted. Furthermore, RAG bases the LLM 36 on a set of external, verifiable facts, resulting in an LLM 36 that has few opportunities to draw on information embedded in its parameters. This reduces the chance and risk of the LLM 36 leaking sensitive data or providing false or misleading answers to user input.More specifically, the RAG tools 38 based on the LLM 36 of this disclosure extend the capabilities of LLM 36 applications and improve their efficiency by effectively utilizing customer-specific data. The customer-specific data can relate to any wide range of topics, but should generally be understood as relating specifically to certain hardware or software applications of the host device 12. In a non-restrictive example, the customer-specific or domain-specific knowledge data stored in a domain knowledge base (DKB) can relate to onboard systems of a vehicle 14, including, but not limited to, navigation systems, powertrain control systems, suspension control systems, heating, ventilation, and air conditioning (HVAC) systems, and the like.The LLMs 36 utilize customer-specific data for the RAG tool 38 that is relevant to a command, question, or task generated by the user 16 and provide the customer-specific data as context for the LLM 36 to generate a response. In many ways, RAG is an effective approach to improving the performance of the LLM 36 and is successful in supporting chatbots and question-answering (Q&A) systems that require access to domain-specific information. Domain-specific information can encompass a wide range of data relating to the specific functions of the host device 12, the vehicle 14, and the back office 30, and additionally to the onboard hardware, software, and applications for both the host device 12, the vehicle 14, and the back office 30. Domain-specific information can also refer to data provided by an original equipment manufacturer (OEM).(Original Equipment Manufacturer) specific proprietary data relating to the specific hardware and software products manufactured by the OEM. The domain-specific information can define an embedded model stored within the memory 26 of the controller 22 on board the host device 12.

[0032] Referring to Fig. 2 and with continued reference to Fig. Figure 1 shows an exemplary schematic logical flow diagram of the CLR application 40 in additional detail. The CLR application 40 receives an input 100 in offline mode. The input 100 can be received via a human-machine interface (HMI) 53 of the host device 12, which includes, but is not limited to, an audio or visual or audiovisual receiver such as a microphone or other audio sensor 52, a camera or other image sensor 54, tactile interfaces including, but not limited to, buttons and touchscreens 56, or the like. In several examples, the input 100 is a command 18 from the user 16, which is subsequently processed by the RAG tool 38, based on the LLM 36, in a series of subroutines before the output 20 is generated.The input 100 can take any of a variety of forms without deviating from the scope or intention of this disclosure. In some non-limiting examples, the input 100 can include verbal or written user commands 18 in any language, such as: commands to use the vehicle's navigation system 14, commands to search for a point of interest, commands to change the vehicle's cabin temperature 14, commands to suspend, wait, or delay, requests to obtain a solution to a mathematical statement, or any other such commands 18. The input 100 is processed by a variety of subroutines within the RAG tool 38, which is based on the LLM 36.

[0033] Output 20 is a response from host device 12 to input 100 from user 16. In several aspects, output 20 directly or indirectly modifies the function of one or more systems of host device 12, including, but not limited to, modifying one or more functions of the navigation system, powertrain control system, suspension control system, HVAC system, or the like. Output 20 may thus include an LLM output command that modifies a destination or route planning function of the navigation system, changes a mode or operation of the powertrain, suspension control, or HVAC system by directly or indirectly changing the positions of host device actuators within relevant systems of host device 12 to adjust the performance of the relevant system in response to input 100 of user 16's command 18.

[0034] The RAG tool 38 includes an Ensemble Retriever 102. Ensemble Retrievers 102 are sophisticated retrieval algorithms that improve the relevance of retrieved contextual information by pooling or merging results from multiple distinct and parallel retrievers. By using multiple parallel retrievers, the strengths of each can be effectively leveraged to obtain results with respect to the user command 18 or the input 100 more accurately than data retriever algorithms could individually. The input 100 is received within the Ensemble Retriever 102 by at least two parallel subroutines of the LLM 36-based RAG tool 38, namely a semantic tool 200 and a syntactic tool 300. The semantic tool 200 receives the input 100 in a semantic retriever 202.The subroutine of semantic retriever 202 first extracts semantic information from input 100. Semantic retriever 202 then calculates a semantic similarity score 204 for the semantic information extracted from input 100. To calculate the semantic similarity score 204, semantic retriever 202 transforms the extracted semantic information from input 100 into a semantic text vector in a vector space, such that the vector is a mathematical, graphical representation of the extracted semantic information from input 100. As used here, the meaning of any semantic vector is implicitly defined by its position and direction in a vector space. In a non-restrictive example, input 100 might contain a verbal command from user 16.It should be understood that the verbal command can be a wide variety of different commands, including, but not limited to: "increase cabin temperature," where user 16 intends that the command in input 100 causes vehicle 14 to use a heating, ventilation, and air conditioning (HVAC) system to change the temperature in the passenger compartment of vehicle 14. Each word, i.e., "increase," "cabin," and "temperature," in user 16's command in input 100 is parsed and recorded in a vector space and then compared with predetermined text vectors in a text vector database 206 stored in memory 26.

[0035] Since the individual commands of the user's speech input 100 may differ in actual diction or word choice, the vector representing the input 100 cannot be accurately represented by text vectors stored in the text vector database 206. The text vector database 206 contains a variety of semantic text vectors that define mathematical, graphic, vectorized representations of predefined semantic inputs 100, which the host device 12 is programmed to accept, respond to, and understand. The variety of text vectors in the text vector database 206 can be selected manually and / or automatically during pre-production programming of the system 10, and the variety of text vectors can be updated with new information upon the occurrence of a specific event, or can be manually or automatically updated or modified continuously, periodically, or the like.In non-restrictive examples, the text vector database 206 is updated by the human experts or reviewers 32.

[0036] Accordingly, the semantic retriever 202 calculates the semantic similarity score 204 based on the text vector representing input 100 and predefined text vector information accessed within the text vector database 206. In several aspects, the semantic similarity score 204 is a numerical, graphical, vectorized representation of the degree of similarity between the semantic structure of input 100 and the semantic structure of the text vectors that most closely approximate or correspond to the semantic text vector of input 100.After calculating the semantic similarity score 204, the semantic tool 200 performs a rank fusion calculation 208, which filters the context of the input 100 based on a threshold T to avoid "low-quality" contexts or low-quality matches between the data in the text vector database 206 and the text vector representing the input data 100. More precisely, the CLR application 40 uses a reciprocal rank fusion algorithm to re-rank and merge the results of both the semantic tool 200 and the syntactic tool 300.

[0037] That means the Ensemble Retriever 102 (Ret i ) receives a retrieved context C i1 and a vector similarity score S i1 and eliminates a context C i , if and only if the similarity score of C i such that the vector similarity score S iis lower than the threshold T. In contrast, “high-quality” contexts are defined based on an importance score, where an importance weight is based on your count of the contexts, C. i , in various retrievers, CC i , of a rank of a context, C i , in various retrievers, RC i , is calculated, and an importance score of a context, C i , is given as IC i = f (CC i , RC i The contexts are then determined based on the importance score IC. i classified. Thus, in some non-restrictive examples, a series of classified semantic similarity scores are generated according to: Reta=[(Ca1,Sa1),(Ca2,Sa2),…(Can,San)] Retb=[(Cb1,Sb1),(Cb2,Sb2),…(Cbn,Sbn)] Retc=[(Cc1,Sc1),(Cc2,Sc2),…(Ccn,Scn)] where the context {C ij}, j ≥ 1 is classified based on the similarity score, i.e.: S i1 > S i2 ... > S in This applies. The importance score IC i defines a relative importance of context C i Regarding the type of input received, 100. In a non-restrictive example, the importance score IC indicates i a low value for contexts C i , such as HVAC functions, which do not imply critical functions of the host device 12, such as powertrain, suspension, or safety-critical systems of a vehicle 14. In contrast, the importance score IC i higher than the lower value when the context is C i indicates that input 100 is closely related to, or directly implies, critical functions of host device 12.

[0038] In contrast, the syntactic tool 300, which operates in parallel with the semantic tool 200, receives the input 100 within a syntactic retriever 302. The subroutine of the syntactic retriever 302, like the subroutine of the semantic retriever 202, first extracts syntactic information from the input 100. The subroutine of the syntactic retriever 302 then calculates a syntactic similarity score 304 based on the raw text of the input 100 and a raw text database 306 stored in memory 26. The raw text database 306 contains a variety of raw text vectors that define mathematical, graphical, vectorized representations of predefined syntactic inputs 100, which the host device 12 is programmed to accept and understand.The multitude of raw text vectors in the raw text database 306 can be selected manually and / or automatically during the pre-production programming of system 10, and the multitude of raw text vectors can be updated with new information upon the occurrence of a specific event, or can be updated or modified manually or automatically, continuously, periodically, or the like. As used herein, the meaning of any raw text vector is implicitly defined by its position and direction in a vector space. In non-restrictive examples, the raw text database 306 is updated by the human experts or reviewers 32.

[0039] To calculate the syntactic similarity score 304, the subroutine of the syntactic retriever 302 transforms the extracted syntactic information from input 100 into a raw text vector in a vector space, such that the vector is a mathematical, graphical representation of the extracted syntactic information from input 100. The syntactic similarity score 304 is a numerical representation of the degree of similarity between the syntactic structure of input 100 and the data in the raw text database 36. More precisely, the syntactic similarity score 304 is calculated based on a Jaccard index and a Levenshtein distance between them. The Jaccard index, (J(A,B)=|A∩B||A∪B|=|A∩B||A|+|B|−|A∩B|), is used to measure the similarity and diversity, or difference, of the information in input 100 to the predefined raw text data stored in raw text database 306. Similarly, the Levenshtein distance is a string metric used to measure the difference between two sequences, or in this case, the difference between the raw text of input 100 and the raw text stored in raw text database 306. For example, the Levenshtein distance between two words is the minimum number of single-letter revisions (i.e., insertions, deletions, or substitutions) required to change one word into the other. The Levenshtein distance can be represented mathematically as follows: lev(a,b)={|a|if f|b|=0,|b|if f|a|=0,lev(tail(a),tail(b)if head(a)=head(b),1+min{lev(tail(a),b)lev(a,tail(b))lev(tail(a),tail(b)otherwise

[0040] It should be understood that the use of the Jaccard index and the Levenshtein distance are only intended as exemplary, non-limiting examples of the types of algorithms or functions that can be used to calculate a syntactic similarity score 304 according to the subject matter of the present disclosure.The semantic and syntactic similarity scores 204, 304 can cover ranges of values ​​that vary from application to application; but in a non-restrictive example, the semantic and syntactic similarity scores 204, 304 are variable between zero (0) and one (1), such that if there is no similarity at all, the semantic and / or syntactic similarity scores 204, 304 are equal to zero, and if there is a perfect identity between the semantic and / or syntactic similarity scores 204, 304 and the information in the text vector database 206 or the raw text database 306, the values ​​of the semantic and / or syntactic similarity scores are equal to one.

[0041] The results or outputs of the rank fusion calculation 208 and the syntactic similarity score 304 are then combined to calculate a confidence score 400. The confidence score 400 is a normalized, weighted sum of the semantic similarity scores 204 and the syntactic similarity scores 304. The confidence score can be represented as follows: Cout=Z(wi∗Scoresem+wj∗Scoresyn) where Z(...) is the normalization function and W i , W j The weights are. The semantic similarity score is 204. sem = f (rank fusion score) is and the syntactic similarity score 304 Score syn = f (Jaccard Index, Levenshtein Distance, ...) is.

[0042] In various aspects, the weights W i , W j The weights W i , W jThey can be chosen by an application developer, an original equipment manufacturer, a supplier, or the like. It should also be understood that the weights W i , W j Depending on the type of queries and the structures of the queries that the system 10 receives as inputs 100, they can be dynamic, variable or constant.

[0043] As described herein, RAG tools based on the LLM 36 generate 38 outputs 20 which, at least in pre-production processes, require verification and validation by human experts or auditors 32.However, the system 10 of the present disclosure offers several advantages, including, but not limited to, the automatic generation, via the CLR application 40 of the present disclosure, of an output confidence score for the outputs 20 of the LLM 36, so that the human experts or reviewers 32 can efficiently prioritize the verification, thereby substantially reducing the amounts of human effort and working hours necessary to validate the outputs 20 of the LLM 36, while increasing the accuracy of the LLM 36, reducing the computational effort and consumption of computing resources, and reducing the potential for human-introduced typographical, syntactic, or other such errors from a first set to a second set that is substantially smaller than the first set.In one example, the experts or reviewers 32 might choose to first verify outputs 20 of the LLM 36 with low confidence levels, and subsequently proceed to verify outputs 20 of the LLM 36 with confidence levels higher than the low confidence levels. Accordingly, confidence scores can be automatically calculated by effectively utilizing vector-based similarity scores from a retriever and the similarity score of the input 100 (e.g., the Jaccard distance) of a given input 100. It should be understood that in either pre-production or production phases or applications, as the CLR 40 is continuously used, evaluated, and updated, the amount of interaction and input from a human expert or reviewer 32 will decrease over time.This means that even in a production application where an end user or customer who is not an engineer interacts with the LLM 36, the CLR application 40 works to interpret the end user's or customer's inputs to the system 10 accurately, consistently, reliably and robustly, and accordingly, with progressively reduced use of computing resources, progressively increased computing efficiency and progressively reduced dependence on verifications by the human verifier 32, to generate a response.

[0044] If one now turns Fig. 3 to, and with continued reference to Fig. 1 and Fig. 2. Once the host device 12 or the vehicle 14 is in production, the system 10, and in particular the UFA 41, is shown in more detail in the form of a flowchart. Starting at block 500, the output 20 of the LLM 36 is received by one or more human experts or inspectors 32. In some examples, the human experts or inspectors 32 may be engineers or experts located in a back office 30 away from the host device 12 or vehicle 14, or the human experts or inspectors 32 may be users of the host device 12, such as customers or the like. Accordingly, in a non-restrictive example, the human experts or inspectors 32 may be users 16 of the vehicle 14, such as a driver, passenger, or the like.The human experts or reviewers 32 use a performance tuning optimizer 502 to verify that the outputs 20 of the LLM 36 are correct and accurate responses to the input 100 from the user 16 and to user feedback 504, such as the provision of correct and accurate reactions or responses by the host device 12 to user commands 18. That is, after using the performance tuning optimizer 502 to verify the outputs 20, the UFA 41, upon receiving user feedback 504 from human experts or reviewers 32, generates a verified LLM output 20. The user feedback 504 can take any of a variety of forms without deviating from the scope or intent of this disclosure.In a non-restrictive example, the user feedback 504 can include one or more inputs to the HMI 53 of the host device 12, including but not limited to acoustic, visual, and / or tactile user input to a microphone or other audio sensor 52, a camera or other vision sensor 54, or to tactile interfaces, including but not limited to buttons and touchscreens 56, and the like. Depending on the degree of similarity between the user's input 100 or command 18 16 and the resulting LLM output 20, the user feedback 504 can be affirmative and / or corrective.

[0045] The Performance Tuning Optimizer 502 comprises at least three different subroutines or control logics: a Domain Knowledge Revision Subroutine (SRDK) 600, a System 10 Prompt Revision Subroutine (SRSP) 700, and a Constraint Regeneration Subroutine (SCR) 800. Referring now to Fig. 4 and with continued reference to Fig. 1-3 shows the SRDK 600 in more detail in the form of a flowchart.

[0046] The SRDK 600 begins by receiving a verification request 602 from the performance tuning optimizer 502. The verification request 602 contains the LLM output 20 and the feedback 504 from user 16. In some non-restrictive examples, the verification request 602 is provided to user 16 via the HMI 53. The verification request 602 may include an audiovisual, tactile, verbal, numeric, alphanumeric, or other such request for confirmation that the LLM output 20 accurately and sufficiently represents the type of information or response from system 10 that user 16 intended via input 100 or user command 18. The confirmation request may, but is not limited to, an audiovisual, tactile, verbal, numeric, or alphanumeric confirmation request via the HMI 53.The appropriateness of the response is assessed based on the personal preferences of User 16. The SRDK 600 then determines in block 604 whether output 20 from LLM 36 has been modified by User 16. If it is determined that output 20 has not been modified, the SRDK 600 proceeds to block 606, where it terminates and combines its results with those of SRSP 700 and SCR 800 before returning to block 500 of UFA 41. However, if it is determined that output 20 has been modified by User 16, the SRDK 600 initiates process 608 to optimize DKB 51, which begins in block 610. In block 610, the SRDK 600 retrieves the confidence score of output 20 modified by User 16. Subsequently, SRDK 600 determines in block 612 whether the version 20 modified by user 16 is already present in DKB 51.If it is determined that the output 20 modified by user 16 is not already present in DKB 51, the SRDK 600 proceeds to block 614, where it requests one or more human experts or verifiers 32 to review and provide input, responses, or other such guidance to refine or improve the outputs of system 10 to respond more accurately, precisely, and consistently to the inputs 100 or commands 18 of user 16. Once the human experts or verifiers 32 have reviewed and provided the responses that improve the outputs 20 of system 10, the SRDK 600 returns to block 612 to re-evaluate whether the output 20 modified by user 16 is present in DKB 51.If block 612 determines that the user-modified edition 20 is present in DKB 51, the SRDK 600 proceeds to block 616, where the SRDK 600 revises the knowledge embedded in DKB 51 and creates an updated DKB 51 containing the new editions 20 modified by user 16 and verified by experts 32.

[0047] If one now turns Fig. 5 to, and with continued reference to Fig. Sections 1-4 provide a more detailed description of SRSP 700 in the form of a flowchart. SRSP 700 begins by receiving a request 702 from the performance tuning optimizer 502 to verify a prompt revision. The request 702 includes the LLM output 20 and the user feedback 504. In block 704, SRSP 700 then determines whether sufficient evidence has been provided to indicate that a prompt change is appropriate. That is, in block 704, System 10 and SRSP 700 compare the host device 12 prompt with the user feedback 504 to determine if a threshold of similarity exists between the host device 12 prompt and the user feedback 504.If it is determined that the similarity threshold has been reached, indicating that the request from host device 12 and user feedback 504 indicate that no changes are required, the SRSP 700 proceeds to block 706, where it terminates and combines its results with those from SRDK 600 and SCR 800 before returning to block 500 of UFA 41. However, if it is determined that the similarity threshold has not been reached, the SRSP 700 proceeds to block 708. In block 708, the SRSP 700 rewrites the request from system 10. When rewriting the System 10 prompt, the SRSP 700 attempts to increase the degree of similarity between the user feedback 504 and the System 10 prompt of the host device 12 from a first degree to a second degree that is greater than the first. Several examples illustrate the rewriting of prompts.The system 10 requests are executed electronically and automatically, or, as shown in the figures, one or more human experts or testers 32 manually rewrite the system 10 requests. Subsequently, in Block 710, the SRSP 700 uses test inputs 712 to perform a regression test on the rewritten system 10 request from Block 708. The regression test performed in Block 710 ensures that new information in the rewritten system 10 request functions correctly and that the system 10 responses have not been adversely affected by the rewritten system 10 request. In Block 714, the SRSP 700 determines, based on predetermined data such as predetermined and / or variable metrics and / or thresholds, whether the newly modified system 10 request functions satisfactorily.If it is determined that performance remains unsatisfactory, the SRSP 700 returns to block 708, where the System 10 request is rewritten in another attempt to more closely align the User Feedback 504 with the System 10 request. It is understood that, depending on the functions of the Host Device 12 implied by the System 10 request and User 16 commands 18, the threshold for determining whether performance is satisfactory or unsatisfactory may be predetermined, variable, or dynamic. In block 714, if it is determined that performance is satisfactory, the SRSP 700 proceeds to block 716, where the System 10 request is recorded in the DKB 51 or such updated DKB 51, including the new System 10 requests modified by User 16 and verified by Expert 32.It is understood that the SRSP 700 is only rarely requested to modify the System 10 requests, as the System 10 requests are an integral part of the LLM 36 functionality. Therefore, a review of the SRSP 700 process by the human expert 32 is used to reduce the possibility of erroneous data being added to the System 10 requests in the updated DKB 51'.

[0048] If one now turns Fig. 6 to, and with continued reference to Fig. In sections 1-5, the SCR 800 is described in more detail in the form of a flowchart. The SCR 800 regenerates contexts and / or reprioritizes existing contexts to regenerate the output of LLM 36 based on user 16's feedback 504. That is, the SCR 800 checks a context of the original LLM output 20 and progressively uses smaller subsets of contexts to filter out and more accurately regenerate an LLM output 20 that is more relevant to user 16's input 100 or command 18. As shown in Fig.As shown in Figure 6, the SCR 800 begins in block 802, where user 16 and / or human expert or reviewer 32 checks the contexts used by the LLM 36 to generate the LLM output 20. Subsequently, in block 804, the SCR 800 determines whether a low-quality context exists. As previously described in relation to the CLR application 40, "low-quality" contexts define low-quality matches between one data record and another, specifically the context used by the LLM 20 and user 16's input 100 in a non-restrictive example. If block 804 determines that a low-quality context exists, the SCR 800 proceeds to block 806, where the low-quality context is removed from use within the DKB 51. Subsequently, in block 808, SCR 800 reprioritizes the contexts within DKB 51 to include remote, low-quality contexts.Referring again to block 804, if it is determined that no low-quality context exists, SCR 800 proceeds directly from block 804 to block 808, where a reprioritization of contexts in DKB 51 is performed if necessary. Subsequently, SCR 800 proceeds to block 810r, where it generates a regenerated LLM output 20" that incorporates the newly prioritized contexts in DKB 51.

[0049] In a non-restrictive example of the SCR 800 in operation, user 16's input command 18 could be a command to the host device 12 or vehicle 14 to navigate to a fast-food restaurant within a 16-kilometer or 10-mile radius of user 16's current location. The LLM 36 processes user 16's input command 18, references various databases and / or DKBs 51 to determine which popular fast-food restaurants are located within a 10-mile radius of user 16's current location. The LLM 36 then produces an LLM output 20 containing a predetermined number of top-ranked contexts, such as a ranking of (i.e., locally, nationally, or internationally most popular) fast-food restaurants.Based on LLM output 20 and user 16's preferences, user 16 can determine, as in block 802, that LLM output 20 is not sufficiently accurate or specific for their own needs and can request that LLM 36 regenerate LLM output 20 based on additional constraints, such as user 16's preferences for ethnic foods, healthy foods, or the like. Subsequently, SCR 800 causes LLM 36 to generate a new, regenerated LLM output 20 constrained by user 16's preferences. That is, SCR 800 causes LLM 36 to regenerate LLM output 20 according to the context constrained by user feedback 504.Accordingly, user 16 can continuously reprioritize the LLM outputs 20 and the regenerated LLM outputs 20 based on user 16's preferences to further refine and specify the LLM outputs 20. That is, the SCR 800 uses user feedback 504 in response to the LLM outputs 20 to revise the LLM outputs 20 based on user constraints.

[0050] A System 10 and a UFA 41, including the performance tuning optimizer 502, of the present disclosure offer several advantages, including providing a systematic approach for automatically determining the accuracy, relevance, and consistency of LLMs 36 supported by RAG tools, ensuring the accuracy, precision, consistency, and reliability of the LLM output 20, and providing redundant and consistent checks to ensure that the accuracy, precision, consistency, and reliability of the LLM output 20 are maintained while maintaining or reducing the complexity of the System 10, and ensuring that user feedback 504 16 is received and implemented to further ensure the accuracy, relevance, and consistency of the LLM outputs 20, while efficiently prioritizing verification and reducing the amounts of human effort and working hours.the resources required to verify the outputs 20 of LLM 36 are substantially reduced. At the same time, System 10 and UFA 41 of the present disclosure increase the accuracy of LLM 36, while reducing the computational effort and consumption of computing resources, improving computational efficiency, and reducing the potential for human-introduced and / or automatically generated typographical, syntactic, semantic, or other such errors from a first set to a second set that is substantially smaller than the first set.

[0051] The description of the present revelation is by its very nature merely exemplary, and variations that do not deviate from the core of the present revelation are to be considered within the scope of the present revelation. Such variations are not to be regarded as a deviation from the spirit and scope of the present revelation. legend

[0052] In the drawing figures, N stands for no and Y for yes.

Claims

[1] System for incorporating user feedback into large language model (LLM) based retrieval augmentation (RAG) tools, wherein the system comprises: a host device with a controller, wherein the controller comprises a processor, memory and input / output (I / O) ports, wherein the I / O ports are connected to a human-machine interface (HMI) and one or more databases, wherein the processor executes programmatic control logic stored in memory, wherein the programmatic control logic includes an application for incorporating user feedback (UFA) into LLM-based RAG tools, wherein the UFA comprises: a first control logic that receives an input into the LLM from a user of the host device; a second control logic that generates an LLM output in response to the input; a third control logic that causes a system user to verify the LLM output and provide user feedback via a human-machine interface (HMI) of the host device; and a fourth control logic that, in response to user feedback, employs a performance tuning optimizer that modifies the LLM output by revising domain knowledge, revising system requests, and performing constrained regeneration of the LLM output, wherein the performance tuning optimizer progressively reduces the use of computational resources, progressively increases computational efficiency, and progressively reduces the reliance on human reviewers and user feedback over time, and wherein the LLM output is a command to one or more systems of the host device. [2] System according to claim 1, wherein the first control logic further comprises: a control logic for receiving input via the human-machine interface (HMI) of the host device. [3] System according to claim 2, wherein the second control logic further comprises: a control logic for employing an ensemble retriever, wherein the ensemble retriever determines a similarity between the input and predetermined data in one or more databases stored in memory; a control logic to cause the ensemble retriever to generate the LLM output and an output confidence score; a control logic to cause a human reviewer to prioritize and verify the LLM output according to the output confidence score; a control logic for mapping the output confidence score to the LLM output, wherein the LLM output commands one or more actuators of the host device to adjust the performance of relevant systems of the host device; a control logic that causes the human reviewer to prioritize and review the LLM output according to the output confidence score and a classified context; and causes the human reviewer to selectively update one or more of a text vector database and a raw text database with data obtained from the input, the LLM output, and the output confidence score, and wherein the host device comprises a vehicle and LLM output commands to one or more actuators of the vehicle modify the functionality of the vehicle according to inputs received from the user. [4] System according to claim 3, wherein the third control logic further comprises: A control logic that prompts the system user for feedback via a request for confirmation that the LLM output is a sufficiently accurate response to the input or user command, wherein the confirmation request further includes an audiovisual, tactile, verbal, numeric or alphanumeric confirmation request, and wherein the sufficiently accurate response is judged based on the user's personal preference. [5] System according to claim 4, wherein the fourth control logic further comprises: a control logic for invoking a domain knowledge revision subroutine (SRDK), wherein the SRDK receives the LLM output and user feedback and determines whether the LLM output has been modified by the user, and, if it is determined that the LLM output has been modified by the user, executes a control logic for optimizing a domain knowledge base (DKB). [6] System according to claim 4, wherein the control logic for optimizing the DKB further comprises: a control logic that retrieves the confidence score of the LLM output modified by the user and determines whether the LLM output modified by the user already exists in the DKB, wherein, If it is determined that the LLM output modified by the user is not already present in the DKB, it receives input from human experts to verify that the LLM output modified by the user has been correctly added to the DKB; and where, if it is determined that the LLM output modified by the user is already present in the DKB, it revises the domain knowledge embedded in the DKB, thereby producing an updated DKB that contains the LLM output modified by the user. [7] System according to claim 4, wherein the fourth control logic further comprises: a control logic for inserting a System Request Revision Subroutine (SRSP), where the SRSP is a request to verify a revision of a request, the LLM output and The SRSP receives user feedback and determines whether there is sufficient evidence to implement revisions to system requests, wherein, in order to determine whether there is sufficient evidence, the SRSP compares a host device request with the user feedback and determines whether there is a threshold for similarity between the host device request and the user feedback, wherein, if it is determined that there is sufficient evidence, it executes control logic to rewrite the system request, subject to a performance review. [8] System according to claim 7, wherein the control logic to rewrite the system request further comprises: a control logic that performs a regression test on a rewritten system request, wherein the regression test uses test inputs stored in the DKB's memory, the regression test verifies that new information in rewritten system requests allows the system to continue functioning without negatively affecting system responses, and wherein, if it is determined that the rewritten system request functions correctly, it executes a control logic that updates the system request in the DKB, and where, if it is determined that the rewritten system request does not function properly, it continues to use user feedback to recursively and continuously rewrite the system request, performs a regression test on the rewritten system request, and tests the system functionality until the regression test indicates that the new information allows the system to continue functioning without negatively affecting system responses. [9] System according to claim 4, wherein the fourth control logic further comprises: a control logic to execute a subroutine for constrained regeneration (SCR), where the SCR uses user feedback in response to LLM outputs to revise LLM outputs based on user constraints. [10] System according to claim 9, wherein the SCR further comprises: a control logic for checking a context that is used by the LLM to generate the LLM output; a control logic for determining whether a low-quality context exists, where low-quality contexts define low-quality matches between the context used by the LLM to generate the LLM output and a user input context, where low-quality contexts are defined according to user preferences; and A control logic that receives user feedback via the HMI indicating that a low-quality context is present, removing the low-quality context and reprioritizing contexts before executing a control logic to regenerate an LLM output, subject to the context constrained by the user feedback.

Images