Calculating confidence score for retrieval enhancement tool based on large language model

Abstract

A system for calculating a confidence score for a large language model (LLM)-based retrieval enhancement (RAG) tool includes a host device having a controller. The controller executes program control logic, the program including an algorithm to calculate a confidence score for an LLM-based RAG tool (CLR application). The CLR application receives input to the LLM from a user and uses an integrated retriever. The integrated retriever determines a similarity between the input and predetermined data in the one or more databases. An integrated retriever generates an output and an output confidence score. The CLR application then causes the human verifier to prioritize and review an output according to the output confidence score, wherein the output is a command to one or more systems of the host device. As time goes on, the system gradually reduces the utilization rate of computing resources, gradually improves the computing efficiency, and gradually reduces the dependence on human verifiers.

Description

Technical Field

[0001] This disclosure relates to systems and methods for generating validation test suites for artificial intelligence-driven tools, and more specifically to automatically generating validation and evaluation test suites for large language model-driven tools. Artificial intelligence (AI) models, including large language models (LLMs), are increasingly used to perform tasks for end-users across a variety of technical and non-technical fields. The output of AI models (including LLMs) can be hampered by a lack of systematic coverage metrics, non-diversified test sets, and uneven test coverage and coverage measurements. Therefore, LLMs are trained on large amounts of data (typically from various sources) and then optimized using a retrieval augmentation generation (RAG) process to ensure accuracy. However, even RAG-assisted LLMs may generate outputs that are less relevant, inaccurate, inappropriate, or compromised for various reasons. Background Technology

[0002] Therefore, while current systems and methods for validating generative AI-driven tools have achieved their intended purpose, there is still a need for new and improved systems and methods that provide a systematic approach to automatically determine the accuracy, relevance, and consistency of RAG tool-assisted LLMs to ensure the accuracy, precision, consistency, and reliability of LLM outputs, and to provide redundancy and consistency checks to ensure that the accuracy, precision, consistency, and reliability of LLM outputs are maintained or reduced while keeping system complexity down. Summary of the Invention

[0003] According to several aspects of this disclosure, a system for calculating confidence scores for a Large Language Model (LLM)-based Retrieval Enhancement (RAG) tool includes a host device with a controller. The controller has a processor, memory, and input / output (I / O) ports. The I / O ports communicate with a Human-Machine Interface (HMI) and one or more databases. The processor executes program control logic stored in memory. The program control logic includes algorithms for calculating confidence scores for an LLM-based RAG tool (CLR application). The CLR application includes at least first, second, third, and fourth control logic. The first control logic receives LLM input from a user on the host device. The second control logic enables an integrated retrieval unit. The integrated retrieval unit determines the similarity between the input and predetermined data stored in one or more databases in memory. The third control logic causes the integrated retrieval unit to generate outputs and output confidence scores. The fourth control logic causes a human verifier to prioritize and review the outputs based on the output confidence scores. The outputs are commands to one or more systems on the host device. Over time, the system gradually reduces computational resource utilization, gradually increases computational efficiency, and gradually reduces reliance on human verifiers.

[0004] In another aspect of this disclosure, the first control logic also includes control logic for receiving input via a human-machine interface (HMI) of the host device.

[0005] In another aspect of this disclosure, the second control logic also includes control logic for enabling a semantic tool that extracts semantic information from the input. The semantic tool converts the semantic information extracted from the input into a vector in a vector space, such that the vector defines a mathematical, graphical, vectorized representation of the semantic information within the input.

[0006] In another aspect of this disclosure, the semantic tool also includes control logic for accessing a text vector database stored in memory, wherein the text vector database contains a plurality of predefined semantic text vectors that define mathematical, graphical, vectorized representations of predefined semantic inputs to which the host device is programmed to accept and respond; and control logic for comparing the predefined semantic text vectors in the text vector database with vectorized semantic information extracted from the input.

[0007] In another aspect of this disclosure, the semantic tool also includes control logic for: using ranking fusion to determine the retrieved context C. i1 Vector similarity score S i Calculate the semantic similarity score, and when the context C i The similarity score makes the vector similarity score S i Eliminate context C when it is less than the threshold T i The semantic tools also include control logic for: fusing “high-quality” contexts based on importance scores, where importance weights are based on different search engine CCs. i Context C i Counting, different retrieval RC i Context C i The sorting is used for calculation, and the context C i The importance score is given as IC i =f(CC) i ,RC i ); and according to

[0008] 〖Ret〗_a=[(C_a1,S_a1),(C_a2,S_a2),…(C an ,S an )]

[0009] 〖Ret〗_b=[(C_b1,S_b1),(C_b2,S_b2),…(C bn ,S bn )]

[0010] 〖Ret〗_c=[(C_c1,S_c1),(C_c2,S_c2),…(C cn ,S cn )]

[0011] Based on importance score IC i Sort the context.

[0012] Where the context {C ij}, j≥1 are sorted based on similarity scores, i.e.: S i1 >S i2 …>S in "Low-quality" contexts have vector similarity scores S less than a threshold T. i The "high quality" context indicates that the input is closely related to or directly involved in the critical host device functionality, thus affecting the importance score of the IC. i .

[0013] In another aspect of this disclosure, the CLR application also includes control logic for enabling syntactic tools to extract syntactic information from the input. The syntactic tools convert the extracted syntactic information from the input into a vector space representing the original text of the input, such that the vector defines a mathematical, graphical, vectorized representation of the syntactic information within the input.

[0014] In another aspect of this disclosure, the CLR application also includes control logic for accessing a raw text database stored in memory. The raw text database contains multiple predefined raw text vectors that define mathematical, graphical, vectorized representations of predefined raw text inputs to which the host device is programmed to accept and respond. The CLR application also includes control logic for comparing the predefined raw text vectors in the raw text database with vectorized syntactic information extracted from the input.

[0015] In another aspect of this disclosure, the CLR application also includes control logic for calculating a syntactic similarity score based on the Jaccard index and the Levenstein distance between the vectorized syntactic information extracted from the input and multiple predefined raw text vectors in the raw text database.

[0016] In another aspect of this disclosure, the CLR application also includes methods for...

[0017] C out =Z(w i *Score sem +w j *Score syn )

[0018] The control logic uses a normalized weighted sum of semantic similarity scores and syntactic similarity scores to calculate the output confidence score.

[0019] Where Z(...) is the normalization function, W i W j The weight is the semantic similarity score: Score sem = f(rankfusion score), where Score is the syntactic similarity score. syn =f(Jaccard Index,LevenshteinDistance,…).

[0020] In another aspect of this disclosure, the third control logic also includes control logic for assigning output confidence scores to outputs. The output commands one or more actuators of the host device to adjust the performance of the relevant host device system. The third control logic also enables a human verifier to prioritize and review the outputs based on the output confidence scores and a sorting context; and enables the human verifier to selectively update one or more of the text vector database and the original text database using new data obtained from the inputs, outputs, and output confidence scores. The host device is a vehicle and outputs commands to one or more actuators of the vehicle to change the vehicle's functionality based on input received from the user.

[0021] In another aspect of this disclosure, a method for calculating a confidence score for a Retrieval Augmentation (RAG) tool based on a Large Language Model (LLM) includes program control logic stored in the memory of a vehicle's controller, executed by a processor of the controller. The controller also includes input / output (I / O) ports for communicating with the vehicle's Human-Machine Interface (HMI). The program control logic includes an algorithm for calculating the confidence score of the LLM-based RAG tool (CLR application). The CLR application includes control logic for receiving LLM input from a vehicle user via the HMI, and enabling an integrated retrieval unit that determines the similarity between the input and predetermined data in one or more databases stored in memory. The CLR application also includes control logic for causing the integrated retrieval unit to generate outputs and output confidence scores, and for prioritizing and reviewing the outputs based on the output confidence scores. The output is a command to one or more systems of the vehicle. Over time, this method gradually reduces computational resource utilization, gradually increases computational efficiency, and gradually reduces reliance on human verifiers.

[0022] In another aspect of this disclosure, the method also includes enabling a semantic tool to extract semantic information from the input. The semantic tool converts the semantic information extracted from the input into a vector in a vector space, such that the vector defines a mathematical, graphical, vectorized representation of the semantic information within the input.

[0023] In another aspect of this disclosure, the method further includes accessing a text vector database stored in memory, the text vector database containing a plurality of predefined semantic text vectors that define mathematical, graphical, vectorized representations of predefined semantic inputs to which the vehicle is programmed to accept and respond, and comparing the predefined semantic text vectors in the text vector database with vectorized semantic information extracted from the input.

[0024] In another aspect of this disclosure, the method also includes using ranking fusion to determine the retrieved context C. i1 Vector similarity score S i Calculate the semantic similarity score, and when the context C i The similarity score makes the vector similarity score S i Eliminate context C when it is less than the threshold T i The method also includes fusing “high-quality” context based on an importance score definition, where the importance weights are based on different retrieval CCs. i Context C i Counting, different retrieval RC i Context C i The ranking is calculated based on the context C. i The importance score is given as IC i =f(CC) i ,RC i ); and according to

[0025] 〖Ret〗_a=[(C_a1,S_a1),(C_a2,S_a2),…(C an ,S an )]

[0026] 〖Ret〗_b=[(C_b1,S_b1),(C_b2,S_b2),…(C bn ,S bn )]

[0027] 〖Ret〗_c=[(C_c1,S_c1),(C_c2,S_c2),…(C cn ,S cn )]

[0028] Based on importance score IC i Sort the context.

[0029] Where the context {C ij}, j≥1 are sorted based on similarity scores, i.e.: S i1 >S i2 …>S in"Low-quality" contexts have vector similarity scores S less than a threshold T. i The "high quality" context indicates that the input is closely related to or directly involved in the critical host device functionality, thus affecting the importance score of the IC. i .

[0030] In another aspect of this disclosure, the method also includes enabling a syntactic tool to extract syntactic information from the input. The syntactic tool converts the syntactic information extracted from the input into a vector space representing the original text vector of the input, such that the vector defines a mathematical, graphical, vectorized representation of the syntactic information within the input.

[0031] In another aspect of this disclosure, the method further includes accessing a raw text database stored in memory, the raw text database containing a plurality of predefined raw text vectors defining mathematical, graphical, vectorized representations of predefined raw text inputs programmed to accept and respond to them, and comparing the predefined raw text vectors in the raw text database with vectorized syntactic information extracted from the input.

[0032] In another aspect of this disclosure, the method further includes calculating a syntactic similarity score based on the Jaccard index and the Levenstein distance between the vectorized syntactic information extracted from the input and multiple predefined raw text vectors in the raw text database.

[0033] In another aspect of this disclosure, the method also includes, according to

[0034] C out =Z(w i *Score sem +w j *Score syn )

[0035] The output confidence score is calculated using a normalized weighted sum of semantic similarity and syntactic similarity scores.

[0036] Where Z(...) is the normalization function, W i W j The weight is the semantic similarity score: Score sem = f(rankfusion score), where Score is the syntactic similarity score. syn =f(Jaccard Index,LevenshteinDistance,…).

[0037] In another aspect of this disclosure, the method further includes assigning output confidence scores to outputs, wherein the outputs command one or more actuators of the vehicle to adjust the performance of the relevant vehicle system, and the outputs are prioritized and reviewed by a human verifier based on the output confidence scores and a sorting context. The method also includes selectively updating one or more of a text vector database and an original text database by a human verifier using new data obtained from the inputs, outputs, and output confidence scores. The output commands to one or more actuators of the vehicle are used to change the function of the vehicle based on input received from the user.

[0038] In another aspect of this disclosure, the method further includes a method for calculating a confidence score for a retrieval enhancement (RAG) tool based on a large language model (LLM). The method includes program control logic stored in the memory of a vehicle controller, executed by a processor of the controller, which also has input / output (I / O) ports for communicating with a human-machine interface (HMI) of the vehicle. The program control logic includes an algorithm for calculating the confidence score of the LLM-based RAG tool (CLR application). The CLR application includes control logic for receiving input to the LLM from a vehicle user via the HMI and for enabling an integrated retrieval unit. The integrated retrieval unit determines the similarity between the input and predetermined data in one or more databases stored in memory. The CLR application also includes control logic for causing the integrated retrieval unit to generate outputs and output confidence scores, and for prioritizing and reviewing the outputs based on the output confidence scores. The outputs are commands to one or more systems of the vehicle, and the method progressively reduces computational resource utilization, progressively increases computational efficiency, and progressively reduces reliance on human verifiers over time. The CLR application also includes control logic for enabling semantic tools that extract semantic information from the input. Semantic tools convert semantic information extracted from the input into vectors in a vector space, such that these vectors define a mathematical, graphical, vectorized representation of the semantic information within the input. The CLR application also includes control logic for accessing a text vector database stored in memory. This text vector database contains multiple predefined semantic text vectors that define the mathematical, graphical, vectorized representation of predefined semantic inputs to which the vehicle is programmed to accept and respond. The CLR application also includes control logic for comparing the predefined semantic text vectors in the text vector database with the vectorized semantic information extracted from the input, and for using ranking fusion based on the retrieved context. i1 Vector similarity score S i Calculate the semantic similarity score, and when the context C i The similarity score makes the vector similarity score S i Eliminate context C when it is less than the threshold T iThe CLR application also includes control logic for fusing “high-quality” contexts based on an importance score definition, where importance weights are based on different retrieval CCs. i Context C i Counting, different retrieval RC i Context C i The sorting is used for calculation, and the context C i The importance score is given as: IC i =f(CC) i ,RC i ); and according to

[0039] 〖Ret〗_a=[(C_a1,S_a1),(C_a2,S_a2),…(C an ,S an )]

[0040] 〖Ret〗_b=[(C_b1,S_b1),(C_b2,S_b2),…(C bn ,S bn )]

[0041] 〖Ret〗_c=[(C_c1,S_c1),(C_c2,S_c2),…(C cn ,S cn )]

[0042] Based on importance score IC i Sort the context.

[0043] Where the context {C ij}, j≥1 are sorted based on similarity scores, i.e.: S i1 >S i2 …>S inThe CLR application also includes control logic for enabling syntactic tools that extract syntactic information from the input. These syntactic tools convert the extracted syntactic information from the input into raw text vectors in a vector space, such that these vectors define a mathematical, graphical, vectorized representation of the syntactic information within the input. The CLR application also includes control logic for accessing a raw text database stored in memory, wherein the raw text database contains multiple predefined raw text vectors that define the mathematical, graphical, vectorized representation of the predefined raw text input to which the vehicle is programmed to accept and respond. The CLR application also includes control logic for comparing the predefined raw text vectors in the raw text database with the vectorized syntactic information extracted from the input, and for calculating a syntactic similarity score based on the Jaccard index and the Levenstein distance between the vectorized syntactic information extracted from the input and the multiple predefined raw text vectors in the raw text database. The CLR application also includes control logic for... out =Z(w i *Score sem +w j *Score syn The output confidence score is calculated using a normalized weighted sum of semantic similarity and syntactic similarity scores, where Z(...) is the normalization function and W... i W j The weight is the semantic similarity score: Score sem = f(rank fusion score), where Score is the syntactic similarity score. syn = f(Jaccard Index, Levenshtein Distance, ...). The CLR application also includes control logic for assigning output confidence scores to outputs, where the outputs command one or more actuators of the vehicle to adjust the performance of the relevant vehicle system, and for prioritizing and reviewing the outputs by a human verifier based on the output confidence scores and sorting context. The CLR application also includes control logic for selectively updating one or more of the text vector database and the original text database by a human verifier using new data obtained from the inputs, outputs, and output confidence scores. Output commands are given to one or more actuators of the vehicle to change the vehicle's functionality based on input received from the user.

[0044] Further applicability will become apparent from the description provided herein. It should be understood that the specification and specific examples are for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0045] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure in any way.

[0046] Figure 1 This is a schematic diagram depicting a system for calculating confidence scores of a retrieval enhancement (RAG) tool based on a large language model (LLM) according to an exemplary embodiment; and

[0047] Figure 2 It is a description of an exemplary embodiment. Figure 1 A flowchart of the logical flow of an application system for calculating confidence scores for an LLM-based RAG tool. Detailed Implementation

[0048] The following description is merely exemplary in nature and is not intended to limit this disclosure, its application, or its uses.

[0049] refer to Figure 1 A system 10 for calculating confidence scores for a Retrieval Augmentation (RAG) tool based on a Large Language Model (LLM) is illustrated schematically. System 10 generally runs in or on host device 12. Host device 12 can take any of a variety of forms, including vehicle 14. However, it should be understood that system 10 of this disclosure is not required to be tied to such a vehicle 14. Rather, vehicle 14 is merely an exemplary, non-limiting embodiment in which system 10 of this disclosure is described herein in conjunction with vehicle 14. System 10 can run in any hardware and software configuration (where the generative AI-driven tool is used to receive input from user 16, such as user command 18) and generate output 20 that alters the functionality of the hardware and / or software configuration or system in which the generative AI-driven tool is being used. Additionally, while vehicle 14 shown is a car, it should be understood that vehicle 14 can be of any type without departing from the scope or intent of this disclosure. In several non-limiting examples, vehicle 14 can be: a car, truck, sport utility vehicle (SUV), semi-trailer truck, tractor trailer, tractor, combine harvester or other such agricultural equipment; powered and unpowered aircraft, such as airplanes, helicopters, gliders or rotorcraft; powered and unpowered vessels, such as ships, sailboats, motorboats, yachts, jet skis, etc. In other non-limiting embodiments, it should be understood that, without departing from the scope or intent of this disclosure, system 10 described herein can be adapted to work with host device 12, such as manned or unmanned spacecraft, for example: satellites, rockets, space stations, and other orbital and extraorbital satellite communications enabled devices. In still other non-limiting examples, host device 12 can include a mobile computing platform, such as a laptop computer, mobile phone, tablet, or any other such host device 12 through which a user can interact with generative AI-driven tools.

[0050] System 10 also includes a controller 22, which is a non-general-purpose electronic control device having a pre-programmed digital computer or processor 24, a non-transitory computer-readable medium or memory 26 for storing data such as control logic, software applications, instructions, computer code, data, lookup tables, etc., and a transceiver or input / output (I / O) port 28. The computer-readable medium or memory 26 includes any type of media accessible by a computer, such as read-only memory (ROM), random access memory (RAM), hard disk drive, optical disc (CD), digital video disc (DVD), or any other type of memory 26. "Non-transitory" computer-readable memory 26 does not include wired, wireless, optical, or other communication links that transmit transient electrical or other signals. Non-transitory computer-readable memory 26 includes media in which data can be permanently stored and media in which data can be stored and subsequently rewritten, such as rewritable optical discs or erasable memory devices. Computer code includes any type of program code, including source code, object code, and executable code. The processor 24 is configured to execute code or instructions.

[0051] When system 10 is running on vehicle 14, controller 22 may include a dedicated Wi-Fi controller or engine control module, transmission control module, body control module, infotainment control module, etc. Transceiver or I / O port 28 is configured to wirelessly communicate with the back-end management system 30 using cellular protocols, including Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Enhanced Data Rate Evolution of GSM (EDGE), Universal Mobile Telecommunications Service (UMTS), High-Speed ​​Packet Access (HSPA), Code Division Multiple Access (CDMA), Evolved Data Optimized (EV-DO / EVDO / 1xEV-DO), Short Message Service (SMS), Wi-MAX, Manufacturing Message Specification (MMS), 2G, 3G, 4G, 5G, wireless, and cellular standards as defined by IEEE 802.1X, IEEE 802 LAN / MAN, and the IEEE Mobile Communications Networks Standards Committee (MobiNet-SC) standards, etc. The back-end management system 30 may include one or more controllers 22 and / or one or more human experts or verifiers 32, which are shown and described in more detail in the following figures.

[0052] Controller 22 also includes one or more applications 34. Application 34 is a software program configured to perform a specific function or set of functions. Application 34 may include one or more computer programs, software components, instruction sets, procedures, functions, objects, classes, instances, associated data, or adaptations thereof implemented as part of suitable computer-readable program code. Application 34 may be stored in memory 26 or in additional or separate memory. Examples of applications 34 include audio or video streaming services, games, browsers, social media, etc., and algorithms for calculating confidence scores for a large language model (LLM) 36-based RAG tool 38 (hereinafter referred to as CLR application 40) for generative AI-driven tools or LLM 36 tools. Generative AI-driven tools or LLM 36 define applications 34 stored locally on vehicle 14 controller 22 and / or in a remote back-end management system 30 or cloud-based computing device. CLR application 40 includes multiple subroutines or control logic sections.

[0053] In examples where System 10 is in or running on Vehicle 14, a vehicle operator or vehicle development engineering user 16 can use System 10 and CLR application 40 to validate tools for developing and testing systems that dynamically adjust the operation of Vehicle 14 and its features. In some examples, Vehicle 14 may be equipped with a navigation system 42, one or more drive motors 44 that provide and change the amount of torque transmitted to the wheels 46 of Vehicle 14 to move, stop, etc., and a steering system 48 that can adjust the direction and heading of Vehicle 14. In other examples, Vehicle 14 is equipped with a braking system 50 that, when activated by controller 22, slows the movement of Vehicle 14. Without departing from the scope or intent of this disclosure, Vehicle 14 may be equipped with various other vehicle body motion control systems that can be used to alter or otherwise control the dynamic performance of Vehicle 14, including but not limited to aerodynamic control surfaces and actuators, active and / or semi-active suspension systems and actuators, etc.

[0054] LLM 36 is trained on large datasets in one or more online and offline modes. The training data provides an efficient way for LLM 36 to generate outputs for tasks such as answering questions, performing mathematical calculations, performing language translation, and / or summarizing. Retrieval-enhanced generation (RAG) is an architectural approach used to improve the quality of responses generated by LLM 36 by supplementing its internal information representation by building the LLM 36 model on top of external knowledge sources. Implementing RAG in an LLM 36-based question-answering system offers at least two main benefits: ensuring that LLM 36 has access to up-to-date, reliable, and relevant data within a defined scope, and that user 16 has access to the sources of LLM 36, ensuring that LLM 36's responses to user 16's input are likely appropriate and credible. Furthermore, by building LLM 36 on a set of externally verifiable facts, RAG minimizes the opportunity for LLM 36 to incorporate information into its parameters, thus reducing the chance of LLM 36 leaking sensitive data or providing incorrect or misleading responses to user 16's input. More specifically, the LLM 36-based RAG tool 38 of this disclosure extends the capabilities and improves the efficiency of LLM 36 applications by utilizing custom data. Custom data can relate to any of a wide range of topics, but should generally be understood to specifically relate to a particular hardware or software application of host device 12. In a non-limiting example, custom or domain-specific knowledge data can relate to the vehicle 14's in-vehicle systems, including but not limited to navigation systems, powertrain control systems, suspension control systems, heating, ventilation, and air conditioning (HVAC) systems, and so on. LLM 36 utilizes custom data from the RAG tool 38 related to commands, questions, or tasks generated by user 16 and provides the custom data as context for LLM 36 to generate responses. In many respects, RAG is an effective method for improving LLM 36 performance and has successfully supported chatbots and question-and-answer (Q&A) systems that require access to domain-specific information. Domain-specific information may include any of a variety of data relating to the functionality of a particular host device 12, vehicle 14, and back-end management system 30, and additionally to the onboard hardware, software, and applications of each of the host device 12, vehicle 14, and back-end management system 30. Domain-specific information may also relate to specific proprietary data held by the original equipment manufacturer (OEM), which is related to specific hardware and software products manufactured by the OEM. Domain-specific information may define an embedded model stored in the memory 26 of the controller 22 on the host device 12.

[0055] Now for reference Figure 2 And continue to refer to Figure 1A more detailed exemplary schematic logic flowchart of the CLR application 40 is shown. The CLR application 40 receives input 100 in offline mode. Input 100 may be received via a human-machine interface (HMI) of the host device 12, including but not limited to audio or visual or audiovisual receivers, such as microphones or other audio sensors 52, cameras or other visual sensors 54, tactile sensor interfaces, including but not limited to buttons and touchscreens 56, etc. In several examples, input 100 is a user command 18 of user 16, which is subsequently processed in a series of subroutines by an LLM-based RAG tool 38 before generating output 20. Input 100 may take any of a variety of forms without departing from the scope or intent of this disclosure. In some non-limiting examples, input 100 may include a spoken or written user command 18 in any language, such as: a command to start the navigation system of vehicle 14, a command to search for points of interest, a command to change the cabin temperature of vehicle 14, a command to pause or wait or delay, a request to obtain a solution to a mathematical statement, or any other such command 18. Input 100 is processed by multiple subroutines within the LLM 36-based RAG tool 38.

[0056] Output 20 is the response of host device 12 to input 100 from user 16. In several aspects, output 20 directly or indirectly alters the function of one or more systems of host device 12, including but not limited to: altering one or more functions of the navigation system, powertrain control system, suspension control system, HVAC system, etc. Therefore, output 20 may include adjusting the performance of the relevant system by directly or indirectly changing the position of the actuator of the host device within the relevant host device 12 system in response to user command 18 or input 100, changing the destination or route planning function of the navigation system, or changing the mode or operation of the powertrain, suspension control, or HVAC system.

[0057] RAG tool 38 includes an ensemble retriever 102. The ensemble retriever 102 is a complex retrieval algorithm that improves the relevance of the retrieved contextual information by aggregating results from multiple different and parallel retrievers. By using multiple parallel retrievers, the advantages of each can be leveraged to obtain results more accurately related to the user command 18 or input 100 than by using a single data retrieval algorithm alone. Input 100 is received within the ensemble retriever 102 by at least two parallel subroutines (i.e., semantic tool 200 and syntactic tool 300) of the LLM 36-based RAG tool 38. Semantic tool 200 receives input 100 in semantic retriever 202. The semantic retriever 202 subroutine first extracts semantic information from input 100. Semantic retriever 202 then calculates a semantic similarity score 204 of the semantic information extracted from input 100. To calculate the semantic similarity score 204, the semantic retrieval unit 202 converts the semantic information extracted from input 100 into a semantic text vector in a vector space, such that the vector is a mathematical graphical representation of the semantic information extracted from input 100. In a non-limiting example, input 100 may include a verbal command from user 16. It should be understood that the verbal command can be any of a variety of different commands, including but not limited to: “Raise the cabin temperature,” where user 16 intends for input 100 to command vehicle 14 to use the heating, ventilation, and air conditioning (HVAC) system to change the temperature of the passenger compartment of vehicle 14. Each word in the command input 100 by user 16, namely “raise,” “cabin,” and “temperature,” is parsed and plotted in the vector space and subsequently compared with a predetermined text vector in the text vector database 206 stored in memory 26.

[0058] Because the language input command 100 of a single user 16 may differ in the actual wording or terminology chosen, the vector representing input 100 may not be accurately represented by the text vectors stored in the text vector database 206. The text vector database 206 contains multiple semantic text vectors that define the mathematical, graphical, and vectorized representation of the predefined semantic input 100 programmed for the host device 12 to accept, respond to, and understand. The multiple text vectors in the text vector database 206 can be manually and / or automatically selected during the pre-programming of the system 10, and the multiple text vectors can be updated with new information when specific events occur, or can be updated or modified manually or automatically, continuously, periodically, etc. In some non-limiting examples, the text vector database 206 is updated by a human expert or verifier 32.

[0059] Therefore, semantic retrieval 202 calculates a semantic similarity score 204 based on the text vector representing input 100 and predefined text vector information accessed within the text vector database 206. In several aspects, the semantic similarity score 204 is a numerical, graphical, vectorized representation of the similarity level between the semantic structure of input 100 and the semantic structure of the text vector that most closely corresponds to the semantic text vector of input 100. After calculating the semantic similarity score 204, semantic tool 200 performs a ranking fusion calculation 208, which filters the context of input 100 based on a threshold T to avoid "low-quality" contexts or low-quality matches between data in the text vector database 206 and the text vectors representing the data of input 100. More specifically, CLR application 40 utilizes an inverse ranking fusion algorithm to reorder and merge the results from each of semantic tool 200 and syntactic tool 300.

[0060] In other words, integrated search engine (Ret i )102 Obtain the retrieved context C i1 And vector similarity score S i1 And if and only if C i The similarity score makes the vector similarity score S i Context C is eliminated only when the value is less than the threshold T. i Conversely, the "high-quality" context is defined based on an importance score, where the importance weights are based on different search engine CCs. i Context C i Counting, different retrieval RC i Context C i The sorting is used for calculation, and the context C i The importance score is given as IC i =f(CC) i ,RC i Then based on the importance score IC i The context is sorted. Therefore, in some non-restricted examples, according to...

[0061] Ret a =[(C a1 ,S a1 ),(C a2 ,S a2 ),…(C an ,S an )]

[0062] Ret b =[(C b1 ,S b1 ),(C b2 ,S b2 ),…(C bn ,Sbn )]

[0063] Ret c =[(C c1 ,S c1 ),(C c2 ,S c2 ),…(C cn ,S cn Generate a series of ranked semantic similarity scores.

[0064] Where the context {C ij {,j≥1} are sorted according to similarity scores, i.e.: S i1 >S i2 …>S in Importance score (IC) i Define context C i The relative importance of the type of input 100 received. Therefore, in a non-limiting example, for context C that does not involve critical host device 12 functions (e.g., the powertrain, suspension, or safety-critical systems of vehicle 14). i (e.g., HVAC function), importance score IC i It has a lower value. Conversely, when context C i When input 100 is more closely related to or directly involved in the function of critical host device 12, the importance score IC indicates that... i Higher than the lower value.

[0065] Conversely, the syntactic tool 300, operating in parallel with the semantic tool 200, receives input 100 within the syntactic retrieval unit 302. Like the semantic retrieval unit 202, the syntactic retrieval unit 302 subroutine first extracts syntactic information from input 100. Then, the syntactic retrieval unit 302 subroutine calculates a syntactic similarity score 304 based on the raw text of input 100 and a raw text database 306 stored in memory 26. The raw text database 306 contains multiple raw text vectors that define the mathematical, graphical, vectorized representation of the predefined syntactic input 100 programmed by the host device 12 to accept and understand. The multiple raw text vectors in the raw text database 306 can be manually and / or automatically selected during pre-programming of the system 10, and can be updated with new information when specific events occur, or can be updated or modified manually or automatically, continuously, periodically, etc. In some non-limiting examples, the raw text database 306 is updated by a human expert or verifier 32.

[0066] To compute the syntactic similarity score 304, the syntactic retrieval subroutine 302 converts the syntactic information extracted from input 100 into a raw text vector in a vector space, such that this vector is a mathematical graphical representation of the syntactic information extracted from input 100. The syntactic similarity score 304 is a numerical representation of the level of similarity between the syntactic structure of input 100 and the data in the raw text database 36. More specifically, the syntactic similarity score 304 is computed based on the Jaccard index and the Levinstein distance. The Jaccard index... This is used to measure the similarity and diversity between input 100 information and predefined original text information stored in the original text database 306. Similarly, the Levenstein distance is a string metric used to measure the difference between two sequences, or in this example, the difference between the original text of input 100 and the original text stored in the original text database 306. In one example, the Levenstein distance between two words is the minimum number of single-character edits (i.e., insertions, deletions, or replacements) required to change one word to another. The Levenstein distance can be mathematically represented as follows:

[0067]

[0068] It should be understood that the use of the Jaccard index and the Levinstein distance is intended only as an exemplary, non-limiting example of an algorithm or function type that can be used to compute the syntactic similarity score 304 according to the purposes of this disclosure. The semantic and syntactic similarity scores 204, 304 may cover a range of values ​​that vary depending on the application, but in one non-limiting example, the semantic and / or syntactic similarity scores 204, 304 vary between values ​​of zero (0) and one (1), such that the semantic and / or syntactic similarity scores 204, 304 equal 0 when there is no similarity whatsoever, and the semantic and / or syntactic similarity scores equal 1 when there is complete consistency between the semantic and / or syntactic similarity scores 204, 304 and the information in the text vector database 206 or the original text database 306.

[0069] Subsequently, the combined sorting and fusion calculations of the outputs of semantic similarity score 208 and syntactic similarity score 304 are performed, and a confidence score of 400 is calculated. The confidence score 400 is a normalized weighted sum of the semantic similarity score 204 and the syntactic similarity score 304. The confidence score can be expressed as:

[0070] C out =Z(w i *Score sem +w j *Score syn )

[0071] Where Z(...) is the normalization function, Wi W j This is the weight. The semantic similarity score of 204 is: Score sem = f(rankfusion score), the syntactic similarity score of 304 is: Score syn =f(Jaccard Index,LevenshteinDistance,…).

[0072] In several aspects, weight W i W j It can vary significantly depending on the application. Weight W i W j This can be selected by application developers, original equipment manufacturers, suppliers, etc. It should also be understood that weight W... i W j It can be dynamic, variable, or constant, depending on the type and structure of the query that system 10 receives as input 100.

[0073] As described herein, the LLM 36-based RAG tool 38 generates outputs 20 that require verification and validation by a human expert or verifier 32 at least during the pre-production process. However, the system 10 of this disclosure offers several advantages, including, but not limited to, automatically generating confidence scores for the LLM 36 outputs 20 via the CLR application 40 of this disclosure, allowing the human expert or verifier 32 to effectively prioritize verification, thereby significantly reducing the human effort and time required to verify the LLM 36 outputs 20, while improving the accuracy of the LLM 36, reducing computational work and resource consumption, and reducing the likelihood of human-introduced typographical, syntactic, or other such errors from a first number to a much smaller number. In one example, the expert or verifier 32 may choose to first verify the LLM 36 outputs 20 with low confidence values ​​and then take action to verify the LLM 36 outputs 20 with confidence levels higher than the low confidence values. Therefore, confidence scores can be automatically calculated by utilizing the retrieval's vector-based similarity score and the input 100 similarity score (e.g., Jaccard distance) of a given input 100. It should also be understood that, in pre-production or production settings, as the CLR 40 is continuously used, evaluated, and updated over time, the amount of interaction and input from human experts or verifiers 26 decreases. That is, even in production applications where non-engineering end-users or customers interact with the LLM 36, the CLR application 40 can run to accurately, consistently, reliably, and robustly interpret end-user or customer input to the system 10 and generate responses accordingly, where computational resource utilization gradually decreases, computational efficiency gradually increases, and reliance on verification by human verifiers 32 gradually decreases.

[0074] The description in this disclosure is merely exemplary in nature, and any changes that do not depart from the spirit of this disclosure are intended to fall within its scope. Such changes should not be considered as departing from the scheme and scope of this disclosure.

Claims

1. A system for computing a confidence score for a Large Language Model (LLM) based Retrieval Augmentation (RAG) tool, the system comprising: a host device having a controller with a processor, a memory, and an input / output (I / O) port in communication with a human-machine interface (HMI) and one or more databases, the processor executing program control logic stored in the memory, the program control logic including an algorithm for computing a confidence score for a LLM based GRA tool (CLR application), the CLR application including: first control logic to receive an input to a LLM from a host device user; second control logic to enable an integrated retriever, wherein the integrated retriever determines a similarity between the input and predetermined data stored in one or more databases in the memory; third control logic to cause the integrated retriever to generate an output and an output confidence score; and fourth control logic to cause a human verifier to prioritize and review the output based on the output confidence score, wherein the output is a command to one or more systems of the host device, and wherein, over time, the system gradually reduces computational resource utilization, gradually increases computational efficiency, and gradually reduces reliance on the human verifier.

2. The system of claim 1, wherein, the first control logic further including: control logic to receive the input via a human-machine interface (HMI) of the host device.

3. The system of claim 1, wherein, the second control logic further including: control logic to enable a semantic tool to extract semantic information from the input, and wherein the semantic tool converts the extracted semantic information from the input into a vector in a vector space, such that the vector defines a mathematical, graphical, vectorized representation of the semantic information within the input.

4. The system of claim 3, wherein, the semantic tool further including: control logic to access a text vector database stored in memory, wherein the text vector database contains a plurality of predefined semantic text vectors that define a mathematical, graphical, vectorized representation of predefined semantic inputs that the host device is programmed to accept and respond to; and control logic to compare the predefined semantic text vectors in the text vector database to the vectorized semantic information extracted from the input.

5. The system of claim 4, wherein, the semantic tool further including: control logic to: Use ranking fusion to determine the retrieved context C i1 Vector similarity score S i Calculate the semantic similarity score, and when the context C i The similarity score makes the vector similarity score S i Eliminate context C when it is less than the threshold T i ; The system integrates "high-quality" contexts defined by importance scores, where importance weights are based on different search engine CCs. i Context C i Counting, different retrieval RC i Context C i The sorting is used for calculation, and the context C i The importance score is given as IC i =f(CC) i ,RC i );as well as determine a similarity between the input and the predefined semantic text vectors in the text vector database based on 〖Ret〗_a = [(C_a1, S_a1), (C_a2, S_a2),... (C an ,S an )] 〖Ret〗_b = [(C_b1,S_b1),(C_b2,S_b2),…(C bn ,S bn )] Ret_c = [(C_c1, S_c1), (C_c2, S_c2),... (C cn ,S cn )] Based on the importance score IC i Ranking contexts, where the context {C ij},j≥1 are ordered based on the similarity score, i.e.: i1 S i2 S in where "low quality" contexts have a vector similarity score S i < less than a threshold T, and "high quality" contexts have an importance score IC i indicative that the input is closely related or directly involved with a critical host device function.

6. The system of claim 5, further comprising: control logic to enable a syntax tool to extract syntax information from the input; and wherein the syntax tool converts the extracted syntax information from the input into an input raw text vector in a vector space, such that the vector defines a mathematical, graphical, vectorized representation of the syntax information within the input.

7. The system of claim 6, further comprising: control logic to access a raw text database stored in memory, wherein the raw text database contains a plurality of predefined raw text vectors that define a mathematical, graphical, vectorized representation of predefined raw text inputs that the host device is programmed to accept and respond to; and control logic for comparing a pre-defined raw text vector in the raw text database to vectorized syntax information extracted from the input.

8. The system of claim 7, further comprising: control logic for computing a syntax similarity score from a Jaccard index and a Levenshtein distance between vectorized syntax information extracted from the input and a plurality of pre-defined raw text vectors in the raw text database.

9. The system of claim 8, further comprising: for calculating a semantic similarity score Ssemantics between a first sentence and a second sentence based on C out = Z(w i * Scoresemantics sem + w j * Scoresyntax syn control logic for calculating the output confidence score using a normalized weighted sum of the semantic similarity score and the syntactic similarity score, where Z(...) is a normalization function, W i ,W j is a weight, the semantic similarity score is: Score sem = f(rankfusionscore), the syntactic similarity score is: Score syn = f(JaccardIndes, LevenshteinDistance,...).

10. The system of claim 9, wherein, the third control logic further comprising: control logic for assigning the output confidence score to the output, wherein the output commands one or more actuators of the host device to adjust a performance of an associated host device system; control logic for causing the human validator to prioritize and review the output according to the output confidence score and a ranking context; and causing the human validator to selectively update one or more of the text vector database and the raw text database with new data obtained from the input, the output, and the output confidence score, and wherein the host device comprises a vehicle and outputs commands to one or more actuators of the vehicle to change a function of the vehicle according to the input received from the user.

Images