LARGE LANGUAGE MODEL AND VECTOR SIMILARITY-BASED INTENTION CLASSIFICATION AND CONFIDENCE SCORE VALIDATION SYSTEM

Abstract

This invention relates to a system (1) that enables the automatic interpretation of user queries and their redirection to appropriate business processes in sectors such as telecommunications, banking, e-commerce, healthcare, and public services, particularly in the fields of natural language processing and artificial intelligence, especially in technologies such as speech-based chatbots, virtual assistants, customer service automation, and interactive voice response (IVR).

Description

1 TARIFF LARGE LANGUAGE MODEL AND VECTOR SIMILARITY-BASED INTENTION CLASSIFICATION AND CONFIDENCE SCORE VERIFICATION SYSTEM Technical Area 5 This breakthrough holds significant ground in the fields of natural language processing and artificial intelligence, particularly speech-based applications. chatbots, virtual assistants, customer service automation, and interactive voice response (IVR) automatic interpretation of user queries in technologies and 10 telecommunications, banking, e- It enables its application in sectors such as trade, health, and public services. It is related to the system. Previous Technique 15 In the known state of the art, existing intent classification systems generally a single method, either just vector similarity or just a language model. It works using vectors, and this approach has significant limitations: It only works with vectors. Semantic-based systems can capture semantic similarities but contextual nuances. And it cannot understand subtle differences in meaning. Only language model-based systems are powerful. It has the ability to understand, but in large sets of intentions, it has all the options. Evaluating it is computationally inefficient. Reliability in current systems. These scores are often uncalibrated and do not reflect the true accuracy rate. This also makes automation decisions risky. Out-of-context or general chat (small 25 Queries containing (talk) can be attributed to incorrect intents. Traditional intent classification methods; low accuracy, unreliable confidence scores, and out-of-context It involves problems such as the misclassification of queries. Automatic classification of user intents, customer service 30 Automation is critical in chatbots and task-oriented dialogue systems. 2 This is a technical requirement. Given the current state of the art, there are various ways to solve this problem. There are approaches: In current technology, user queries can be pre-prepared. Vector similarity search for assignment to defined intent categories (vector) Similarity search is widely used. In this approach: Intent Descriptions and user queries are converted into numerical vectors (embedding), 5 Using metrics such as cosine similarity or Euclidean distance, the most similar Intent is found. Fast search with vector databases such as FAISS and Pinecone. This is done. However, in this approach, even if semantic similarity is achieved, contextual similarity is also important. Nuances and subtle differences in meaning are overlooked, and similar words are used interchangeably. It cannot distinguish between situations where it expresses different intentions in different contexts, it is ambiguous. 10 It is insufficient for correctly classifying (ambiguous) queries and similarity The scores do not reflect the true confidence level, only the proximity in the vector space. This shows. In recent years, major language models such as GPT-4, Claude, and Mistral have been used less frequently. For intention classification through learning of few-shot or zero-shot shots. It has started to be used. Academic studies (for example, "Intent Detection in the 15 "Age of LLMs", arXiv 2410.01627) this approach is advantageous in terms of accuracy. This has been shown. However, when there are hundreds or thousands of categories of intention, Including all intents in the command exceeds the token limit. All intents for each query. Evaluating the set is computationally inefficient and leads to high latency. (latency) causes. Confidence scores generated by large language models are calibrated to 20. has not been verified and does not reflect the true accuracy rate (Gong et al., "Confidence Calibration for Intent Detection via Hyperspherical Space", AAAI 2022). Model can invent a non-existent category or prompt injection They may be vulnerable to attacks. Systems that use the Large Language Model (LLM) alone, As the number of intentions increases (100+, 1000+), the request size explodes and hits the token limits. Systems that use vector search alone are good at capturing semantic similarity, but... It cannot understand complex contexts. Vector similarity scores (cosine between 0-1) (Similarity) does not reflect the true accuracy rate. LLM softmax outputs are typically 30 This would be overconfident. Current systems, which predictions... 3 It cannot provide a reliable metric to determine if it can be trusted. "Hello", General conversational expressions like "thank you" can be mistakenly used in a business context. It can be assigned to that purpose. Systems without a relevance pre-filter force all queries to be assigned to a specific purpose. It assigns intent. Evaluating the entire set of intents for each query (with an LLM) provides a high level of accuracy. It is costly and slow. The real 5 in the form of chatbots and interactive voice responses. In time-sensitive applications, delay is critical. Therefore, considering the studies and shortcomings in the current technique... When this is the case, vector similarity search can be performed using a large intention set to find the relevant information. the candidates are quickly filtered, then the language model is used to select these candidates 10 the language model's ability to make a final classification through contextual understanding. log-probability values ​​using temperature scaling method by calibrating and generating real confidence scores, the user query's task Determining whether it is a focused request or a general conversation, vector Indexing allows working with hundreds or thousands of intent categories and 15 the easy addition of new intents, the structured functions of the language model By using it with a call, the consistency and processability of the output format are guaranteed. It appears that a system is needed to enable this. United States Code 20, numbered US12367223B1, which is included in the known state of the art. in his patent, by one or more computing devices and a large language input of a calibration data sample to be classified by the model and previously unseen but with known and tagged correct answers The text describes a system that classifies data according to calibration data. The invention in question involves combining pre-established probability values ​​with the actual 25. Labels are compared to calculate a suitability score. These scores are then organized into a series. stored and analyzed within a certain margin of error, to make future predictions. A threshold value is set for each classification value to be valid for the sample. while representing the probability of output, the results from the large language model are maximized. By filtering from the highest probability down to this threshold value, only the reliable result 30 It presents its suite of data in the user interface and constantly updates itself with current data. 4 a recalibrating verification system, an AI that monitors itself It is presented. Brief Description of the Invention The aim of this invention is to advance in the fields of natural language processing and artificial intelligence, particularly speech. chatbots, virtual assistants, customer service automation, and based on In interactive voice response technologies, user queries are automated. It should be interpreted as such and directed towards appropriate business processes. telecommunications, banking, e-commerce, healthcare, public services – these are the 10 sectors. a system developed to enable its implementation in various sectors to accomplish. Another aim of this invention is to analyze queries that users express in natural language and Automatic classification of requests and their placement in the correct business process 15 The goal is to implement a system developed to facilitate its guidance. Detailed Description of the Invention The “Large Language Model and Vector 20” project was carried out to achieve the purpose of this invention. "Similarity-Based Intent Classification and Confidence Score Validation System" is attached. This is shown in the figure; Figure 1 shows a schematic view of the system that is the subject of the invention. The parts shown in the figure are individually numbered, and these numbers correspond to... The corresponding answers are given below. 1. System 2. Client Application 30 3. Authentication Module 4. Intent Database and Vector Indexing Module 5. Embedded Service Module 6. Vector Similarity Search Module 7. Relevance Filtering Module 8. Major Language Model Integration Module 5 9. Intent Classification Module 10. Confidence Score Calibration Module 11. Request Template Management Module In the field of natural language processing and artificial intelligence, 10 things users express in natural language. to enable automatic classification of queries and requests The developed invention subject system (1); - Making REST API calls over the http protocol, on a mobile device in the form of an application, web interface and / or other microservice at least one client application configured to run (2), 15 - Verify the carrier token in the security header of each incoming request, authenticate tokens with a predefined public key, At least one identity configured to prevent unauthorized access verification module (3), - 20 for each of the predefined intention categories within it to include explanatory texts, one or more for each category of intent Including too many example sentences and explanations, these statements of intent, Converting to high-dimensional vectors via the embedding model, The generated vectors are used in a FAISS format for fast similarity searching. Indexing using a vector indexing library, this index, 25 to load into memory and keep it in memory throughout the entire query process, thus ensuring that disk access latency is eliminated At least one intent database and vector indexing are configured. module (4), - processing the query text from the client application (2), an external 30 by calling the embedded application programming interface, the query text 6 at least one embedding configured to convert to a digital vector. service module (5), - After the query vector is created, the cosine of the FAISS index is calculated. using a metric such as similarity or Euclidean distance Finding the top-K intent vector closest to the query vector, where K is 5. The maximum value is configured to be set between 10-50. a small vector similarity search module (6), - working in the query processing process, coming from the client application (2) Is the user query a task-oriented request or a general chat? At least one relevance filter is configured to determine if it is 10 module (7), - an abstraction aimed at interacting with large language models using the layer, various software libraries with this layer through a single platform, it provides access to various major language model providers. to provide from the interface, user 15 coming from the client application (2) to turn the inquiry and candidate intentions into a request and the big language sending it to the model, the large language model, based on this information, will be able to do the best. at least one structured to enable the identification of possible intent Big Language Model Integration Module (8), - Embedding the user query from the client application (2) service 20 to pass to module (5) and obtain the query vector, this vector using the vector similarity search module (6), top-K most similar to collect intentions, to group these intentions according to their similarity scores, and to record the highest score for each unique intention at least one intent classification module configured (9), 25 - calibrating the raw logarithmic probability values ​​of the large language model at least one structured to convert into verified confidence scores confidence score calibration module (10), - requests sent to the large language model are in external YAML files to ensure it is hidden, to enable system engineering and 30 different Enabling testing of client versions, YAML file 7 to ensure the inclusion of variable placeholders and the operation at least one structured to be filled with real values ​​in a timely manner The request includes the template management module (11). The client application (2) in the system (1) that is the subject of the invention, Hypertext Transfer 5 The protocol is represented via the HyperText Transfer Protocol (HTTP) protocol. State Transfer Application Programming Interface (Representational State) Making Transfer Application Programming Interface (REST API) calls, It operates as a mobile application, web interface, and / or another microservice. It is structured accordingly. 10 The authentication module (3) in the system (1) that is the subject of the invention, each incoming the bearer identifier in the request's authorization header token verification, tokens with a predefined public key It is configured to verify and prevent unauthorized access. 15 The intention database and vector indexing module included in the system (1) are the subject of the invention. (4), descriptive for each of the predefined categories of intent within it. to include texts, one or more example sentences for each intent category and to include explanations, these statements of intent, an embedding model 20 converting them into high-dimensional vectors, and then rapidly processing the generated vectors. FAISS (Facebook AI Similarity Search) is a vector for similarity searching. indexing using an indexing library, loading this index into memory, and keeping it in memory throughout the entire query process, thus reducing disk access latency. It is structured to ensure its elimination. 25 The embedded service module (5) in the system (1) that is the subject of the invention, client To process the query text from the application (2), use an external embedded application. by calling the programming interface (Application Programming Interface - API) Converting the query text to a numeric vector, and then using the generated query vector, intent 30 to be compared with vectors in the database and vector indexing module 8 Passing to the vector similarity search module (6) with different embedding models by working to select the model that provides the best performance and different embedded systems. It is structured to facilitate the transition between providers. The vector similarity search module (6) in the system (1) which is the subject of the invention, query 5 After the vector is generated, cosine similarity or Euclidean similarity is applied to the FAISS index. Using a metric in the form of distance, the top-K intent closest to the query vector. To find the vector, ensure that the K value is set between 10 and 50. It is structured in such a way. The vector similarity search module (6) finds each intention. for, between 0 and 1, which intentions come from the client application (2) user 10 a similarity score that shows the query is semantically close to calculate these scores using the Large Language Model (LLM) It functions as a filtering mechanism to determine the intentions of the candidates to be sent. It is configured to enable visualization. Vector similarity search module (6), using the similarity score threshold, automatically detects low-scoring intentions. To eliminate, intentions with a similarity score below 0.3 will be removed from the candidate list. extracting, with this threshold, the prompt sent to the large language model to optimize its size and prevent the evaluation of irrelevant intentions It is being structured. The relevance filtering module (7) in the system (1) which is the subject of the invention, query processing to work in the process of the user query coming from the client application (2) task Is it a task-oriented request or a general chat (small talk, chitchat)? It is structured to determine that it is related. The relatedness filtering module (7), Labeled data consisting of task-oriented and general conversation examples 25 Using a separate FAISS index generated from the cluster allows the client application to... (2) search the incoming user query in this relevance index and find the closest examples to find, include the found examples in a large language model request, and the large With the language model, a binary decision is made as to whether the query is task-oriented or not. giving, when making this decision with the large language model, call function (function 30 It is configured to use the (calling) mechanism. Relevance filtering 9 module (7) is a predefined “task objective” of the large language model. It calls the "set_task_intent_flag" function and asks "is it relevant:" It should produce an output structured as "true / false" (is_relevant: true / false). to ensure that if the query is not relevant (is_relevant: false status), intent classification is applied. To ensure that the next stage is not reached and to send the client application (2) "5 on this subject Responding with "I can't help," thus preventing misclassifications. significantly reduce and use resources efficiently It is structured to provide this. The large language model integration module (8) in the system (1) that is the subject of the invention, 10 an abstraction layer for the purpose of interacting with large language models Using an abstraction layer; this layer can be used with various software libraries. through a single interface access to different major language model providers It is structured to provide the large language model integration module (8), User query and candidate intents from client application (2) a request 15 to transform it into and send it to the large language model, the large language model, this to enable it to determine the most likely intention based on the information It is structured. The large language model integration module (8), large language the model uses a structured function call instead of generating free text. It is structured to enable the use of the mechanism. Large language model 20 integration module (8), an artificial intelligence protocol of the large language model using; it chooses one of the letter options from A, B, C, D. requiring, each letter to correspond to one of the candidate intentions, and the "intention letter" to make it call a function named "classify" (classify_intent_letter, Selection of the major language model Structured JavaScript Object Representation 25 To ensure it returns in JavaScript Object Notation (JSON) format. It is structured. The large language model integration module (8), large language model application programming interface (LLM API); function call (function The logarithmic probability (log-) for each word segment (token) is calculated along with the (calling) response. It is configured to return probability values. 30 The intention classification module (9) in the system (1) which is the subject of the invention, client to forward the user query from application (2) to the embedding service module (5) and obtaining the query vector, then using this vector to perform vector similarity search. from the module (6), to get the most similar intentions top-K, these intentions, similarity Grouping them according to their scores and recording the highest score for each unique intention 5 It is structured as follows: Intent classification module (9), a threshold By applying the value filter, low-scoring intentions are filtered, and the remaining candidate intentions are presented in a to place the command template into the request template; by humans configuration files and data that are easy to read and can be processed by machines. a widely used, user-friendly data serialization language for storage, and 10 Storing it in an external file in a format called "YAML" and In the file; user query from client application (2), candidate intent The list should include information in the form of explanations of each intention and example sentences. This provides the ability to call functions from the large language model. Ensuring that it is sent by enabling the response of the large language model, “intent 15 the function call should return "classify_intent_letter". to provide, decode the selected letter, large language model implementation programming interface to record the logarithmic probability value returned from its face It is being structured. The confidence score calibration module (10) in the system (1) which is the subject of the invention, is large the raw logarithmic probability values ​​of the language model are converted into calibrated confidence scores. It is configured to convert. Confidence score calibration module (10), calibration process, temperature scaling method to achieve this using, in machine learning and deep learning, especially in the 25 In multi-class classification problems, a neural network converting raw outputs (logits) into probability values ​​between 0 and 1. a mathematical activation function called "softmax" It is configured to add a temperature parameter (T) to a function. Confidence score calibration module (10), temperature parameter greater than 1 30 When this happens, the temperature parameter helps to smooth the probability distribution. 11 To ensure that its distribution is sharpened when it is less than 1. It is structured. Confidence score calibration module (10), temperature Storing the parameter as an adjustable value, this parameter, confidence estimated on a validation dataset during the development phase a metric that measures the difference between scores and actual accuracy rates, and is 5 Expected Calibration Error (ECE) It is structured to determine this by minimizing a metric called [name of metric]. Confidence score calibration module (10), confidence obtained after calibration. Returning the score as a value between 0 and 1 to the client application (2), this Providing a critical metric for automation decisions with a score, confidence score 10 Automatic action is taken if it's above 0.9, and human approval is required if it's between 0.5 and 0.9. to request, if below 0.5, to consider the classification as failed. It is being structured. The invention concerns the request template management module (11) located in the system (1), large language 15 requests sent to the model should be stored in external YAML files. to enable prompt engineering and different prompts To enable testing of versions, variable placement within the YAML file. to ensure placeholders are in place and to ensure accuracy during operation. It is structured to be filled with values. 20 Industrial application of the invention The system of invention (1) searches for vector similarity from a large set of intentions. the rapid filtering of relevant candidates, then the language model of these candidates 25 the language model's ability to make a final classification through contextual understanding. By calibrating log-probability values ​​using a temperature scaling method, the actual results are obtained. generating confidence scores, determining whether the user query is a task-oriented request or not. Determining whether it is a general chat or not is possible thanks to vector indexing, hundreds of or working with thousands of intention categories and easily creating new intentions 30 by adding it using structured function calls to the language model, 12 ensuring the consistency and processability of the output format It provides. Around these fundamental concepts, the subject of the invention is "The Large Language Model and Vector". Similarity-Based Intent Classification and Confidence Score Validation System (1)” 5 It is possible to develop a wide variety of applications related to this, and the invention is presented here. It cannot be limited to the examples given; it is essentially as stated in the claims.

Claims

13 REQUESTS 1. In the field of natural language processing and artificial intelligence, users can express themselves in natural language. automatic classification of the queries and requests they make developed to provide; 5 - Making REST API calls over the http protocol, on a mobile device in the form of an application, web interface and / or other microservice at least one client application configured to run (2), - Verify the carrier token in the security header of each incoming request, authenticating tokens with a predefined public key, 10 At least one identity configured to prevent unauthorized access Containing verification module (3); - for each of the predefined intention categories within it to include explanatory texts, one or more for each category of intent including too many example sentences and explanations, these statements of intent, a 15 Converting to high-dimensional vectors via the embedding model, The generated vectors are used in a FAISS format for fast similarity searching. indexing using a vector indexing library, this index, to load into memory and keep it in memory throughout the entire query process, thus ensuring that disk access latency is eliminated 20 At least one intent database and vector indexing are configured. module (4), - processing the query text coming from the client application (2), an external by calling the embedded application programming interface, the query text At least one embedded 25 configured to convert a digital to a vector. service module (5), - After the query vector is created, the cosine of the FAISS index is calculated. using a metric such as similarity or Euclidean distance Finding the top-K intent vector closest to the query vector, the value of K The minimum setting (EN 30) is configured to allow adjustment between 10 and 50. a small vector similarity search module (6), 14 - working in the query processing process, coming from the client application (2) Is the user query a task-oriented request or a general chat? at least one relevance filter configured to determine this. module (7), - an abstraction for the purpose of interacting with large language models 5 using the layer, various software libraries with this layer through a single platform, it provides access to various major language model providers. to provide from the interface, user from client application (2) to turn the inquiry and candidate intentions into a request and the big language sending it to the model, the large language model, based on this information, will at least 10 at least one structured to enable the identification of possible intent Big Language Model Integration Module (8), - Embedding service for user query from client application (2) to pass to module (5) and obtain the query vector, this vector using the vector similarity search module (6), top-K most similar 15 to collect intentions, to group these intentions according to their similarity scores, and to record the highest score for each unique intention at least one intent classification module configured (9), - calibrating the raw logarithmic probability values ​​of the large language model at least one 20 structured to convert into verified confidence scores confidence score calibration module (10), - requests sent to the large language model are in external YAML files to ensure it is hidden, to enable system engineering and different Enabling testing of client versions, YAML file to ensure the inclusion of variable placeholders and study 25 at least one structured to be filled with real values ​​in a timely manner a system characterized by the request template management module (11) (1).

2. Representative State Transfer via Hypertext Transfer Protocol 30 Making Application Programming Interface calls, a mobile application, to operate as a web interface and / or another microservice As in Claim 1, characterized by the configured client application (2). a system (1).

3. Verify the carrier token in the security header of each incoming request, 5 Authenticating tokens with a predefined public key, unauthorized authentication module configured to block access (3) as in any of the above claims characterized by system (1).

4. For each of the predefined categories of intention within it. to include explanatory texts, one or more for each category of intent Including example sentences and explanations, these statements of intent, in an embedded form. converting the generated vectors into high-dimensional vectors through the model vectors, a vector in the form of FAISS for fast similarity search 15 indexing using an indexing library, storing this index in memory. to load and keep in memory throughout all query operations, so that the disk configured to eliminate access latency characterized by intent database and vector indexing module (4) a system like any of the above requests (1). 20 5. To process the query text from the client application (2), use an external embedding. By calling the application programming interface, the query text is numerical. converting to vector, the generated query vector, intent database and Vector 25 to be compared with vectors in the vector indexing module to pass to the similarity search module (6), with different embedding models by working to select the model that provides the best performance and different configured to facilitate the transition between embedding providers from the above requests characterized by the embedded service module (5) a system like any of them (1). 30 16 6. After the query vector is created, cosine similarity is applied to the FAISS index. or by using a metric in the form of Euclidean distance, the most accurate value for the query vector. Finding the near top-K intention vector, where K is a value between 10 and 50. Vector similarity search 5 configured to enable adjustment any of the above requests characterized by module (6) a system like one of them (1).

7. For each intent found, select a scale of 0 to 1 indicating which intents the client has. 10 that is semantically close to the user query from application (2) calculating a similarity score that shows that these scores are related to a large language A filter to determine candidate intents to be sent to the model. a vector structured to function as a mechanism from the above requests characterized by the similarity search module (6) a system like any other (1). 15 8. Automatically identify low-scoring intentions using the similarity score threshold. to eliminate intentions with a similarity score below 0.3 from the candidate list by extracting, through this thresholding, the request size sent to the large language model. to optimize and prevent the evaluation of irrelevant intentions 20 The vector similarity search module (6) is structured to characterize a system like any of the above-mentioned requests (1).

9. Working in the query processing process, the user coming from the client application (2) whether the inquiry was a task-oriented request or a general conversation 25 with the relevance filtering module (7) configured to determine and provide as in any of the above characterized claims system (1). 17 10. Labeled data consisting of task-oriented and general conversation examples. Using a separate FAISS index generated from the cluster, the client Searching for the user query from application (2) in this relevance index and finding the closest examples, and using the found examples to create a large language model. to include in the request and with the large language model, the query is task-oriented and 5 making a binary decision that it is not, while making this decision with the grand language model, Relevance configured to use the function call mechanism from the above requests characterized by the filtering module (7) a system like any other (1).

11. The major language model has a predefined “mission objective flag” It should call the "set" function and ask "is it relevant: true / false". to ensure it produces a structured output, if the query is not relevant, to ensure that the intent classification stage is not reached and the client (2) A response to the application is "I cannot help with this" 15 to rotate, thereby significantly reducing misclassifications and to ensure the efficient use of resources characterized by the structured relevance filtering module (7) a system like any of the above requests (1).

12. An abstraction for the purpose of interacting with large language models. using the layer, through various software libraries providing access to different major language model providers from a single interface. with the large language model integration module (8) structured for a 25 as in any of the above characterized claims system (1).

13. The client application receives the user query and candidate intents from (2). to make a request and send it to the large language model, large language 30 Enabling the model to determine the most likely intention based on this information 18 with the large language model integration module (8) structured for as in any of the above characterized claims system (1).

14. The large language model should generate a structured 5 instead of free text. to enable the use of a function call mechanism characterized by the structured large language model integration module (8) a system like any of the above-mentioned requests (1).

15. Using an artificial intelligence protocol, the large language model A, B, C, D 10 each The letter corresponds to one of the candidate intentions and "classify the intention letter". to make it call a function named, the large language model select structured JavaScript Object Notation format large language model integration structured to enable its return 15 any of the above requests characterized by module (8) a system like one of them (1).

16. Function calls in the large language model application programming interface. The logarithmic probability values ​​for each word fragment, along with the answer, are 20. large language model integration structured to enable its return any of the above requests characterized by module (8) a system like one of them (1).

17. Embedding the user query from the client application (2) service 25 to pass to module (5) and obtain the query vector, using this vector From the vector similarity search module (6), retrieve top-K most similar intents, grouping these intentions according to their similarity scores and identifying each unique intention Intent classification structured to record the highest score for 19 any of the above requests characterized by module (9) a system like one of them (1).

18. By applying a threshold value, low-scoring intentions can be filtered out, leaving the remaining candidates. their intentions, placing them into a request template, a command template; people 5 easy to read, machine-processable, configuration A commonly used, user-friendly data storage solution for files and data. an external serialization language in a format called "YAML" to store in a file and in this file; from the client application (2) The incoming user query, candidate intent list, description of each intent, and example 10. to ensure that information is included in the form of sentences, this request, the big language the model should be sent with the function call feature enabled. to provide, the response of the large language model is, "classify the intent letter" To return a function call, decode the selected character, The large language model returns a logarithmic 15 from the application programming interface. Intent classification configured to record probability values. any of the above requests characterized by module (9) a system like one of them (1).

19. Calibrated raw logarithmic probability values ​​of the large language model 20 confidence score structured to convert into confidence scores from the above requests characterized by the calibration module (10) a system like any other (1).

20. Perform the calibration process using the temperature scaling method. 25 to implement, in machine learning and deep learning, especially in very In multi-class classification problems, the raw outputs of a neural network can be scaled between 0 and 1. a mathematical activation that converts between probability values a function that is called "softmax" Confidence score configured to include the temperature parameter (T) is 30. from the above requests characterized by the calibration module (10) a system like any other (1).

21. When the temperature parameter is greater than 1, the probability distribution... to ensure softening, 5 when the temperature parameter is less than 1. trust structured to ensure a sharpening of its distribution The score above is characterized by the calibration module (10) a system like any of the requests (1).

22. Storing the temperature parameter as an adjustable value, this 10 the parameter on a validation dataset during the development phase the difference between estimated confidence scores and actual accuracy rates a metric that measures and is called expected calibration error. a confidence score structured to determine a metric by minimizing it 15 of the above requests characterized by the calibration module (10). a system like any other (1).

23. After calibration, the confidence score obtained is a value between 0 and 1. Returning to the client application (2) with this score and automation Providing a critical metric for their decisions, if the confidence score is above 0.9, then 20 take automatic action, request human approval if it's between 0.5-0.9, and if it's above 0.5 confidence structured to consider classification a failure if it is below a certain level. The score above is characterized by the calibration module (10) a system like any of the requests (1).

24. Requests sent to the large language model are in external YAML files. to ensure storage, to enable requests engineering and different requests to enable testing of versions within the YAML file to ensure the inclusion of variable placeholders and real-time operation. Request template management module 30 configured to be populated with values 21 as in any of the above claims characterized by (11). a system (1). 10 20 30

Images