Conversational rag pipeline for an llm-based automotive assistant

Abstract

An apparatus for interacting with an occupant in a vehicle includes an infotainment system that has been integrated into the vehicle, an automotive assistant that is configured to execute in the infotainment system, a speech interface that is configured to receive, from the occupant, an original utterance that is to be processed by the automotive assistant, and a classifier that determines that the original utterance has either anaphora or ellipsis. A model that provides a response to the occupant based on a prompt that includes the original utterance and that has been augmented by a function specification selected based at least in part on a rewritten utterance that has been derived from the original utterance.

Description

CONVERSATIONAL RAG PIPELINE FOR AN LLM-BASED AUTOMOTIVE ASSISTANTCross-Reference to Related Applications

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 561,462, filed on March 5, 2024, which is incorporated herein by reference.Technical Field

[0002] Disclosed herein is a system for conversation RAG pipeline for an LLM based automotive assistant.Background

[0003] This application relates to processing natural language commands, and more particularly to an approach to retrieval augmented generation (RAG) using large language models (LLMs).

[0004] A typical automobile includes an infotainment system that executes various applications, among which is an automotive assistant. Such an automotive assistant uses a speech interface to communicate with an occupant of the vehicle.

[0005] In some cases, the automotive assistant uses a large language model (LLM), referred to herein as simply a “model,” to assist it in providing information to the occupant.

[0006] A model can be viewed as a system that attempts to provide a meaningful response to a prompt by availing itself of a corpus of information. In some cases, the model fails to do so. Instead, it provides an inappropriate response.

[0007] An inappropriate response most often arises because of a flaw in the prompt or a flaw in the corpus used to respond to the prompt.

[0008] A prompt must be constructed with some care to ensure that the interaction with a model is likely to succeed. In some cases, a casual user may have neither the talent nor the inclination to provide an utterance that forms the basis of a good prompt. In other cases,a speech interface may fail to correctly interpret an occupant’s utterance, thus dooming the prompt that relies on the utterance. As a result, it is known to provide a prompt-rewriter that has been trained to spot flaws in a prompt, to rewrite the prompt, and to submit that prompt to the model instead of the original one.

[0009] An inappropriate response can also arise because of irrelevant subject matter in the corpus that the model consults when responding to the prompt. This difficulty can be overcome by augmenting the prompt so that the model uses only carefully curated information for providing the response. A known procedure for carrying this out is “retrieval augmented generation” or “RAG.”Summary

[0010] In real world conversations with a large language model (LLM) based automotive assistant, users do not always use full-fledged standalone requests. More often than not, user requests are contextually dependent utterances and contain ellipsis, anaphora and other signs of contextual dependence. Contextually dependent utterances contain fewer semantic cues, and using such utterances for retrieval in RAG may result in incorrect results. In a general aspect, this problem is addressed introducing a rewrite module prior to the vector database search in our RAG pipeline. This module receives both the query and conversational history or “context” (e.g., restricted to a certain window) and transforms (e.g., rewrites) the conversational query into a standalone one that can be sent to the retriever model. The rewriter is an LLM with a carefully tuned prompt, but can also be a specially finetuned smaller generative model. In another aspect, the system determines whether the query needs to be rewritten. The rewrite module contains or makes use of an ellipsis / anaphora classifier. In some examples, this is implemented as a support vector machine (SVM) classifier that takes sentence embeddings as input. The classifier does not contribute to the processing latency in a significant manner, but having this classifier in the pipeline allows one to reduce the number of LLM calls for rewriting and also helps to improve the accuracy of rewriter outputs. For instance, when the LLM tries to rewrite utterances that do not need it, accuracy may be lower than if the original utterance was used for the RAG stage. On the other hand, selectively rewriting the query before RAGretrieval has been found to significantly enhance the conversational capabilities of the LLM-based automotive assistant.

[0011] A typical interaction with an automotive assistant is conversational in nature. In such interactions, the meaning of any one utterance will often depend on utterances that preceded it. A typical utterance is so riddled with ellipsis, anaphora, and other cues that indicate context dependence that to use it as a prompt will likely result in an inappropriate response from a typical model.

[0012] It is possible to provide the model with additional context to promote its ability to provide a meaningful response to a prompt. However, in the case of an augmented prompt, the utterance enters a different pipeline in which it is used to ultimately select information for augmenting the prompt. This pipeline is more sensitive to any flaws in the utterance, such as anaphora and ellipsis.

[0013] The aspects disclosed herein contemplate rewriting a flawed utterance before it has a chance to adversely affect the outcome of this second pipeline. The rewriting operation is carried out by a rewriter that receives both the utterance and a portion of conversation preceding the utterance to provide context. The rewriter uses both of these to construct a modified utterance that can be used to retrieve information for augmenting the prompt.

[0014] In one aspect, the disclosure features an apparatus for interacting with an occupant in a vehicle. Such an apparatus includes an infotainment system that has been integrated into the vehicle, an automotive assistant that is configured to execute in the infotainment system, a speech interface that is configured to receive, from the occupant, an original utterance that is to be processed by the automotive assistant, and a classifier that determines whether the original utterance is an utterance of a first kind or an utterance of a second kind. An utterance of the first kind comprises at least one of anaphora and ellipsis. An utterance of the second kind is the converse of that of the first kind. It lacks both anaphora and ellipsis. The apparatus further includes a model that provides a response to the occupant based on a prompt that includes the original utterance. This prompt hasbeen augmented by a function specification that was selected based at least in part on either the original utterance or a rewritten utterance. In the latter case, the rewritten utterance is one that has been derived from the original utterance.

[0015] Some embodiments include a function retriever as well as a rewriter that is called into action when the utterance is of the first kind. The rewriter is one that receives the original utterance from the classifier and generates therefrom the rewritten utterance. The function retriever receives the rewritten utterance and retrieves the function specification from a database based at least in part on the rewritten utterance.

[0016] Embodiments also include those that include a function retriever that receives an utterance of the first kind from the classifier and retrieves the function specification from a database based at least in part on the original utterance.

[0017] In some embodiments, the prompt further includes context that promotes resolution of the original utterance when the original utterance is an utterance of the first kind.

[0018] Also, among the embodiments are those that include a rewriter that receives both the original utterance and context. The rewriter is configured to generate a rewritten utterance based on the original utterance and the context.

[0019] In another aspect, the disclosure features a method for augmenting a prompt based on a retrieved function specification. Such a method includes receiving, from an automotive assistant executing in an infotainment system of a vehicle, an original utterance from an occupant of the vehicle, generating a prompt that includes the original utterance, determining whether the original utterance is an utterance of a first kind or an utterance of a second kind, wherein an utterance of the first kind comprises at least one of anaphora and ellipsis and an utterance of the second kind lacks both anaphora and ellipsis, retrieving the functional specification based at least in part on either the original utterance or a rewritten utterance, the rewritten utterance having been derived from the original utterance, and augmenting the prompt with the functional specification.

[0020] In another aspect, in general, a method automated processing of a user query comprises receiving a text-based query (which may be a question or a command), computing a vector representation of said query, and retrieving a plurality of content items according to the vector representation. A prompt is formed based at least on the text-based query and the plurality of content items. A large language model is used to process the prompt to yield a model output, and the model output is processed to act on the query. Computing the vector representation comprises selecting between a plurality of ways of computing the vector representation. This plurality of ways includes a first way of computing said vector representation that comprises processing the query with a first embedder to form the vector representation (i.e., without any use of the context), and a second way that comprises rewriting the query by processing said query and a context of said query to form a rewritten query, and then processing the rewritten query with a second embedded to form the vector representation. The vector representation is computed using the selected way.

[0021] Aspects can include one or more of the following features.

[0022] A trained machine learning classifier is used to select between said ways.

[0023] A first vector representation is computed from the query using the first embedder, and using the trained machine-learning classifier includes processing said first vector representation.

[0024] Using the trained machine learning classifier comprise using a support vector machine (SVM).

[0025] Rewriting the query includes processing a combination of the query and the context with a large language model to yield the rewritten query as an output of the large language model.

[0026] At least one of the first embedder and the second embedded comprises a neural network.

[0027] The neural network of said embedder comprises a transformer based neural network.

[0028] The first embedder and the second embedder have the same structure and configuration parameter values.

[0029] The content items comprise function specifications of a software library.

[0030] The model output to act on the query comprises calling a function specified in model output, said function being a function specification of one or the plurality of retrieved content items.

[0031] Retrieving the plurality of content items comprises retrieving said content items from a database formed from the software library.

[0032] The database comprises a plurality of records, each record comprises a function specification of a function of the software library, and a vector representation of said function. Retrieving the plurality of content items comprises selecting function specifications of functions in said database according to a similarity of the vector representations of said functions and the vector representation computed from the query.

[0033] The classifier is trained using a data set comprising queries, contexts, and corresponding functions.

[0034] The training of the classifier comprises adjusting values of parameters of said classifier according to a similarity of a vector representation selected by an output of said classifier and a vector representation of a corresponding function.

[0035] A machine-readable medium comprising instructions stored thereon, execution of said instructions causes a processor of a data processing system to perform all the steps of any method set forth above.

[0036] These and other features of the disclosure will be apparent from the following detailed description and the accompanying figures, in which:Description of Drawings

[0037] FIG. 1 shows a vehicle having an infotainment system that executes an automotive assistant; and

[0038] FIG. 2 shows a dual pipeline architecture used by the automotive assistant and model of FIG. 1.Detailed Description

[0039] FIG. 1 shows a vehicle 10 carrying an occupant 12. The vehicle 10 includes an infotainment system 14 that is in communication with a remote server 16. The infotainment system 14 executes an automotive assistant 18 that communicates with the occupant 12 via a speech interface 20.

[0040] As a result of its communication with the remote server 16, the automotive assistant 18 is able to make use of computational resources at the remote server 16 while using the speech interface 20 to interact with the occupant 12. Among the resources at the remote server 16 is a model 22. The model 22 is typically a large language model.

[0041] The remote server 16 may be located at a separate and distinct location from the vehicle 10. The remote server 16 may be at a remote location or a cloud server and may be accessible by the vehicle 10. The vehicle 10 may be equipped with certain communication networks, wireless transceivers, vehicle communication busses, etc., to deliver and receive information between the vehicle and the server 16. The communication network may be referred to as a “cloud” and may involve data transfer via wide area and / or local area networks, such as the Internet, a global navigation satellite system (GNSS), cellular networks, Wi-Fi, Bluetooth, etc. The communication network may provide for communication between the vehicle 10 and the external or remote server 16 and / or databases, as well as other external applications, systems, vehicles, etc. This communication network may provide navigation, music or other audio, program content, marketing content, internet access, speech recognition, cognitive computing, artificial intelligence, to the vehicle 10.

[0042] The speech interface 20 may include a microphone, speaker, and / or display to receive and provide information to the occupant 12. The microphone may be configured receive audio signals from within the vehicle cabin, such as acoustic utterances including spoken words, phrases, or commands from a user. The microphone may include a plurality of microphones arranged throughout the vehicle cabin.

[0043] The speaker may provide audio playback may include audio from sources such as the vehicle virtual assistant, as well as a vehicle radio, including satellite radio, decoded amplitude modulated (AM) or frequency modulated (FM) radio signals, and audio signals from compact disc (CD) or digital versatile disk (DVD) audio playback, streamed audio from a mobile device, commands from a navigation system, etc. The virtual assistant may provide audible instructions and responses to the occupant 12.

[0044] Referring to FIG. 2, during use of the automotive assistant 18, the occupant 12 of the vehicle utters a command, which may be one in a series of voice exchanges with the assistant, and the speech interface 20 acquires the speech (e.g., via a microphone in the vehicle) and processes the speech signal using a machine-implemented automatic speech recognizer to yield a text based query 28. The overall system processes this query 28 and ultimately generates a text based response 56, which is presented to the occupant typically in a synthesized voice form, although other forms of response 56, such as graphical response on the dashboard, or not explicit response when there is an implicit response by virtue of an action that is taken by vehicle systems in response to the command.

[0045] In the course of processing the query 28, the model 22 receives a prompt 24 that is assembled from a carrier 26, the occupant’s query 28 or occupant’s utterance 28, as well as certain information, referred to herein as “context 30” that promotes the model’s ability to correctly understand the query 28, for example, to resolve an anaphora or ellipsis in the occupant’s utterance 28. Typical context 30 includes previous utterances from which it is possible, in many cases, to resolve such anaphora or ellipsis in the occupant’s utterance 28. The context 30 can also include other information, such as vehicle state (e.g., speed, location, status of windows being open or closed, etc.), generally expressed in a text form to be compatible in form with the query 28.

[0046] As used herein, “ellipsis” is not to be construed as the punctuation mark that shares its name but rather in the sense used in the study of linguistics to indicate that subject matter that is essential to divining the meaning of the utterance is missing from that utterance.

[0047] Also in the course of processing the query 28, the system may make use of a software library 54, for example, software that is executed locally in the vehicle or remotely in the cloud. Examples of such software include software that performs vehicle functions or retrieves vehicle information (e.g., opening a window or retrieving the amount of fuel remaining), and software that acts on or retrieves remote information (e.g., sending a message, turning on garage lights, or retrieving a weather forecast). As can be imagined, there may be a large number of software functions that may be invoked via the library, and collectively, the specification of how those functions are invoked is referred to as an application programming interface (API). Therefore, in the course of processing the query 28, the system may need to determine which function is to be invoked (“called”), and the values of parameters (“arguments”) of that function to use in that invocation. The system makes the call, and uses returned information from the call to form the response 56.

[0048] To accommodate the large number of functions that can be invoked depending on the occupant’ s query 28, the system uses a technique referred to as a retrieval augmented generation (RAG). In this technique, the processing of the query 28 has several stages. In the first stage, a subset of available functions is identified, and represented as a set of retrieved functions 34 in a text form as the most likely to be relevant functions 34 that are available. In the second stage, the prompt 24 is assembled using the query 28, the retrieved functions 34, as well as the context 30 and the relatively static carrier 26 in which the other components of the prompt are textually embedded. This prompt 24 is processed by the model 22 to yield an output. The output is not a response to the query 28. Rather, it has sufficient information to specify the function of the software library 54 to invoke it. In the third stage, the output of the model 22 is processed by an API processor 52 to interpret the model output and to make the appropriate software call to the software library 54, and to receive the associate response from the library. The API processor 54 interprets theresponse, which may be in a structured data form, and form the corresponding text response 56.

[0049] Turning to the processing in the first stage, the retrieving makes use of a function database 36, which includes a record for each of a plurality of functions 44 that can be invoked. The identification of the relevant functions 44 makes use of a vector-based similarity. In particular, the information characterizing each function 34 in the software database 36 is pre-processed by a library pre-processor 35, and for each function 34, the pre-processor 35 computes a vector representation 48. For example, the pre-processor 35 may process a text description of the function 34 (e.g., from documentation of the function and its arguments), using a neural network, such as a transformer-based network, which can be a bidirectional encoder representations from transformers (BERT) network. A function retriever 32 uses a vector representation of the query 28 to find the most relevant functions 44 by comparing the vector representation of the query 28 with the vector representations 48 of the functions, for example, according to an inner product, cosine similarity, or Euclidean distance computation between the two vectors. As part of the preprocessing of the software library 54, a text representation 46 of each function 44 is also stored in the database 36, and the text representations 48 of the closest functions are returned to the retriever 32, which passes them on for inclusion in the prompt 24.

[0050] An important aspect for the overall quality of the response 56 is the quality of the retrieved functions 44, and this in turn is dependent on the quality of the vector representation 48 of the occupant’s query 28. The approach taken here is to provide two ways of computing the vector representation of the query 28. One way is to process the query 28 with an embedder 37 to form the vector representation. For example, the embedder 37 is of the same type as used in the library pre-processor 35, and takes the text of the query 28, and forms a vector representation of the same size as the vector representations 48 of the functions. As introduced above, there are situations in which such a direct conversion may not yield particularly relevant functions, for example, if the query 28 is “open it” without the context of what “it” means, when with context it would be clear that “it” referred to the trunk. While the retriever 32 might return functions 34 that relateto opening something, they would not in any way be relevant opening a trunk, as opposed to opening a window or a garage door. Another way to process the query 28 is to use a rewriter 40, which takes the query 28, and the context 30, and aims to form a more fully formed text presentation of the query 28. For instance, the rewriter 40 is a transformer network based large language model that receives the query 28 and context 30 and a directive to clarify the query 28. Once the query 28 is rewritten to form a rewritten query 42, the rewritten query 42 is processed using an embedder 41 to form a vector representation of the query 28. In some cases, the embedder 41 is the same as the embedder 37, but they may have the same structure and be trained differently based on the nature of their inputs, or could have different structures as well. In using this second way of processing the query 28, it was found that there are instances in which the second way is not always better than the first way. Issues with the second way include “hallucination” of content in the rewritten query 42, which ultimately results in retrieval of less relevant functions than in the first way. Another issue with the second way is that it is more computationally costly to determine the rewritten query 42, and can introduce computational delays that affect the interaction with the occupant by delaying when they receive the response 56. In cases in which the query 28 is precise and fully formed, the first way, which computes the vector representation directly, results in retrieving more relevant functions and incurs lower computational cost and introduces less delay. So there remains the technical problem of deciding when to use the first approach and when to use the second approach.

[0051] To distinguish between these two cases, the system uses a classifier 38 to classify the query 28 as belonging to either a first set that should be processed the second way including rewriting of the query, or a second set that are best processed the first way directly without rewriting. As introduced above occupant’s utterance 28 is in the first set if it contains at least one of an ellipsis and an anaphora, or otherwise has ambiguity or imprecise meaning that could be addressed with context 30. The second set is the complement of the first set.

[0052] The classifier 38 can take various forms. In the implementation illustrated in FIG. 2, the classifier 38 is a support vector machine classifier that receives, as its input, the vector representation of the occupant’s query 28 as output from the embedder 37. Another suitable classifier 38 is one based on a multilayer perceptron. It has been found that such classifiers are easy and inexpensive to train. Such training can include input queries and context, and the vector representation of the most relevant function and use of a criterion to yield the closest matching representation of the query 28. Embodiments include those in which the classifier 38 is a monoglot classifier and those in which it is a polyglot classifier. It has been found that, in some cases, a monoglot classifier works acceptably well in a language other than that in which it has been trained. In either case, it is particularly useful to divide the training data into first and second portions, to train the classifier 38 using only the first portion, and to tune the classifier 38 using the second portion.

[0053] In alternative embodiments, the classifier 38 does not necessarily use the output of the embedder 37 and uses the text of the query 28 itself. For instance, classification of the query 28 based on the words spoken (e.g., as a bag of words), or other processing of the word sequence, for example, using a neural network (e.g., a recurrent network or a relatively small and efficient transformer based model) may be used. Processing of a combination of the vector representation 48 and the word sequence of the query 28 can also be performed in this way with a neural network receiving both the vector and the word sequence.

[0054] In FIG. 2, the output of the classifier 38 passes to a switch 39, which logically selects the output of the embedder 37 or the embedder 41. The computation involved in the rewriter 40 should be deferred until after the selection by the output of the classifier 38 is made in order to avoid unnecessary computational expense but can be performed in anticipation of the selection to reduce delay in response.

[0055] As introduced above, the embedders 37 and 41, rewriter 40, and the classifier 38, may be parameterized components, and setting of the values of the parameters may affect the quality of the retrieved functions and thereby the quality of the overall response 56. In one approach, all but the classifier are pre-trained, for example, the rewriter being ageneral purpose large language model, and the embedders 37 and 41, and as used in the library preprocessor 35, are pretrained BERT models, with the token-by-token outputs being pooled (e.g., averaged) to produce a single fixed length vector representation. The classifier, such as an SVM implementation, is then trained to select the closest of the two vectors produced by the embedders 37 and 41 using a training set that includes the queries and the vector representation of the “correct” function. Other approaches to training may use the quality of the ultimate response 56 as the training criterion. In some implementations, multiple of the components may be jointly trained, for example, by introducing different processing of the token-by-token outputs of transformers that process the query of the rewritten query 42 to best match the vector representation of the most relevant function.

[0056] In the descriptions above, a particular architecture may be described in which some of the components illustrated in FIGS. 1 and 2 are performed locally (e.g., in the vehicle) and some are performed remotely (e.g., in the “cloud”). It should be understood that the described selection of the two ways of computing the vectors for the RAG retrieval is applicable in many other architectures, and is not limited to automotive assistant applications. Furthermore, while the RAG approach is described in the context of retrieving the most relevant functions for calling from a software library, this approach can be used for retrieving documents from a document library rather than a function database, with the model 22 generating a response directly without making an API call. In yet other implementations, the RAG retrieval may access both relevant functions or relevant documents, for example, in a vehicle in which the occupant may both issue a command to perform a software implemented function and make a request to retrieve information from a document, such as an owner’s manual (e.g., “what tire pressure should I have in my rear tires?”).

[0057] Implementations of the approaches described above may be software based in which case a non-transitory machine-readable medium contains instructions that when processed by a data processing system causes a processor (i.e., computational circuitry, such as a central processing unit (CPU) and / or a graphics processing unit (GPU)) of thesystem to perform the operations described above. The system and / or the medium containing the instructions may be distributed with some parts being in a local environment (e.g., a vehicle) and some parts being remote to that environment. Such implementations may make use of special purpose circuity (“hardware”) for some functions, such as application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). Some implementations use a combination of software and hardware.

[0058] Having described the aspects and a examples thereof, what is claimed as new and secured by letters patent is:

Claims

CLAIMS1. An apparatus for interacting with an occupant in a vehicle, said apparatus comprising an infotainment system that has been integrated into said vehicle, an automotive assistant that is configured to execute in said infotainment system, a speech interface that is configured to receive, from said occupant, an original utterance that is to be processed by said automotive assistant, a classifier that determines whether said original utterance is an utterance of a first kind or an utterance of a second kind, wherein an utterance of the first kind comprises at least one of anaphora and ellipsis and an utterance of the second kind lacks both anaphora and ellipsis, and a model that provides a response to said occupant based on a prompt that includes said original utterance, said prompt having been augmented by a function specifications that was selected based at least in part on either said original utterance or a rewritten utterance, said rewritten utterance having been derived from said original utterance.

2. The apparatus of claim 1, wherein said original utterance is an utterance of the first kind and wherein said apparatus further comprises a rewriter that receives said original utterance from said classifier and generates therefrom said rewritten utterance and a function retriever that receives said rewritten utterance and retrieves said function specifications from a database based at least in part on said rewritten utterance.

3. The apparatus of claim 1, wherein said original utterance is an utterance of the second kind and wherein said apparatus further comprises a function retriever that receives said original utterance from said classifier and retrieves said function specifications from a database based at least in part on said original utterance.

4. The apparatus of claim 1, wherein said original utterance comprises at least one of anaphora and ellipsis and wherein said prompt further comprises context that promotes resolution of said original utterance.

5. The apparatus of claim 1, further comprising a rewriter that receives said original utterance and context and that is configured to generate a rewritten utterance based on said original utterance and said context.

6. A method for augmenting a prompt based on a retrieved function specifications, said method comprising receiving, from an automotive assistant executing in an infotainment system of a vehicle, an original utterance from an occupant of said vehicle, generating a prompt that includes said original utterance, determining whether said original utterance is an utterance of a first kind or an utterance of a second kind, wherein an utterance of the first kind comprises at least one of anaphora and ellipsis and an utterance of the second kind lacks both anaphora and ellipsis, retrieving said functional specifications based at least at least in part on either said original utterance or a rewritten utterance, said rewritten utterance having been derived from said original utterance, and augmenting said prompt with said functional specifications.

7. A method automated processing of a user query comprising: receiving a text-based query; computing a vector representation of said query; retrieving a plurality of content items according to the vector representation; forming a prompt based at least on the text-based query and the plurality of content items; using a large language model to process the prompt to yield a model output; and processing the model output to act on the query; wherein computing the vector representation of the comprises, selecting between a plurality of ways of computing the vector representation, said plurality of ways including a first way of computing said vector representation comprising processing the query with a first embedder to form the vector representation, and a second way comprising rewriting the query by processing said query and a context of said query to form a rewritten query, and processing the rewritten query with a second embedder to form the vector representation, and computing the vector representation using the selected way.

8. The method of claim 7, wherein the method includes using a trained machine learning classifier to select between said ways.

9. The method of claim 8, wherein the method includes computing a first vector representation from the query using the first embedded, and using the trained machine-learning classifier includes processing said first vector representation.

10. The method of any one of claims 8 and 9, wherein using the trained machine learning classifier comprises using a support vector machine (SVM).

11. The method of claim 7, wherein rewriting the query includes processing a combination of the query and the context with a large language model to yield the rewritten query as an output of the large language model.

12. The method of any one of claims 7 to 11, wherein at least one of the first embedder and the second embedded comprises a neural network.

13. The method of claim 12, wherein the neural network of said embedder comprises a transformer based neural network.

14. The method of claim 12, wherein the first embedder and the second embedder have the same structure and configuration parameter values.

15. The method of any one of claims 7 to 14, wherein the content items comprise function specifications of a software library, and wherein processing the model output to act on the query comprises calling a function specified in model output, said function being a function specifications of one or the plurality of retrieved content items.

16. The method of claim 15, wherein retrieving the plurality of content items comprises retrieving said content items from a database formed from the software library.

17. The method of claim 16, wherein the database comprises a plurality of records, each record comprises a function specifications of a function of the software library, and a vector representation of said function, and wherein retrieving the plurality of content items comprises selecting function specifications of functions in said database according to a similarity of the vector representations of said functions and the vector representation computed from the query.

18. The method of any one of claims 7 to 17, further comprising training the classifier using a data set comprising queries, contexts, and corresponding functions.

19. The method of claim 18, wherein said training comprises adjusting values of parameters of the classifier according to a similarity of a vector representation selected by an output of the classifier and a vector representation of a corresponding function.

20. A machine-readable medium comprising instructions stored thereon, execution of said instructions causes a processor of a data processing system to perform all the steps of any one of claims 7 through 19.

Images