Computer-implemented method for processing a prompt

Abstract

According to an embodiment, a computer-implemented method (100) for processing a prompt, the method comprising: obtaining (101) a prompt; providing (102) the prompt to a large language model a plurality of times, thus obtaining a plurality of responses; calculating (103) at least one vector space distance based on the plurality of responses; and providing (104) a response to a user based on the plurality of responses and the at least one vector space distance.

Description

COMPUTER- IMPLEMENTED METHOD FOR PROCESSING A PROMPTTECHNICAL FIELD

[0001] The present disclosure relates to prompt processing, and more particularly to a computer-implemented method for processing a prompt , a computing device, and a computer program product .BACKGROUND

[0002] Large language models (LLM) are typically provided as cloud-based services . A user can use these models by providing so-called prompts , comprising for example questions and / or requests , and the LLM can provide results in response to these prompts . However, the reliabil ity of the responses provided by an LLM can be difficult to assess .SUMMARY

[0003] This summary is provided to introduce a selection of concepts in a s implif ied form that are further described below in the detailed description . This summary is not intended to identify key features or essential features of the claimed subj ect matter, nor is it intended to be used to limit the scope of the claimed subj ect matter .

[0004] It is an obj ective to provide a computer-implemented method for processing a prompt , a computing device , and a computer program product . The foregoingand other obj ectives are achieved by the features of the independent claims . Further implementation forms are apparent from the dependent claims , the description and the figures .

[0005] According to a first aspect, a computer-implemented method for processing a prompt comprises : obtaining a prompt ; providing the prompt to a large language model a plurality of times , thus obtaining a plurality of responses ; calculating at least one vector space distance based on the plurality of responses ; and providing a response to a user based on the plurality of responses and the at least one vector space distance .

[0006] In an implementation form of the first aspect , the method further comprises , before providing the prompt to the large language model the plurality of times , setting a temperature parameter of the large language model .

[0007] In another implementation form of the first aspect , the setting the temperature parameter of the large language model comprises setting the temperature parameter to be greater than a threshold temperature .

[0008] In another implementation form of the first aspect , the calculating the at least one vector space distance based on the plurality of responses comprises calculating the at least one vector space distance using cosine similarity .

[0009] In another implementation form of the first aspect , the calculating the at least one vector space distance based on the plurality of responses comprises :calculating a plurality of vector space distances , wherein each vector space distance in the plural ity of vector space distances corresponds to a vector space di stance between two responses in the plurality of responses ; and calculating at least one vector space distance stati stic based on the plurality of vector space distances ; wherein the providing the response to the user based on the plurality of responses and the at least one vector space distance comprises providing the response to the user based on the plurality of responses and the at least one vector space distance statistic .

[0010] In another implementation form of the first aspect , the at least one vector space distance statistic comprises at least one of : a vector space distance mean ; a vector space distance variance ; and / or a vector space distance standard deviation .

[0011] In another implementation form of the first aspect , the providing the response to the user based on the plurality of responses and the at least one vector space distance comprises , in response to the at least one vector space distance statistic not fulfilling at least one preconfigured condition, providing an error response .

[0012] In another implementation form of the first aspect , the providing the response to the user based on the plurality of responses and the at least one vector space distance comprises choosing at least one response from the plurality of responses based on the at least one vector space distance .

[0013] In another implementation form of the first aspect , the providing the response to the user based on the plurality of responses and the at least one vector space distance comprises , in response to the at least one vector space distance being greater than a threshold vector space distance , providing an error response .

[0014] In another implementation form of the first aspect , the providing the response to the user based on the plurality of responses and the at least one vector space distance comprises : selecting a subset of responses from the plurality of responses based on the at least one vector space distance ; and providing the response to the user based on the subset of responses .

[0015] In another implementation form of the first aspect , the providing the response to the user based on the subset of responses comprises : providing a summary prompt to the large language model , wherein the summary prompt requests to summari ze the subset of responses , thus obtaining a summary response ; and providing the summary response to the user .

[0016] In another implementation form of the first aspect , the large language model is configured to provide the plurality of responses based on a custom knowledge base .

[0017] According to a second aspect , a computing device compri ses at least one processor and at least one memory including computer program code , the at least one memory and the computer program code being configured to , with the at least one proces sor, cause the computingdevice to perform the method according to the first aspect .

[0018] According to a third aspect , a computer program product comprises program code configured to perform the method according to the first aspect when the computer program product is executed on a computer .

[0019] Many of the attendant features wil l be more readily appreciated as they become better understood by reference to the following detailed description considered in connection with the accompanying drawings .DESCRIPTION OF THE DRAWINGS

[0020] In the following, example embodiments are described in more detail with reference to the attached figures and drawings , in which :

[0021] Fig . 1 illustrates a flow chart representation of a method according to an embodiment ;

[0022] Fig . 2 illustrates a flow chart representation of a method according to another embodiment ;

[0023]

[0024] Fig . 3 illustrates a schematic representation data flow according to an embodiment ;

[0025] Fig . 4 illustrates a schematic representation of a plurality of responses according to an embodiment ;

[0026] Fig . 5 illustrates a flow chart representation of a method according to another embodiment ; and

[0027] Fig . 6 illustrates a schematic representation of a computing device according to an embodiment .

[0028] In the following, like reference numerals are used to des ignate li ke parts in the accompanying drawings .DETAILED DESCRIPTION

[0029] In the following description, reference is made to the accompanying drawings , which form part of the disclosure , and in which are shown, by way of illustration, specific aspects in which the present disclosure may be placed . It is understood that other aspects may be utilised, and structural or logical changes may be made without departing from the scope of the present disclosure . The following detailed description, therefore , is not to be taken in a limiting sense , as the scope of the present disclosure is defined by the appended claims .

[0030] For instance , it is understood that a disclosure in connection with a described method may also hold true for a corresponding device or system configured to perform the method and vice versa . For example , if a specific method step is described, a corresponding device may include a unit to perform the described method step, even if such unit is not explicitly described or il lustrated in the f igures . On the other hand, for example , if a specific apparatus is described based on functional units , a corresponding method may include a step performing the described functionality, even if such step is not explicitly described or illustrated in the figures . Further, it is understood that the featuresof the various example aspects described herein may be combined with each other, unless specifically noted otherwise .

[0031] Fig . 1 illustrates a flow chart representation of a method according to an embodiment .

[0032] According to an embodiment , a computer-implemented method 100 for processing a prompt comprises obtaining 101 a prompt .

[0033] Herein a prompt may refer to , for example , a request or a question to be provided to a large language model . The prompt or at least a part of the prompt may be provided, for example , by a user .

[0034] The method 100 may further comprise providing 102 the prompt to a large language model (LLM) a plurality of times , thus obtaining a plurality of responses .

[0035] In some embodiments , the prompt may further comprise other information in addition to the information provided by the user, such as a question . For example , the prompt may indicate what information sources the LLM should use to provide a response to the prompt , such as an answer to the question . For example , it can be advantageous that a chatbot answers questions from users based on a custom knowledge base of a company instead of generic information from the internet . This may also be referred to as grounding or retrieval-augmented generation (RAG) . In other embodiments , such information may be provided to the LLM separately from the prompt .

[0036] In some embodiments , the prompt may comprise a request section and a data section . The request section of the prompt may comprise , for example , a request to process the data provided in the data section in some manner . The data section may comprise any data to be used by the LLM . For example , if the request section comprises a question or some other type or request , the data section can comprise data based on which the LLM should provide the answer .

[0037] In other embodiments , the prompt may comprise , alternatively or additionally, other sections . For example , the prompt may comprise a system section and a user section . The user section may comprise , for example , a question and the system section may comprise instructions to provide an answer to the question using a specific information source , such as specific documents .

[0038] The providing 102 the prompt to the LLM may comprise , for example , providing the prompt to the LLM via a telecommunications network, such as over the internet . The LLM may be , for example , provided as a cloud service . Alternatively, the LLM may be running on a server in a local area network (LAN) or locally, for example on a computing device performing the method 100 .

[0039] The method 100 may further comprise calculating 103 at least one vector space distance based on the plurality of responses .

[0040] Herein, calculating may also be referred to as computing .

[0041] Each response in the plurality of responses can be vectori zed . The vector space representation of a response can represent the semantic content of the response . Each vector representing a response may comprise a plurality of floating-point numbers . Thus , each response can be represented as a vector in a vector space . This may al so be referred to as embedding . Each vector representing a response may be referred to as an embedding vector .

[0042] The vector space distance may comprise any metric that can be used to quantify how similar the responses in the plurality of responses are . For example, in some embodiments , the vector space distance may measure and / or correlate with an angle between vectors representing two responses . In other embodiments , the vector space distance may measure and / or correlate with a distance between the end points of two vectors when the two vectors are arranged to start from the same point .

[0043] Herein, a vector space distance may also be referred to as a vector space similarity, a similarity, a distance , or similar .

[0044] The method 100 may further comprise providing 104 a response to a user based on the plurality of responses and the at least one vector space distance .

[0045] The providing 104 the response to user may comprise , for example , providing the response to the user via a telecommunications network, such as over the internet . The method 100 may be , for example , running as on a server and be provided as a cloud service to theuser . Alternatively, the method 100 may be running on a server in a local area network (LAN) or locally on a device of the user . In such cases , the providing 104 the response to the user may comprise providing the response via LAN and / or via a user interface .

[0046] It should be appreciated that at least some of the operations of the method 100 may be performed in various orders .

[0047] The quality of responses provided by an LLM can be di fficult to assess . The qual ity can be af fected by various factors , such as by a question provided in the prompt , any instructions provided in the prompt , and / or information based on which the response needs to be provided . Further, LLMs are often trained with large amounts of generic data, such as data from the internet . Thus , when an LLM should, for example , answer a question based on a custom knowledge base , the LLM may provide incorrect answers based on the generic training data .

[0048] The method 100 can, for example , enable assessing the quality of responses provided by an LLM .

[0049] The method 100 can, for example , enable the use of LLMs in various applications .

[0050] The method 100 can, for example , reduce the need to manually check responses provided by an LLM .

[0051] Fig . 2 illustrates a flow chart representation of a method according to another embodiment .

[0052] According to an embodiment , the method 100 further comprises , before providing 102 the prompt to thelarge language model the plurality of times , setting 201 a temperature parameter of the large language model .

[0053] The temperature parameter can define the randomness of the responses provided by the LLM . A greater value of the temperature parameter may cause the LLM to provide more diverse responses . A smal ler value of the temperature parameter may cause the LLM to provide less diverse and more deterministic responses .

[0054] According to an embodiment , the setting 201 the temperature parameter of the large language model comprises setting the temperature parameter to be greater than a threshold temperature .

[0055] In some embodiments , the threshold temperature may be , for example , 50 % of a maximum value of the temperature parameter, 70 % of a maximum value of the temperature parameter, 80 % of a maximum value of the temperature parameter, 90 % of a maximum value of the temperature parameter, 95% of a maximum value of the temperature parameter, 97 % of a maximum value of the temperature parameter, 99% of a maximum value of the temperature parameter, or 99 . 5% of a maximum value of the temperature parameter .

[0056] For example , in some embodiments , the setting the temperature parameter of the LLM may comprise setting the temperature parameter to a maximum value of the temperature parameter .

[0057] It should be appreciated that the setting 201 the temperature parameter of the LLM may be performed at any time before providing 102 the prompt to the LLMthe plurality of times . For example , the setting 201 the temperature parameter may be performed before obtaining 101 the prompt or after obtaining 101 the prompt . Similarly, if any other operations are performed before providing 102 the prompt to the LLM the plurality of times , setting 201 the temperature parameter of the LLM may be performed at any time .

[0058] Fig . 3 illustrates a schematic representation data flow according to an embodiment .

[0059] In the embodiment of Fig . 3 , four responses 301 are obtained from the LLM . The responses 301 may have been obtained by providing the same prompt four times to the LLM . Based on the responses , a plurality of vector space distances 302 are calculated . In the notation used in Fig . 3 , "distancexy" corresponds to a vector space distance between response x and response y . For example , distancel2 corresponds to a distance between responsel and response2 , distance23 corresponds to a vector space distance between response2 and responses and so on .

[0060] In the embodiment of Fig . 3 , a vector space distance 302 is calculated for each pair of responses in the plurality of responses 301 . In other embodiments , a different number of vector space distances may be calculated . For example, a vector space may not be calculated for each pair of responses in the plurality of responses 301 .

[0061] According to an embodiment , the calculating the at least one vector space distance based on the plurality of responses comprises calculating the at least one vector space distance using cosine similarity .

[0062] The cosine similarity SC(A, B) between vectors A and B can be calculated aswhere Atand Btare the components of the vectors A and B, respectively .

[0063] In other embodiments , the vector space distance may be calculated using some other distance metric, such a Euclidian distance or a squared Euclidian distance . The Euclidian distance SE(AB) between vectors A and B can be calculated asand the squared Euclidian distanceSE(A, B) as

[0064] According to an embodiment , the calculating the at least one vector space distance based on the plurality of responses comprises : calculating a plurality of vector space distances , wherein each vector space distance in the plurality of vector space distances corresponds to a vector space distance between two responses in the plurality of responses ; and calculating at leastone vector space distance statistic based on the plurality of vector space distances ; wherein the providing the response to the user based on the plurality of responses and the at least one vector space distance comprises providing the response to the user based on the plurality of responses and the at least one vector space distance statistic .

[0065] For example , in the embodiment of Fig . 3 , a vector space distance statistic 303 is calculated based on the plurality of vector space distances 302 . In other embodiments , different numbers of vector space distance statistics can be calculated .

[0066] According to an embodiment , the at least one vector space distance statistic comprises at least one of : a vector space distance mean ; a vector space distance variance ; and / or a vector space distance standard deviation .

[0067] According to an embodiment , the at least one vector space distance statistic comprises at least two of : a vector space distance mean ; a vector space distance variance ; and / or a vector space distance standard deviation .

[0068] According to an embodiment , the at least one vector space distance statistic comprises a vector space distance mean and a vector space distance variance .

[0069] According to an embodiment , the at least one vector space distance statistic comprises a vector space di stance mean and a vector space distance standard deviation .

[0070] According to an embodiment , the providing the response to the user based on the plurality of responses and the at least one vector space distance comprises , in response to the at least one vector space distance statistic not fulfilling at least one preconfigured condition, providing an error response .

[0071] The error response may comprise , for example , an indication that the LLM cannot provide a valid response . For example , if the prompt comprises a question, the error response may compri se an indication that the LLM cannot answer the question . In some cases , at least some of the responses provided by the LLM may comprise such an error response . In some embodiments , even if the LLM does not provide an error response , an error response can be generated when, for example , the LLM provides inconsistent responses in the plurality of responses , which can be detected by the at least one vector space distance statistic not fulfilling the at least one preconfigured condition .

[0072] For example , if the variance and / or standard deviation of the at least one vector space di stance is large , this may indicate that the responses in the plurality of responses are not similar . Thus , it may be concluded that the responses provided by the LLM are not consistent with each other .

[0073] Fig . 4 illustrates a schematic representation of a plurality of responses according to an embodiment .

[0074] According to an embodiment , the providing the response to the user based on the plurality of responsesand the at least one vector space distance comprises choosing at least one response from the plurality of responses based on the at least one vector space distance .

[0075] It should be appreciated that the vector space in which the responses are represented may comprise, for example, hundreds, thousands, tens of thousands, hundreds of thousands, millions, or more dimensions. Thus, the examples illustrated herein only illustrate simplified representations of the responses in a two-dimensional vector space.

[0076] For example, in the embodiment of Fig. 4, response! 301_l, responses 301_3, and response4 301_4 are similar, since the vector space distance between these responses is small, while response2 301_2 is not similar to the other responses since the vector space distance between responses 301_2 and the other responses is large. Thus, responsel 301_l, response3 301_3, and re- sponse4 301_4 likely are semantically close to each other, while responses 301_2 is not. Semantically close responses may also be referred to as coherent. Due to this, responsel 301_l, response3 301_3, and response4 301_4 are more likely to be satisfactory responses to the prompt. Based on this, one of the responses responsel 301_l, response3 301_3, response4 301_4, and / or a response based on these responses may be provided to the user.

[0077] According to an embodiment, the providing the response to the user based on the plurality of responsesand the at least one vector space distance comprises : selecting a subset of responses from the plurality of responses based on the at least one vector space distance and providing the response to the user based on the subset of responses .

[0078] The subset of responses may represent similar responses in the plurality of responses .

[0079] For example , a threshold vector space distance can be preconfigured . The subset of responses may be chosen such that the vector space distance between each response pair in the subset of responses is less than the threshold vector space distance . For example , in the embodiment of Fig . 4 , responsel 301_l , response3 301_3 , and response4 301_4 may form the subset of responses 401 .

[0080] In other embodiments , the subset of responses may be selected in some other way, such as using a clustering algorithm .

[0081] In some embodiments , the subset of responses can be selected by removing outl ier responses from the plurality of responses . The outlier responses can comprise , for example , responses that have a large vector space distance , such as greater than the threshold vector space distance , to other responses , such as response! 301_2 in the embodiment of Fig . 4 , and / or special case responses . A special case response may comprise , for example , an error response provided by the LLM, such as a response in which the LLM indicates thatit cannot provide an answer to the question presented in the prompt .

[0082] According to an embodiment , the providing the response to the user based on the subset of responses comprises : providing a summary prompt to the large language model , wherein the summary prompt requests to summari ze the subset of responses , thus obtaining a summary response ; and providing the summary response to the user .

[0083] The summary prompt may comprise , for example , in a data section, the subset of response , and, in a request section, a request to summari ze the subset of responses .

[0084] In other embodiments , the providing the response to the user based on the subset of responses may comprise , for example : selecting a response from the subset of responses and providing the selected response to the user . The selection can be performed, for example , randomly or based on preconfigured criteria .

[0085] According to an embodiment , the providing the response to the user based on the plurality of responses and the at least one vector space distance comprises , in response to the at least one vector space distance being greater than a threshold vector space distance , providing an error response .

[0086] For example , in some embodiments , if one or more of the plurality of responses is far away from the other responses in the plurality of responses , an error response can be provided . For example, the vector spacedistance between a response in the plurality of responses and other responses in the plurality of responses may be greater than the threshold vector space distance .

[0087] For example , in the embodiment of Fig . 2 , the vector space distance between response2 301_2 and any of the responses responsel 301_l , responses 301_3 , re- sponse4 301_4 may be greater than the threshold vector space distance .

[0088] In some embodiments , at least one vector space distance statistic can be calculated based on the subset of responses . For example , if only a small number of responses , such as less than a preconfigured threshold number, are different from the subset of responses , the at least one vector space distance statistic can be calculated based on the subset of responses . For example , in the embodiment of Fig . 4 , the at least one vector space di stance stati stic can be calculate based on the subset of responses 401 , since only response2 301_2 is different from the other responses . The response different from the other responses may comprise , for example , an error response provided by the LLM .

[0089] Fig . 5 illustrates a flow chart representation of a method according to another embodiment .

[0090] According to an embodiment , the large language model is configured to provide the plurality of responses based on a custom knowledge base .

[0091] After obtaining 101 the prompt , the method 100 may comprise obtaining 501 information from a knowledgebase . The information may comprise , for example , any data based on which the LLM should provide the response to the prompt .

[0092] Herein a knowledge base may comprise any source of information that can be provided to the LLM for the LLM to answer questions based on information in the knowledge based . A custom knowledge base may comprise information that should be used by the LLM to answer questions provided to the LLM in prompts . For example , the custom knowledge base may comprise documentation of a company based on which the LLM should answer question from users .

[0093] In operation 201 , the temperature parameter of the LLM can be set and then the prompt can be provided 102 to the LLM a plurality of times .

[0094] It should be appreciated that various operations di sclosed herein can be performed in various orders . For example , in the embodiment of Fig . 5 , operations 101 , 501 , and 201 can be performed in any order at least in some embodiments . In some embodiments , the information can be obtained 501 based on the prompt that is provided in operation 101 . In such embodiments , the prompt may need to be obtained 101 before obtaining 501 the information .

[0095] In operation 502 , each response in the plurality of responses can be vectori zed . In operation 103 , at least one vector space distance can be calculated .

[0096] In operation 503 , at least one vector space distance statistic can be calculated .

[0097] In operation 504, it can be checked whether the at least one vector space distance statistic fulfils at least one preconfigured condition. If the at least one condition is fulfilled, the procedure can move to providing 102 the response to the user.

[0098] If the at least one preconfigured condition is not fulfilled, the procedure can move to checking 505 whether further tries are left. If no further tries are left, the procedure can move to providing 506 an error response to the user. If further tries are left, the procedure can move to operation 501 so that the prompt can be provided a plurality of times to the LLM again.

[0099] The number of tries in the embodiment of Fig. 5 may be preconfigured. Each time the operations 501, 201, 102, 502, 103, 503 are performed, the number of tries left can decremented by one.

[0100] The embodiment of Fig. 5 only illustrates one possible implementation of at least some parts of the method 100. The method 100 can also be implemented in various other ways.

[0101] Fig. 6 illustrates a schematic representation of a computing device according to an embodiment.

[0102] According to an embodiment, a computing device 600 comprises at least one processor 601 and at least one memory 602 including computer program code, the at least one memory 602 and the computer program code configured to, with the at least one processor 601, cause the computing device 600 to perform the method 100.

[0103] The computing device 600 may comprise at least one processor 601. The at least one processor 601 may comprise, for example, one or more of various processing devices, such as a co-processor, a microprocessor, a digital signal processor (DSP) , a processing circuitry with or without an accompanying DSP, or various other processing devices including integrated circuits such as, for example, an application specific integrated circuit (ASIC) , a field programmable gate array (FPGA) , a microprocessor unit (MCU) , a hardware accelerator, a special-purpose computer chip, or the like.

[0104] The computing device 600 may further comprise a memory 602. The memory 602 may be configured to store, for example, computer programs and the like. The memory 602 may comprise one or more volatile memory devices, one or more non-volatile memory devices, and / or a combination of one or more volatile memory devices and nonvolatile memory devices. For example, the memory 602 may be embodied as magnetic storage devices (such as hard disk drives, magnetic tapes, etc.) , optical magnetic storage devices, and semiconductor memories (such as mask ROM, PROM (programmable ROM) , EPROM (erasable PROM) , flash ROM, RAM (random access memory) , etc.) .

[0105] The computing device 600 may further comprise other components not illustrated in the embodiment of Fig. 6. The computing device 600 may comprise, for example, an input / output bus for connecting the computing device 600 to other devices. Further, a user may control the computing device 600 via the input / output bus.

[0106] When the computing device 600 is configured to implement some functionality, some component and / or components of the computing device 600 , such as the at least one processor 601 and / or the memory 602 , may be configured to implement this functionality . Furthermore , when the at least one processor 601 is configured to implement some functionality, this functionality may be implemented using program code comprised, for example , in the memory .

[0107] The computing device 600 may be implemented at least partially using, for example , a computer, some other computing device , or similar .

[0108] The method 100 and / or the computing device 600 may be utilised in, for example , chatbot applications . For example , a chatbot may be configured to answer questions provided by users via a chat user interface . The chatbot can answer the questions by using an LLM via the method 100 and assess the qual ity of the answers provided by the LLM . The chatbot may be, for example , running on a server of a company and be provided as a service to the clients of the company . The LLM may be running, for example , a third party server provided as a service to the company .

[0109] The method 100 and / or the computing device 600 may be utilised in, for example, voicebot applications . A voicebot may be configured to obtain information from users by, for example , phone and convert the voice information into text information using automatic speech recognition (ASR) . The text information may be used asprompts for the method 100 and / or the computing device 600 . Responses obtained from the LLM can then be converted to speech using speech syntheti zation and the synthesi zed speech can be provided to the user . Thus , the voicebot can function similarly to a chatbot as disclosed above but using audio instead of a chat interface . The voicebot can answer the questions by using an LLM via the method 100 and assess the quality of the answers provided by the LLM . The voicebot may be , for example , running on a server of a company and be provided as a service to the clients of the company . The LLM may be running, for example , a third party server provided as a service to the company .

[0110] Any range or device value given herein may be extended or altered without losing the effect sought . Also any embodiment may be combined with another embodiment unless explicitly disallowed .[01 1 1 ] Although the subj ect matter has been described in language specific to structural features and / or acts , it is to be understood that the subj ect matter defined in the appended claims is not necessarily limited to the specific features or acts described above . Rather, the specific features and acts described above are disclosed as examples of implementing the claims and other equivalent features and acts are intended to be within the scope of the claims .

[0112] It will be understood that the benefits and advantages described above may relate to one embodiment or may relate to several embodiments . The embodimentsare not limited to those that solve any or all of the stated problems or those that have any or all of the stated benefits and advantages . It wil l further be understood that reference to ' an ' item may refer to one or more of those items .[01 1 3] The steps of the methods described herein may be carried out in any suitable order, or simultaneously where appropriate . Additionally, individual blocks may be deleted from any of the methods without departing from the spirit and scope of the subj ect matter described herein . Aspects of any of the embodiments described above may be combined with aspects of any of the other embodiments described to form further embodiments without losing the effect sought .

[0114] The term ' comprising ' is used herein to mean including the method, blocks or elements identified, but that such blocks or elements do not comprise an exclusive list and a method or apparatus may contain additional blocks or elements .[01 1 5] It will be understood that the above description is given by way of example only and that various modif ications may be made by those ski lled in the art . The above specification, examples and data provide a complete description of the structure and use of exemplary embodiments . Although various embodiments have been described above with a certain degree of particularity, or with reference to one or more individual embodiments , those skilled in the art could make numerous alterations to the disclosed embodiments withoutdeparting from the spirit or scope of this specification .

Claims

CLAIMS :

1. A computer-implemented method (100) for processing a prompt, the method comprising: obtaining (101) a prompt; providing (102) the prompt to a large language model a plurality of times, thus obtaining a plurality of responses; calculating (103) at least one vector space distance based on the plurality of responses; and providing (104) a response to a user based on the plurality of responses and the at least one vector space distance by choosing at least one response from the plurality of responses based on the at least one vector space distance.

2. The computer-implemented method (100) according to claim 1, wherein the method (100) further comprises, before providing the prompt to the large language model the plurality of times, setting (201) a temperature parameter of the large language model.

3. The computer-implemented method (100) according to claim 2, wherein the setting the temperature parameter of the large language model comprises setting the temperature parameter to be greater than a threshold temperature .

4. The computer-implemented method (100) according to any preceding claim, wherein the calculatingthe at least one vector space distance based on the plurality of responses comprises calculating the at least one vector space distance using cosine similarity .5 . The computer-implemented method ( 100 ) according to any preceding claim, wherein the calculating the at least one vector space distance based on the plurality of responses comprises : calculating a plurality of vector space distances , wherein each vector space distance in the plurality of vector space distances corresponds to a vector space distance between two responses in the plurality of responses ; and calculating at least one vector space distance statistic based on the plurality of vector space distances ; wherein the providing the response to the user based on the plurality of responses and the at least one vector space distance comprises providing the response to the user based on the plurality of responses and the at least one vector space distance statistic .6 . The computer-implemented method ( 100 ) according to claim 5 , wherein the at least one vector space distance statistic comprises at least one of : a vector space distance mean ; a vector space distance variance ; and / or a vector space distance standard deviation .7 . The computer-implemented method ( 100 ) according to claim 5 or claim 6 , wherein the providing ( 104 ) the response to the user based on the plurality of responses and the at least one vector space distance comprises , in response to the at least one vector space distance statistic not fulfilling at least one preconfigured condition, providing an error response .8 . The computer-implemented method ( 100 ) according to any preceding claim, wherein the providing the response to the user based on the plurality of responses and the at least one vector space distance comprises , in response to the at least one vector space distance being greater than a threshold vector space distance , providing an error response .9 . The computer-implemented method ( 100 ) according to any preceding claim, wherein the providing the response to the user based on the plurality of responses and the at least one vector space distance comprises : selecting a subset of responses from the plurality of responses based on the at least one vector space distance ; and providing the response to the user based on the subset of responses .

10. The computer-implemented method (100) according to claim 9, wherein the providing the response to the user based on the subset of responses comprises: providing a summary prompt to the large language model, wherein the summary prompt requests to summarize the subset of responses, thus obtaining a summary response; and providing the summary response to the user.

11. The computer-implemented method (100) according to any preceding claim, wherein the large language model is configured to provide the plurality of responses based on a custom knowledge base.

12. A computing device (600) , comprising at least one processor (601) and at least one memory (602) including computer program code, the at least one memory (602) and the computer program code configured to, with the at least one processor (601) , cause the computing device (600) to perform the method (100) according to any preceding claim.

13. A computer program product comprising program code configured to perform the method (100) according to any of claims 1 - 11 when the computer program product is executed on a computer.

Images