Cached response

Abstract

According to an example aspect of the present invention, there is provided a method comprising receiving a query, determining, distance values between the received query and each one of plural cached queries wherein a first distance value, between the received query and a first cached query, is the smallest distance value among the determined distance values, responsive to the first distance value being smaller than a first threshold, providing a cached response to the first cached query as a final response to the received query, responsive to the first distance value being larger than the first threshold and smaller than a second threshold, providing the cached response to the first cached query as a provisional response to the received query, and responsive to the first distance value being larger than the second threshold, obtaining the result to the received query and providing it as a final response to the query.

Description

CACHED RESPONSEFIELD

[0001] The present disclosure relates to the field of computer systems, for example large language model computer systems.BACKGROUND

[0002] Automated systems may be configured to return a deliverable as a response to a task assigned to the automated system. For example, a large language model, or another kind of automated system, may call on plural automated tools when preparing the deliverable. Automated systems may include automated configuration generators, such as for a cellular telecommunication system or part thereof, for an industrial site, for a vehicle or for a vehicle fleet.SUMMARY

[0003] According to some aspects, there is provided the subject-matter of the independent claims. Some embodiments are defined in the dependent claims.

[0004] According to a first aspect of the present disclosure, there is provided a computer-implemented method comprising receiving a query, determining, using a predetermined metric, distance values between the received query and each one of plural cached queries wherein a first distance value, between the received query and a first cached query, is the smallest distance value among the determined distance values, responsive to the first distance value being smaller than a first threshold, providing a cached response to the first cached query as a final response to the received query, responsive to the first distance value being larger than the first threshold and smaller than a second threshold, providing the cached response to the first cached query as a provisional response to the received query, and responsive to the first distance value being larger than the second threshold, obtaining the result to the received query and providing the obtained result as a final response to the received query.

[0005] According to a second aspect of the present disclosure, there is provided an apparatus comprising means for receiving a query, means for determining, using a predetermined metric, distance values between the received query and each one of plural cached queries wherein a first distance value, between the received query and a first cached query, is the smallest distance value among the determined distance values, means for, responsive to the first distance value being smaller than a first threshold, providing a cached response to the first cached query as a final response to the received query, means for, responsive to the first distance value being larger than the first threshold and smaller than a second threshold, providing the cached response to the first cached query as a provisional response to the received query, and means for, responsive to the first distance value being larger than the second threshold, obtaining the result to the received query and providing the obtained result as a final response to the received query.

[0006] According to a third aspect of the present disclosure, there is provided a non- transitory computer readable medium having stored thereon a set of computer readable instructions that, when executed by at least one processor, cause an apparatus to at least receive a query, determine, using a predetermined metric, distance values between the received query and each one of plural cached queries wherein a first distance value, between the received query and a first cached query, is the smallest distance value among the determined distance values, responsive to the first distance value being smaller than a first threshold, provide a cached response to the first cached query as a final response to the received query, responsive to the first distance value being larger than the first threshold and smaller than a second threshold, provide the cached response to the first cached query as a provisional response to the received query, and responsive to the first distance value being larger than the second threshold, obtain the result to the received query and provide the obtained result as a final response to the received query.

[0007] According to a fourth aspect of the present disclosure, there is provided an apparatus comprising at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to at least perform: receiving a query, determining, using a predetermined metric, distance values between the received query and each one of plural cached queries wherein a first distance value, between the received query and a first cached query, is the smallest distance value among the determined distance values, responsive to the first distance value being smaller than a first threshold, providing a cached response to the first cached query as a final response to thereceived query, responsive to the first distance value being larger than the first threshold and smaller than a second threshold, providing the cached response to the first cached query as a provisional response to the received query, and responsive to the first distance value being larger than the second threshold, obtaining the result to the received query and providing the obtained result as a final response to the received query.

[0008] According to a fifth aspect of the present disclosure, there is provided an apparatus comprising at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to at least send a query to an automated computer tool, determine whether a response to the query, received in the apparatus from the automated computer tool, is a provisional response or a final response, use the response in preparation of a deliverable responsive to determining the response is the final response, responsive to determining the response is the provisional response, determine whether the provisional response is usable in preparation of the deliverable, responsive to the provisional response being usable in preparation of the deliverable, use the provisional response in preparing the deliverable before receiving the final response from the automated computer tool, responsive to the provisional response not being usable in preparation of the deliverable, wait for the final response from the automated computer tool, and provide the deliverable as a response to a task assigned to the apparatus.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] FIGURE 1 illustrates an example system in accordance with at least some embodiments of the present invention;

[0010] FIGURE 2 is an example signalling diagram according to at least some embodiments of the present invention;

[0011] FIGURE 3 illustrates an example apparatus capable of supporting at least some embodiments of the present invention;

[0012] FIGURE 4 illustrates a process diagram in accordance with at least some embodiments of the present invention;

[0013] FIGURE 5 illustrates a process diagram in accordance with at least some embodiments of the present invention;

[0014] FIGURE 6 is a flow graph of a method in accordance with at least some embodiments of the present invention, and

[0015] FIGURE 7 is a flow graph of a method in accordance with at least some embodiments of the present invention.EMBODIMENTS

[0016] Disclosed herein are methods to reduce delays in automated systems, such that actions may be taken sooner than without using the herein disclosed methods. In detail, an automated tool is configured to provide, as a response to a query it receives from the automated system, a cached response to a cached query which has been generated for the earlier cached query. The cached response is provided as either a provisional response or a final response, depending on how closely the received query matches the cached query. In case it is provided as a provisional response, the automated tool may generate a separate final response by processing the received query, and sending the final response after sending the provisional response. The automated system may be able to take steps to prepare its deliverable already based on the provisional response, reducing the overall time it takes for the automates system to generate the deliverable. For example, where the automated system is a configuration generator for a cellular telecommunication system, a re-configuration may thus be initiated sooner as a response to a fault condition, using a provisional configuration update, enhancing the dependability of the underlying cellular telecommunication system.

[0017] FIGURE 1 illustrates an example system in accordance with at least some embodiments of the present invention. Automated system 110, which may comprise, for example, a large language model, LLM, small language model, SLM, or an expert system, receives a task A from a calling function 101. Automated system 110 may be based on LangChain, Haystack or Semantic Kernel architecture, for example. Task A may comprise, for example, a request to prepare a configuration update in a cellular communication system to respond to a fault condition, or task A may comprise a sequence of error messages received from the cellular communication system or other underlying system. Alternatively to the cellular communication system, an industrial automation process, vehicle, vehicle fleet, cellular user equipment or group of cellular user equipments may be the underlying system the task relates to, for example in terms of devising a configuration update to theunderlying system as a response to error indications in the task. The calling function may be a configuration function or error management function of the underlying system, for example.

[0018] When the underlying system is the cellular communication system, a configuration update may comprise a set of new parameter values to be taken into use in a set of base stations of the cellular communication system and / or a core network of the cellular communication system, for example. Examples of such parameters include frequency bands in use, discontinuous reception or transmission patterns and beamforming parameters.

[0019] Automated system 110 prepares a deliverable J as a response to task A. The contents of deliverable J are dependent on the configuration of automated system 110 and the content of task A. For example, task A may comprise descriptors of fault indications received from the underlying system, or a desired configuration outcome for the underlying system, and deliverable J may comprise a set of configuration parameters for the underlying system. The underlying system is a system that task A relates to.

[0020] A further example of the underlying system is an industrial process, such as a paper mill, where the query may comprise, or be based on, an error signal or alert based on, or originating in, a sensor device of the industrial process. The automated system prepares as the deliverable a decision to trigger measurements to increase the situational awareness of personnel. The measurements may be automatically performed as a response to the trigger, and / or the decision to trigger may be presented to the personnel on a display, or an audible signal.

[0021] A yet further example of the underlying system is an electric vehicle. The query may in this case take the form of data including one or more of a driver’s calendar, the driver’s email messages, weather forecasts and car drive batter charge level. The deliverable in this case comprises a drive battery charging schedule which aims to maintain a sufficient and not excessive charge in the drive battery, as an almost-fully charged state, in excess of 80% of maximum charge, is straining to the drive battery. The automated system calls on the driver’s calendar and email system to discover future driving needs, and on a weather forecast to discover a predicted temperature, as driving in colder temperatures is more draining on the battery. Based on these information, the automated system, such as an LLM, for example, may prepare as the deliverable the charging schedule, to maximize drive batterylife and availability of the electric vehicle, such as an electric car. The automated system may re-generate the deliverable, for example hourly or at another fixed time periodicity, to adapt to changes and updates in the driver’s situation.

[0022] Automated system 110, in preparing deliverable J, is configured to call on one or more automated tools 122, 124, 126, 128. Which automated tools are called on may depend on the nature of task A, for example. Calling on an automated tool comprises transmitting a query to this tool. Automated tools 122, 124, 126, 128 may comprise, for example, tools configured to retrieve information from sources external to automated system 110, such as data repositories. Examples of data repositories include status information repositories of devices or nodes of the underlying system. Additionally or alternatively, automated tools 122, 124, 126, 128 may be configured to generate data usable by automated system 110 using a simulation or other complex mathematical process. Yet further, additionally or alternatively at least one of automated tools 122, 124, 126, 128 may be a web search engine that automated system 110 queries with a search string generated based on task A and / or a result from an earlier query to another one of the automated tools 122, 124, 126, 128. Alternatively or additionally to a web search engine, one or more of the automated tools 122, 124, 126, 128 may be configured to perform a query-based search in a document repository which is not publicly accessible, such as a corporate repository.

[0023] In the example of FIGURE 1, automated system 110 calls four automated tools 122, 124, 126, 128. In detail, query B to automated tool 122 results in response C from automated tool 122, query D to automated tool 124 results in response E from automated tool 124, query F to automated tool 126 results in response G from automated tool 126 and query H to automated tool 128 results in response I from automated tool 128. In general, automated system 110 may determine which automated tools to call, and in which order, based on task A.

[0024] As automated tools 122, 124, 126, 128 may each need some time to return their respective responses to automated system 110, and especially if queries to these tools need to be performed in sequence such that a response from an earlier automated tool is needed to compile a query to a subsequent one of the automated tools, overall the generation of deliverable J will take some time. In one extreme case, query H needs as input data from response G, query F needs as input data from response E, and query D needs as input data from response C. As deliverable J is a technical deliverable comprising, for example,configuration data applicable in the underlying system which may have a fault condition, it would be beneficial to obtain deliverable J sooner rather than later. In some cases, automated system 110 calls a same automated tool 122, 124, 126, 128 more than once in sequence, for example if a web search initially does not return results that automated system 110 can use in preparing deliverable J, automated system 110 may call the same search tool again using a modified search string. Further, individual ones of automated tools 122, 124, 126, 128 may be highly loaded, increasing their response time.

[0025] To facilitate a faster generation of deliverable J, at least one of automated tools 122, 124, 126, 128 is configured to ascertain whether it can respond to a query received from automated system 110 using a cached response, generated as a response to an earlier query. The automated tool may thus store earlier queries and responses generated to serve these earlier queries.

[0026] Determining whether a response to a cached query is usable in serving a new query is performed using a predetermined metric. In detail, the automated tool determines, using the predetermined metric, distance values between the newly received query and each one of the cached queries. Inevitably, one of the distance values will be smallest, indicating the cached query which is closest, in terms of the predetermined metric, to the received query. Herein the distance value between the received query and a closest cached query will be referred to as a first distance value. In case two cached queries have the same, smallest, distance value, these may both be treated as the closest cached query. Optionally, instead of determining the distance values to all the cached queries, the automated tool may determine distance values only to cached queries which are less than a threshold age old. The threshold age may be provided to the automated tool in the received query, or the automated tool may select the threshold age based on the received query.

[0027] The first distance value is then compared to thresholds. Responsive to the first distance value being smaller than a first threshold, the automated tool provides to the automated system the cached response to the closest cached query as a response to the received query. This is a final response, and processing of the received query will then stop in the automated tool. The response may comprise an indicator indicating that the response is final. The automated tool may provide, along with the cached response, also the corresponding cached query to enable automated system 110 to determine, how relevant the response is to preparing deliverable J.

[0028] If the first distance value is larger than the first threshold and smaller than a second threshold, the automated tool provides the cached response to the closest cached query as a provisional response to the received query, obtains a result to the received query and then provides the obtained result to the system as a final response. The provisional response may comprise an indicator indicating that the response is provisional in nature. The calling automated system will thus get the provisional response before it gets the final response, potentially allowing the calling system to act on the provisional response before it gets the final response. Again, the automated tool may provide, along with the cached response, the cached query which was earlier used to generate the cached response.

[0029] In case the first distance value is larger than the second threshold, the automated tool obtains a result to the received query and provides the obtained result to the system normally, without using the cached responses. In this latter case, the automated tool doesn’t have a usefully close cached response. The automated tool may cache the received query and its final response among the cached queries and responses.

[0030] The predetermined metric may be based on time instants specified in queries, such that the distance values obtained using the predetermined metric are greater when the time instants specified in the queries are further from each other in time and the distance values obtained using the predetermined metric are smaller when the time instants specified in the queries are closer to each other in time. For example, a query arriving very soon after an earlier query may thus be associated, in terms of the predetermined metric, with the earlier query owing to the small difference in transmission timestamps of the queries. The relevant time, however, may alternatively be a different time than the transmission timestamp of the query, for example the queries may be directed to specific past instants in time, and these past instants may be close to each other resulting in a closeness between the queries, according to the predetermined metric.

[0031] The predetermined metric may be based on geographical separation, such that the distance values obtained using the predetermined metric are greater when geographical locations specified in the queries are further from each other and the distance values obtained using the predetermined metric are smaller when the geographical locations specified in the queries are closer to each other. For example, a query relating to meteorological conditions at a cell site may be close, in terms of the predetermined metric, to another query which relates to meteorological conditions of the same, or a neighbouring, cell site.

[0032] The predetermined metric may be based on semantic distance, such that the distance values obtained using the predetermined metric are greater when the queries are further from each other in a semantic space and the distance values obtained using the predetermined metric are smaller when the queries are closer to each other in the semantic space. For example, the semantic space may be a semantic vector space, which may be utilized in the determining of the semantic distance. Parameters of the received query, such as a web search query, may be transformed into a vector. This may be accomplished using e.g. the Microsoft Azure OpenAI interface, using the embeddings feature therein. The vectorized form of the received request may then be compared to vectorized forms of cached queries to determine the distance values using, for example, a distance between vector endpoints or similarity, such as a cosine similarity.

[0033] The predetermined metric may be based on numerical parameters, such that the distance values obtained using the predetermined metric are greater when values of corresponding numerical parameters specified in the queries are further from each other and the distance values obtained using the predetermined metric are smaller when the values of the corresponding numerical parameters specified in the queries are closer to each other. When comparing numerical values, numerical values in distinct queries are compared to corresponding numerical values, that is, e.g. temperature values are compared to temperature values or pressure values to pressure values.

[0034] The automated tool may be configured to determine each one of the distance values as an absolute value of a distance vector, each component of the distance vector being obtained using a specific property of the queries. Examples of such properties are described above, including time instants, geographical separation, semantic distance and numerical parameters. Thus one component of the distance vector could be, for example, the distance between the vectorized form of the received query and a vectorized form of a cached query and another component of the distance vector could be a difference in time instants specified in the received and cached query.

[0035] The automated tool may obtain the first threshold and / or the second threshold from the received query. Likewise, the automated tool may obtain the predetermined metric from the received query. This way the automated system, which calls the tool, may control which kind of cached queries the tool may use in handling the new query. In particular, if the calling system knows that an aspect of the new query is central to preparing deliverable J, the calling system may send, along with or in the new query, to the automated tool apredetermined metric where the role of this aspect of the queries is dominant, eliminating the possibility that the automated tool would send as a provisional response a response to a cached query which significantly deviates from the new query as to this dominant aspect. In other words, the automated system may select the predetermined metric based on one or more characteristics of deliverable J, or task A. In detail, automated system 110 may select a predetermined metric to send to the automated tool which is more responsive than a default predetermined metric to differences between queries in a property defined in task A.

[0036] In receipt of a provisional response, automated system 110 may determine whether it can use the provisional response in preparing deliverable J, for example based on types of other responses it has received. For example, if most responses automated system 110 has received up to that point are final in nature, then automated system 110 may proceed based on the provisional response without waiting for, or requesting, a final-type response. For example, the responses may originate in sensors measuring a same property using different measuring principles. If two of three responses, for example, are final in nature, automated system 110 may proceed without waiting for a final response from the third automated tool.

[0037] As another example, the query may define conditions under which a provisional response is usable in preparing deliverable J. For example, the query may specify that provisional responses from specific automated tools cannot be used, but that provisional responses from other, specified automated tools, or specified types of automated tools, may be used in preparing deliverable J. In the case of the electric vehicle charging solution, the query may specify that weather forecasts which are older than a threshold age are provisional and that the automated system should wait for an updated weather forecast before finalizing the deliverable J.

[0038] Deliverable J is provided from automated system 110 to receiving function 130. Receiving function 130 may be the same function as calling function 101, or a different one. For example, receiving function 130 may be a function configured to disseminate configuration data obtained from automated system 110 in the form of deliverable J to the underlying system.

[0039] FIGURE 2 is an example signalling diagram according to at least some embodiments of the present invention. The vertical axes correspond to structure described in connection to FIGURE 1 , like numbering denoting like structure. Time advances from the top toward the bottom.

[0040] In phase 210, calling function 101 sends task A to automated system 110. Automated system 110 begins processing task A, and as part of this processing automated system 110 sends, phase 220, query B to automated tool 122. Automated tool 122 determines that query B is close enough to a query cached in automated tool 122 to enable a provisional response 230 being sent to automated system 110. The provisional response may comprise an indication that it is provisional in nature, and it may likewise comprise the cached earlier query which resulted in the cached response. In some embodiments, the presence of the cached query in the provisional response is itself the indication that the response is provisional in nature.

[0041] In receipt of provisional response 230, automated system 110 determines that it can use provisional response 230 in preparing deliverable J, in detail, it may use information from provisional response 230 in compiling query D to automated tool 124, phase 240. For example, automated system 110 may perform this determination based on the cached query present in provisional response 230. Automated tool 124 determines that it can respond to query D using a provisional response 250, which may, as described herein above, comprise the cached query used in generating it. In the example of FIGURE 2, however, automated system 110 determines, for example based on the cached query present in provisional response 250, that it cannot use this provisional response in preparing deliverable J.

[0042] Automated system thus waits for automated tool 124 to send the final response E, phase 270, to query D of phase 240. In the meanwhile, also the final response C to query B is received in automated system 110, in phase 260. The final response of phase 260 need not be acted on, as query D, phase 240, was already performed based on provisional response 230. Automated system 110 may then proceed to send query F to automated tool 126, phase 280, and query H to automated tool 128, phase 290. These queries may, in this example, be sent without waiting for the other to complete, since they are not dependent on each other. Responses G and I are received, respectively, from automated tools 126 and 128, respectively, in phases 2100 and 2110.

[0043] Finally, deliverable J is provided, phase 2120, from automated system 110 to receiving function 130. Use of provisional response 230 accelerates all subsequent phases of the signalling diagram of FIGURE 2, providing a technical effect of accelerated processing.

[0044] FIGURE 3 illustrates an example apparatus capable of supporting at least some embodiments of the present invention. Illustrated is device 300, which may comprise, for example, a computing substrate configured to perform as an automated system 110 or as an automated tool as herein described. Comprised in device 300 is processor 310, which may comprise, for example, a single- or multi-core processor wherein a single-core processor comprises one processing core and a multi-core processor comprises more than one processing core. Processor 310 may comprise, in general, a control device. Processor 310 may comprise more than one processor. When processor 310 comprises more than one processor, device 300 may be a distributed device wherein processing of tasks takes place in more than one physical unit. Processor 310 may be a control device. A processing core may comprise, for example, a Cortex- A8 processing core manufactured by ARM Holdings or a Zen processing core designed by Advanced Micro Devices Corporation. A processing core or processor may be, or may comprise, at least one qubit. Processor 310 may comprise at least one Qualcomm Snapdragon and / or Intel Atom processor. Processor 310 may comprise at least one application-specific integrated circuit, ASIC. Processor 310 may comprise at least one field-programmable gate array, FPGA. Processor 310, optionally together with memory and computer instructions, may be means for performing method steps in device 300, such as receiving, determining, providing, obtaining, sending, using and waiting. Processor 310 may be configured, at least in part by computer instructions, to perform actions.

[0045] Device 300 may comprise memory 320. Memory 320 may comprise randomaccess memory and / or permanent memory. Memory 320 may comprise at least one RAM chip. Memory 320 may be a computer readable medium. Memory 320 may comprise solid- state, magnetic, optical and / or holographic memory, for example. Memory 320 may be at least in part accessible to processor 310. Memory 320 may be at least in part comprised in processor 310. Memory 320 may be means for storing information. Memory 320 may comprise computer instructions that processor 310 is configured to execute. When computer instructions configured to cause processor 310 to perform certain actions are stored in memory 320, and device 300 overall is configured to run under the direction of processor 310 using computer instructions from memory 320, processor 310 and / or its at least one processing core may be considered to be configured to perform said certain actions. Memory 320 may be at least in part external to device 300 but accessible to device 300. Memory 320 may be transitory or non-transitory. The term “non-transitory”, as used herein, is a limitationof the medium itself (that is, tangible, not a signal) as opposed to a limitation on data storage persistency (for example, RAM vs. ROM).

[0046] Device 300 may comprise a transmitter 330. Device 300 may comprise a receiver 340. Transmitter 330 and receiver 340 may be configured to transmit and receive, respectively, information in accordance with at least one cellular or non-cellular standard. Transmitter 330 may comprise more than one transmitter. Receiver 340 may comprise more than one receiver. Transmitter 330 and / or receiver 340 may be configured to operate in accordance with a suitable communication standard.

[0047] Device 300 may comprise user interface, UI, 360. UI 360 may comprise at least one of a display, a keyboard, a touchscreen, a vibrator arranged to signal to a user by causing device 300 to vibrate, a speaker or a microphone. A user may be able to operate device 300 via UI 360, for example to input rules for generating configurations.

[0048] Processor 310 may be furnished with a transmitter arranged to output information from processor 310, via electrical leads internal to device 300, to other devices comprised in device 300. Such a transmitter may comprise a serial bus transmitter arranged to, for example, output information via at least one electrical lead to memory 320 for storage therein. Alternatively to a serial bus, the transmitter may comprise a parallel bus transmitter. Likewise processor 310 may comprise a receiver arranged to receive information in processor 310, via electrical leads internal to device 300, from other devices comprised in device 300. Such a receiver may comprise a serial bus receiver arranged to, for example, receive information via at least one electrical lead from receiver 340 for processing in processor 310. Alternatively to a serial bus, the receiver may comprise a parallel bus receiver.

[0049] Device 300 may comprise further devices not illustrated in FIGURE 3. For example, where device 300 comprises a smartphone, it may comprise at least one digital camera. Some devices 300 may comprise a back- facing camera and a front-facing camera, wherein the back-facing camera may be intended for digital photography and the frontfacing camera for video telephony. Device 300 may comprise a fingerprint sensor arranged to authenticate, at least in part, a user of device 300. In some embodiments, device 300 lacks at least one device described above.

[0050] Processor 310, memory 320, transmitter 330, receiver 340 and / or UI 360 may be interconnected by electrical leads internal to device 300 in a multitude of different ways. For example, each of the aforementioned devices may be separately connected to a master bus internal to device 300, to allow for the devices to exchange information. However, as the skilled person will appreciate, this is only one example and depending on the embodiment various ways of interconnecting at least two of the aforementioned devices may be selected without departing from the scope of the present invention.

[0051] FIGURE 4 illustrates a process diagram in accordance with at least some embodiments of the present invention. The illustrated process is performed in one of the automated tools. In phase 410, a new query is received in the automated tool and processing advances to phase 420. In phase 420, the automated tool determines which one of the earlier, cached queries it has processed is closest, in terms of the predetermined metric, to the query received in phase 410. Processing advances to phase 430, where it is determined if the distance between the query received in phase 410 and the closest cached query, that being the first distance as discussed above, is smaller than the first threshold. If this is the case, Y, processing advances to phase 440 where the cached response to the cached closest query is returned to the calling system as a final response to the query received in phase 410. The final response may comprise an indication that it is a final response. Processing optionally advances to phase 450, where the new query is stored in the cache, along with a copy of the response sent in phase 440. When processing does not advance from phase 440 to phase 450, processing ends in phase 440.

[0052] If the first distance is not smaller than the first threshold, processing advances from phase 430 to phase 460, where it is determined whether the first distance is smaller than the second threshold. If this is the case, Y, processing advances from phase 460 to phase 470, where the cached response to the closest cached query is returned to the calling system as a provisional response. As described herein above, the provisional response may comprise an indicator that the response is of provisional type and / or the closest cached query. Processing advances from phase 470 to phase 480, where the automated tool processes the query received in phase 410 to obtain a response to this query, after which this response is returned, phase 490, as a final response to the calling system. The request received in phase 410 is then cached, along with the final response of phase 490, in phase 450.

[0053] On the other hand, if the first distance exceeds the second threshold, processing advances from phase 460 to phase 4100, where it is determined whether the first distance exceeds the second threshold. If this is the case, processing advances to phases 480, 490 and 450, which have been described herein above. Phase 4100 is optional, and in its absence processing advances from the “N” fork of phase 460 directly to phase 480.

[0054] In some embodiments the tool is configured to abstain from sending the final response after the provisional response, unless the calling system explicitly requests the final response. In that case, the final response is then sent to the calling system as a response to the explicit request for the final response, in addition to the provisional response.

[0055] FIGURE 5 illustrates a process diagram in accordance with at least some embodiments of the present invention. The illustrated process is performed in one the automated system calling on automated tools.

[0056] In phase 510, the automated system sends a query to an automated tool and receives a response to the query. Processing advances to phase 520, where the automated system determined, whether the response is a final response or a provisional response. If the response is final, processing advances to phase 570 where the automated system proceeds with the response in the preparation of deliverable J.

[0057] On the other hand if the response is not final, that is, the response is provisional, processing advances from phase 520 to phase 530, where the automated system determined, whether it can proceed with the preparation of deliverable J based on the provisional response. If this is the case, processing advances to phase 540 where the automated system takes at least one action using the provisional response before receipt of the final response. In phase 550, optionally, the automated system may perform further actions in the preparation of deliverable J, not being directly based on the provisional response. Processing then advances to phase 560, where the final result is received, in some embodiments the final result is received responsive to an explicit request from the automated system to receive the final response. In some cases processing proceeds directly from phase 540 or 550 to phase 570.

[0058] In case the automated system determines, in phase 530, that the provisional response cannot be used in preparing deliverable J, processing advances from phase 530 to phase 560 where the final response is obtained, optionally by explicitly requesting it, andthen processing advances to phase 570, described herein above. In phase 530 the automated system may determine whether the provisional response is usable in preparing deliverable J based on, for example, the cached query used earlier by the automated tool in obtaining the provisional response. As described above, the automated tool may include the cached query in the provisional response, and the automated system may be configured to determine the provisional response is unusable if the cached query differs from the query of phase 510 in an aspect central to the task A the automated system is performing.

[0059] FIGURE 6 is a flow graph of a method in accordance with at least some embodiments of the present invention. The phases of the illustrated method may be performed in an automated tool 122, or in a control device configured to control the functioning thereof, when installed therein.

[0060] Phase 610 comprises receiving a query from a system. Phase 620 comprises determining, using a predetermined metric, distance values between the received query and each one of plural cached queries wherein a first distance value, between the received query and a first cached query, is the smallest distance value among the determined distance values. Phase 630 comprises responsive to the first distance value being smaller than a first threshold, providing to the system a cached response to the first cached query as a response to the received query. Phase 640 comprises, responsive to the first distance value being larger than the first threshold and smaller than a second threshold, providing the cached response to the first cached query as a provisional response to the received query, obtaining a result to the received query and providing the obtained result to the system. Phase 650 comprises, responsive to the first distance value being larger than the second threshold, obtaining the result to the received query and providing the obtained result to the system.

[0061] FIGURE 7 is a flow graph of a method in accordance with at least some embodiments of the present invention. The phases of the illustrated method may be performed in the automated system 110, for example, or in a control device configured to control the functioning thereof, when installed therein

[0062] Phase 710 comprises sending a query to an automated computer tool. Phase 720 comprises determining whether a response to the query, received in the apparatus from the automated computer tool, is a provisional response or a final response. Phase 730 comprises using the response in preparation of a deliverable responsive to determining the response is the final response. Phase 740 comprises, responsive to determining the responseis the provisional response, determining whether the provisional response is usable in preparation of the deliverable. Phase 750 comprises, responsive to the provisional response being usable in preparation of the deliverable, using the provisional response in preparing the deliverable before receiving the final response from the automated computer tool, Phase 760 comprises, responsive to the provisional response not being usable in preparation of the deliverable, waiting for the final response from the automated computer tool. Finally, phase 770 comprises providing the deliverable as a response to a task assigned to the apparatus.

[0063] In some embodiments, automated system 110 is a text-based chatbot or a voice-based automatic speech recognition, ASR, customer service bot. This automated system is configured to perform a configuration update in a cellular telecommunication system automatically as a response to inputs it receives from a user, the user inputs forming task A, as described herein above. The configuration update generated by automated system 110 as deliverable J may comprise an instruction to modify a subscription of the user interacting with automated system 110. This instruction to modify the subscription may comprise an instruction to delete or disable the subscription, or an instruction to limit communications capability of the subscription to emergency calls only. For example, the automated system 110 may determine that the subscription and the user inputs match a profile of a subscription employed in transmitting spam emails or participation in denial-of- service, DoS, attacks. Training data used in enabling classification of a user as malicious may include prior logged information of subscriptions participating in DoS attacks, for example such subscriptions may initiate a large number of protocol connections within a short space of time, or they may seek to re-configure firewall settings of the network in a manner that normal users rarely, if ever, do.

[0064] The cached response accelerates response of automated system 110, for example if a cached response is provided as a provisional response, such as a provisional configuration update, the subscription of the user may be limited for the time needed to obtain the final response. If the final response confirms the subscription is malicious, the subscription will remain in the limited, or disabled, state, but if the final response, contrary to the provisional response, indicates the subscription is not malicious, the subscription may be automatically reverted back to its un-limited or un-disabled configuration. Thus a DoS attack may be substantially impaired quickly if the user is malicious, but if the user is benign the subscription is limited for a brief while only, yielding a clear technical advantage.

[0065] It is to be understood that the embodiments of the invention disclosed are not limited to the particular structures, process steps, or materials disclosed herein, but are extended to equivalents thereof as would be recognized by those ordinarily skilled in the relevant arts. It should also be understood that terminology employed herein is used for the purpose of describing particular embodiments only and is not intended to be limiting.

[0066] Reference throughout this specification to one embodiment or an embodiment means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Where reference is made to a numerical value using a term such as, for example, about or substantially, the exact numerical value is also disclosed.

[0067] As used herein, a plurality of items, structural elements, compositional elements, and / or materials may be presented in a common list for convenience. However, these lists should be construed as though each member of the list is individually identified as a separate and unique member. Thus, no individual member of such list should be construed as a de facto equivalent of any other member of the same list solely based on their presentation in a common group without indications to the contrary. In addition, various embodiments and example of the present invention may be referred to herein along with alternatives for the various components thereof. It is understood that such embodiments, examples, and alternatives are not to be construed as de facto equivalents of one another, but are to be considered as separate and autonomous representations of the present invention.

[0068] Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the preceding description, numerous specific details are provided, such as examples of lengths, widths, shapes, etc., to provide a thorough understanding of embodiments of the invention. One skilled in the relevant art will recognize, however, that the invention can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of the invention.

[0069] While the forgoing examples are illustrative of the principles of the present invention in one or more particular applications, it will be apparent to those of ordinary skill in the art that numerous modifications in form, usage and details of implementation can bemade without the exercise of inventive faculty, and without departing from the principles and concepts of the invention. Accordingly, it is not intended that the invention be limited, except as by the claims set forth below.

[0070] The verbs “to comprise” and “to include” are used in this document as open limitations that neither exclude nor require the existence of also un-recited features. The features recited in depending claims are mutually freely combinable unless otherwise explicitly stated. Furthermore, it is to be understood that the use of "a" or "an", that is, a singular form, throughout this document does not exclude a plurality.

[0071] As used herein, “at least one of the following: ” and “at least one of ” and similar wording, where the list of two or more elements are joined by “and” or “or”, mean at least any one of the elements, or at least any two or more of the elements, or at least all the elements.INDUSTRIAL APPLICABILITY

[0072] At least some embodiments of the present invention find industrial application in computer systems.ACRONYMS LISTLLM large language modelSLM small language modelREFERENCE SIGNS LIST

Claims

CLAIMS:

1. A computer-implemented method comprising:- receiving a query;- determining, using a predetermined metric, distance values between the received query and each one of plural cached queries wherein a first distance value, between the received query and a first cached query, is the smallest distance value among the determined distance values;- responsive to the first distance value being smaller than a first threshold, providing a cached response to the first cached query as a final response to the received query;- responsive to the first distance value being larger than the first threshold and smaller than a second threshold, providing the cached response to the first cached query as a provisional response to the received query, and- responsive to the first distance value being larger than the second threshold, obtaining the result to the received query and providing the obtained result as a final response to the received query.

2. The computer-implemented method according to claim 1, wherein the query is received from a large language model or a small language model, and the cached response and / or the obtained result is provided to the large language model or the small language model.

3. The computer-implemented method according to any of claims 1 - 2, comprising, as a response to the first distance value being larger than the second threshold, including the received query among the cached queries and caching the obtained result.

4. The computer-implemented method according to any of claims 1 - 3, wherein the predetermined metric is based on time instants specified in queries, such that the distance values obtained using the predetermined metric are greater when the time instants specified in the queries are further from each other in time and the distance values obtained using the predetermined metric are smaller when the time instants specified in the queries are closer to each other in time.

5. The computer-implemented method according to any of claims 1 - 4, wherein the predetermined metric is based on geographical separation, such that the distance values obtained using the predetermined metric are greater when geographical locations specified in the queries are further from each other and the distance values obtained using the predetermined metric are smaller when the geographical locations specified in the queries are closer to each other.

6. The computer-implemented method according to any of claims 1 - 5, wherein the predetermined metric is based on semantic distance, such that the distance values obtained using the predetermined metric are greater when the queries are further from each other in a semantic space and the distance values obtained using the predetermined metric are smaller when the queries are closer to each other in the semantic space.

7. The computer-implemented method according to any of claims 1 - 6, wherein the predetermined metric is based on numerical parameters, such that the distance values obtained using the predetermined metric are greater when values of corresponding numerical parameters specified in the queries are further from each other and the distance values obtained using the predetermined metric are smaller when the values of the corresponding numerical parameters specified in the queries are closer to each other.

8. The computer-implemented method according to any of claims 1 - 7, comprising determining each one of the distance values as an absolute value of a distance vector, each component of the distance vector being obtained using a specific aspect of the queries.

9. The computer-implemented method according to any of claims 1 - 8, comprising obtaining the first threshold and the second threshold from the received query.

10. The computer-implemented method according to any of claims 1 - 9, comprising obtaining the predetermined metric from the received query.

11. The computer-implemented method according to any of claims 1 - 10, comprising:- determining whether the response to the query is a provisional response or a final response;- using the response in preparation of a deliverable responsive to determining the response is the final response;- responsive to determining the response is the provisional response, determining whether the provisional response is usable in preparation of the deliverable;- responsive to the provisional response being usable in preparation of the deliverable, using the provisional response in preparing the deliverable before receiving the final response;- responsive to the provisional response not being usable in preparation of the deliverable, waiting for the final response, and- providing the deliverable as a response to a task.

12. The computer-implemented method according to claim 11, wherein the using of the provisional response in the preparing of the deliverable comprises sending a second query, the second query comprising a parameter which is comprised in the provisional response or is derived from the provisional response.

13. The computer-implemented method according to claim 11 or 12, comprising performing the providing of the deliverable after receiving the provisional response and before receiving the final response.

14. The computer-implemented method according to any of claims 11 - 13, further comprising sending the query and including in the query at least one of: a first threshold, a second threshold and a predetermined metric for use in determining whether to send the provisional response.

15. The computer-implemented method according to any of claims 1 - 14, wherein the receiving of the query and the providing of the provisional and / or final response is performed by an automated tool.

16. An apparatus comprising at least one processing core and at least one memory storing instructions that, when executed by the at least one processing core, cause the apparatus at least to perform a computer-implemented method according to at least one of claims 1 - 15.

17. An apparatus comprising:- means for receiving a query;- means for determining, using a predetermined metric, distance values between the received query and each one of plural cached queries wherein a first distance value, between the received query and a first cached query, is the smallest distance value among the determined distance values;- means for, responsive to the first distance value being smaller than a first threshold, providing a cached response to the first cached query as a final response to the received query;- means for, responsive to the first distance value being larger than the first threshold and smaller than a second threshold, providing the cached response to the first cached query as a provisional response to the received query, and- means for, responsive to the first distance value being larger than the second threshold, obtaining the result to the received query and providing the obtained result as a final response to the received query.

18. A non-transitory computer readable medium having stored thereon a set of computer readable instructions that, when executed by at least one processor, cause an apparatus to at least:- receive a query;- determine, using a predetermined metric, distance values between the received query and each one of plural cached queries wherein a first distance value, between the received query and a first cached query, is the smallest distance value among the determined distance values;- responsive to the first distance value being smaller than a first threshold, provide a cached response to the first cached query as a final response to the received query;- responsive to the first distance value being larger than the first threshold and smaller than a second threshold, provide the cached response to the first cached query as a provisional response to the received query, and- responsive to the first distance value being larger than the second threshold, obtain the result to the received query and provide the obtained result as a final response to the received query.

Images