Query handling

Abstract

There is provided a method for handling queries associated with a RL model. The RL model is configured to control an environment within a domain. The method comprises obtaining (102) a user query comprising a condition associated with an operation of the RL model. The method also comprises obtaining (104), based on the condition, at least one state of the environment and at least one corresponding action. The at least one corresponding action corresponds to an action recommended by the RL model to be executed on the environment in the at least one state. The method further comprises generating (106) a first response to the user query. The first response is generated based on a statistical analysis of the at least one state and the at least one corresponding action. The method also comprises generating (108) a second response to the user query. The second response comprises information indicative of a feature analysis of one or more input features of the RL model, and the one or more input features are associated with the at least one state. The method further comprises providing (110), to the user associated with the user query, a query response based on the generated first and second responses to the user query.

Description

[0001] QUERY HANDLING

[0002] Technical Field

[0003] The present disclosure relates to methods for handling queries associated with a reinforcement learning (RL) model, and entities configured to operate in accordance with those methods.

[0004] Background

[0005] Reinforcement learning (RL) is a method where an agent learns from its interaction with an environment to make decisions with the highest long-term benefit. In a complex scenario, the agent is implemented using state-of-the-art methods where understanding the behaviour of the agent is not straightforward. Thus, explainable reinforcement learning (XRL) methods have been developed to analyse the behaviour of RL agents. In XRL methods, an explanation of an RL agent's behaviour can generate additional information about several elements associated with the RL agent, such as input states, rewards, actions, or internal properties of the agent.

[0006] There currently exist methods for performing XRL. For example, some methods involve generating an explanation for the behaviour of the RL agent in a natural language format. Such methods enable a conversation-like interaction between a human user and the performing RL agent. For example, the human user is able to receive an answer to the following questions: "When is an action performed?”, "Why is an action not taken?”, and "What action will be taken in a certain situation?”. While some existing techniques may provide explanation of RL agent behaviour in a natural language format, other techniques may provide an explanation by measuring the contributions of input features to reward components of the RL agent.

[0007] However, there are challenges associated with such existing techniques for performing XRL. In particular, that many of the existing techniques fail to provide a comprehensive explanation for the behaviour of the RL agent. Indeed, many of the existing techniques which provide natural language format explanations do so without considering the internal properties of the RL model being used by the RL agent. As such, these existing techniques fail to thoroughly understand the decision making and prediction mechanisms that result in certain behaviour being executed by the RL model. Furthermore, existing XRL techniques which rely on measuring contributions of input features do not provide a mechanism to tailor the analysed data. As such, these techniques suffer from a lack of precision.

[0008] As mentioned above, there exist certain challenges associated with current techniques for handling queries associated with a reinforcement learning (RL) model. In particular, existing techniques fail to provide a thorough explanation of RL model behavior (e.g. as requested in a query), and as such do not provide the necessary level of detail in order to allow for a proper understanding of the (e.g. choices and / or decisions made by) RL model.

[0009] It is an object of the disclosure to elimi nate or obviate at least some of these challenges associated with existing techniques.

[0010] Therefore, according to an aspect of the disclosure, there is provided a method for handling queries associated with a RL model. The RL model is configured to control an environment within a domain. The method comprises obtaining a user query comprising a condition associated with an operation of the RL model. The method also comprises obtaining, based on the condition, at least one state of the environment and at least one corresponding action. The at least one corresponding action corresponds to an action recommended by the RL model to be executed on the environment in the at least one state. The method further comprises generating a first response to the user query. The first response is generated based on a statistical analysis of the at least one state and the at least one corresponding action. The method also comprises generating a second response to the user query. The second response comprises information indicative of a feature analysis of one or more input features of the RL model, and the one or more input features are associated with the at least one state. The method further comprises providing, to the user associated with the user query, a query response based on the generated first and second responses to the user query.

[0011] According to another aspect of the disclosure, there is provided a first entity comprising processing circuitry configured to operate in accordance with the method described earlier. In some embodiments, the first entity may comprise at least one memory for storing instructions which, when executed by the processing circuitry, cause the first entity to operate in accordance with the method described earlier.

[0012] According to another aspect of the disclosure, there is provided a computer program comprising instructions which, when executed by processing circuitry, cause the processing circuitry to perform the method described earlier.

[0013] According to another aspect of the disclosure, there is provided a computer program product, embodied on a non-transitory machine-readable medium, comprising instructions which are executable by processing circuitry to cause the processing circuitry to perform the method described earlier.

[0014] Thus, in the manner described above, improved techniques for handling queries associated with a reinforcement learning (RL) model are provided. Advantageously, the techniques provide for a more thorough response (e.g. explanation) of a query associated with an RL model by providing a query response based on a combination of a first response and a second response. The first response provides insight into the statistical properties which are associated with a particular RL model's behaviour, while the second response provides an insight into the internal properties of the RL model and how those properties contribute to said RL model behaviour. As such, the first response and second response are complementary. Moreover, the generation of both the first response and the second response are based on the same at least one state of the environment, which further enables the provision of a more thorough query response, and clarifies potential ambiguity which may arise in existing techniques for handling queries. In a particular example, the improved techniques can be used by a user to provide query responses which address different situations, identify (e.g. unwanted) behaviour exhibited by the RL model, and provide a detailed analysis of this behaviour.

[0015] For a better understanding of the techniques, and to show how they may be put into effect, reference will now be made, by way of example, to the accompanying drawings, in which:

[0016] Figures 1 is a schematic illustration of an example domain;

[0017] Figures 2-5 are graphs illustrating the results of an RL model analysis;

[0018] Figure 6 is a block diagram illustrating an entity according to an embodiment;

[0019] Figure 7 is a block diagram illustrating a method performed according to an embodiment;

[0020] Figure 8 is a signalling diagram illustrating a method performed by a system according to an embodiment; and

[0021] Figures 9-12 are graphs illustrating the results of an RL model analysis.

[0022] Detailed

[0023] Generally, all terms used herein are to be interpreted according to their ordinary meaning in the relevant technical field, unless a different meaning is clearly given and / or is implied from the context in which it is used. All references to a / an / the element, apparatus, component, means, step, etc. are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any methods disclosed herein do not have to be performed in the exact order disclosed, unless a step is explicitly described as following or preceding another step and / or where it is implicit that a step must follow or precede another step. Any feature of any of the embodiments disclosed herein may be applied to any other embodiment, wherever appropriate. Likewise, any advantage of any of the embodiments may apply to any other embodiments, and vice versa. Other objectives, features and advantages of the enclosed embodiments will be apparent from the following description.

[0024] Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Other embodiments, however, are contained within the scope of the subjectmatter disclosed herein, the disclosed subject-matter should not be construed as limited to only the embodiments set forth herein; rather, these embodiments are provided by way of example to convey the scope of the subject-matter to those skilled in the art.

[0025] As mentioned earlier, this disclosure relates to methods for handling queries associated with a RL model. As such, the methods described herein may be referred to as examples of explainable reinforcement learning (XRL). As mentioned herein, XRL methods can be used to analyse behaviour of RL models. One method for performing XRL can be referred to as autonomous policy explanation (APE) (see, for example, Hayes et al, ""Improving robot controller transparency through autonomous policy explanation"). APE methods can involve summarising a mapping between recorded states (e.g. of an environment) to selected actions of the RL model. APE methods can be configured to generate explanations to a query in a natural language format. This can enable a conversation-like interaction, for example between a human and a performing RL model (e.g. RL agent). As such, APE methods can be used to provide answers to questions such as: "When is an action performed?”, "Why is an action not taken?”, and "What action will be taken in a certain situation?”. As APE methods can be configured to analyse a statistical occurrence of states and / or actions, (e.g. human) users can query such questions without actually running the RL model (e.g. agent) to arrive at a certain state. Furthermore, some APE methods are capable of generating contrastive explanations which explain, for example, why an action is not taken for a certain state. In general, APE methods can be applied to any RL model / agent. Thus, when a different RL model (e.g. algorithm and / or agent) is used, many components of APE methodology can be reused with minimal adjustment.

[0026] While APE methods are able to summarise a state-action mapping in natural language format, other methods of XRL are able to generate (e.g. more) detailed explanations by measuring (e.g. analysing) input features for the RL mode. An example of such a method is Both Ends Explanations for RL (BEERL), as described in International patent application No. WO 2023 / 136771. BEERL can be configured to generate detailed explanations (e.g. of RL model behaviour) by measuring contributions of input features to reward components. BEERL is a method internally developed in Ericsson and it can be implemented based on a Deep Q-Network (DQN) algorithm with reward decomposition. In addition to analysing (e.g. and showing) feature contributions, BEERL is also able to identify bias in a reward component of the RL model. Furthermore, reward components can be exposed to provide a better understanding of how each reward component contributes to action selection (e.g. by the RL model). In this way, the analysis provided by BEERL can be used as feedback to configure weights of the RL model, or to mitigate bias by adjustment. BEERL methods can also generate contrastive explanation which, for example, explain why an action is selected over another (e.g. in terms of feature contribution).

[0027] The techniques described herein involve an RL model. For the purposes of the present disclosure, the term "RL model” can encompass, within its scope, an RL algorithm, comprising processes or instructions through which data may be used in a training process to generate a model artefact for performing a given task, or for representing a real world process or system. Alternatively, or in addition, the term "RL model” may encompass the model artefact that is created by such a training process, and which comprises the computational architecture that performs the task. Alternatively, or in addition, the term "RL model” may encompass the process performed by the model artefact in order to complete the task. References to "RL model”, "model”, model parameters”, "model information”, etc., may thus be understood as relating to any one or more of the above concepts encompassed within the scope of "RL model”. The RL model may be referred to herein as a "RL agent”.

[0028] Figure 1 is a schematic illustration of a domain according to an example. As illustrated in Figure 1 , in some examples, the domain referred to herein may comprise a network. Therefore, in some examples, the domain referred to herein may be a (tele)communications network. In such examples, the environment referred to herein may comprise one or more network entities (e.g. of the domain).

[0029] An RL model, such as the RL model referred to herein, may be applied to the network of Figure 1 in order to optimise (e.g. the key performance indicators (KPIs) of) the network. For example, as illustrated in Figure 1, the RL model can be applied to a remote electrical antenna-tilt (RET) use case where the electrical tilt of the antenna is controlled to optimise the KPIs of the network. In this use case, each antenna can be controlled by the RL model. For example, the RL model can obtain (e.g. take) properties and / or KPIs of the network as input (e.g. states) to produce one of the following actions: tilt-up, no-tilt, or tilt-down. As such, in some examples, the RL model can be trained to optimise the tilt (e.g. value) of one or more antennas in the environment.

[0030] In addition to the above-mentioned input, multiple frequency bands may be configured in the environment, making the environment more complex. More specifically, the configuration of multiple frequency bands in the environment may require the RL agent to optimise over different frequency layers. Figure 1 illustrates such a multi-band RET scenario use case. In Figure 1 , the use of different types of shading illustrates different frequency bands that can be supported by each antenna. As illustrated in Figure 1 , each sector can be covered by one to three antennas. As mentioned herein, one method for performing XRL is autonomous policy explanation (APE). APE relies on the statistical analysis of states and actions (e.g. as performed by an RL model) to generate explanations. The explanation generated by

[0031] APE can be in a natural language format. To utilise APE for handling queries associated with an RL agent, APE may require one or more kinds of information.

[0032] For example, APE may utilise experience replay data. In particular, an APE process may comprise generating a summarisation of a policy using experience replay data and / or state-action pairs (e.g. associated with the RL model). State data can be collected (e.g. from the RL model) during training and / or from a testing phase (e.g. process). The action (e.g. of the state-action pair) can be obtained during an exploitation mode (e.g. of the latest trained version of the RL model). It can be beneficial to obtain the action in this way since actions recommended by the RL model during testing or training (e.g. "exploration actions”) can generate misleading results. Collecting data from a testing phase can be advantageous in comparison to collecting data from a training phase since, during training, the RL model may be converging and thus still being improved. Nevertheless, collecting data from training can also be beneficial to understand the evolution of the model. Experience replay data can be saved in a record. Each record may include a state, a (e.g. corresponding) action and a state identifier and / or number.

[0033] Alternatively, or in addition, an APE process may utilise predicate definition. Predicates can be defined as simple Boolean expressions. The Boolean expression can be defined based on state variables. Predicates can be used to map each state (e.g. of the environment controlled by RL model) to a binary string (e.g. 11000). Each predicate can describe a property of the state using one or more of its variables. Each predicate can have a Boolean expression and natural language description for a True outcome and a False outcome of the expression. Examples of such a Boolean expression are given below:

[0034] 'true' : ‘cell is more loaded that its neighbours (traffic) ', 'false' : ‘cell is less or equally loaded as its neighbours' , 'verify' : lambda s: s[‘trafficShare'] >1

[0035] An APE method may utilise state and action labels. Ideally, names of state observables (e.g. KPIs) and actions (e.g., uptilt, downtilt, etc.) are provided in a specific (e.g. right) order (e.g. action 0 corresponds to uptilt, action 1 corresponds to downtilt, and action 2 corresponds to no tilt). To generate explanations, an APE method may accept the following questions:

[0036] When do you <action name>?

[0037] The action name can be one of the actions defined in a set of action labels.

[0038] Why didn't you <action name> in state <state number>?

[0039] The action name can be one of the actions defined in a set of action labels. The state number can be a valid identifier (e.g. for obtaining the state from experience replay records).

[0040] What will you do when <state bit stri ng>?

[0041] The state bit string can correspond to defined predicates. The string length can be equal to the number of predicates. The possible value for each bit is as follows:

[0042] • 1 : The predicate is True

[0043] • 0: The predicate is False

[0044] • The value of this predicate doesn't matter

[0045] Some example outputs for possible queries (e.g. questions) asked in the multi-band RET use case of Figure 1 are shown below. The use case of Figure 1 may involve a single RL model or multiple RL models.

[0046] 1. When do you uptilt?

[0047] I uptilt when cell has low RRC congestion and cell neighbours have normal percent of bad traffic and cell edge coverage is acceptable and cell is not causing interference in its area and neighbours are not highly interfering with the cell and cell has same or lower di utilization as neighbour frequencies ( 52.53%).

[0048] OR cell has low RRC congestion and overshooting is acceptable and cell is used for capacity and cell has same or lower di utilization as neighbour frequencies (39.85%).

[0049] The percentages shown in brackets above are a measure of the percentage likelihood that the action "uptilt” is recommended (e.g. by the RL model) in the corresponding state of the environment.

[0050] 2. Why didn’t you uptilt in state 2803?

[0051] I didn't uptilt because cell has normal percent of bad traffic and neighbours interference is affecting the cell and cell is used for coverage and cell has same or lower di utilization as neighbour frequencies.

[0052] 3. What will you do when 1--01-1— ? I will downtilt if cell is used for coverage.

[0053] OR cell neighbours have high RRC congestion and cell has normal percent of bad traffic.

[0054] OR neighbours interference is affecting the cell

[0055] OR cell is less or equally loaded as its neighbours.

[0056] For questions 1 and 3, in addition to the explanations provided by APE, a set of state identifiers (e.g. indexes) can also be generated for the states that fulfil the queried condition.

[0057] As also mentioned herein, another method for performing XRL is Both Ends Explanation for RL (BEERL). BEERL methods can involve generating detailed explanation(s) showing the contributions of inputs to different reward components (e.g. of the RL model). A BEERL method may require some of the following types of information.

[0058] Multiple reward component

[0059] In general, BEERL methodology is built on top of reward decomposition methods. As such, a BEERL method can require access to reward components instead of, for example, a single value of the reward during the training of the RL model. Thus, in some examples, the use of BEERL may require the environment and / or the RL agent to be adjusted to generate and / or receive reward components.

[0060] Core Explainer

[0061] Since BEERL tends to use post-hoc methodology to generate an explanation. A core method can be specified to be used as a core explainer for BEERL. Examples of techniques that can be used for the core method include GradientSHAP, local interpretable model-agnostic explanations (LIME), and Deeplift.

[0062] In some cases, the use of BEERL may require a label and / or a name to be assigned to each input feature (e.g. of the RL model), and / or each reward component (e.g. of the RL model).

[0063] Figures 2 to 5 show graphs illustrating an outcome of analysis of an RL model performed using a BEERL based method, according to some examples. In particular, the graphs illustrated in Figures 2 to 5 are results of an analysis performed in respect of the use case as described with reference to Figure 1 . As can be seen from Figures 2 to 5, the BEERL based method is able to generate detailed explanations of RL model behaviour.

[0064] For example, Figure 2 illustrates a local explanation for the RL model. More specifically, Figure 2 illustrates the results of a BEERL explanation of the RL model's decision making for a particular state (e.g. of the environment of Figure 1). The graph of Figure 2 is a visualisation of a local BEERL explanation, which illustrates how each input feature of the RL model contributes to different reward components. In the example illustrated in Figure 2, the different rewards components comprise a legacy reward component, a multiband reward component, and a congestion reward component.

[0065] As mentioned above, Figures 2 to 5 correspond to the use case described with reference to Figure 1. In such an example, the domain can be a telecommunications network, and the environment can comprise one or more antennas (e.g. of the telecommunications network). As illustrated in Figure 2, the input features of the RL model can comprise: a position of an antenna in an x-axis ("cell_x_norm”), a position of the antenna in an y-axis ("cell_y_norm”), a height of the antenna ("cell_z_norm”), traffic generated in good radio conditions ("goodTraffic_norm”), traffic generated in bad radio conditions ("badTraffic_norm”), traffic generated in good radio conditions of a neighbouring cell / antenna ("goodTrafficNeigh_norm”), traffic generated in bad radio conditions of a neighbouring cell / antenna ("badTrafficNeigh_norm”), tilting angle of the antenna ("retNorm”), distance of one antenna to another (“interSiteDistNorm”), KPI of radio resource control congestion ("rrcCongestionRate”), percentage of traffic handled by a cell / antenna (“trafficshare”), type of cell (e.g. capacity and / or coverage cell) (“cellType”). As illustrated in Figure 2, many of the input features have been normalised to give a normalised (e.g. "_norm”) value. The input feature values can be normalised into the value range 0 to 1 (e.g. to assist training and / or learning of the RL model).

[0066] Figure 3 illustrates an example of a global explanation for the RL model. More specifically, Figure 3 explains overall input feature contribution towards different reward components. As such, Figure 3 illustrates a global explanation of a trained RL model. The reward components and input features illustrated in Figure 3 can be as described with reference to Figure 2.

[0067] Figure 4 illustrates an example of a contrastive explanation analysis that can be generated from a reward decomposition method. The reward decomposition method can involve, for example, comparing Q-values for each reward component (e.g. of the reward components illustrated in Figures 2 and 3). As such, Figure 4 is an example of an analysis which serves to explain why an action is chosen (e.g. by the RL model) instead of another action.

[0068] As illustrated in Figure 4, the reward associated with each reward component is greater for the action labelled "1 2” than the action labelled "1 0”.

[0069] Figure 5 illustrates a contrastive explanation that is generated using a BEERL method which compares the contribution of each input feature to each reward component. The reward components and input features illustrated in Figure 5 can be as described with reference to Figure 2. Some of the features associated with XRL methods have been described above. However, as mentioned herein, existing (e.g. XRL) methods for handling queries associated with an RL model present certain challenges. For example, the use of APE can generate explanations of RL model behaviour by analysing statistical properties of recorded (e.g. RL model) data, but fails to take into account any internal properties of the RL model Itself. In addition, although the use of BEERL may explain certain internal properties of the RL model, BEERL methods can only present either local (a single instance) data or whole datasets (e.g. all instances). Such a configuration is inefficient and there is no mechanism to select a certain type of data to be explained. Furthermore, using APE for contrastive explanation can produce a non-dominance action where the explanation is counter intuitive. For example, a user may ask "Why didn't you downtilt instead of uptilt in state 4?” and the explanation generated using APE may state "I didn't downtilt because cell has low RRC congestion and cell edge coverage is acceptable”. However, if further analysis is performed, it can be revealed that the most dominant action of the given explanation (i.e. low RRC congestion and cell edge coverage is acceptable) is downtilt. In scenarios such as these, an APE explanation alone may provide an incorrect and / or unsuitable explanation (e.g. to the user).

[0070] The improved techniques described herein address the challenges associated with existing techniques, such as those described above. In particular, the improved techniques discussed herein provides for an understanding of the internal properties of a RL model in a certain condition to evaluate decisions or predictions of the RL model.

[0071] Figure 6 illustrates a first entity 10 in accordance with an embodiment. The first entity 10 is for handling queries associated with a RL model. In some embodiments, the first entity 10 referred to herein can refer to equipment capable, configured, arranged and / or operable to communicate directly or indirectly with the RL model referred to herein, and / or with other entities or equipment to enable and / or to perform the functionality described herein. In some embodiments, the first entity 10 referred to herein can, for example, be a physical entity (e.g. a physical machine or server) or a virtual entity (e.g. a virtual machine, VM). In some examples, the first entity 10 referred to herein may comprise, and / or be configured to perform the functionality of, the RL model referred to herein. The first entity 10 may be comprised in the domain referred to herein. Alternatively, or in addition, the first entity 10 may be comprised in the environment referred to herein.

[0072] As illustrated in Figure 6, the first entity 10 comprises processing circuitry (or logic) 12. The processing circuitry 12 controls the operation of the first entity 10 and can implement the method described herein in respect of the first entity 10. The processing circuitry 12 can be configured or programmed to control the first entity 10 in the manner described herein. The processing circuitry 12 can comprise one or more hardware components, such as one or more processors, one or more processing units, one or more multi- core processors and / or one or more modules. In particular implementations, each of the one or more hardware components can be configured to perform, or is for performing, individual or multiple steps of the method described herein in respect of the first entity 10. In some embodiments, the processing circuitry 12 can be configured to run software to perform the method described herein in respect of the first entity 10. The software may be containerised according to some embodiments. Thus, in some embodiments, the processing circuitry 12 may be configured to run a container to perform the method described herein in respect of the first entity 10.

[0073] Briefly, the processing circuitry 12 of the first entity 10 can be configured to obtain a user query comprising a condition associated with an operation of an RL model. The processing circuitry 12 of the first entity 10 can also be configured to obtain, based on the condition, at least one state of an environment and at least one corresponding action. The at least one corresponding action corresponds to an action recommended by the RL model to be executed on the environment in the at least one state. The processing circuitry 12 of the first entity 10 can also be configured to generate a first response to the user query. The first response is generated based on a statistical analysis of the at least one state and the at least one corresponding action. The processing circuitry 12 of the first entity 10 can also be configured to generate a second response to the user query. The second response comprises information indicative of a feature analysis of one or more input features of the RL model, and the one or more input features are associated with the at least one state. The processing circuitry 12 of the first entity 10 can also be configured to provide, to the user associated with the user query, a query response based on the generated first and second responses to the user query.

[0074] As illustrated in Figure 6, in some embodiments, the first entity 10 may optionally comprise a memory 14. The memory 14 of the first entity 10 can comprise a volatile memory or a non-volatile memory. In some embodiments, the memory 14 of the first entity 10 may comprise a non-transitory media. Examples of the memory 14 of the first entity 10 include, but are not limited to, a random access memory (RAM), a read only memory (ROM), a mass storage media such as a hard disk, a removable storage media such as a compact disk (CD) or a digital versatile disk (DVD), and / or any other memory.

[0075] The processing circuitry 12 of the first entity 10 can be communicatively coupled (e.g. connected) to the memory 14 of the first entity 10. In some embodiments, the memory 14 of the first entity 10 may be for storing program code or instructions which, when executed by the processing circuitry 12 of the first entity 10, cause the first entity 10 to operate in the manner described herein in respect of the first entity 10. For example, in some embodiments, the memory 14 of the first entity 10 may be configured to store program code or instructions that can be executed by the processing circuitry 12 of the first entity 10 to cause the first entity 10 to operate in accordance with the method described herein in respect of the first entity 10. Alternatively or in addition, the memory 14 of the first entity 10 can be configured to store any information, data, messages, requests, responses, indications, notifications, signals, or similar, that are described herein. The processing circuitry 12 of the first entity 10 may be configured to control the memory 14 of the first entity 10 to store any of the information, data, messages, requests, responses, indications, notifications, signals, or similar, that are described herein.

[0076] In some embodiments, as illustrated in Figure 6, the first entity 10 may optionally comprise a communications interface 16. The communications interface 16 of the first entity 10 can be communicatively coupled (e.g. connected) to the processing circuitry 12 of the first entity 10 and / or the memory 14 of the first entity 10. The communications interface 16 of the first entity 10 may be operable to allow the processing circuitry 12 of the first entity 10 to communicate with the memory 14 of the first entity 10 and / or vice versa. Similarly, the communications interface 16 of the first entity 10 may be operable to allow the processing circuitry 12 of the first entity 10 to communicate with any one or more entities referred to herein and / or any other entity. The communications interface 16 of the first entity 10 can be configured to transmit and / or receive any of the information, data, messages, requests, responses, indications, notifications, signals, or similar, that are described herein. In some embodiments, the processing circuitry 12 of the first entity 10 may be configured to control the communications interface 16 of the first entity 10 to transmit and / or receive any of the information, data, messages, requests, responses, indications, notifications, signals, or similar, that are described herein.

[0077] Although the first entity 10 is illustrated in Figure 6 as comprising a single memory 14, it will be appreciated that the first entity 10 may comprise at least one memory (i.e. a single memory or a plurality of memories) 14 that operate in the manner described herein. Similarly, although the first entity 10 is illustrated in Figure 6 as comprising a single communications interface 16, it will be appreciated that the first entity 10 may comprise at least one communications interface (i.e. a single communications interface or a plurality of communications interfaces) 16 that operate in the manner described herein. It will also be appreciated that Figure 6 only shows the components required to illustrate an embodiment of the first entity 10 and, in practical implementations, the first entity 10 may comprise additional or alternative components to those shown.

[0078] Figure 7 illustrates a computer-implemented method performed in accordance with an embodiment. The method is for handling queries associated with a reinforcement RL model. The RL model is configured to control an environment within a domain. The first entity 10 described earlier with reference to Figure 6 can be configured to operate in accordance with the method of Figure 7. The method can be performed by or under the control of the processing circuitry 12 of the first entity 10 according to some embodiments. With reference to Figure 7, as illustrated at block 102, a user query comprising a condition associated with an operation of the RL model is obtained. In some examples, the condition can comprise information indicative of a queried state of the environment. For example, if the user query is a question of the form "What will you do in <queried state>?”, then the condition may comprise the information indicative of the "<queried state>”. In some examples, the condition may comprise information indicative of a queried action recommended by the RL model. For example, if the user query is a question of the form "When do you <queried action>?”, then the condition may comprise information of the "<queried action>”. In some examples, the condition may comprise both information indicative of a queried state and information indicative of a queried action. For example, if the user query is a question of the form "Why did you not <queried action> in <queried state>?”, then the condition may comprise information indicative of the "<queried action>” and information indicative of the "<queried state>”. In some examples, the user query may be obtained by the first entity 10 referred to herein (e.g. via the communications interface 16 of the first entity 10).

[0079] As illustrated at block 104 of Figure 7, at least one state of the environment and at least one corresponding action are obtained based on the condition. The at least one corresponding action corresponds to an action recommended by the RL model to be executed on the environment in the at least one state. For example, the at least one corresponding action may correspond to a (e.g. stored) action that is recommended by the RL model when the environment is configured in the at least one state. Therefore, the at least one corresponding action may represent the suggested behaviour of the RL model for the environment when the environment is in the at least one state. The at least one corresponding action may be a past action or a future action. For example, the at least one corresponding action may be an action already recommended by the RL model (e.g. and executed on the environment). Alternatively, or in addition, the at least one corresponding action may be an action that will be recommended by the RL model (e.g. when the environment is (e.g. eventually) in the at least one state). In the example use case as described with reference to Figure 1 above, the RL model may recommend an antenna uptilt action if the cell of the antenna has low radio resource configuration (RRC) congestion. In this example, antenna uptilt can be referred to as the corresponding action for the at least one state of low RRC congestion.

[0080] As illustrated at block 106 of Figure 7, a first response to the user query is generated. The first response is generated based on a statistical analysis of the at least one state and the at least one corresponding action. As such, the first response may be based on a frequency and / or likelihood of the RL model recommending a certain action for a given state of the environment. In a particular example, the generation of the first response may consider how often the condition referred to herein occurs. In some examples, the first response may be generated based on an APE process as described herein. The first response may be a natural language response to the user query. For example, the first response may be in the form of a natural language response which serves as an explanation for the user query. It will be understood that the at least one state can comprise any number of states. For example, the at least one state can comprise a plurality of states. Similarly, the at least one corresponding action can comprise any number of corresponding actions (e.g. one or more).

[0081] As illustrated at block 108 of Figure 7, a second response to the user query is generated. The second response comprises information indicative of a feature analysis of one or more input features of the RL model. The one or more input features are associated with the at least one state. The second response can be indicative of (e.g. show) a contribution of each input feature to (e.g. each of) one or more reward components. In the example use case described with reference to Figure 1 , the one or more input features may comprise any of the input features a described above with reference to Figure 2. The second response may be generated using a feature analysis technique, such as BEERL, SHAP, and / or LIME. As mentioned above, the one or more input features are associated with the at least one state referred to herein. As such, the generation of the first response and the second response can be based on the same at least one state referred to herein. That is, the data used to generate the first response can also be used to generate the second response.

[0082] As illustrated at block 108 of Figure 7, a query response is provided to the user associated with the user query. The query response is based on the generated first and second responses to the user query. As such, the method advantageously provides a query response which takes into account both the first and second responses. In this way, the provided query response provides the user with a thorough response (e.g. answer) to the user query.

[0083] In some examples, the at least one state and the at least one corresponding action can be obtained from a memory (e.g. the memory 14 of the first entity 10). The memory can be a database of the domain (e.g. a telecommunications network). The memory may store all relevant information about any RL elements (e.g. of the RL model). For example, the memory may be a database of all actions and / or all states of the environment encountered by the RL model. The database may store such information using one or more state-action tuples.

[0084] Although not illustrated in Figure 7, in some examples, obtaining the at least one state and the at least one corresponding action may comprise obtaining a plurality of states associated with the environment, and a plurality of corresponding actions. For example, data corresponding to a plurality of states and a plurality of corresponding actions can be obtained (e.g. taken) from the memory referred to herein. In some examples, obtaining the at least one state and the at least one corresponding action may comprise identifying, based on the condition, the at least one state from the plurality of states, and the at least one corresponding action from the plurality of corresponding actions. For example, the plurality of states and / or the plurality of corresponding actions can be filtered, based on the condition, to extract the at least one state and the at least one corresponding action.

[0085] Although not illustrated in Figure 7, in some examples, the method may comprise generating a binary representation of the condition. For example, the condition may be mapped to a binary string. The generation of the binary representation can be based on one or more (e.g. preconfigured) predicates. Each predicate of the one or more predicates may be related to an input feature of the RL model. Each predicate may be defined in terms of a range of input values. The (e.g. definition of the) one or more predicates can be used to convert the condition (e.g. the queried state referred to herein, and / or the queried action referred to herein) to the binary representation of the condition. Identifying (e.g. filtering) the at least one state from the plurality of states, as mentioned herein, may comprise identifying, based on the binary representation of the (e.g. queried state of the) condition, the at least one state from a binary representation of each state of the plurality of states. As such, in some examples, the method may comprise generating a binary representation of each state of the plurality of states referred to herein.

[0086] In a particular example, obtaining the at least one state of the environment and the at least one corresponding action may comprise obtaining (e.g. taking) one or more (e.g. whole) recorded state and action tuples from a deployment of the (e.g. trained) RL agent. Generating the first response can comprise generating (e.g. via APE) an explanation in natural language format. In some examples, the states of the recorded state and action tuple can be converted into binary representations (e.g. based on defined predicates as mentioned herein). The binary representation information (e.g. data) may then be analysed (e.g. using one or more different algorithms) based on the user query. In examples in which a noncontrastive explanation is generated for the first response, the generation of the first response can result in a set of states which fulfils the queried condition. These states may then be used to generate a (e.g. more) detailed second response (e.g. using a BEERL method).

[0087] As mentioned herein, the condition referred to herein may comprise information indicative of a queried state of the environment. The at least one state can be associated with the queried state. Alternatively, or in addition, the condition may comprise information indicative of a queried action recommended by the RL model. The at least one corresponding action can be associated with the queried action. As such, in some examples, the condition referred to herein can itself comprise a queried state and / or a queried action (e.g. "Why did you do <queried action> in <queried state>?”). The at least one state may be associated with the queried state by virtue of the at least one state being the same state as the queried state, a state which comes before / after the queried state (e.g. with respect to time), and / or a state that has a particular relationship with the queried state. Similarly, in some examples, the at least one corresponding action may be associated with the queried action by virtue of the at least one action being the same action as the queried action, and / or an action that has a relationship with the queried action.

[0088] Although not illustrated in Figure 7, in some examples, the method may comprise generating the feature analysis of the one or more input features of the RL model. Generating the feature analysis may comprise determining, for each of the one or more input features of the RL model, a value representing a contribution of each of the one or more input features to an output of the RL model. In some examples, generating the feature analysis may comprise determining, for each of the one or more input features of the RL model, a value of a reward function representing a contribution of each of the one or more input features to at least one reward component of the RL model. It will be understood that the feature analysis may be generated using one or more different techniques. For example, the feature analysis may be generated using a BEERL technique, a SHAP technique (e.g. GradientSHAP technique), a LIME technique, a Deeplift technique, and / or any other suitable technique for generating the feature analysis referred to herein.

[0089] As mentioned above, the second response (e.g. the feature analysis) may be generated using a BEERL technique (method). The use of a BEERL technique can provide for the generation of the second response in a local manner, meaning that a single explanation can be generated for each state of the at least one state referred to herein. When generating a global explanation, a BEERL technique can involve aggregating local explanations using a mean-absolute function. An example of such a mean-absolute function is provided below as Equation 1 :

[0090] Equation 1

[0091] The selected states produced by the first (e.g. APE) explanation can be explained (e.g. aggregated) using the BEERL technique to generate the second response. For example, for the at least one state on which the first response is based, the BEERL technique can explain what (e.g. important) features and / or reward components are specific for that at least one state. Thus, a query response based on such a second response can enable the user (e.g. an expert) to evaluate whether important elements for a particular condition make sense or not (i.e. instead of knowing only the action taken).

[0092] In a particular example, in which the generation of the first response utilises an APE technique and the generating of the second response utilises a BEERL technique, the combination of the first and second response can provide a more useful contrastive explanation. For example, for the first response, APE can be used to provide a statistical analysis such as "the agent does not take action b because feature 1 is below average and feature 2 is above average”. However, there may a situation in which such an explanation is not clear (e.g. for the user). Thus, the generation of the second response, using BEERL, enables the generation of a contrastive explanation for the same at least one state which complements (e.g. clarifies) the first response. The combination of the first response and the second response can produce a query response such as "the agent predicts that action b has compromised performance for reward component Y and Z, while the chosen action has advantage in all reward components”. Without this additional information, a user may wrongly perceive the first response. This could lead to a lack of trust in the (e.g. deployed) RL model. Therefore, the generation of the second (e.g. contrastive) explanation (e.g. via BEERL) can clarify the justification for the RL model selecting a non-dominance action. The second response (e.g. via BEERL) can explain (e.g. show) why choosing a certain action has more advantage over other possible actions (e.g. which may not be clear from the first response (e.g. via APE) alone).

[0093] In some examples, the user query referred to herein can be associated with one of a first user type and a second user type. The first user type and the second user type can be different. In some examples, the first user type may correspond to an end user of the domain (e.g. an end user of a telecommunications network). In some examples, if the user query is associated with the first user type, the method may comprise truncating the method to omit the step of generating the second response, and providing, to the user associated with the user query, a query response based on the generated first response to the user query (e.g. only). Thus, the method (e.g. and the content of the response query) described herein can be tailored according to the type of user that makes the user query. For example, the second user type may correspond to an internal developer (e.g. of the domain referred to herein), and / or a monitor of the RL model. Providing a query response based on both the generated first response and second response can be useful for such a second user type, as they may require more detailed analysis of RL model behaviour(e.g. than an end user).

[0094] As mentioned herein, in some examples, the domain referred to herein is a communications network. In some of these examples, the at least one state may correspond to at least one of: a value of a network coverage parameter; a value of a network capacity parameter; a value of a network congestion parameter; a network resource allocation; a network resource configuration; a network usage parameter; a network power consumption; a network power source; a network type; a network parameter of a neighbour communications network cell; a value of a network signal quality parameter; a value of a network signal interference parameter; a value of a network power parameter; a value of a network signal length; a network frequency band; a network frequency bandwidth a network antenna down-tilt angle; a network antenna vertical beamwidth; a network antenna horizontal beamwidth; a network antenna height; a network antenna beam width; a network geolocation; and a network inter-site distance.

[0095] The parameter of the neighbour communications network cell, mentioned above, may comprise any of the parameters mentioned herein. That is, in examples in which the at least one state corresponds to the network parameter of the neighbour communications network cell, the at least one state may correspond to any one or more of the values and / or parameters mentioned above in relation to the neighbour communications network cell. The neighbour communications network cell may comprise, for example, a neighbouring antenna of the communications network.

[0096] Alternatively, or in addition, in examples in which the domain is a communications network, the at least one corresponding action may correspond to at least one of: an allocation decision for a network resource; a configuration for a network node; a configuration for network equipment; a configuration for a network operation; a configuration for a network power consumption; a configuration for a network power supply; a configuration for prioritisation of network traffic; a configuration for cell association; a decision relating to provision of network services for a wireless device; and a configuration for an operation performed by a wireless device in relation to the network. In some examples, the at least one state may correspond to a network antenna down-tilt angle. In such examples, the at least one corresponding action may correspond to a configuration of a network node. In this case, the at least one corresponding action may comprise an uptilt of an antenna, a downtilt of the antenna, and / or not tilting the antenna.

[0097] In some examples, the environment referred to herein may comprise a robot. In such examples, the at least one state referred to herein may correspond to at least one of: a configuration of the robot; a location of the robot; a measurement of the environment; a location of an object; a configuration and / or a property of the object; a goal of the robot; a travel path of the robot; a safety level of the robot; a safety level of the environment; and a speed of the robot.

[0098] Alternatively, or in addition, the at least one corresponding action referred to herein may correspond to an action executed by the robot.

[0099] In some examples, the RL model referred to herein can be an explainable RL (XRL) model. In some examples, the RL model may be a trained RL model. For example, the RL model may be an RL model that has been trained using training data (e.g. associated with the domain and / or environment). In some examples, the trained RL model may be specific to the environment. For example, the RL model may be specific to a network system in cases in which the domain referred to herein is a network.

[0100] Although not illustrated in Figure 7, in some examples, the method may comprise initiating transmission of a request towards the environment. The request can be a request to configure the environment based on the first response and / or the second response. Thus, the generation of the first response and / or the second response can provide useful information which can be used to (re)configure the environment.

[0101] There is also provided a system comprising the first entity 10 described herein. A method performed by the system comprises the method as described with reference to Figure 7.

[0102] Figure 8 is a signalling diagram illustrating an exchange of signals in a system according to an embodiment. As illustrated in Figure 8, the system may comprise an environment 202, an RL model 204 (“rl_agent"), a memory 206 (e.g. a database), a user 208, a manager module 210 (“explainerjnanager”), a first module 212 ("ape_explainer”), and a second module 214 ("beerl_explainer”). Although Figure 8 illustrates the entities of the system as separate entities, it will be understood that some or all of the entities illustrated in Figure 8 may be combined (e.g. at the same location). For example, the manager module 210, the first module 212, and the second module 214 may all be comprised in a first entity 10 as defined herein. Thus, in some examples, the first entity 10 as referred to herein can be configured to perform the functionality of the manager module 210, the first module 212, and the second module 214. In some examples, the RL model 204 may be comprised in the first entity 10 as referred to herein.

[0103] As illustrated by arrow 216 of Figure 8, the RL model 204 may provide (e.g. transmit) information indicative of the RL model (e.g. one or more algorithms) to the second module 214. Thus, the second module can receive the information indicative of the RL model from the RL model 204. The information indicative of the RL model may comprise the configuration of the RL model and / or be indicative of the manner in which the RL model operates. In the example system illustrated in Figure 8, the RL model 204 may be an artificial intelligence (Al) model which recommends actions for execution on the environment, as described herein. The RL model 204 may be implemented as part of a cognitive software (CSW) product (e.g. running at a data center).

[0104] As illustrated by arrow 218 of Figure 8, the environment 202 may provide (e.g. transmit) information indicative of one or more states of the environment to the RL model 204. Thus, the RL model 204 can receive the information indicative of the one or more states from the environment 202. The one or more states of the environment may be referred to herein as one or more observations. In some examples, the environment 202 can be the system where the RL model 204 is trained and / or deployed. For example, in scenarios in which the domain referred to herein is a communications network, the environment 202 may be an area where user equipments (UEs) are spread and interact with one or more antennas (e.g. in base stations) of the network. In such an example, performance KPIs can be the at least one state (observation) referred to herein and the reward to be optimised (e.g. by the RL model), and the corresponding action can be an antenna tilting value.

[0105] As illustrated by arrow 220 of Figure 8, the environment 202 may provide (e.g. transmit) the information indicative of the one or more states of the environment 202 to the memory 206. Thus, the information indicative of the one or more states can be stored in the memory 206. As illustrated in Figure 8, the memory 206 may be a database. The memory 206 may store all (e.g. relevant) information about RL model elements. More specifically, the memory 206 may store the at least one states and the at least one corresponding action referred to herein. Thus, the memory 206 can store state observations and a resulting action taken (e.g. as recommended by the RL model 204). As illustrated by arrow 222 of Figure 8, the environment 202 can provide (e.g. transmit) the information indicative of the one or more states of the environment to the second module 214. Thus, the second module 214 can receive the information indicative of the one or more states of the environment from the environment 202.

[0106] As illustrated by arrow 224 of Figure 8, the RL model can determine (e.g. compute) a corresponding action to recommend to be executed on the environment 202 based on the at least one state of the environment 202. As illustrated by arrow 226 of Figure 8, the RL model 204 can provide information indicative of the corresponding action to the environment 202. Thus, the environment 202 can obtain the information indicative of the corresponding action from the RL model 204. As illustrated by arrow 228 of Figure 8, the RL model 204 can provide information indicative of the corresponding action to the memory 206. Thus, the information indicative of the corresponding action can be stored in the memory 206. As illustrated by arrow 230 of Figure 8, the RL model 204 can provide information indicative of the corresponding action to the second module 214. Thus, the second module 214 can obtain the information indicative of the corresponding action from the RL model 204.

[0107] As illustrated by arrow 232 of Figure 8, the environment 202 may execute the corresponding action recommended by the RL model 204. For example, in the example in which the domain is a communications network, the corresponding action recommended by the RL model 204 may be to (re)configure an antenna tilt of an antenna in the communications network.

[0108] As illustrated by arrow 234 of Figure 8, the environment 202 may provide information indicative of a reward and / or a feedback, associated with executing the corresponding action recommended by the RL model 204, to the RL model 204. Thus, the RL model 204 can receive the information indicative of the reward (e.g. and / or feedback) from the environment 202. The information indicative of the reward may comprise a reward value associated with a reward component. As illustrated by arrow 234 of Figure 8, the environment 202 may provide the information indicative of the reward to the memory 206. Thus, the information indicative of the reward can be stored in the memory 206.

[0109] As illustrated by arrow 238 of Figure 8, the RL model 204 may transmit a request for training data towards the memory 206. Thus, the memory 206 can receive the request from the RL model 204. As illustrated by arrow 240 of Figure 8, the memory 206 can transmit a response to the request towards the RL model. Thus, the RL model 204 can receive the response form the memory 206. The response to the request can comprise training data for the RL model 204. As illustrated by arrow 242 of Figure 8, the RL model may be trained based on the training data received from the memory 206. As such, the RL model 204 can be updated based on data associated with the environment 202. The method steps illustrated by arrows 216 to 242 of Figure 8 can be referred to herein as a training phase of the RL model 204.

[0110] As illustrated by arrow 244 of Figure 8, the second module 214 may generate an initial response (e.g. explanation) which represents an attributive explanation of the corresponding action obtained with reference to arrow 230 of Figure 8. As illustrated by arrow 246 of Figure 8, the second module 214 may provide the initial response (e.g. explanation) to the memory 206. Thus, the initial response may be stored in the memory 206.

[0111] As illustrated by arrow 248 of Figure 8, the manager module 210 can obtain a user query comprising a condition associated with an operation of the RL model, as defined herein. As illustrated in Figure 8, the user query can be received from the user 208. In some examples, the user query may correspond to a question. Alternatively, or in addition, the user query can comprise information indicative of a user type of the user 208. The user type can vary depending on the user making the user query. For example, the user type may comprise a first user type and / or a second user type. The first user type and the second user type can be different. The second user type may comprise an internal developer user and / or a model monitor user. An internal developer user may correspond to a developer of the domain referred to herein and / or the environment referred to herein. An internal developer user may usefully take help from the query response as defined herein, for example, during an offline training of the RL model 204 to achieve a correct analysis. A model monitor user may use the user query defined herein to make sure that the RL model 204 is behaving as expected (e.g. in production). The first user type referred to herein may comprise an end user type. An end user type may only require a high level insight into the performance of the RL model 204 (e.g. in order to trust the RL model 204 functionality). As mentioned herein, a query response can be tailored (e.g. to provide different levels of explain-ability) based on the user type of the user 208. Further explanation of potential user types can be found below.

[0112] For an internal developer user type the provision of a query response based on the generated first and second responses referred to herein may assist in developing better RL models (e.g. via offline design and / or training based on, for example, simulations and / or datasets). For an internal developer user type, it can be useful to be provided with (e.g. very) detailed and insightful information. The generated second response, as referred to herein can be especially useful for this purpose. In some examples, a reward weight mechanism may be incorporated into the generation of the second response. In these examples, the query response (e.g. as based on the second response) can be used to tune the RL model 204. For example, a reward weight can be adjusted to suit specific needs and / or improve performance of the RL model 204. In some examples, the generation of the first response (e.g. via APE) can be used to find a certain condition to be analysed. Further, the combination of the first response and the second response referred to herein provides detail and thorough insights for checking RL model 204 behaviour in different situations, especially when critical or suboptimal condition occurs.

[0113] For a model monitoring user type, the provision of a query response as defined herein helps in monitoring the performance, or behaviour in production, of the RL model 204. More specifically, basing the query response on the second response (e.g. BEERL explanation) can inform a model monitoring user as to the contribution of each input feature for each decision made by the RL model 204. As such, the user can be provided with insights on how each input feature affects the RL model's decision making. Furthermore, contrastive explanation generated through the first and second responses can show the rationale of why an action is taken instead of another. As such, a user can be provided with better clarity in justifying and / or debugging RL model 204 behaviour.

[0114] For an end user user type, the provision of a query response as defined herein can help in increasing customer confidence in RL model decisions. In some examples, it may be useful to omit certain information about the RL model 204 when providing a query response to an end user. That is, although it is useful to provide understandable information to the end user (e.g. that is closer to the expertise of the end user), it may not be as useful to provide an end user with complex diagrams and / or charts (e.g. which are more suitable for deep debugging capabilities). The generated first response as described herein may fit the needs of an end user who desires insights into the performance of the RL model 204.

[0115] As illustrated by arrow 250 of Figure 8, the manager module 210 may provide the condition as referred to herein to the first module 212. Thus, the first module 212 may obtain the condition from the manager module 210. In some examples, the manager module 210 may provide the question referred to herein to the first module 212. As such, the manager module 210 may initiate the generation of the first response (e.g. APE explanation). In some examples, the manager module 210 may determine the user type of the user 208 and determine the manner in which the query response will be provided based on the user type. Although some examples of user type have been described herein, it will be understood that the techniques described herein can be implemented in respect of other (e.g. first and / or second) user types.

[0116] As illustrated by arrow 252 of Figure 8, the first module 212 may transmit a request towards the memory 206. The request can be for a plurality of states associated with the environment 202 and / or a plurality of corresponding actions, as defined herein. As illustrated by arrow 254 of Figure 8, the memory 206 may provide information indicative of the plurality of states and / or the plurality of corresponding actions towards the first module 212. Thus, the first module 212 can obtain the plurality of states associated with the environment, and / or the plurality of corresponding actions (e.g. from the memory 206). As such, in some examples, the first module 212 may obtain (e.g. take) data (e.g. state and action pairs) from the memory 206 (e.g. where encountered situations are stored). The first module 212 may obtain (e.g. take) all data stored in the memory 206 in order to generate the first response as referred to herein.

[0117] As illustrated by arrow 256 of Figure 8, the first module 212 may identify (e.g. filter) based on the condition, the at least one state, as referred to herein, from the plurality of states, and / or the at least one corresponding action, as referred to herein, from the plurality of corresponding actions. The identification may be based on a binary representation of the condition as described herein.

[0118] As illustrated by arrow 258 of Figure 8, a first response to the user query may be generated by the first module 212. The first response can be generated based on a statistical analysis of the at least one state and the at least one corresponding action. As illustrated in Figure 8, the first response may be generated using APE. However, it will be understood that this is merely an example of the first response, and that the first response can be generated using other types of statistical analysis. The first response (e.g. APE explanation) may be generated for any user type described herein.

[0119] As illustrated by arrow 260 of Figure 8, the first module 212 may provide the first response to the manager module 210. Thus, the manager module 210 may receive the first response from the first module 212. The manager module 210 may determine how to proceed with the method in response to obtaining the first response. For example, as illustrated by arrow 262 of Figure 8, if the user query is associated with a first user type as defined herein, the manager module 210 may filter the first response (e.g. APE explanation) to remove sensitive and / or confidential information (e.g. data) from the first response. Thus, in some examples, the query response can be filtered to remove sensitive and / or confidential information.

[0120] An end-user may not be permitted to have full access to the system of Figure 8, and the corresponding information and / or data. Therefore, if the user query is associated with a first user type (e.g. an end user), only the first response (e.g. APE explanation) may be generated. That is, in some examples, if the user query is associated with the first user type, the method may be truncated to omit the step of generating the second response. The end-user may more easily consume the query response based on the first response. As mentioned above, in some examples, the generated first response (e.g. APE explanation) may comprise some sensitive data (e.g. that is only needed for a developer and / or model monitor). In examples in which the first response comprises one or more predicates, each predicate may be associated with an indicator to differentiate information that can or cannot be comprised in the query response (e.g. depending on the user type referred to herein).

[0121] In examples in which the user query is not associated with the first user type (e.g. the user query is associated with the second user type as defined herein), the manager module 210 may initiate provision of information from the first module 212 to the second module 214 (e.g. such that the second module 214 can generate the second response as referred to herein). As illustrated by arrow 264 of Figure 8, the first module 212 may provide information indicative of the at least one state of the environment, as defined herein, to the second module 214. Thus, the second module 214 can receive the information indicative of the at least one state of the environment from the first module 212. As such, the (e.g. selected) at least one state (e.g. selected by the first module 212 based on the condition) can be forwarded to the second module 214.

[0122] As illustrated by arrow 266 of Figure 8, the second module 214 may generate the second response to the user query. The second response can comprise information indicative of a feature analysis of one or more input features of the RL model 204. The one or more input features can be associated with the at least one state referred to herein.

[0123] The second response can represent a contrastive explanation of the user query. For example, the second module 214 (e.g. BEERL explainer) may generate an attributive explanation of different action(s) (e.g. by running inference without executing any actions in the environment 202). In some examples, the user query may correspond to different types of question. For example, the user query may correspond to a noncontrastive question. In these examples, the second module 214 may obtain (e.g. take) previously generated second responses (e.g. as stored in the memory 206). The second module 214 may aggregate the generated second responses to summarise general features and rewards contributions. The second module 214 may perform this aggregation using Equation 2 below.

[0124] Equation 2

[0125] As illustrated by arrow 268 of Figure 8 (e.g. in examples in which the user query is associated with a noncontrastive question), the second module 214 may provide information indicative of an index of the at least one state to the memory 206. In some examples, the second module 214 may request any (e.g. previous) generated second responses associated with the index of the at least one state. As illustrated by arrow 270 of Figure 8, the memory 206 may provide information indicative of the generated second responses associated with the index of the at least one state to the second module 214. Thus, the second module 214 can receive the information indicative of the generated second responses associated with the index of the at least one state from the memory 206. As illustrated by arrow 271 of Figure 8, the second module 214 can aggregate each of the generated second responses associated with the index of the at least one state. The aggregation can be used to generate the second response. Therefore, in some examples, from the at least one state of the environment 202, relevant generated second responses (e.g. BEERL explanations) can be obtained using an (e.g. state) index. As such, the data used to generate both the first and second responses can be aligned. Thus, in some examples, instead of getting a fixed global explanation, the second response can be generated based on an aggregation of previously generated second responses (e.g. obtained based on a set of indices). In some examples, the recorded states and second responses can be stored at the (e.g. same) memory 206. In some examples, generating the relevant second responses (e.g. BEERL explanations) can be done directly from the at least one state.

[0126] As illustrated by arrow 274 of Figure 8, the second module 214 can provide the second response to the manager module 210. Thus, the manager module 210 can obtain the second response from the second module 214. As illustrated by arrow 276, the manager module 210 can provide, to the user 208, the query response based on the generated first and second responses. Thus, the user 208 can obtain the query response from the manager module 210.

[0127] Therefore, as described herein, a query response to a user query can be provided which serves to explain the behaviour of an RL model. The generated second response to the user query, as described herein, complements and clarifies the generated first response to the user query, as described herein, by providing detailed insight into the contribution of the RL model decision making.

[0128] The method disclosed herein is described further below with reference to the use case as described with reference to Figure 1 above.

[0129] In particular, in some examples, the user query may be associated with a question. For example, the user query may be "When do you do <action>?”. In this example, the condition comprises information indicative of a queried action recommended by the RL model (i.e. "<action>”).

[0130] In a particular example, the user query may be "When do you make no change (no tilt)?”. In this example, the queried action recommended by the ML model is "no change” and / or "no tilt”. Thus, the condition comprised in the user query can comprise information indicative of "no change” and / or "no tilt”. Based on this condition, the first response may be generated as described herein. The first response may take the form below (i.e. in a natural language format):

[0131] "I make no change when cell has normal percent of bad traffic and cell neighbours have normal percent of bad traffic and overshooting is acceptable and cell edge coverage is acceptable and cell is not causing interference in its area and neighbours are not highly interfering with the cell (39.06%)” OR "cell has low RRC congestion and cell has normal percent of bad traffic and cell neighbours have normal percent of bad traffic and cell edge coverage is acceptable and neighbours are not highly interfering with the cell and cell is used for capacity (33.72%)” OR

[0132] "cell has low RRC congestion and cell has normal percent of bad traffic and cell neighbours have normal percent of bad traffic and cell edge coverage is acceptable and cell is not causing interference in its area and cell is used for capacity (32.31 %)”.

[0133] The percentages indicated above can correspond to the percentage likelihood that no change is recommended by the RL model in the corresponding states of the environment.

[0134] A second response, as referred to herein, can also be generated in response to the above exemplary user query as described below.

[0135] Figure 9 shows a graph illustrating information which may be comprised in the second response as referred to herein.

[0136] The information in Figure 9 can correspond to a generated second response to the user query "When do you make no change (no tilt)?”, as mentioned above. In the example of Figure 9, the second response may be generated using BEERL.

[0137] The information shown in Figure 9 illustrates aggregated feature importance for a no tilt action (e.g. from the user query mentioned above). It can see that most of the features (on the left hand side of the graph of Figure 9) only contribute to a multiband reward component and a congestion reward component. Furthermore, it can be seen from Figure 9 that the information (e.g. corresponding to the second response) is significantly different to the global explanation as illustrated in Figure 3. Indeed, even though the information shown in Figure 3 corresponds to more than 33% of a tilt action dataset (i.e. no tilt action), the compositions of feature contributions are different. For a no-tilt action, the information in Figure 9 shows that no feature contributes significantly to a legacy reward component. This explanation can be important for evaluating the RL model in a certain situation (e.g. when a no-tilt action is taken).

[0138] In some examples, the user query may be associated with a question. For example, the user query may be "What will you do on <condi tion>?” . In a particular example, the user query may be "What will you do when '1-01-1—'?”. In this examples, the condition comprises the predicate '1 -01-1—'. Based on this condition the first response may be generated as described herein. The first response may take the form below (i.e. in a natural language format): "I will downtilt if cell is used for coverage”

[0139] Statistics: matching states: 78 percent of actions taken in the matching states: downtilt 57.69% uptilt 12.82% make no change 29.49%

[0140] The corresponding predicates for the user query can be: rrcCongestionRate is high & badTrafficNeigh_norm is low & taOvershootingFactor is high & numCellsHighOverlapHighRsrpSrcAgg is high.

[0141] A second response, as referred to herein, can also be generated in response to the above exemplary user query as described below.

[0142] Figure 10 shows a graph illustrating information which may be comprised in the second response as referred to herein.

[0143] The information in Figure 10 can correspond to a generated second response to the user query "What will you do when '1 -01-1— '?”, as mentioned above. In the example of Figure 10, the second response may be generated using BEERL. The second generated response can complement the first generated response mentioned above.

[0144] In the example illustrated in Figure 10, the second response (e.g. BEERL explanation) shows an aggregated feature importance for the condition '1-01-1—' (e.g. rrcCongestionRate is high & badTrafficNeigh_norm is low & taOvershootingFactor is high & numCellsHighOverlapHighRsrpSrcAgg is high). It can be seen that the information illustrated in Figure 10 is different from that illustrated in Figure 3 and Figure 9.

[0145] In some examples, the user query may be "Why don't you do <action> in state <x>?”. In this example, the condition comprises information indicative of a queried state of the environment (i.e. “<x>”), and information indicative of a queried action recommended by the RL model (i.e. "<action>”). In a particular example, the user query may be "Why didn't you downtilt in state 5?”. In this example, the queried state of the environment is "state 5”, and the queried action recommended by the ML model is “downtilt”. Thus, the condition comprised in the user query can comprise information indicative of "state 5” and “downtilt”. Based on this condition the first response may be generated as described herein. The first response may take the form below (i.e. in a natural language format):

[0146] "I didn't downtilt because cell has low RRC congestion and cell neighbours have high RRC congestion and cell has normal percent of bad traffic and cell neighbours have high percent of bad traffic and cell has same or lower dl utilization as neighbour frequencies”.

[0147] Action frequency for states matching the explanation: matching states: 3 percent of actions taken in the matching states: uptilt 33.33% make no change 66.67%

[0148] A second response, as referred to herein, can also be generated in response to the above exemplary user query as described below.

[0149] Figure 11 shows a graph illustrating information which may be comprised in the second response as referred to herein.

[0150] The information in Figure 11 can correspond to a generated second response to the user query "Why didn't you downtilt in state 5?”, as mentioned above. In the example of Figure 11, the second response may be generated using BEERL. The second generated response can complement the first generated response mentioned above.

[0151] As illustrated in Figure 11, the generated second response can be indicative of one or more Q-values illustrating a contrastive explanation for the generated first response to the user query. More specifically, for the condition comprised in the user query, Figure 11 shows that choosing action "0_2” (e.g. choosing no tilt over downtilt) has more advantages (e.g. in terms of Q-values) compared to action “0_1” (e.g. choosing no tilt over uptilt). Showing a difference in Q-values is useful for the user. Additional information may be included in the second response (e.g. and thus the query response) as illustrated in Figure 12 below. Figure 12 shows a graph illustrating information which may be comprised in the second response as referred to herein.

[0152] The information in Figure 12 can correspond to a generated second response to the user query "Why didn't you downtilt in state 5?”, as mentioned above. In the example of Figure 12, the second response may be generated using BEERL. The second generated response can complement the first generated response mentioned above.

[0153] The information in Figure 12 shows a contrastive explanation for the user query which may be comprised in the generated second response (e.g. as a complement to the respective generated first response). Figure 12 shows that, for the condition comprised in the user query, choosing action "0_2” (e.g. choosing no tilt over downtilt) has more advantages (e.g. in terms of Q-values) compared to other actions such as “0_1” (e.g. choosing no tilt over uptilt). This information (e.g. explanation) shows the model's "expectation” on different reward components rather than comparing it with other states (e.g. as may be performed in generating the first response).

[0154] There is also provided a computer program comprising instructions which, when executed by processing circuitry (such as the processing circuitry 12 of the first entity 10 described herein), cause the processing circuitry to perform at least part of the method described herein. There is provided a computer program product, embodied on a non-transitory machine-readable medium, comprising instructions which are executable by processing circuitry (such as the processing circuitry 12 of the first entity 10 described herein) to cause the processing circuitry to perform at least part of the method described herein. There is provided a computer program product comprising a carrier containing instructions for causing processing circuitry (such as the processing circuitry 12 of the first entity 10 described herein) to perform at least part of the method described herein. In some embodiments, the carrier can be any one of an electronic signal, an optical signal, an electromagnetic signal, an electrical signal, a radio signal, a microwave signal, or a computer-readable storage medium.

[0155] In some embodiments, the first entity 10 functionality described herein can be performed by hardware. Thus, in some embodiments, the first entity 10 described herein can be a hardware entity. However, it will also be understood that optionally at least part or all of the first entity described herein can be virtualised. For example, the functions performed by the first entity 10 described herein can be implemented in software running on generic hardware that is configured to orchestrate the first entity functionality described herein. Thus, in some embodiments, the first entity 10 described herein can be a virtual entity. In some embodiments, at least part or all of the first entity functionality described herein may be performed in a network enabled cloud. Thus, the method described herein can be realised as a cloud implementation according to some embodiments. The first entity functionality described herein may be distributed, e.g. the first entity functionality may be performed by one or more different entities.

[0156] Therefore, as described herein, there are provided improved techniques for handling queries associated with a RL model. The techniques enable a user to be provided with a more detailed (e.g. thorough) explanation of a user query. By providing a query response based on both a first response and a second response, the query response is indicative of complementary explanations which detail both statistical contributions and internal model contributions to the RL model behaviour. As such, the techniques described herein enable a user to query different situations and potentially identify unwanted behaviour and analyse said behaviour. In scenarios in which the user query is a contrastive query, basing the query response on both the first and second response s described herein enables the query response to clarify a justification for the RL model recommending a non-dominance action. For example, the query response can indicate why recommending a particular action is more advantageous over other possible actions. Moreover, in some examples of the techniques described herein, the query response can be tailored to suit the type of user associated with the user query.

[0157] It should be noted that the above-mentioned embodiments illustrate rather than limit the idea, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. The word "comprising” does not exclude the presence of elements or steps other than those listed in a claim, "a” or "an” does not exclude a plurality, and a single processor or other unit may fulfil the functions of several units recited in the claims. Any reference signs in the claims shall not be construed so as to limit their scope.

Claims

CLAIMS1. A computer-implemented method for handling queries associated with a reinforcement learning, RL, model (204), wherein the RL model (204) is configured to control an environment (202) within a domain, the method comprising: obtaining (102) a user query comprising a condition associated with an operation of the RL model (204); obtaining (104, 254), based on the condition, at least one state of the environment (202) and at least one corresponding action, wherein the at least one corresponding action corresponds to an action recommended by the RL model (204) to be executed on the environment (202) in the at least one state; generating (106, 258) a first response to the user query, wherein the first response is generated based on a statistical analysis of the at least one state and the at least one corresponding action; generating (108, 266) a second response to the user query, wherein the second response comprises information indicative of a feature analysis of one or more input features of the RL model (204), and wherein the one or more input features are associated with the at least one state; and providing (110, 276), to the user (208) associated with the user query, a query response based on the generated first and second responses to the user query.

2. The method as claimed in claim 1, wherein the at least one state and the at least one corresponding action are obtained from a memory (206).

3. The method as claimed in claim 2, wherein the memory (206) is a database of the domain.

4. The method as claimed in any of the preceding claims, wherein obtaining (104, 254) the at least one state and the at least one corresponding action comprises: obtaining (254) a plurality of states associated with the environment (202), and a plurality of corresponding actions; and identifying (256), based on the condition, the at least one state from the plurality of states, and the at least one corresponding action from the plurality of corresponding actions.

5. The method as claimed in claim 4, wherein the method comprises: generating a binary representation of the condition; and identifying (256) the at least one state from the plurality of states comprises: identifying, based on the binary representation of the condition, the at least one state from a binary representation of each state of the plurality of states.

6. The method as claimed in any of the preceding claims, wherein the condition comprises information indicative of a queried state of the environment (202), and wherein the at least one state is associated with the queried state.

7. The method as claimed in any of the preceding claims, wherein the condition comprises information indicative of a queried action recommended by the RL model (204), and wherein the at least one corresponding action is associated with the queried action.

8. The method as claimed in any of the preceding claims, the method comprising: generating the feature analysis of the one or more input features of the RL model (204).

9. The method as claimed in claim 8, wherein generating the feature analysis comprises: determining, for each of the one or more input features of the RL model (204), a value representing a contribution of each of the one or more input features to an output of the RL model (204).

10. The method as claimed in claim 8 or 9, wherein generating the feature analysis comprises: determining, for each of the one or more input features of the RL model (204), a value of a reward function representing a contribution of each of the one or more input features to at least one reward component of the RL model (204).11 . The method as claimed in any of the preceding claims, wherein the user query is associated with one of a first user type and a second user type, and wherein the first user type and the second user type are different.

12. The method as claimed in claim 11, wherein the first user type corresponds to an end user of the domain.

13. The method as claimed in claim 11 or 12, wherein, if the user query is associated with the first user type, the method comprises: truncating the method to omit the step of generating the second response; and providing, to the user associated with the user query, a query response based on the generated first response to the user query.

14. The method as claimed in any of the preceding claims, wherein the domain is a communications network.

15. The method as claimed in claim 14, wherein the at least one state corresponds to at least one of: a value of a network coverage parameter; a value of a network capacity parameter; a value of a network congestion parameter; a network resource allocation; a network resource configuration; a network usage parameter; a network power consumption; a network power source; a network type; a network parameter of a neighbour communications network cell; a value of a network signal quality parameter; a value of a network signal interference parameter; a value of a network power parameter; a value of a network signal length; a network frequency band; a network frequency bandwidth a network antenna down-tilt angle; a network antenna vertical beamwidth; a network antenna horizontal beamwidth; a network antenna height; a network antenna beam width; a network geolocation; and a network inter-site distance; and wherein the at least one corresponding action corresponds to at least one of: an allocation decision for a network resource; a configuration for a network node; a configuration for network equipment; a configuration for a network operation; a configuration for a network power consumption; a configuration for a network power supply; a configuration for prioritisation of network traffic; a configuration for cell association; a decision relating to provision of network services for a wireless device; and a configuration for an operation performed by a wireless device in relation to the network.

16. The method as claimed in claim 15, wherein the at least one state corresponds to the network antenna down-tilt angle, the at least one corresponding action corresponds to the configuration of the communications network node, and wherein the at least one corresponding action comprises: an uptilt of an antenna; a downtilt of the antenna; and / or not tilting the antenna.

17. The method as claimed in any of the preceding claims, wherein the environment (202) comprises a robot, wherein the at least one state corresponds to at least one of: a configuration of the robot; a location of the robot; a measurement of the environment (202); a location of an object; a configuration of the object; a goal of the robot; a travel path of the robot; a safety level of the robot; a safety level of the environment (202); and a speed of the robot; and wherein the at least one corresponding action corresponds to an action executed by the robot.

18. The method as claimed in any of the preceding claims, wherein the RL model (204) is an explainable RL, XRL, model.

19. The method as claimed in any of the preceding claims, wherein the RL model (204) is a trained RL model.

20. The method as claimed in claim 19, wherein the trained RL model is specific to the environment (202).21 . The method as claimed in any of the preceding claims, wherein the first response is a natural language response to the user query.

22. The method as claimed in any of the preceding claims, the method comprising: initiating transmission of a request towards the environment (202), wherein the request is a request to configure the environment (202) based on the first response and / or the second response.

23. A first entity (10) comprising: processing circuitry (12) configured to operate in accordance with any of claims 1 to 22.

24. A first entity (10) as claimed in claim 23, wherein: the first entity (10) comprises: at least one memory (14) for storing instructions which, when executed by the processing circuitry (12), cause the first entity (10) to operate in accordance with any of claims 1 to 22.

25. A computer program comprising instructions which, when executed by processing circuitry, cause the processing circuitry to perform the method according to any of claims 1 to 22.

26. A computer program product, embodied on a non-transitory machine-readable medium, comprising instructions which are executable by processing circuitry to cause the processing circuitry to perform the method according to any of claims 1 to 22.

Images