Computer-implemented method, computing device, radio resource scheduler for a wireless communication network, and computer software

Abstract

The present invention discloses a computer-implemented method for selecting, at a time step, an action to be performed by a system in a current state among a set of possible states of the system from a set of possible actions, the method comprising: selecting an action that causes the system to most likely follow a path from the current state that corresponds to a sequence of actions and state transitions in successive time steps that have a payoff of at least a predetermined value; and causing the system to execute the action.

Description

[Technical Field]

[0001] The present disclosure relates to the field of selecting actions to be performed by a system, and more particularly, to the field of selecting optimal actions for a system when performing actions whose outputs are not deterministically known. [Background technology]

[0002] Many systems can be modeled to determine the actions the system should take to optimize its output. Examples of such systems include robots, schedulers, or more generally, any physical system that can perform actions to achieve a goal.

[0003] In many systems, the output of the same action may be different outputs. For example, if a robot is commanded to search for an object when its battery is low, it is usually not possible to know in advance whether the robot will find the object or stop because its battery has completely discharged. In another example, if a scheduler in a wireless communication network provides a user terminal with radio resources to transmit a message, the user terminal may either succeed in transmitting the message or fail to transmit the message, depending on unpredictable network conditions. Certain outputs, such as the robot finding the object or successfully transmitting the message, may be considered desirable, while other outputs, such as the robot's battery being completely discharged or failing to transmit the message, may be considered undesirable. Therefore, improving the probability of obtaining the most desirable output is a common objective of system control.

[0004] However, the situation becomes complicated when continuous operation leads to long-term outputs that cannot be deterministically predicted. For example, a scheduler in a wireless communication network may grant radio resources to various user terminals without knowing in advance whether their messages will be successfully transmitted. On the other hand, some user terminals may be bound by application constraints (e.g., constraints from applications requiring message transmission) and may fail to comply with the constraints if they transmit messages too late. This is considered a highly undesirable output. Therefore, optimizing the decision process of such a system is an exhaustive problem.

[0005] Markov Decision Processes (MDPs) provide a framework for solving such problems. MDPs model systems using actions, rewards, and state transition probabilities. At each time step, the system is in a current state, in which it can perform multiple different actions. Once an action is performed, a state transition from the current state to one or more subsequent states can occur with varying probabilities. The probability that a state transition occurs depends on the current state and the action being performed.

[0006] Each state transition is also associated with a reward. The reward can be positive or negative depending on the desirability of the state transition. From this, various paths of successive actions and state transitions can be defined. Each path can be associated with a payoff defined according to the successive rewards of the state transitions on that path, and a probability corresponding to the combined probabilities of the state transitions on that path. Thus, the MDP framework can associate with each path a payoff that defines whether the long-term outcome of that path is desirable or not, and the probability that that path will occur when executing the sequence of actions in the successive states of the path.

[0007] An MDP-based decision algorithm is described, for example, in Non-Patent Document 1. An MDP-based decision algorithm typically selects an action in the current state that maximizes the expected payoff. More specifically, the algorithm calculates the average payoff for each state, and for each possible action in the current state, determines the payoff and associated probability associated with each possible subsequent state. The algorithm then selects the action that provides the highest expected payoff.

[0008] Such decision processes generally do not take into account the probability of actions that result in potentially catastrophic outcomes when they increase the expected gain of the system. Indeed, actions associated with high expected gains may result in only paths with high gains, but may also result in many paths with very high gains, or a small number of paths with very low gains with low probability. Thus, a decision process based on maximum expected gains fails to minimize the likelihood of potentially catastrophic outcomes.

[0009] Therefore, in systems where the long-term effects of each action are not deterministically known in advance, a decision process is needed that reduces the probability of selecting an action that will have very negative long-term consequences. [Prior art documents] [Non-patent literature]

[0010] [Non-Patent Document 1] “Reinforcement Learning, An Introduction” by RS Sutton and AG Barto (2nd ed., The MIT Press, 2020) Summary of the Invention [Problem to be solved by the invention]

[0011] The present disclosure improves this situation. [Means for solving the problem]

[0012] A computer-implemented method is proposed for selecting, at a time step, an action from a set of possible actions to perform on a system in a current state of the set of possible states of the system, the method comprising: selecting an action that causes the system to most likely follow a path from the current state that corresponds to a sequence of actions and state transitions in successive time steps that have a payoff of at least a predetermined value; and causing the system to execute the action.

[0013] "System" refers to a representation of a real-world system that can be modeled according to various states and that can perform various actions, each of which causes the system to switch from an initial state to one or more possible subsequent states.

[0014] A "path" refers to a set of consecutive transitions between consecutive states of a system at consecutive time steps, where each transition may or may not be associated with an action.

[0015] "A system follows a path" means that the sequence of states and state transitions of the system over successive time steps corresponds to the sequence of states and state transitions defined by the path.

[0016] "Causing the system to follow a path with a certain probability" means that when the system is in a certain state and performs a certain action, the system is expected to have a given probability of following that path.

[0017] "Gain" refers to a value associated with a path, the value indicating the degree to which the sequence of states defined by the path corresponds to a good or bad outcome.

[0018] A "path resulting from the current state during the execution of an operation" refers to a path whose initial state is the current state and whose first transition is possible from the current state during the execution of an operation.

[0019] This allows for the selection of an action that exhibits the lowest probability of producing a gain below a predefined threshold in a system where each action may produce a different outcome with a defined probability.

[0020] This therefore allows for the taking of actions most likely to avoid potentially negative outcomes in a system where the long term effects of each action are not deterministically known in advance.

[0021] In another aspect, a computing device is proposed, comprising at least one processing unit configured to perform at least a part of the methods defined herein.

[0022] In another aspect, a radio resource scheduler for a wireless communication network is proposed, comprising at least one processing unit configured to implement at least a part of the method defined herein.

[0023] In another aspect, computer software is proposed comprising instructions for carrying out at least part of the methods defined herein when the software is executed by a processor.

[0024] In another aspect, a computer-readable non-transitory storage medium is proposed on which software is registered to perform the methods defined herein when the software is executed by a processor.

[0025] The following features can optionally be implemented separately or in combination with each other.

[0026] Advantageously, said selection is based on obtaining a transition model that defines, for at least one state transition from an initial state to a subsequent state between two successive time steps, the probability of making the transition from the initial state to the subsequent state upon execution of an action belonging to a set of actions executable in the initial state and the reward associated with the transition, and calculating, for a plurality of paths, the payoffs of these paths and the probabilities of these paths using at least the transition model.

[0027] "Transition model" refers to a model that defines state transitions and determines the probabilities and rewards associated with each state transition, where at least one of the probabilities for a state transition depends on the action taken by the system in a state.

[0028] "Reward" refers to a value that indicates the desirability of a transition. If the transition produces a desirable outcome, the reward is positive; if the transition produces an undesirable outcome, the reward is negative.

[0029] "Obtaining" refers to various means of obtaining the probability and payoff for each path of a plurality of paths. For example, the obtaining step may include: - Calculating probabilities and payoffs - Reading or receiving pre-calculated probabilities and payoffs during the training phase may include:

[0030] This allows the probabilities and outcomes associated with possible transitions to be taken into account in order to determine the path gain, thus making it possible to obtain a path gain calculation that accurately takes into account the evolution of the state of the system.

[0031] These features are advantageous in themselves, and a method is also proposed which includes only the steps described above.

[0032] Therefore, a computer-implemented method is also proposed, comprising: obtaining a transition model that defines, for at least one state transition from an initial state to a subsequent state between two successive time steps, the probability of making the transition from the initial state to the subsequent state upon execution of an action belonging to a set of actions executable in the initial state and the reward associated with the transition; and calculating, for a plurality of paths, the payoffs of the multiple paths and the probabilities of the multiple paths using the transition model.

[0033] Advantageously, the payoffs of the multiple paths are calculated as the cumulative rewards of a series of successive state transitions and actions of each path of the multiple paths at successive time steps, and the probabilities of the multiple paths are calculated as the combined probabilities of each combination of state transitions and actions of each path of the multiple paths at successive time steps.

[0034] "Cumulative reward" refers to the sum of rewards corresponding to successive state transitions in a path. The reward may depend only on the state transitions, or on both the state transitions and the corresponding actions. The sum may be a weighted sum. For example, the weight associated with each reward may decrease with each time step.

[0035] "Combinatorial probability" refers to the probability of a path defined as a combination of state transition probabilities. Combinatorial probability is usually the product of the probabilities of successive state transitions.

[0036] This allows all transitions within a finite or infinite time window to be used to determine the path gain and probability, and therefore allows reliable calculations of the path gain and probability to be obtained.

[0037] Advantageously, a binning rule is applied to the reference gains of the paths originating from each state to determine the size of a reference set associated with each state, the reference set being a predetermined size of the gains of the paths originating from the state and the probabilities of these paths.

[0038] A "binning rule" refers to a rule that merges multiple paths associated with similar gains into a single path with a probability equal to the cumulative probability of the paths belonging to the bin.

[0039] This allows limiting the number of paths considered by merging paths associated with similar gains, thereby limiting the computational complexity while obtaining reliable results for gains and probabilities.

[0040] Advantageously, calculating the gains and probabilities using the model includes iteratively calculating a reference set of states including, for each state in the set of possible states, a reference gain for a path that results from the state when the action is performed and a reference probability for causing the system to follow the path that results from the state when the action is performed, the iterative calculation applying a binning rule in each iteration.

[0041] This allows the payoffs and probabilities of paths starting from each possible state to be calculated recursively while maintaining the size of the set representing the probabilities and payoffs of paths starting from each state, thereby enabling recursive calculations even when the number of possible paths is large.

[0042] Advantageously, the iterative computation includes: for each state in the set of possible states, obtaining an initial reference set; for each state in the set of possible states, selecting an action to perform at that state according to the policy until a stopping criterion is met; for each subsequent state to which the probability of transitioning from that state using the selected action is non-zero, redefining the initial reference set for that state by concatenating the reference gain in the reference set of that subsequent state multiplied by a discount factor, added to the reward for transitioning from that state to the subsequent state, and the reference probability of the subsequent state multiplied by the non-zero probability of the transition; and reducing the size of the reference set for that state to a predetermined size by applying a binning rule to the reference gain of that state.

[0043] A "policy" refers to a set of rules that define what actions should be taken in what state of the system.

[0044] This provides an efficient way of performing recursive computations while ensuring that stopping criteria are respected at the end of the computation.

[0045] Advantageously, the iterative calculation comprises: for each state in the set of possible states, obtaining an initial reference set; for each state in the set of possible states, obtaining, for each possible action at that state, the probability of progressing to each subsequent state upon execution of the possible action at that state until a stopping criterion is met; selecting from among the possible actions an action that causes the system to most likely follow from that state a path corresponding to a sequence of actions and state transitions at successive time steps having a payoff of at least a predefined value; for each subsequent state for which the probability of transitioning from the state using the selected action is non-zero, redefining the initial reference set for that state by concatenating the reference gain in the reference set of that subsequent state multiplied by a discount factor, added to the reward of the transition from that state to the subsequent state, and the reference probability of the subsequent state multiplied by the non-zero probability of the transition; and reducing the size of the reference set for that state to a predefined size by applying a binning rule to the reference gain of that state.

[0046] This makes it possible to obtain an optimized policy with the aim of maximizing the probability of selecting an action that brings a payoff higher than or at least equal to the threshold.

[0047] Advantageously, the stopping criterion is selected from the group comprising at least one of the following criteria: a maximum number of iterations and a convergence criterion of the gain between two successive iterations.

[0048] Advantageously, the transition model belongs to the Markov decision process model, and the payoff of a path is calculated as a weighted sum of the rewards of each successive transition in the path, where the weight is a discount factor between 0 and 1 raised to the power of the transition index.

[0049] The "discount factor" refers to a factor that reduces the relative importance of rewards corresponding to more time steps depending on the time difference from the current time step.

[0050] The Markov decision process model provides an efficient way to calculate the payoff, where successive rewards are considered and the weighting factors associated with the rewards decrease over time.

[0051] Advantageously, the step of using the transition model further comprises using a machine learning model trained to predict, from the states of the paths resulting from that state, the path probabilities having gains belonging to each interval among a plurality of intervals of possible gains.

[0052] This allows only the trained model to be stored, thus providing accurate predictions of probabilities while reducing the size of the model parameters.

[0053] Advantageously, the machine learning model is iteratively trained using an error signal calculated as a cost function based on predictions of path probabilities from an initial state and from a subsequent state multiplied by the probability of transitioning from that initial state to that subsequent state when performing an action that causes the system to most likely follow a path from that initial state that has a gain of at least a predetermined value.

[0054] This allows machine learning models to be efficiently trained and converge to predictions that are consistent with optimal action choices.

[0055] Advantageously, the iterative calculation comprises performing a loop of selecting a plurality of states in a set of possible states until a stopping criterion is met, and for each state in the plurality of states, calculating using the machine learning model a prediction of a path probability from the initial state, and for each possible action in that state, obtaining from the transition model the probability of progressing to each subsequent state upon execution of that possible action and the associated reward in the transition model, calculating using the machine learning model a prediction of the path probability from each of the subsequent states, and from the prediction of the path probability from each of the subsequent states and the probability of progressing to each subsequent state upon execution of the possible action and the associated reward, The method includes performing a loop of calculating predictions of path probabilities from an initial state when a possible action is performed, selecting an action from the predictions of path probabilities from the initial state when a possible action is performed that will cause the system to most likely follow a path from the initial state that has a gain of at least a predetermined value, calculating values of a cost function based on the predictions of path probabilities from the initial state when a possible action is performed that will cause the system to most likely follow a path from the initial state that has a gain of at least a predetermined value, and updating the machine learning model based on multiple values of the cost function calculated for the multiple states.

[0056] This provides an effective way to calculate the path probability for each action in each state and train a machine learning model to decide which action to choose.

[0057] Advantageously, the method is implemented by a resource allocation device of the system, the set of possible actions corresponding to possible resource allocations in the states, and the rewards being defined according to the results of the resource allocations and the results of the state transitions of the system.

[0058] A "resource allocation device" refers to a module that can allocate physical resources of a physical system or computing resources of a computing system.

[0059] This allows the resource allocation device to allocate resources in a manner that reduces the probability of resource allocations that are detrimental to the system.

[0060] Advantageously, the method is implemented by a radio resource scheduler of a wireless communication network, wherein the set of possible actions corresponds to possible allocations of radio resources to users of the wireless communication network, at least one probability of making a state transition depends on a probability that a user of the wireless communication network will successfully transmit a packet over the network when the radio resources are allocated to the user, and at least one reward depends on compliance with a time constraint for the transmission of the at least one packet.

[0061] A "time constraint for the transmission of at least one packet" refers to a constraint on the time at which one or more packets should be transmitted. The time constraint may be, for example, a constraint on a time limit for transmitting and / or receiving a packet, such as a round trip time (RTT), a packet delay budget (PDB), etc. The time constraint may be, for example, a time constraint formulated by an application that requires the transmission of a packet. In such a case, the time constraint may be referred to as an "application constraint."

[0062] This enables the radio resource scheduler to allocate radio resources to users of the wireless communication network and reduces the potential for adverse effects related to failure to meet telecommunication constraints.

[0063] Other features, details and advantages are set forth in the following detailed description and in the figures. [Brief explanation of the drawings]

[0064] [Figure 1]FIG. 1 illustrates an example of payoffs as a function of policies in an exemplary system with a mobile robot modeled according to an MDP framework in which the present invention can be implemented for control. [Figure 2] FIG. 1 illustrates an example of a radio resource scheduler for a wireless communication network in which the present invention may be implemented for control. [Figure 3] FIG. 1 illustrates an example computer-implemented method according to embodiments of the present invention. [Figure 4] FIG. 10 illustrates a first example calculation of reference gains and reference probabilities according to one embodiment. [Figure 5] FIG. 10 illustrates a second example calculation of reference gains and reference probabilities according to one embodiment. [Figure 6] FIG. 10 illustrates a third example calculation of reference gains and reference probabilities according to one embodiment. [Figure 7] FIG. 2 is a diagram illustrating an example of multiple paths in an embodiment of the present invention. [Figure 8] FIG. 2 depicts a section of a trellis illustrating possible paths in multiple embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0065] To facilitate presentation of the present invention, some of the embodiments are presented with respect to a model conforming to a Markov Decision Process (MDP) framework. However, it should be noted that this is provided only as a non-limiting example, and the present invention is not limited to this example framework. To facilitate presentation of the present invention, some of the notations conventionally used in the MDP framework are reminded below. It should be noted that the following definitions are provided only as non-limiting examples of definitions in some embodiments of the present invention.

[0066] In the MDP framework, the payoff at time t is, for example, G t and is expressed as follows:

number

[0067] In the above formula (1), -r i is the reward received at each time step i; - λ represents a discount, meaning that the weight allocated to the reward decreases over time towards 0, 0<λ<1.

[0068] It is also possible to calculate the gain within a finite time window, in which case λ=1 can be used.

[0069] In the MDP framework, a policy π represents one or more decision rules used at each time step. In other words, a policy π defines the actions to be taken by the system at each time step t depending on at least one state s∈S, where S is the set of possible system states the system can be in at time step t.

[0070] In a particular case, several different policies are available, so one of the problems to be solved in a system modeled by the MDP framework is to select the optimal policy from the set of possible policies.

[0071] A widely used solution in the prior art is to calculate the expected payoff E[G t The goal is to select a policy that maximizes [expected payoff]. The expected payoff is calculated based on the probability of making various state transitions in the model, depending on the actions taken by the policy at each time step. Here, the transition probabilities model the randomness in the system's environment.

[0072] Since a given policy π deterministically selects an action in a state, the value of a state s∈S obtained using a given policy π can be defined as the expected payoff when the system is in state s at time t and the given policy π is used:

number

[0073] Policy π * is considered optimal if:

number

[0074] In other words, a policy is considered to be an optimal policy if it provides a more desirable value for each possible state than possible alternative policies.

[0075] Optimal policy π * Under this condition, the value of the state s can be expressed as the following equations (4) and (5) via the Bellman equation.

number

[0076] where: -A s is the set of allowable actions in state s; - p(s j |s,a) is the transition from state s to state s given action a. j represents the transition probability to ; - r(s j , s) is the transition from state s to state s j is the reward obtained when proceeding to; - R(s,a)=E[r|s,a]=Σ j p(s j |s,a)r(s j ,s).

[0077] π * Given , the optimal action at state s is obtained as follows:

number

[0078] Within this framework, it is possible to compare existing policies.

[0079] It is also possible to find a policy. In particular, it is possible to define a policy that optimizes the maximum expected payoff of the system. The standard technique for obtaining an optimal policy is to use a value iteration algorithm to iterate over the values.

number

number

[0080] This technique is described in, for example, Non-Patent Document 1.

[0081] In the above formula (7), v n teeth,

number

[0082] As explained above, this policy definition allows for obtaining a policy that provides the maximum expected payoff, but does not take into account whether the policy may occasionally result in a very low payoff. In other words, it is possible to obtain a maximum expected payoff policy that has a very high probability of resulting in a state transition with a high payoff, but may occasionally result in a state transition with a very low payoff. This situation is problematic because a low payoff may correspond to a catastrophic outcome in a physical system.

[0083] Next, refer to Figure 1.

[0084] FIG. 1 illustrates an example of payoffs as a function of policies in an exemplary system with a mobile robot modeled according to an MDP framework in which the present invention can be implemented for control.

[0085] The example in Figure 1 concerns the recycling robot problem. In this example, the problem is as follows: - A battery-powered mobile robot must collect empty cans; - The robot can be in two battery states that form a set of possible states S={s1="low",s2="high"}; - In each of the two states, the possible actions form a set of possible actions A = {a1 = “search”, a2 = “wait”, a3 = “recharge”}; - In the case of the action "wait", the robot i |s i , a2)=1 and stay in the same state, and the reward r wait get; - For the action "search", the robot moves from state "high" to state "low" with probability p(s1|s2,a1)=1-β and receives reward r with probability 1. search >r wait In the state “low”, the robot stays in the same state with probability β and gets reward r search otherwise, a negative reward r rescue Finally, in the case of the action "recharge", the robot obtains the probability p(s2|s i ,a3)=1, proceed to state “high” and get reward 0.

[0086] In the simulation, the following parameters were considered: r rescue =-1, r wait =0.4, r search = 0.9, β = 0.8, and discount λ = 0.8 are used.

[0087] In this example, two policies are considered: - Maximum expected payoff policy that performs the action "search" for both states, i.e. the robot always searches regardless of the state; - An alternative policy where the robot performs the action "wait" in the state "low".

[0088] Based on this, the payoffs and probabilities of each possible sequence of state transitions for the two policies are calculated when the robot is initially in state s1="low" and when it is initially in state s2="high".

[0089] Figure 1 shows the probability that a given policy will provide a gain higher than a certain value. More specifically, - The horizontal axis represents the number x; - The vertical axis is the probability P(G t >x).

[0090] Figure 1 shows the probability of different values being higher for different gains. - Pmax high represents the probability of the maximum expected payoff policy when the robot is initially in state s2 = “high”; - Pmax low represents the probability of the maximum expected payoff policy when the robot is initially in state s1 = “low”; - Palt represents the probability of alternative policies, whatever the initial state of the robot.

[0091] More specifically, Figure 1 shows the experimental complement cumulative density functions (CCDFs) obtained by simulating both policies.

[0092] Figure 1 shows the following: - The alternative policy always provides a payoff equal to 2; - The maximum expected payoff policy has a very high probability of delivering a payoff higher than 2, but also has a non-zero probability of delivering a payoff lower than 2 in a small number of cases.

[0093] When the maximum expected payoff policy results in a payoff lower than 2, this usually corresponds to the robot's battery discharging prematurely and the robot needing to be rescued early.

[0094] This example demonstrates that even though the maximum expected payoff policy generally provides good results, it can sometimes produce negative outcomes. In particular, the maximum expected payoff strategy is such that p(G t >α|s1) is not optimal.

[0095] The present invention can be used, for example, to improve the policies used by such robots.

[0096] Next, refer to Figure 2.

[0097] The present invention can be used to define policies for many different purposes, especially resource allocation policies. For example, the present invention can be used by a resource allocation device to optimize resource allocation. The resource allocation device can allocate physical resources of a physical system or computing resources of a computing system. For example, the resource allocation device can allocate computing resources, memory resources, network resources, etc.

[0098] The invention can be used, for example, to improve the control of radio resource schedulers in wireless communication networks.

[0099] FIG. 2 illustrates an example of a radio resource scheduler for a wireless communication network that the present invention may be implemented to control.

[0100] In the example of FIG. 2, the radio resource scheduler can allocate a single radio resource to one user among six different users at each time step. The user to which the radio resource is allocated can transmit a packet. However, the packet transmission may fail due to multiple factors, such as the user itself and the state of the wireless communication network. Meanwhile, each user must adhere to a time constraint. For example, each user must regularly transmit packets successfully. The time constraint may be, for example, an application constraint, e.g., a constraint formulated by an application that requires packet transmission.

[0101] In the example of Figure 2, the horizontal axis represents successive time steps. The current time step is time step 8, and the resource allocation device has allocated radio resources to user 5, as indicated by the black dot in the figure. The vertical axis represents different users. Each user is associated with a horizontal line, and the boxes represent the number of remaining time steps a user needs to successfully transmit one packet.

[0102] In this example: - Users 1 and 5 must transmit packets at time step 8 at the latest; - User 2 must send a packet at time step 11 at the latest; - User 3 must send a packet at time step 12 at the latest; User 4 must transmit a packet at time step 14 at the latest.

[0103] So in this example, - the set of possible actions includes allocating radio resources to users 1, 2, 3, 4, 5 or 6, respectively; - The state of the system includes the number of remaining time steps required to successfully transmit a packet by each user; - When a packet is sent by a user, this packet transmission may succeed with probability p and fail with probability 1-p; If the time constraint is not respected, i.e., in this case, if the user is unable to successfully transmit a packet by the transmission deadline, the reward will be a large negative value. For example, if the time constraint is an application constraint, this may cause the application to fail.

[0104] As will be explained in more detail below, the method according to the invention makes it possible to obtain a policy that increases the probability of obtaining a payoff above a threshold, in other words, it reduces the probability of obtaining a payoff below a threshold, which for example corresponds to an undesirable outcome.

[0105] In the example of FIG. 2, the present invention allows for example to obtain a policy that reduces the probability of having a low payoff, and therefore reduces the probability of missing an application deadline.

[0106] However, the example of FIG. 2 is provided only as a non-limiting example of a radio resource allocation device that can implement the present invention for control, and the present invention can also be applied to other resource allocation devices in different embodiments.

[0107] Next, refer to Figure 3.

[0108] FIG. 3 illustrates an example computer-implemented method P3 according to multiple embodiments of the present invention.

[0109] Method P3 is a method for selecting an action a from a set of possible actions A(s) to be performed on a system that is in a current state s among a set S of possible states of the system at time step t.

[0110] For example, method P3 can be implemented to control a system such as the systems illustrated in Figure 1 or 2. Method P3 can, for example, be implemented by the system itself or by a computing device that controls the system. More generally, method P3 can be implemented by any computing unit that is capable of determining the actions to be performed by the system.

[0111] Method P3 therefore represents what is commonly referred to as the "inference phase," which is the phase in which actions are selected to control the system.

[0112] In embodiments of the present invention, several steps may be performed within the inference phase and / or within the training phase on which the inference phase depends, which may, for example, determine path probabilities or policies to be applied to control the system.

[0113] The following disclosure may indicate method steps that may be performed during the inference phase or the training phase, where applicable.

[0114] Returning to FIG. 3, method P3 further includes a first step S33 of selecting, in the current state, an action that causes the system with the highest probability to follow a path corresponding to a sequence of actions and state transitions in successive time steps from the current state that has a gain of at least a predetermined value.

[0115] For example, the first step S33 may select an action that has the highest sum of the probabilities of paths resulting from the current state when the action is executed, which results in a gain of at least a predetermined value.

[0116] In other words, step S33 does not select the action that results in the largest expected gain, as in the prior art, but rather selects the action that minimizes the probability of resulting in a path having a gain less than or equal to a defined limit.

[0117] Method P3 further includes a second step S34 of causing the system to perform the selected action.

[0118] Thus, method P3 causes the system to perform actions that reduce the probability that a path has a gain below or equal to a predetermined limit. This predetermined limit may be referred to as α in the remainder of this disclosure. Because the gain may be defined according to the outcome of the actions performed by the system and the desirability of the state transition, the method thereby limits the probability that an undesirable outcome will ultimately occur. For example, when the method is implemented by a resource allocation device, the method may reduce the probability of causing a resource allocation that results in a highly undesirable outcome, such as non-compliance with communication constraints, computing resources for successful program execution, etc.

[0119] Determining the probability of following a path with a payoff at least equal to a predefined value can be done in a variety of ways. The steps described below can be performed in part or in whole during the training phase. As described in more detail below, the payoffs and probabilities can be calculated for a predefined policy or for a policy that is iteratively defined by selecting the optimal action at each state during the training and / or inference phases.

[0120] One solution to associating with an action a probability that the system will follow a path corresponding to a sequence of actions and state transitions at successive time steps that has a payoff at least equal to a predetermined value is to execute or simulate the system during the execution of the sequence of actions, and associate with the path taken by the system during the execution or simulation of the system a payoff that depends on the outcome of the execution or simulation.

[0121] For example, in the example scheduler shown in FIG. 2, the scheduler can be simulated or run multiple times, e.g., with different actions corresponding to different policies, or an exhaustive selection of all possible actions corresponding to giving radio resources to different users in each state, and then each resulting path taken by the system (i.e., the successive states and state transitions of the system) can be associated with: - the probability of observing the system follow a path from a state when performing an action. Such a probability can be calculated by dividing the number of times the system follows a path from a state when the action occurs by the number of times the system performs the action in that state; - The payoff calculated according to the outcome of the path. For example, in the example in Figure 2, a negative reward is added to the payoff of the path each time the application deadline passes.

[0122] Another option for determining the probabilities is to use a transition model to calculate the probabilities before step S33. This option is represented by steps S31 and S32 in FIG.

[0123] In FIG. 3, steps S31 and S32 are depicted as preliminary steps to method P3, but may be performed in part or in whole by a training method executed before method P3.

[0124] A first preliminary step S31 performed by method P3 or during the training phase comprises obtaining a transition model that defines, for at least one state transition from an initial state to a subsequent state between two consecutive time steps, the probability of making the transition from the initial state to the subsequent state upon execution of an action belonging to a set of possible actions to be performed in the initial state, and the reward associated with this transition.

[0125] For example, step S31 may involve creating a model that conforms to the MDP framework and the associated transition probabilities p(s j |s, a) and reward r(s j, s).

[0126] A second preliminary step S32 performed by method P3 or during the training phase involves obtaining probabilities and gains corresponding to a plurality of paths corresponding to a sequence of actions and state transitions at successive time steps, the probabilities and gains being calculated using a model for calculating the probabilities and gains of the paths.

[0127] In other words, step S32 makes it possible to obtain, for each path among the multiple paths, the path gain and path probability to be followed. The path gain and probability can be calculated during the execution of method P3 or can be calculated in advance during the training phase. Examples of such determinations are shown in Figures 4, 5, 6 and 7.

[0128] Next, refer to FIG.

[0129] FIG. 7 illustrates an example of multiple paths in one embodiment of the present invention.

[0130] The example of Figure 7 is applicable to, but not limited to, the implementation of the method within an MDP framework.

[0131] FIG. 7 more specifically depicts a trellis showing the possible paths.

[0132] In embodiments of the present invention, the gain and probability of each path is calculated based on the successive state transitions that make up the path. Thus, step S32 may include calculating: - the payoff of multiple paths as the cumulative reward for each combination of state transitions and actions for each path of multiple paths at successive time steps; - Probabilities of multiple paths as the combinatorial probability of each combination of state transitions and actions for each path of multiple paths at successive time steps.

[0133] Figure 7 provides an example of such a calculation.

[0134] In the example of Figure 7, three consecutive time steps t, t+1 and t+2 are considered, and the system can go through six different states s 17 , s 27 , s 37 , s 47 , s 57 and s 67 It can be in.

[0135] Some example transitions are shown in Figure 7. For clarity, only a small subset of the possible actions are shown in Figure 7, such as: - The system is in state 27 When two actions a 217 and a 227 is possible; - The system is in state 17 When two actions a 117 and a 127 is possible; - The system is in state 27 When in, action a 217 can cause two transitions: ○ Probability p 217 and state s with reward r=1 27 From state s 17 transition to; ○ Probability 1-p 217 and state s with reward r=0 27 From state s 47 transition to; - The system is in state 27 When in, action a 227 can cause two transitions: ○ Probability p 227 State in s 27 From state s 57 transition to; ○ Probability 1-p 227 State in s 27 From state s 67 transition to; - The system is in state 17 When in, action a 117 can cause two transitions: ○ Probability p 117 and state s with reward r=0 17 From state s 27 transition to; ○ Probability 1-p 117 and state s with reward r=-1 17 From state s 47 transition to; - The system is in state 17 When in, action a 127 can cause two transitions: ○ Probability p 127 State in s 27 From state s 57 transition to; ○ Probability 1-p 127 State in s 27 From state s 67 Transition to.

[0136] In the example of Figure 7, when the discount λ=1; - Status 27 First example of a path resulting from

number

number

number

number

number

number

[0137] More generally, - The path probability of the kth path arising from state s is P s (k), and the path gain of this path is G s It can be shown in (k); - The path probability of the kth path arising from state s when action a is executed is P s,a(k), and the path gain of this path is G s,a This can be shown in (k).

[0138] The example shown in Figure 7 is provided as a simple illustration and non-limiting example, and many more examples of transitions and paths can be envisioned. For example: - An action may cause a number of possible state transitions other than two. For example, an action may cause a single state transition with probability equal to 1, or it may cause three or more state transitions. The sum of the probabilities of state transitions from the same state during the same action is equal to 1; - Two different actions may cause the same state transition. In that case, the probability and reward may be different. For example, in the example of Figure 7, action a 227 is the state s 27 From s 57 It may also cause a state transition to 217 may differ from 1; - A state transition is a transition from a state to itself, e.g., state s 27 From s 27 may include state transitions to; - A path can contain more than two transitions; - The path gains and probabilities can be calculated over a finite or an infinite time horizon, i.e. in a finite or an infinite time window; - In the above example, the payoff is calculated as the sum of the rewards. It can also be calculated as a weighted sum by applying a discount factor multiplied by the transition index, for example as shown in equation (1) above.

[0139] In the remainder of this disclosure, the state the system is in at time t and from which an action is performed will be referred to as the "current state," and for a given transition, the state from which this transition occurs will be referred to as the "initial state," and the state to which this transition occurs will be referred to as the "successor state." For example, for state s 27 From s 17 In the transition to s27 is the initial state, and s 17 is the subsequent state.

[0140] The path gains and probabilities can be calculated in a variety of ways, for example, the complete combination of all possible paths can be calculated over the time horizon, or, as described below, the gains and probabilities can be calculated iteratively using binning rules.

[0141] Thus, the action can be selected, for example, by calculating the following formula:

number

[0142] where: · s is the current state; a is a set of actions that can be performed in the current state. s is an action that belongs to; · α is the gain limit; P s,a (k) is the occurrence probability of the kth possible path arising from the current state s when performing action a; 1{G s,a (k)>α} is ○ The payoff G of the kth possible path resulting from the current state s when executing action a s,a equals 1 if (k) is greater than α; ○ otherwise equals 0; a * is the selected action that exhibits the larger value Q(s, a, α) among all possible actions.

[0143] Thus, at the output of step S33, the action that most likely causes the system to follow a path corresponding to a sequence of actions and state transitions at successive time steps with a gain at least equal to the predetermined value α is selected.

[0144] As explained above, according to different embodiments of the present invention, some steps can be performed during the training phase or the inference phase, and therefore, according to different embodiments of the present invention, the output of the training phase and the input of the inference phase may be different.

[0145] For example, in some embodiments of the present invention, the training phase may involve calculating the payoff G for each of the kth possible paths arising from each state s during the execution of action a. s,a (k) and occurrence probability P s,a (k) is finally calculated.

[0146] Thus, in such an example, the inference phase takes as input all values G s,a (k) and P s,a (k), and in step S33, the above equations (8) and (9) are calculated to select an action that causes the system to follow with the highest probability a path corresponding to a series of actions and state transitions at consecutive time steps that has a gain at least equal to a predetermined value.

[0147] In another embodiment of the invention, the training phase involves calculating the payoff G for each of the k possible paths arising from each state s, under the assumption that the best action is selected in each state. s (k) and occurrence probability P s (k) is finally calculated.

[0148] Thus, in such an example, the inference phase takes as input all values G s (k) and P s (k) and the transition model. In step S33, the probability of the path resulting from the current state during the execution of the action can be calculated based on: - The probability of making a transition from the current state to each successive state in the transition model; - Probability P of following each path resulting from each subsequent step s (k).

[0149] Meanwhile, the gain associated with each path resulting from the current state during the execution of an action can be calculated based on: - The reward associated with each transition from the current state to each successive state in the transition model; - the payoff G of each path arising from each successive state to which a discount factor can be applied s (k).

[0150] Thus, in these embodiments, step S33 is performed by determining the value of the current state G s,a (k) and P s,a (k) and then using equations (8) and (9) to select the action to perform.

[0151] In yet another embodiment of the present invention, the training phase completely determines the action to take at each state. Thus, the output of the training phase is the optimal policy π * and step S33 is to determine the policy π * Applying policy π to select the action to take in the current state. * can be represented, for example, in the form of a table that associates an action to be performed with each state.

[0152] Note that in such cases, the policy is predetermined for a given constraint α, and therefore the use of the policy during the inference phase is necessarily optimized for the given constraint α. In contrast, in embodiments that rely on the use of Equations (8) and (9), or similar equations, during the inference phase, the parameter α can be set during the inference phase.

[0153] Next, refer to FIG.

[0154] FIG. 8 illustrates a section of a trellis of possible paths in several embodiments of the present invention.

[0155] In some embodiments of the present invention, path probabilities and gains are calculated for paths defined by a series of consecutive states.

[0156] In another embodiment of the invention, each state is associated with a reference set containing reference gains for paths originating from that state and reference probabilities for following those paths from that state, and a binning rule is applied to the reference gains for paths originating from each state to assign a predetermined size to the size of the reference set associated with each state containing the gains for paths originating from that state and the probabilities for those paths. Thus, the reference set, reference gains, and reference probabilities define the possible paths originating from the state with their respective gains and probabilities.

[0157] In other words, paths with similar gains are put into "bins" that define intervals of gains. Thus, in such a representation, a state, or a combination of state and action, is associated with a probability of causing the system to follow a path in each of the bins, i.e., a path with a gain in the defined interval. For computational purposes, all paths included in a defined bin can be associated with the same gain, e.g., the gain at the center of the bin. In such a representation, the exact state transitions used by a path are irrelevant; the only relevant values are the gains and probabilities of the various bins.

[0158] The bins can be predefined so that the interval of gain forming each bin is known in advance. In such a case, the bins are predefined for all states, so the value G s,a (k) and G s (k) is actually the same for all states. For example, multiple bins can be predefined by the user around the expected gain from the state, or a regularly spaced interval can be defined between the minimum and maximum gain.

[0159] Therefore, in embodiments of the present invention, the calculation of the gains and probabilities of the paths relies on iterative calculations and binning rules that allow the size of the reference set of gains and probabilities to be maintained at a predetermined size K. The principles of the iterative calculations and binning rules are further explained below.

[0160] First, in each iteration, several paths emerge from each state. Recursively calculating all possible states and paths can result in an excessively large number of possible path combinations. To limit the number of combinations while still achieving good results, paths with similar probabilities can be merged to obtain K paths for one state path. The probabilities of these K paths emerging from a state can be shown as follows:

number

[0161] The associated gain can be shown as follows:

number

[0162] Therefore, the state is a criterion set Ω of the following formula: s It can be expressed as:

number

[0163] Therefore, within the framework of the present invention, the value of state s can be defined as follows:

number

[0164] In other words, the value of a state can be expressed as the probability that, when the system is in that state, it will end up following a path with a payoff higher than α.

[0165] Note that the above notation does not depend on the action a, because in the iterative computation, an action is automatically associated with each state, either by applying a predefined policy or by automatically selecting the action that maximizes the value of the state.

[0166] In the example of Figure 8, the trellis is 18 and two possible successor states s ’ 18 and s ’ 28 The two possible state transitions are: - Probability p and reward r(s ’ 18 ) in 18 From s ’ 18 transition to; - Probability (1-p) and reward r(s ’ 28 ) in s 18 From s ’ 28 Transition to.

[0167] In the iterative process, s 18 The reference set of state s ’ 18 The reference set of

number

number

number

number

number

number

number

number

[0168] Status 18 The reference set of

number

number

[0169] where λ is the discount factor, 0<λ≦1.

[0170] At this stage, the collection

number

[0171] To limit the size of the set, the binning rule is

number

number

number

number

[0172] The binning function is ’ 18 and s ’ 28 has already been applied to, and the vector of payoffs associated with these two states is, as a result,

number

number

number

[0173] moreover,

number

[0174] Here, Bin(G s, x) represents the application of the binning rule to the vector x.

[0175] To provide concrete examples of binning rules,

number

number

number

number

number

[0176] Below is a concrete example of the iterative calculation of path gains and probabilities.

[0177] Here, reference is made to FIG.

[0178] 4 shows an iterative method P4 for computing the gains and probabilities of paths given an input policy π. Method P4 can therefore be performed during the training phase. The baseline gains and probabilities thus obtained can then be used during the inference phase, for example by method P3.

[0179] Method P4 consists in determining, for each state in the set S of possible states, the initial basis set Ω s ={G s ref ,P s ref}, where G s ref and P s ref Both are size K.

[0180] Then, a loop S44 is performed for each state s in the set of possible states S. For each state, the following steps are performed:

[0181] In a first substep S41, an action a to be executed in a state s is selected according to a policy π.

[0182] A second substep S42 uses the selected operation to redefine the initial reference set of states by concatenating, for each successor state that has a non-zero transition probability from that state, the base gain in the reference set of the successor state multiplied by a discount factor and added to the reward for the transition from that state to the successor state, and the base probability of the successor state multiplied by the non-zero transition probability.

[0183] The gain can be calculated as follows:

number

[0184] For each state, the non-zero transition probability is p(s jk |s,a).

[0185] The probabilities are then calculated as follows:

number

[0186] Next, the iteration includes a substep S43 of reducing the size of the reference set of states to a predetermined size K by applying a binning rule to the reference gains of the states.

[0187] Once sub-steps S41 to S43 have been performed for a state, sub-step S44 checks whether they need to be performed for another state. If so, sub-steps S41 to S43 are performed for the other state. If not, step S45 is performed to check whether a stopping criterion is met.

[0188] If the stopping criterion is not met, a new iteration is performed in which sub-steps S41 to S43 are performed for all states.

[0189] If not, the method is complete and the baseline gains and probabilities are calculated for each state.

[0190] This therefore provides an efficient way to calculate the baseline payoff and baseline probability for a given policy.

[0191] It is of course possible to repeat method P4 for a number of different policies in order to select the policy that maximizes the probability of providing a payoff above a predetermined value in a particular state.

[0192] Next, refer to Figure 5.

[0193] FIG. 5 illustrates a computer-implemented method P5 for calculating baseline gains and baseline probabilities.

[0194] Method P5 does not take a default policy as input, but iteratively constructs an optimized policy by selecting the optimized behavior of each state at each iteration.

[0195] Therefore, sub-step S41 is replaced by sub-steps S51 and S52, and the other steps and sub-steps are the same as those of method P4.

[0196] Step S51 calculates the probability P of taking a path with a payoff in each bin k in state s when executing action a.s,a The objective is to obtain (k). To this end, step S51 involves obtaining, for each possible action in the state, the probability of proceeding to each successor state '' upon execution of a possible action a in state s. Such probabilities p(s'|s, a) can be obtained, for example, from a transition model. The reward for each transition and the reference set {P s’ (kg s’ ref (k)}, the probability P of taking a path with a payoff in each bin k in state s when executing action a s,a (k) can be found.

[0197] Step S52 involves selecting from among the possible actions that will most likely cause the system to follow a path from that state that corresponds to a sequence of actions and state transitions at successive time steps that have a gain at least equal to a predetermined value.

[0198] For example, knowing the transition probabilities from a state to possible successor states and a reference set of successor states, the probability that an action will result in a payoff of at least a predetermined value α can be calculated by the following formula:

number

[0199] Next, the action to be performed is a * can be selected as follows:

number

[0200] Therefore, when applying the binning rule, the base probability is the probability of the selected action a * It will be acquired along with the

[0201] The state values can also be defined as follows:

number

[0202] In fact, this equation is equivalent to equation (13).

[0203] Thus, the optimized policy is constructed iteratively.

[0204] A different stopping criterion can be used for each of methods P4 and P5. For example, the stopping criterion can be: - Maximum number of iterations; - the convergence criterion of the gain between two successive iterations, etc.

[0205] Next, refer to FIG.

[0206] FIG. 6 shows a third example P6 of a computer-implemented method for calculating baseline gains and baseline probabilities.

[0207] In the example of Figure 6, step S32 of using the transition model further includes using a machine learning model trained to predict the probability of a path having a gain that falls within each interval among a plurality of intervals of possible gains from the states of the paths resulting from that state.

[0208] Thus, a machine learning model is trained to predict the probability of following a path from a given state, under the assumption that optimal actions are taken at each state.

[0209] For example, the machine learning model can be iteratively trained using an error signal calculated as a cost function based on predicted path probabilities from an initial state and predicted path probabilities from a subsequent state multiplied by the transition probability from the initial state to the subsequent state when performing an action that causes the system to most likely follow a path from the initial state that has a gain of at least a predetermined value.

[0210] For example, the learning phase is now performed by determining the probability P that, from an initial state s, the system follows a path from state s that has a payoff in bin k.s A machine learning model can be trained to predict (k). In the example of FIG. 6, these probabilities do not represent the probability of following a defined path, but rather the probability of following a path whose payoff falls within a defined bin. The bins can be predefined. For example, multiple bins can be predefined by the user around the expected payoff from a state, or multiple bins can be defined at regularly spaced intervals between the minimum and maximum payoffs.

[0211] Then, in the inference phase, the probability of the path that will result from the current state in the execution of the action can be calculated based on: - The probability of making a transition from the current state to each successive state in the transition model; - Probability P of taking each path at each successive state s (k).

[0212] On the other hand, the gain associated with each path resulting from the current state during the execution of an action can be calculated based on: - the rewards associated with each transition from the current state to each subsequent state in the transition model; - The payoff G for following each path along each successive state, to which a discount factor may be applied. s (k).

[0213] Therefore, the inference phase is based on the value of the current state G s,a (k) and P s,a (k) and then using equations (8) and (9) to select the action to perform.

[0214] In the example of FIG. 6, training method P6 includes a double loop that trains a machine learning model to minimize a cost function.

[0215] In loop S67, a loop is performed over a plurality of states to calculate the value of the cost function of the states.

[0216] The loop S67 may be, for example, a loop through all states or a loop through a sub-portion thereof in order to obtain multiple values of the cost function for multiple states, respectively.

[0217] At the output of loop S67, step S68 updates the machine learning model based on the multiple values of the cost function calculated for the multiple states.

[0218] At the output of step S68, step S69 checks whether a stopping criterion is met: if so, the training phase ends and the machine learning model can be used for inference by method P3.

[0219] If not, a new loop of steps S61 to S68 is executed until the stopping criterion is met.

[0220] The stopping criteria may, for example, relate to one or more of a number of iterations or may be a convergence criterion. The stopping criteria may, for example, include one or more of the following: - Comparison of the number of iterations with a threshold; - comparison of the mean value of the cost function at the output of the update in step S68; - Comparison of the difference in the mean value of the cost function between two consecutive time steps.

[0221] In the loop S67, the value of the cost function for the selected state is calculated by steps S61 to S66.

[0222] In step S61, a machine learning model is used to calculate predictions of path probabilities from the initial state, thus the currently trained machine learning model is applied to the selected state.

[0223] Steps S62 to S64 are then repeated for each possible action that can be performed from the selected state.

[0224] In step S62, the probability of progressing to each successor state upon execution of a possible action and the associated reward in the transition model are obtained from the transition model.

[0225] Thus, at the output of step S62, all state transitions to each possible successor state are known, along with the associated transition probabilities and transition rewards.

[0226] In step S63, a machine learning model is used to calculate a prediction of the path probability from each of the subsequent states.

[0227] Thus, at the output of step S63, it is known, based on the current learning of the model, for each successor state what the probability is of following a path from the successor state that has a gain in each of the bins.

[0228] In step S64, a prediction of the path probabilities from each successor state and the probability of proceeding to each successor state when performing a possible action is calculated from the associated rewards.

[0229] In step S64, the probability of the transition at each subsequent state, the reward of the transition, and the probability and payoff of each bin can be used to calculate the probability of following a path from the selected state that has a payoff in each bin when performing each possible action.

[0230] In step S65, from the prediction of the path probability from the initial state when performing possible actions, an action is selected that causes the system to follow with the highest probability a path having a gain from the initial state that is at least equal to or greater than a predetermined value.

[0231] In other words, the optimal behavior of the state during the current training of the model is selected in step S65, for example, using equations (8) and (9).

[0232] In step S66, a value of the cost function is calculated based on a prediction of the path probability from the initial state and a prediction of the path probability from the initial state when performing possible actions that cause the system to most likely follow a path from the initial state that has a gain of at least a predetermined value.

[0233] In essence, the cost function estimates the difference between two calculations of path probabilities: - Computational results obtained by directly applying machine learning models to selected states; - A computational result obtained by applying a machine learning model to subsequent states and combining this information with transition probabilities and rewards from a transition model.

[0234] If the machine learning model is properly trained, the difference between the two calculations will be small.

[0235] Furthermore, this means that the machine learning model is trained so that successive updates of the model based on this cost function provide path probabilities associated with bins consistent with the transition model. Thus, during the inference phase, the machine learning model can be directly used to predict the probability of following a path in each bin from the current state. Thus, the selection of an action in step S33 within the inference phase will in fact cause the system to most likely follow a path corresponding to a sequence of actions and state transitions in successive time steps that has a gain at least equal to a predetermined value.

[0236] Thus, method P6 provides an efficient way to train a machine learning model to predict the probability that, upon execution of an action in a state, the system will follow a path whose gain is included in a defined bin, but is provided only as an example of the application of a machine learning model to predict the probability that the system will follow a path with a gain defined in one embodiment of the present invention.

[0237] The present disclosure is not limited to the methods, apparatus, and computer programs described herein, which are merely examples, and the present invention encompasses all alternatives that would occur to one skilled in the art upon reading this document.

Claims

1. 1. A computer-implemented method for selecting, at a time step, an action from a set of possible actions to perform on a system at a current state of the system in a set of possible states, the method comprising: selecting an action that causes the system to most likely follow a path from the current state that corresponds to a sequence of actions and state transitions in successive time steps that have a gain of at least a predetermined value; causing the system to perform the operation; 20. A computer-implemented method comprising:

2. The selection may be obtaining a transition model that defines, for at least one state transition from an initial state to a subsequent state between two consecutive time steps, a probability of making the state transition from the initial state to the subsequent state upon execution of an action belonging to a set of possible actions to be performed in the initial state, and a reward associated with the state transition; Using at least the transition model, for a plurality of paths, gains of the multiple paths; and the probabilities of the plurality of paths; and The computer-implemented method of claim 1 , wherein the method is based on

3. a gain for the plurality of paths is calculated as a cumulative reward of a series of successive state transitions and actions for each path of the plurality of paths at successive time steps; The computer-implemented method of claim 2 , wherein the probabilities of the multiple paths are calculated as the combined probability of each combination of state transitions and actions for each path of the multiple paths at successive time steps.

4. 4. The computer-implemented method of claim 3, wherein a binning rule is applied to the reference gains of the paths originating from each state to bring the size of a reference set associated with each state containing the gains of the paths originating from the state and the probabilities of the paths to a predetermined size.

5. 5. The computer-implemented method of claim 4, wherein calculating gains and probabilities using the transition model includes iteratively calculating a reference set of states including, for each state in the set of possible states, a reference gain for a path resulting from the state when the action is performed and a reference probability for causing the system to follow a path resulting from the state when the action is performed, the iterative calculation applying the binning rule in each iteration.

6. The iterative calculation is For each state in the set of possible states, obtaining an initial reference set; until a stopping criterion is met. For each state in the set of possible states: selecting an action to perform in said state according to a policy; For each successor state that has a non-zero probability of state transition from said state using the selected action, a reference gain in a reference set of the successor state multiplied by a discount factor added to the reward of the state transition from the state to the successor state; a base probability of the successor state multiplied by a non-zero probability of the state transition; redefining the initial reference set of states by concatenating reducing the size of the reference set of states to a predetermined size by applying the binning rule to the reference gains of the states; The computer-implemented method of claim 5 , comprising:

7. The iterative calculation is For each state in the set of possible states, obtaining an initial reference set; until a stopping criterion is met. For each state in the set of possible states: obtaining, for each possible action in the state, a probability of progressing to each successor state upon execution of the possible action in the state; selecting from among the possible actions an action that most likely causes the system to follow a path from the state corresponding to a sequence of actions and state transitions at successive time steps that has a gain at least equal to a predetermined value; For each successor state that has a non-zero probability of transitioning from said state using the selected action, the reference gain in the reference set of the successor state multiplied by a discount factor added to the reward of the transition from the state to the successor state; a base probability of the successor state multiplied by a non-zero probability of the transition; redefining the initial reference set of states by concatenating reducing the size of the reference set of states to a predetermined size by applying the binning rule to the reference gains of the states; The computer-implemented method of claim 5 , comprising:

8. 3. The computer-implemented method of claim 2, wherein using the transition model further comprises using a machine learning model trained to predict path probabilities from the state having gains that belong to each interval among a plurality of intervals of possible gains for paths resulting from the state.

9. 9. The computer-implemented method of claim 8, wherein the machine learning model is iteratively trained using an error signal calculated as a cost function based on predictions of path probabilities from an initial state and predictions of path probabilities from a subsequent state multiplied by the probability of a transition from the initial state to the subsequent state upon execution of an operation that causes the system to most likely follow a path from the initial state that has a gain of at least a predetermined value.

10. The repetitive training may include: until a stopping criterion is met. executing a loop comprising: selecting a plurality of states in the set of possible states; and for each state in the plurality of states: using the machine learning model to calculate predictions of path probabilities from the initial state; For each possible state in the state: obtaining from the transition model a probability of proceeding to each successor state upon execution of the possible action and an associated reward in the transition model; using the machine learning model to calculate a prediction of a path probability from each of the successor states; calculating a prediction of a path probability from the initial state during said execution of said possible actions from path probabilities from each of said successor states and a prediction of the probability of proceeding to each successor state during said execution of said possible actions and an associated reward; selecting, from the predictions of the path probabilities from the initial state during the execution of the possible actions, the action that causes the system to most likely follow a path from the initial state that has a gain of at least a predetermined value; calculating a value of the cost function based on a prediction of the path probabilities from the initial state and a prediction of the path probabilities from the initial state upon execution of the possible actions that cause the system to most likely follow a path from the initial state that has a gain of at least a predetermined value; and executing a loop of updating the machine learning model based on a plurality of values of the cost function calculated for the plurality of states; 10. The computer-implemented method of claim 9, comprising:

11. implemented by a resource allocation device of the system; the set of possible actions corresponds to possible resource allocations in the states; The computer-implemented method of any one of claims 1 to 10, wherein the reward is defined according to the result of the resource allocation and the state transition of the system.

12. Implemented by a radio resource scheduler of a wireless communication network, the set of possible actions corresponds to possible allocations of radio resources to users of the wireless communication network; at least one probability of making a state transition is dependent on a probability that a user of the wireless communication network will successfully transmit a packet over the wireless communication network when the radio resources are allocated to the user; The computer-implemented method of any one of claims 1 to 11, wherein at least one reward depends on compliance with a time constraint for the transmission of at least one packet.

13. A computing device comprising at least one processing unit configured to perform the computer-implemented method of any one of claims 1 to 12.

14. A radio resource scheduler for a wireless communication network, comprising at least one processing unit configured to perform the computer-implemented method of any one of claims 1 to 12.

15. Computer software comprising instructions for performing at least part of the computer-implemented method of any one of claims 1 to 12 when said computer software is executed by a processor.

Images