Method for determining an optimised time sequence of movements to be performed by at least one entity in order to reach a set of predetermined locations

Abstract

The invention relates to a method for determining an optimised time sequence of movements to be performed by at least one entity in order to reach a set of predetermined locations. Such a method comprises: - instantiating (11) a reinforcement learning environment, as a function of location data of the locations, temporal data associated with a predefined time period, and constraint data comprising at least one time constraint; - a training phase (12) of the reinforcement learning type, wherein an agent is trained to generate a time sequence of movements to be performed by the at least one entity in order to reach the set of predetermined locations over the predefined time period while complying with the constraints, on the basis of at least one data exchange between the agent and the environment; and - an inference phase (13), wherein the trained agent is used to deliver the optimised time sequence.

Description

Method for determining an optimized time sequence of movements to be carried out by at least one entity to reach a set of predetermined locations.

[0001] The invention lies in the field of optimizing the provision of resources (human, material) at different geographical locations. More particularly, the invention relates to the problems of optimizing routes to be taken to visit a set of locations, in the presence of various predefined constraints (temporal, professional, etc.). Prior art

[0002] The Vehicle Routing Problem (VRP) is a classic optimization problem that involves identifying the best routes for a set of vehicles, objects, or messages that must pass through a set of predetermined locations. The term "best routes" generally refers to the set of routes that minimizes the total distance traveled by all vehicles, or minimizes the total time spent by all vehicles to visit all predetermined locations (each location generally having to be visited at least once by at least one of the vehicles). When such a problem also incorporates certain time constraints, it is known as a "Vehicle Routing Problem with Time Windows" (VRPTW).Solving this type of problem is of interest to many stakeholders (companies, organizations, etc.), for example when it comes to establishing schedules (or schedules) for the intervention of technicians or more generally of visitors to customers, who must take into account possible periods of unavailability of certain technicians and / or certain customers to be visited during previously identified time slots (e.g. in the event of a customer being temporarily absent from their home, communicated in advance, in the event of a planned meeting that a technician must attend, etc.).

[0003] Optimization tools exist to facilitate the resolution of the vehicle routing problem with time windows, such as those proposed in the OR-Tools software suite developed by Google™ for example. However, these existing solutions have limitations when specific constraints, including more specific time constraints or other types of constraints, are to be taken into consideration. For example, in certain fields such as telecommunications or energy supply, certain types of interventions are likely to require the use of specific equipment and / or the simultaneous presence (i.e. in the same time slot) on site of several technicians. Thus, an intervention at height (e.g.at the top of a pole or on the facade of a multi-story building) may require that cradle-type equipment be available and that two technicians be present (instead of one for a more traditional intervention), and it is therefore necessary to also take into account these specific business constraints when establishing the technicians' daily intervention schedule.

[0004] Generally speaking, it has been found that such specific constraints, whether temporal or business-related, are poorly managed by current optimization tools. On the one hand, they are difficult to integrate into these tools, which remain relatively generic. This difficulty arises in particular from the large number and diversity of the types of business constraints likely to be encountered. On the other hand, even when a means of modeling them in the tool is identified and implemented, the increase in complexity caused by their presence significantly degrades the performance of these existing tools, and they no longer produce optimal solutions to the problem at hand.

[0005] There is therefore a need for a solution to better manage the consideration of disparate constraints in vehicle routing problems with time windows.

[0006] The present technique makes it possible to propose a solution aimed at remedying certain drawbacks of the prior art. According to one aspect, the present technique relates to a method for determining an optimized time sequence of movements to be carried out by at least one entity to reach a set of predetermined locations.

[0007] Such a method is implemented by an electronic device, and it comprises:

[0008] - the instantiation of a reinforcement learning environment, based on configuration data provided as input to said electronic device, said configuration data comprising location data of said locations, time data associated with a predefined time period for the implementation of said movements, and constraint data comprising at least one time constraint;

[0009] - a training phase of reinforcement learning type, in which at least one agent instantiated by said electronic device is trained to generate at least one temporal sequence of movements to be carried out by said at least one entity to reach said set of predetermined locations over said predefined time period while respecting said constraints, on the basis of at least one exchange of data between said at least one agent and said environment;

[0010] - an inference phase, in which said at least one trained agent is used to deliver said optimized time sequence.

[0011] In this way, the present technique proposes a clever solution based on reinforcement learning techniques to determine an optimized temporal sequence of movements to be carried out by at least one entity to reach a set of predetermined locations. Such a solution is particularly flexible and efficient for solving a problem deemed to be complex. More particularly, it is flexible insofar as it allows for the simple integration of multiple constraints, particularly temporal ones, to be taken into account. It is efficient insofar as the proposed trial-and-error approach allows for exploring a wide range of possible solutions to this problem, and thus training a model to identify the most optimal solutions to this problem.

[0012] In a particular embodiment, said at least one time constraint comprises data representative of an unavailability of at least one of said at least one entity and / or of at least one of said locations over at least one time slice of said predefined time period.

[0013] In this way, time constraints frequently encountered in scheduling operations, namely the unavailability in certain time slots of one or more actors and / or resources involved in the activities that are the subject of the schedule, are easily taken into account for the determination of an optimized time sequence of movements according to the present technique.

[0014] In a particular embodiment, said constraint data further comprises at least one additional constraint. According to a particular characteristic, said at least one additional constraint is associated with an activity which is the subject of said movements.

[0015] In this way, the proposed technique allows for the simple consideration of potentially numerous and varied, even heterogeneous, additional constraints, including for example business constraints that are very specific to a given activity.

[0016] In a particular embodiment, said at least one data exchange between said at least one agent and said environment comprises:

[0017] - the emission, by said at least one agent, of action data to said learning environment, said action data being representative of a next envisaged location of said at least one entity relative to a current location of said at least one entity in the generated time sequence of movement;

[0018] - receiving, by said at least one agent, in response to said transmission, state data and reward data from said learning environment;

[0019] the successive action data sent to said learning environment within the framework of said at least one exchange being determined so as to tend to maximize the reward data received.

[0020] In this way, the proposed technique makes it possible to explore and test, without operational impact, a large number of movement sequences, and thus learn to identify the best time sequences of movements allowing the different entities to reach all the predetermined locations over a predefined time period considered.

[0021] In a particular embodiment, said training phase comprises training a plurality of agents, and said inference phase comprises delivering by a plurality of said trained agents a plurality of candidate movement time sequences, and selecting said optimized movement time sequence from among said plurality of candidate movement time sequences, based on reward data associated with each of said candidate movement time sequences.

[0022] In this way, the performance of the method according to the present technique is further increased, by the possibility of benefiting during the inference phase from the capacities of a group of pre-trained agents rather than a single agent.

[0023] According to a particular characteristic of this embodiment, the agents of said plurality of agents are trained sequentially.

[0024] In this way, training a plurality of agents does not require significant hardware resources (in terms of processors, memories, etc.) and can be implemented on electronic devices with modest technical capabilities. More particularly, the hardware resources required for training a single agent are sufficient for training several agents, only the duration required for training being increased.

[0025] In a particular embodiment, said training phase comprises the implementation of a plurality of successive training episodes, an increasing number of constraints from a set of constraints defined by said constraint data being progressively added for the configuration of said episodes, as said successive episodes are implemented, until all of said constraints defined by said constraint data are reached.

[0026] In this way, learning is progressive, which ultimately allows for a more accurate model to be obtained than if all the constraints had been introduced from the start of learning.

[0027] In a particular embodiment, said method comprises, prior to said reinforcement learning type training phase, an offline learning phase comprising training of said at least one agent by means of a learning base comprising data representative of a plurality of pre-existing time sequences of movements.

[0028] In this way, an imitation learning phase is implemented prior to the reinforcement learning phase. The pre-modeling of the agent thus obtained on the basis of pre-existing temporal sequences of movements allows the subsequent implementation of a faster and more efficient reinforcement learning phase, since it has as its starting point an already pre-trained model and not a virgin model.

[0029] According to a particular characteristic of this embodiment, said method comprises a feedback loop used to supplement said learning base with data representative of said optimized time sequence delivered during said inference phase.

[0030] In this way, the experience of a human supervisor in charge of controlling and possibly adapting the time sequences delivered by the electronic device, or even feedback from field experience and in particular the management of imponderables, can be taken into account by re-injecting the corresponding modified time sequences into the system, thus contributing to further enriching the learning by an agent. In addition, the implementation of such a feedback loop can also be used to identify in real time and / or continuously an optimized time sequence of movements remaining to be carried out, for example when the reality of the field imposes deviations from an initially determined time sequence of movements.

[0031] According to another aspect, the present technique relates to an electronic device for determining an optimized time sequence of movements to be carried out by at least one entity to reach a set of predetermined locations. Such a device comprises:

[0032] - means for instantiating a reinforcement learning environment, as a function of configuration data provided as input to said electronic device, said configuration data comprising location data of said locations, time data associated with a predefined time period for the implementation of said movements, and constraint data comprising at least one time constraint;

[0033] - means for implementing a training phase of the reinforcement learning type, in which at least one agent instantiated by said electronic device is trained to generate at least one temporal sequence of movements to be carried out by said at least one entity to reach said set of predetermined locations over said predefined time period while respecting said constraints, on the basis of at least one exchange of data between said at least one agent and said environment;

[0034] - means for implementing an inference phase, in which said at least one trained agent is used to deliver said optimized time sequence.

[0035] Such an electronic device may of course have the various characteristics relating to the determination method according to the present disclosure, which may be combined or considered in isolation. Thus, the characteristics and advantages of this device are the same as those of the method for determining an optimized time sequence of movements, and are not detailed further.

[0036] According to another aspect, the proposed technique also relates to a computer program product downloadable from a communication network and / or stored on a computer-readable medium and / or executable by a microprocessor, comprising program code instructions for executing a method for determining an optimized temporal sequence of movements as described above in any of its embodiments, when executed on a computer.

[0037] The proposed technique also relates to a computer-readable recording medium on which is recorded a computer program comprising program code instructions for executing the steps of the method as described above, in any of its embodiments.

[0038] Such a recording medium may be any entity or device capable of storing the program. For example, the medium may include a storage medium, such as a ROM, for example a CD ROM or a microelectronic circuit ROM, or a magnetic recording medium, for example a USB flash drive or a hard disk.

[0039] On the other hand, such a recording medium may be a transmissible medium such as an electrical or optical signal, which may be conveyed via an electrical or optical cable, by radio or by other means, so that the computer program contained therein is remotely executable. The program according to the present disclosure may in particular be downloaded over a network, for example the Internet.

[0040] The various embodiments mentioned above can be combined with each other for the implementation of the present disclosure. Figures

[0041] Other characteristics and advantages of the invention will appear more clearly on reading the following description of a preferred embodiment, given as a simple illustrative and non-limiting example, and the appended drawings, among which:

[0042] illustrates the general principle of a method for determining an optimized time sequence of movements to be carried out by at least one entity to reach a set of predetermined locations, in a particular embodiment of the proposed technique;

[0043] presents an example of mapping illustrating locations of locations to be reached, in a particular embodiment of the proposed technique;

[0044] presents the data exchanges carried out between an agent and a reinforcement learning environment for training said agent, in a particular embodiment of the proposed technique;

[0045] presents an example of a temporal sequence of movements explored during a training phase or delivered at the output of an inference phase, in a particular embodiment of the proposed technique;

[0046] describes a simplified architecture of an electronic device for implementing the proposed technique, in a particular embodiment. Detailed description of the invention

[0047] This application makes it possible to remedy some of the aforementioned drawbacks.

[0048] In all figures in this document, elements and steps of the same nature are designated by the same reference.

[0049] According to a first aspect, the present technique relates to a method for determining an optimized time sequence of movements to be carried out by at least one entity to reach a set of predetermined locations. An entity within the meaning of the invention may, for example, correspond to an object or an element, physical or virtual, capable of moving (for example, a vehicle, a robotic tool, a message in a network, etc.), or even to a person (for example, a technician or a craftsman having to intervene on construction sites).Depending on the applications considered for the present technique, the predetermined locations may, for example, correspond to different areas of an object to be worked on (for example, an object to be repaired or manufactured), or to geographical locations corresponding to construction sites or work sites to be carried out, to material storage locations, to locations where people are domiciled, to addresses where virtualized functions are accessible, etc. The present technique aims more specifically to determine optimal routes to be traveled by these entities so that all the predetermined locations are visited by at least one of them, while respecting predefined constraints (for example, temporal, spatial, professional, etc.). By "optimal routes", we mean, for example, those which make it possible to minimize a cost so that all the locations are processed (i.e.visited), this cost can for example be expressed in total distance traveled by the entities, in total time spent by the entities, in financial cost, or using any other metric of interest to the user of the proposed technique.

[0050] In the rest of the document, the solution is mainly described in the context of generating a schedule (or "planning" in English) intended to plan a period of interventions, typically the next working day, of technicians at customers' premises.The purpose of such a schedule is to schedule the technicians' interventions, so that they know which location to go to at each moment of their working day, the generated schedule also having to respect a set of predefined constraints (for example the need for a technician to have to go through a warehouse to retrieve particular equipment required for a subsequent intervention, the need to coordinate the movements of several technicians so that they meet at the same customer's premises at a given time slot when the intervention at this customer's premises cannot be carried out by a single technician, taking into account the unavailability of certain customers and / or technicians at certain time slots, etc.).Such an example of use is however purely illustrative and non-limiting, and the present technique can also be implemented in many other contexts (for example to coordinate and optimize the movement of a plurality of tools to a plurality of areas or locations to be processed of a part to be repaired and / or manufactured on a robotic production line, etc.). Another example of use of the proposed technique is provided later, in relation to the optimization of the routing of messages in a communication network. Thus, in a communication network, the movements of a message or a packet via a plurality of functions / equipment in charge of processing and / or routing of the message or the packet can represent a valid implementation for the solution(s) proposed in this disclosure.The locations may correspond to the functions while the movements may concern the messages or the packets. Such routing of messages in a communication network may also induce constraints including processing times by the functions, requirements in terms of points of passage of a message in the functions, an order to be respected for the scheduling of the functions in charge of routing or providing processing to the message during its routing or even constraints in terms of availability of elementary processing supported by a function. This implementation in a communication network is particularly valid when the communication network is instantiated in virtualized form and a chaining of the functions allows the movement and routing of a message via the functions according to constraints specific to the message and / or the communication network.

[0051] The general principle of the proposed method for determining an optimized time sequence of movements to be performed by at least one entity to reach a set of predetermined locations is illustrated in relation to the, in a particular embodiment. This method is implemented within an electronic device described later in this document.

[0052] In a step 11, a reinforcement learning environment is instantiated by the electronic device, based on configuration data provided as input to this device. Such data may, for example, in whole or in part, be entered by a human operator by means of a human-machine interface made available by the electronic device, or even be obtained automatically from various data sources via a communication network to which the electronic device is connected.

[0053] These configuration data include location data, temporal data, and constraint data, detailed below. In a game analogy commonly used to present reinforcement learning techniques, location data and temporal data can be seen as representative of a game environment, while constraint data defines the rules of that game.

[0054] More specifically, the location data defines, in a given geographic reference system, the coordinates (for example in the form of longitude and latitude data, abscissas and ordinates, etc.) of various locations of interest, including in particular predetermined target locations to be reached by the entities in the context of a given activity (in other words, to use the previous analogy, in the context of the “game”), and additional locations of importance such as, for example, the starting locations of these entities, locations where certain resources (e.g., materials) are available, etc. In the context where the proposed technique is used to plan interventions by technicians at customers' premises, these locations of interest correspond to the possible locations to which the technicians may go during their travels.As illustrated in the, which presents an example of mapping such locations, the location data then includes, for example, the geographical coordinates of the depots of the company that employs the technicians (a depot D in the example of the, where the technicians can, for example, collect or drop off specific equipment, for example a cradle for working at height), the technicians' homes (two homes E1 and E2 in the example of the), and the customers' homes where they can work (six intervention locations C1 to C6 in the example of the), all for a given geographical area ZG (for example the perimeter of a district, a city, a canton, etc.).According to a particular characteristic, such location data may comprise or be used to consolidate a table of distances between the locations of interest, also serving as configuration data when instantiating the reinforcement learning environment. The distances present within this table may be expressed in various ways, for example in average time required to reach two locations, in kilometer distance, in monetary cost, etc. The distances between each pair of locations may be obtained directly from the coordinates of these locations. These distances may for example be automatically acquired from software programs of the "estimated time of arrival" (ETA) type, such as Google Maps™ for example.

[0055] The time data defines for its part a total time period over which all the movements must have been carried out (also called operation time period in the rest of the document), as well as possibly a division of this time period into elementary time units (e.g. into time slots or time blocks). In the context where the proposed technique is used to plan technician interventions at customers' premises, the considered operation time period corresponds for example typically to a working day (e.g. extending from 8 a.m. to 6 p.m.), divided into half-hour or one-hour time slots (e.g. slot 1 from 8 a.m. to 8:30 a.m., slot 2 from 8:30 a.m. to 9 a.m., slot 3 from 9 a.m. to 9:30 a.m., etc.) which may for example correspond to slots for passages in the locations D, E1, E2, C1 to C6 mentioned above.

[0056] Finally, the constraint data defines a set of constraints to be respected, the existence of which has an impact on determining the optimized time sequence of movements to be carried out by the different entities. In the context of the present technique, these constraints comprise at least one time constraint associated with at least one entity or at least one of the locations. In certain particular embodiments, these constraints further comprise at least one additional constraint associated with an activity that is the subject of the movements of the different entities. In the context where the proposed technique is used to plan interventions by technicians at customers' premises, the time constraints may concern technicians (e.g. the entities) and / or customers (e.g. the locations) indifferently.These time constraints take the form, for example, of scheduling constraints, representing the unavailability of at least one customer and / or at least one technician over one or more given time slots, or even constraints relating to the order of visits, etc. Additional constraints are typically business constraints, associated, for example, with the need to have access to certain resources – specific equipment or tools, a particular qualification (e.g., a driving license for a particular type of vehicle, a particular skill, etc.), a minimum number of technicians to carry out the intervention, etc. – at the time when an intervention must take place at certain predetermined target locations.

[0057] The electronic device generally obtains this configuration data before the operating time period during which the movements must be made. Typically, the acquisition of this data takes place, for example, a few hours before the time period to be planned (for example, the day before the workday to be planned), in order to allow time for the electronic device to implement the training 12 and inference 13 phases described below to deliver an optimized time sequence of movements to be made by the different entities, and possibly to allow time for a human supervisor to verify the relevance of the time sequence thus determined.

[0058] In a step 12, a reinforcement learning type training phase is implemented using the learning environment configured in step 11. During this phase, at least one agent instantiated by said electronic device is trained to generate time sequences of movements to be carried out by at least one entity to reach all of the predetermined locations over the predefined time period, while respecting the time constraints and the additional constraints previously set.

[0059] Each attempt by an agent to generate (or explore) such a temporal sequence of movements during the training phase can be called an episode, according to a common terminology used in the field of reinforcement learning. The training phase 12 generally includes the implementation of a multitude of episodes (from several tens to several thousands).

[0060] More specifically, during this reinforcement learning training phase, using a trial-and-error approach, multiple data exchanges are carried out between the agent and the learning environment, with a view to broadly exploring the results obtained in response to different sequences of actions corresponding to temporal sequences of movements tested.

[0061] Each of these exchanges includes, according to the usual vocabulary in reinforcement learning and as illustrated in relation to:

[0062] - the emission, by an AG agent, of action data D_ACT to the learning environment ENV;

[0063] - the reception, by the agent AG, in response to the transmission of said action data, of state data D_ETA and reward data D_REC from the learning environment ENV.

[0064] The state data D_ETA is representative of a new state of the reinforcement learning environment ENV, resulting from the action performed on this environment via the action data D_ACT.

[0065] The D_REC reward data are representative of one or more metrics of the effectiveness of the action carried out on the ENV environment, with regard to the objective sought, namely the determination of an optimized temporal sequence of movements to be carried out by at least one entity to reach all of the predetermined target locations while respecting the fixed constraints (i.e. the temporal constraints and possibly the additional specific business constraints, as well as possibly any other type of predefined constraints).

[0066] The successive actions sent to the digital replica are determined by a reinforcement algorithm, so as to tend to maximize during an episode the weighted sum of the reward data D_REC received in response to these actions.

[0067] In a particular embodiment of the present technique, from an initial geographical position of the different entities, for example, a location of the different entities is successively associated with each time slot (or time slice) of the operating time period with respect to the different available locations, as illustrated later in relation to the. The action data D_ACT transmitted by an agent to the learning environment ENV then comprises a set of data representative of the location where each entity must go at the following time slot (or time slice) (it being understood that an absence of movement, i.e. the fact that an entity remains positioned at its current location, is an available option). In other words, via the action data D_ACT, the agent AG indicates to the learning environment ENV the next location where each entity must go.The action data are thus representative of a next envisaged location of each entity with respect to its current location in the temporal sequence of movement being generated (i.e. explored).

[0068] As described previously, different information, delivered by the reinforcement algorithm, is then obtained by the agent AG, in response to the modification of the ENV environment resulting from the implementation of the action defined by the action data D_ACT.

[0069] First, reward data D_REC is calculated. This reward data typically takes the form of a simple numerical value, typically a real number, which is the sum of different reward (positive values) and / or punishment (negative values) values ​​associated with the new locations of the entities, as defined by the action data D_ACT.

[0070] For example, the value r can be the result of a sum r = α + β + γ + δ + ε, in which:

[0071] - the value of the parameter α is a negative value (i.e. a punishment) added to each time slot in order to encourage the agent to arrive as quickly as possible at the end of an episode, i.e. to ensure that a solution is found so that all target locations are visited in the fewest possible "moves" (i.e. time elapsed, for example evaluated in number of required time slots);

[0072] - the value of the parameter β is a negative value (i.e. a punishment) proportional to the sum of the distances between the old and new location of each entity, so as to favor the shortest possible movements for each entity;

[0073] - the value of the parameter γ is a positive value (i.e. a reward) of average value assigned each time a target location is visited for the first time successfully (i.e., without violating one of the predefined constraints) by an entity;

[0074] - the value of the parameter δ is a large negative value (i.e. a significant punishment) assigned each time the new location of an entity generates non-compliance with one of the predefined constraints (in other words, the new location violates the rules of the game), such as for example a second visit to a target location already visited in a previous time slot, a visit to a target location in a time slot corresponding to the unavailability of an associated customer, a visit to a target location not respecting the human and / or material resource conditions required for the intervention (e.g. the need for two technicians to be present, the need for the technician present to have a particular qualification, the need for particular equipment or tools such as a cradle, etc.);

[0075] - the value of the parameter ε is a large positive value (i.e. a significant reward) assigned when all target locations have been successfully visited (with possibly the additional optional condition, depending on the context of use, that the entities have also returned to their initial positions or to a predefined position, which can be translated, in the context of the example mainly described in this document, by the fact that all customers have received a visit from technicians, and that the latter have returned to their respective homes or to the depot of the company that employs them at the end of their tour).

[0076] The previous example is of course given for purely illustrative and non-limiting purposes, other parameters (positive reward values ​​or negative punishment values) can be taken into account for the calculation of the reward data D_REC that the agent seeks to maximize during each episode.

[0077] Second, state data D_ETA (or “observation”) are delivered, in order to allow the agent AG to obtain at least a partial view of the state of the learning environment ENV (in other words, to use the analogy previously introduced, of the state of the game) following the implementation of an action. Such data include for example:

[0078] - the table of distances between locations;

[0079] - the total distance traveled by each entity;

[0080] - the visited or unvisited status of each location;

[0081] - a table of necessary resources associated with each location;

[0082] - a table of resources available at each location;

[0083] - a schedule of time constraints associated with each location;

[0084] - a schedule of time constraints associated with each entity;

[0085] - etc.

[0086] Furthermore, a boolean – possibly part of the D_ETA state data – is also returned to the AG agent, in order to indicate whether the current episode is in an “episode finished” or “episode not finished” state. This boolean takes more specifically a value representative of a finished episode to mark what can be assimilated to the end of a game or part. Thus, for example, it takes the value “episode finished”:

[0087] - when all locations have been visited with overall satisfactory compliance with the imposed constraints (for example with a D_REC reward level reached above a predetermined threshold) and possibly all entities have returned to their initial positions (case of a game considered as “won”); or

[0088] - when too many constraints have not been respected, for example because all the locations could not be visited within a given time limit corresponding to the predefined operating time period, or because a reward value associated with the D_REC reward data has not reached or has fallen below a certain predetermined critical threshold due to the accumulation of too many punishments due to non-compliance with certain constraints (in particular temporal or additional trade constraints) or because of a movement policy that is too inefficient (case of a game considered as “lost”).

[0089] As long as the boolean is not set to the "episode finished" state (in other words, as long as it is in the "episode not finished" state), the game is not over, and the agent can transmit new action data D_ACT to the learning environment ENV.

[0090] Thus, at the end of learning phase 12, at least one agent has been trained on a large number of episodes – in other words, the weights of a model associated with the agent have been progressively adjusted over the episodes – so that the agent is able to determine temporal sequences of movements as optimal as possible to reach a set of predetermined target locations while respecting predefined constraints.

[0091] In a step 13, at the end of the training phase, an inference phase is implemented, in which at least one agent trained in step 12 is used to determine and deliver the optimized time sequence of movements to be carried out by the different entities to reach all of the predetermined target locations.

[0092] In the context where the proposed technique is used to plan technician interventions at customers' premises on a given working day, such an inference phase 13 is typically implemented the evening before or early in the morning of said working day, in order to generate the schedule of tours (i.e. interventions) to be carried out during the day by each of the technicians.

[0093] An example of a time sequence of movements explored during the training phase 12 or delivered as an output of the inference phase 13 is presented in relation to the, in such a context of planning technician interventions at customers' premises. This purely illustrative example is based on configuration data similar to those of the, namely the determination of a schedule for the tour of two technicians T1 and T2, domiciled respectively at locations E1 and E2, having to intervene during a working day divided into seven successive time slots t1 to t7 at the homes C1 to C6 of six customers. The company employing technicians T1 and T2 also has a warehouse D for storing equipment. As illustrated in the, the time sequence of movements defines for each of the technicians T1 and T2, at each time slot, their respective position with respect to one of the locations of interest E1, E2, D, C1, C2, C3, C4, C5 or C6.According to the explored (step 12) or delivered (step 13) schedule, it is for example planned that technicians T1 and T2 are at their respective homes E1 and E2 at the first time slot t1 and the last time slot t7 of the day, and that all the customers' homes C1 to C6 are visited by at least one of the technicians in the time slots t2 to t5. In the illustrated example, at least one constraint (here respected) concerns the intervention at the home of customer C2, namely for example a long intervention, requiring at least part of the intervention the presence of two technicians and a nacelle-type device. Also, according to the proposed schedule, the two technicians T1 and T2 are present together at location C2 in the time slot t5, and time has been planned for technician T1 to collect beforehand, in the time slot t4, the nacelle required from the company's warehouse D.In time slot t6, for example, it is planned that technician T2 completes the intervention alone at the home of customer C2 (thus characterizing an absence of travel for technician T2 between time slots t5 and t6), while technician T1 is responsible for returning the cradle, which is useless for this second part of the intervention, to the company's warehouse D. Each of the time slots t2 to t7 is also associated with a reward value D_REC, reflecting the relevance associated with the new locations of technicians T1 and T2 in the time slot considered, compared to their previous locations. These rewards are, for example, calculated according to the methods presented previously in relation to the, and their sum over all the time slots makes it possible to evaluate the quality of the produced schedule, particularly with regard to compliance with the predefined constraints set.

[0094] Of course, other modes of representation than a table as presented can be used within the framework of the present technique to communicate to a user a temporal sequence of movements according to the present technique. For illustration purposes, such a sequence can also be represented in a cartographic form, the movements determined for each entity then taking the form of routes plotted on a map of the type illustrated in relation to the for example.

[0095] In particular embodiments, various approaches presented below may be adopted, optionally and possibly complementary, in order, for example, to make the method for determining an optimized time sequence of movements according to the present technique more efficient and / or more robust.

[0096] According to a first approach, rather than presenting the problem with all the constraints to be solved from the beginning of training, it is possible to implement progressive difficulty training techniques (known as "curriculum learning"). According to this approach, a progressive increase in the difficulty level is implemented during phase 12 of training, as the agent manages to successfully complete episodes. In other words, new constraints are added in stages, each time the agent's reinforcement algorithm successfully completes a given number of episodes (i.e. reaches a predetermined reward threshold for a given number of episodes), until all the predefined constraints are finally integrated.Such progressive learning allows for faster convergence and a more accurate model to be obtained than if all the constraints had been introduced from the start of learning.

[0097] According to a second approach, conditional action masking techniques are implemented during training phase 12, consisting of reducing the number of possible actions (i.e. the range of possibilities) as an episode progresses. For example, in a particular embodiment, a masking technique consists of configuring the agent so that it is no longer allowed to visit a location that has already been successfully visited in a previous time slot of the episode. In this way, the agent learns more quickly, with the possible actions to be explored by the agent decreasing over time due to the fact that certain actions become unavailable in certain time slots.

[0098] According to a third approach, it is proposed in a particular embodiment to instantiate and train not a single agent during the training phase 12, but a plurality (i.e. at least two) agents. During the inference phase 13, an inference is carried out by each of the agents thus previously trained, and the sequence retained as the optimized temporal sequence of movements to be carried out is that delivered by the agent having maximized the reward data obtained during this inference phase. Such a technique makes it possible to benefit during the inference phase from the capacities of a group of pre-trained agents (rather than a single agent), and to ultimately retain the best of the optimized temporal sequences of movements determined by these different agents, thus increasing the performance of the proposed technique.According to a particular characteristic, during the training phase 12, the different agents of the plurality of agents are trained sequentially (i.e. one after the other), in order to limit the number of resources required (in terms of computing power, memory, etc.) to carry out the training, which makes it possible to implement the present technique even on machines with potentially limited capacities.

[0099] According to a fourth approach, the method comprises, prior to reinforcement learning during which at least one agent interacts with the learning environment, a so-called offline learning phase during which the agent does not interact with such an environment, but learns (i.e. gradually adjusts the weights of its model) by means of a learning base provided as input to the agent. More particularly, this AHL offline learning phase illustrated by the optional step 10 of the can be described as an imitation learning phase, aimed at carrying out pre-modeling of the agent on the basis of pre-existing temporal sequences of movements (or pre-existing agendas) forming said learning base.Such pre-existing time sequences of movements correspond, for example, to sequences identified by human operators and provided during the solution configuration operations according to the present technique, or generated by elementary scripts. These pre-existing time sequences of movements do not necessarily correspond to the most optimal solutions, but they allow an agent to which they are communicated not to start the reinforcement learning phase – called online learning AEL – from scratch, and therefore allow it to learn more quickly and more efficiently.In other words, by analogy with the field of video games for example, such prior offline learning AHL (step 10 of the) allows for example the agent to acquire a “human” level of play, and the subsequent online learning AEL (steps 11 and 12 of the) then allows it to progress, by reinforcement, until reaching a level of play comparable to a “superhuman” level. Optionally, according to a particular characteristic and as illustrated in the, a feedback loop BR can also be implemented in this context, in order to continuously feed the learning base used for offline learning with data from the implementation of the method according to the present technique. Such data include for example all or part of the optimized time sequences of movements delivered at the end of the inference phases at each use of the method.They may also include traces (or logs) of the time sequences of movements that actually took place, in other words field interventions. These time sequences actually implemented in the field may indeed differ more or less significantly from those delivered at the end of the inference phase for various reasons. The optimized time sequence delivered at the end of the inference phase may, for example, have been subject to adaptations during a check by a human supervisor before its implementation. It may also have been modified in the field itself to take into account unforeseen circumstances, for example when the technicians in charge are forced to adapt the initial schedule to deal with unforeseen situations (localized weather phenomena, roadworks, uncommunicated unavailability of a customer, etc.).Some of the solutions thus adapted may be suboptimal, i.e. for example not minimizing the total time elapsed to visit all the locations and / or the total distance of journeys traveled by the fleet of technicians for this purpose, but they nevertheless constitute data of interest to feed the learning base and further improve the performance of the process according to the proposed technique. According to a particular characteristic, when the traces of the time sequences of movements actually carried out are available in real time or near real time, the BR feedback loop can also be used to identify in real time and / or continuously an optimized time sequence of movements remaining to be carried out, in particular when the reality of the field has imposed deviations from the time sequence of movements initially calculated.In such an embodiment, the entities have, for example, means (e.g., communication devices or interfaces) enabling them to exchange, via a communication network, notification messages with the electronic device responsible for implementing the method for determining optimized time sequences of movements according to the present technique. In this way, technicians can, for example, send useful information in real time to the electronic device, including in particular confirmation that the visit to a location (e.g., of a customer) has just been successfully carried out in accordance with the initially planned schedule, or, if necessary, information that a visit initially planned in the schedule cannot ultimately be carried out on time for a particular reason that may be specified in the message.In this second case, upon receipt of this notification, the electronic device can then trigger, on the basis of this new information, the determination of a new optimized time sequence of movements, and communicate this updated sequence to all the technicians in the field, for consideration. The determination of a new updated optimized time sequence of movements can also in certain situations be triggered at the initiative of the electronic device itself, or upon receipt by this device of information from data sources other than the entities (for example in the event of detection of alerts or reception of alert messages such as weather alerts, congestion alerts at particular points of a network, etc.).

[0100] As indicated at the beginning of the document, although mainly described in the context of generating a schedule intended to plan the trips of technicians to customers, the present technique is not limited to this use case, and it can be used in many different fields. In a particular embodiment, it can thus for example be used to determine an optimized time sequence for routing messages within a communication network infrastructure, in particular in the context of the implementation of functionalities based on service chains, in which the services in question are potentially hosted at different locations in the communication network, for example within different physical servers and / or different virtual machines.According to a particular characteristic, the messages are for example representative of network tasks to be executed, which must be processed within the different services, taking into account time constraints (e.g. task A must be processed before task B by a given service, which defines constraints relating to the order of reception of at least certain messages within this service), and possible additional constraints (e.g. task C must imperatively be processed by a given service, which is therefore an obligatory point of passage for the message associated with this task).In other words, the implementation of the present technique then makes it possible to determine a schedule for optimal routing of the different messages within the communication network, while taking into account constraints specific to this type of use – for example the possibility for the same message to be routed several times to the same virtual machine, if the general processing of the tasks associated with this message requires it, but the prohibition for the same virtual machine to simultaneously process several different messages – which can in this sense be described as business constraints.In this context, "optimal routing" can, for example, be understood as one that makes it possible to limit the carbon footprint resulting from the management of these messages within the communication network and / or to be able to ensure that the chaining of services respects a certain delay corresponding, for example, to a quality of experience requirement for a user. Thus, an application requiring low latency may require a chaining of elementary services or network functions executed in a time less than a value representative of the latency. The different types of data presented in the context of the present technique – in particular configuration data, action data, state data, and reward data – are then adapted to this context.For example, the reward data may take into account carbon cost values ​​associated with the network links used to route a message between two virtual machines or servers (a negative value, or punishment, proportional to the sum of the carbon costs associated with the network links between the old and new location of each message being assigned, for example, so as to favor the least carbon-emitting paths possible for each message). Similarly, the reward data may take into account the successful completion of a task (by associating a reward of average positive value) or of all tasks (by associating a reward of large positive value).

[0101] According to another aspect, the proposed technique also relates to a device for determining an optimized time sequence of movements to be carried out by at least one entity to reach a set of predetermined locations. Such an electronic device is capable of carrying out the method previously described in any of its embodiments. More particularly, such a device according to the present technique comprises, in a particular embodiment:

[0102] - means for instantiating a reinforcement learning environment, as a function of configuration data provided as input to said electronic device, said configuration data comprising location data of said locations, time data associated with a predefined time period for the implementation of said movements, and constraint data comprising at least one time constraint and at least one additional constraint;

[0103] - means for implementing a training phase of the reinforcement learning type, in which at least one agent instantiated by said electronic device is trained to generate time sequences of movements to be carried out by said at least one entity to reach said set of predetermined locations over said predefined time period while respecting said constraints, on the basis of at least one exchange of data between said at least one agent and said environment;

[0104] - means for implementing an inference phase, in which said at least one trained agent is used to deliver said optimized time sequence.

[0105] Larepresents, in a schematic and simplified manner, the structure of such an electronic device, in a particular embodiment. The device, according to the proposed technique, comprises for example a memory 51 consisting of a buffer memory M, a processing unit 52, equipped for example with a microprocessor μP, and controlled by the computer program Pg 53, implementing steps of the method for determining an optimized temporal sequence of movements according to at least one embodiment of the invention. To this end, the electronic device also comprises at least one communication interface (for example an Ethernet communication interface) and / or at least one human-machine interface, allowing it to receive and transmit data from and to other equipment present in a communication network to which it is connected and / or human operators.

[0106] Upon initialization, the code instructions of the computer program 53 are loaded into the buffer memory before being executed by the processor of the processing unit 52. The processing unit 52 receives as input E, for example, configuration data by means of which it can instantiate a reinforcement learning environment.

[0107] The microprocessor of the processing unit 52 then carries out the following steps of the method for determining an optimized time sequence, according to the instructions of the computer program 53. More particularly, the processing unit 52 instantiates at least one agent, and trains it – on the basis of data exchanges with said reinforcement learning environment – ​​to generate time sequences of movements to be carried out by at least one entity to reach the set of predetermined locations, while respecting constraints predefined in the configuration data. At the end of this training, the agent thus trained is then used to generate an optimized time sequence of movements, which is then delivered by the processing unit 52 as output S.

Claims

Method for determining an optimized temporal sequence of movements to be performed by at least one entity to reach a set of predetermined locations, said method being implemented by an electronic device, said method being characterized in that it comprises: - the instantiation (11) of a reinforcement learning environment (ENV), as a function of configuration data provided as input to said electronic device, said configuration data comprising location data of said locations, temporal data associated with a predefined temporal period for the implementation of said movements, and constraint data comprising at least one temporal constraint;- a training phase (12) of the reinforcement learning type, in which at least one agent (AG) instantiated by said electronic device is trained to generate at least one temporal sequence of movements to be carried out by said at least one entity to reach said set of predetermined locations over said predefined temporal period while respecting said constraints, on the basis of at least one exchange of data between said at least one agent (AG) and said environment (ENV); - an inference phase (13), in which said at least one trained agent is used to deliver said optimized temporal sequence.; Method according to claim 1, characterized in that said at least one time constraint comprises data representative of an unavailability of at least one of said at least one entity and / or of at least one of said locations over at least one time slice of said predefined time period. Method according to claim 1, characterized in that said constraint data further comprises at least one additional constraint. Method according to claim 3, characterized in that said at least one additional constraint is associated with an activity which is the subject of said movements. Method according to claim 1, characterized in that said at least one data exchange comprises:- the transmission, by said at least one agent, of action data to said learning environment, said action data being representative of a next envisaged location of said at least one entity relative to a current location of said at least one entity in the generated time sequence of movement;- the reception, by said at least one agent, in response to said transmission, of state data and reward data from said learning environment; the successive action data transmitted to said learning environment within the framework of said at least one exchange being determined so as to tend to maximize over time the reward data received. Method according to claim 1, characterized in that said training phase comprises training a plurality of agents, and in that said inference phase comprises the delivery by a plurality of said trained agents of a plurality of candidate time sequences of movements, and the selection of said optimized time sequence of movements from among said plurality of candidate time sequences of movements, as a function of reward data associated with each of said candidate time sequences of movements. Method according to claim 6, characterized in that the agents of said plurality of agents are trained sequentially. Method according to claim 1, characterized in that said training phase comprises the implementation of a plurality of successive training episodes, an increasing number of constraints from a set of constraints defined by said constraint data being progressively added for the configuration of said episodes, as said implementation of said successive episodes progresses, until reaching all of said constraints defined by said constraint data. Method according to claim 1, characterized in that it comprises, prior to said reinforcement learning type training phase, an offline learning phase comprising training of said at least one agent by means of a learning base comprising data representative of a plurality of pre-existing time sequences of movements. Method according to claim 9, characterized in that it comprises a feedback loop (BR) used to supplement said learning base with data representative of said optimized time sequence delivered during said inference phase. Electronic device for determining an optimized temporal sequence of movements to be carried out by at least one entity to reach a set of predetermined locations, said device being characterized in that it comprises: - means for instantiating a reinforcement learning environment, as a function of configuration data provided as input to said electronic device, said configuration data comprising location data of said locations, temporal data associated with a predefined temporal period for the implementation of said movements, and constraint data comprising at least one temporal constraint;- means for implementing a reinforcement learning type training phase, in which at least one agent instantiated by said electronic device is trained to generate at least one time sequence of movements to be carried out by said at least one entity to reach said set of predetermined locations over said predefined time period while respecting said constraints, on the basis of at least one exchange of data between said at least one agent and said environment; - means for implementing an inference phase, in which said at least one trained agent is used to deliver said optimized time sequence.; Computer program product downloadable from a communications network and / or stored on a computer-readable medium and / or executable by a microprocessor, characterized in that it comprises program code instructions for executing a method according to any one of claims 1 to 10, when executed by a computer.

Images