Method for determining an optimized time sequence of movements to be carried out by at least one entity to reach a set of predetermined locations.

Abstract

Method for determining an optimized time sequence of movements to be performed by at least one entity to reach a set of predetermined locations. The invention relates to a method for determining an optimized time sequence of movements to be performed by at least one entity to reach a set of predetermined locations.Such a method comprises: - instantiating (11) a reinforcement learning environment, based on location data of said locations, time data associated with a predefined time period, and constraint data comprising at least one time constraint; - a training phase (12) of the reinforcement learning type, in which an agent is trained to generate a time sequence of movements to be performed by said at least one entity to reach said set of predetermined locations over said predefined time period while respecting said constraints, on the basis of at least one exchange of data between said agent and said environment; - an inference phase (13), in which said trained agent is used to deliver said optimized time sequence. Abstract figure: Figure 1.

Description

Title of the invention: Method for determining an optimized time sequence of movements to be carried out by at least one entity to reach a set of predetermined locations. Technical field

[0001] The invention lies in the field of optimizing the provision of resources (human, material) at different geographical locations. More particularly, the invention relates to the problems of optimizing routes to be taken to visit a set of locations, in the presence of various predefined constraints (temporal, professional, etc.). Prior art

[0002] The vehicle routing problem (VRP) is a classic optimization problem that involves identifying the best routes to be taken by a set of vehicles or objects or even messages that must pass through a set of predetermined locations. What is meant by "best routes" generally corresponds to the set of routes that makes it possible to minimize the total distance traveled by all the vehicles, or to minimize the total time spent by all the vehicles to visit all the predetermined locations (each location generally having to have been visited at least once by at least one of the vehicles). When such a problem also incorporates certain time constraints, it is then known as a "vehicle routing problem with time windows" (VRPTW).Solving this type of problem is of interest to many stakeholders (companies, organizations, etc.), for example when it comes to establishing schedules (or timetables) for the intervention of technicians or more generally of visitors to customers, who must take into account possible periods of unavailability of certain technicians and / or certain customers to be visited during previously identified time slots (e.g. in the event of a customer being temporarily absent from their home, in the event of a planned meeting that a technician must attend, etc.).

[0003] Optimization tools exist to facilitate the resolution of the vehicle routing problem with time windows, such as those proposed in the OR-Tools software suite developed by Google™ for example. These existing solutions, however, have limitations when particular constraints, including more specific time constraints or other types of constraints, are to be taken into consideration. For example, in certain fields such as tele communications or energy supply, certain types of interventions are likely to require the use of specific equipment and / or the simultaneous presence (i.e. in the same time slot) on site of several technicians. Thus, an intervention at height (e.g. at the top of a pole or on the facade of a multi-story building) may require that cradle-type equipment be available and that two technicians be present (instead of one for a more traditional intervention), and it is therefore necessary to also take into account these specific business constraints when establishing the technicians' daily intervention schedule.

[0004] Generally speaking, it has been found that such specific constraints, whether temporal or business-related, are poorly managed by current optimization tools. On the one hand, they are difficult to integrate into these tools, which remain relatively generic, this difficulty arising in particular from the large number and diversity of the types of business constraints likely to be encountered. On the other hand, even when a means of modeling them in the tool is identified and implemented, the increase in complexity caused by their presence significantly degrades the performance of these existing tools, and they no longer produce optimal solutions to the problem posed.

[0005] There is therefore a need for a solution that allows for better management of the consideration of disparate constraints in vehicle routing type problems with time windows. Summary of the invention

[0006] The present technique makes it possible to propose a solution aimed at remedying certain drawbacks of the prior art. According to one aspect, the present technique relates to a method for determining an optimized time sequence of movements to be carried out by at least one entity to reach a set of predetermined locations.

[0007] Such a method is implemented by an electronic device, and it comprises:

[0008] - the instantiation of a reinforcement learning environment, depending on configuration data provided as input to said electronic device, said configuration data comprising location data of said locations, time data associated with a predefined time period for the implementation of said movements, and constraint data comprising at least one time constraint;

[0009] - a reinforcement learning type training phase, in which at at least one agent instantiated by said electronic device is trained to generate at least one temporal sequence of movements to be carried out by said at least one entity for reaching said set of predetermined locations over said predefined time period while respecting said constraints, on the basis of at least one exchange of data between said at least one agent and said environment;

[0010] - an inference phase, in which said at least one trained agent is used to deliver said optimized time sequence.

[0011] In this way, the present technique proposes a clever solution based on reinforcement learning techniques to determine an optimized temporal sequence of movements to be carried out by at least one entity to reach a set of predetermined locations. Such a solution is particularly flexible and effective for solving a problem deemed to be complex. More particularly, it is flexible insofar as it allows for the simple integration of multiple constraints, particularly temporal constraints, to be taken into account. It is effective insofar as the proposed trial-and-error approach makes it possible to explore a wide range of possible solutions to this problem, and thus to train a model to identify the most optimal solutions to this problem.

[0012] In a particular embodiment, said at least one time constraint comprises data representative of an unavailability of at least one of said at least one entity and / or of at least one of said locations over at least one time slice of said predefined time period.

[0013] In this way, time constraints frequently encountered in scheduling operations, namely the unavailability in certain time slots of one or more actors and / or resources involved in the activities that are the subject of the schedule, are easily taken into account for the determination of an optimized time sequence of movements according to the present technique.

[0014] In a particular embodiment, said constraint data further comprises at least one additional constraint. According to a particular characteristic, said at least one additional constraint is associated with an activity which is the subject of said movements.

[0015] In this way, the proposed technique allows for the simple consideration of potentially numerous and varied, even heterogeneous, additional constraints, including for example business constraints that are very specific to a given activity.

[0016] In a particular embodiment, said at least one data exchange between said at least one agent and said environment comprises:

[0017] - the emission, by said at least one agent, of action data to said learning environment, said action data being representative of a next envisaged location of said at least one entity relative to a current location of said at least one entity in the time sequence of de- placement generated;

[0018] - the reception, by said at least one agent, in response to said transmission, of data status and reward data from said learning environment;

[0019] the successive action data sent to said learning environment within the framework of said at least one exchange being determined so as to tend to maximize the reward data received.

[0020] In this way, the proposed technique makes it possible to explore and test, without operational impact, a large number of movement sequences, and thus learn to identify the best time sequences of movements allowing the different entities to reach all of the predetermined locations over a predefined time period considered.

[0021] In a particular embodiment, said training phase comprises training a plurality of agents, and said inference phase comprises delivering by a plurality of said trained agents a plurality of candidate movement time sequences, and selecting said optimized movement time sequence from among said plurality of candidate movement time sequences, as a function of reward data associated with each of said candidate movement time sequences.

[0022] In this way, the performance of the method according to the present technique is further increased, by the possibility of benefiting during the inference phase from the capacities of a group of pre-trained agents rather than a single agent.

[0023] According to a particular characteristic of this embodiment, the agents of said plurality of agents are trained sequentially.

[0024] In this way, the training of a plurality of agents does not require significant hardware resources (in terms of processors, memories, etc.) and can be implemented on electronic devices with modest technical capabilities. More particularly, the hardware resources required for training a single agent are sufficient for training several agents, only the duration required for training being increased.

[0025] In a particular embodiment, said training phase comprises the implementation of a plurality of successive training episodes, an increasing number of constraints from a set of constraints defined by said constraint data being progressively added for the configuration of said episodes, as said successive episodes are implemented, until all of said constraints defined by said constraint data are reached.

[0026] In this way, learning is progressive, which ultimately makes it possible to obtain a more precise model than if all the constraints had been introduced from the start of learning.

[0027] In a particular embodiment, said method comprises, prior to said reinforcement learning type training phase, an offline learning phase comprising training said at least one agent by means of a learning base comprising data representative of a plurality of pre-existing time sequences of movements.

[0028] In this way, an imitation learning phase is implemented prior to the reinforcement learning phase. The pre-modeling of the agent thus obtained on the basis of pre-existing temporal sequences of movements allows the subsequent implementation of a faster and more efficient reinforcement learning phase, insofar as it has as its starting point an already pre-trained model and not a virgin model.

[0029] According to a particular characteristic of this embodiment, said method comprises a feedback loop used to supplement said learning base with data representative of said optimized time sequence delivered during said inference phase.

[0030] In this way, the experience of a human supervisor in charge of controlling and possibly adapting the time sequences delivered by the electronic device, or even feedback from field experience and in particular the management of imponderables, can be taken into account by re-injecting the corresponding modified time sequences into the system, thus contributing to further enriching the learning by an agent. In addition, the implementation of such a feedback loop can also be used to identify in real time and / or continuously an optimized time sequence of movements remaining to be carried out, for example when the reality of the field imposes deviations from an initially determined time sequence of movements.

[0031] According to another aspect, the present technique relates to an electronic device for determining an optimized time sequence of movements to be carried out by at least one entity to reach a set of predetermined locations. Such a device comprises:

[0032] - means for instantiating a reinforcement learning environment, based on configuration data provided as input to said electronic device, said configuration data comprising location data of said locations, time data associated with a predefined time period for the implementation of said movements, and constraint data comprising at least one time constraint;

[0033] - means for implementing a learning-type training phase by reinforcement, in which at least one agent instantiated by said electronic device electronics is trained to generate at least one time sequence of movements to be carried out by said at least one entity to reach said set of predetermined locations over said predefined time period while respecting said constraints, on the basis of at least one exchange of data between said at least one agent and said environment;

[0034] - means for implementing an inference phase, in which said at least a trained agent is used to deliver said optimized time sequence.

[0035] Such an electronic device can of course have the different characteristics relating to the determination method according to the invention, which can be combined or considered in isolation. Thus, the characteristics and advantages of this device are the same as those of the method for determining an optimized time sequence of movements, and are not detailed further.

[0036] According to another aspect, the proposed technique also relates to a computer program product downloadable from a communication network and / or stored on a computer-readable medium and / or executable by a microprocessor, comprising program code instructions for executing a method for determining an optimized time sequence of movements as described above in any of its embodiments, when executed on a computer.

[0037] The proposed technique also relates to a computer-readable recording medium on which is recorded a computer program comprising program code instructions for executing the steps of the method as described above, in any of its embodiments.

[0038] Such a recording medium may be any entity or device capable of storing the program. For example, the medium may comprise a storage means, such as a ROM, for example a CD ROM or a microelectronic circuit ROM, or a magnetic recording means, for example a USB key or a hard disk.

[0039] On the other hand, such a recording medium may be a transmissible medium such as an electrical or optical signal, which may be conveyed via an electrical or optical cable, by radio or by other means, so that the computer program contained therein is remotely executable. The program according to the invention may in particular be downloaded over a network, for example the Internet.

[0040] The different embodiments mentioned above can be combined with each other for implementing the invention. Figures

[0041] Other characteristics and advantages of the invention will appear more clearly in the reading the following description of a preferred embodiment, given as a simple illustrative and non-limiting example, and the appended drawings, among which:

[0042] [Fig. 1] illustrates the general principle of a method for determining an optimized time sequence of movements to be carried out by at least one entity to reach a set of predetermined locations, in a particular embodiment of the proposed technique;

[0043] [Fig.2] presents an example of mapping illustrating locations of places to be reached, in a particular embodiment of the proposed technique;

[0044] [Fig.3] presents the data exchanges carried out between an agent and a reinforcement learning environment for training said agent, in a particular embodiment of the proposed technique;

[0045] [Fig.4] presents an example of a temporal sequence of movements explored during a training phase or delivered at the output of an inference phase, in a particular embodiment of the proposed technique;

[0046] [Fig.5] describes a simplified architecture of an electronic device for implementing the proposed technique, in a particular embodiment. Detailed description of the invention

[0047] The present application makes it possible to remedy certain of the aforementioned drawbacks.

[0048] In all the figures of this document, the elements and steps of the same nature are designated by the same reference.

[0049] According to a first aspect, the present technique relates to a method for determining an optimized time sequence of movements to be carried out by at least one entity to reach a set of predetermined locations. An entity within the meaning of the invention may, for example, correspond to an object or an element, physical or virtual, capable of moving (for example, a vehicle, a robotic tool, a message in a network, etc.), or even to a person (for example, a technician or a craftsman having to intervene on construction sites).Depending on the applications considered for the present technique, the predetermined locations may, for example, correspond to different areas of an object to be worked on (for example, an object to be repaired or manufactured), or to geographical locations corresponding to construction sites or work sites to be carried out, to material storage locations, to locations where people are domiciled, to addresses where virtualized functions are accessible, etc. The present technique aims more specifically to determine optimal routes to be traveled by these entities so that all the predetermined locations are visited by at least one of them, while respecting predefined constraints (for example, temporal, spatial, professional, etc.). etc.). By "optimal routes" we mean, for example, those which make it possible to minimize a cost so that all locations are processed (i.e. visited), this cost being able, for example, to be expressed in total distance traveled by the entities, in total time spent by the entities, in financial cost, or using any other metric of interest to the user of the proposed technique.

[0050] In the remainder of the document, the solution is mainly described in the context of generating a schedule (or "planning" in English) intended to plan a period of interventions, typically the next working day, of technicians at customers' premises.The purpose of such a schedule is to schedule the technicians' interventions, so that they know which location to go to at each moment of their working day, the generated schedule also having to respect a set of predefined constraints (for example the need for a technician to have to go through a warehouse to retrieve particular equipment required for a subsequent intervention, the need to coordinate the movements of several technicians so that they meet at the same customer's premises at a given time slot when the intervention at this customer's premises cannot be carried out by a single technician, taking into account the unavailability of certain customers and / or technicians at certain time slots, etc.).Such an example of use is however purely illustrative and not limiting, and the present technique can also be implemented in many other contexts (for example to coordinate and optimize the movement of a plurality of tools towards a plurality of zones or locations to be treated of a part to be repaired and / or manufactured on a robotic production line, etc.). Another example of use of the proposed technique is provided later, in relation to the optimization of the routing of messages in a communication network.

[0051] The general principle of the proposed method for determining an optimized time sequence of movements to be performed by at least one entity to reach a set of predetermined locations is illustrated in relation to [Fig.l], in a particular embodiment. This method is implemented within an electronic device described later in this document.

[0052] In a step 11, a reinforcement learning environment is instantiated by the electronic device, based on configuration data provided as input to this device. Such data may, for example, in whole or in part, be entered by a human operator by means of a human-machine interface made available by the electronic device, or even be obtained automatically from various data sources via a communication network to which the electronic device is connected.

[0053] This configuration data includes location data, time data, and constraint data, detailed below. According to an analogy with In the gaming domain commonly used to present reinforcement learning techniques, location data and temporal data can be seen as representative of a game environment, while constraint data somehow defines the rules of this game.

[0054] More particularly, the location data define, in a given geographical reference system, the coordinates (for example in the form of longitude and latitude data, abscissas and ordinates, etc.) of various locations of interest, including in particular predetermined target locations to be reached by the entities in the context of a given activity (in other words, to use the previous analogy, in the context of the “game”), and additional locations of importance such as, for example, the starting locations of these entities, locations where certain resources (e.g., materials) are available, etc. In the context where the proposed technique is used to plan interventions by technicians at customers' premises, these locations of interest correspond to the possible locations to which the technicians may go during their travels. As illustrated in [Fig.2], which presents an example of mapping such locations, the location data then includes, for example, the geographical coordinates of the depots of the company employing the technicians (a depot D in the example of [Fig.2], where the technicians can, for example, collect or drop off specific equipment, for example a cradle for working at height), the technicians' homes (two homes El and E2 in the example of [Fig.2]), and the customers' homes where they can work (six intervention locations Cl to C6 in the example of [Fig.2]), all for a given geographical area ZG (for example the perimeter of a district, a town, a canton, etc.).According to a particular characteristic, such location data may comprise or be used to consolidate a table of distances between the locations of interest, also acting as configuration data when instantiating the reinforcement learning environment. The distances present within this table may be expressed in various ways, for example in average time required to reach two locations, in kilometer distance, in monetary cost, etc. The distances between each pair of locations may be obtained directly from the coordinates of these locations. These distances may for example be automatically acquired from software programs of the “estimated time of arrival” (ETA) type, such as Google Maps™ for example.

[0055] The time data defines for its part a total time period over which all the movements must have been carried out (also called the operation time period in the remainder of the document), thus possibly actually than a division of this time period into elementary time units (e.g. into time slots or time slices). In the context where the proposed technique is used to plan technician interventions at customers' premises, the operating time period considered corresponds for example typically to a working day (e.g. extending from 8 a.m. to 6 p.m.), divided into half-hour or one-hour slots (e.g. slot 1 from 8 a.m. to 8:30 a.m., slot 2 from 8:30 a.m. to 9 a.m., slot 3 from 9 a.m. to 9:30 a.m., etc.) which may for example correspond to slots for passages in locations D, El, E2, Clà C6 mentioned above.

[0056] Finally, the constraint data defines a set of constraints to be respected, the existence of which has an impact on determining the optimized time sequence of movements to be carried out by the different entities. In the context of the present technique, these constraints comprise at least one time constraint associated with at least one entity or at least one of the locations. In certain particular embodiments, these constraints further comprise at least one additional constraint associated with an activity that is the subject of the movements of the different entities. In the context where the proposed technique is used to plan interventions by technicians at customers' premises, the time constraints may concern technicians (e.g. the entities) and / or customers (e.g. the locations) indifferently.These time constraints take the form, for example, of scheduling constraints, representing the unavailability of at least one customer and / or at least one technician over one or more given time slots, or even constraints on the order of visits, etc. Additional constraints are typically business constraints, associated, for example, with the need to have access to certain resources - specific equipment or tools, a particular qualification (e.g., a driving license for a particular type of vehicle, a particular skill, etc.), a minimum number of technicians to carry out the intervention, etc. - at the time when an intervention must take place at certain predetermined target locations.

[0057] The obtaining of this configuration data by the electronic device is generally carried out upstream of the operating time period, during which the movements must be carried out. Typically, the acquisition of this data takes place for example a few hours before the time period to be planned (for example the day before the working day to be planned), in order to allow time for the electronic device to implement the training 12 and inference 13 phases described below to deliver an optimized time sequence of movements to be carried out by the different entities, and possibly to allow time for a human supervisor to verify the relevance of the time sequence thus determined.

[0058] In a step 12, a reinforcement learning type training phase is implemented using the learning environment configured in step 11. During this phase, at least one agent instantiated by said electronic device is trained to generate time sequences of movements to be carried out by at least one entity to reach all of the predetermined locations over the predefined time period, while respecting the time constraints and the additional constraints previously set.

[0059] Each attempt by an agent to generate (or explore) such a temporal sequence of movements during the training phase can be described as an episode, according to a common terminology used in the field of reinforcement learning. The training phase 12 generally comprises the implementation of a multitude of episodes (from several tens to several thousands).

[0060] More particularly, during this training phase by reinforcement learning, according to a trial-and-error type approach, multiple data exchanges are carried out between the agent and the learning environment, with a view to broadly exploring the results obtained in response to different sequences of actions corresponding to temporal sequences of movements tested.

[0061] Each of these exchanges includes, according to the usual vocabulary in reinforcement learning and as illustrated in relation to [Fig.3]:

[0062] - the transmission, by an AG agent, of action data D_ACT to the ENV learning environment;

[0063] - the reception, by the agent AG, in response to the transmission of said action data, of D_ETA state data and D_REC reward data from the ENV learning environment.

[0064] The state data D_ETA are representative of a new state of the reinforcement learning environment ENV, resulting from the action performed on this environment via the action data D_ACT.

[0065] The reward data D_REC are representative of one or more metrics of effectiveness of the action carried out on the ENV environment, with regard to the objective sought, namely the determination of an optimized temporal sequence of movements to be carried out by at least one entity to reach all of the predetermined target locations while respecting the fixed constraints (i.e. the temporal constraints and possibly the additional specific business constraints, as well as possibly any other type of predefined constraints).

[0066] The successive actions sent to the digital replica are determined by a reinforcement algorithm, so as to tend to maximize during an episode the weighted sum of the reward data D_REC received in response to these actions.

[0067] In a particular embodiment of the present technique, from a initial geographical position of the different entities, for example, we successively associate with each time slot (or time slice) of the time period of operation a location of the different entities with regard to the different available locations, as illustrated later in relation to [Fig.4]. The action data D_ACT transmitted by an agent to the learning environment ENV then includes a set of data representative of the location where each entity must go at the next time slot (or time slice) (it being understood that an absence of movement, that is to say the fact for an entity to remain positioned at its current location, is an available option). In other words, via the action data D_ACT, the agent AG indicates to the learning environment ENV the next location where each entity must go.The action data are thus representative of a next envisaged location of each entity with respect to its current location in the time sequence of movement being generated (i.e. explored).

[0068] As described previously, different information, delivered by the reinforcement algorithm, is then obtained by the agent AG, in response to the modification of the environment ENV resulting from the implementation of the action defined by the action data D_ACT.

[0069] First, reward data D_REC is calculated. This reward data typically takes the form of a simple numerical value r, typically a real number, the sum of different reward (positive values) and / or punishment (negative values) values ​​associated with the new locations of the entities, as defined by the action data D_ACT.

[0070] For example, the value r can be the result of a sum r=a + [3 + y + ô + e, in which:

[0071] - the value of the parameter a is a negative value (i.e. a punishment) added to each time slot in order to encourage the agent to arrive as quickly as possible at the end of an episode, i.e. to ensure that a solution is found so that all target locations are visited in the fewest possible "moves" (i.e. time elapsed, for example evaluated in number of required time slots);

[0072] - the value of the parameter [3 is a negative value (i.e. a punishment) proportional to the sum of the distances between the old and new location of each entity, so as to promote the shortest possible travel for each entity;

[0073] - the value of the parameter y is a positive value (i.e. a reward) of value average assigned each time a target location is first visited successfully (i.e., without violating one of the predefined constraints) by an entity;

[0074] - the value of the parameter ô is a large negative value (i.e. an im punishment bearing) assigned each time the new location of an entity generates non-compliance with one of the predefined constraints (in other words, the new location violates the rules of the game), such as for example a second visit to a target location already visited in a previous time slot, a visit to a target location in a time slot corresponding to the unavailability of an associated customer, a visit to a target location not respecting the human and / or material resource conditions required for the intervention (e.g. the need for two technicians to be present, the need for the technician present to have a particular qualification, the need for particular equipment or tools such as a cradle, etc.);

[0075] - the value of the parameter e is a large positive value (i.e. an im reward bearing) assigned when all target locations have been successfully visited (with possibly the additional optional condition, depending on the context of use, that the entities have also returned to their initial positions or to a predefined position, which can be translated, in the context of the example mainly described in this document, by the fact that all customers have received a visit from technicians, and that the latter have returned to their respective homes or to the depot of the company that employs them at the end of their tour).

[0076] The preceding example is of course given purely for illustrative purposes and is not limiting, other parameters (positive reward values ​​or negative punishment values) can be taken into account for the calculation of the reward data D_REC that the agent seeks to maximize during each episode.

[0077] Secondly, state data D_ETA (or “observation”) are delivered, in order to allow the agent AG to obtain at least a partial view of the state of the learning environment ENV (in other words, to use the analogy previously introduced, of the state of the game) following the implementation of an action. Such data include for example:

[0078] - the table of distances between locations;

[0079] - the total distance traveled by each entity;

[0080] - the visited or unvisited state of each location;

[0081] - a table of necessary resources associated with each location;

[0082] - a table of resources available at each location;

[0083] - a schedule of time constraints associated with each location;

[0084] - a schedule of time constraints associated with each entity;

[0085] - etc.

[0086] Furthermore, a boolean - possibly part of the D_ETA state data - is also returned to the AG agent, in order to indicate to it whether the current episode is in a "finished episode" or "unfinished episode" state. This boolean takes more particularly a value representative of a finished episode to mark what can be assimilated to an end of game or part. Thus, it takes for example the value "episode finished":

[0087] - when all the locations have been visited with a globally sa respect satisfying the imposed constraints (for example with a D_REC reward level reached above a predetermined threshold) and possibly that all the entities have returned to their initial positions (case of a game considered as “won”); or

[0088] - when too many constraints have not been respected, for example example because all locations could not be visited within a given time limit corresponding to the predefined operating time period, or because a reward value associated with the D_REC reward data did not reach or fell below a certain predetermined critical threshold due to the accumulation of too many punishments due to non-compliance with certain constraints (in particular temporal or additional trades) or because of a movement policy that was too inefficient (case of a game considered as "lost").

[0089] As long as the boolean is not positioned in the “episode finished” state (in other words as long as it is in the “episode not finished” state), the game is not over, and the agent can transmit new action data D_ACT to the learning environment ENV.

[0090] Thus, at the end of the learning phase 12, at least one agent has been trained on a large number of episodes - in other words, the weights of a model associated with the agent have been progressively adjusted during the episodes - so that the agent is able to determine temporal sequences of movements that are as optimal as possible to reach a set of predetermined target locations while respecting predefined constraints.

[0091] In a step 13, at the end of the training phase, an inference phase is implemented, in which at least one agent trained in step 12 is used to determine and deliver the optimized time sequence of movements to be carried out by the different entities to reach all of the predetermined target locations.

[0092] In the context where the proposed technique is used to plan interventions by technicians at customers' premises on a given working day, such an inference phase 13 is typically implemented the evening before or early in the morning of said working day, in order to generate the schedule of the rounds (i.e. the interventions) to be carried out on the day by each of the technicians.

[0093] An example of a temporal sequence of movements explored during the training phase 12 or delivered at the output of the inference phase 13 is presented in relation with [Fig.4], in such a context of planning technician interventions at customers' premises. This purely illustrative example is based on configuration data similar to those of [Fig.2], namely the determination of a schedule for the tour of two technicians Tl and T2, domiciled respectively at locations El and E2, to intervene during a working day divided into seven successive time slots tl to t7 at the homes Cl to C6 of six customers. The company employing technicians Tl and T2 also has a warehouse D for storing equipment. As illustrated in [Fig.4], the time sequence of movements defines for each of the technicians Tl and T2, at each time slot, their respective position with respect to one of the locations of interest El, E2, D, Cl, C2, C3, C4, C5 or C6.According to the explored (step 12) or delivered (step 13) schedule, it is for example planned that technicians T1 and T2 are at their respective homes E1 and E2 at the first time slot t1 and the last time slot t7 of the day, and that all the customers' homes C1 to C6 are visited by at least one of the technicians in the time slots t2 to t5. In the example illustrated, at least one constraint (here respected) concerns the intervention at the home of customer C2, namely for example a long intervention, requiring at least part of the intervention the presence of two technicians and a nacelle-type device. Also, according to the proposed schedule, the two technicians T1 and T2 are present together at location C2 in the time slot t5, and time has been planned for technician T1 to retrieve in advance, in the time slot t4, the nacelle required from the company's warehouse D.In time slot t6, for example, it is planned that technician T2 completes the intervention alone at the home of customer C2 (thus characterizing an absence of travel for technician T2 between time slots t5 and t6), while technician T1 is responsible for returning the nacelle, which is useless for this second part of the intervention, to the company's warehouse D. Each of the time slots t2 to t7 is also associated with a reward value D_REC, reflecting the relevance associated with the new locations of technicians T1 and T2 in the time slot considered, compared to their previous locations. These rewards are, for example, calculated according to the methods presented previously in relation to [Fig. 3], and their sum over all the time slots makes it possible to evaluate the quality of the produced schedule, particularly with regard to compliance with the predefined constraints set.

[0094] Of course, other modes of representation than a table such as presented in [Fig.4] can be used within the framework of the present technique to communicate to a user a temporal sequence of movements according to the present technique. By way of illustration, such a sequence can also be represented in a cartographic form, the movements determined for each entity then taking the form of routes plotted on a map of the type illustrated in relation to [Fig.2] by example.

[0095] In particular embodiments, various approaches presented below can be adopted, optionally and possibly complementary, in order, for example, to make the method for determining an optimized time sequence of movements according to the present technique more efficient and / or more robust.

[0096] According to a first approach, rather than presenting the problem with all the constraints to be solved from the start of training, it is possible to implement training techniques with progressive difficulty (known by the English term "curriculum learning"). According to this approach, a progressive increase in the level of difficulty is implemented during training phase 12 of [Fig.l], as the agent manages to successfully complete episodes. In other words, new constraints are added in stages, each time the agent's reinforcement algorithm successfully completes a given number of episodes (i.e. reaches a predetermined reward threshold for a given number of episodes), and this until all the predefined constraints are finally integrated.Such progressive learning allows for faster convergence and a more precise model to be obtained than if all the constraints had been introduced from the start of learning.

[0097] According to a second approach, conditional action masking techniques are implemented during training phase 12, consisting of reducing the number of possible actions (i.e. the range of possibilities) as an episode progresses. For example, in a particular embodiment, a masking technique consists of configuring the agent so that it is no longer allowed to visit a location that has already been successfully visited in a previous time slot of the episode. In this way, the agent learns more quickly, with the possible actions to be explored by the agent decreasing over time due to the fact that certain actions become unavailable in certain time slots.

[0098] According to a third approach, it is proposed in a particular embodiment to instantiate and train not a single agent during the training phase 12, but a plurality (i.e. at least two) agents. During the inference phase 13, an inference is carried out by each of the agents thus previously trained, and the sequence retained as the optimized temporal sequence of movements to be carried out is that delivered by the agent having maximized the reward data obtained during this inference phase. Such a technique makes it possible to benefit during the inference phase from the capacities of a group of pre-trained agents (rather than a single agent), and to ultimately retain the best of the optimized temporal sequences of movements determined by these different agents, thus increasing the performance of the proposed technique. According to a particular characteristic, during the training phase 12, the different agents of the plurality of agents are trained sequentially (i.e. one after the other), in order to limit the number of resources required (in terms of computing power, memory, etc.) to carry out the training, which makes it possible to implement the present technique even on machines with potentially limited capacities.

[0099] According to a fourth approach, the method comprises, prior to reinforcement learning during which at least one agent interacts with the learning environment, a so-called offline learning phase during which the agent does not interact with such an environment, but learns (i.e. gradually adjusts the weights of its model) by means of a learning base provided as input to the agent. More particularly, this AHL offline learning phase illustrated by the optional step 10 of [Fig.l] can be described as an imitation learning phase, aimed at carrying out pre-modeling of the agent on the basis of pre-existing temporal sequences of movements (or pre-existing agendas) forming said learning base.Such pre-existing time sequences of movements correspond, for example, to sequences identified by human operators and provided during the solution configuration operations according to the present technique, or generated by elementary scripts. These pre-existing time sequences of movements do not necessarily correspond to the most optimal solutions, but they allow an agent to which they are communicated not to start the reinforcement learning phase - called online learning AEL - from scratch, and therefore allow it to learn more quickly and more efficiently. In other words, by analogy with the field of video games for example, such prior offline learning AHL (step 10 of [Fig. l]) allows, for example, the agent to acquire a “human” level of play, and the subsequent online learning AEL (steps 11 and 12 of [Fig.l]) then allows it to progress, by reinforcement, until reaching a level of play comparable to a “superhuman” level. Optionally, according to a particular characteristic and as illustrated in [Fig.l], a BR feedback loop can also be implemented in this context, in order to continuously feed the learning base used for offline learning with data from the implementation of the method according to the present technique. Such data include, for example, all or part of the optimized time sequences of movements delivered at the end of the inference phases at each use of the method. They can also include traces (or logs) of the time sequences of movements that actually took place, in other words field interventions.These time sequences actually implemented on the ground may in fact differ more or less significantly from those delivered at the end of the phase. inference for various reasons. The optimized time sequence delivered at the end of the inference phase may, for example, have been subject to adaptations during a check by a human supervisor before its implementation. It may also have been modified in the field itself to take into account unforeseen circumstances, for example when the technicians in charge are forced to adapt the initial schedule to deal with unforeseen situations (localized weather phenomena, roadworks, uncommunicated unavailability of a customer, etc.). Some of the solutions thus adapted may be suboptimal, i.e., for example, not minimizing the total time elapsed to visit all locations and / or the total distance traveled by the fleet of technicians for this purpose, but they nonetheless constitute interesting data to feed the learning base and further improve the performance of the process according to the proposed technique.According to a particular characteristic, when the traces of the time sequences of movements actually carried out are available in real time or quasi-real time, the feedback loop BR can also be used to identify in real time and / or continuously an optimized time sequence of movements remaining to be carried out, in particular when the reality of the terrain has imposed deviations from the time sequence of movements initially calculated. In such an embodiment, the entities have for example means (e.g. devices or communication interfaces) allowing them to exchange, via a communication network, notification messages with the electronic device responsible for implementing the method for determining optimized time sequences of movements according to the present technique.In this way, technicians can, for example, send useful information to the electronic device in real time, including confirmation that the visit to a location (e.g., a customer's) has been successfully completed in accordance with the initially planned schedule, or, if necessary, information that a visit initially planned in the schedule cannot ultimately be carried out on time for a particular reason that may be specified in the message. In this second case, upon receipt of this notification, the electronic device can then trigger, on the basis of this new information, the determination of a new optimized time sequence of movements, and communicate this updated sequence to all the technicians in the field, for consideration.The determination of a new updated optimized time sequence of movements can also in certain situations be triggered at the initiative of the electronic device itself, or upon receipt by this device of information from data sources other than the entities (for example in the event of detection of alerts or receipt of alert messages such as weather alerts, congestion alerts at points par-. individuals of a network, etc.).

[0100] As indicated at the beginning of the document, although mainly described in the context of generating a schedule intended to plan the trips of technicians to customers, the present technique is not limited to this use case, and it can be used in many different fields. In a particular embodiment, it can thus for example be used to determine an optimized time sequence for routing messages within a communication network infrastructure, in particular in the context of the implementation of functionalities based on chaining of services, in which the services in question are potentially hosted at different locations in the communication network, for example within different physical servers and / or different virtual machines.According to a particular characteristic, the messages are for example representative of network tasks to be executed, which must be processed within the different services, taking into account time constraints (e.g. task A must be processed before task B by a given service, which defines constraints relating to the order of reception of at least certain messages within this service), and possible additional constraints (e.g. task C must imperatively be processed by a given service, which is therefore an obligatory point of passage for the message associated with this task).In other words, the implementation of the present technique then makes it possible to determine a schedule for optimal routing of the different messages within the communication network, while taking into account constraints specific to this type of use - for example the possibility for the same message to be routed several times to the same virtual machine, if the general processing of the tasks associated with this message requires it, but the prohibition for the same virtual machine to simultaneously process several different messages - which can in this sense be described as business constraints.In this context, "optimal routing" can, for example, be understood as one that makes it possible to limit the carbon footprint resulting from the management of these messages within the communication network and / or to be able to ensure that the chaining of services respects a certain delay corresponding, for example, to a quality of experience requirement for a user. Thus, an application requiring low latency may require a chaining of elementary services or network functions executed in a time less than a value representative of the latency. The different types of data presented in the context of the present technique - in particular configuration data, action data, state data, and reward data - are then adapted to this context.For example, reward data may consider carbon cost values ​​associated with network links used to route a message between two virtual machines or . servers (a negative value, or punishment, proportional to the sum of the carbon costs associated with the network links between the old and new location of each message being assigned, for example, so as to favor the least carbon-emitting routes possible for each message). Similarly, reward data can take into account the successful completion of a task (by associating it with a reward of average positive value) or of all tasks (by associating it with a reward of large positive value).

[0101] According to another aspect, the proposed technique also relates to a device for determining an optimized time sequence of movements to be carried out by at least one entity to reach a set of predetermined locations. Such an electronic device is capable of carrying out the method previously described in any of its embodiments. More particularly, such a device according to the present technique comprises, in a particular embodiment:

[0102] - means for instantiating a reinforcement learning environment, based on configuration data provided as input to said electronic device, said configuration data comprising location data of said locations, time data associated with a predefined time period for the implementation of said movements, and constraint data comprising at least one time constraint and at least one additional constraint;

[0103] - means for implementing a learning-type training phase by reinforcement, in which at least one agent instantiated by said electronic device is trained to generate time sequences of movements to be carried out by said at least one entity to reach said set of predetermined locations over said predefined time period while respecting said constraints, on the basis of at least one exchange of data between said at least one agent and said environment;

[0104] - means for implementing an inference phase, in which said at least a trained agent is used to deliver said optimized time sequence.

[0105] [Fig. 5] represents, in a schematic and simplified manner, the structure of such an electronic device, in a particular embodiment. The device, according to the proposed technique, comprises for example a memory 51 consisting of a buffer memory M, a processing unit 52, equipped for example with a microprocessor pP, and controlled by the computer program Pg 53, implementing steps of the method for determining an optimized temporal sequence of movements according to at least one embodiment of the invention. To this end, the electronic device also comprises at least one communication interface (for example an Ethernet communication interface) and / or at least one human- machine, enabling it to receive and transmit data from and to other equipment present in a communications network to which it is connected and / or human operators.

[0106] At initialization, the code instructions of the computer program 53 are loaded into the buffer memory before being executed by the processor of the processing unit 52. The processing unit 52 receives as input E, for example, configuration data by means of which it can instantiate a reinforcement learning environment.

[0107] The microprocessor of the processing unit 52 then carries out the following steps of the method for determining an optimized time sequence, according to the instructions of the computer program 53. More particularly, the processing unit 52 instantiates at least one agent, and trains it - on the basis of data exchanges with said reinforcement learning environment - to generate time sequences of movements to be carried out by at least one entity to reach the set of predetermined locations, while respecting constraints predefined in the configuration data. At the end of this training, the agent thus trained is then used to generate an optimized time sequence of movements, which is then delivered by the processing unit 52 as output S.

Claims

Claims

1. Method for determining an optimized temporal sequence of movements to be performed by at least one entity to reach a set of predetermined locations, said method being implemented by an electronic device, said method being characterized in that it comprises: - the instantiation (11) of a reinforcement learning environment (ENV), as a function of configuration data provided as input to said electronic device, said configuration data comprising location data of said locations, temporal data associated with a predefined temporal period for the implementation of said movements, and constraint data comprising at least one temporal constraint;- a training phase (12) of the reinforcement learning type, in which at least one agent (AG) instantiated by said electronic device is trained to generate at least one temporal sequence of movements to be carried out by said at least one entity to reach said set of predetermined locations over said predefined temporal period while respecting said constraints, on the basis of at least one exchange of data between said at least one agent (AG) and said environment (ENV); - an inference phase (13), in which said at least one trained agent is used to deliver said optimized temporal sequence.;

2. Method according to claim 1, characterized in that said at least one time constraint comprises data representative of an unavailability of at least one of said at least one entity and / or of at least one of said locations over at least one time slice of said predefined time period.

3. Method according to claim 1, characterized in that said constraint data further comprises at least one additional constraint.

4. Method according to claim 3, characterized in that said at least one additional constraint is associated with an activity which is the subject of said movements.

5. Method according to claim 1, characterized in that said at least one data exchange comprises: - the transmission, by said at least one agent, of action data to- destination of said learning environment, said action data being representative of a next envisaged location of said at least one entity relative to a current location of said at least one entity in the generated time sequence of movement; - the reception, by said at least one agent, in response to said transmission, of state data and reward data from said learning environment; the successive action data transmitted to said learning environment within the framework of said at least one exchange being determined so as to tend to maximize over time the reward data received.

6. A method according to claim 1, characterized in that said training phase comprises training a plurality of agents, and in that said inference phase comprises delivering by a plurality of said trained agents a plurality of candidate movement time sequences, and selecting said optimized movement time sequence from among said plurality of candidate movement time sequences, based on reward data associated with each of said candidate movement time sequences.

7. Method according to claim 6, characterized in that the agents of said plurality of agents are trained sequentially.

8. Method according to claim 1, characterized in that said training phase comprises the implementation of a plurality of successive training episodes, an increasing number of constraints from a set of constraints defined by said constraint data being progressively added for the configuration of said episodes, as said implementation of said successive episodes progresses, until reaching all of said constraints defined by said constraint data.

9. Method according to claim 1, characterized in that it comprises, prior to said reinforcement learning type training phase, an offline learning phase comprising training of said at least one agent by means of a learning base comprising data representative of a plurality of pre-existing time sequences of movements.

10. Method according to claim 9, characterized in that it comprises a feedback loop (BR) used to complete said base learning with data representative of said optimized time sequence delivered during said inference phase.

11. Electronic device for determining an optimized temporal sequence of movements to be carried out by at least one entity to reach a set of predetermined locations, said device being characterized in that it comprises: - means for instantiating a reinforcement learning environment, as a function of configuration data provided as input to said electronic device, said configuration data comprising location data of said locations, temporal data associated with a predefined temporal period for the implementation of said movements, and constraint data comprising at least one temporal constraint;- means for implementing a reinforcement learning type training phase, in which at least one agent instantiated by said electronic device is trained to generate at least one time sequence of movements to be carried out by said at least one entity to reach said set of predetermined locations over said predefined time period while respecting said constraints, on the basis of at least one exchange of data between said at least one agent and said environment; - means for implementing an inference phase, in which said at least one trained agent is used to deliver said optimized time sequence.;

12. Computer program product downloadable from a communications network and / or stored on a computer-readable medium and / or executable by a microprocessor, characterized in that it comprises program code instructions for executing a method according to any one of claims 1 to 10, when executed by a computer.

Images