Method and electronic system for developing a navigation module for an autonomous drone based on navigation rules; navigation module for an autonomous drone

Abstract

Method, and electronic system, for developing a navigation module for an autonomous drone based on navigation rules, navigation module for an autonomous drone. Method for developing a navigation module (22) for an autonomous drone (1) comprising: selection of an action (At) from among an action determined by reinforcement learning and a navigation action (At, 2) determined by a rule-based module based on a first received state value of the current environment; updating the reinforcement learning based on the second current state value of the environment obtained following the implementation of the selected navigation action; and if there is no rule in the rule base whose said first current state value satisfies the condition, then creation of a new rule;and / or for a given rule in the rule base whose first current state value satisfies the condition and whose result is said selected action, replacement of said given rule with a modified rule. Figure for the abbreviation: Figure 1;

Description

Title of the invention: Method and electronic system for developing a navigation module for an autonomous drone based on navigation rules, navigation module for an autonomous drone

[0001] Drone piloting can be classified into three categories: remotely piloted drones, semi-autonomous drones and fully autonomous drones.

[0002] Methods for developing a navigation module are known from the article "Au-tonomous Learning Of Sequential Tasks: Experiments And Analyses" by Sun R et Al, and from document US 2020 / 193333 Al.

[0003] The present invention relates more specifically to autonomous drone navigation, via the development of a completely autonomous navigation model.

[0004] Autonomous drones are particularly useful in many types of missions: surveillance, information gathering, inspection, rescue, etc.

[0005] A precise definition of what is meant by autonomous drone can be found here, for example: https: / / en.rn.wikipedia.org / wiki / Unmanned_aerial_vehicle.

[0006] The first technologies developed for drone navigation are based on graph traversal algorithms. These methods require complete knowledge of the environment via a map that is pre-constructed and stored in memory.

[0007] Several artificial intelligence technologies have made it possible to limit the need for knowledge to only the environment close to the drone:

[0008] - Expert systems: these systems contain navigation rules previously defined by domain experts and are implemented in the form of code in the drone's computing unit. These rules define the navigation actions to be performed by the drone based on the states of the sensors or any data that can be used by the drone. Often based on physical / mathematical equations or control models such as PD, PI and PID. - Machine learning models: these systems go through a learning phase to learn the navigation actions that the drone must perform based on the states of the sensors or any data that can be used by the drone.

[0009] In the case of expert systems, the rule sets developed by experts may suffer from certain weaknesses. The rules may not be complete or some rules may be wrong.

[0010] In the context of models created by machine learning, these are black box systems that suffer from a lack of interpretability and explainability. However, these properties are essential in critical systems to understand the decisions made by the system, especially in the event of problems

[0011] There is therefore a need to have an autonomous drone navigation solution that is efficient, explainable and interpretable, and thus more easily certifiable.

[0012] To this end, according to a first aspect, the invention proposes a method for developing a navigation module for an autonomous drone based on navigation rules, characterized in that it is implemented by an electronic system comprising:

[0013] - a reinforcement learning machine module adapted to determine at least one drone navigation action based on a current state of the drone's environment; and

[0014] - a rule-based module, comprising a navigation rule base where a rule is defined by a condition and by an action which is the result of said rule if and only if said condition is respected,

[0015] the state of the environment of the drone being a vector of dimension N, said state of the environment and the navigation actions of the drone having each been discretized into a finite number of values,

[0016] and in that the following steps are implemented iteratively:

[0017] i. receiving a first value of the current state of the drone's environment;

[0018] ii. determination, by the automatic reinforcement learning module, of one or more first navigation actions as a function of said first received current state value and determination by the rule-based module of one or more second navigation actions as a function of said first received current state value;

[0019] iii. selection, by said system, of an action from among said first and second actions;

[0020] iv. updating the reinforcement learning of the reinforcement learning module as a function of the second current state value of the environment obtained following the implementation of the selected navigation action; and in parallel

[0021] v. updating the rules of the rule-based module, said updating comprising at least one of the following steps:

[0022] vi if there is no rule in the rule base for which said first current state value satisfies the condition, then creation of a new rule in the rule base, the condition of which is that the value of the current state is equal to said first current state value, and the conclusion of which is said selected action;

[0023] v.ii. for a given rule of the rule base whose first current state value verifies the condition and whose result is said selected action, said given rule is replaced, in the rule base, by a modified rule whose result is said selected action and whose condition includes said condition of the given rule and an additional dimension among the N dimensions not taken into account in the condition and / or by a modified rule whose condition includes said condition of the given rule from which a dimension has been removed, depending on the comparison of the relevance of the rule to the relevance of the modified rule.

[0024] The invention is based on an architecture that hybridizes reinforcement learning and rule-based systems to extract a navigation model in the form of a set of rules.

[0025] The invention allows the development of a navigation system which is at the same time autonomous, interpretable, explainable and therefore, by extension, easier to certify.

[0026] The rules generated during the learning phase are based on the definition of the states of the environment and the possible actions of the drone. They are therefore more easily interpretable and explainable than black box systems which are currently the majority of navigation models created from artificial intelligence techniques.

[0027] The ruleset is then used by the drone to manage its navigation and its mission in general.

[0028] In embodiments, a method according to the invention further comprises one or more of the following features:

[0029] - the determination by the reinforcement learning module of said, or said first navigation actions as a function of said first current state value received is a function of the current values ​​of evaluation parameters associated with the respective pairs (first current state value, action) and which evaluate the relevance of performing said action in said state;

[0030] - updating the reinforcement learning of the learning module by reinforcement includes an update of the value of the evaluation parameter associated with the pair (first current state value, selected action) based on the second current state value of the environment obtained; and

[0031] - the relevance of the rule, respectively of the modified rule, is evaluated in function of the evaluation parameters associated with the respective pairs (state, action) such that the state verifies the condition of the rule, respectively the condition of the modified rule, and the action is the action of said rule which is also that of the modified rule;

[0032] in step vi, said new rule is only created if it verifies a utility criterion determined as a function of at least said updated value of the evaluation parameter associated with the pair (first current state value, selected action);

[0033] said action selection step comprises the following steps:

[0034] for each of said actions among said first and second actions, calculation of a score associated with said action and equal to a weighted sum of the current value of the evaluation parameter associated with the pair (first current state value, said action) and k,

[0035] where k takes the value 1 if said action is part of the first actions and takes the value 0 if said action is not part of the first actions, the respective weights being predefined values.

[0036] According to a second aspect, the present invention proposes a navigation module for an autonomous drone based on navigation rules, said navigation module being characterized in that it is generated by an electronic system for developing a navigation rules module for an autonomous drone comprising:

[0037] - a reinforcement learning machine module adapted to determine at least one drone navigation action based on a current state of the drone's environment; and

[0038] - a rule-based module, comprising a navigation rule base where a rule is defined by a condition and by an action which is the result of said rule if and only if said condition is respected,

[0039] the state of the environment of the drone being a vector of dimension N, said state of the environment and the navigation actions of the drone having each been discretized into a finite number of values, such that iteratively:

[0040] i. said system receives a first value of the current state of the environment of the drone;

[0041] ii. the reinforcement learning module determines one or more first navigation actions based on said first received current state value and the rule-based module determines one or more second navigation actions based on said first received current state value;

[0042] iii. said system selects an action from among said first and second actions;

[0043] iv. the reinforcement learning module updates its reinforcement learning based on the second current state value of the environment obtained following the implementation of the selected navigation action; and in parallel

[0044] v. said system updates the rules of the rule-based module, in the following manner

[0045] vi if there is no rule in the rule base for which said first current state value satisfies the condition, then creation of a new rule in the rule base, the condition of which is that the value of the current state is equal to said first current state value, and the conclusion of which is said selected action; and / or

[0046] v.ii. for a given rule of the rule base whose first current state value verifies the condition and whose result is said selected action, replacement, in the rule base, of said given rule by a modified rule whose result is said selected action and whose condition includes said condition of the given rule and an additional dimension among the N dimensions not taken into account in the condition and / or by a modified rule whose condition includes said condition of the given rule from which a dimension has been removed, depending on the comparison of the relevance of the rule to the relevance of the modified rule.

[0047] According to a third aspect, the present invention proposes an electronic system for developing a navigation rules module for an autonomous drone comprising:

[0048] - a reinforcement learning machine module adapted to determine at least one drone navigation action based on a current state of the drone's environment; and

[0049] - a rule-based module, comprising a navigation rule base where a rule is defined by a condition and by an action which is the result of said rule if and only if said condition is respected,

[0050] the state of the environment of the drone being a vector of dimension N, said state of the environment and the navigation actions of the drone having each been discretized into a finite number of values,

[0051] i. said system is adapted to iteratively receive a first value of the current state of the environment of the drone;

[0052] ii. the automatic reinforcement learning module is adapted to iteratively determine one or more first navigation actions as a function of said first received current state value and the rule-based module is adapted to iteratively determine one or more second navigation actions as a function of said first received current state value;

[0053] iii. said system is adapted to iteratively select an action from among said first and second actions;

[0054] iv. the reinforcement learning module is adapted to iteratively update its reinforcement learning as a function of the second current state value of the environment obtained following the implementation of the selected navigation action; and in parallel

[0055] v. said system is adapted to iteratively update the rules of the rule-based module, in the following manner

[0056] vi if there is no rule in the rule base for which said first current state value satisfies the condition, then creation of a new rule in the rule base, the condition of which is that the value of the current state is equal to said first current state value, and the conclusion of which is said selected action; and / or

[0057] v.ii. for a given rule of the rule base whose first current state value verifies the condition and whose result is said selected action, replacement, in the rule base, of said given rule by a modified rule whose result is said selected action and whose condition includes said condition of the given rule and an additional dimension among the N dimensions not taken into account in the condition and / or by a modified rule whose condition includes said condition of the given rule from which a dimension has been removed, depending on the comparison of the relevance of the rule to the relevance of the modified rule.

[0058] In embodiments, the electronic system for developing a navigation rules module for an autonomous drone according to the invention further comprises at least one of the following characteristics:

[0059] - the determination by the reinforcement learning module of said, or said first navigation actions as a function of said first current state value received is a function of the current values ​​of evaluation parameters associated with the respective pairs (first current state value, action) and which evaluate the relevance of performing said action in said state;

[0060] - updating the reinforcement learning of the learning module by reinforcement includes an update of the value of the evaluation parameter associated with the pair (first current state value, selected action) based on the second current state value of the environment obtained; and

[0061] - the relevance of the rule, respectively of the modified rule, is evaluated in function of the evaluation parameters associated with the respective pairs (state, action) such that the state verifies the condition of the rule, respectively the condition of the modified rule, and the action is the action of said rule which is also that of the modified rule;

[0062] - in the updating by said system of the rules of the rule-based module, if it there is no rule in the rule base for which said first current state value satisfies the condition, then a new rule is created in the rule base, the condition of which is that the value of the current state is equal to said first current state value, and the conclusion of which is said selected action, the system being adapted to create said new rule only if it satisfies a utility criterion determined as a function of at least said updated value of the evaluation parameter associated with the pair (first current state value, selected action);

[0063] - when selecting the action, the system is adapted to calculate for each of said actions among said first and second actions, a score associated with said action and equal to a weighted sum of the current value of the evaluation parameter associated with the pair (first current state value, said action) and k, where k takes the value 1 if said action is part of the first actions and takes the value 0 if said action is not part of the first actions, the respective weights being predefined values.

[0064] These characteristics and advantages of the invention will appear on reading the description which follows, given solely by way of example, and made with reference to the appended drawings, in which:

[0065] [Fig.l] [Fig.l] represents a schematic view of an autonomous navigation system of a drone in one embodiment of the invention;

[0066] [Fig.2] [Fig.2] represents a system for developing a rules module for the autonomous navigation system of a drone of [Fig.l];

[0067] [Fig.3] [Fig.3] is a flowchart of steps implemented in one embodiment of the invention;

[0068] [Fig.4] [Fig.4] illustrates an environment of an autonomous drone considered in a embodiment of the invention;

[0069] [Fig.5] [Fig.5] illustrates the actions of the autonomous drone considered in [Fig.4].

[0070] [Fig.l] represents a schematic view of an autonomous navigation system 20 of a drone 1 in one embodiment of the invention.

[0071] This navigation system, carried on board the drone during its operational mission, comprises a set 21 of sensors, a MODNav navigation module 22 and a set of effectors 23.

[0072] The set 21 of sensors comprises sensors of physical quantities representative of the environment of the drone 1 or of time derivatives of these physical quantities, for example, of distance, of rotation (roll angles, yaw, heading), of temperature, of humidity, of pressure etc.

[0073] The set of effectors 23 comprises for example a set of motor(s), driving a set of propeller(s), control surfaces, etc.

[0074] The set of sensors 21 is adapted to determine the state of the environment of the drone 1 and provide it to the M0DNAV navigation module 22. The state of the environment is represented in the form of an N-dimensional vector in a discrete space.

[0075] The state of the environment is therefore a vector of physical quantities representative of the environment of the drone 1 or of time derivatives of these physical quantities, for example, of distance, of rotation (roll angles, yaw, heading), of temperature, of humidity, of pressure etc. (it comprises all the dimensions necessary for the hybrid learning block 42 to represent the world and to determine the necessary navigation actions). The vector is for example of dimension N, N for example is strictly greater than 1; the different dimensions of the vector are for example diml, ..., dimN.

[0076] The navigation module M0DNAV 22 is a rule-based module comprising a base of navigation rules each expressed in the form:

[0077] if CONDITION then action no. i;

[0078] indicating that if and only if the condition is verified, the result of the application of the rule is action no. i;

[0079] where the condition is for example aiAa2A... Aap where the aj are each associated with a distinct dimension of the state and have a particular value fixed among the values possible discrete values ​​in this dimension for j=l to p and where p is strictly less than or, in cases equal, to N, for example p = 1 (p can take distinct values ​​following the rules)

[0080] action no. i is a specific action in a predetermined discrete action space, each action corresponding to specific effector commands of the effector set 23.

[0081] The navigation module M0DNAV 22 is adapted, depending on the current state received, to determine the rule(s) of the rule base whose conditions are met and to command the effectors to perform the corresponding navigation actions indicated by this, or these, rules.

[0082] The rule-based navigation module M0DNAV 22 was obtained by implementing a hybrid learning method 102 according to the invention as described in [Fig.3] and implemented on the navigation module development system 40 shown in [Fig.2].

[0083] Thus the navigation module development system 40 comprises an electronic simulator 41 and a hybrid electronic learning block 42.

[0084] The hybrid learning electronic block 42 comprises a reinforcement learning module 43, a rule-based module 22_a and a selection module 45.

[0085] Iterative steps are implemented by the navigation module development system 40, during which the reinforcement learning module 43 and the rule-based module 22_a evolve, until the latter becomes a completed M0DNAV navigation module 22, ready to be inserted into a drone 1 (this drone thus becoming autonomous).

[0086] The electronic simulator 41: it is adapted to represent as faithfully as possible the operational context and the mission to which the drone is subject. In particular, it is adapted to determine the state of the environment St+i at time t+1 as a function of actions At which are “commanded” to it at time t, in state St. As such, the electronic simulator 41 is adapted to calculate all the possible transitions St (At) St+i. In the context of a drone mission, this will often result in the implementation of a physical calculation function such as applying a force to an object. The rules of the mission are also added, such as, for example, the calculation of the exit from a specific zone. Thus, the learning module is adapted to use it to learn the transitions.

[0087] The automatic reinforcement learning module 43 is adapted to implement an automatic learning method of the reinforcement learning type from states of the environment to determine the navigation actions according to said states and reward values ​​evaluating the determined actions. previously.

[0088] As is known, reinforcement learning consists of learning the actions to take, depending on the current state of the environment from interactions with the simulator, so as to optimize the quantitative reward over time which can be positive or negative. A reinforcement learning module seeks, through iterated experiments, optimal decision-making behavior (called strategy or policy, and which is a function associating the action to be executed with the current state), in the sense that it maximizes the expected gain (for example, the sum of rewards over time):

[0089] The rule-based module 22_a: it comprises a rule base, the form of these rules being similar to that set out above for the rule-based module 22.

[0090] The invention could for example be based on Clarion® (www.clarioncognitivearchitecture.com / downloads), which is a cognitive architecture model which, although it comes from a field of research other than that of artificial intelligence, nevertheless offers an interesting basis for hybridizing connectionist artificial intelligence and symbolic artificial intelligence in the present solution for obtaining navigation rules which are based on the representation of the states of the environment.

[0091] The basic postulate is as follows: the low-level knowledge corresponding to the reinforcement learning of the reinforcement learning module 43 corresponds in humans to procedural knowledge or “know-how”, whereas the high-level knowledge corresponding to that of the rule-based module 22 corresponds to declarative knowledge, i.e. to rules that can be stated in clear language. The representation of the low-level knowledge corresponds to a digital representation (i.e. the link between actions and states), in reinforcement learning, based on the Q-value. The high-level knowledge is represented by the set of logical rules. The hybrid learning logic according to the invention, from the bottom up, corresponds to the fact that the low-level knowledge is first learned and then the rules that constitute the high-level knowledge are extracted from this knowledge.Both levels of knowledge then participate in decision-making for continued learning, as detailed later.

[0092] [Fig. 3] represents a method 100 according to the invention, which comprises an initialization phase 101, a hybrid learning phase 102 and an operational phase 103.

[0093] The initialization phase 101 comprises the preliminary steps 1011, 1012, 1013 and 1014.

[0094] Step 1011 is a step of fine-tuning the simulator 41 as defined above.

[0095] Step 1012 is a step of defining the reward function. As its As the name suggests, reinforcement learning is based on "reinforcement", or more commonly called "reward", to learn. It is therefore necessary to define a reward function that allows the hybrid learning block 42 to learn the correct "state-action" associations (also called policy). Concretely, the unwanted states must be penalizing and the desired states rewarded. Depending on the embodiments, the reward is calculated by the simulator 41 or the hybrid learning block 42.

[0096] Step 1013 is a step of defining the discretization of actions and states. The hybrid learning block 42 implements a reinforcement learning algorithm which requires having discrete states and actions. However, the automatic piloting of a drone in a real operational context is based on continuous variables (speed, angle, etc.).

[0097] Step 1014 is a step for defining parameters of the system 40, in particular:

[0098] - the discount factor E [0,1] is used in the reinforcement learning module. During learning, the discount factor is used to determine the importance of future rewards. - the learning rate e [0,1], determines how the newly acquired information outperforms the old. At 0, no learning takes place, the new information never being taken into account. At 1, the agent does not consider the past and only takes into account the new information. - the success threshold of a state-action association: this threshold determines from when a State -> Action association present at the low level, in the reinforcement learning module, must be created as a rule within the rule module. - the stopping or convergence criterion of the learning phase, indicating the number of iterations in step 102 or an overall reward value over N episodes.

[0099] The hybrid learning phase 102 is now described, comprising the reiteration, at each time step t, of the steps below.

[0100] At time step t, the simulator 41 determines the current state of the environment Sten as a function of the action At.i previously transmitted by the hybrid learning block 42. The simulator 41 further determines the updated value Rt of the reward as a function of the state St. The simulator 41 delivers this current state St and the updated reward Rt to the hybrid learning block 42.

[0101] At the beginning of hybrid learning, the rule base does not contain any rules and for the reinforcement learning module, all Q-values ​​are initialized to 0.43 for example.

[0102] At time step t, in a step 1021, the hybrid learning block 42 receives the current state St (and the updated reward Rt 0 from the simulator 41.

[0103] In a step 1022, one or more navigation actions are determined in parallel by the reinforcement learning module 43 and the rule-based module 22_a, as a function of the current state St:

[0104] - the reinforcement learning module 43 determines the action At4 or the set of possible actions {At>i}, at the current stage of its learning and calculates Q(St, A i) for each of these possible actions At4; Q() is Q-value of the association “perform action At4 in the current state St” defined by the Bellman equation as the expected gain of choosing action A in state S.

[0105] Q(s,a)~ E(Gt\St = s, At = a)

[0106] with Gt the gain the sum of the rewards expected in the future weighted by the discount factor.

[0107] Gt= R;+1+ yRt^_+ ... =

[0108] The Q-value update equation differs depending on the reinforcement learning algorithm chosen;

[0109] - the rule-based module 22_a determines the action At>2 or the set of actions {At>2] such that each of these actions is the action resulting from a rule whose condition is verified by the current state St;

[0110] In a step 1023, the selection module 45 selects an action, named At from the actions At,i and At 2 by applying a choice method making it possible to select “the best” action, for example by a vote between the two levels of knowledge (high level for the actions determined by the rule-based module 22_0 and low level for the actions determined by the reinforcement learning module), that is to say that for example:

[0111] - for each action A in the set of actions {At4} or {Ai2} a sum weighted wl.q + w2.k is calculated to represent the value (utility) of this action, q corresponding to the Q-value of the association “perform action A in the current state St” calculated, ke {0, 1} since the rules are binary (k = 0 when A is not an element of the set {At>i} and is equal to 1 otherwise) and wl and w2 are weights associated with each value (The weights are hyperparameters, they are determined before the learning phase even if it remains possible to make them evolve during the learning;

[0112] a probabilistic selection is then carried out using a softmax function on these weighted sums representing "a utility" of applying action A in state St;

[0113] In a step 1024, the selected action At is indicated by the hybrid learning electronic block 42 and the simulator 41 determines the next state St+i as a function of the action At and further determines the updated reward value Rt as a function of the state St+i (depending on how the reward function is defined) that the stimulator 41 delivers to the hybrid learning block 42.

[0114] In a step 1025, the learning is updated:

[0115] - in a sub-step 1025_l: the continuation of the learning by the reinforcement learning module 43 comprises the updating of the Q-value of At as a function of the reward Rt; - in a sub-step 1025_2: the updating of the rule base in the rule-based module 22_a is carried out in the manner detailed below and consisting of a rule extraction and refinement algorithm: it makes it possible to create declarative knowledge from procedural knowledge according to the following principle: if an action decided by the low level (i.e. the reinforcement learning module) is a success, i.e. if it satisfies a certain criterion, then the hybrid learning electronic block 42 constructs a new rule for the rule-based module 22_a from the current state leading to the chosen action. Subsequently, during the iterations, the rule in the rule-based module 22_a is refined by the hybrid learning electronic block 42: if it seems relevant, it will try to generalize it, otherwise it will try to specialize it.

[0116] Thus, in one embodiment, the updating of the rule base in a sub-step 1025_2 in the rule-based module 22_a comprises the following steps:

[0117] in a sub-step 1025_21: if no rule in the rule base has its condition satisfied by the state St, creation in the rule base, under certain conditions detailed below, of a new rule including:

[0118] • the condition is that the state must be equal to the state as defined by St, considering all dimensions (ie N); and • the action resulting from the rule (i.e. only when the condition is fulfilled) is At;

[0119] otherwise in a sub-step 1025_22: for any rule of the rule base whose condition is satisfied by the state St, it is evaluated whether said rule once specified (by removing a dimension from the condition of the rule) or once generalized (by adding a dimension to the condition of the rule) is more satisfactory than said rule.

[0120] To do this, the gain provided by said rule is compared to the respective gains of all the rules whose resulting action is the same and whose condition is the condition of said rule increased by one dimension (not yet part of the condition), and this for each additional dimension and for each value that the additional dimension considered can take; and depending on at least said comparison, the rule is replaced by one of said increased rules;

[0121] and

[0122]

[0123]

[0124]

[0125]

[0126]

[0127]

[0128]

[0129]

[0130]

[0131]

[0132]

[0133]

[0134]

[0135]

[0136]

[0137]

[0138] we compare the gain brought by said rule to the respective gains of all the rules whose resulting action is the same and whose condition is the condition of the rule reduced by one dimension. In one embodiment, this sub-step 1025_2 of updating the rule base in the rule-based module 22_a which will be detailed below in an example, is based on a set of metrics, which are for example defined as follows: - for a condition C, we calculate the current value of two indicators, defined as follows: - PMa(C): Number of times - since the start of learning - where the current value of the state of the environment having verified condition C, action “A” was selected and the result was positive; - NMa(C): Number of times - since the start of learning - where the current value of the state of the environment having verified condition C, action “A” was selected and the result was negative; with the positive / negative result criterion: “positive” when y.maxA(Q(st+i,A'))-Q(st,A) +rt+j> Tsuœès “negative' when y.maxA(Q(st+i,A'))-Q(st,A) +rt+j <réchec où : maxA' is the function returning the maximum value considering any action A' in the discrete space of actions as defined; rt+i is the reward; there is the discount factor; AuccèSetTfailureAre predefined threshold values. - we also define a Gain Index, called IG, which allows us to compare two rules with respective conditions C, C' with the result being the same action A: IG (C, C')>0 means that the rule [if C, then action A] is better than the rule [if C', then action A]: PMa(C)+l , PMa(C')+l tog2 pM« +NMn (C)+2~lOë2 PMa (C') + NMa (C’) +2 The operations in sub-step 1025_2 are then schematized as follows: - Update rule metrics - IF no rule matches the current state THEN • Creation of a new rule (dimension N) ELSE For each rule whose condition is satisfied by the current state, according to the modification criteria: Rule specification • Generalization of the rule

[0139] This gives more detail in one embodiment:

[0140] - in the rule creation sub-step 1025_21, if and only if y maxA (Q(st+i,A'))-Q(st,A) + rt+i> rsuccess then add the rule [if C then A ] is created in the rule base of the rule-based module 22_a (where C is the vector of dimension N equal to the current current environment state st); - in sub-step 1025_22 of specialization or generalization for any rule of the rule base whose condition C is satisfied by the state Stet resulting in the action At:

[0141] for each of these rules, we are interested in its condition C, as well as in all the conditions which are close to its condition: the neighborhood of a condition is defined as a literal of more or less with respect to said condition (for example the condition "aAb" has as neighborhood a, b, aAbAc, in the case where a, b and c are values ​​taken by three respective dimensions among the N dimensions of the state of the environment).

[0142] To know whether to generalize or specialize a rule C present in the rule-based module 22_a, the current rule C is compared with the new possible versions, in the following manner:

[0143] let C be the condition of a rule satisfied by the current state and ail is the condition which is always satisfied (i.e. the tautology):

[0144] Generalization:

[0145] If IG(C,all) > rgen and maxC'IG(C,,C) > 0 then, in the rule base of module 22_a:

[0146] - add the rule [if argmaxC'(IG(C,,C)), then At ] and

[0147] - remove the old rule [if C, then At ],

[0148] (here C' designates C enriched with an additional literal)

[0149] Specialization:

[0150] If IG(C,all) < Tspec and maxC' IG(C",C) > 0 then, in the rule base of module 22_a

[0151] - add the rule [if argmaxC'(IG(C,,C)) then At ] and

[0152] - remove the old rule [if C, then At ],

[0153] (here C' denotes C minus a literal)

[0154] rgen and Tspec are predefined values.

[0155] In a step 1026, the time step is incremented and the phase 102 of steps is repeated.

[0156] The operational phase 103 comprises the implementation in the drone of the generated rule-based module 22, and then, the autonomous navigation of the drone piloted by this rule-based module 22 once it has been implemented in the drone.

[0157] The rules created according to the invention are a function of the discretization of the states carried out during step 1013. This discretization must therefore be found in the operational phase 103. As described above, relative to [Fig.l], the environment of the drone 1 is read via all of the sensors 21, and this reading of the current state activates or not the conditions of the rules which then trigger the actions, if the condition of the rule is verified. These actions are then translated by the effectors of the drone to act in the environment.

[0158] The transition from the state of the environment to the tests of the conditions of the rules must therefore follow the discretization of the environment carried out during the learning phase. It is therefore necessary, in addition to the set of rules transferred into the memory of the drone in step 103, to add a program carrying out the discretization of the state.

[0159] The operational phase 103 comprises in embodiments a certification and interpretation step:

[0160] - Validation by a business expert of the rule set

[0161] - Verification of the consistency and completeness of the rule set

[0162] -Perform unit tests on the ruleset.

[0163] The prior art automatic pilot models based on machine learning do not recover here a system of rules as we propose in this invention, but a black box system, most of the time a neural network (a weight vector) which does not have the explainable and interpretable properties necessary for the certification of critical systems. Here, the set of rules allows experts to understand the behavior of the drone and therefore to verify it, approve it and certify it before transferring it into a drone for an application in an operational context.

[0164] In order to illustrate the creation of a set of rules for a drone 1 as described generally with reference to figures 2 and 3, the simple example of a drone 1 which must go from a point A to a point B while remaining confined within a corridor 50 is now considered, with reference to [Fig.4].

[0165] In order to be able to carry out the simulation necessary for learning, it is necessary to develop a simulator of the operational context.

[0166] Physical simulation engines are available on the market. Generally, software such as Unity 3D or Unreal Engine makes it easier to develop this type of simulator by implementing the basic physical reactions. They make it possible to develop this type of simulator environment without having to re-implement it, or in another embodiment then a complete implementation is implemented.

[0167] Then the state of the environment and the possible controllable actions are discretized.

[0168] As for the state of the environment, in one embodiment, it is re- presented by the following dimensions (cf. [Fig.4]):

[0169] • Di: The distance from drone 1 to point B • D2: The angle between the drone-point B axis and the directional axis of the drone • D3: The distance between the nose of drone 1 and the top of corridor 50 vertically • D4: The distance between the nose of drone 1 and the bottom of corridor 50 vertically • D5: The distance between the nose of drone 1 and corridor 50 along the drone axis • D6: The distance between the tip of the right wing of drone 1 and corridor 50 (located to the right of the drone) • D7: The distance between the tip of the left wing of drone 1 and corridor 50 (located to the left of the drone).

[0170] The discretization of the space of these dimensions is as follows:

[0171] • Only values ​​that Di can take: Du = [0,1], D[2 =] 1,10], D[3 = ]10,100], Di 4 = ] 100,+infinity] (in meters) • Only values ​​that D2 can take: D2ji = [-90,-45], D22 = -45,0], D23 = 0,45], D2j4 = 45,90 (in degrees) • Only values ​​that D3 can take: D3ji = [0,1], D3 2 =] 1,10], D3 3 = ]10,+infinity] (in meters) • Only values ​​that D4 can take: D4ji = [0,1], D42 =] 1,10], D43 = ]10,+infinity] (in meters) • Only values ​​that D5 can take: D5ji = [0,1], D5 2 =] 1,10], D5 3 = ]10,+infinity] (in meters) • Only values ​​that D6 can take: D61 = [0,1], D62 =] 1,10], D63 = ]10,+infinity] (in meters) • Only values ​​that D7 can take: D71 = [0,1], D7 2 =] 1,10], D7 3 = ]10,+infinity] (in meters)

[0172] As for actions, an action is a combination of an acceleration in the three dimensions represented in [Fig.5]. (A1, A2 and A3 determine the positioning in space and the acceleration determines how one will move according to this positioning.

[0173] • Ai: The yaw angle (in English, Yaw) • A2: The roll angle (in English, Roll) • A3: The pitch angle (in English, Pitch) • A4: Acceleration

[0174] The discretization of the dimensions of the action is as follows:

[0175] • Only values ​​that Ai can take: • Au = -1 (yaw action in the direction shown by the corresponding arrow in figure 5), • Aij2 = 0 (zero yaw action), Aij3 = 1 (yaw action in the direction shown by the corresponding arrow in figure 5), Only values ​​that A2 can take: • A24 = -1 (rolling action in the direction shown by the corresponding arrow) laying in figure 5), • A2>2 = 0 (zero rolling action), • A2j3 = 1 (rolling action in the direction shown by the corresponding arrow) laying in figure 5), Only values ​​that A3 can take: • A34 = -1 (pitching action in the direction shown by the corresponding arrow) laying in figure 5), • A3>2 = 0 (zero pitching action), • A3>3 = 1 (pitching action in the direction shown by the corresponding arrow) laying in figure 5), Only values ​​that A3 can take: • A4ji = -1 (deceleration), • A4j2 = 0 (maintain speed), • A4j3 = 1 (acceleration).

[0176] For learning to allow the drone to learn the correct behavior to adopt in its environment, it is necessary to define a reward function, which encourages this good behavior. The good behavior chosen in the context of the example considered here is that the drone moves from point A to point B as quickly as possible, remaining within the corridor. The reward function is therefore the following, for example:

[0177] Reward = -1000 if the drone leaves the corridor Reward = -1 for each decision (so that the learning agent prioritizes actions that make it reach point B most quickly and avoids the drone making unnecessary loops within the corridor) Reward = 1000 if the drone reaches point B

[0178] Finally, before launching the learning phase of reinforcement learning, for example with the CLARION architecture, the following values ​​are chosen for example:

[0179] The discount factor: 0.99 (commonly used order of magnitude) The learning rate: 10 4 (commonly used order of magnitude) The success threshold for a state-action association: 0.08 The criterion for stopping or converging the learning phase: 107 steps

[0180] Once the simulation is completed, as described in step 102, a set of rules will be recovered. The rules are of the form: IF (CONDITION) is verified, THEN de- completion of the ACTION to be implemented, for example IF Di = Di,j THEN An = An,m

[0181] Di,j being the discrete values ​​of the dimensions listed above and An,m being the discrete values ​​of the actions listed above.

[0182] For example, such a rule is: IF D5 = D5,l THEN A3,l

[0183] This rule translates that if the drone is heading down the corridor and is very close to it, it must change course and head up (see [Fig.4]). A set of rules of this type (and also including more complex rules, for example a rule with several concomitant conditions, or any other combination of defined dimensions) allows for automatic piloting based on rules understandable by an expert who will be more easily able to interpret and validate them. The fact is that the expert can give a semantic value to the discretizations, for example for Dl, an expert could say Du = "near", Di 3 = "far" (we assume that he will be able to define when it is close and when it is far in the discretization) by making this replacement with semantic variables that belong to the expertise vocabulary, we facilitate the understanding by the expert.

[0184] Note that for example the rule IF Ds,i THEN A3,i is a rule with only one dimension in condition. This would not have been possible with pure reinforcement learning. In a reinforcement learning policy, the state is represented by all its dimensions (even if they are not useful in decision making).

[0185] The invention offers a compromise, with systems based on artificial intelligence, between the precision of the model (or generalization), which allows for optimal behavior in the dedicated operational context, and the interpretability and explainability of the model which allows for the behavior to be interpreted, understood and certified. The model obtained at the output is a set of rules, therefore with interpretability and explainability properties making it easier to move towards certification. In addition, the reinforcement learning phase makes it possible to have more efficient output models and to address greater complexity (number of dimensions in the states and actions).The advantage of this invention is therefore to enable the production of an interpretable and efficient autonomous drone piloting model by taking advantage of the benefits of the hybridization of connectionist artificial intelligence and symbolic artificial intelligence.

[0186] The simulation of the operational context as proposed above allows:

[0187] - to avoid loss of material in the event that it would be risky to use it to develop the autonomous piloting model; - not needing a mass of data like deep learning, which can be long and expensive.

[0188] Simulating the environment allows the learning phase to have learned from a large number of situations. This allows for a more accurate and robust set of rules than a set of rules determined by experts in the operational context, but in other embodiments, no simulation is used and the learning is performed with operational data.

[0189] To date, the most recent version of CLARION (http: / / www.clarioncognitivearchitecture.com / downloads) is provided in C# language. The Unity 3D physics simulation engine (https: / / unity.com / fr) also uses C# for its development and has environments, tools and objects that can facilitate the development of the simulation. The interfacing of Unity and CLARION is reduced to the simple integration of the CLARION library within the Unity project.

[0190] Otherwise, it is entirely possible to interface CLARION with “non-C#” simulators, for example by using a message broker type RabbitMQ (https: / / www.rabbitmq.com / ).

[0191] The solution according to the invention is based on standard logic (i.e. a variable is either true or false) and not on fuzzy logic (i.e. the variables have a degree of probability which can take values, including values ​​other than 0 and 1, in the interval [0;1]. It is scalable since the rules are constructed and adapted to the context during the learning phase and finally it is based on reinforcement learning.

[0192] Autonomous drones are particularly useful in many types of missions: surveillance, information gathering, inspection, rescue, etc.

[0193] Among these, the invention makes it possible to provide an autonomous navigation system for: (in each case, a simulation of the mission will be necessary for the learning phase):

[0194] • Power line monitoring mission • Mission to assist in the inspection of the railway network • Track inspection • Inspection of catenaries • Presence of people on the railway tracks • Parcel delivery mission • Military reconnaissance / scouting (espionage) mission • Information gathering mission (meteorological type) • Target elimination mission (military) • Flying Weapon Mission • Drone joumalism mission (https: / / en.wikipedia.org / wiki / Dronejoumalism) • Mission specific to agriculture • Recognition of soil heterogeneity • Varying seeding density to create a plot • Establish crop needs by analyzing plant density • Rapid recognition of damage and its causes on a plot • Mission to distribute access to WIFI • 3D soil and building modeling mission • Life Saving Mission • Transport of life jacket and defibrillator • Any other possible assistance • Law enforcement support mission • Reconnaissance and surveillance of hard-to-access areas • Observation • Monitoring of road networks • Space mission

[0195] The invention makes it possible to create a model of rules, a technology belonging to so-called “Symbolic” artificial intelligence, by relying on automatic learning, a technology belonging to so-called “connectionist” artificial intelligence in order to allow a drone to carry out a specific mission in a completely autonomous manner.

[0196] Symbolic LTA and connectionist LTA have advantages that can be combined and thus compensate for the weakness of the other. Symbolic LTA has the advantage of working with symbols, which allows it to be better understood by humans. Expert systems, which are a variation of this, make it possible to capture knowledge already acquired and use it to automate reasoning. This results in models that are more easily interpretable and explainable and therefore more easily certifiable and validated. Unfortunately, they have the disadvantage of not being flexible and not being able to manage unstructured data. In addition, the capabilities of the system are often limited to those of the expert. Connectionist LTA has the ability to create complex models based on correlations calculated on large masses of data.Given a fairly substantial amount of computing power, it is possible to create very powerful prediction and classification models, most of the time much better than expert systems. The main problem with this technology is that the models created are what are commonly called "black boxes", that is to say that we cannot . cannot explain the decision taken by this type of model.

[0197] The major advantage of the proposed symbolic AI-connectionist AI hybridization makes it possible to rely on automatic learning, here reinforcement learning via interaction with a simulator, and to extract rules of drone behavior which have better interoperability and explainability, making it possible to move towards certification and validation of these autonomous drone piloting models.

[0198] Finally, an essential characteristic to highlight is the autonomy property of the drone. The invention allows the drone to adapt to all new situations (within the framework of the simulated mission), that is to say that it is not necessary to restart a calculation phase for each new mission. The model of rules learned during the learning phase knows how to fulfill a mission, whatever the parameters of the mission (shape of the corridor, presence of obstacles, etc.).

Claims

1. Claims Method for developing a navigation module (22) for an autonomous drone (1) based on navigation rules, said method being characterized in that it is implemented by an electronic system (42) comprising: - a reinforcement learning module (43) adapted to determine at least one drone navigation action based on a current state of the drone's environment; and - a rule-based module (22_a), comprising a navigation rule base where a rule is defined by a condition and by an action which is the result of said rule if and only if said condition is respected, the state of the environment of the drone being a vector of dimension N, said state of the environment and the navigation actions of the drone having each been discretized into a finite number of values, and in that the following steps are implemented iteratively: i. receiving a first value of the current state of the drone's environment; ii. determination, by the automatic reinforcement learning module (43), of one or more first navigation actions (At4) as a function of said first current state value received and determination by the rule-based module of one or more second navigation actions (At>2) as a function of said first current state value received; iii. selection, by said system (42), of an action (At) from among said first and second actions; iv. updating the reinforcement learning of the reinforcement learning module based on the second current state value of the environment obtained following the implementation of the selected navigation action; and in parallel v. updating the rules of the rule-based module (22_a), said updating comprising at least one of the following steps: vi if there is no rule in the rule base whose condition is satisfied by said first current state value, then creation of a new rule in the rule base, whose condition is that the current state value is equal to said first current state value, and whose conclusion is said selected action; v.ü. for a given rule of the rule base whose first current state value verifies the condition and whose result is said selected action, said given rule is replaced, in the rule base, by a modified rule whose result is said selected action and whose condition includes said condition of the given rule and an additional dimension among the N dimensions not taken into account in the condition and / or by a modified rule whose condition includes said condition of the given rule of which a dimension has been deleted, depending on the comparison of the relevance of the rule with the relevance of the modified rule, the method further comprising a step of certification and interpretation of the rules of the rule-based module and a step of implementation, in the drone, of the rule-based module, the drone then being able to navigate autonomously while being piloted by the rule-based module.

2. Method for developing a navigation module (22) of an autonomous drone based on navigation rules according to claim 1, according to which - the determination by the reinforcement learning module (43) of said first navigation action(s) as a function of said first current state value received is a function of the current values ​​of evaluation parameters associated with the respective pairs (first current state value, action) and which evaluate the relevance of carrying out said action in said state; - the reinforcement learning update of the reinforcement learning module comprises an update of the value of the evaluation parameter associated with the pair (first current state value, selected action) as a function of the second current state value of the environment obtained; and - the relevance of the rule, respectively of the modified rule, is evaluated according to the evaluation parameters associated with the respective pairs (state, action) such that the state verifies the condition of the rule, respectively the condition of the modified rule, and the action is the action of said rule which is also that of the modified rule.

3. Method for developing a navigation module (22) for an autonomous drone based on navigation rules according to claim 2, according to which, in step vi, said new rule is only created if it verifies a utility criterion determined as a function of at least said value ac evaluation parameter associated with the pair (first current state value, selected action).

4. Method for developing a navigation module (22) of an autonomous drone based on navigation rules according to one of claims 2 or 3, according to which said step of selecting the action comprises the following steps: for each of said actions among said first and second actions, calculating a score associated with said action and equal to a weighted sum of the current value of the evaluation parameter associated with the pair (first current state value, said action) and k, where k takes the value 1 if said action is part of the first actions and takes the value 0 if said action is not part of the first actions, the respective weights being predefined values.

5. Navigation module (22) of an autonomous drone based on navigation rules, said navigation module being characterized in that it is generated by an electronic system (40) for developing a navigation rules module of an autonomous drone comprising: - a reinforcement learning module (43) adapted to determine at least one drone navigation action based on a current state of the drone's environment; and - a rule-based module (22_a), comprising a navigation rule base where a rule is defined by a condition and by an action which is the result of said rule if and only if said condition is respected, the state of the environment of the drone being a vector of dimension N, said state of the environment and the navigation actions of the drone having each been discretized into a finite number of values, such that iteratively: i. said system receives a first value of the current state of the drone's environment; ii. the reinforcement learning module (43) determines one or more first navigation actions (At4) based on said first received current state value and the rule-based module determines one or more second navigation actions (At,2) based on said first received current state value; iii. said system (42) selects an action (At) from among said first and second actions; iv. the reinforcement learning module updates its ap

6. reinforcement learning based on the second current state value of the environment obtained following the implementation of the selected navigation action; and in parallel v. said system updates the rules of the module (22_a) based on rules, in the following manner vi if there is no rule in the rule base whose condition is satisfied by said first current state value, then creation of a new rule in the rule base, whose condition is that the current state value is equal to said first current state value, and whose conclusion is said selected action; and / or v.ii. for a given rule of the rule base whose first current state value verifies the condition and whose result is said selected action, replacement, in the rule base, of said given rule by a modified rule whose result is said selected action and whose condition includes said condition of the given rule and an additional dimension among the N dimensions not taken into account in the condition and / or by a modified rule whose condition includes said condition of the given rule from which a dimension has been deleted, depending on the comparison of the relevance of the rule with the relevance of the modified rule the rules of the rule-based module being suitable for being certified and interpreted by the system, the rule-based module being suitable for being implemented in the drone, the drone then being able to navigate autonomously while being piloted by the rule-based module. Electronic system (40) for developing a navigation rules module for an autonomous drone comprising: - a reinforcement learning module (43) adapted to determine at least one drone navigation action based on a current state of the drone's environment; and - a rule-based module (22_a), comprising a navigation rule base where a rule is defined by a condition and by an action which is the result of said rule if and only if said condition is respected, the state of the environment of the drone being a vector of dimension N, said state of the environment and the navigation actions of the drone having each been discretized into a finite number of values, i. said system is adapted to iteratively receive a first value of the current state of the drone's environment;

7. ii. the automatic reinforcement learning module (43) is adapted to iteratively determine one or more first navigation actions (At4) as a function of said first received current state value and the rule-based module is adapted to iteratively determine one or more second navigation actions (At>2) as a function of said first received current state value; iii. said system (42) is adapted to iteratively select an action (At) from among said first and second actions; iv. the reinforcement learning module is adapted to iteratively update its reinforcement learning based on the second current state value of the environment obtained following the implementation of the selected navigation action; and in parallel v. said system is adapted to iteratively update the rules of the rule-based module (22_a), in the following manner vi if there is no rule in the rule base whose condition is satisfied by said first current state value, then creation of a new rule in the rule base, whose condition is that the current state value is equal to said first current state value, and whose conclusion is said selected action; and / or v.ii. for a given rule of the rule base whose first current state value verifies the condition and whose result is said selected action, replacement, in the rule base, of said given rule by a modified rule whose result is said selected action and whose condition includes said condition of the given rule and an additional dimension among the N dimensions not taken into account in the condition and / or by a modified rule whose condition includes said condition of the given rule of which a dimension has been deleted, depending on the comparison of the relevance of the rule with the relevance of the modified rule, the system being capable of certifying and interpreting the rules of the rule-based module and of implementing, in the drone, the rule-based module, the drone then being able to navigate autonomously while being piloted by the rule-based module. Electronic system (40) for developing a navigation rules module for an autonomous drone according to claim 6, wherein - the determination by the reinforcement learning module (43) of said first navigation action(s) as a function of said first current state value received is a function of the values current evaluation parameters associated with the respective pairs (first current state value, action) and which evaluate the relevance of carrying out said action in said state; - the reinforcement learning update of the reinforcement learning module comprises an update of the value of the evaluation parameter associated with the pair (first current state value, selected action) as a function of the second current state value of the environment obtained; and - the relevance of the rule, respectively of the modified rule, is evaluated according to the evaluation parameters associated with the respective pairs (state, action) such that the state verifies the condition of the rule, respectively the condition of the modified rule, and the action is the action of said rule which is also that of the modified rule.

8. Electronic system (40) for developing a navigation rules module for an autonomous drone according to claim 7, wherein, in the updating by said system of the rules of the rule-based module (22_a), if there is no rule in the rule base for which said first current state value satisfies the condition, then there is creation of a new rule in the rule base, the condition of which is that the value of the current state is equal to said first current state value, and the conclusion of which is said selected action, said new rule is only created by said system if it satisfies a utility criterion determined as a function of at least said updated value of the evaluation parameter associated with the pair (first current state value, selected action).

9. Electronic system (40) for developing a navigation rules module for an autonomous drone according to one of claims 7 or 8, adapted, when selecting the action, to calculate for each of said actions among said first and second actions, a score associated with said action and equal to a weighted sum of the current value of the evaluation parameter associated with the pair (first current state value, said action) and k, where k takes the value 1 if said action is part of the first actions and takes the value 0 if said action is not part of the first actions, the respective weights being predefined values.

Images