Task planning method and related device based on reinforcement learning under temporal logic constraints

Abstract

The present application discloses a task planning method based on reinforcement learning under temporal logic constraints and related devices, the method includes converting the task to be planned into a deterministic finite automaton; determining the state-action trajectory based on the deterministic finite automaton and the initial strategy; inputting the state-action trajectory and the external rewards of each state-action pair into a feedforward neural network, and outputting the internal rewards of each state-action pair through the feedforward neural network; determining the first objective function and the first return value of the initial strategy based on each external reward and each internal reward, and updating the strategy parameters of the initial strategy based on the first objective function and the first return value to obtain the target strategy corresponding to the task to be planned. The present application captures the temporal characteristics of the task through the attention mechanism, so that the execution end can quickly learn tasks with temporal relationships in a sparse reward environment, so that the sparse reward problem under LTL constraints can be solved in different environments, so that reinforcement learning can learn the optimal strategy.

Description

Technical Field

[0001] The present application relates to the technical field of reinforcement learning, and in particular to a task planning method and related devices based on reinforcement learning under temporal logic constraints. Background Art

[0002] Nowadays, Linear Temporal Logic (LTL) has received widespread attention and has shown excellent performance in a wide range of applications. The main advantage of LTL lies in its expressive power, which allows the definition of various advanced behaviors beyond the scope of traditional motion planning. For example, many strategy generation methods under LTL constraints have been proposed in the literature (Baier C, Katoen J P. Principles of model checking [M]. MIT press, 2008.).

[0003] Many scholars have focused on the application of reinforcement learning in temporal logic planning because it can handle more complex tasks and more uncertain environments. However, when combining linear temporal logic (LTL) with reinforcement learning, it is necessary to consider the historical dependency and temporality of linear temporal logic (LTL) to construct appropriate instant rewards. Otherwise, due to the delayed rewards and sparsity of rewards in LTL tasks, reinforcement learning methods will not be able to converge quickly or even learn the optimal strategy.

[0004] In order to solve the above problems, existing reinforcement learning algorithms generally complete the management and transfer of system states by converting LTL tasks into deterministic Rabin automata and constructing product automata with the environment, and in order to obtain immediate rewards for the agent, the acceptable maximum end component representing the accepted state is calculated, and all immediate rewards that can be converted to the acceptable maximum end component are 1, otherwise they are 0. However, reinforcement learning algorithms using this method still have the problem of generating optimal strategies.

[0005] Therefore the prior art still needs to be improved and enhanced. Summary of the invention

[0006] The technical problem to be solved by the present application is to provide a task planning method and related devices based on reinforcement learning under temporal logic constraints in view of the deficiencies of the existing technology.

[0007] In order to solve the above technical problems, the first aspect of the embodiment of the present application provides a task planning method based on reinforcement learning under temporal logic constraints, the method comprising:

[0008] Convert the task to be planned into a deterministic finite automaton;

[0009] Determining a state-action trajectory corresponding to the task to be planned based on the deterministic finite automaton and the initial strategy corresponding to the task to be planned, wherein each state-action pair in the dynamic action trajectory corresponds to an external reward;

[0010] Inputting the state-action trajectory and the external rewards corresponding to each state-action pair into a preset feedforward neural network, and outputting the internal rewards corresponding to each state-action pair through the feedforward neural network, wherein the feedforward neural network is configured with a self-attention mechanism;

[0011] Determine a first objective function and a first return value corresponding to the initial strategy based on each external reward and each internal reward, and update a strategy parameter of the initial strategy based on the first objective function and the first return value;

[0012] Continue to execute the step of determining the state action trajectory corresponding to the task to be planned based on the deterministic finite automaton and the initial strategy corresponding to the task to be planned, until a target strategy corresponding to the task to be planned is obtained.

[0013] The task planning method based on reinforcement learning under the constraints of temporal logic, wherein the determining of the state action trajectory corresponding to the task to be planned based on the deterministic finite automaton and the initial strategy corresponding to the task to be planned specifically includes:

[0014] Get the current status of the environment where the execution end of the task to be planned is located;

[0015] Perform action sampling based on the current state and the initial strategy corresponding to the task to be planned to obtain an action;

[0016] Controlling the execution end to execute the action to obtain a next state, and checking a transition state of the next state in a deterministic finite automaton;

[0017] If the transition state satisfies the first preset condition, a corresponding external reward is configured for the state-action pair and the state-action pair is terminated to obtain a state-action trajectory, wherein the state-action pair includes a state and an action;

[0018] If the transition state satisfies a second preset condition, configuring a preset external reward for the state-action pair;

[0019] The next state is taken as the current state, and the step of sampling actions based on the current state and the initial strategy corresponding to the task to be planned is continued to obtain actions until the transition state violates the timing logic or belongs to an acceptable state set or the trajectory length of the state action trajectory reaches a preset length threshold.

[0020] The task planning method based on reinforcement learning under the constraints of temporal logic, wherein the first preset condition is that the transition state violates the temporal logic or belongs to an acceptable state set; the second preset condition is that the transition state does not violate the temporal logic and does not belong to the acceptable state set, or the transition state of the next state in the deterministic finite automaton is not checked.

[0021] The task planning method based on reinforcement learning under the constraints of temporal logic, wherein if the transition state does not violate the temporal logic and does not belong to the acceptable state set, after configuring a preset external reward for the state-action pair consisting of the current state and the action, the method further includes:

[0022] The state of the deterministic finite automaton is reset to an initial state of the deterministic finite automaton.

[0023] The task planning method based on reinforcement learning under the constraints of temporal logic, wherein the feedforward neural network includes a self-attention module and a fully connected module; the state-action trajectory and the external rewards corresponding to each state-action pair are input into a preset feedforward neural network, and the internal rewards corresponding to each state-action pair are output through the feedforward neural network specifically include:

[0024] Inputting the state-action trajectory and the external reward input corresponding to each state-action pair into the self-attention module, and outputting the time series characteristic vector corresponding to each state-action pair through the self-attention module;

[0025] The time series characteristic vectors corresponding to each state-action pair are input into the fully connected module, and the internal rewards corresponding to each action state are input through the fully connected module.

[0026] The task planning method based on reinforcement learning under the temporal logic constraint, wherein the time slot characteristic vector corresponding to the action state is:

[0027] y=ωv+x

[0028]

[0029] Among them, v represents the value vector, x represents the time series feature vector, q represents the query vector, k represents the key vector, softmax represents the softmax function, and dimv represents the spatial dimension of the value vector v.

[0030] The task planning method based on reinforcement learning under the constraints of temporal logic, wherein the calculation formulas of the first objective function and the return value are respectively:

[0031]

[0032]

[0033] Among them, J ex+in represents the first objective function, γ represents the loss factor, λ represents the hyperparameter, represents the intrinsic reward at time i, represents the external reward at time t, η represents the network parameters of the feedforward neural network, T represents the number of action states, s t represents the state at time t, a t represents the action at time t, s i represents the state at time i, a i Represents the action at time i.

[0034] The task planning method based on reinforcement learning under the constraints of temporal logic, wherein the step of continuing to perform the step of determining the state action trajectory corresponding to the task to be planned based on the deterministic finite automaton and the initial strategy corresponding to the task to be planned, until the target strategy step corresponding to the task to be planned is obtained, the method further includes:

[0035] A second objective function and a second return value corresponding to the initial strategy are determined based on each external reward, and the network parameters of the feedforward neural network are updated using the second objective function and the second return value.

[0036] A second aspect of an embodiment of the present application provides a task planning device based on reinforcement learning under temporal logic constraints, the task planning device comprising:

[0037] A conversion module, used for converting the task to be planned into a deterministic finite automaton;

[0038] A determination module, configured to determine a state-action trajectory corresponding to the task to be planned based on the deterministic finite automaton and an initial strategy corresponding to the task to be planned, wherein each state-action pair in the dynamic action trajectory corresponds to an external reward;

[0039] A feedforward network module, used for inputting the state-action trajectory and the external rewards corresponding to each state-action pair into a preset feedforward neural network, and outputting the internal rewards corresponding to each state-action pair through the feedforward neural network, wherein the feedforward neural network is configured with a self-attention mechanism;

[0040] An updating module, configured to determine a first objective function and a first return value corresponding to the initial strategy based on each external reward and each internal reward, and to update a strategy parameter of the initial strategy based on the first objective function and the first return value;

[0041] The execution module is used to continue to execute the step of determining the state action trajectory corresponding to the task to be planned based on the deterministic finite automaton and the initial strategy corresponding to the task to be planned, until the target strategy corresponding to the task to be planned is obtained.

[0042] A third aspect of an embodiment of the present application provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the task planning method based on reinforcement learning under the constraints of temporal logic as described above.

[0043] A fourth aspect of the embodiments of the present application provides a terminal device, comprising: a processor, a memory, and a communication bus; the memory stores a computer-readable program that can be executed by the processor;

[0044] The communication bus realizes the connection and communication between the processor and the memory;

[0045] When the processor executes the computer-readable program, the steps in the task planning method based on reinforcement learning under the temporal logic constraints as described above are implemented.

[0046] Beneficial effect: Compared with the prior art, the present application provides a task planning method and related device based on reinforcement learning under temporal logic constraints, the method comprising converting the task to be planned into a deterministic finite automaton; determining the state-action trajectory corresponding to the task to be planned based on the deterministic finite automaton and the initial strategy corresponding to the task to be planned; inputting the state-action trajectory and the external rewards corresponding to each state-action pair into a preset feedforward neural network, and outputting the internal rewards corresponding to each state-action pair through the feedforward neural network; determining the first objective function and the first return value corresponding to the initial strategy based on each external reward and each internal reward, and updating the strategy parameters of the initial strategy based on the first objective function and the first return value; continuing to perform the step of determining the state-action trajectory corresponding to the task to be planned based on the deterministic finite automaton and the initial strategy corresponding to the task to be planned, until the target strategy corresponding to the task to be planned is obtained. The present application captures the temporal characteristics of the task through the attention mechanism, so that the execution end can quickly learn tasks with temporal relationships in a sparse reward environment, so that the sparse reward problem under LTL constraints can be solved in different environments, so that reinforcement learning can learn the optimal strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without inventive work.

[0048] Figure 1 A flowchart of a task planning method based on reinforcement learning under temporal logic constraints provided in this application.

[0049] Figure 2 This is a flowchart of the principle of the task planning method based on reinforcement learning under the constraints of temporal logic provided in this application.

[0050] Figure 3 This is a structural schematic diagram of the task planning device based on reinforcement learning under the constraints of temporal logic provided in this application.

[0051] Figure 4 This is a schematic diagram of the structure of the terminal device provided in this application. DETAILED DESCRIPTION

[0052] The present application provides a task planning method based on reinforcement learning under temporal logic constraints and related devices. In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

[0053] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0054] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0055] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not mean the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0056] The inventors have found through research that Linear Temporal Logic (LTL) has received widespread attention and has shown excellent performance in a wide range of applications. The main advantage of LTL is its expressive power, which allows the definition of various advanced behaviors beyond the scope of traditional motion planning. For example, many strategy generation methods under LTL constraints have been proposed in the literature (Baier C, Katoen J P. Principles of model checking [M]. MIT press, 2008.).

[0057] Many scholars have focused on the application of reinforcement learning in temporal logic planning because it can handle more complex tasks and more uncertain environments. However, when combining linear temporal logic (LTL) with reinforcement learning, it is necessary to consider the historical dependency and temporality of linear temporal logic (LTL) to construct appropriate instant rewards. Otherwise, due to the delayed rewards and sparsity of rewards in LTL tasks, reinforcement learning methods will not be able to converge quickly or even learn the optimal strategy.

[0058] In order to solve the above problems, existing reinforcement learning algorithms generally complete the management and transfer of system states by converting LTL tasks into deterministic Rabin automata and constructing product automata with the environment, and in order to obtain immediate rewards for the agent, the acceptable maximum end component representing the accepted state is calculated, and all immediate rewards that can be converted to the acceptable maximum end component are 1, otherwise they are 0. However, reinforcement learning algorithms using this method still have the problem of generating optimal strategies.

[0059] In order to make the algorithm converge to the optimal strategy, scholars have considered long-term performance optimization based on Markov decision processes. In order to reveal the impact of uncertainty on this convergence process, some scholars have proposed to estimate the Markov environment first. Since the calculation of the acceptable maximum end component consumes a lot of computing resources, in order to avoid this calculation and reduce the amount of calculation, a more popular method is to collect rewards based on the Rabin acceptance condition of the product automaton. Another equally effective method is to convert LTL into a restricted deterministic automaton to generate a product automaton that is smaller than the original system. It is worth noting that all of the above work requires a product automaton, which requires mastering all environmental information, which is not always feasible in practical applications. Therefore, the most common method is a model-free method that approximates state action values ​​through neural networks and derives real-time rewards from deterministic Rabin automata.

[0060] In the case of complex scenarios in the real world, although rewards can be specified based on LTL tasks, standard reinforcement learning algorithms are difficult to learn and converge quickly due to the sparsity of rewards. In addition, in a sparse reward environment, given that the transmission of the impact of an action on the result may be delayed, the robot cannot clearly understand the temporal relationship between actions and states. This problem can be modeled as a temporal credit assignment problem (Hung CC, Lillicrap T, Abramson J, et al. Optimizing agent behavior over long time scales by transporting value [J]. Nature communications, 2019, 10 (1): 1-12.). To solve this problem, the literature (Ng AY, Harada D, Russell S. Policy invariance under reward transformations: Theory and application to reward shaping [C]. Icml. 1999, 99: 278-287.) proposed a reward formation method that incorporates domain knowledge into reinforcement learning, guides the reinforcement learning algorithm to find more promising solutions faster, and handles the credit assignment problem in time. In addition, the intrinsic reward mechanism proposed in the literature (Singh S, Lewis RL, Barto A G. Where do rewards come from [C]. Proceedings of the annual conference of the cognitive science society. Cognitive Science Society, 2009: 2601-2606.) can also solve the sparse reward problem in time-based credit allocation. This method uses prediction errors as curiosity rewards to encourage diversified exploration. In the absence of external rewards, an intrinsic reward defined by distance is designed to expand similar trajectories. However, the designed intrinsic reward cannot fully describe the temporal properties of the entire task, which results in the algorithm being unable to quickly find the optimal strategy in a sparse reward environment with temporal dependence such as LTL.

[0061] Based on this, in an embodiment of the present application, the task to be planned is converted into a deterministic finite automaton; the state-action trajectory corresponding to the task to be planned is determined based on the deterministic finite automaton and the initial strategy corresponding to the task to be planned; the state-action trajectory and the external rewards corresponding to each state-action pair are input into a preset feedforward neural network, and the internal rewards corresponding to each state-action pair are output through the feedforward neural network; based on each external reward and each internal reward, the first objective function and the first return value corresponding to the initial strategy are determined, and the strategy parameters of the initial strategy are updated based on the first objective function and the first return value; the step of determining the state-action trajectory corresponding to the task to be planned based on the deterministic finite automaton and the initial strategy corresponding to the task to be planned is continued until the target strategy corresponding to the task to be planned is obtained. The present application captures the timing characteristics of the task through the attention mechanism, so that the execution end can quickly learn tasks with timing relationships in a sparse reward environment, so that the sparse reward problem under the LTL constraint can be solved in different environments, so that reinforcement learning can learn the optimal strategy.

[0062] The application content is further explained below through the description of embodiments in conjunction with the accompanying drawings.

[0063] This embodiment provides a task planning method based on reinforcement learning under temporal logic constraints, such as Figure 1 As shown, the method includes:

[0064] S10, converting the task to be planned into a deterministic finite automaton.

[0065] Specifically, the planning task is a pre-given task, and the task to be planned can be declared in a syntax of time linear logic LTL in a form that can be understood by a terminal device, wherein the terminal device can be a mobile phone, a computer, a cloud server, etc. For example, the task to be planned can be that the robot must first go to the key storage area to get the key before walking to the box storage area. In addition, when converting the task to be planned into a deterministic finite automaton, a standard conversion method for converting time linear logic LTL into a deterministic finite automaton can be used, for example, the method described in the document Baier C, Katoen JP. Principles of model checking [M]. MIT press, 2008, which will not be repeated here.

[0066] S20: Determine a state action trajectory corresponding to the task to be planned based on the deterministic finite automaton and the initial strategy corresponding to the task to be planned.

[0067] Specifically, the state-action trajectory includes several state-action pairs, each of which includes a state and an action corresponding to the state, and each of the several state-action pairs corresponds to an external reward, which is an immediate reward obtained by executing the action at the execution end. The initial strategy is pre-set, for example, the initial strategy can be a uniformly distributed random strategy, that is, each of the several actions in the initial strategy has the same probability, so that one of the several actions is sampled as an action.

[0068] In an implementation of this embodiment, determining the state action trajectory corresponding to the task to be planned based on the deterministic finite automaton and the initial strategy corresponding to the task to be planned specifically includes:

[0069] Get the current status of the environment where the execution end of the task to be planned is located;

[0070] Perform action sampling based on the current state and the initial strategy corresponding to the task to be planned to obtain an action;

[0071] Controlling the execution end to execute the action to obtain a next state, and checking a transition state of the next state in a deterministic finite automaton;

[0072] If the transition state satisfies the first preset condition, a corresponding external reward is configured for the state-action pair and the state-action pair is terminated to obtain a state-action trajectory, wherein the state-action pair includes a state and an action;

[0073] If the transition state satisfies a second preset condition, configuring a preset external reward for the state-action pair;

[0074] The next state is taken as the current state, and the step of sampling actions based on the current state and the initial strategy corresponding to the task to be planned is continued to obtain actions until the transition state violates the timing logic or belongs to an acceptable state set or the trajectory length of the state action trajectory reaches a preset length threshold.

[0075] Specifically, the current state s is the state of the environment in which the execution end is located. For example, if the execution end is a robot, then the current state is the state of the environment in which the robot is currently located. After obtaining the current state s, action a can be sampled based on the current state s and the initial strategy π to obtain the action a corresponding to the current state s, where action a is the action to be performed by the execution end based on the initial strategy π under the current state s. After obtaining the action, the execution end executes action a to obtain the next state s', where the next state s' is the next state after the current state in time order, that is, after the execution end executes the action in the current state, the current state changes to the next state.

[0076] In the process of executing an action to obtain the next state s', the label function is used to find the transition state of the next state s' in the deterministic finite automaton, that is, to find the corresponding transition of the process (s, a, s') in the deterministic finite automaton B, and then the state-action pair (s, a) is given external rewards based on the preset conditions satisfied by the transition state, wherein the preset conditions include a first preset condition and a second preset condition, wherein the first preset condition is that the transition state violates the temporal logic or belongs to the acceptable state set; the second preset condition is that the transition state does not violate the temporal logic and does not belong to the acceptable state set, or the transition state of the next state in the deterministic finite automaton is not checked. In other words, when the transition state is that the transition state violates the temporal logic or belongs to the acceptable state set, the corresponding external reward is configured for the state-action pair and the state-action pair is terminated to obtain the state-action trajectory; when the transition state does not violate the temporal logic and does not belong to the acceptable state set, or the transition state of the next state in the deterministic finite automaton is not checked, the preset external reward is configured for the state-action pair.

[0077] In one implementation of this embodiment, the violation of temporal logic and belonging to the acceptable state set each have corresponding external rewards. When the transition state is a violation of temporal logic, the external reward corresponding to the violation of temporal logic is configured, and when the transition state is a belonging to the acceptable state set, the external reward corresponding to the belonging to the acceptable state set is configured. In a specific implementation, the external reward corresponding to the violation of temporal logic is -1, and the external reward corresponding to the belonging to the acceptable state set is 1. That is, when the transition state is a violation of temporal logic, it means that (s, a, s') violates the LTL task constraint, and at this time, a negative reward value external reward r will be assigned to the state-action pair. ex = -1, and terminate the state-action trajectory D; when the state is changed to belong to the acceptable state set, it means that after executing (s, a, s'), the current state in the deterministic finite automaton belongs to the acceptable state set, indicating that the task has been completed. At this time, the state-action pair is assigned an external reward r with a positive reward value. ex =1, and terminate the state action trajectory. In addition, the preset external reward is pre-configured to promote the learning process, for example, the preset external reward is -0.2, etc. Of course, in actual application, it can be adjusted according to the actual reinforcement learning training process.

[0078] Furthermore, when the transition state of the next state in the deterministic finite automaton is not checked, it means that the next state s' is not in the deterministic finite automaton B, that is, (s, a, s') does not trigger the transition in the deterministic finite automaton B. At this time, the preset external reward is configured for the state-action pair. In addition, in order to avoid the execution end from running all the time and causing an infinite loop, in the process of detecting the transition state, the trajectory length of the state-action trajectory can be monitored. If the transition state does not reach the point of violating the temporal logic or belongs to the acceptable state set, and the trajectory length of the state-action trajectory reaches the preset length threshold T, the state-action trajectory is also terminated and the external reward of the last state-action pair is set to the external reward when violating the temporal logic. For example, if the external reward corresponding to the violation of the temporal logic is -1, then the external reward of the last state-action pair is set to -1.

[0079] In an implementation of this embodiment, if the transition state does not violate the sequential logic and does not belong to the acceptable state set, after configuring a preset external reward for the state-action pair consisting of the current state and the action, the method further includes: resetting the state of the deterministic finite automaton to the initial state of the deterministic finite automaton. In this way, when the state-action trajectory is continuously acquired after the policy parameters of the initial policy are subsequently updated, the state in the deterministic finite automaton can be guaranteed to be the initial state, so as to restart the task.

[0080] S30, inputting the state-action trajectory and the external rewards corresponding to each state-action pair into a preset feedforward neural network, and outputting the internal rewards corresponding to each state-action pair through the feedforward neural network.

[0081] Specifically, the feedforward neural network is configured with a self-attention mechanism for obtaining the internal rewards corresponding to each state-action pair in the state-action trajectory, that is, the input items of the feedforward neural network are the state-action trajectory and the external rewards corresponding to each state-action pair, and the output items are the internal rewards corresponding to each state-action pair. The internal rewards corresponding to each state-action pair are used as dense rewards relative to the external rewards to carry out the learning process, and the internal rewards corresponding to each state-action pair are used only to promote the update of the policy parameters of the initial policy to ensure the maximization of the accumulated external rewards.

[0082] In an implementation of this embodiment, the feedforward neural network includes a self-attention module and a fully connected module; the inputting of the state-action trajectory and the external rewards corresponding to each state-action pair into a preset feedforward neural network, and outputting the internal rewards corresponding to each state-action pair through the feedforward neural network specifically includes:

[0083] Inputting the state-action trajectory and the external reward input corresponding to each state-action pair into the self-attention module, and outputting the time series characteristic vector corresponding to each state-action pair through the self-attention module;

[0084] The time series characteristic vectors corresponding to each state-action pair are input into the fully connected module, and the internal rewards corresponding to each action state are input through the fully connected module.

[0085] Specifically, the timing characteristic vector is determined by the self-attention module based on the state-action trajectory and the external rewards corresponding to each state-action pair. That is, the state-action trajectory and the external rewards corresponding to each state-action pair are used to encode the timing characteristics of the linear temporal logic LTL task, and the timing characteristics encoded by the state-action trajectory and the external rewards corresponding to each state-action pair are captured by the self-attention module, wherein the self-attention module is configured with a self-attention mechanism. In a specific implementation, for each state-action pair (s i ,a i ), the concatenated time feature vector x i It needs to undergo three separate linear transformations. The self-attention module regards the encoded representation of the state-action pair and its corresponding external reward as a set of query key-value pairs (q, k, v). The dimension of each vector in the query key-value pair (q, k, v) is n, where n is the trajectory length of the state-action trajectory D.

[0086] In addition, after the query key-value pair (q, k, v) is randomly initialized, during the training process of the feedforward neural network, the parameter update of the query key-value pair (q, k, v) is synchronously updated, wherein the update formula of the parameter update of the query key-value pair (q, k, v) can be:

[0087] q=W q x

[0088] k=W k x

[0089] v=W v x

[0090] Among them, q is the query vector, k is the key vector, v is the value vector, x is the time series feature vector, and W q , W k , W v is the weight coefficient.

[0091] Furthermore, the output of the self-attention module can be the sum of the weighted average of the value vector v and the time series feature vector x, where the weight ω of the value vector is determined by using the softmax function to query the dot product of q and all k. Based on this, the time slot feature vector corresponding to the action state is:

[0092] y=ωv+x

[0093]

[0094] Among them, v represents the value vector, x represents the time series feature vector, q represents the query vector, k represents the key vector, softmax represents the softmax function, and dimv represents the spatial dimension of the value vector v.

[0095] In one implementation of this embodiment, before performing the softmax operation, a preset mask can be applied to the weight matrix so that each layer in the feedforward neural network focuses mainly on its own observations, wherein the mask is used to separate different training stages (i.e., different state-action pairs) and is set according to different training stages so that the current training will not be affected by other training stages through the mask. For example, the mask of other training stages except the current training stage is set to 0, and after multiplying by the weight matrix, other training stages have no effect on this training. In addition, the mask does not cover up future observations, so the feedforward neural network can understand the relationship between current operations and future rewards. Then, the fully connected module of the feedforward neural network can use the output of the self-attention module to generate an internal reward r in , where the internal reward r in The calculation formula can be:

[0096] r in =FC η (y)

[0097] Among them, r in represents the internal reward, y represents the output of the self-attention module, FC η (.) indicates a fully connected module.

[0098] S40. Determine a first objective function and a first return value corresponding to the initial strategy based on each external reward and each internal reward, and update a strategy parameter of the initial strategy based on the first objective function and the first return value.

[0099] Specifically, the first objective function and the first return value are both the objective function and the return value of the policy gradient algorithm, wherein the first objective function and the first return value are calculated based on the external reward and the internal reward, wherein the calculation formulas of the first objective function and the first return value are respectively:

[0100]

[0101]

[0102] Among them, J ex+inrepresents the first objective function, γ represents the loss factor, and λ represents the hyperparameter used to balance the internal reward and the external reward. represents the intrinsic reward at time i, represents the external reward at time t, η represents the network parameters of the feedforward neural network, T represents the number of action states, s t represents the state at time t, a t represents the action at time t, s i represents the state at time i, a i Represents the action at time i.

[0103] Further, after obtaining the first objective function and the first return value, the policy parameters of the initial policy are updated in a regular policy gradient manner, wherein the update formula of the policy parameters can be:

[0104]

[0105] Among them, θ represents the policy parameters before updating, θ' represents the policy parameters after updating, α represents the weight coefficient, G ex+in (s t ,a t ) is the first return value, π is the initial strategy, s t represents the state at time t, a t Represents the action at time t.

[0106] In an implementation of this embodiment, the feedforward neural network is trained synchronously with the initial strategy. Accordingly, before continuing to perform the step of determining the state action trajectory corresponding to the task to be planned based on the deterministic finite automaton and the initial strategy corresponding to the task to be planned, until the target strategy step corresponding to the task to be planned is obtained, the method further includes:

[0107] A second objective function and a second return value corresponding to the initial strategy are determined based on each external reward, and the network parameters of the feedforward neural network are updated using the second objective function and the second return value.

[0108] Specifically, the second objective function and the second return value are both the objective function and the return value of the policy gradient algorithm, wherein the second objective function and the second return value are calculated based on the external reward, wherein the calculation formulas of the second objective function and the second return value are respectively:

[0109]

[0110]

[0111] Among them, J ex represents the second objective function, γ represents the loss factor, represents the external reward at time t, T represents the number of action states, and s t represents the state at time t, a t Represents the action at time t.

[0112] In addition, when updating the feedforward neural network, only the second objective function and the second return value determined based on the external reward are used to update the network parameters of the feedforward neural network, which can ensure that the internal reward helps to maximize the accumulated external reward. In a specific implementation, the partial derivative is as follows:

[0113]

[0114]

[0115]

[0116]

[0117] S50, continue to execute the step of determining the state action trajectory corresponding to the task to be planned based on the deterministic finite automaton and the initial strategy corresponding to the task to be planned, until a target strategy corresponding to the task to be planned is obtained.

[0118] Specifically, after the policy parameters of the initial policy and the network parameters of the feedforward neural network are updated, the updated initial policy is used as the initial test, and the updated feedforward neural network is used as the feedforward neural network for the next round of reinforcement learning, and so on until the target policy corresponding to the task to be planned is obtained.

[0119] In summary, this embodiment provides a task planning method based on reinforcement learning under temporal logic constraints, the method comprising converting the task to be planned into a deterministic finite automaton; determining the state-action trajectory corresponding to the task to be planned based on the deterministic finite automaton and the initial strategy corresponding to the task to be planned; inputting the state-action trajectory and the external rewards corresponding to each state-action pair into a preset feedforward neural network, and outputting the internal rewards corresponding to each state-action pair through the feedforward neural network; determining the first objective function and the first return value corresponding to the initial strategy based on each external reward and each internal reward, and updating the strategy parameters of the initial strategy based on the first objective function and the first return value; continuing to execute the step of determining the state-action trajectory corresponding to the task to be planned based on the deterministic finite automaton and the initial strategy corresponding to the task to be planned, until the target strategy corresponding to the task to be planned is obtained. This application captures the temporal characteristics of the task through the attention mechanism, so that the execution end can quickly learn tasks with temporal relationships in a sparse reward environment, so that the sparse reward problem under LTL constraints can be solved in different environments, so that reinforcement learning can learn the optimal strategy.

[0120] Based on the above-mentioned task planning method based on reinforcement learning under temporal logic constraints, this embodiment provides a task planning device based on reinforcement learning under temporal logic constraints, such as Figure 3 As shown, the task planning device includes:

[0121] A conversion module 100, used to convert the task to be planned into a deterministic finite automaton;

[0122] A determination module 200, configured to determine a state-action trajectory corresponding to the task to be planned based on the deterministic finite automaton and the initial strategy corresponding to the task to be planned, wherein each state-action pair in the dynamic action trajectory corresponds to an external reward;

[0123] A feedforward network module 300, for inputting the state-action trajectory and the external rewards corresponding to each state-action pair into a preset feedforward neural network, and outputting the internal rewards corresponding to each state-action pair through the feedforward neural network, wherein the feedforward neural network is configured with a self-attention mechanism;

[0124] An updating module 400, configured to determine a first objective function and a first return value corresponding to the initial strategy based on each external reward and each internal reward, and to update a strategy parameter of the initial strategy based on the first objective function and the first return value;

[0125] The execution module 500 is used to continue to execute the step of determining the state action trajectory corresponding to the task to be planned based on the deterministic finite automaton and the initial strategy corresponding to the task to be planned, until the target strategy corresponding to the task to be planned is obtained.

[0126] Based on the above-mentioned task planning method based on reinforcement learning under temporal logic constraints, this embodiment provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the task planning method based on reinforcement learning under temporal logic constraints as described in the above-mentioned embodiment.

[0127] Based on the above-mentioned task planning method based on reinforcement learning under the constraints of temporal logic, the present application also provides a terminal device, such as Figure 4 As shown, it includes at least one processor (processor) 20; display screen 21; and memory (memory) 22, and may also include a communications interface (Communications Interface) 23 and a bus 24. Among them, the processor 20, the display screen 21, the memory 22 and the communication interface 23 can communicate with each other through the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setting mode. The communication interface 23 can transmit information. The processor 20 can call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0128] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0129] The memory 22 is a computer-readable storage medium that can be configured to store software programs, computer executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions or modules stored in the memory 22, that is, implementing the methods in the above embodiments.

[0130] The memory 22 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 22 may include a high-speed random access memory and may also include a non-volatile memory. For example, a variety of media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, may also be a transient storage medium.

[0131] In addition, the specific working process of the above-mentioned training sample set acquisition device, the specific process of loading and executing multiple instruction processors in the storage medium and the terminal device have been described in detail in the above-mentioned method and will not be described here one by one.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A task planning method based on reinforcement learning under temporal logic constraints, characterized in that: The method comprises: Convert the task to be planned into a deterministic finite automaton; Determine a state-action trajectory corresponding to the task to be planned based on the deterministic finite automaton and the initial strategy corresponding to the task to be planned, wherein each state-action pair in the state-action trajectory corresponds to an external reward, and the state-action includes a state of the robot in the environment and an action to be performed by the robot based on the initial strategy in the state; Inputting the state-action trajectory and the external rewards corresponding to each state-action pair into a preset feedforward neural network, and outputting the internal rewards corresponding to each state-action pair through the feedforward neural network, wherein the feedforward neural network is configured with a self-attention mechanism; Determine a first objective function and a first return value corresponding to the initial strategy based on each external reward and each internal reward, and update a strategy parameter of the initial strategy based on the first objective function and the first return value; Continue to execute the step of determining the state action trajectory corresponding to the task to be planned based on the deterministic finite automaton and the initial strategy corresponding to the task to be planned, until a target strategy corresponding to the task to be planned is obtained; The feedforward neural network includes a self-attention module and a fully connected module; the step of inputting the state-action trajectory and the external rewards corresponding to each state-action pair into a preset feedforward neural network, and outputting the internal rewards corresponding to each state-action pair through the feedforward neural network specifically includes: Input the state-action trajectory and the external rewards corresponding to each state-action pair into the self-attention module, and output the time series characteristic vector corresponding to each state-action pair through the self-attention module; Inputting the time series characteristic vectors corresponding to each state-action pair into the fully connected module, and inputting the internal rewards corresponding to each action state through the fully connected module; The time slot characteristic vector corresponding to the action state is: y=ωv+x Among them, v represents the value vector, x represents the time series feature vector, q represents the query vector, k represents the key vector, softmax represents the softmax function, dim v represents the spatial dimension of the value vector v; The calculation formulas of the first objective function and the first return value are respectively: Among them, J ex+in represents the first objective function, G ex+in represents the first return value, γ represents the loss factor, and λ represents the hyperparameter. represents the intrinsic reward at time i, represents the external reward at time t, η represents the network parameters of the feedforward neural network, T represents the number of action states, s t represents the state at time t, a t represents the action at time t, s i represents the state at time i, a i Represents the action at time i.

2. The task planning method based on reinforcement learning under temporal logic constraints according to claim 1 is characterized in that: The determining of the state action trajectory corresponding to the task to be planned based on the deterministic finite automaton and the initial strategy corresponding to the task to be planned specifically includes: Get the current status of the environment where the execution end of the task to be planned is located; Perform action sampling based on the current state and the initial strategy corresponding to the task to be planned to obtain an action; Controlling the execution end to execute the action to obtain a next state, and checking a transition state of the next state in a deterministic finite automaton; If the transition state satisfies the first preset condition, a corresponding external reward is configured for the state-action pair and the state-action pair is terminated to obtain a state-action trajectory, wherein the state-action pair includes a state and an action; If the transition state satisfies a second preset condition, configuring a preset external reward for the state-action pair; The next state is taken as the current state, and the step of sampling actions based on the current state and the initial strategy corresponding to the task to be planned is continued to obtain actions until the transition state violates the timing logic or belongs to an acceptable state set or the trajectory length of the state action trajectory reaches a preset length threshold.

3. The task planning method based on reinforcement learning under temporal logic constraints according to claim 2 is characterized in that: The first preset condition is that the transition state violates the sequential logic or belongs to an acceptable state set; the second preset condition is that the transition state does not violate the sequential logic and does not belong to the acceptable state set, or the transition state of the next state in the deterministic finite automaton is not checked.

4. The task planning method based on reinforcement learning under temporal logic constraints according to claim 2 is characterized in that: If the transition state does not violate the sequential logic and does not belong to the acceptable state set, after configuring a preset external reward for the state-action pair consisting of the current state and the action, the method further includes: The state of the deterministic finite automaton is reset to an initial state of the deterministic finite automaton.

5. The task planning method based on reinforcement learning under temporal logic constraints according to claim 1 is characterized in that: The method further comprises: continuing to execute the step of determining the state action trajectory corresponding to the task to be planned based on the deterministic finite automaton and the initial strategy corresponding to the task to be planned until the target strategy corresponding to the task to be planned is obtained: A second objective function and a second return value corresponding to the initial strategy are determined based on each external reward, and the network parameters of the feedforward neural network are updated using the second objective function and the second return value.

6. A task planning device based on reinforcement learning under temporal logic constraints, characterized in that: The task planning device comprises: A conversion module, used for converting the task to be planned into a deterministic finite automaton; A determination module, configured to determine a state-action trajectory corresponding to the task to be planned based on the deterministic finite automaton and an initial strategy corresponding to the task to be planned, wherein each state-action pair in the state-action trajectory corresponds to an external reward; A feedforward network module, used for inputting the state-action trajectory and the external rewards corresponding to each state-action pair into a preset feedforward neural network, and outputting the internal rewards corresponding to each state-action pair through the feedforward neural network, wherein the feedforward neural network is configured with a self-attention mechanism; An updating module, configured to determine a first objective function and a first return value corresponding to the initial strategy based on each external reward and each internal reward, and to update a strategy parameter of the initial strategy based on the first objective function and the first return value, wherein the state action includes a state of the robot in the environment and an action to be performed by the robot based on the initial strategy in the state; an execution module, configured to continue to execute the step of determining the state action trajectory corresponding to the task to be planned based on the deterministic finite automaton and the initial strategy corresponding to the task to be planned, until a target strategy corresponding to the task to be planned is obtained; The feedforward neural network includes a self-attention module and a fully connected module; the step of inputting the state-action trajectory and the external rewards corresponding to each state-action pair into a preset feedforward neural network, and outputting the internal rewards corresponding to each state-action pair through the feedforward neural network specifically includes: Input the state-action trajectory and the external rewards corresponding to each state-action pair into the self-attention module, and output the time series characteristic vector corresponding to each state-action pair through the self-attention module; Inputting the time series characteristic vectors corresponding to each state-action pair into the fully connected module, and inputting the internal rewards corresponding to each action state through the fully connected module; The time slot characteristic vector corresponding to the action state is: y=ωv+x Among them, v represents the value vector, x represents the time series feature vector, q represents the query vector, k represents the key vector, softmax represents the softmax function, dim v represents the spatial dimension of the value vector v; The calculation formulas of the first objective function and the first return value are respectively: Among them, J ex+in represents the first objective function, G ex+in represents the first return value, γ represents the loss factor, and λ represents the hyperparameter. represents the intrinsic reward at time i, represents the external reward at time t, η represents the network parameters of the feedforward neural network, T represents the number of action states, s t represents the state at time t, a t represents the action at time t, s i represents the state at time i, a i Represents the action at time i.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the task planning method based on reinforcement learning under the constraints of temporal logic as described in any one of claims 1-5.

8. A terminal device, characterized in that: include: Processor, memory and communication bus; The memory stores a computer-readable program executable by the processor; The communication bus realizes the connection and communication between the processor and the memory; When the processor executes the computer-readable program, the steps in the task planning method based on reinforcement learning under the temporal logic constraints as described in any one of claims 1 to 5 are implemented.

Images