Improved state space based signal temporal logic reinforcement learning planning method

Abstract

This invention discloses a signal-temporal logic reinforcement learning planning method and apparatus based on an improved state space, relating to the field of robot intelligent control and autonomous navigation technology. The method includes: acquiring a robot path planning task described in natural language; converting the planning task into signal-temporal logic constraints using a signal-temporal logic-based approach; constructing a K-MDP augmented state space model, defining flag bits for each subtask using flag bit operators and encoding them as one-hot encoded vectors; constructing the robot's reinforcement learning state space based on the one-hot encoded vectors and the robot's physical position; constructing the main reward function and reward shaping function for the currently activated subtask; summing the two functions to construct a single-step total reward function; and using the robot's reinforcement learning state space and the single-step total reward function, employing a Q-learning reinforcement learning algorithm to solve the robot's path planning action sequence through iterative interaction between the agent and the environment. This invention can improve the success rate of path planning.

Description

Technical Field

[0001] This invention relates to the field of robot intelligent control and autonomous navigation technology, and in particular to a signal temporal logic reinforcement learning planning method and apparatus based on improved state space. Background Technology

[0002] With the accelerating pace of intelligentization in fields such as autonomous driving, unmanned systems, and intelligent service robots, robot motion planning technology has become a core component of autonomous decision-making for intelligent agents. Its performance directly impacts the system's safety, robustness, and task execution efficiency. However, in practical applications, robots often need to perform complex tasks involving strict temporal logic and spatial constraints, which traditional methods struggle to effectively model. Signal temporal logic provides a rigorous mathematical framework that allows the temporal relationships of tasks to be represented in formulas. As a variant of temporal logic, signal temporal logic can constrain continuous time and space, accurately describing real-world task specifications related to time requirements and sequence. Therefore, applying signal temporal logic to robot motion planning can improve the reliability and efficiency of completing complex tasks.

[0003] Traditional solutions to robot path planning problems under signal-sequential logic constraints primarily rely on mixed-integer linear programming. This method transforms the logical operators in the signal-sequential logic formula into mixed-integer constraints, constructing the path planning problem as a constrained optimization problem. A mathematical solver directly calculates the globally optimal trajectory that meets the timing requirements. Compared to traditional mixed-integer linear programming, this endpoint backtracking-based planning method, through optimized search strategies, offers faster computation speeds in online solutions and better meets real-time requirements. A funnel feedback control strategy is employed to achieve seamless integration of planning and control. Furthermore, existing techniques utilize obstacle functions to model the workspace and task specifications, and based on this, conduct collaborative design of planning and control to ensure the agent satisfies safety constraints while completing the task. This approach optimizes robot trajectory planning and real-time control response in complex environments, thereby improving the overall system performance and adaptability.

[0004] To overcome the dependence of traditional optimization methods on environmental models and improve the adaptability of robots in unknown environments, path planning methods for unknown environments have been explored in recent years. One important method is reinforcement learning, which learns strategies through trial and error in the interaction between the agent and the environment, making it naturally suitable for decision-making problems in unknown environments. This approach abstracts and models the interaction process between the robot and the environment as a Markov decision process model, describes the robot's task requirements in a given environment as a signal-temporal logic task, designs a reward function based on robustness, and trains a path planning strategy for the signal-temporal logic task using a reinforcement learning algorithm.

[0005] Research on how to generate paths that satisfy Signal Temporal Logic (STL) constraints for robot reinforcement learning controllers in unknown environments is still in the exploratory stage. Existing methods construct a high-dimensional τ-MDP model that extends the multi-step history, directly concatenating the robot's recent historical state sequence into the current state. By inputting partial historical trajectories, the agent can determine the progress of satisfying the temporal logic, thus enabling the training of a path that meets the STL task requirements using the extended reinforcement learning model. However, this method heavily relies on partial historical trajectories. As the horizon length of the largest subtask increases, the state space grows exponentially, making training extremely difficult.

[0006] To overcome the curse of dimensionality caused by recording historical trajectories, instead of storing continuous historical trajectories, this method introduces a set of discrete Boolean flags to abstract and record the progress of each subtask. Specifically, this method decomposes the complex STL global task into several subtasks and assigns a flag variable to each subtask. The robot's physical state is combined with all the flag sets to construct an augmented state space. Although this simplifies the state representation to some extent compared to τ-MDP, it requires inputting the flags of all subtasks into the neural network simultaneously. When the subtask duration is long, the state space grows linearly, leading to high computational complexity and making it difficult for the agent to focus on the most pressing task. Furthermore, this method suffers from poor guidance due to sparse reward signals. During the intermediate stages of task completion, the system consistently provides the same reward. This sparse reward mechanism results in a lack of effective feedback signals for the robot in the early stages of training, making it difficult for the agent to determine whether the current action brings it closer to the goal. This forces the robot to perform a large amount of ineffective random exploration in the environment, severely reducing learning efficiency and further decreasing the success rate of path planning. Therefore, a novel path planning method that can provide dense guidance rewards is needed. Summary of the Invention

[0007] To address the problems of existing technologies that rely on recording a large number of historical trajectories or subtask completion states, leading to a linear growth of the state space with increasing time, resulting in extremely high computational load and difficulty in convergence, thus reducing the success rate of path planning; and the difficulty of effectively mapping the temporal robustness of STL to the immediate rewards of reinforcement learning in existing reward function designs, easily generating sparse rewards, leading to low robot training efficiency, slow convergence speed, and difficulty in generating smooth trajectories, this invention provides a signal temporal logic reinforcement learning planning method and apparatus based on an improved state space. The technical solution is as follows: On the one hand, a signal temporal logic reinforcement learning planning method based on an improved state space is provided. This method is implemented by a signal temporal logic reinforcement learning planning device based on an improved state space, and includes: S1. Obtain the robot path planning task described in natural language; use signal-time logic to transform the robot path planning task into signal-time logic constraints and define multiple sub-tasks; S2. Construct a K-MDP augmented state space model; input the signal timing logic constraints into the K-MDP augmented state space model, define a flag bit for each subtask through the flag bit operator to characterize the current satisfaction level of the subtask, and dynamically update the flag bit according to the physical state of the robot; directly compare the flag bit sizes of all subtasks at the current time, extract the subtask corresponding to the minimum value, and encode it into a one-hot encoded vector. S3. Based on the one-hot encoded vector and the robot's physical position, construct the robot's reinforcement learning state space; based on the one-hot encoded vector indicating the currently activated sub-task, and combined with the signal temporal logic space robustness of the sub-task, construct the main reward function; use the spatial robustness difference of the currently activated sub-task at adjacent time steps to construct the reward shaping function; sum the main reward function and the reward shaping function to construct the single-step total reward function. S4. Based on the robot's reinforcement learning state space and single-step total reward function, the Q-learning reinforcement learning algorithm is used to solve the robot path planning action sequence through the interaction and iteration between the agent and the environment.

[0008] On the other hand, a signal-temporal logic reinforcement learning planning device based on an improved state space is provided. This device is applied to a signal-temporal logic reinforcement learning planning method based on an improved state space. The device includes: The acquisition unit is used to acquire the robot path planning task described in natural language; and to convert the robot path planning task into signal-time logic constraints and define multiple sub-tasks using signal-time logic-based methods. The first building unit is used to construct the K-MDP augmented state space model. The signal timing logic constraints are input into the K-MDP augmented state space model. A flag bit is defined for each subtask through the flag bit operator to represent the current satisfaction level of the subtask. The flag bit is dynamically updated according to the physical state of the robot. The flag bit values ​​of all subtasks at the current time are directly compared, the subtask corresponding to the minimum value is extracted, and it is encoded into a one-hot encoded vector. The second construction unit is used to construct the robot's reinforcement learning state space based on the one-hot encoded vector and the robot's physical position; construct the main reward function based on the one-hot encoded vector indicating the currently active subtask and combining the signal temporal logic space robustness of the subtask; construct the reward shaping function using the spatial robustness difference of the currently active subtask at adjacent time steps; and sum the main reward function and the reward shaping function to construct the single-step total reward function. The solver unit is used to solve the robot path planning action sequence based on the robot's reinforcement learning state space and single-step total reward function. It employs the Q-learning reinforcement learning algorithm to solve the robot's path planning action sequence through the interaction and iteration between the agent and the environment.

[0009] On the other hand, a signal temporal logic reinforcement learning planning device based on an improved state space is provided. The improved state space-based signal temporal logic reinforcement learning planning device includes: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, they implement any of the methods described above for signal temporal logic reinforcement learning planning based on an improved state space.

[0010] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement any of the above-described signal-sequential logic reinforcement learning planning methods based on improved state space.

[0011] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: This invention proposes a robot path planning method based on K-MDP state space simplification and logical potential field reward shaping, which improves the success rate of path planning. The method includes standardized modeling of signal-sequential logic tasks, construction of the K-MDP state space based on one-hot encoding, and design of a two-layer reward mechanism.

[0012] The use of one-hot encoding to construct the K-MDP state space is a key aspect of this invention. Unlike existing technologies (such as F-MDP) that record all subtask flags leading to exponential state growth, this invention uses one-hot encoded vectors to dynamically indicate the most urgent subtask, successfully reducing the state space growth from exponential to linear and solving the "curse of dimensionality" problem. Furthermore, the two-layer reward mechanism based on the logical potential field accurately and efficiently provides action guidance for the robot. By utilizing the difference in STL robustness between adjacent time steps to construct a guidance signal, the sparse reward problem in traditional STL tasks is effectively solved, allowing the robot to obtain continuous optimization direction even during the intermediate stages of task completion, significantly accelerating the algorithm's convergence speed. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of a signal temporal logic reinforcement learning planning method based on an improved state space, provided by an embodiment of the present invention. Figure 2 This is a schematic diagram of the algorithm system framework of a signal temporal logic reinforcement learning planning method based on an improved state space, provided by an embodiment of the present invention. Figure 3 This is a schematic diagram of a K-MDP augmented state construction and policy network structure provided in an embodiment of the present invention; Figure 4 This is a block diagram of a signal temporal logic reinforcement learning planning device based on an improved state space, provided by an embodiment of the present invention. Figure 5 This is a schematic diagram of the structure of a signal temporal logic reinforcement learning planning device based on an improved state space, provided by an embodiment of the present invention. Detailed Implementation

[0015] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0016] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0017] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0018] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0019] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0020] This invention provides a signal temporal logic reinforcement learning planning method based on an improved state space. This method can be implemented by a signal temporal logic reinforcement learning planning device based on an improved state space, which can be a terminal or a server. Figure 1 The flowchart shown is for a signal-temporal logic reinforcement learning planning method based on an improved state space. The processing flow of this method may include the following steps: S1. Obtain the robot path planning task described in natural language; use signal-time logic to transform the robot path planning task into signal-time logic constraints and define multiple sub-tasks.

[0021] Among them, such as Figure 2 The diagram shows a schematic of the algorithm system framework of a signal temporal logic reinforcement learning planning method based on an improved state space, provided by an embodiment of the present invention. In one feasible implementation, signal temporal logic is used, a formal language widely used to describe high-level logical behavior on continuous signals. In this embodiment of the present invention, signal temporal logic is used to transform the input task described in natural language into mathematical logical constraints, expressed by the following formula (1): (1) in, This represents the top-level task formula, used to describe global constraints throughout the entire time window; This represents a mid-level logic formula used to describe specific sequential subtasks; subscripts are used to indicate the underlying logic. Indicates a time interval; This represents the first-level logic formula; G represents the second intermediate-level logic formula; G represents the time operator, indicating that the logic formula must be true at every moment within the time interval; F represents the final time operator, indicating that there exists at least one moment within the time interval for the logic formula to be true. The underlying predicate formula represents a subtask; T and The time interval symbol under the time operator. Two non-negative constants representing a time interval; Indicates an atomic predicate; This represents the first-level predicate formula; This represents the second-level predicate formula; Indicates logical NOT; This indicates a logical AND operation. Represents a logical truth value.

[0022] Natural language descriptions only provide an intuitive understanding of the task; rigorous formal semantics must be introduced to enable algorithmic solutions. To establish mathematical criteria for evaluating whether the continuous state trajectory generated by the robot's planning effectively follows the predefined STL task formula, a mathematical criterion is used. For signal s, ... To represent its value at time t, similarly, using The signal s satisfies the formula at time t, and the specific process is represented by the following formula (2): (2) Among them, the function The meaning is the distance of the state signal s from the boundary of the task μ (i.e., the degree of satisfaction); when When the task is satisfied, This indicates a violation of the mission.

[0023] Under the aforementioned semantic definition, given a sequence of robot positions in the environment, while it's possible to determine whether this sequence satisfies the task, this is a binary Boolean semantic, meaning the result is simply 'satisfied' or 'not satisfied'. To transform the logical satisfaction level into a continuous reward signal required for reinforcement learning, this embodiment of the invention further employs spatial robustness as a quantification indicator. Spatial robustness not only determines whether the logic holds true but also calculates the 'distance' of the state trajectory from satisfying (or violating) the logical formula; whereby the spatial robustness function... This can be expressed by the following formula (3):

[0024] in, Indicates status signal exist Moment about atomic predicates Spatial robustness; Indicates status signal exist Time with respect to logical non-formula Spatial robustness; Indicates status signal exist Time about logical formulas Spatial robustness; Indicates status signal exist Time with respect to the first underlying predicate formula Spatial robustness; Indicates status signal exist Time with respect to the second-level predicate formula Spatial robustness; Indicates status signal exist Formula for time with respect to the final time operator Spatial robustness; Indicates status signal exist Time-of-start and end time operator formula Spatial robustness.

[0025] In one feasible implementation, the robot path planning task described by natural language in this embodiment of the invention includes multiple sub-tasks.

[0026] In one feasible implementation method, embodiments of the present invention list three typical tasks: arrival task, charging task, and patrol task. Among them, the arrival task... This means that the robot must eventually reach the designated target area within a specified time. Among them, charging task This means the robot needs to arrive at the charging station within a specified time T. Each time it arrives at the charging station, the robot needs to stay within the charging area for a certain period of time. To complete charging, if the robot leaves the charging area during charging, it will need to start charging from scratch. Among them, patrol missions This means the robot needs to continuously patrol two target areas within a specified time; the robot's patrol time cycle is... For the entire task time period Within, the arbitrary time length is All robot path segments must have corresponding waypoints that can be found in all target areas.

[0027] S2. Construct a K-MDP augmented state space model; input the signal timing logic constraints into the K-MDP augmented state space model, define a flag bit for each subtask through the flag bit operator to characterize the current satisfaction level of the subtask, and dynamically update the flag bit according to the physical state of the robot; directly compare the flag bit sizes of all subtasks at the current time, extract the subtask corresponding to the minimum value, and encode it into a one-hot encoded vector.

[0028] To address the exponential explosion of the state space caused by recording complete historical trajectories in traditional extended Markov decision processes, this invention proposes a K-MDP augmented state space model. The core of this model lies in introducing a flag operator instead of directly storing historical trajectories. , where n represents the number of subtasks. Specifically, for each subtask... Define the flag bit This is used to characterize the current satisfaction level of the subtask. The flag for the next time step... The update follows the following logic: if the current state If the subtask requirements are met, the flag is incremented or increased; if the current state does not meet the subtask requirements (violation of requirements), the flag is reset or decremented. The state update formula is expressed by the following formula (4): (4) in, Indicates the first Each sub-task is The size of the time flag; Indicates the first Each sub-task is The size of the time flag; Represents the time interval constant under the time operator; Indicates that the robot is The physical state at any given moment; Indicates the final time operator; Indicates until the time operator; Indicates the first Sub-tasks; Indicates the first The underlying predicate formulas for each subtask.

[0029] In order to extract the most critical decision information from the flags of multiple subtasks, this embodiment of the invention introduces operator K. The flag group is encoded into a one-hot encoded vector. Each element... The value is determined by the corresponding flag bit. Whether it is the minimum value among all current flags determines its importance. Physically, it points to the subtask with the highest urgency or priority.

[0030] Optionally, the process of comparing the flag values ​​of all subtasks at the current time, extracting the subtask corresponding to the minimum value, and encoding it into a one-hot encoded vector is represented by the following formula (5): (5) in, One-hot encoding of time t The i-th component; The flag bit represents the i-th subtask at time t; i represents the i-th subtask, which also corresponds to the i-th component in one-hot encoding; t represents time.

[0031] The robot's reinforcement learning state space is represented as follows: Where S is the physical position of the robot. This is a one-hot encoded vector; if multiple minimum values ​​exist, the index is selected by default. The minimum task is set to 1. This design reduces the dimension of the state space from exponential growth to linear growth, significantly reducing computational complexity.

[0032] S3. Based on the one-hot encoded vector and the robot's physical position, construct the robot's reinforcement learning state space; based on the one-hot encoded vector indicating the currently activated sub-task, and combined with the signal temporal logic space robustness of the sub-task, construct the main reward function; use the spatial robustness difference of the currently activated sub-task at adjacent time steps to construct the reward shaping function; sum the main reward function and the reward shaping function to construct the single-step total reward function.

[0033] To address the reward sparsity problem caused by the fact that signal-sequential logic tasks typically only award rewards upon task completion, this embodiment of the invention designs a dense logic potential field reward function based on the robustness of signal-sequential logic tasks and combined with the artificial potential field method. The first layer is the main reward function, which uses the robustness of signal-sequential logic to quantify the degree of task satisfaction. This embodiment of the invention uses a one-hot encoded vector to calculate the main reward of the currently active subtask.

[0034] This invention incorporates a reward-shaping framework based on artificial potential fields, but unlike traditional methods that construct potential fields based on physical distance, this invention innovatively utilizes the robustness of STL tasks. This invention maps the robustness value of the current K-MDP activation subtask to a kind of logical potential energy. Since STL robustness can quantify the degree to which complex logic is satisfied (i.e., the closer the state is to satisfying the requirements, the greater the robustness / potential energy), the reward shaping function constructed based on the robustness difference between adjacent time steps essentially captures the potential energy change trend of the agent in the logical potential field. When this occurs, it indicates that the agent is moving in the direction that satisfies the logical constraints (i.e., the direction of increasing potential energy), thereby obtaining a positive reward. This provides the agent with dense guiding signals that are not limited to spatial location but also include the logical progress of the task.

[0035] Optionally, the main reward function of the currently active subtask is represented by the following formula (6): (6) in, Represents a one-hot encoded vector The first time t One component; This represents a weighting coefficient indicating that the robot's robustness to the target task is greater than 0. Indicates the next state after the action is performed. robustness value; express The value of the main reward function at time t; t represents time t. This indicates the total number of subtasks.

[0036] The second layer is a reward shaping function. To provide continuous and dense feedback signals during the exploration phase when the robot has not completed its task, this embodiment of the invention introduces a potential energy-based reward shaping mechanism based on the main reward function. Specifically, it calculates the difference in robustness value of the currently active subtask before and after the robot performs an action.

[0037] Alternatively, the reward shaping function can be expressed by the following formula (7): (7) in, Represents a one-hot encoded vector The One component; This represents the weighting coefficient for reward shaping; it indicates; Indicates the next state after the action is performed. robustness value; Indicates the current state Corresponding subtasks Spatial robustness value; express The reward shaping function value at time t; t represents time t.

[0038] The reward shaping function constructs a potential field based on robustness values. When the robot moves in the direction that increases robustness, i.e., Those who perform well will receive positive shaping rewards; conversely, those who perform poorly will receive negative penalties.

[0039] In one feasible implementation, the total reward function for a single step is expressed by the following formula (8): (8) in, express The reward shaping function value at any given moment; express The value of the main reward function at time 1; This indicates the total reward for a single step.

[0040] S4. Based on the robot's reinforcement learning state space and single-step total reward function, the Q-learning reinforcement learning algorithm is used to solve the robot path planning action sequence through the interaction and iteration between the agent and the environment.

[0041] In one feasible implementation, to enable the robot to autonomously learn the optimal motion strategy that satisfies temporal logic constraints, this embodiment of the invention employs the Q-learning algorithm as the solver. This algorithm can transform state definitions and reward signals into specific path planning action sequences through iterative interaction between the agent and the environment.

[0042] Optionally, the specific implementation process of S4 includes S41-S48: S41. Model the task and abstract it as a Markov decision process; define the state space S, action space A, state transition probabilities and reward function R; create a Q table and set its initial value to 0, and set hyperparameters including: learning rate, discount factor, exploration rate and maximum number of rounds; the hyperparameters remain unchanged after initialization until the end of the iteration. S42. Single-round learning initialization: At the beginning of each training round, the environment is reset to the initial state, the robot returns to the initial position, and the flag bit is reset to 0. The one-hot encoded vector at the initial moment is calculated. S43, Adopt A greedy strategy selects the next action based on the probability of exploration. Randomly select an action, or with a probability of 1- Select the current status in the Q table. The action that maximizes the Q value; Among them, such as Figure 3 The diagram shown is a schematic of a K-MDP augmented state construction and policy network structure provided by an embodiment of the present invention; wherein, the exploration probability Initially, the learning curve is high, allowing the robot to explore many new samples. As the number of learning iterations increases, the learning curve gradually diminishes, increasing the weight of the samples and causing the algorithm to converge to the optimal strategy.

[0043] Optionally, adopt The process of the greedy strategy choosing the next action is represented by the following formula (9): (9) in, Indicates the state Select action Strategies; Indicates the state Select action The corresponding Q value; Indicates the robot's physical location; Indicates the robot's actions; represents the exploration rate (i.e., the probability of randomly selecting an action); t represents the current time. This represents the one-hot encoded vector of the most urgent subtask.

[0044] S44. Execute the selected action, obtain the physical location coordinates at the next moment, calculate the flag bits of each subtask according to the flag bit update rule, and update the one-hot encoding vector to obtain the complete K-MDP state at the next moment. ; S45. Calculate the current single-step total reward based on the updated one-hot encoding vector; S46. Based on the current total reward for a single step, update the Q table using a time-series difference equation; Optionally, the process of updating the Q-table using the time-series difference equation based on the current total reward for a single step is represented by the following formula (10): (10) in, Representing state Next action Q value; express Time is determined by physical location The augmented state formed by combining the one-hot encoded K vector; express Actions selected and executed at any time; This represents the one-hot encoded vector of the most urgent subtask. This represents the learning rate used to control the degree of sample estimation updates; express Total reward per step at any given moment; This represents the discount factor used to set the weight of future rewards; Indicates the state at the next moment. Next action Q value; express The augmented state of time; Indicates the state at the next moment. The corresponding action below.

[0045] Among them, learning rate Then control the update degree of the sample estimate, if Then completely replace it with the new value, if Then it will not be updated at all.

[0046] S47. Update the current state to the next state. Determine whether the update state has reached the termination state. If it has not reached the termination state, return to step S43 and continue iterating. If it has reached the termination state, end the current round. S48. After training is complete, for any state Choose the action with the highest Q value. That is, directly select the action with the largest Q value in the current state; at this time, the action sequence from the initial state to the target state is the optimal strategy learned by the robot in the early stage.

[0047] This invention proposes a robot path planning method based on K-MDP state space simplification and logical potential field reward shaping, which improves the success rate of path planning. The method includes standardized modeling of signal-sequential logic tasks, construction of the K-MDP state space based on one-hot encoding, and design of a two-layer reward mechanism.

[0048] The use of one-hot encoding to construct the K-MDP state space is a key aspect of this invention. Unlike existing technologies (such as F-MDP) that record all subtask flags leading to exponential state growth, this invention uses one-hot encoded vectors to dynamically indicate the most urgent subtask, successfully reducing the state space growth from exponential to linear and solving the "curse of dimensionality" problem. Furthermore, the two-layer reward mechanism based on the logical potential field accurately and efficiently provides action guidance for the robot. By utilizing the difference in STL robustness between adjacent time steps to construct a guidance signal, the sparse reward problem in traditional STL tasks is effectively solved, allowing the robot to obtain continuous optimization direction even during the intermediate stages of task completion, significantly accelerating the algorithm's convergence speed.

[0049] Figure 4 This is a block diagram of a signal temporal logic reinforcement learning planning device based on an improved state space, provided by an embodiment of the present invention. This device is used for a signal temporal logic reinforcement learning planning method based on an improved state space. (Refer to...) Figure 4 The device includes an acquisition unit 410, a first construction unit 420, a second construction unit 430, and a solution unit 440. Wherein: The acquisition unit 410 is used to acquire the robot path planning task described in natural language; and to convert the robot path planning task into signal-time logic constraints and define multiple sub-tasks using signal-time logic-based methods. The first building unit 420 is used to build a K-MDP augmented state space model. The signal timing logic constraints are input into the K-MDP augmented state space model. A flag bit is defined for each subtask through a flag bit operator to represent the current satisfaction level of the subtask. The flag bit is dynamically updated according to the physical state of the robot. The flag bit values ​​of all subtasks at the current time are directly compared, the subtask corresponding to the minimum value is extracted, and it is encoded into a one-hot encoded vector. The second construction unit 430 is used to construct the robot's reinforcement learning state space based on the one-hot encoded vector and the robot's physical position; construct the main reward function based on the one-hot encoded vector indicating the currently activated sub-task and combining the signal temporal logic space robustness of the sub-task; construct the reward shaping function using the spatial robustness difference of the currently activated sub-task at adjacent time steps; and sum the main reward function and the reward shaping function to construct the single-step total reward function. Solver 440 is used to solve the robot path planning action sequence based on the robot's reinforcement learning state space and single-step total reward function. It adopts the Q-learning reinforcement learning algorithm to solve the robot path planning action sequence through the interaction and iteration between the agent and the environment.

[0050] Optionally, the process of defining a flag bit for each subtask using the flag bit operator to characterize the current satisfaction level of the subtask, and dynamically updating the flag bit according to the physical state of the robot, is represented by the following formula (1): (1) in, Indicates the first Each sub-task is The size of the time flag; Indicates the first Each sub-task is The size of the time flag; Represents the time interval constant under the time operator; Indicates that the robot is The physical state at any given moment; Indicates the final time operator; Indicates until the time operator; Indicates the first Sub-tasks; Indicates the first The underlying predicate formulas for each subtask.

[0051] Optionally, the process of comparing the flag values ​​of all subtasks at the current time, extracting the subtask corresponding to the minimum value, and encoding it into a one-hot encoded vector is represented by the following formula (2): (2) in, One-hot encoding of time t The i-th component; The flag bit represents the i-th subtask at time t; i represents the i-th subtask, which also corresponds to the i-th component in one-hot encoding; t represents time.

[0052] Optionally, the main reward function of the currently active subtask is represented by the following formula (3): (3) in, Represents a one-hot encoded vector The first time t One component; This represents a weighting coefficient indicating that the robot's robustness to the target task is greater than 0. Indicates the next state after the action is performed. robustness value; express The value of the main reward function at time t; t represents time t. This indicates the total number of subtasks.

[0053] Optionally, the reward shaping function is represented by the following formula (4): (4) in, Represents a one-hot encoded vector At time t One component; This represents the weighting coefficient for rewarding shaping; Indicates the next state after the action is performed. robustness value; Indicates the current state Corresponding subtasks Spatial robustness value; express The reward shaping function value at time t; t represents time t.

[0054] Optionally, the solving unit 440 is used for: S41. Model the task and abstract it as a Markov decision process; define the state space S, action space A, state transition probabilities and reward function R; create a Q table and set its initial value to 0, and set hyperparameters including: learning rate, discount factor, exploration rate and maximum number of rounds; the hyperparameters remain unchanged after initialization until the end of the iteration. S42. Single-round learning initialization: At the beginning of each training round, the environment is reset to the initial state, the robot returns to the initial position, and the flag bit is reset to 0. The one-hot encoded vector at the initial moment is calculated. S43, Adopt A greedy strategy selects the next action based on the probability of exploration. Randomly select an action, or explore with a probability of 1- Select the current status in the Q table. The action that maximizes the Q value; S44. Execute the selected action, obtain the physical location coordinates at the next moment, calculate the flag bits of each subtask according to the flag bit update rule, and update the one-hot encoding vector to obtain the complete K-MDP state at the next moment. ; S45. Calculate the current single-step total reward based on the updated one-hot encoding vector; S46. Based on the current total reward for a single step, update the Q table using a time-series difference equation; S47. Update the current state to the next state. Determine whether the update state has reached the termination state. If it has not reached the termination state, return to step S43 and continue iterating. If it has reached the termination state, end the current round. S48. After training is complete, for any state The action with the highest Q-value is selected as the optimal strategy learned by the robot.

[0055] Optionally, the adoption The process of the greedy strategy choosing the next action is represented by the following formula (5): (5) in, Indicates the state Select action Strategies; Indicates the state Select action The corresponding Q value; Indicates the robot's physical location; Indicates the robot's actions; represents the exploration rate (i.e., the probability of randomly selecting an action); t represents the current time. This represents the one-hot encoded vector of the most urgent subtask. The process of updating the Q-table based on the current single-step total reward through a time-series difference equation is represented by the following formula (6): (6) in, Representing state Next action Q value; express Time is determined by physical location The augmented state formed by combining the one-hot encoded K vector; express Actions selected and executed at any time; This represents the one-hot encoded vector of the most urgent subtask. This represents the learning rate used to control the degree of sample estimation updates; express Total reward per step at any given moment; This represents the discount factor used to set the weight of future rewards; Indicates the state at the next moment. Next action Q value; express The augmented state of time; Indicates the state at the next moment. The corresponding action below.

[0056] This invention proposes a robot path planning method based on K-MDP state space simplification and logical potential field reward shaping, which improves the success rate of path planning. The method includes standardized modeling of signal-sequential logic tasks, construction of the K-MDP state space based on one-hot encoding, and design of a two-layer reward mechanism.

[0057] The use of one-hot encoding to construct the K-MDP state space is a key aspect of this invention. Unlike existing technologies (such as F-MDP) that record all subtask flags leading to exponential state growth, this invention uses one-hot encoded vectors to dynamically indicate the most urgent subtask, successfully reducing the state space growth from exponential to linear and solving the "curse of dimensionality" problem. Furthermore, the two-layer reward mechanism based on the logical potential field accurately and efficiently provides action guidance for the robot. By utilizing the difference in STL robustness between adjacent time steps to construct a guidance signal, the sparse reward problem in traditional STL tasks is effectively solved, allowing the robot to obtain continuous optimization direction even during the intermediate stages of task completion, significantly accelerating the algorithm's convergence speed.

[0058] Figure 5 This is a schematic diagram of the structure of a signal temporal logic reinforcement learning planning device based on an improved state space, as provided in an embodiment of the present invention. Figure 5 As shown, the signal temporal logic reinforcement learning planning device based on improved state space can include the above-mentioned... Figure 4 The illustrated signal-temporal logic reinforcement learning planning apparatus based on an improved state space is shown. Optionally, the signal-temporal logic reinforcement learning planning apparatus 510 based on an improved state space may include a first processor 2001.

[0059] Optionally, the signal-sequential logic reinforcement learning planning device 510 based on the improved state space may also include a memory 2002 and a transceiver 2003.

[0060] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0061] The following is combined Figure 5 A detailed introduction to each component of the signal-sequential logic reinforcement learning planning device 510 based on improved state space is provided below: The first processor 2001 is the control center of the signal-temporal logic reinforcement learning planning device 510 based on the improved state space. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0062] Optionally, the first processor 2001 can execute various functions of the signal-temporal logic reinforcement learning planning device 510 based on the improved state space by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0063] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 5 CPU0 and CPU1 are shown in the diagram.

[0064] In a specific implementation, as one example, the signal-temporal logic reinforcement learning planning device 510 based on the improved state space may also include multiple processors, for example... Figure 5 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0065] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0066] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be interfaced through the signal-timing logic reinforcement learning planning device 510 based on the improved state space. Figure 5 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0067] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0068] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 5 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0069] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be interfaced with the signal-timing logic reinforcement learning planning device 510 based on the improved state space. Figure 5 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0070] It should be noted that, Figure 5 The structure of the signal timing logic reinforcement learning planning device 510 based on the improved state space shown does not constitute a limitation on the router. Actual signal timing logic reinforcement learning planning devices based on the improved state space may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0071] Furthermore, the technical effects of the signal temporal logic reinforcement learning planning device 510 based on the improved state space can be referred to the technical effects of the signal temporal logic reinforcement learning planning method based on the improved state space described in the above method embodiments, and will not be repeated here.

[0072] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0073] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0074] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0075] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0076] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0077] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0078] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0079] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0080] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0081] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0082] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0083] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0084] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An improved state space based signal temporal logic reinforcement learning planning method, characterized in that, The method includes: S1. Obtain the robot path planning task described in natural language; use signal-time logic to transform the robot path planning task into signal-time logic constraints and define multiple sub-tasks; S2. Construct a K-MDP augmented state space model; input the signal timing logic constraints into the K-MDP augmented state space model, define a flag bit for each subtask through the flag bit operator to characterize the current satisfaction level of the subtask, and dynamically update the flag bit according to the physical state of the robot; directly compare the flag bit sizes of all subtasks at the current time, extract the subtask corresponding to the minimum value, and encode it into a one-hot encoded vector. S3. Based on the one-hot encoded vector and the robot's physical position, construct the robot's reinforcement learning state space; based on the one-hot encoded vector indicating the currently activated sub-task, and combined with the signal temporal logic space robustness of the sub-task, construct the main reward function; use the spatial robustness difference of the currently activated sub-task at adjacent time steps to construct the reward shaping function; sum the main reward function and the reward shaping function to construct the single-step total reward function. S4. Based on the robot's reinforcement learning state space and single-step total reward function, the Q-learning reinforcement learning algorithm is used to solve the robot path planning action sequence through the interaction and iteration between the agent and the environment. 2.The improved state space based signal timing logic reinforcement learning planning method according to claim 1, characterized in that, The process of defining a flag bit for each subtask using the flag bit operator to characterize the current satisfaction level of the subtask, and dynamically updating the flag bit according to the robot's physical state, is represented by the following formula (1): （1） wherein, denotes the size of the flag for the th subtask at time ; denotes the size of the flag for the th subtask at time ; denotes the time interval constant under the time operator; denotes the physical state of the robot at time ; denotes the final time operator; denotes the until time operator; denotes the th subtask; denotes the th subtask corresponding bottom predicate formula. 3.The improved state space based signal timing logic reinforcement learning planning method according to claim 1, characterized in that, The process of comparing the flag values ​​of all subtasks at the current time, extracting the subtask corresponding to the minimum value, and encoding it into a one-hot encoded vector is represented by the following formula (2). （2） wherein, a one-hot encoding representing time instant t the i-th component of the one-hot encoding; a flag bit representing the i-th subtask at time instant t; i represents the i-th subtask, while corresponding to the one-hot encoding as the i-th component; t represents the time instant. 4.The improved state space based signal timing logic reinforcement learning planning method according to claim 1, wherein, The main reward function of the currently active subtask is expressed by the following formula (3): （3） in, Represents the one-hot encoded vector at time t The One component; This represents a weighting coefficient indicating that the robot's robustness to the target task is greater than 0. Indicates the next state after the action is performed. Corresponding subtasks The robustness value of the space robustness value, state One-hot encoded vector With physical location constitute; express The value of the main reward function at time t; t represents time t. This indicates the total number of subtasks.

5. The improved state-space based signal temporal logic reinforcement learning planning method according to claim 1, wherein, The reward shaping function is expressed by the following formula (4): （4） in, Represents a one-hot encoded vector At time t One component; This represents the weighting coefficient for rewarding shaping; Indicates the next state after the action is performed. robustness value; Indicates the current state Corresponding subtasks Spatial robustness value; express The reward shaping function value at time t; t represents time t.

6. The signal temporal logic reinforcement learning planning method based on improved state space according to claim 1, characterized in that, The robot-based reinforcement learning state space and single-step total reward function of S4 employ the Q-learning reinforcement learning algorithm to solve the robot path planning action sequence through the interaction and iteration between the agent and the environment, including: S41. Model the task and abstract it as a Markov decision process; define the state space S, action space A, state transition probabilities and reward function R; create a Q table and set its initial value to 0, and set hyperparameters including: learning rate, discount factor, exploration rate and maximum number of rounds; the hyperparameters remain unchanged after initialization until the end of the iteration. S42. Single-round learning initialization: At the beginning of each training round, the environment is reset to the initial state, the robot returns to the initial position, and the flag bit is reset to 0. The one-hot encoded vector at the initial moment is calculated. S43, Adopt A greedy strategy selects the next action based on the probability of exploration. Randomly select an action, or explore with a probability of 1- Select the current status in the Q table. The action that maximizes the Q value; S44. Execute the selected action, obtain the physical location coordinates at the next moment, calculate the flag bits of each subtask according to the flag bit update rule, and update the one-hot encoding vector to obtain the complete K-MDP state at the next moment. ; S45. Calculate the current single-step total reward based on the updated one-hot encoding vector; S46. Based on the current total reward for a single step, update the Q table using a time-series difference equation; S47. Update the current state to the next state. Determine whether the update state has reached the termination state. If it has not reached the termination state, return to step S43 and continue iterating. If it has reached the termination state, end the current round. S48. After training is complete, for any state The action with the highest Q-value is selected as the optimal strategy learned by the robot.

7. The signal temporal logic reinforcement learning planning method based on improved state space according to claim 5, characterized in that, The use The process of the greedy strategy choosing the next action is represented by the following formula (5): （5） in, Indicates the state Select action Strategies; Indicates the state Select action The corresponding Q value; Indicates the robot's physical location; Indicates the robot's actions; represents the exploration rate (i.e., the probability of randomly selecting an action); t represents the current time. This represents the one-hot encoded vector of the most urgent subtask. The process of updating the Q-table based on the current single-step total reward through a time-series difference equation is represented by the following formula (6): （6） in, Representing state Next action Q value; express Time is determined by physical location The augmented state formed by combining the one-hot encoded K vector; express Actions selected and executed at any time; This represents the one-hot encoded vector of the most urgent subtask. This represents the learning rate used to control the degree of sample estimation updates; express Total reward per step at any given moment; This represents the discount factor used to set the weight of future rewards; Indicates the state at the next moment. Next action Q value; express The augmented state of time; Indicates the state at the next moment. The corresponding action below.

8. A signal temporal logic reinforcement learning planning device based on an improved state space, wherein the improved state space-based signal temporal logic reinforcement learning planning device is used to implement the signal temporal logic reinforcement learning planning method based on an improved state space as described in any one of claims 1-7, characterized in that, The device includes: The acquisition unit is used to acquire the robot path planning task described in natural language; and to convert the robot path planning task into signal-time logic constraints and define multiple sub-tasks using signal-time logic-based methods. The first building unit is used to construct the K-MDP augmented state space model. The signal timing logic constraints are input into the K-MDP augmented state space model. A flag bit is defined for each subtask through the flag bit operator to represent the current satisfaction level of the subtask. The flag bit is dynamically updated according to the physical state of the robot. The flag bit values ​​of all subtasks at the current time are directly compared, the subtask corresponding to the minimum value is extracted, and it is encoded into a one-hot encoded vector. The second construction unit is used to construct the robot's reinforcement learning state space based on the one-hot encoded vector and the robot's physical position; construct the main reward function based on the one-hot encoded vector indicating the currently active subtask and combining the signal temporal logic space robustness of the subtask; construct the reward shaping function using the spatial robustness difference of the currently active subtask at adjacent time steps; and sum the main reward function and the reward shaping function to construct the single-step total reward function. The solver unit is used to solve the robot path planning action sequence based on the robot's reinforcement learning state space and single-step total reward function. It employs the Q-learning reinforcement learning algorithm to solve the robot's path planning action sequence through the interaction and iteration between the agent and the environment.

9. A signal temporal logic reinforcement learning planning device based on an improved state space, characterized in that, The signal temporal logic reinforcement learning planning device based on the improved state space includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images