Double-layer control method and system in dynamic environment based on reinforcement learning

Abstract

The invention discloses a double-layer control method and system in a dynamic environment based on reinforcement learning, a double-layer reinforcement learning architecture is designed for the path planning problem of multi-task operation in a man-machine coexistence environment, and the method comprises the following steps: S1, according to the multi-task operation to be executed by a robot, determining the path of the multi-task operation to be executed by the robot; multi-task atomic proposition is carried out by using security cooperation linear sequential logic, and a corresponding finite state automaton is constructed; s2, according to the observation state of the manually-driven vehicle in the environment, by constructing a dynamic factor discrete motion model, environment uncertainty measurement is completed; s3, performing risk measurement according to the environmental uncertainty measurement; s4, performing path planning based on a double-layer reinforcement learning algorithm; and S5, if the robot encounters operation interruption in the operation process, returning to the step S4, and carrying out risk calculation and path planning again. According to the method, effective combination of local and global optimal paths can be realized, and finally the global optimal multi-task operation path is generated.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology, and in particular to a double-layer control method and system in a dynamic environment based on reinforcement learning. Background Art

[0002] In automated factories and logistics warehousing scenarios, when AGV robots work together with manually driven forklifts, they face vehicle path conflicts and uncertainties brought about by sudden changes in the gate control system status. For example, the random movement of the manual forklift may cause the robot to make emergency avoidance, or the manual forklift may occupy the path, preventing the AGV robot from reaching the next workshop, causing the AGV to interrupt the preset path. As a result, the AGV robot has a low success rate when performing multiple tasks, making it difficult to improve operating efficiency.

[0003] Existing control systems are often based on static environment assumptions or single obstacle avoidance strategies, making them incapable of effectively handling complex dynamic random risks. For example, the A* algorithm is a classic solution for path planning, but it lacks the ability to anticipate global risks and relies solely on local obstacle avoidance, failing to avoid conflicts caused by random worker movements during the planning phase. Reinforcement learning algorithms can effectively address single-task tasks in dynamic scenarios, but when performing multi-task path planning, they typically require defining multiple reward functions and incur high learning costs, requiring further improvement. While real-time path planning algorithms can optimize local obstacle avoidance, they are unable to plan a globally optimal path from a global, multi-task perspective. Furthermore, the constant need for local obstacle avoidance and re-planning of task paths results in excessive computational costs, making it easy to fall into local optimal traps and leading to low operational efficiency. These algorithms struggle to meet the real-time requirements of real-world scenarios and are impractical for logistics warehouses and factory environments where workers operate in the same environment.

[0004] Therefore, there is an urgent need for a control method that is suitable for human-machine collaborative operation scenarios and can face the random distribution of risks in dynamic environments, so as to avoid operation conflicts at the planning level and plan the optimal operation path for multiple tasks, thereby improving the safety and robustness of operations in complex dynamic environments. Summary of the Invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a two-layer control method and system in a dynamic environment based on reinforcement learning, aiming to take into account the human uncertainty in the human-machine collaborative operation environment, provide optimal control strategies for the robot's multi-tasking operations and plan the optimal operation path.

[0006] Technical solution: A two-layer control method in a dynamic environment based on reinforcement learning. In this method, in a dynamic environment where humans and robots coexist, the robot needs to go to I workbenches to collect workpieces and send them to different destinations. For any workbench i, i∈{1,2,…,I}, o i is the position of workbench i, gi The destination of the workpieces collected by workstation i; the robot visits any o i And collect the workpiece of workbench i as the pickup task, and send the workpiece of workbench i to g i For delivery tasks, The time limit for pickup and delivery tasks includes the following steps:

[0007] S1, based on the multi-task operations to be performed by the robot, use safe cooperative linear temporal logic to atomically formulate the multi-tasks and construct the corresponding finite state automaton;

[0008] S2, based on the observed state of the human-driven vehicle in the environment, completes the environmental uncertainty measurement by constructing a discrete motion model of dynamic factors;

[0009] S3, risk measurement based on environmental uncertainty measurement;

[0010] S4, path planning based on a two-layer reinforcement learning algorithm;

[0011] S5: If the robot encounters an interruption during operation, it returns to step S4 to recalculate the risk and plan the path.

[0012] Furthermore, by constructing a dynamic factor discrete motion model, the specific steps to complete the environmental uncertainty measurement are as follows:

[0013] S21, by constructing a discrete motion model of dynamic factors;

[0014] Define the state set of K manually driven vehicles observed at time t as in is the state of vehicle k (k∈{1,…,K}), Indicates the vehicle's position; dire k ∈{←,→} represents its direction of movement;

[0015] The vehicle's action space is {fowrad, wait}. The vehicle selects the fowrad action to move forward through the corridor and the direction of movement remains unchanged during the passage. The wait action is only triggered in emergency avoidance situations. f is the probability of the forklift moving forward, p w Corresponding to the waiting action probability; at time t, the vehicle position is Direction k ←, select forward action, the state at time t+1 is The state transition is The distance between the two states of the vehicle or 2;

[0016] S21, dynamic stochastic risk assessment;

[0017] Assume that each door n has a protection area B n When the robot plans to cross the corridor from door n to the adjacent workshop, its operation process is as follows:

[0018] First, the robot sends a door-opening request when approaching the target door;

[0019] Then, the gate control system controls the protection area B. n Perform real-time risk assessment. If there is no conflict risk in the area, the confirmation command "Y" will be fed back and the door opening operation will be started. On the contrary, when there is potential danger, the door opening will be refused and the rejection command "N" will be returned.

[0020] Furthermore, the protection zone B n The risk assessment mechanism is as follows:

[0021] Each access control n corresponds to the protection area B n It is defined by its surrounding grid set, denoted as:

[0022] B n ={v n 1 ,v n 2 ,…}

[0023] Among them, v n 1 ,v n 2 ,… represents the grid covered by the access control protection area;

[0024] When vehicle k enters the corridor at the initial observation time t0, its state information is expressed as:

[0025]

[0026] in, is the initial position, dire k is the direction of travel;

[0027] The trajectory prediction algorithm is used to simulate the robot's motion trajectory until it leaves the corridor, and finally the grid v∈B is calculated. n Conditional probability of being occupied by a robot The calculation formula is as follows:

[0028]

[0029] Among them, simulation times is the number of simulation predictions, and visited times of v is the number of times v is visited at time t;

[0030] Protection Zone B n The risk assessment formula at any time t is:

[0031]

[0032] Then we get all the * The environmental risk set R isk :

[0033] R isk ={{r(B1,t0+1),…,r(B1,T * )},…,{r(B N ,T * ),…,r(B N ,T * )}}

[0034] Where N is the number of gates, T * Indicates the upper limit of the time domain;

[0035] Thus, the protection zone B is obtained n In the time domain [t0+1,T k ] within a dynamic risk assessment framework, where T k represents the time when robot k leaves the corridor.

[0036] Furthermore, the two-layer reinforcement learning algorithm consists of an upper-layer FSA-RL learning algorithm and a lower-layer fusion environment risk safety path planning. Through a layered architecture, it achieves the coordinated optimization of multi-task global optimal strategies and local risk avoidance paths in a dynamic environment.

[0037] The implementation steps for risk safety path planning in the lower-layer fusion environment are as follows:

[0038] Assume that the robot's environment workshop is a 2D grid map, and the lower MDP learner Defined as:

[0039]

[0040] Among them, S l is a state set; A l =A is the set of robot actions; P l :S l ×A l ×S l →[0,1], is the state transition probability function; R l is the lower layer reward function, s l ,s l ′∈S l , a l ∈A l ; is the initial state, is the target state, input from the upper layer; R isk is the environmental risk set; L is the label function;

[0041] State set S l Defined by a 2D grid graph, containing all grids; the robot starts from a given initial state Start, choose an action a l ∈A l And explore the environment to gain experience; the lower layer reward function R l The expression is as follows:

[0042]

[0043] Among them, goal_reward is a positive real number, which is the reward obtained by the robot when it reaches the target state; step_cost is a negative real number, which represents the cost paid by the robot for each state transition; t′ represents the time when the robot transitions to state s l ′, u(·) is used to calculate the reward brought by environmental uncertainty;

[0044] Introducing uncertainty risk R isk To predict whether the door is likely to be opened, for each door n, we have:

[0045]

[0046] Among them, θ is a threshold; when the risk value of gate n is less than θ, the risk of the gate is low and can be passed; if r(B n ,t)≥θ, it means that passing through the door will bring collision risk, and the door is refused to open;

[0047] u(·) is defined as:

[0048]

[0049] Where c_cost is the collision cost, which is set to a negative real number; Penalty represents the penalty cost the robot receives when the door refuses to open;

[0050] At any time t, the robot is based on the current state s l , adopt greedy strategy to select action a l ; Randomly select an action with probability ε, and select the action with the largest current Q value with probability 1-ε; Then, the robot executes the selected action a l , and observe the transition to the next state s l ’ and the immediate reward r; the robot updates the Q table based on Q learning:

[0051]

[0052] Where α is the learning rate, γ is the discount factor, and r = R l (s l ,a,s l ′);

[0053] The condition for the task to be completed is that the robot reaches the target state within the budget time Each task has a predetermined time budget and represents the time budget for the pickup task, represents the time budget for the delivery task; all tasks must be completed within the budget to be considered successful; if the robot reaches the target state within the budget time, the task completion label L(s l ) returns True; otherwise, returns False;

[0054] After the task is completed, the lower layer learner will combine the path x with L(s l ) Return to the upper learning algorithm;

[0055] The implementation steps of the upper-level FSA-RL learning algorithm are as follows:

[0056] Building an upper-level learning framework The expression is as follows:

[0057]

[0058] Among them, S h =Q represents the state set; A h ={move, wait} represents an action set; P h ∶S h ×A h ×S h →[0,1] represents the state transition probability; δ h ∶S h ×PrP×L(S l )→S h Represents the state transfer function; L:S l →2 PrP represents a label function; PrP represents a set of atomic propositions; R h Represents the upper layer reward function, which is determined by the lower layer MDP; q0 represents the initial state, q F Indicates an accepting state;

[0059] The robot starts from state s h to s′ h Select action a h And perform state transition with probability p(s′ h|s h ,a h ); h = move, the robot moves with probability p(s′ h |s h ,a h ) moves to the next state s' h , and 1-p(s′ h |s h ,a h ) has a probability of staying in s h , s′ h ≠s h ;

[0060] Then the state transition is determined by δ h (·) is defined as:

[0061] s′ h =δ h (s h ,prp,L(s l ))

[0062] Among them, prp represents the atomic proposition of the task; L(s l ) is a label;

[0063] The action set is defined as follows:

[0064] A h ={pick i ,deliver i ,null i}

[0065] Among them, the moving state move is composed of the operation set {pick i ,deliver i}, while the static state stay is represented by null i express;

[0066] Given state s h =q∈Q, let neighbors be the set of all possible transition states of q; for There is a corresponding atomic proposition prp; each atomic proposition prp represents an online pickup task τ i Or delivery task τ I+i ; Define f(·) as the mapping from the atomic proposition prp to the corresponding task position, the expression is as follows:

[0067] s l =f(prp)

[0068] Among them, s lSelect the next target position corresponding to the atomic proposition prp for the robot, s l ∈{o1,…,o I ,g1,…,g I};o i is the position of workbench i, g i is the destination of the workpieces collected by workstation i, i∈{1,2,…,I}, I is the total number of workstations;

[0069] definition The target position of the previous state transition; the robot sends it to the lower layer Call the lower layer to search for the local optimal path and obtain the label L(s l );

[0070] If the task τ i or τ i+1 Completed online, in the lower layer, L(s l ) will return True, the robot will start from s h =q transfers to s h ′=q′, written as δ h (q,prp,L(s l )=True)→q′; otherwise, L(s l )=False,δ h (q,prp,L(s l )=False)→q; and return the local path x q,q′ .

[0071] Furthermore, the upper reward function R h The expression is as follows:

[0072]

[0073] Among them, rE is the reward obtained by successful state transfer, which is a positive real number; It is returned from the lower layer h to s′ h Path length; step cost is the cost of each move, a negative real number.

[0074] A two-layer control system in a dynamic environment based on reinforcement learning, used to execute any of the above two-layer control methods in a dynamic environment based on reinforcement learning, comprising:

[0075] Based on the scLTL search state space construction module, it is used to construct the upper-level algorithm search state space; first, the job tasks are atomically propositioned, and the atomic proposition set PrP is constructed. All tasks are written into scLTL formulas in the form of atomic propositions. Finally, a finite state automaton FSA is constructed to form the upper search state space;

[0076] Environmental uncertainty measurement module, used to measure the risks caused by dynamic factors in the environment;

[0077] A path planning module based on two-layer reinforcement learning is used to plan the global optimal operation path of the robot in a dynamic environment. The upper layer is the global optimal strategy learning based on scLTL, and the lower layer is the local optimal path planning in a dynamic environment.

[0078] Compared with the prior art, the present invention has the following significant effects:

[0079] 1. The control system of this invention integrates an environmental risk model during the planning phase, enabling the prediction of conflict areas and proactive avoidance, thus reducing the computational burden of real-time obstacle avoidance. This global risk prediction capability enables the robot to better adapt to complex situations such as interference from operating vehicles and uncertainty in the door control system in dynamic environments.

[0080] 2. The control method of the present invention realizes efficient path optimization of robot multi-task operations in dynamic environments by constructing a two-layer reinforcement learning architecture. The upper layer realizes task control based on scLTL (safe co-safe Linear Temporal Logic) and constructs an upper-layer search space to achieve global optimal path planning; the lower layer generates a local optimal operation path based on the Markov decision process (MDP) and the dynamic risk fusion of the environment to avoid risks, thereby realizing the effective combination of local and global optimal paths, and finally generating a global optimal multi-task operation path. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 This is an architectural diagram of the control system of the present invention;

[0082] Figure 2 Schematic diagram of building modules for scLTL-based search state space;

[0083] Figure 3 This is a schematic diagram of the environmental uncertainty measurement module;

[0084] Figure 4 Schematic diagram of the path planning module based on two-layer reinforcement learning;

[0085] Figure 5 This is a schematic diagram of the motion model and state transition of a forklift;

[0086] Figure 6 This is a schematic diagram of the interaction between the robot and the door control system;

[0087] Figure 7 is a flow chart of the control method of the present invention;

[0088] Figure 8 Schematic diagram of the robot's starting position, task settings, and the forklift's observation position;

[0089] Figure 9 (a) is a schematic diagram of the path generated by the traditional reinforcement learning algorithm, and (b) is a schematic diagram of the path generated by the algorithm proposed in this invention. DETAILED DESCRIPTION

[0090] The present invention will be described in further detail below with reference to the accompanying drawings and specific implementations.

[0091] like Figure 1 As shown, a mobile control system for a mobile robot performing multi-tasking operations in a dynamic environment taking into account environmental risks provided by the present invention includes a state space construction module based on scLTL (syntactically co-safe linear temporal logic), an environmental uncertainty measurement module, and a path planning module based on two-layer reinforcement learning;

[0092] like Figure 2 As shown in the figure, it is a module for building the search state space based on scLTL, which is used to build the search state space of the upper algorithm. Specifically, the task is firstly transformed into an atomic proposition, and an atomic proposition set PrP is constructed. All tasks are written into the scLTL formula in the form of atomic propositions. Finally, a finite state automaton FSA is constructed to form the upper search state space.

[0093] like Figure 3 As shown in Figure 1, it is an environmental uncertainty measurement module, which is used to measure the risks caused by dynamic factors in the environment.

[0094] like Figure 4 As shown in the figure, it is a path planning module based on two-layer reinforcement learning, which is used to plan the global optimal operation path of the robot in a dynamic environment.

[0095] In the embodiment, the robot is mainly considered to work in a dynamic environment. The robot needs to go to I workbenches to collect workpieces and send them to different destinations. For any workbench i, i∈{1,2,…,I}, o i For its position, g i is the destination of the workpieces collected by workstation i.

[0096] For any workstation i, it is defined as follows:

[0097] Robot access to any o i And collecting the workpiece on workbench i is the picking task;

[0098] Send the workpiece of table i to g i For delivery tasks;

[0099] They are the time limits for pickup and delivery tasks respectively.

[0100] Therefore, I workstations define 2I tasks. The task state at any time t is defined as τ t ={τ1,…,τ i ,…,τ I ,τ I+1 ,…,τ I+i ,…,τ 2I}, where τ i is the pickup task status, τ I+i is any state of sending documents, and τ i ,τ I+i ∈{False,True}, where False means the task is not completed and True means the task is completed. The initial state of the robot task is τ 0 ={False,…,False,…,False}. When the robot completes the task of picking up the item from workstation i at time t, its task state becomes τ t ={False,…,True,…,False}; when all tasks are completed, the task status becomes {True,…,True,…,True}. The definition is as follows:

[0101] If a task is completed within the specified time, it is called "online"; otherwise, it is called "offline";

[0102] If all tasks are completed online, the job is considered successful; otherwise, the job fails.

[0103] Dynamic uncertainty in the environment is caused by random human behavior. For example, a human-driven forklift is used as an example. The robot and the human-driven vehicle operate in the same environment. The human-driven vehicle needs to pass through a corridor, which is shared by the robot and the human-driven vehicle. The robot passes through the corridor to another workshop to perform a delivery task. There are doors for the robot to pass through. Each door n (n∈{1,…,N}) has two states: "closed" and "open", which are controlled by the door control system. Definition is the state of gate n at time t, and is expressed as follows:

[0104]

[0105] like Figure 6As shown in Figure 1, when the robot needs to pass through door n, it sends a door opening request to the door control system. If door n can be opened, the door control system replies to the robot with an approval and opens the door; otherwise, the door control system replies to the robot with a rejection command.

[0106] A = {forward, backward, left, right, wait} is the robot's action set.

[0107] This invention proposes a two-layer control method in a dynamic environment based on reinforcement learning. The core of the method is a two-layer control algorithm in a dynamic environment based on reinforcement learning: the upper layer is global optimal strategy learning based on scLTL, and the lower layer is local optimal path planning in a dynamic environment. The purpose of this invention is to avoid risks in the environment at the planning level, generate the optimal safe path for mobile robots to perform multi-task operations in a dynamic environment, and avoid conflicts with operating vehicles in advance. The flow chart is as follows Figure 7 The detailed implementation steps are as follows:

[0108] Step 1: Initialize various parameters;

[0109] Including environmental map information G=<V,Edge> and task information; task information includes the workbench location {o1,…,o I} and the delivery position {g1,…,g I}.

[0110] Step 2: Based on the tasks to be performed by the robot, scLTL is used to atomically formulate multi-task propositions and construct the corresponding finite state automaton (FSA).

[0111] Step 21, constructing a search state space based on scLCL;

[0112] Step 211, converting the multi-task atomic propositions;

[0113] This invention mainly uses scLTL logic specifications to control the robot's multi-tasks, writing multiple tasks into a formula for unified control. The technical principles are as follows:

[0114] First, turn the multi-task into atomic propositions and organize the expressions.

[0115]

[0116] Where T represents the Boolean operator "true"; PrP is a set of atomic propositions; Represents an atomic proposition; X(next) represents the next step; express The next state is true; F (future) represents the future; express The future is real; U(until): express True until satisfy.

[0117] Through the above logical expression, the atomic propositions of all tasks can be written into formula (2) for unified control and management.

[0118] According to the above 2I tasks, the expression of the set of atomic propositions PrP is defined as follows:

[0119]

[0120] PrP i ={null i ,pick i ,deliver i}

[0121] Among them, PrP i is the set of atomic propositions corresponding to workbench i, null i Indicates that the robot has not completed the task of picking up the item at workbench i; pick i Indicates that the robot completes the task of picking up items at workstation i; deliver i Indicates that the robot completes the delivery task of workbench i.

[0122] Step 212, constructing the scLTL formula;

[0123] All tasks can be written into scLTL formulas through the above atomic propositions The specific definitions are as follows:

[0124]

[0125] Among them, f pick i Indicates that the robot will complete the online pickup task of workstation i at a certain moment in the future;

[0126] F deliver i Indicates that the robot will complete the online delivery task of workstation i at a certain moment in the future;

[0127] This means that the delivery task will not be executed until the online pickup task is completed.

[0128] Step 213: Generate a finite state automaton (FSA)

[0129] In scLTL formula (3), Use existing off-the-shelf computing tools to convert into finite state automata The expression is as follows:

[0130]

[0131] Where Q represents a finite set of task states, PrP represents a set of atomic propositions, and δ represents Q×2 PrP →Q state transition function, q0 represents the initial state, q F represents a finite set of accepting states.

[0132] By constructing a finite state automaton Thus constructing a new search state space.

[0133] Step three, observe environmental information;

[0134] By constructing a discrete motion model of dynamic factors, the environmental uncertainty measurement is completed;

[0135] The environmental uncertainty in this invention mainly comes from the random movement of workers and work vehicles. We use the forklift in the automated factory scenario as an example for analysis. The set of K forklift states observed at time t is defined as in is the state of forklift k (k∈{1,…,K}), that is Indicates the position of the forklift; dire k ∈{←,→} represents the direction of the forklift's movement. The forklift's action space is {fowrad,wait}. Normally, the forklift chooses the fowrad action to move forward through the corridor and its direction of movement remains unchanged during the passage; the wait action is only triggered in special circumstances such as emergency avoidance. f is the probability of the forklift moving forward, p w Corresponding to the waiting action probability. Considering the driver's autonomous adjustment of the driving speed, the actual displacement of the forklift presents an uncertain characteristic. Specifically, at time t, the forklift position state is observed to be Direction k ←, select forward action, the state at time t+1 is The state transition is Note that this state transition is uncertain. Specifically, due to the influence of speed fluctuations, the forklift may advance 1 or 2 unit distances (usually one grid or two grids) at the next moment, that is, the distance between the two states or 2, such as Figure 5 shown.

[0136] Step 4: Conduct risk measurement based on environmental uncertainty measurement;

[0137] Step 41, dynamic random risk assessment;

[0138] Each door n has a protection zone B n Only robots need to go through the door from one workshop to another. Human-driven forklifts only pass through the corridor and pass in front of the door. When the robot plans to go through the corridor from door n to the adjacent workshop, its operation process is as follows:

[0139] First, the robot sends a door-opening request when approaching the target door;

[0140] Then, the gate control system controls the protection area B. n Perform real-time risk assessment. If there is no conflict risk in the area (such as no moving forklift), the confirmation instruction "Y" is fed back and the door opening operation is started. On the contrary, when there is a potential danger (such as a forklift entering the area around door n in the embodiment), the door is refused to open and the rejection instruction "N" is returned.

[0141] Protection Zone B n The risk assessment mechanism is as follows:

[0142] Each access control n corresponds to the protection area B n Defined by its surrounding grid set, denoted as B n ={v n 1 ,v n 2 ,…}. Among them, v n 1 ,v n 2 ,…indicates the grids covered by the access control protection area.

[0143] When forklift k enters the corridor at the initial observation time t0, its state information can be expressed as (in is the initial position, dire k The trajectory prediction algorithm is used to simulate the forklift's trajectory until it leaves the corridor, and finally the grid v∈B is calculated. n Conditional probability of being occupied by a forklift The specific calculation formula is as follows:

[0144]

[0145] Among them, simulation times is the number of simulation predictions, and visited times of v is the number of times v is visited at time t.

[0146] According to the conditional probability that each grid v is occupied by a forklift Get protection zone B n In the time domain [t0+1,Tk ] within a dynamic risk assessment framework, where T k The time when forklift k leaves the corridor. n The risk assessment formula at any time t is:

[0147]

[0148] Step 42, environmental risk aggregation;

[0149] By calculating formula (6), we can get all the * The environmental risk set R isk ={{r(B1,t0+1),…,r(B1,T * )},…,{r(B N ,T * ),…,r(B N ,T * )}}, where N is the number of gates, T * Indicates the upper limit of the time domain (i.e. the time when the last forklift leaves).

[0150] Therefore, the uncertainty measure can fully quantify the B n In [t0+1,T * ] dynamic risks and provide a quantitative basis for security decision-making.

[0151] Step 5: Path planning based on a two-layer reinforcement learning algorithm;

[0152] The two-layer reinforcement learning algorithm consists of an upper-layer FSA-RL learning algorithm and a lower-layer fusion environment risk safety path planning. Through a layered architecture, it achieves the coordinated optimization of multi-task global optimal strategies and local risk avoidance paths in a dynamic environment.

[0153] Step 51: planning a risk safety path for the lower-layer fusion environment;

[0154] Input: Map information G =<V,Edge> , initial state Target state Where V is the 2D map grid set and Edge is the edge set.

[0155] Output: path x, task completion label L(s l ).

[0156] The lower layer is the fusion environment risk R isk The MDP learner is used for local optimal path planning. In the lower layer learning, the robot's environment workshop is a 2D grid map, and the lower layer MDP learner It can be defined as:

[0157]

[0158] Among them, S l is a state set; A l =A is the set of robot actions; P l :S l ×A l ×S l →[0,1], is the state transition probability function; R l is the lower layer reward function, s l ,s l ′∈S l , a l ∈A l ; Initial state Target state Input from the upper layer; R isk is the environmental risk set; L is the label function.

[0159] State set S l It is defined by a 2D grid map, which contains all grids. The robot starts from a given initial state. Starting, the robot chooses an action a l ∈A l And explore the environment to gain experience.

[0160] In order to improve learning efficiency in an uncertain environment, this embodiment adjusts the lower layer reward function R l The improved reward function is as follows:

[0161]

[0162] Among them, goal_reward is a positive real number, which is the reward obtained by the robot when it reaches the target state; step_cost is a negative real number, which represents the cost paid by the robot for each state transition; t′ represents the time when the robot transitions to state s l ′, u(·) is used to calculate the reward caused by environmental uncertainty. When the robot approaches a door and sends a request, the door control system may return a message "Y" to indicate opening the door, or "N" to indicate rejection. However, whether the door is available is unknown. Therefore, this embodiment introduces the uncertainty risk R isk To predict whether the door is likely to open. For each door n, we have:

[0163]

[0164] Where θ is a threshold. As shown in formula (9), when the risk value r(B n ,t)<θ, the risk of this gate is low and it can be passed. n,t)≥θ, it means that passing through the door will bring collision risk, and the door is refused to open. u(·) is defined as:

[0165]

[0166] Among them, c_cost is the collision cost, which is set to a negative real number; Penalty represents the penalty cost obtained by the robot when the door refuses to open.

[0167] At any time t, the robot is based on the current state s l , adopt greedy strategy to select action a l . Randomly select an action (exploration) with probability ε, and select the action with the largest current Q value with probability 1-ε. Then, the robot executes the selected action a l , and observe the transition to the next state s l ’ and the immediate reward r. The robot updates the Q table based on Q learning:

[0168]

[0169] Where α is the learning rate, γ is the discount factor, and r = R l (s l ,a,s l Through continuous exploration, the robot learns the optimal strategy to complete tasks safely and efficiently in the environment, and ultimately generates a safe planned path for the robot.

[0170] The condition for the task to be completed is that the robot reaches the target state within the budget time (Budget) Each task has a predetermined time budget (Pickup Task) and (Delivery task). All tasks must be completed within the budget to be considered successful. If the robot reaches the target state within the budget time, the task completion label L(s l ) returns True; otherwise, returns False. The specific definitions are as follows:

[0171]

[0172] After the task is completed, the lower layer learner will combine the path x with L(s l ) returns to the upper learning algorithm.

[0173] Table 1 Specific steps for risk safety path planning in the lower-layer fusion environment

[0174]

[0175] Step 52, upper layer FSA-RL learning algorithm;

[0176] enter:

[0177] Output: Path χ for executing multiple tasks

[0178] The upper layer FSA-RL is a reinforcement learning search algorithm based on the fusion of FSA and MDP, which plans the optimal path for the robot to perform multiple tasks.

[0179] based on This embodiment builds an upper-level learning framework The expression is as follows:

[0180]

[0181] Among them, S h =Q represents the state set; A h ={move, wait} represents an action set; P h ∶S h ×A h ×S h →[0,1] represents the state transition probability; δ h ∶S h ×PrP×L(S l )→S h Represents the state transfer function; L:S l →2 PrP represents a label function; PrP represents a set of atomic propositions; R h Represents the upper-level algorithm reward function, which is determined by the lower-level MDP; q0 represents the initial state, q F Indicates acceptance status.

[0182] The robot starts from state s h to s′ h Select action a h And perform state transition with probability p(s′ h |s h ,a h ). h = move, the robot moves with probability p(s′ h |s h ,a h ) Move to the next state s′ h (s′ h ≠s h ), and possibly 1-p(s′ h |s h ,a h ) has a probability of staying in s h .

[0183] Specifically, the state transition is determined by δ h(·) is defined as:

[0184] s′ h =δ h (s h ,prp,L(s l )) (13)

[0185] where prp∈PrP i Represents the atomic proposition of the task, L(s l ) is a label.

[0186] The action set is defined as follows:

[0187] A h ={pick i ,deliver i ,null i}, (14)

[0188] Among them, the moving state move is composed of the operation set {pick i ,deliver i}, while the static state stay is represented by null i express.

[0189] δ h (·)and The state transition δ(·) in is different. It should be noted that δ(·) is a deterministic transition, δ h (·) is an uncertain transfer, whose uncertainty mainly depends on the fact that the label L(·) is unknown, so its state transfer is uncertain.

[0190] For δ(·), given a state q and the corresponding label L(·), when the next state transition is q′, it is written as q′←δ(q,L(·)). Given a state s h =q∈Q, let neighbors be the set of all possible transition states of q. There is a corresponding atomic proposition prp. Each prp represents an online pickup task τ i Or delivery task τ I+i ; Define f(·) as the mapping from the atomic proposition prp to the corresponding task position, the expression is as follows:

[0191] s l =f(prp) (15)

[0192] Among them, s l ∈{o1,…,o I ,g1,…,g I}; Once the robot chooses to visit a neighbor, that is, the robot selects the atomic proposition prp, the corresponding next target position is sl ;

[0193] definition The target position of the previous state transition, that is, the current position of the robot. The robot sends ( s l are the starting points in the lower MDP respectively With the goal ), call the lower layer to search for the local optimal path and obtain the label L(s l ).

[0194] If the task τ i or τ i+1 Completed online, that is, the vehicle arrives at s l In the lower layer, L(s l ) will return True, the robot will start from s h =q transfers to s h ′=q′, written as δ h (q,prp,L(s l )=True)→q′. Otherwise, L(s l )=False,δ h (q,prp,L(s l )=False)→q.

[0195] The upper model calls the lower model to learn the local control strategy, and the lower model returns the label L(s l ). At the same time, if the corresponding task is completed online, the local path x is returned. q,q′ .

[0196] scLTL formula Provides the definition of global reward. When satisfied, the robot receives a global reward, defined as:

[0197] R h (s h ,a h ,s′ h )=E,if s′ h =q F (16)

[0198] Among them, E is a learning method that satisfies the scLTL formula The reward obtained by the global strategy is a positive real number.

[0199] In addition, in order to accelerate the learning process, the present invention redesigns the upper reward function R h , the expression is as follows:

[0200]

[0201] Among them, rE is the reward obtained by successful state transfer, which is a positive real number; It is returned from the lower layer h to s′ h The path length, step cost is the cost of each move, a negative real number.

[0202] Table 2 Specific execution steps of the upper-level FSA-RL learning algorithm

[0203]

[0204] Step 6: The robot performs multi-tasking operations. If the operation is interrupted during the operation, it returns to step 4 and recalculates the risk and plans the path.

[0205] Step 7: Complete the path planning and the operation is completed.

[0206] by Figure 8 For this example, the robot's starting position is s0 = (0,0), and the worktable's positions are {o1,o2,o3} = {(6,3),(13,3),(13,7)}. The corresponding delivery targets are {g1,g2,g3} = {(3,14),(17,15),(14,19)}. Two forklifts are observed, located at (6,10) and (18,11), respectively. Their movement directions are indicated by the red arrows. Assume that the forklifts begin moving at t = 30.

[0207] Figure 9 Figure (a) shows a path generated by a traditional reinforcement learning algorithm. After completing the pickup task, the robot passes through Gates 3 and 6 and enters another workshop to deliver the item. However, this path is not optimal in a dynamic environment. When passing through Gates 3 and 6, the robot may collide with a human-driven forklift. In 100 simulations, the average number of conflicts was 0.26, and the task success rate was 74%.

[0208] Figure 9 (b) shows the path generated by the dual-layer reinforcement learning algorithm proposed in this paper. After completing the pickup task, the robot passes through Gates 1 and 4 and enters another workshop to deliver the package. This path is equivalent to the path generated by a traditional reinforcement learning algorithm and avoids the risk of collision with a human-driven forklift. In 100 simulations, the average number of collisions was 0, indicating no collisions, resulting in a 100% success rate.

Claims

1. A two-layer control method in a dynamic environment based on reinforcement learning, where: In a dynamic environment where humans and robots coexist, the robot needs to go to I workbenches to collect workpieces and send them to different destinations. For any workbench i, i∈{1,2,…,I}, o i is the position of workbench i, g i The destination of the workpieces collected by workstation i; the robot visits any o i And collect the workpiece of workbench i as the pickup task, and send the workpiece of workbench i to g i For delivery tasks, The time limit for the pickup task and the delivery task is characterized by including the following steps: S1, based on the multi-task operations to be performed by the robot, use safe cooperative linear temporal logic to atomically formulate the multi-tasks and construct the corresponding finite state automaton; S2, based on the observed state of the human-driven vehicle in the environment, completes the environmental uncertainty measurement by constructing a discrete motion model of dynamic factors; S3, risk measurement based on environmental uncertainty measurement; S4, path planning based on a two-layer reinforcement learning algorithm; S5: If the robot encounters an interruption during operation, it returns to step S4 to recalculate the risk and plan the path.

2. The two-layer control method in a dynamic environment based on reinforcement learning according to claim 1 is characterized in that: By constructing a dynamic factor discrete motion model, the specific steps to complete the environmental uncertainty measurement are as follows: S21, by constructing a discrete motion model of dynamic factors; Define the state set of K manually driven vehicles observed at time t as in is the state of vehicle k (k∈{1,…,K}), Indicates the vehicle's position; dire k ∈{←,→} represents its direction of movement; The vehicle's action space is {fowrad, wait}. The vehicle selects the fowrad action to move forward through the corridor and the direction of movement remains unchanged during the passage. The wait action is only triggered in emergency avoidance situations. f is the probability of the vehicle moving forward, p w Corresponding to the waiting action probability; at time t, the vehicle position is Direction k ←, select forward action, the state at time t+1 is The state transition is The distance between the two states of the vehicle or 2; S21, dynamic stochastic risk assessment; Assume that each door n has a protection area B n When the robot plans to cross the corridor from door n to the adjacent workshop, its operation process is as follows: First, the robot sends a door-opening request when approaching the target door; Then, the gate control system controls the protection area B. n Perform real-time risk assessment. If there is no conflict risk in the area, the confirmation command "Y" is fed back and the door opening operation is initiated. Conversely, if there is potential danger, the door opening is rejected and the rejection command "N" is returned.

3. The two-layer control method in a dynamic environment based on reinforcement learning according to claim 2 is characterized in that: The protection zone B n The risk assessment mechanism is as follows: Each access control n corresponds to the protection area B n It is defined by its surrounding grid set, denoted as: B n ={v n 1 ,v n 2 ,…} Among them, v n 1 ,v n 2 ,… represents the grid covered by the access control protection area; When vehicle k enters the corridor at the initial observation time t0, its state information is expressed as: in, is the initial position, dire k is the direction of travel; The trajectory prediction algorithm is used to simulate the robot's motion trajectory until it leaves the corridor, and finally the grid v∈B is calculated. n Conditional probability p of being occupied by a robot k,t (v|Z t0 ), the calculation formula is as follows: Among them, simulation times is the number of simulation predictions, visitedtimes of v is the number of times v is visited at time t; Protection Zone B n The risk assessment formula at any time t is: Then we get all the * The environmental risk set R isk : R isk ={{r(B1,t0+1),…,r(B1,T * )},…,{r(B N ,T * ),…,r(B N ,T * )}} Where N is the number of gates, T * Indicates the upper limit of the time domain; Thus, the protection zone B is obtained n In the time domain [t0+1,T k ] within a dynamic risk assessment framework, where T k represents the time when robot k leaves the corridor.

4. The two-layer control method in a dynamic environment based on reinforcement learning according to claim 3 is characterized in that: The two-layer reinforcement learning algorithm consists of an upper-layer FSA-RL learning algorithm and a lower-layer fusion environment risk safety path planning. Through a layered architecture, it achieves the coordinated optimization of multi-task global optimal strategies and local risk avoidance paths in a dynamic environment. The implementation steps for risk safety path planning in the lower-layer fusion environment are as follows: Assume that the robot's environment workshop is a 2D grid map, and the lower MDP learner Defined as: Among them, S l is a state set; A l =A is the set of robot actions; P l :S l ×A l ×S l →[0,1], is the state transition probability function; R l is the lower layer reward function, is the initial state, is the target state, input from the upper layer; R isk is the environmental risk set; L is the label function; State set S l Defined by a 2D grid graph, containing all grids; the robot starts from a given initial state Start, choose an action a l ∈A l And explore the environment to gain experience; the lower layer reward function R l The expression is as follows: Among them, goal_reward is a positive real number, which is the reward obtained by the robot when it reaches the target state; step_cost is a negative real number, which represents the cost paid by the robot for each state transition; t′ represents the time when the robot transitions to state s l ′, u(·) is used to calculate the reward brought by environmental uncertainty; Introducing uncertainty risk R isk To predict whether the door is likely to be opened, for each door n, we have: Among them, θ is a threshold; when the risk value of gate n is less than θ, the risk of the gate is low and can be passed; if r(B n ,t)≥θ, it means that passing through the door will bring collision risk, and the door is refused to open; u(·) is defined as: Where c_cost is the collision cost, which is set to a negative real number; Penalty represents the penalty cost the robot receives when the door refuses to open; At any time t, the robot is based on the current state s l , adopt greedy strategy to select action a l ; Randomly select an action with probability ε, and select the action with the largest current Q value with probability 1-ε; Then, the robot executes the selected action a l , and observe the transition to the next state r l ’ and the immediate reward r; the robot updates the Q table based on Q learning: Where α is the learning rate, γ is the discount factor, and r = R l (s l ,a,s l ′); The condition for the task to be completed is that the robot reaches the target state within the budget time Each task has a predetermined time budget and represents the time budget for the pickup task, represents the time budget for the delivery task; all tasks must be completed within the budget to be considered successful; if the robot reaches the target state within the budget time, the task completion label L(s l ) returns True; otherwise, returns False; After the task is completed, the lower layer learner will combine the path x with L(s l ) Return to the upper learning algorithm; The implementation steps of the upper-level FSA-RL learning algorithm are as follows: Building an upper-level learning framework The expression is as follows: Among them, S h =Q represents the state set; A h ={move, wait} represents an action set; P h ∶S h ×A h ×S h →[0,1] represents the state transition probability; δ h ∶S h ×PrP×L(S l )→S h Represents the state transfer function; L:S l →2 PrP represents a label function; PrP represents a set of atomic propositions; R h Represents the upper layer reward function, which is determined by the lower layer MDP; q0 represents the initial state, q F Indicates an accepting state; The robot starts from state s h to s′ h Select action a h And perform state transition with probability p(s′ h |s h ,a h ); h = move, the robot moves with probability p(s′ h |s h ,a h ) Move to the next state s′ h , and 1-p(s′ h |s h ,a h ) has a probability of staying in s h , s′ h ≠s h ; Then the state transition is determined by δ h (·) is defined as: s′ h =δ h (s h ,prp,L(s l )) Among them, prp represents the atomic proposition of the task; L(s l ) is a label; The action set is defined as follows: A h ={pick i ,deliver i ,null i } Among them, the moving state move is composed of the operation set {pick i ,deliver i }, while the static state stay is represented by null i express; Given state s h =q∈Q, let neighbors be the set of all possible transition states of q; for There is a corresponding atomic proposition prp; each atomic proposition prp represents an online pickup task τ i Or delivery task τ I+i ; Define f(·) as the mapping from the atomic proposition prp to the corresponding task position, the expression is as follows: s l =f(prp) Among them, s l Select the next target position corresponding to the atomic proposition prp for the robot, s l ∈{o1,…,o I ,g1,…,g I };o i is the position of workbench i, g i is the destination of the workpieces collected by workstation i, i∈{1,2,…,I}, I is the total number of workstations; definition The target position of the previous state transition; the robot sends it to the lower layer Call the lower layer to search for the local optimal path and obtain the label L(s l ); If the task τ i or τ i+1 Completed online, in the lower layer, L(s l ) will return True, the robot will start from s h =q transfers to s h ′=q′, written as δ h (q,prp,L(s l )=True)→q′; otherwise, L(s l )=False,δ h (q,prp,L(s l )=False)→q; and return the local path x q,q′ .

5. The two-layer control method in a dynamic environment based on reinforcement learning according to claim 4 is characterized in that: Upper layer reward function R h The expression is as follows: Among them, rE is the reward obtained by successful state transfer, which is a positive real number; It is returned from the lower layer h to s′ h Path length; step cost is the cost of each move, a negative real number.

6. A two-layer control system in a dynamic environment based on reinforcement learning, used to execute the two-layer control method in a dynamic environment based on reinforcement learning according to any one of claims 1 to 5, characterized in that: include: Based on the scLTL search state space construction module, it is used to construct the upper-level algorithm search state space; first, the job tasks are atomically propositioned, and the atomic proposition set PrP is constructed. All tasks are written into scLTL formulas in the form of atomic propositions. Finally, a finite state automaton FSA is constructed to form the upper search state space; Environmental uncertainty measurement module, used to measure the risks caused by dynamic factors in the environment; A path planning module based on two-layer reinforcement learning is used to plan the global optimal operation path of the robot in a dynamic environment. The upper layer is the global optimal strategy learning based on scLTL, and the lower layer is the local optimal path planning in a dynamic environment.

Images