Narrow environment robot autonomous escape system and method based on end-to-end reinforcement learning

Abstract

The invention provides a narrow environment robot autonomous escape system and method based on end-to-end reinforcement learning, and the system comprises an end-to-end reinforcement learning module which carries out the reinforcement learning of an action strategy of a robot in an environment according to the real-time scanning data of a radar; and the action mask mechanism module is used for optimizing the action strategy of the robot and avoiding an invalid or collision action for getting out of trouble in a narrow environment. According to the method, an end-to-end reinforcement learning architecture is adopted, real-time decision directly based on sensor input can be realized, and the technical problem of poor dynamic environment adaptability caused by dependence on accurate environment modeling in a traditional method is solved. An efficient action mask mechanism is adopted, safe and reliable real-time obstacle avoidance can be achieved, and the technical problems of invalid exploration and collision risks caused by a high-dimensional action space are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot autonomous navigation and reinforcement learning, in particular to a narrow environment robot autonomous escape system and method based on end-to-end reinforcement learning. BACKGROUND

[0002] The application of existing robot autonomous escape technology in narrow environments has significant technical bottlenecks, mainly in the following two aspects:

[0003] Although traditional rule-based path planning methods perform well in structured environments, they face multiple challenges in actual narrow and complex scenes such as homes. First, they are highly dependent on accurate environment modeling and static maps, and cannot effectively deal with dynamic obstacles such as moving furniture, suddenly appearing pets, etc. Second, in narrow environments, due to the nonholonomic constraints of robots, especially the motion limitations of differential drive models, the paths generated by algorithms are often theoretically feasible but can easily cause collisions or get stuck in actual execution. More seriously, when encountering narrow passages with sizes smaller than the robot's circumscribed circle radius, the failure rate of these methods will rise sharply.

[0004] Although reinforcement learning-based navigation methods have the advantage of environmental adaptation, they still have obvious shortcomings in practical applications. The most prominent problem is low training efficiency, which is mainly due to two factors: one is the sparsity of rewards, in long-distance navigation tasks, robots have difficulty obtaining effective learning signals; the second is that there are a large number of invalid or even dangerous action options in the high-dimensional action space (continuous space containing angular velocity and linear velocity), leading to low exploration efficiency. In addition, these methods often exhibit repeated oscillation or get stuck in local optima when encountering long corridors, sparse obstacles, and other specific scenarios.

[0005] As can be seen, the current mainstream autonomous escape methods have low overall success rate in complex scenes containing narrow exits, long corridors, etc. This performance bottleneck seriously restricts the practicality and reliability of household cleaning robots in real-world home environments.

[0006] Therefore, there is an urgent need in the industry to develop a new type of end-to-end, lightweight autonomous escape navigation solution that can significantly improve the robot's autonomous escape ability in narrow and complex environments while ensuring real-time performance.

[0007] Through retrieval, a robot escape processing method is disclosed in Chinese patent application No. 202410226556.6. In this method, the robot obtains the current state of the robot when performing a task; in response to the current state being a trapped state, the robot determines an escape strategy; and the robot uses the escape strategy to escape to continue performing the task. However, it needs to rely on accurate environment modeling and also has invalid exploration. SUMMARY

[0008] In view of the defects in the prior art, the purpose of the present application is to provide an end-to-end reinforcement learning-based robot autonomous escape system and method in a narrow environment.

[0009] In a first aspect, the present application provides an end-to-end reinforcement learning-based robot autonomous escape system in a narrow environment, comprising:

[0010] An end-to-end reinforcement learning module learns the robot's action policy in the environment according to real-time radar scanning data;

[0011] An action mask mechanism module optimizes the robot's action policy to avoid invalid or collision actions in the escape from the narrow environment.

[0012] Optionally, the end-to-end reinforcement learning module comprises:

[0013] A state representation submodule obtains lidar input data, target position, and robot action distribution, and respectively performs multi-layer perception encoding to obtain laser encoding, target position encoding, and robot action distribution encoding, which together form a multi-dimensional state feature vector for reinforcement learning;

[0014] An action space design submodule comprises a discretized action set for controlling robot motion;

[0015] A reinforcement learning network submodule learns the robot's action policy in the discretized action set based on a Soft Actor-Critic framework, combining a Transformer encoder and a multi-layer perception decoder through the multi-dimensional state feature vector.

[0016] Optionally, the discretized action set comprises 42 discrete actions, which are two actions of rotating in place, 4*10 actions of circular motion, where the number 4 represents left front, right front, left rear, and right rear, and the number 10 represents 10 different curvatures.

[0017] Optionally, the action mask mechanism module comprises:

[0018] An action space dimension reduction submodule maps a two-dimensional action space composed of angular velocity and linear velocity to a one-dimensional linear space through a fixed turning radius to generate 42 different one-dimensional discrete actions;

[0019] A collision pre-detection submodule compares the radar real-time scanning data with the boundary points of the motion trajectory formed by the action policy of the reinforcement learning network submodule under the one-dimensional discrete action premise to determine whether the current action policy has a collision risk, determine a maximum collision-free step, form an action mask, and clip an effective action policy.

[0020] Optionally, the method further comprises a hybrid training strategy module, comprising:

[0021] An A* algorithm guiding mechanism submodule uses the A* algorithm to generate a collision-free global path in the early stage of training, and calculates action instructions through inverse kinematics to guide the robot to perform safe actions.

[0022] A two-stage curriculum learning submodule ensures the generalization ability of the robot in the later stage of training through the training mode of a fixed target stage and a random target stage.

[0023] Optionally, the A* algorithm guiding mechanism submodule comprises:

[0024] An input unit inputs a grid map and dilates the grid map to obtain a dilated grid map;

[0025] A planning unit calculates a coordinate sequence of a feasible path based on the dilated grid map through the A* method;

[0026] An inverse kinematics unit converts the coordinate sequence into discrete actions.

[0027] Optionally, the coordinate sequence output by the A* algorithm is converted into discrete actions through inverse kinematics, and is equivalent to the motion strategy of the reinforcement learning robot in the environment.

[0028] In a second aspect of the present application, a method for autonomous escape of a robot in a narrow environment based on end-to-end reinforcement learning is provided, comprising:

[0029] An action strategy of a reinforcement learning robot in an environment is determined according to radar real-time scanning data;

[0030] The action strategy of the robot is optimized to avoid invalid or collision actions for escape in a narrow environment.

[0031] In a third aspect of the present application, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to execute the method or run the system.

[0032] In a fourth aspect of the present application, a computer readable storage medium is provided, which stores a computer program executable by a processor to execute the method or run the system.

[0033] The end-to-end reinforcement learning-based narrow environment robot autonomous escape system provided in the application adopts an end-to-end reinforcement learning architecture, can realize real-time decision based on sensor input directly, and solves the technical problem of poor dynamic environment adaptability caused by the dependence of traditional methods on accurate environment modeling. An efficient action mask mechanism is adopted, which can realize safe and reliable real-time obstacle avoidance, and solves the technical problem of invalid exploration and collision risk caused by high-dimensional action space.

[0034] Other technical effects brought by additional features will be further described in corresponding embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0035] Other features, objects and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments with reference to the attached drawings:

[0036] Figure 1 is a structural diagram of an end-to-end reinforcement learning-based robot autonomous escape system according to an exemplary embodiment;

[0037] Figure 2 is a framework diagram of an end-to-end reinforcement learning-based robot autonomous escape system according to an exemplary embodiment;

[0038] Figure 3 is a schematic diagram of inverse calculation of robot action sequence based on A* algorithm according to an exemplary embodiment;

[0039] Figure 4 is a schematic diagram of efficient action space for differential kinematics and equal proportion dimension reduction according to an exemplary embodiment;

[0040] Figure 5 is a schematic diagram of pre-computed action mask identifying invalid states and filtering out unsafe actions according to an exemplary embodiment;

[0041] Figure 6 is a flowchart of an end-to-end reinforcement learning-based robot autonomous escape method according to an exemplary embodiment. DETAILED DESCRIPTION

[0042] The application will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the application, and these all belong to the protection scope of the application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0043] The traditional method relies on a pre-constructed static environment map, and due to the existence of certain approximation loss error of the grid map, the failure rate of navigation in a narrow environment is significantly increased. The existing method has a large amount of invalid exploration in a narrow channel due to the redundancy of the action space, resulting in long planning time and low success rate. Based on the above problems, the embodiment of the application provides a narrow environment robot autonomous escape system based on end-to-end reinforcement learning to solve the above problems.

[0044] As shown in Figure 1 and Figure 2 , a narrow environment robot autonomous escape system based on end-to-end reinforcement learning 100, comprising:

[0045] An end-to-end reinforcement learning module 110 learns the action strategy of the robot in the environment according to the real-time scanning data of the radar;

[0046] An action mask mechanism module 120 optimizes the action strategy of the robot to avoid invalid or collision actions in the narrow environment.

[0047] Specifically, invalid refers to actions that exceed the mechanical performance, such as greater than the maximum speed, maximum angular velocity.

[0048] The above embodiment of the application adopts an end-to-end reinforcement learning architecture, which can realize real-time decision-making based on sensor input, and solves the technical problem of poor dynamic environment adaptability caused by the dependence of traditional methods on accurate environment modeling. An efficient action mask mechanism is adopted, which can realize safe and reliable real-time obstacle avoidance, and solves the technical problem of invalid exploration and collision risk caused by high-dimensional action space.

[0049] In order to better improve the sensitivity of environment modeling error, in some specific embodiments of the application, the end-to-end reinforcement learning module, which is the core part of the robot motion decision, is divided into a state representation submodule, an action space design submodule, and a reinforcement learning network submodule.

[0050] The state representation submodule receives the laser radar input data, obtains the target position, and obtains the robot action distribution. Then, the three types of data are encoded by a multi-layer perception MLP to form a multi-dimensional state feature vector representing the robot state for use by the reinforcement learning network.

[0051] For example, the laser radar input data is a 500-dimensional vector representing the distance of 500 laser points, and after encoding, the laser radar data encoding is obtained. The target position is a 5-dimensional vector. The action distribution of the robot includes current linear velocity, angular velocity, and other information.

[0052] The action space design submodule is a key part of the end-to-end reinforcement learning module, and mainly controls the movement of the robot through a discretized action set, including multiple combinations of rotational angular velocity and linear velocity. Each action is accurately mapped to a specific physical movement and optimized through a reinforcement learning network.

[0053] For example, the discretized action set includes 42 discrete actions, which are two actions of rotating in place, 4*10 actions of circular motion, where 4 represents left front, right front, left rear and right rear, and 10 represents 10 different curvatures.

[0054] The reinforcement learning network submodule is based on the SoftActor-Critic (SAC) framework, combining a Transformer encoder with an MLP decoder to learn the action policy of the robot.

[0055] Specifically, the network balances exploration and utilization by maximizing the objective function to improve the diversity and stability of the policy.

[0056] The following objective function can be used:

[0057]

[0058] V π (s) represents the soft state value function of state s under the policy π, where the state S represents the state representation submodule robot state.

[0059] Q π (s,a) is the soft state-action value function, and the 42 discrete actions of the action space design submodule correspond to a in the formula.

[0060] π(a|s) represents the probability of selecting action a under state s;

[0061] α is a temperature parameter that adjusts the balance between rewards and exploration;

[0062] H is the entropy of the policy under state s.

[0063] It should be noted that the action policy output by the reinforcement learning network submodule refers to the probability distribution of the 42 discrete actions.

[0064] The above embodiments of the present application directly process real-time scanning data of the lidar through an end-to-end perception decision architecture, avoiding the dependence on a static environment model (only the current scanning data is needed, and no static environment model needs to be pre-stored), enabling the system to respond to environmental changes in real time and to deal with dynamic obstacles in real time.

[0065] To better address the problem of low efficiency of narrow space motion planning, it is necessary to improve the motion decision efficiency of the robot and reduce the collision risk. In some embodiments of the present application, the action mask mechanism module performs dimensionality reduction processing on the action space and provides real-time collision pre-detection. The action mask mechanism module is divided into an action space dimensionality reduction submodule and a collision pre-detection submodule.

[0066] The action space dimensionality reduction submodule, as shown in Figure 4 , maps the two-dimensional action space (velocity and angular velocity) to a one-dimensional linear space, optimizes the selection of the action space, and generates 42 different discrete actions. These actions cover a variety of modes from sharp turns to straight-line movements, ensuring that the robot can move flexibly in various environments. Specifically, through equal proportion mapping, which is set according to the maximum linear velocity and maximum angular velocity of the robot, dimensionality reduction refers to a fixed turning radius for an action, thereby fixing the ratio of linear velocity and angular velocity, and the angular velocity and linear velocity can change proportionally at the same time, which is related to the maximum non-collision step length.

[0067] Specifically, first, the set is defined as follows:

[0068] Fixed-radius turning

[0069]

[0070] The formula represents the composition of the 42 discrete actions generated after dimensionality reduction, which includes two parts: one is 2 stationary rotation actions represented by (±ω max , 0), meaning only the maximum angular velocity (positive and negative representing left and right directions) and linear velocity of 0; the other is 40 circular motion actions formed by the combination of different angular velocities (±ω i ) and linear velocities (±v i ), covering movements of different directions and turning radii.

[0071] Then, the parameters are calculated as follows:

[0072] For each radius r i (i = 0, 1,..., 9), the angular velocity ω i and linear velocity v i are calculated as follows:

[0073]

[0074] ω max : maximum angular velocity (unit: rad / s).

[0075] v max : maximum linear velocity (unit: m / s).

[0076] r iTurning radius (unit: meters), increasing exponentially (starting from 0.01 meters and doubling each time);

[0077] ω i Actual angular velocity, limited by ω max or v max / r i The smaller value.

[0078] v i Actual linear velocity, limited by v max or ω max ·r i The smaller value.

[0079] r i =ω i / v i .

[0080] The movements of a robotic vacuum cleaner are composed of both velocity and angular velocity, forming a two-dimensional motion space. Determining whether each discrete movement will result in a collision within this two-dimensional space is computationally intensive and costly. However, by performing dimensionality reduction, transforming the two-dimensional motion space into 42 discrete one-dimensional movements, the computational complexity can be effectively reduced, thereby improving the robot's real-time response and training efficiency.

[0081] Collision pre-detection submodule, such as Figure 5 As shown, by comparing the boundary points of the LiDAR point cloud data with those of the robot's motion trajectory (a trajectory composed of action policies output by a reinforcement learning network) in real time, the system can quickly determine whether there is a collision risk in the current action. Through this mechanism, the system can filter out potentially dangerous actions before they are executed, thereby improving the safety of the robot when performing tasks.

[0082] Specifically, collision detection obtains the maximum collision-free step size for 42 discrete actions, which is then used to trim (i.e., mask) the action itself.

[0083] For example:

[0084] For example, when checking the "small left turn" action: based on the trajectory boundary points, calculate that when performing this action, the leftmost edge of the robot will be 0.1 meters away from the left chair leg (safe), and it can move forward 0.8 meters before it may hit an obstacle (this is the "maximum collision-free step length").

[0085] Further inspection of the "turn right in place" action revealed that the robot's right side would directly rub against the wall during execution (the trajectory boundary point and the wall scanned by the radar overlapped), which was determined to be "risk of collision". The maximum collision-free step length was almost 0.

[0086] That is, the output from the collision pre-detection submodule is: from the 42 discrete actions output by the mask operation, there is no collision risk (still 42 after pruning, the optimal action is the action probability distribution generated by reinforcement learning to sample the optimal action. If the maximum collision-free step is 0, his sampling probability is also 0).

[0087] The above embodiments of the present application significantly improve the action screening efficiency by designing a 42-dimensional discrete action space and a real-time action mask mechanism.

[0088] In order to better solve the problem of convergence difficulty in complex scene training, some specific embodiments of the present application further include a hybrid training strategy module, which combines the advantages of A* algorithm and reinforcement learning algorithm to improve the learning efficiency of the robot in a narrow environment. It can be realized by the following submodules:

[0089] The A* algorithm guiding mechanism submodule is used to generate a collision-free global path at the beginning of training, and the action command is calculated through inverse kinematics, as shown in Figure 3 , guiding the robot to perform safe actions. This mechanism provides a reliable preliminary demonstration for the reinforcement learning network, ensuring that the robot can safely learn at the beginning.

[0090] The two-stage curriculum learning submodule ensures that the robot can quickly learn simple tasks in the initial stage and has the ability to handle more complex tasks in the later stage through fixed target stage and random target stage training methods. The fixed target stage helps the robot accumulate basic motor skills, while the random target stage helps the robot adapt to changing environments and improve the generalization ability of the strategy.

[0091] Specifically, the fixed map and random map are input into the A* planning algorithm. If the A* algorithm has a collision-free path, the corresponding discrete action is obtained through inverse kinematics of the global path. If there is no collision-free path, the action distribution output by the reinforcement learning is subjected to an efficient action mask mechanism to obtain a collision-free action distribution, and then the action distribution is sampled to obtain the final action, which is executed on the robot.

[0092] It should be noted that, as Figure 3 shown, inverse kinematics calculation refers to converting the path points planned by the A* algorithm into discrete actions. For example, two points are converted into in-place rotation + straight walking + in-place rotation, which is equivalent to the action output by the reinforcement learning, realizing seamless switching and helping the reinforcement learning to converge.

[0093] In addition, in the collision detection optimization of the A* planning algorithm, the core idea is to improve the calculation efficiency through shape simplification and equivalent conversion. First, the robot is approximated as an inscribed circle (the smallest circle that can completely wrap the robot, with a radius of r), at which time it is only necessary to determine whether the inscribed circle collides with the obstacle: if the inscribed circle does not collide, the robot itself will certainly not collide, which simplifies the originally complex shape edge collision judgment. On this basis, further equivalent conversion is performed: the inscribed circle of the robot is simplified to a particle (i.e., the center of the inscribed circle), and the edges of all obstacles are inflated outward by r (the inflation distance is equal to the radius of the inscribed circle) to form an "inflated environment". The equivalence of this conversion is that: whether the inscribed circle collides with the original obstacle, and whether the particle intersects with the inflated environment are completely equivalent - if the particle enters the inflated environment, it is equivalent to the inscribed circle colliding with the original obstacle. In order to further improve the efficiency, a fast lookup table mechanism is used: the information of "whether the particle will collide" in the inflated environment is stored in the table in advance (for example, each grid in the grid map is marked as "whether it collides"), and real-time detection does not need to calculate the geometric relationship, but only needs to query the table value corresponding to the position of the particle to judge, which greatly reduces the real-time calculation amount and makes the collision detection more efficient.

[0094] The above embodiments of the present application, through the innovative hybrid training strategy, combine the prior knowledge of the classic planning algorithm and the curriculum learning method of progressive difficulty improvement, so that the training efficiency of the system in a complex scene is significantly improved, and finally the system performs very robustly in a test scene containing multiple obstacle combinations.

[0095] Based on the same technical concept, in some other embodiments of the present application, a narrow environment robot autonomous escape method based on end-to-end reinforcement learning includes the following steps: Figure 6 As shown in the figure, the method includes the following steps:

[0096] First, according to the real-time scanning data of the radar, the motion strategy of the robot in the environment is optimized through reinforcement learning;

[0097] Second, the motion strategy of the robot is optimized to avoid invalid or collision actions.

[0098] The specific implementation of each step in the above examples of the present application can refer to the implementation technology of the corresponding modules / sub-modules of the end-to-end reinforcement learning-based narrow environment robot autonomous escape system in the above embodiments, which will not be described here.

[0099] The above embodiments of the present application are a new type of end-to-end, lightweight autonomous escape navigation scheme, which can significantly improve the autonomous escape ability of the robot in a narrow and complex environment while ensuring real-time performance. Lightweight means that there is no complex network structure, so the real-time performance is good and the calculation efficiency is very high. It does not rely on historical information and does not need to store too much information, and can be re-planned in real time to cope with dynamic scenes.

[0100] In one embodiment, the end-to-end reinforcement learning-based narrow environment robot autonomous escape method has the following detailed process:

[0101] S1. Obtain radar scan data, target position encoding, and robot state distribution to form state information and form a multi-dimensional state feature vector.

[0102] Specifically, the state information is constructed as follows:

[0103] The robot obtains environmental information through sensors, including:

[0104] Radar scan data: single-line 2D laser radar collects 500 distance measurement points, covering a 360° range, forming obstacle distribution information.

[0105] Target position encoding: a five-tuple is used to represent the distance, azimuth angle, and robot orientation deviation of the target point. d represents the straight-line distance from the current position of the robot to the target position. θ t refers to the angle between the current orientation of the robot as the reference and the "robot-target connecting line", refers to the angle between the current orientation of the robot and the "target direction" (i.e., the direction of the connecting line from the robot to the target);

[0106] Robot action distribution: includes current linear velocity, angular velocity, and other information.

[0107] These data are subjected to feature extraction and splicing to form a multi-dimensional state feature vector as input to the reinforcement learning network.

[0108] S2. The multi-dimensional state feature vector is input to the RL-network (reinforcement learning network) to obtain a spatial action design (which is the action policy, i.e., the probability distribution of 42 actions). The spatial action design includes 42 discrete actions, each corresponding to an angular velocity and a linear velocity.

[0109] S3. The 42 discrete actions are input to the action mask mechanism module for action space dimension reduction and collision pre-detection to obtain the optimal action. The optimal action here is obtained by sampling the 42 discrete actions after masking.

[0110] S4. Implement the optimal action.

[0111] The preferred features in the above embodiments can be used individually in any embodiment, and can also be used in any combination as long as they do not conflict. In addition, parts not described in detail in the embodiments can be implemented using existing technology.

[0112] The present application will be further described below in conjunction with specific application examples / contrastive examples, so as to better understand the above technical solutions of the present application. It should be understood that the following are only some examples and do not limit the present application.

[0113] Application Example 1: Household Sweeping Robot Trapped Escape

[0114] Application scenario:

[0115] When the sweeping robot is trapped between the legs of a dining table during cleaning (for example, the leg spacing is only 40 cm), the system automatically starts the escape mode.

[0116] Implementation process:

[0117] 1. The robot identifies the distribution of obstacles around by laser radar scanning and detects that the front, back, left and right are surrounded by obstacles

[0118] 2. The system automatically selects the narrow environment escape mode and activates the reinforcement learning decision module

[0119] 3. Based on real-time sensor data, the system generates an escape path in real time:

[0120] 4. No manual intervention is required throughout the process, with an average escape time of 10 seconds

[0121] Application Example 2: Passage of Commercial Service Robot

[0122] Application scenario:

[0123] The service robot encounters a temporarily placed shelf in the corridor of the mall and needs to pass through a narrow passage with a width of only 50 cm.

[0124] Implementation process:

[0125] 1. The robot detects that the passage in front narrows and automatically switches to autonomous escape mode

[0126] 2. Combine pre-stored maps and real-time laser data to efficiently plan a feasible path

[0127] 3. Use real-time action mask strategy to adjust the motion direction in real time according to the distance on both sides

[0128] From the above two specific applications, it can be seen that the end-to-end reinforcement learning narrow environment robot autonomous escape system of the present application can realize completely autonomous decision-making without the need for remote control or manual intervention; it can adapt to various unexpected situations and complex environments; while ensuring safety, it can maintain high passage efficiency; it can be widely used in various scenarios such as home, business, and industry. These practical application cases demonstrate the practical value and reliability of the present application in real environments, solving the key problem of limited action of existing robot systems in narrow spaces.

[0129] Based on the same technical concept, in other embodiments of the present application, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, is configured to execute the method for autonomous escape of a robot in a narrow environment based on end-to-end reinforcement learning, or run the system for autonomous escape of a robot in a narrow environment based on end-to-end reinforcement learning.

[0130] Based on the same technical concept, in other embodiments of the present application, a computer readable storage medium is provided, which stores a computer program executable by a processor to execute the method for autonomous escape of a robot in a narrow environment based on end-to-end reinforcement learning, or run the system for autonomous escape of a robot in a narrow environment based on end-to-end reinforcement learning.

[0131] Optionally, the memory is configured to store the program; the memory can include volatile memory (e.g., random-access memory (RAM), such as static random-access memory (SRAM), Double Data Rate Synchronous Dynamic Random-Access Memory (DDR SDRAM), etc.), and / or non-volatile memory (e.g., flash memory). The memory is configured to store computer programs (e.g., application programs, functional modules, etc. for implementing the above method), computer instructions, etc. The computer programs, computer instructions, etc. described above can be stored in one or more memories in a partitioned manner. Furthermore, the computer programs, computer instructions, data, etc. described above can be invoked by the processor.

[0132] The computer programs, computer instructions, etc. described above can be stored in one or more memories in a partitioned manner. Furthermore, the computer programs, computer instructions, data, etc. described above can be invoked by the processor.

[0133] The processor is configured to execute the computer program stored in the memory to implement each step in the method described above in the embodiments. For details, please refer to the related description in the method embodiments above.

[0134] The processor and the memory can be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor can be coupled and connected through a bus.

[0135] Those skilled in the art will appreciate that embodiments of the application can be readily used as software, hardware, or a combination of software and hardware. In one embodiment, the application can be implemented in software and can be stored on a computer readable medium, which can include random access memory (RAM), read only memory (ROM), magnetic disk or optical disk, or the like. The application can also be implemented as a combination of both software and hardware. In addition, the application can be implemented as a computer program product that can include a computer readable medium having stored computer program code thereon.

[0136] The present application is described in terms of exemplary embodiments, apparatus (systems) and computer program products according to the embodiments of the application as described above. It is to be understood that each of the figures or flows and / or combinations of the flows or flows and / or blocks in the flow and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flow or flows and / or blocks.

[0137] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow or flows and / or blocks Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flow or flows and / or blocks.

[0138] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow or flows and / or blocks Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flow or flows and / or blocks.

[0139] The above description is only specific embodiments of the present application. It is to be understood that the present application is not limited to the above specific embodiments, and various modifications or changes can be made by those skilled in the art within the scope of the claims, without departing from the spirit of the present application. The above preferred features can be combined in any manner, without conflict.

Claims

1. An end-to-end reinforcement learning based narrow environment robot autonomous escape system, characterized in that, The method comprises: an end-to-end reinforcement learning module for learning the robot's action policy in the environment according to radar real-time scanning data; an action mask mechanism module for optimizing the robot's action policy to avoid invalid or collision actions in a narrow environment.

2. The end-to-end reinforcement learning based robot autonomous escape system for narrow environment according to claim 1, wherein, The end-to-end reinforcement learning module comprises: a state representation submodule for obtaining lidar input data, target position and robot action distribution, and performing multi-layer perception encoding respectively to obtain laser encoding, target position encoding and robot action distribution encoding, which together form a multi-dimensional state feature vector for reinforcement learning; an action space design submodule comprising a discrete action set for controlling the robot's movement; a reinforcement learning network submodule based on a Soft Actor-Critic framework, combining a Transformer encoder and a multi-layer perception decoder, for learning the robot's action policy in the discrete action set through the multi-dimensional state feature vector.

3. The end-to-end reinforcement learning based robot autonomous escape system in a narrow environment according to claim 2, characterized in that, The discrete action set comprises 42 discrete actions, including two actions of rotating in place and 4*10 actions of circular motion, wherein the number 4 represents left front, right front, left rear and right rear, and the number 10 represents 10 different curvatures.

4. The end-to-end reinforcement learning based robot autonomous escape system in a narrow environment according to claim 3, characterized in that, The action mask mechanism module comprises: an action space dimension reduction submodule for mapping a two-dimensional action space composed of angular velocity and linear velocity to a one-dimensional linear space through a fixed turning radius to generate 42 one-dimensional discrete actions; a collision pre-detection submodule for comparing boundary points of a motion trajectory formed by the radar real-time scanning data and the action policy of the reinforcement learning network submodule under the premise of the one-dimensional discrete actions to determine whether the current action policy has a collision risk, determine a maximum collision-free step length, form an action mask, and clip an effective action policy.

5. The end-to-end reinforcement learning based autonomous escape system for robots in narrow environments according to claim 1, wherein, The method further comprises a hybrid training strategy module comprising: an A* algorithm guiding mechanism submodule for using an A* algorithm to generate a collision-free global path in the early stage of training, and calculating action instructions through inverse kinematics to guide the robot to perform safe actions; a two-stage curriculum learning submodule for ensuring the robot's generalization ability in the later stage of training through fixed target and random target training methods.

6. The end-to-end reinforcement learning based narrow environment robot autonomous escape system according to claim 5, wherein, The A* algorithm guiding mechanism submodule comprises: an input unit for inputting a grid map and obtaining an inflation grid map by inflation; a planning unit for calculating a coordinate sequence of a feasible path based on the inflation grid map through an A* method; an inverse kinematics unit for converting the coordinate sequence into discrete actions.

7. The end-to-end reinforcement learning based narrow environment robot autonomous escape system according to claim 6, wherein, The coordinate sequence output by the A* algorithm is converted into discrete actions through inverse kinematics, which is equivalent to the motion strategy of the reinforcement learning robot in the environment.

8. An end-to-end reinforcement learning based method for autonomous escape of a robot in a narrow environment, characterized in that, The method comprises: learning the robot's action policy in the environment according to radar real-time scanning data; optimizing the robot's action policy to avoid invalid or collision actions in a narrow environment.

9. A terminal comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to perform the method of claim 8, or runs the system of any one of claims 1-7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program, when executed by the processor, is operable to perform the method of claim 8, or to operate the system of any one of claims 1-7.

Images