Autonomous navigation method and system for mobile robots in cross-regional map-free scenarios

Abstract

The present disclosure relates to the field of autonomous navigation of mobile robots, and in particular to a method and system for autonomous navigation of mobile robots in cross-regional map-free scenarios. The method introduces auxiliary exploration tasks and combines extrinsic and intrinsic reward mechanisms to accelerate algorithm training while improving the exploration ability of the mobile robot in a local optimal area, guiding it to move in a direction close to a target point and escape from an oscillating state, thereby achieving efficient exploration and path optimization of the mobile robot in an unknown environment.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous navigation of mobile robots, and in particular to a method and system for autonomous navigation of mobile robots in cross-regional map-free scenarios. Background Art

[0002] Mobile robots are a key branch of mobile robotics technology. They possess autonomous locomotion capabilities and can perform tasks in an environment without relying on fixed trajectories or pre-set paths. Compared to traditional industrial mobile robots, mobile robots are highly flexible and adaptable, and are widely used in logistics and transportation, environmental monitoring, security inspections, medical services, and disaster relief, possessing significant academic and engineering significance. When mobile robots perform tasks such as workshop operations, biological sampling, and environmental mapping, planning reasonable paths and executing motion control based on real-time information acquired by sensors are essential for their successful completion. Autonomous navigation technology, a core issue in mobile robot research, directly impacts the intelligence level and task execution efficiency of mobile robots. However, autonomous navigation of mobile robots in map-free, cross-regional scenarios presents significant technical challenges. These tasks require mobile robots to efficiently plan paths and reach their destinations in unknown environments, relying solely on local sensory information. Traditional navigation methods primarily rely on path planning algorithms based on grid maps or classic heuristic planning algorithms (such as A* and Dijkstra). However, these methods often assume prior knowledge of the environment and are difficult to adapt to dynamic and complex scenarios. In addition, model-based navigation methods are sensitive to environmental uncertainties and are easily limited by local optimal solutions, making it difficult to achieve universal navigation capabilities across regions.

[0003] In recent years, deep reinforcement learning (DRL) has provided a new solution for mobile robot navigation tasks due to its powerful learning capabilities in high-dimensional state spaces. By combining deep neural networks, DRL can directly learn policy mappings from high-dimensional sensor data (such as lidar and cameras) to generate autonomous navigation behaviors. However, traditional DRL methods still face several key challenges when solving mapless cross-region tasks. First, external reward signals typically rely on goal-oriented designs, such as shortening the distance to the goal or feedback on task completion. However, such rewards are often sparse and difficult to balance multiple objectives (such as obstacle avoidance and goal arrival), which can easily lead to policies overfitting to specific paths and ignoring the global optimal solution. Second, traditional algorithms primarily rely on external signals to drive policy updates, lacking the ability to actively explore environmental changes and struggling to effectively escape local optima. Consequently, in cross-region, mapless scenarios, mobile robots are prone to becoming trapped in local optima and unable to proactively escape inefficient policies. Summary of the Invention

[0004] The present disclosure provides a method and system for autonomous navigation of a mobile robot in a cross-regional map-free scenario to address the problems existing in related technologies. The technical solutions are as follows:

[0005] In a first aspect, the present disclosure provides a method for autonomous navigation of a mobile robot in a cross-regional map-free scenario, comprising the following steps:

[0006] The mobile robot observes the state information s at time step t t , the state information s t Input into the Actor network;

[0007] The Actor network sends the state information s t After forward propagation, according to the strategy π θ (s t ) Output an action selection a t , and send the action selection a t Give the mobile robot, the mobile robot performs the action selection a t After that, the environment changes to a new state s t+1 , where θ is the network parameter of Actor;

[0008] Calculate the extrinsic reward r e,t and intrinsic reward r i,t , for each moment t, the extrinsic reward r e,t is defined as r e,t =r g +r c +r s , where r g is the goal-oriented part reward, r c Part of the reward for safe obstacle avoidance, r s To assist exploration rewards; intrinsic rewards r i,t The status information s t The input is generated by the intrinsic reward mechanism structure, which consists of four parts: feature extraction network, comparator network, context memory library and intrinsic reward estimation model;

[0009] The resulting quadruple <s t ,a t ,r t ,s t+1 >Stored in the experience replay unit, the critic network is based on the four-tuple in the experience replay unit<s t ,a t ,r t ,s t+1 > To approximate the action-value function Qφ (s t ,a t ) to train and update the parameters of the Actor network and Critic network, where φ is the Critic network parameter, r t =r e,t +r i,t ;

[0010] The trained Actor network and Critic network models are used to guide mobile robots to perform real-time cross-regional map-free autonomous navigation tasks.

[0011] Optionally, the status information s t Contains the scanning information of the single-line two-dimensional laser radar, the mobile robot's target point position information and the mobile robot's own position information.

[0012] Optionally, the action selection a t Includes the speed and angular velocity of the mobile robot.

[0013] Optionally, the intrinsic reward r i,t The generation steps are as follows:

[0014] The mobile robot observes the current lidar scanning data and the state information s of the target point at each time step t t , after processing through the feature extraction network, the state feature φ(s t );

[0015] The comparator network is used to evaluate the state feature s i and s j Reachability R(s) within k steps i ,s j )=f(φ(s i ),φ(s j )), by training a multi-layer fully connected neural network f to predict the state pair (s i ,s j ) and uses the Sigmoid activation function to map the output to [0,1], which represents the reachability score R(s i ,s j ), a score close to 1 indicates that the states are relatively “close” or “reachable”, while a score close to 0 indicates that the two states are far away or unreachable.

[0016] At the current time step t, the mobile robot maintains a context memory M t-1 , used to memorize the historical observation information of the mobile robot; the current state feature φ(s t ) and episodic memory bank M t-1 The reachability score R(s) between elements int ,M t-1 ), when the accessibility score is lower than the threshold τ, φ(s t ) is embedded in the episodic memory bank and defines the intrinsic reward r i,t =β·R(s t ,M t-1 ), where β is the intrinsic reward coefficient.

[0017] Optionally, the network structures of the Actor network and the Critic network are both composed of a 512*512*512*2 fully connected layer and an activation function Relu, the parameter optimizer is Adam, and the experience replay unit capacity is 2×10 6 , respectively calculate the network loss function to update the corresponding parameters θ, φ.

[0018] Optionally, before using the trained Actor network and Critic network models to guide the mobile robot to perform a real-time cross-region map-free autonomous navigation task, the following steps are also included:

[0019] Evaluation indicators are used to evaluate the safety and reliability of mobile robots in map-free cross-region navigation tasks, where the evaluation indicators include at least one of the following indicators: navigation success rate, collision rate, timeout rate, or number of execution steps.

[0020] Optionally, the goal-oriented reward is defined as follows:

[0021]

[0022] Among them, P t-1 and P t is the position of the mobile robot at the last moment and time step, P target Indicates the location of the target point, c1 is the reward coefficient when approaching the target point, and c2 is the positive reward obtained when the mobile robot reaches the target point; when the range between the mobile robot and the target point is less than a certain threshold d goal When , it is considered to have reached the target point and a large reward value is given;

[0023] Safety obstacle avoidance part reward c The definition is as follows:

[0024]

[0025] Among them, P obs,i It is represented as the position information of the i-th obstacle, c3 is the reward coefficient of the safe obstacle avoidance part, and c4 is the penalty value obtained after the collision; when the range between the mobile robot and the obstacle is less than a certain threshold d collision When , it is considered to have collided with the obstacle and a large penalty value is given;

[0026] Assisted Exploration Rewards s The definition is as follows:

[0027]

[0028] Among them, S new is the newly added area scanned by the mobile robot radar after a single action is executed, S lidar It represents the field of view that the mobile robot can scan; c5 is the reward coefficient of the auxiliary exploration part, which is used to adjust the weight of the auxiliary reward to balance the exploration and goal-oriented rewards.

[0029] In a second aspect, the embodiments of the present disclosure further provide a mobile robot autonomous navigation system for cross-regional map-free scenarios, including:

[0030] Sensor unit, used to observe the mobile robot time step t to obtain state information s t , the state information s t Input into the Actor network;

[0031] Action selection unit, the Actor network sends the state information s t After forward propagation, according to the strategy π θ (s t ) Output an action selection a t , and send the action selection a t Give the mobile robot, the mobile robot performs the action selection a t After that, the environment changes to a new state s t+1 , where θ is the network parameter of Actor;

[0032] External reward and intrinsic reward calculation unit, calculate the external reward r e,t and intrinsic reward r i,t , for each moment t, the extrinsic reward r e,t is defined as r e,t =r g +r c +r s , where r g is the goal-oriented part reward, r c Part of the reward for safe obstacle avoidance, r s To assist exploration rewards; intrinsic rewards r i,t The status information s t The input is generated by the intrinsic reward mechanism structure, which consists of four parts: feature extraction network, comparator network, context memory library and intrinsic reward estimation model;

[0033] The Actor network and Critic network training units will obtain the quadruple <s t ,a t ,r t ,s t+1 >Stored in the experience replay unit, the critic network is based on the four-tuple in the experience replay unit<s t ,a t ,r t ,s t+1 > To approximate the action-value function Q φ (s t ,a t ) to train and update the parameters of the Actor network and Critic network, where φ is the Critic network parameter, r t =r e,t +r i,t ;

[0034] The control execution unit uses the trained Actor network and Critic network models to guide the mobile robot to perform real-time cross-region map-free autonomous navigation tasks.

[0035] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned mobile robot autonomous navigation method when executing the computer program.

[0036] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned mobile robot autonomous navigation method when the program is executed by a processor.

[0037] The advantages or beneficial effects of the above technical solution include at least:

[0038] By introducing auxiliary exploration tasks and combining external rewards with intrinsic reward mechanisms, the exploration ability of the mobile robot in the local optimal area is improved while accelerating the algorithm training, guiding it to move in the direction close to the target point and get rid of the oscillation state, thereby realizing efficient exploration and path optimization of the mobile robot in an unknown environment. In addition, an intrinsic reward mechanism driven by situational curiosity is proposed, and the concept of state reachability is introduced as an exploration driving force to motivate the mobile robot to actively explore unseen areas, thereby effectively avoiding the local optimal problem and improving the training efficiency and adaptability of the algorithm under sparse reward conditions. The disclosed embodiment not only overcomes the limitations of traditional external reward sparsity, but also significantly enhances the algorithm's exploration ability, convergence efficiency and navigation task success rate, providing an innovative solution for autonomous navigation in map-free cross-region scenarios.

[0039] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present disclosure will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present disclosure and should not be regarded as limiting the scope of the present disclosure.

[0041] Figure 1 This is a flow chart of a method for autonomous navigation of a mobile robot in a cross-region map-free scenario according to an embodiment of the present disclosure;

[0042] Figure 2 Schematic diagram of the framework of the map-free cross-region navigation method for a mobile robot in an embodiment of the present disclosure;

[0043] Figure 3 A schematic diagram of the auxiliary exploration reward in an embodiment of the present disclosure;

[0044] Figure 4 Schematic diagram of the intrinsic reward mechanism of the situation-based curiosity mechanism in an embodiment of the present disclosure;

[0045] Figure 5 Schematic diagram of the structure of the feature extraction network in the embodiment of the present disclosure;

[0046] Figure 6 is a schematic diagram of the structure of a comparator network in an embodiment of the present disclosure;

[0047] Figure 7 This is a block diagram of the autonomous navigation system of a mobile robot in a cross-regional map-free scenario in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] Hereinafter, only certain exemplary embodiments are briefly described. As will be appreciated by those skilled in the art, the described embodiments may be modified in various ways without departing from the spirit or scope of the present disclosure. Therefore, the drawings and description are to be considered as illustrative in nature and not restrictive.

[0049] The present disclosure provides a method for autonomous navigation of a mobile robot in a cross-regional map-free scenario. Figure 1 As shown, Figure 1 The flowchart of the autonomous navigation method of the mobile robot includes the following steps:

[0050] S10, the mobile robot observes and obtains state information s at time step t t , the state information s t Input into the Actor network;

[0051] Figure 2 This is a schematic diagram of the framework of the map-free cross-region navigation method for mobile robots. It integrates a storage unit and a sensor unit. The sensor unit includes an inertial odometer, a single-line two-dimensional laser radar, and a GPS. It is used to obtain the target point orientation, its own position and posture, and local observable environment information. It can collect the observation information, position information, and collision detection information of the mobile robot in real time, providing accurate perception input for navigation decision-making. At time step t, the mobile robot observes the time step state information s through the sensor unit. t In this embodiment, the status information is defined as follows:

[0052] s t =(s lidar ,s target ), where s lidar is the single-line two-dimensional lidar data, s target is the orientation information of the target point, which is calculated by the mobile robot's own position and the target point orientation information.

[0053] S20, the Actor network sends the status information s t After forward propagation, according to the strategy π θ (s t ) Output an action selection a t , and send the action selection a t Give the mobile robot, the mobile robot performs the action selection a t After that, the environment changes to a new state s t+1 , where θ is the network parameter of Actor;

[0054] In this embodiment, action selection a t It can be defined as the velocity v and angular velocity ω of the mobile robot: t =(v,ω), in order to ensure the safe navigation of the mobile robot, an upper limit is set on the speed and angular velocity output by it.

[0055] S30. Calculate the external reward r e,t and intrinsic reward r i,t , for each time step t, the extrinsic reward r e,t is defined as r e,t =r g +r c +r s , where r gis the goal-oriented part reward, r c Part of the reward for safe obstacle avoidance, r s To assist exploration rewards; intrinsic rewards r i,t The status information s t The input is generated by the intrinsic reward mechanism structure, which consists of four parts: feature extraction network, comparator network, context memory library and intrinsic reward estimation model;

[0056] In this embodiment, the goal-oriented reward can be defined as follows:

[0057]

[0058] Among them, P t-1 and P t is the position of the mobile robot at the last moment and time step, P target Indicates the location of the target point, c1 is the reward coefficient when approaching the target point, and c2 is the positive reward obtained when the mobile robot reaches the target point. When the range between the mobile robot and the target point is less than a certain threshold d goal When , we believe that it has reached the target point and give it a larger reward value.

[0059] In this embodiment, the safety obstacle avoidance reward r c It can be defined as follows:

[0060]

[0061] Among them, P obs,i It is represented as the position information of the i-th obstacle, c3 is the reward coefficient of the safe obstacle avoidance part, and c4 is the penalty value obtained after the collision. When the range between the mobile robot and the obstacle is less than a certain threshold d collision When , we consider that it collides with an obstacle and give it a larger penalty value.

[0062] In this embodiment, the auxiliary exploration reward r s It can be defined as follows:

[0063]

[0064] Among them, S new is the newly added area scanned by the mobile robot radar after a single action is executed, S lidar Indicates the area of ​​the field of view that the mobile robot can scan. c5 is the reward coefficient of the auxiliary exploration part, which is used to adjust the weight of the auxiliary reward to balance the exploration and goal-oriented rewards, so as to avoid deviation from the main navigation task due to excessive exploration, such as Figure 3 Shown is a schematic diagram of assisted exploration rewards.

[0065] In this embodiment, Figure 4 The figure shows the intrinsic reward mechanism, which moves the current state information s of the robot to t As input, generate intrinsic reward r i,t At each time step t, the mobile robot observes the current lidar scanning data and the target point orientation information s t =(s lidar ,s goal ); process these data separately through two independent network branches and fuse their features to form a feature extraction network φ. The extracted state features are expressed as φ(s t ), specific network structure, such as Figure 5 The figure shows the structure of the feature extraction network. Furthermore, a comparator network is designed to calculate the reachability R(s) of the state information observed by the mobile robot at two different times within k steps. i ,s j )=f(φ(s i ),φ(s j )), by training a multi-layer fully connected neural network f to predict a pair of states (s i ,s j ) is the reachability of the mobile robot. t ), a buffer named episodic memory bank M is designed in the intrinsic reward mechanism, with a capacity of N. The memory bank is initialized before the start of the round and is updated to M at time step t. t-1 Specifically, at each time step, the current observation value s is calculated using the feature extraction network and the comparator network. t and episodic memory bank M t-1 The reachability score R(s) between elements in t ,M t-1 )=max{f(φ(s t ),m)|m∈M t-1}∈[0,1]. This score is the maximum value obtained by comparing the current observation value with the feature state values ​​in the memory bank to avoid completely ignoring the contribution of old memory elements. If the reachability score is lower than the threshold, that is, it is far away from the historical state of the mobile robot, the current observation value is embedded in the context memory bank. The specific formula is as follows:

[0066]

[0067] Where τ is the accessibility threshold parameter, and the episodic memory bank will be reset after each round.

[0068] Furthermore, when the accessibility score R(s t ,M t-1) < τ, it means that the current state cannot be reached from any state in the memory bank through k steps, which means that the state has sufficient "novelty" and is therefore added to the episodic memory bank. Therefore, the intrinsic reward estimation model is defined as:

[0069]

[0070] Among them, r i,t is the intrinsic reward obtained by the mobile robot at time t, and β is the intrinsic reward coefficient.

[0071] S40, the obtained quadruple <s t ,a t ,r t ,s t+1 >Stored in the experience replay unit, the critic network is based on the four-tuple in the experience replay unit<s t ,a t ,r t ,s t+1 > To approximate the action-value function Q φ (s t ,a t ) to train and update the parameters of the Actor network and Critic network, where φ is the Critic network parameter, r t =r e,t +r i,t ;

[0072] In this embodiment, the data generated by the above-mentioned state conversion process is t ,a t ,r t ,s t+1 > As an experience data, it is stored in the experience playback unit of the algorithm, where r t =r e,t +r i,t , and enter the next state transition process.

[0073] As an optional embodiment, the network structures of the Actor network and the Critic network are composed of a 512*512*512*2 fully connected layer and an activation function ReLU, the parameter optimizer is Adam, and the experience replay unit capacity is 2×10 6 , respectively calculate the network loss function to update the corresponding parameters θ, φ. A batch of experience data is randomly selected from the experience replay unit and sent to the Critic and Actor networks for training. The Critic network approximates the action value function Q based on the data in the experience replay unit. φ (s t ,a t), where φ is the parameter of the Critic network. During each training process, the parameters θ and φ of the Actor and Critic networks are also updated by the Adam optimizer to ensure that the action decisions output by the mobile robot tend to the optimal strategy.

[0074] Repeat the above steps until a better strategy is trained, and save the model parameters of the Actor network and Cr itic network.

[0075] S50. Use the trained Actor network and Critic network models to guide the mobile robot to perform real-time cross-regional map-free autonomous navigation tasks.

[0076] In this embodiment, at the beginning of each training round, the mobile robot's starting point is randomly set in the environment, the size and structure of different areas in the room are randomly set, and the target point location is randomly set. At each time step, the mobile robot selects control instructions within the action space to move. If the mobile robot collides with an obstacle or boundary, the round ends in failure and a new round of training begins. If the mobile robot reaches the target point and receives a large reward, the round ends in success and a new round of training begins. After each training round, the mobile robot's reward value, number of execution steps, and training success rate are recorded. When these indicators tend to stabilize, the algorithm has converged and the mobile robot can make autonomous decisions in map-free cross-region scenarios, ensuring the safety and stability of navigation tasks. The algorithm and network model parameters are deployed on the mobile robot platform to realize its autonomous navigation tasks in map-free cross-region scenarios.

[0077] As an optional embodiment, the status information s t Contains the scanning information of the single-line two-dimensional laser radar, the mobile robot's target point position information and the mobile robot's own position information.

[0078] As an optional embodiment, the action selection a t Includes the speed and angular velocity of the mobile robot.

[0079] As an optional embodiment, the intrinsic reward r i,t The generation steps are as follows:

[0080] The mobile robot observes the current lidar scanning data and the state information s of the target point at each time step t t , after processing through the feature extraction network, the state feature φ(s t );

[0081] like Figure 6 The diagram shows a comparator network used to evaluate the state characteristic s.i and s j Reachability R(s) within k steps i ,s j )=f(φ(s i ),φ(s j )), by training a multi-layer fully connected neural network f to predict the state pair (s i ,s j ) and uses the Sigmoid activation function to map the output to [0,1], which represents the reachability score R(s i ,s j ), a score close to 1 indicates that the states are relatively “close” or “reachable”, while a score close to 0 indicates that the two states are far away or unreachable.

[0082] At the current time step t, the mobile robot maintains a context memory M t-1 , used to memorize the historical observation information of the mobile robot; the current state feature φ(s t ) and episodic memory bank M t-1 The reachability score R(s) between elements in t ,M t-1 ), when the accessibility score is lower than the threshold τ, φ(s t ) is embedded in the episodic memory bank and defines the intrinsic reward r i,t =β·R(s t ,M t-1 ), where β is the intrinsic reward coefficient.

[0083] As an optional embodiment, before using the trained Actor network and Critic network models to guide the mobile robot to perform a real-time cross-region map-free autonomous navigation task, the following steps are also included:

[0084] Evaluation indicators are used to evaluate the safety and reliability of mobile robots in map-free cross-region navigation tasks, where the evaluation indicators include at least one of the following indicators: navigation success rate, collision rate, timeout rate, or number of execution steps.

[0085] The present disclosure also provides a mobile robot autonomous navigation system 100 for cross-regional map-free scenarios. Figure 7 Shown, including:

[0086] Sensor unit 1 is used to observe the mobile robot at time step t to obtain state information s t , the state information s t Input into the Actor network;

[0087] Action selection unit 2, the Actor network sends the state information s tAfter forward propagation, according to the strategy π θ (s t ) Output an action selection a t , and send the action selection a t Give the mobile robot, the mobile robot performs the action selection a t After that, the environment changes to a new state s t+1 , where θ is the network parameter of Actor;

[0088] External reward and intrinsic reward calculation unit 3, calculate the external reward r e,t and intrinsic reward r i,t , for each moment t, the extrinsic reward r e,t is defined as r e,t =r g +r c +r s , where r g is the goal-oriented part reward, r c Part of the reward for safe obstacle avoidance, r s To assist exploration rewards; intrinsic rewards r i,t The status information s t The input is generated by the intrinsic reward mechanism structure, which consists of four parts: feature extraction network, comparator network, context memory library and intrinsic reward estimation model;

[0089] Actor network and Critic network training unit 4, the obtained quadruple <s t ,a t ,r t ,s t+1 >Stored in the experience replay unit, the critic network is based on the four-tuple in the experience replay unit<s t ,a t ,r t ,s t+1 > To approximate the action-value function Q φ (s t ,a t ) to train and update the parameters of the Actor network and Critic network, where φ is the Critic network parameter, r t =r e,t +r i,t ;

[0090] The control execution unit 5 uses the trained Actor network and Critic network models to guide the mobile robot to perform real-time cross-region map-free autonomous navigation tasks.

[0091] In the absence of any contradiction, the above-mentioned units in the system 100 of the embodiment of the present disclosure can implement any implementation of the above-mentioned corresponding methods.

[0092] The present disclosure also provides an electronic device comprising: a processor; and a memory for storing instructions executable by the processor. The processor is configured to implement the aforementioned method for autonomous navigation of a mobile robot in a cross-region, map-free scenario. The electronic device can be provided as a terminal, server, or other device.

[0093] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon. When executed by a processor, the computer program instructions implement the aforementioned method for autonomous navigation of a mobile robot in a cross-regional, map-free scenario. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0094] Those skilled in the art will understand that in the specific implementation method and system of the above-mentioned cross-regional map-free scene mobile robot autonomous navigation method, the writing order of each step does not mean a strict execution order and constitutes any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0095] In the description of this specification, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and integrate different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless they are mutually inconsistent.

[0096] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the present disclosure, "plurality" means two or more, unless otherwise specifically defined.

[0097] The above are only specific embodiments of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any person skilled in the art can easily conceive of various modifications or substitutions within the technical scope disclosed in this disclosure, and such modifications or substitutions should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A method for autonomous navigation of a mobile robot in a cross-regional map-free scenario, characterized in that: The steps include: The mobile robot observes the state information s at time step t t , the state information s t Input into the Actor network; The Actor network sends the state information s t After forward propagation, according to the strategy π θ (s t ) Output an action selection a t , and send the action selection a t Give the mobile robot, the mobile robot performs the action selection a t After that, the environment changes to a new state s t+1 , where θ is the network parameter of Actor; Calculate the extrinsic reward r e,t and intrinsic reward r i,t , for each moment t, the extrinsic reward r e,t is defined as r e,t =r g +r c +r s , where r g is the goal-oriented part reward, r c Part of the reward for safe obstacle avoidance, r s To assist exploration rewards; intrinsic rewards r i,t The status information s t The input is generated by the intrinsic reward mechanism structure, which consists of four parts: feature extraction network, comparator network, context memory library and intrinsic reward estimation model; The resulting quadruple <s t ,a t ,r t ,s t+1 >Stored in the experience replay unit, the critic network is based on the four-tuple in the experience replay unit<s t ,a t ,r t ,s t+1 > To approximate the action-value function Q φ (s t ,a t ) to train and update the parameters of the Actor network and Critic network, where φ is the Critic network parameter, r t =r e,t +r i,t ; The trained Actor network and Critic network models are used to guide mobile robots to perform real-time cross-regional map-free autonomous navigation tasks.

2. The mobile robot autonomous navigation method according to claim 1, wherein: The status information s t Contains the scanning information of the single-line two-dimensional laser radar, the mobile robot's target point position information and the mobile robot's own position information.

3. The mobile robot autonomous navigation method according to claim 1 or 2, characterized in that: The action selection a t Includes the speed and angular velocity of the mobile robot.

4. The mobile robot autonomous navigation method according to claim 1 or 2, characterized in that: The intrinsic reward r i,t The generation steps are as follows: The mobile robot observes the current lidar scanning data and the state information s of the target point at each time step t t , after processing through the feature extraction network, the state feature φ(s t ); The comparator network is used to evaluate the state feature s i and s j Reachability R(s) within k steps i ,s j )=f(φ(s i ),φ(s j )), by training a multi-layer fully connected neural network f to predict the state pair (s i ,s j ) and uses the Sigmoid activation function to map the output to [0,1], which represents the reachability score R(s i ,s j ), a score close to 1 indicates that the states are relatively "close" or "reachable", while a score close to 0 indicates that the two states are far away or unreachable; At the current time step t, the mobile robot maintains a context memory M t-1 , used to memorize the historical observation information of the mobile robot; the current state feature φ(s t ) and episodic memory bank M t-1 The reachability score R(s) between elements in t ,M t-1 ), when the accessibility score is lower than the threshold τ, φ(s t ) is embedded in the episodic memory bank and defines the intrinsic reward r i,t =β·R(s t ,M t-1 ), where β is the intrinsic reward coefficient.

5. The mobile robot autonomous navigation method according to claim 1 or 2, characterized in that: The network structures of the Actor network and the Critic network are composed of a 512*512*512*2 fully connected layer and an activation function ReLU. The parameter optimizer is Adam, and the experience replay unit capacity is 2×10 6 , respectively calculate the network loss function to update the corresponding parameters θ, φ.

6. The mobile robot autonomous navigation method according to claim 1 or 2, characterized in that: Before using the trained Actor and Critic network models to guide a mobile robot in real-time cross-region, map-free autonomous navigation, the following steps are also required: Evaluation indicators are used to evaluate the safety and reliability of mobile robots in map-free cross-region navigation tasks, where the evaluation indicators include at least one of the following indicators: navigation success rate, collision rate, timeout rate, or number of execution steps.

7. The mobile robot autonomous navigation method according to claim 1 or 2, characterized in that: The goal-oriented reward is defined as follows: Among them, P t-1 and P t is the position of the mobile robot at the last moment and time step, P target Indicates the location of the target point, c1 is the reward coefficient when approaching the target point, and c2 is the positive reward obtained when the mobile robot reaches the target point; when the range between the mobile robot and the target point is less than a certain threshold d goal When , it is considered to have reached the target point and a large reward value is given; Safety obstacle avoidance part reward c The definition is as follows: Among them, P obs,i It is represented as the position information of the i-th obstacle, c3 is the reward coefficient of the safe obstacle avoidance part, and c4 is the penalty value obtained after the collision; when the range between the mobile robot and the obstacle is less than a certain threshold d collision When , it is considered to have collided with the obstacle and a large penalty value is given; Assisted Exploration Rewards s The definition is as follows: Among them, S new is the newly added area scanned by the mobile robot radar after a single action is executed, S lidar It represents the field of view that the mobile robot can scan; c5 is the reward coefficient of the auxiliary exploration part, which is used to adjust the weight of the auxiliary reward to balance the exploration and goal-oriented rewards.

8. The autonomous navigation system for mobile robots in cross-regional map-free scenarios is characterized by: include: Sensor unit, used to observe the mobile robot time step t to obtain state information s t , the state information s t Input into the Actor network; Action selection unit, the Actor network sends the state information s t After forward propagation, according to the strategy π θ (s t ) Output an action selection a t , and send the action selection a t Give the mobile robot, the mobile robot performs the action selection a t After that, the environment changes to a new state s t+1 , where θ is the network parameter of Actor; External reward and intrinsic reward calculation unit, calculate the external reward r e,t and intrinsic reward r i,t , for each moment t, the extrinsic reward r e,t is defined as r e,t =r g +r c +r s , where r g is the goal-oriented part reward, r c Part of the reward for safe obstacle avoidance, r s To assist exploration rewards; intrinsic rewards r i,t The status information s t The input is generated by the intrinsic reward mechanism structure, which consists of four parts: feature extraction network, comparator network, context memory library and intrinsic reward estimation model; The Actor network and Critic network training units will obtain the quadruple <s t ,a t ,r t ,s t+1 >Stored in the experience replay unit, the critic network is based on the four-tuple in the experience replay unit<s t ,a t ,r t ,s t+1 > To approximate the action-value function Q φ (s t ,a t ) to train and update the parameters of the Actor network and Critic network, where φ is the Critic network parameter, r t =r e,t +r i,t ; The control execution unit uses the trained Actor network and Critic network models to guide the mobile robot to perform real-time cross-region map-free autonomous navigation tasks.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the mobile robot autonomous navigation method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the mobile robot autonomous navigation method according to any one of claims 1 to 7 is implemented.

Images