Robot autonomous navigation coordination control method and system based on st-td3 strategy

Abstract

The present application relates to the technical field of autonomous navigation of unmanned cleaning trolley, and particularly relates to a robot autonomous navigation coordination control method and system based on ST-TD3 strategy. The present application trains a motion decision model to generate a motion based on a current state of the unmanned cleaning trolley, and the unmanned cleaning trolley executes the motion to adjust a speed and a yaw angle. The state includes an environmental depth feature and an obstacle density feature of an environment where the unmanned cleaning trolley is located, and a relative distance and a relative azimuth angle between the unmanned cleaning trolley and a target point. The environmental depth feature and the obstacle density feature are extracted based on a depth image feature collected by the unmanned cleaning trolley. The motion includes a linear speed and an angular speed, and both of them satisfy a linear speed maximum value constraint and an angular speed maximum value constraint of the unmanned cleaning trolley. The angular speed is represented by positive and negative values to indicate a deflection direction. The present application can realize efficient, safe and stable driving of the unmanned cleaning trolley in a complex cabin environment.

Description

Technical Field

[0001] This invention relates to the field of autonomous navigation technology for unmanned cleaning vehicles, and in particular to a robot autonomous navigation coordination control method and system based on the ST-TD3 (gradient strategy) strategy. Background Technology

[0002] With the rapid development of unmanned cleaning vehicle (UGV) technology, its automated operations in the cabin environments of large cargo ships are widely used, such as cargo inspection, environmental monitoring, and localized cleaning. This technology is based on the application of UGVs inside ship cabins. Ship cabins are typically complex in structure and space-constrained, with various static and dynamic obstacles such as cargo stacking and equipment obstructions. This places higher demands on the autonomous navigation capabilities of UGVs. Autonomous navigation, as a fundamental capability for UGVs to perform tasks within ship cabins, requires reliable obstacle avoidance and target point arrival in unknown or partially observable environments. Therefore, while ensuring system real-time performance and reliability, improving the navigation success rate and movement efficiency of UGVs in complex ship cabin environments, and achieving high-precision, highly adaptive positioning, has become a key challenge for current unmanned systems applications within ship cabins.

[0003] In the area of ​​autonomous navigation for unmanned cleaning vehicles, methods can be broadly categorized into traditional approaches, supervised learning-based methods, and reinforcement learning-based methods. Traditional methods plan feasible paths after creating an environmental map, but map creation requires significant computing power and memory and is unsuitable for unknown environments. Supervised learning-based methods do not require map creation but necessitate large amounts of labeled data and also struggle to handle unknown environments.

[0004] Reinforcement learning-based methods do not require graph construction, enabling an end-to-end model from sensor data to action output. Training data is obtained from the interaction between the unmanned cleaning vehicle and its environment, making it an important method for solving this problem. Existing work has made some progress in extracting obstacle features, but considering only the influence of the nearest obstacle can easily lead to the unmanned cleaning vehicle getting stuck in more obstacles while trying to avoid the nearest one. How to solve these problems of traditional methods is also the main objective of this invention. Summary of the Invention

[0005] To overcome the problems of narrow cabin structure, high obstacle density, many turns, and limited vision in the existing technology of unmanned cleaning vehicles for cargo ship cleaning, this invention proposes a robot autonomous navigation coordination control method based on the ST-TD3 strategy, which can realize the efficient, safe and stable driving of unmanned cleaning vehicles in complex cabin environments.

[0006] This invention proposes a robot autonomous navigation coordination control method based on the ST-TD3 strategy, which trains an action decision model based on the current state of the unmanned cleaning vehicle. Generate Actions The unmanned cleaning cart performs the action. To adjust speed and yaw angle;

[0007] The status includes: the environmental depth features and obstacle density features of the environment in which the unmanned cleaning vehicle is located, as well as the relative distance and relative azimuth angle between the unmanned cleaning vehicle and the target point; the environmental depth features and obstacle density features are extracted based on the original depth images collected by the unmanned cleaning vehicle;

[0008] The motion includes linear velocity and angular velocity, and both satisfy the constraints of the maximum linear velocity and maximum angular velocity of the unmanned cleaning vehicle, respectively. The angular velocity is represented by positive and negative values ​​to indicate the deflection direction.

[0009] The obstacle density features are obtained as follows:

[0010] First, the raw depth images collected by the unmanned cleaning vehicle are rasterized to obtain... × The depth feature matrix M;

[0011] Obstacle density features are obtained by applying Gaussian weights to the depth feature matrix M. :

[0012] ;

[0013] ;

[0014] ;

[0015] in, and A weighted vector that satisfies a normal distribution; and It is an ordinal number; To define the variance, It is a natural number.

[0016] Preferably, the action decision model is trained using an Actor-Critic network, and the rewards used in training include distance penalties. Collision penalty Angle punishment and step length penalty Multiple items in the middle; step size penalty To set a value;

[0017] ;

[0018] As a preset constant, The radius of the unmanned cleaning vehicle; A negative constant is defined; The relative distance between the unmanned cleaning vehicle and the target point;

[0019] ;

[0020] The number of collision detection values ​​obtained from the simulation; These are preset constants; It represents the average depth of pixels within a specified range in the middle of the original depth image; This represents the safe distance between the unmanned cleaning vehicle and obstacles. , It is a negative constant; Indicates obstacle density characteristics;

[0021] ;

[0022] and These represent the yaw angles of the unmanned cleaning vehicle at the previous and current moments, respectively. The relative azimuth angle between the unmanned cleaning vehicle and the target point. and It is a negative constant.

[0023] Preferably, environmental depth features are expressed as depth feature vectors. Representation, deep feature vector is the row concatenation vector of the depth feature matrix M.

[0024] Preferably, the method for training an Actor-Critic network includes the following steps:

[0025] Construct a basic model that corresponds one-to-one with the unmanned cleaning vehicle, which includes a gated loop module, an online Actor network, an online Critic network, an online Actor network, and a target Critic network; the online Actor network and the target Actor network have the same structure, and the online Critic network and the target Critic network have the same structure.

[0026] Gated loop module for state Extracting temporal features, the Actor online network generates actions based on these temporal features. The Actor target network is based on the state of the unmanned cleaning vehicle. Execute action at time The state after Generate Actions ,right After adding noise and cropping, the next action is obtained. ; The parameters of the Actor target network are represented; the Critic online network is state-based. and actions Output evaluation value; the Critic target network is state-based. and actions Output evaluation value;

[0027] Let each basic model randomly sample the state of the unmanned cleaning vehicle. Process and construct experience samples Stored in buffer D; The unmanned cleaning cart is in operation. Execute action at time The state after that, For state-based Calculated reward;

[0028] Whenever experience sample Increase quantity setting value Then, a batch of experience samples are randomly selected for each basic model to form the experience pool corresponding to the basic model.

[0029] Each basic model is updated locally on its corresponding experience pool until the number of basic model updates reaches a set value. Then, the gated loop module and the Actor online network are extracted to form an action decision model.

[0030] The preferred method for locally updating the base model on the corresponding experience pool is as follows:

[0031] SD4. Draw N experience samples from the experience pool. The input base model is processed, and the loss function is calculated based on the evaluation values ​​output by the Critic online network and the Critic target network. The Critic online network is updated through backpropagation using the loss function, and the updated Critic online network parameters are copied to the Critic target network.

[0032] SD5. Determine whether the number of updates to the Critic online network and the Critic target network is an integer multiple of d, where d is a set value;

[0033] No, then return to update buffer D;

[0034] If yes, proceed to step SD6;

[0035] SD6. The gated loop module and the Actor online network are updated with gradients based on the evaluation values ​​generated by the Critic online network, and then the Actor target network and the Critic target network are updated with sliding updates.

[0036] SD7. Determine if the number of times the Actor's online network updates have reached the set value;

[0037] No, then return to update buffer D;

[0038] Yes, then the fixed-gated loop module and the Actor online network form an action decision model.

[0039] Preferably, the loss function is calculated in step SD4 as follows:

[0040] ;

[0041] ;

[0042] in, For the first A sample of experiences The corresponding estimated suppression, For Critic online network, the input is and The evaluation value generated at that time For the Critic target network, the input is and The evaluation value generated at that time For the Actor target network, the input is The output at that time is processed by adding noise and cropping to obtain the motion; and These represent the parameters of the Critic online network and the parameters of the Critic target network, respectively. is the discount factor, and N is the sample size.

[0043] Preferred, for After adding noise and cropping, the next action is obtained. The method is as follows:

[0044] ;

[0045] in, Represents the boundary function. To cut noise, and ; and These are the lower and upper limits of the action, respectively; , , and These represent the maximum speed and maximum angular velocity of the unmanned cleaning vehicle, respectively.

[0046] Preferably, the states that serve as inputs to the Actor online network and the action decision model are obtained by normalizing each state variable; the state variables are environmental depth features, obstacle density features, relative distance, and relative azimuth angle.

[0047] The present invention proposes a robot autonomous navigation coordination control system based on the ST-TD3 strategy, which includes a memory and a processor. The memory stores a computer program, and the processor is connected to the memory. The processor is used to execute the computer program to realize the robot autonomous navigation coordination control method based on the ST-TD3 strategy.

[0048] The present invention proposes a storage medium storing a computer program, which, when executed, implements the robot autonomous navigation coordination control method based on the ST-TD3 strategy.

[0049] The advantages of this invention are:

[0050] (1) The present invention proposes a robot autonomous navigation coordination control method based on ST-TD3 strategy, which combines environmental depth features, obstacle density modeling and the relative orientation of the vehicle and the target point to realize the autonomous navigation strategy of machine learning, so as to realize the efficient, safe and stable driving of the unmanned cleaning vehicle in the complex cabin environment.

[0051] (2) In this invention, the obstacle density features are obtained by processing the original depth image collected by the vehicle through Gaussian noise, and the obstacle information in the environment is processed into numerical representation features, which facilitates model learning. In addition, this invention adopts the Actor-Critic network to realize the extraction and utilization of temporal features, which further improves the reliability of model prediction.

[0052] (3) This invention introduces a gated loop module, which combines time-series feature extraction, obstacle density modeling and sparse reward enhancement autonomous navigation strategy. Based on the obstacle density, a new reward function is designed to drive the car to find a faster path with fewer collisions and turns.

[0053] (4) This invention introduces obstacle density into the state space, comprehensively considering the distribution of obstacles ahead, and optimizes the safety of obstacle avoidance decisions. In the Actor-Critic network, historical time series and obstacle density features are extracted to solve the cyclic path problem in long trajectory scenarios. Furthermore, an obstacle avoidance penalty based on obstacle density perception is designed, and obstacle avoidance penalty, distance penalty, angle penalty, and step size penalty are integrated into the reward function. This effectively improves the success rate in complex ship cabin environments, resulting in safer and more efficient movement trajectories. Using this invention, unmanned cleaning vehicles performing cleaning tasks in large cargo holds achieve more accurate navigation. Attached Figure Description

[0054] Figure 1 This is a schematic diagram showing the positions of the vehicle and the target point;

[0055] Figure 2 A simplified schematic diagram of the model;

[0056] Figure 3 The flowchart is a coordinated control method for autonomous navigation of an unmanned cleaning vehicle based on the ST-TD3 strategy proposed in this invention.

[0057] Figure 4 This is a schematic diagram of the model training topology in the embodiment;

[0058] Figure 5 For the experimental environment;

[0059] Figure 6 For the average reward of the action decision model to converge;

[0060] Figure 7 This demonstrates the performance of the converged action decision model. Detailed Implementation

[0061] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0062] This invention proposes a coordinated control method for autonomous navigation of an unmanned cleaning vehicle based on the ST-TD3 strategy, using an action decision model to determine the current state of the unmanned cleaning vehicle. The system processes the data and generates actions to control the speed and yaw angle of the unmanned cleaning vehicle.

[0063] Specifically, the action decision model proposed in this invention includes a gated loop module and an Actor network. The gated loop module controls the current state of the unmanned cleaning vehicle. The process involves extracting temporal features; the Actor network then generates actions based on these temporal features.

[0064] In this method, the state ,action and reward function The definition is as follows.

[0065] (a) Status

[0066] The state space of the unmanned cleaning vehicle (hereinafter referred to as the vehicle) includes: environmental depth features, obstacle density features, relative distance to the target point, and relative azimuth angle.

[0067] The environmental depth features are obtained by: acquiring raw depth images using the depth camera of the unmanned cleaning vehicle and then rasterizing them. × The depth feature matrix M is obtained, where each element of M is the minimum depth value within the corresponding grid cell of the original depth image; then, each row of M is concatenated to obtain a one-dimensional depth feature vector. It is used to characterize the depth features of the environment.

[0068] obstacle density characteristics The method for obtaining the obstacle density features is as follows: Gaussian weighting is applied to the depth feature matrix M to obtain the obstacle density features. It is used to perceive the overall distribution of obstacles ahead, and the calculation formula is as follows:

[0069] ;

[0070] ;

[0071] ;

[0072] in, and A weighted vector that satisfies a normal distribution; for The One element, for No. One element; To set the variance, the value range is [0.5, 3], and in subsequent embodiments, it is taken as 1; The smaller the value, the sharper the distribution curve.

[0073] Reference Figure 1 The relative distance between the unmanned cleaning vehicle and the target point is denoted as :

[0074] ;

[0075] in, The absolute coordinates of the current position of the unmanned cleaning vehicle are provided by the vehicle's built-in positioning system; These are the absolute coordinates of the target point, obtained from environmental monitoring.

[0076] The relative azimuth angle between the unmanned cleaning vehicle and the target point is denoted as :

[0077] ;

[0078] in, The yaw angle of the unmanned cleaning vehicle is provided by the vehicle's built-in sensors.

[0079] Thus, the unmanned cleaning cart is in a state of... ={ , , , }

[0080] In practice, to facilitate model training, the state variables can be normalized before constructing the state. , , , and These are deep feature vectors. Obstacle density characteristics Relative distance and relative azimuth The normalization result;

[0081] ;

[0082] ;

[0083] ;

[0084] ;

[0085] ;

[0086] in, for The One element, for One item, , and They are respectively The 1st, 2nd and × Item element; This is the maximum range of the depth camera. This represents the maximum diagonal distance of the environment.

[0087] (ii) Actions

[0088] The motion space is determined by linear velocity and angular velocity It consists of two parts, namely .

[0089] Because the depth camera is mounted at the front of the automated cleaning vehicle, it can only obtain information about the environment in front of it; therefore, the automated cleaning vehicle is not allowed to move backward. The motion space is continuous. The range of values ​​is , This is the maximum linear velocity (i.e., the maximum linear velocity). The range of values ​​is , This is the maximum angular velocity (i.e., the maximum angular velocity). The negative sign indicates the opposite direction.

[0090] (III) Reward Function

[0091] After the unmanned cleaning vehicle performs an action, an immediate reward is calculated based on the new status and the result of the action to evaluate the quality of the action.

[0092] reward function ;

[0093] in, , , and These are distance penalty, collision penalty, angle penalty, and step size penalty.

[0094] Distance penalty If the unmanned cleaning vehicle reaches the target point, a large positive reward is given; otherwise, a penalty is imposed, with the penalty increasing the further away from the target point. The formula is as follows:

[0095] ;

[0096] in, As a preset constant, The radius of the unmanned cleaning vehicle; The value is a negative constant with a range of [-0.1, -5.0], and in subsequent embodiments, it is 0.3.

[0097] Collision Penalty A significant penalty is imposed when the unmanned cleaning vehicle collides with an obstacle; additionally, to guide the vehicle towards a safer area with fewer obstacles, a penalty is imposed based on the density of obstacles ahead; the calculation formula is as follows:

[0098] ;

[0099] in, This is the binary number of collision detection obtained from the simulation. A value of 1 indicates a collision, while a value of 0 indicates that no collision has occurred. These are preset constants; This represents the average depth of pixels within a 50×50 area in the center of the original depth image; This represents the safe distance between the unmanned cleaning vehicle and the obstacle, with a value ranging from [0.5, 1.2], in meters. In subsequent embodiments, the value is 1.0 meter. , It is a negative constant; The value range is [-0.5, -5.0]. The value range is [-0.1, -3.0], as will be discussed in subsequent embodiments. The value is -2.0. The value is -0.5; This indicates the density characteristics of obstacles.

[0100] Angle penalty To ensure the unmanned cleaning vehicle moves towards the target point while avoiding instability caused by frequent and large-amplitude turns, the following angle penalty is set:

[0101] ;

[0102] in, and These represent the yaw angles of the unmanned cleaning vehicle at the previous and current moments, respectively. The relative azimuth angle between the unmanned cleaning vehicle and the target point. and It is a negative constant; The value range is [-0.5, -3.0]. The value range is [-0.05, -0.5], as will be discussed in subsequent embodiments. The value is -1.0. The value is -0.2.

[0103] Step length penalty To enable the unmanned cleaning vehicle to reach the target point more quickly, a step length penalty is added. To set the value, subsequent embodiments will use -0.02.

[0104] Reference Figure 2 , Figure 3 The model is obtained through the following steps SD1-SD7:

[0105] SD1. Construct a basic model that corresponds one-to-one with the unmanned cleaning vehicle, which includes a gated loop module, an online Actor network, an online Critic network, an online Actor network, and a target Critic network. The online Actor network and the target Actor network have the same structure, and the online Critic network and the target Critic network have the same structure.

[0106] Gated loop module for state Extracting temporal features, the Actor online network generates actions based on these temporal features. The Actor target network is based on the state of the unmanned cleaning vehicle. Execute action at time The state after (i.e., the next state) generates an action. ,right After adding noise and cropping, the next action is obtained. ;Right now:

[0107] ;

[0108] Set transition term E= ;

[0109] ;

[0110] ;

[0111] in, Represents the boundary function. To cut noise, ,Right now for The random value on; c is the noise clipping boundary; and These are the lower and upper limits of the action, respectively; , ;

[0112] visible, , and Both are binary arrays corresponding to angular velocity and linear velocity. It is a univariate value; This is a binary array corresponding to angular velocity and linear velocity.

[0113] Critic online network is state-based and actions Output evaluation value, denoted as Q value in this example; the Critic target network is state-based. and actions Output the evaluation value, which is denoted as the target Q value in this example.

[0114] SD2, randomly sample the state of the unmanned cleaning vehicle from each basic model. The Actor online network processes the status. Process and generate actions The unmanned cleaning vehicle performs the following actions. The state after that is recorded as the next state. Combine with the next state Calculate rewards ; Constructing experience samples Stored in a fixed-size circular experience replay buffer D;

[0115] SD3. Determine if the number of iterations in step SD2 has been reached. An integer multiple of, i.e., newly added empirical samples in buffer D since initialization or the last update. Has the quantity reached Integer multiples of; Set value;

[0116] No, then return to step SD2;

[0117] If yes, then for each basic model, a batch of experience samples is randomly selected from the buffer D as the experience pool corresponding to that basic model, and then each basic model is updated on the corresponding experience pool.

[0118] The steps for updating the base model on the experience pool are as follows:

[0119] SD4. Draw N experience samples from the experience pool. The input base model is processed by the Critic online network. and Process the data to obtain an evaluation value; then apply the Actor target network to... Process the data to obtain the next action. ; Critic target network and Process the data to obtain an evaluation value; calculate the loss function. Used to update the Critic online network and copy the updated Critic online network parameters to the Critic target network;

[0120] ;

[0121] ;

[0122] in, and These represent the parameters of the Critic online network and the parameters of the Critic target network, respectively.

[0123] This indicates that the Critic target network is based on the input... and The evaluation value at that time, referred to as the target Q value; For the Actor target network, the input is The output at that time is processed by adding noise and cropping to obtain the motion; As a reward, As a discount factor, For empirical samples The corresponding estimated suppression; For the first A sample of experiences The corresponding estimated suppression; For Critic online network, the input is and Evaluation value at that time; Figure 3 middle For Critic online network, the input is and The evaluation value at that time is called the Q value.

[0124] SD5. Determine whether the number of updates to the Critic online network and the Critic target network is an integer multiple of d, where d is a set value;

[0125] No, then return to step SD2;

[0126] If yes, proceed to step SD6;

[0127] SD6. Perform gradient updates on the gated loop module and the Actor online network, and then perform sliding updates on the Actor target network and the Critic target network.

[0128] The formula for gradient updates in the Actor online network is:

[0129] ;

[0130] in, and This represents the state and action of the k-th sample. This indicates that the Critic online network uses the following input: and Evaluation value at that time; This indicates that the Actor online network has the following parameters: And the input is Actions generated in time; among them, The parameters represent the online network of Actors. This indicates that the Actor online network is targeting the state. The generated action; The objective function is denoted as .

[0131] The formula for sliding update of the Actor target network and the Critic target network is:

[0132] ;

[0133] ;

[0134] and These represent the current parameters of the Critic target network and the Actor target network, respectively. and These represent the updated parameters of the Critic target network and the Actor target network, respectively. To update the coefficient, it is set to 0.005 in this example.

[0135] SD7. Determine if the number of times the Actor's online network updates have reached the set value;

[0136] No, then return to step SD2;

[0137] If so, then the Actor online network is fixed as the action decision model.

[0138] When applying the application, the unmanned cleaning vehicle will be in a certain state. Input decision model to obtain action .

[0139] In practice, the buffer D is continuously updated based on the actions of each decision model; at set intervals, steps SD1-SD7 are executed to update the decision model to ensure effective coordinated control of each unmanned cleaning vehicle.

[0140] The following specific embodiments verify the above-mentioned robot autonomous navigation coordination control method based on the ST-TD3 strategy.

[0141] In this embodiment, the experimental steps are as follows:

[0142] Step 1: System Initialization and Environment Setup

[0143] Initialize the simulation environment for the unmanned cleaning vehicle, configuring its initial position, target point position, and obstacle distribution. Create an indoor environment with static obstacles within the Gazebo simulation platform. The unmanned cleaning vehicle is equipped with a forward-facing RGB-D camera to perceive environmental depth information. Set the maximum linear velocity of the unmanned cleaning vehicle. =2m / s, maximum angular velocity =1 rad / s.

[0144] Step 2: State Space Feature Extraction

[0145] In this embodiment, the state variables of the state space include: depth feature vector. Obstacle density characteristics The relative distance between the car and the target point and relative azimuth .

[0146] like Figure 2 As shown The coordinates of the unmanned cleaning vehicle. Given the coordinates of the target point, the relative position between the two can be represented as: .

[0147] The relative distance between the unmanned cleaning vehicle and the target point is:

[0148] .

[0149] The angle between the current orientation of the unmanned cleaning vehicle and the line connecting the unmanned cleaning vehicle and the target point is defined as the relative azimuth angle between the unmanned cleaning vehicle and the target point; [This is added to the state space]. This allows the unmanned cleaning vehicle to move towards the target point.

[0150] ;

[0151] in, and They represent × The minimum pixel depth of the first and second grid cells in the first row of the rasterized original depth image; express × The minimum pixel depth of the raster in the m-th row and n-th column of the rasterized original depth image.

[0152] In this embodiment, all obtained state variables are normalized to facilitate network training. That is, the states input to the Actor online network are normalized. The normalization result of the state variables is denoted as .

[0153] state The input is fed into a gated recurrent algorithm (GRU) for processing. Through update and reset gate mechanisms, the weight relationship between historical states and the current input is dynamically adjusted to capture the temporal dependencies in the decision-making process, thus solving the cyclic path problem in long-trajectory scenarios. Specifically, the GRU outputs a 128-dimensional temporal feature vector. The Actor online network receives the temporal features output by the GRU and generates continuous action vectors through fully connected layers.

[0154] In this invention, cyclical experience playback and network updates are used. The network update process is as follows: Figure 3 As shown, the state transition data is stored in the experience replay buffer D. When the amount of data updated in buffer D reaches a threshold, the network is updated according to the following process: 256 sets of data are randomly sampled from the buffer. Then, the target Q-value (i.e., the evaluation value output by the Critic target network) is calculated through the Critic target network. Then, a target policy smoothing technique is used, adding clipped Gaussian noise to the target action. After completion, the Critic online network loss is calculated and the parameters are updated using gradient descent. Specifically, the Actor network is updated once every two Critic updates, using a delayed update strategy and a soft update method to update the target network parameters and coefficients. =0.005;

[0155] Furthermore, in this embodiment, as Figure 4 As shown, to avoid overestimation, a dual Critic network structure is adopted, namely, two Critic target networks are designed in parallel, and the minimum target Q value calculated by the dual Critic target networks is used in the calculation in each update; two Critic online networks are designed in parallel, and the minimum Q value (evaluation value output by the Critic online network) calculated by the two Critic online networks is used in the calculation in each update.

[0156] This embodiment employs commonly used evaluation metrics in reinforcement learning, including Average Reward (AR) and three performance metrics for evaluating the autonomous navigation method of the unmanned cleaning vehicle: Success Rate (SR), Crash Rate (CR), and Lost Rate (LR). SR represents the percentage of times the unmanned cleaning vehicle successfully reaches the target point out of the total number of navigation attempts; CR represents the percentage of times the unmanned cleaning vehicle collides with obstacles out of the total number of navigation attempts; and LR represents the percentage of times the unmanned cleaning vehicle neither reaches the target point nor collides with obstacles out of the total number of navigation attempts. The calculation formula is as follows:

[0157] ;

[0158] In this embodiment, the training batch size is 5, meaning that each training session uses 5 training samples. The maximum number of training rounds, Indicates the first The average reward over a training round. The higher the SR value, the better the performance of the method.

[0159] In such Figure 5 In the simulation environment shown, the experimental results tested by this invention are as follows: Figure 6 , Figure 7 As shown in the diagram. In this simulation environment, slender cuboids are used to simulate obstacles that may be encountered inside the ship's cabin.

[0160] Figure 6 The convergence trend of the average reward AR during training is shown. It can be seen that in this embodiment, the average reward begins to converge after about 150 rounds, which proves the model training efficiency and convergence speed of the method of the present invention.

[0161] Figure 7 The success rate (SR), collision rate (CR), and loss rate (LR) of the converged action decision model are shown. The method of this invention achieves a loss rate of 0 and a success rate of 93.8%, which is excellent.

[0162] Of course, those skilled in the art will recognize that the present invention is not limited to the details of the exemplary embodiments described above, but also includes the same or similar structures that can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0163] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0164] The technologies, shapes, and structures not described in detail in this invention are all known technologies.

Claims

1. A robot autonomous navigation coordination control method based on the ST-TD3 strategy, characterized in that, The action decision model is trained based on the current state of the unmanned cleaning vehicle. Generate Actions The unmanned cleaning cart performs the action. To adjust speed and yaw angle; The status includes: the environmental depth features and obstacle density features of the environment in which the unmanned cleaning vehicle is located, as well as the relative distance and relative azimuth angle between the unmanned cleaning vehicle and the target point; the environmental depth features and obstacle density features are extracted based on the original depth images collected by the unmanned cleaning vehicle; The motion includes linear velocity and angular velocity, and both satisfy the constraints of the maximum linear velocity and maximum angular velocity of the unmanned cleaning vehicle, respectively. The angular velocity is represented by positive and negative values ​​to indicate the deflection direction. The obstacle density features are obtained as follows: First, the raw depth images collected by the unmanned cleaning vehicle are rasterized to obtain... × The depth feature matrix M; Obstacle density features are obtained by applying Gaussian weights to the depth feature matrix M. : in, and A weighted vector that satisfies a normal distribution; and It is an ordinal number; To define the variance, It is a natural number; The action decision model is trained using an Actor-Critic network, and the rewards used in training include distance penalties. Collision penalty Angle punishment and step length penalty Multiple items in the middle; step size penalty To set a value; As a preset constant, The radius of the unmanned cleaning vehicle; A negative constant is defined; The relative distance between the unmanned cleaning vehicle and the target point; The number of collision detection values ​​obtained from the simulation; These are preset constants; It represents the average depth of pixels within a specified range in the middle of the original depth image; This represents the safe distance between the unmanned cleaning vehicle and obstacles. , It is a negative constant; Indicates obstacle density characteristics; and These represent the yaw angles of the unmanned cleaning vehicle at the previous and current moments, respectively. The relative azimuth angle between the unmanned cleaning vehicle and the target point. and It is a negative constant.

2. The robot autonomous navigation coordination control method based on the ST-TD3 strategy as described in claim 1, characterized in that, Environmental depth features are represented by depth feature vectors Representation, deep feature vector is the row concatenation vector of the depth feature matrix M.

3. The robot autonomous navigation coordination control method based on the ST-TD3 strategy as described in claim 1 or 2, characterized in that, The steps to train an Actor-Critic network are as follows: Construct a basic model that corresponds one-to-one with the unmanned cleaning vehicle, which includes a gated loop module, an online Actor network, an online Critic network, an online Actor network, and a target Critic network; the online Actor network and the target Actor network have the same structure, and the online Critic network and the target Critic network have the same structure. Gated loop module for state Extracting temporal features, the Actor online network generates actions based on these temporal features. The Actor target network is based on the state of the unmanned cleaning vehicle. Execute action at time The state after Generate Actions ,right After adding noise and cropping, the next action is obtained. ; The parameters of the Actor target network are represented; the Critic online network is state-based. and actions Output evaluation value; the Critic target network is state-based. and actions Output evaluation value; Let each basic model randomly sample the state of the unmanned cleaning vehicle. Process and construct experience samples Stored in buffer D; The unmanned cleaning cart is in operation. Execute action at time The state after that, For state-based Calculated reward; Whenever experience sample Increase quantity setting value Then, a batch of experience samples are randomly selected for each basic model to form the experience pool corresponding to the basic model. Each basic model is updated locally on its corresponding experience pool until the number of basic model updates reaches a set value. Then, the gated loop module and the Actor online network are extracted to form an action decision model.

4. The robot autonomous navigation coordination control method based on the ST-TD3 strategy as described in claim 3, characterized in that, The method for locally updating the base model on the corresponding experience pool is as follows: SD4. Draw N experience samples from the experience pool. The input base model is processed, and the loss function is calculated based on the evaluation values ​​output by the Critic online network and the Critic target network. The Critic online network is updated through backpropagation using the loss function, and the updated Critic online network parameters are copied to the Critic target network. SD5. Determine whether the number of updates to the Critic online network and the Critic target network is an integer multiple of d, where d is a set value; No, then return to update buffer D; If yes, proceed to step SD6; SD6. The gated loop module and the Actor online network are updated with gradients based on the evaluation values ​​generated by the Critic online network, and then the Actor target network and the Critic target network are updated with sliding updates. SD7. Determine if the number of times the Actor's online network updates have reached the set value; No, then return to update buffer D; Yes, then the fixed-gated loop module and the Actor online network form an action decision model.

5. The robot autonomous navigation coordination control method based on the ST-TD3 strategy as described in claim 4, characterized in that, The loss function is calculated in step SD4 as follows: in, For the first A sample of experiences The corresponding estimated suppression, For Critic online network, the input is and The evaluation value generated at that time For the Critic target network, the input is and The evaluation value generated at that time For the Actor target network, the input is The output at that time is processed by adding noise and cropping to obtain the motion; and These represent the parameters of the Critic online network and the parameters of the Critic target network, respectively. is the discount factor, and N is the sample size.

6. The robot autonomous navigation coordination control method based on the ST-TD3 strategy as described in claim 3, characterized in that, right After adding noise and cropping, the next action is obtained. The method is as follows: in, Represents the boundary function. To trim noise, and ; and These are the lower and upper limits of the action, respectively; , , and represents the maximum speed and maximum angular velocity of the unmanned cleaning vehicle, respectively; c is the noise clipping boundary.

7. The robot autonomous navigation coordination control method based on the ST-TD3 strategy as described in claim 3, characterized in that, The states that serve as inputs to the Actor online network and action decision model are obtained by normalizing the state variables; the state variables are environmental depth features, obstacle density features, relative distance, and relative azimuth.

8. A robot autonomous navigation coordination control system based on the ST-TD3 strategy, characterized in that, It includes a memory and a processor. The memory stores a computer program, and the processor is connected to the memory. The processor is used to execute the computer program to implement the robot autonomous navigation coordination control method based on the ST-TD3 strategy as described in any one of claims 1-7.

9. A storage medium, characterized in that, The system contains a computer program that, when executed, implements the robot autonomous navigation coordination control method based on the ST-TD3 strategy as described in any one of claims 1-7.

Images