Training method of navigation model and navigation method

Abstract

The invention provides a training method of a navigation model and a navigation method. The navigation model training method comprises the following steps: acquiring a first high-precision global map and an initial deep learning model; acquiring state data and radar data of the first robot at the current moment; calculating a target reward value according to the state data and the first high-precision global map; determining a first input data set and a second input data set according to the state data, the radar data and the first high-precision global map; processing the first input data set and the second input data set by using the initial deep learning model to obtain a target loss function; adjusting the weight of the initial deep learning model by using the target loss function and the target reward value to obtain a first deep learning model; the first deep learning model comprises a first Actor network and a first Critic network; updating the current training times; detecting whether the current training frequency is greater than a training frequency threshold; and taking the first Actor network as a navigation model under the condition that the first Actor network is determined to be larger than the first Actor network.

Description

Technical Field

[0001] The present application relates to the field of robot navigation control technology, and in particular to a training method for a navigation model and a navigation method. Background Art

[0002] The trained deep learning model can be used to output the navigation information of the robot at different times to achieve navigation control of the robot. Common deep learning models include Actor networks and Critic networks. The Actor network is used to predict and output the navigation information at the current moment, and the Critic network is used to evaluate the accuracy of the navigation information. In the prior art, the PPO (Proximal Policy Optimization) algorithm is usually used to construct a loss function, and the objective function is used to train the Actor network and the Critic network. The loss function used in this way is simple in structure and does not fully utilize the feature information of the input data. Therefore, the trained deep learning model has low accuracy. Summary of the invention

[0003] The present application provides a navigation model training method and a navigation method to solve the technical problem of being unable to accurately perform robot navigation control.

[0004] In a first aspect, the present application provides a method for training a navigation model, the method comprising:

[0005] Obtain a first high-precision global map and an initial deep learning model; obtain the current state data and radar data of the first robot;

[0006] Calculating a target reward value at the current moment according to the current state data and the first high-precision global map;

[0007] Determine a first input data set and a second input data set according to the state data at the current moment, the radar data at the current moment, and the first high-precision global map;

[0008] Processing the first input data set and the second input data set using the initial deep learning model to obtain a target loss function;

[0009] Using the target loss function and the target reward value to adjust the weight of the initial deep learning model, a first deep learning model is obtained; wherein the first deep learning model includes a first Actor network and a first Critic network;

[0010] Update the current number of training times;

[0011] Check whether the current number of training times is greater than the training times threshold;

[0012] When it is determined that the current number of training times is greater than the training times threshold, the first Actor network is used as the navigation model.

[0013] In some implementations, the first high-precision global map includes a target obstacle scene; the target obstacle scene includes at least one of the following: a threshold with a height less than a first threshold, a channel with a width less than a second threshold, a channel closed at one end, a scene with multiple obstacles densely stacked, and a scene with multiple obstacles discretely distributed.

[0014] In some implementations, the state data includes: current position, noise-free navigation information, noisy navigation information, velocity vector, maximum forward speed, and collision force information of the first robot; and the radar data includes: noisy radar data and noise-free radar data.

[0015] In some implementations, calculating the target reward value at the current moment according to the state data at the current moment and the first high-precision global map includes:

[0016] Obtaining a first low-precision global map using the first high-precision global map;

[0017] Selecting a first target destination of the first robot from the first low-precision global map;

[0018] According to the first target endpoint, selecting a first global sampling point from the first low-precision global map;

[0019] Calculating a target endpoint reward value according to the first global sampling point and the current position;

[0020] Calculate a dense reward value according to the first global sampling point, the current position, the velocity vector, and the maximum forward speed;

[0021] Calculating a collision reward value according to the collision force information of the first robot;

[0022] Perform a weighted sum operation on the target endpoint reward value, dense reward value, and collision reward value to obtain the target reward value at the current moment.

[0023] In some implementations, selecting a first global sampling point from the first low-precision global map according to the first target endpoint includes:

[0024] According to the first target destination, path planning is performed on the first low-precision global map to obtain a first global path;

[0025] Sampling is performed according to a preset sampling interval, and two first global sampling points are selected from the first global path.

[0026] In some implementations, determining the first input data set and the second input data set according to the current state data, the current radar data, and the first high-precision global map includes:

[0027] From the first high-precision global map, extract a circular area with the first robot as the center as a local map;

[0028] Determine noise-free distance information and noise-containing distance information between the current position and the first target end point;

[0029] combining the noisy distance information, the noisy navigation information, the first global sampling point, and the noisy radar data as a first input data set;

[0030] The noise-free distance information, the noise-free navigation information, the first global sampling point, the noise-free radar data, and the local map are combined as a second input data set.

[0031] In some implementations, the initial deep learning model includes an initial Actor network and an initial Critic network;

[0032] The initial Actor network includes: a first convolutional neural network model, a second convolutional neural network model, a first multi-layer perceptron, a second multi-layer perceptron, and a gated recurrent unit;

[0033] The initial critic network includes: a third convolutional neural network model, a fourth convolutional neural network model, a fifth convolutional neural network model, a third multilayer perceptron, a fourth multilayer perceptron, and a fifth multilayer perceptron.

[0034] In some implementations, processing the first input data set and the second input data set using the initial deep learning model to obtain a target loss function includes:

[0035] Inputting the first input data set into the initial Actor network to obtain a first hidden state of the initial Actor network at a current moment;

[0036] Inputting the second input data set into the initial critic network to obtain the second hidden state of the initial critic network at the current moment;

[0037] The target loss function of the initial deep learning model is constructed using the first hidden state at the current moment and the second hidden state at the current moment.

[0038] In some implementations, inputting the first input data set into the initial Actor network to obtain a first hidden state of the initial Actor network at a current moment includes:

[0039] Obtain the predicted navigation information of the previous moment and the first hidden state of the previous moment;

[0040] Splicing the noisy distance information and the noisy navigation information to obtain a first splicing result;

[0041] Inputting the first concatenation result into the first multilayer perceptron to obtain a first calculation result;

[0042] Inputting the first global sampling point into the first convolutional neural network model for feature extraction to obtain a first eigenvalue;

[0043] Inputting the noisy radar data into the second convolutional neural network model for feature extraction to obtain a second eigenvalue;

[0044] splicing the first calculation result, the first eigenvalue, the second eigenvalue, and the predicted navigation information at the previous moment to obtain a second splicing result;

[0045] The second concatenation result and the first hidden state at the previous moment are input into the gated recurrent unit to obtain the first hidden state of the initial Actor network at the current moment.

[0046] In some implementations, after obtaining the first hidden state of the initial Actor network at the current moment, the method further includes: inputting the first hidden state at the current moment into the second multilayer perceptron to obtain the predicted navigation information at the current moment.

[0047] In some implementations, inputting the second input data set into the initial critic network to obtain the second hidden state of the initial critic network at the current moment includes:

[0048] splicing the noise-free distance information and the noise-free navigation information to obtain a third splicing result;

[0049] Inputting the third concatenation result into the third multilayer perceptron to obtain a second calculation result;

[0050] Inputting the local map into the third convolutional neural network model for feature extraction to obtain a third eigenvalue;

[0051] Inputting the first global sampling point into the fourth convolutional neural network model for feature extraction to obtain a fourth eigenvalue;

[0052] Inputting the noise-free radar data into the fifth convolutional neural network model for feature extraction to obtain a fifth eigenvalue;

[0053] The second calculation result, the third eigenvalue, the fourth eigenvalue, and the fifth eigenvalue are converted into one-dimensional vectors and then concatenated to obtain a fourth concatenation result;

[0054] The fourth concatenation result is input into the fourth multilayer perceptron to obtain the second hidden state of the initial Critic network at the current moment.

[0055] In some implementations, the method further includes: inputting the second hidden state at the current moment into the fifth multilayer perceptron to obtain a state value function of the initial Critic network.

[0056] In some implementations, after detecting whether the current number of training times is greater than the training times threshold, the method further includes:

[0057] When it is determined that the current number of training times is less than or equal to a training times threshold, using the first deep learning model as an initial deep learning model;

[0058] The state data of the first robot at the next moment and the radar data at the next moment are obtained as the state data at the current moment and the radar data at the current moment.

[0059] In a second aspect, the present application further proposes a navigation method, the method comprising:

[0060] Obtaining the noisy navigation information, the noisy radar data, and the current position of the second robot at the current moment;

[0061] Acquire a second low-precision global map and a navigation model; the navigation model is trained by any navigation model training method;

[0062] Selecting a second global sampling point and a second target endpoint from the second low-precision global map;

[0063] Determining noisy distance information between the current position and the second target end point;

[0064] Inputting noisy navigation information, noisy radar data, a second global sampling point, and noisy distance information into the navigation model to obtain predicted navigation information at the current moment;

[0065] A control instruction is generated according to the predicted navigation information at the current moment; and the control instruction is used to perform navigation control on the motion behavior of the second robot at the next moment.

[0066] In some implementations, selecting a second global sampling point and a second target end point from the second low-precision global map includes:

[0067] Selecting a second target destination from the second low-precision global map;

[0068] According to the second target destination, performing path planning on the second low-precision global map to obtain a second global path;

[0069] Sampling is performed according to a preset sampling interval, and two second global sampling points are selected from the second global path.

[0070] In a third aspect, the present application proposes a training device for a navigation model, comprising:

[0071] A first acquisition module is used to acquire a first high-precision global map and an initial deep learning model; and to acquire the current state data and radar data of the first robot.

[0072] A first calculation module, used for calculating a target reward value at a current moment according to the state data at the current moment and the first high-precision global map;

[0073] A construction module, configured to determine a first input data set and a second input data set according to the state data at the current moment, the radar data at the current moment, and the first high-precision global map;

[0074] A processing module, configured to process the first input data set and the second input data set using the initial deep learning model to obtain a target loss function;

[0075] An adjustment module, configured to adjust the weight of the initial deep learning model using the target loss function and the target reward value to obtain a first deep learning model; wherein the first deep learning model includes a first Actor network and a first Critic network;

[0076] Update module, used to update the current training times;

[0077] A detection module is used to detect whether the current number of training times is greater than a training times threshold;

[0078] The generation module is used to use the first Actor network as a navigation model when it is determined that the current training number is greater than a training number threshold.

[0079] In a fourth aspect, the present application proposes a navigation device, comprising:

[0080] A second acquisition module is used to acquire the noisy navigation information, the noisy radar data, and the current position of the second robot at the current moment;

[0081] A third acquisition module is used to acquire a second low-precision global map and a navigation model; the navigation model is trained by any navigation model training method;

[0082] A selection module, used for selecting a second global sampling point and a second target end point from the second low-precision global map;

[0083] A second calculation module, used to determine the noise-containing distance information between the current position and the second target end point;

[0084] A prediction module, used for inputting the noisy navigation information, the noisy radar data, the second global sampling point, and the noisy distance information into the navigation model to obtain the predicted navigation information at the current moment;

[0085] The control module is used to generate control instructions according to the predicted navigation information at the current moment; and use the control instructions to perform navigation control on the movement behavior of the second robot at the next moment.

[0086] In a fifth aspect, the present application proposes an electronic device, comprising a processor and a memory for storing instructions executable by the processor, wherein when the processor executes the instructions, the processor implements the steps of any one of the above-mentioned navigation model training methods, or the steps of any one of the above-mentioned navigation methods.

[0087] In a sixth aspect, the present application further proposes a storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of any of the above-mentioned navigation model training methods, or the steps of any of the above-mentioned navigation methods.

[0088] The present application can achieve the following beneficial effects: the present application uses a reinforcement learning algorithm to construct a target loss function using hidden states, uses the target loss function to pre-construct a navigation model, and uses the navigation model in actual scenarios to achieve navigation control of the robot; further, the navigation model can be applied to a variety of complex terrains; and the navigation model does not require the input of a high-precision map but only a low-precision map, which effectively reduces the difficulty of map construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0090] Figure 1A flowchart of a navigation model training method provided in an embodiment of the present application;

[0091] Figure 2 A schematic diagram of the structure of an initial deep learning model provided in an embodiment of the present application;

[0092] Figure 3 A flowchart of a navigation method provided in an embodiment of the present application;

[0093] Figure 4 A schematic diagram of the structure of a navigation model training device provided in an embodiment of the present application;

[0094] Figure 5 A schematic structural diagram of a navigation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0095] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0096] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally indicates that the associated objects before and after are in an "or" relationship.

[0097] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0098] The trained deep learning model can be used to output the navigation information of the robot at different times to achieve navigation control of the robot. Common deep learning models include Actor networks and Critic networks. The Actor network is used to predict and output the navigation information at the current moment, and the Critic network is used to evaluate the accuracy of the navigation information. In the prior art, the PPO (Proximal Policy Optimization) algorithm is usually used to construct a loss function, and the objective function is used to train the Actor network and the Critic network. The loss function used in this way is simple in structure and does not fully utilize the feature information of the input data. Therefore, the trained deep learning model has low accuracy.

[0099] Furthermore, in the prior art, high-precision maps are usually required to achieve accurate navigation control. However, building high-precision maps is difficult and costly, so how to use low-precision maps to achieve navigation control is an urgent problem to be solved.

[0100] This application proposes a navigation model training method and a navigation method, which can use the target loss function to improve the training accuracy of the model, thereby achieving better navigation effects.

[0101] The following is an introduction to a navigation model training method and a navigation method provided by the present application in conjunction with the accompanying drawings.

[0102] In some embodiments, please refer to Figure 1 The present application proposes a method for training a navigation model. It should be noted that the method for training a navigation model is implemented in a computer simulation environment (for example, a deep learning model is trained in a simulation environment in the form of sim2real), and the training data used are all simulation data. In specific implementation, the method may include the following contents.

[0103] S101: Obtain a first high-precision global map and an initial deep learning model; obtain the current state data and radar data of the first robot.

[0104] In some embodiments, the first robot is not a real robot but a simulation object. The first robot moves in the first high-precision global map, avoiding obstacle scenes or obstacles while moving from a starting point to an end point (the end point of the first robot is recorded as the first target end point), so the first robot needs to be navigated and controlled.

[0105] In some embodiments, the first high-precision global map includes a target obstacle scene; the target obstacle scene includes at least one of the following: a threshold with a height less than a first threshold, a channel with a width less than a second threshold, a channel closed at one end, a scene with multiple obstacles densely stacked, and a scene with multiple obstacles discretely distributed.

[0106] In some embodiments, a first high-precision global map including the target obstacle scene can be constructed through a wave function collapse algorithm.

[0107] In some embodiments, the first threshold may be set to 2 cm. Further, the height of the threshold may be set to be greater than 1 cm.

[0108] In some embodiments, the second threshold value can be set to a value greater than the diameter of the first robot. For example, assuming that the diameter of the first robot is 20 cm, the second threshold value is set to 30 cm. A passage with a width less than the second threshold value is relatively narrow, which is not conducive to the passage of the first robot.

[0109] Based on the above embodiments, a variety of target obstacle scenarios can be arranged in the first high-precision global map to provide complex and diverse terrain conditions for the subsequent training of the deep learning model, so that the trained deep learning model (i.e., the navigation model) can be adapted to different obstacle scenarios, thereby improving the applicability and generalization of the deep learning model, and being able to complete navigation tasks in most common scenarios in daily life.

[0110] In some embodiments, the first high-precision global map is a grid map, and the target obstacle scene in the first high-precision global map can also be expanded, and the expansion radius is equal to the diameter of the first robot, that is, the first robot is regarded as a particle, and each target obstacle scene is expanded outward by the distance of the expansion radius to obtain the expanded first high-precision global map. In the expanded first high-precision global map, the grids not occupied by the target obstacle scenes are the grids that the first robot can pass.

[0111] In some embodiments, the state data includes: current position, noise-free navigation information, noise-containing navigation information, velocity vector, maximum forward speed, and collision force information of the first robot.

[0112] In some embodiments, the current position is the coordinate information of the first robot at the current moment.

[0113] In some embodiments, the noise-free navigation information includes: a unit vector (sinα, cosα) corresponding to a noise-free simulated yaw angle (denoted as α), a noise-free simulated linear velocity, and a noise-free simulated angular velocity.

[0114] In some embodiments, the noisy navigation information includes: a unit vector (sinβ, cosβ) corresponding to a noisy simulated yaw angle (denoted as β), a noisy simulated linear velocity, and a noisy simulated angular velocity.

[0115] In some embodiments, the velocity vector refers to the velocity vector of the first robot in the world coordinate system.

[0116] In some embodiments, when the first robot does not collide with the target obstacle scene, the collision force information of the first robot is equal to 0; when the first robot collides with the target obstacle scene, the collision force information of the first robot is equal to the collision external force received by the first robot.

[0117] In some embodiments, a simulated laser radar is arranged on the first robot for generating radar data.

[0118] In some embodiments, the radar data includes: noisy radar data and noise-free radar data. The noisy radar data is a simulated laser radar range ray containing noise. The noise-free radar data is a simulated laser radar range ray without noise.

[0119] In some embodiments, radar data may be used during model training to avoid obstacles that do not appear in the map.

[0120] S102: Calculate the target reward value at the current moment according to the state data at the current moment and the first high-precision global map.

[0121] In some embodiments, the target reward value indicates the navigation quality of the first robot at the current moment, and also indicates the training effect at the current moment. The larger the target reward value, the better the training effect.

[0122] In some embodiments, the target reward value at the current moment is calculated according to the state data at the current moment and the first high-precision global map, specifically including:

[0123] S1: Obtain a first low-precision global map using the first high-precision global map;

[0124] S2: Selecting a first target destination of the first robot from the first low-precision global map;

[0125] S3: selecting a first global sampling point from the first low-precision global map according to the first target endpoint;

[0126] S4: Calculating a target endpoint reward value according to the first global sampling point and the current position;

[0127] S5: Calculate a dense reward value according to the first global sampling point, the current position, the velocity vector, and the maximum forward speed;

[0128] S6: Calculating a collision reward value according to the collision force information of the first robot;

[0129] S7: Perform a weighted sum operation on the target endpoint reward value, the dense reward value, and the collision reward value to obtain the target reward value at the current moment.

[0130] In some embodiments, taking the first high-precision global map of 1cm×1cm as an example, a 5cm window is set, and the window is controlled to move on the first high-precision global map. The maximum value of the raster data in the window is taken each time to form a first low-precision global map of 5cm×5cm.

[0131] In some embodiments, the endpoint of the first robot is selected as the first target endpoint.

[0132] In some embodiments, according to the first target endpoint, selecting a first global sampling point from the first low-precision global map specifically includes:

[0133] S1: performing path planning on the first low-precision global map according to the first target destination to obtain a first global path;

[0134] S2: performing sampling according to a preset sampling interval, and selecting two first global sampling points from the first global path.

[0135] In some embodiments, the Dijkstra algorithm can be used to implement path planning and generate a first global path. The first global path is the shortest path connecting the current position and the first target destination. The first global path is obtained through the state data at the current moment, and the accuracy is very low, and the closer to the first target destination, the lower the accuracy. Taking the current position as the sampling starting point, the sampling interval is set to 0.5 meters. Starting from the sampling starting point in the first global path, 2 points are taken forward at an interval of 0.5 meters to obtain two first global sampling points. The first global sampling point can provide guidance information with higher accuracy, which can improve the efficiency of reaching the first target destination.

[0136] In some embodiments, the current position may be projected into the first global path, and sampling may be performed using the projection point as a sampling starting point.

[0137] In some embodiments, the “target endpoint reward value” represents a sparse reward that encourages the first robot to track the first global sampling point. The target endpoint reward value may be calculated according to the following formula:

[0138]

[0139] Among them, r g1 represents the target end point reward value, represents the first global sampling point, p robot Indicates the current location.

[0140] In some embodiments, the “dense reward value” represents a dense reward for encouraging the first robot to track the first first global sampling point. The dense reward value may be calculated according to the following formula:

[0141]

[0142] Among them, r g2 Represents a dense reward value. The clip() function limits the value input to the function to a preset range. world represents the velocity vector, express The unit vector, v max Indicates the maximum forward speed.

[0143] In some embodiments, the "collision reward value" represents a reward for punishing the first robot for colliding with an obstacle or obstacle scene. The collision reward value may be calculated according to the following formula:

[0144]

[0145] Among them, r c represents the collision reward value, and F represents the collision force information of the first robot.

[0146] In some embodiments, a weighted sum operation is performed on the target endpoint reward value, the dense reward value, and the collision reward value to obtain the target reward value at the current moment, specifically including: setting positive weights for the target endpoint reward value and the dense reward value, setting a negative weight for the collision reward value, and then performing weighted summation to obtain the target reward value at the current moment.

[0147] S103: Determine a first input data set and a second input data set according to the status data at the current moment, the radar data at the current moment, and the first high-precision global map.

[0148] In some embodiments, the first input data set includes data containing noise; and the second input data set includes noise-free data.

[0149] In some embodiments, determining a first input data set and a second input data set according to the current state data, the current radar data, and the first high-precision global map specifically includes:

[0150] S1: extracting a circular area with the first robot as the center from the first high-precision global map as a local map;

[0151] S2: Determine noise-free distance information and noise-containing distance information between the current position and the first target end point;

[0152] S3: combining the noisy distance information, the noisy navigation information, the first global sampling point, and the noisy radar data as a first input data set;

[0153] S4: Combine the noise-free distance information, the noise-free navigation information, the first global sampling point, the noise-free radar data, and the local map as a second input data set.

[0154] In some embodiments, in the first high-precision global map, a region with a radius of 1 m is cut out with the first robot as the center to obtain a local map. Alternatively, a square region with a size of 1 m×1 m is cut out with the first robot as the center to obtain a local map. The resolution of the local map is 1 cm.

[0155] In some embodiments, the distance information includes: a unit vector of the first target endpoint relative to the current position, and a Euclidean distance from the first target endpoint to the current position.

[0156] In some embodiments, the noise-free distance information may be calculated first, and then some noise may be added to form the noisy distance information.

[0157] S104: Use the initial deep learning model to process the first input data set and the second input data set to obtain a target loss function.

[0158] In some embodiments, the initial deep learning model includes an initial Actor network and an initial Critic network. The initial Actor network is used to predict and output navigation information at the current moment; the initial Critic network is used to evaluate the accuracy of the navigation information, and then to assist in adjusting the initial Actor network.

[0159] In some embodiments, see Figure 2 As shown, the initial Actor network includes: a first convolutional neural network model (Convolutional Neural Networks, CNN), a second convolutional neural network model, a first multilayer perceptron (Multilayer Perceptron, MLP), a second multilayer perceptron, and a gated recurrent unit (Gate Recurrent Unit, GRU).

[0160] In some embodiments, see Figure 2 As shown, the initial Critic network includes: a third convolutional neural network model, a fourth convolutional neural network model, a fifth convolutional neural network model, a third multilayer perceptron, a fourth multilayer perceptron, and a fifth multilayer perceptron.

[0161] In some embodiments, processing the first input data set and the second input data set using the initial deep learning model to obtain a target loss function specifically includes:

[0162] S1: Input the first input data set into the initial Actor network to obtain the first hidden state of the initial Actor network at the current moment;

[0163] S2: Input the second input data set into the initial critic network to obtain the second hidden state of the initial critic network at the current moment;

[0164] S3: Construct the target loss function of the initial deep learning model using the first hidden state at the current moment and the second hidden state at the current moment.

[0165] In some embodiments, the current time is recorded as t. The hidden state of the initial Actor network can represent the feature information of the first input data set, recorded as the first hidden state h t (θ), θ represents the weight of the initial Actor network. The hidden state of the initial Critic network can represent the feature information of the second input data set, which is recorded as the second hidden state

[0166] In some embodiments, see Figure 2 As shown, the first input data set is input into the initial Actor network to obtain the first hidden state of the initial Actor network at the current moment, specifically including:

[0167] S1: Obtain the predicted navigation information of the previous moment and the first hidden state of the previous moment;

[0168] S2: splicing the noisy distance information and the noisy navigation information to obtain a first splicing result;

[0169] S3: inputting the first concatenation result into the first multilayer perceptron to obtain a first calculation result;

[0170] S4: inputting the first global sampling point into the first convolutional neural network model for feature extraction to obtain a first eigenvalue; the first convolutional neural network model is a one-dimensional model;

[0171] S5: Inputting the noisy radar data into the second convolutional neural network model for feature extraction to obtain a second eigenvalue; the second convolutional neural network model is a one-dimensional model;

[0172] S6: splicing the first calculation result, the first eigenvalue, the second eigenvalue, and the predicted navigation information at the previous moment to obtain a second splicing result;

[0173] S7: Input the second concatenation result and the first hidden state at the previous moment into the gated recurrent unit to obtain the first hidden state of the initial Actor network at the current moment.

[0174] In some embodiments, see Figure 2 As shown, after obtaining the first hidden state of the initial Actor network at the current moment, the method further includes: inputting the first hidden state at the current moment into the second multilayer perceptron to obtain the predicted navigation information at the current moment.

[0175] In some embodiments, the "predicted navigation information" is the output result of the initial Actor network (or the first initial Actor network), including the linear velocity prediction value and angular velocity prediction value of the first robot (or the second robot). The "predicted navigation information" is used to perform navigation control on the robot's behavior at the next moment.

[0176] In some embodiments, the specific definition of the linear velocity prediction value is "the linear velocity prediction value of the x-axis in the body coordinate system of the first robot (or the second robot)". The specific definition of the angular velocity prediction value is "the angular velocity prediction value of the yaw angle of the first robot (or the second robot) rotating around the z-axis". Here, the coordinate system of the first robot (or the second robot) is defined as the x-axis pointing to the front of the first robot (or the second robot), the y-axis pointing to the left, and the z-axis pointing to the top.

[0177] In some embodiments, if the current moment is the initial moment and there is no previous moment, the predicted navigation information of the previous moment can be set equal to 0, and the first hidden state of the previous moment can be set equal to 0.

[0178] In some embodiments, see Figure 2 As shown, the second input data set is input into the initial Critic network to obtain the second hidden state of the initial Critic network at the current moment, specifically including:

[0179] S1: splicing the noise-free distance information and the noise-free navigation information to obtain a third splicing result;

[0180] S2: Inputting the third concatenation result into the third multilayer perceptron to obtain a second calculation result;

[0181] S3: Inputting the local map into the third convolutional neural network model for feature extraction to obtain a third eigenvalue; the third convolutional neural network model is a two-dimensional model;

[0182] S4: inputting the first global sampling point into the fourth convolutional neural network model for feature extraction to obtain a fourth eigenvalue; the fourth convolutional neural network model is a one-dimensional model;

[0183] S5: inputting the noise-free radar data into the fifth convolutional neural network model for feature extraction to obtain a fifth eigenvalue; the fifth convolutional neural network model is a one-dimensional model;

[0184] S6: converting the second calculation result, the third eigenvalue, the fourth eigenvalue, and the fifth eigenvalue into one-dimensional vectors and then concatenating them to obtain a fourth concatenation result;

[0185] S7: Input the fourth concatenation result into the fourth multilayer perceptron to obtain the second hidden state of the initial critic network at the current moment.

[0186] In some embodiments, see Figure 2 As shown, the method also includes: inputting the second hidden state at the current moment into the fifth multilayer perceptron to obtain the state value function of the initial Critic network; the state value function is recorded as V(s).

[0187] In some embodiments, the state value function may be used to adjust the initial Actor network to improve the training accuracy of the initial Actor network.

[0188] In some embodiments, the target loss function of the initial deep learning model is constructed using the first hidden state at the current moment and the second hidden state at the current moment, specifically including: constructing the target loss function according to the mean square error between the first hidden state and the second hidden state.

[0189] In some embodiments, the target loss function may be determined according to the following formula:

[0190]

[0191] Among them, θ represents the weight of the initial Actor network, represents the weight of the initial Critic network, h t (θ) represents the first hidden state, represents the second hidden state, the MSE() function represents the mean square error, L aux (θ) represents the auxiliary loss function, Represents the basic loss function obtained using the PPO algorithm, represents the target loss function.

[0192] In some embodiments, the basic loss function may be of the form:

[0193]

[0194] In the formula, π θ Represents the initial Actor network; represents the initial Critic network; t represents the current moment; γ represents the discount factor; k represents the number of training times; N represents the number of time steps; r represents the target reward value; π θ (a|s) represents the current conditional probability function, a is the action (the action in this application is the predicted navigation information), s is the state (the state in this application is the first input data set and the second input data set); represents the conditional probability function of the kth optimization; ∈ represents the hyperparameter.

[0195] Based on the above embodiment, the first hidden state of the Actor network is used to predict the second hidden state of the Critic network, helping the Actor network to use noisy and partially observable input data (first input data set) to predict noise-free output data (predict navigation information).

[0196] S105: Using the target loss function and the target reward value to adjust the weight of the initial deep learning model to obtain a first deep learning model; wherein the first deep learning model includes a first Actor network and a first Critic network.

[0197] Based on the objective loss function and reinforcement learning algorithm, by calculating the mean square loss between the first hidden state and the second hidden state, and updating the weights of the initial Actor network and the initial Critic network through gradient backpropagation, this can effectively help the initial Actor network predict global navigation information from the noisy and partially observable first input data set, and it also has good robustness in noise resistance.

[0198] In some embodiments, the weight of the initial Actor network and the weight of the initial Critic network may be adjusted according to the following formula:

[0199]

[0200] Among them, θ k+1 represents the adjusted weight of the initial Actor network, Represents the weight of the adjusted initial Critic network.

[0201] In some embodiments, the adjusted initial Actor network is used as the first Actor network, and the adjusted initial Critic network is used as the first Critic network. The structure of the initial Actor network is the same as that of the first Actor network, and the structure of the initial Critic network is the same as that of the first Critic network.

[0202] S106: Update the current number of training times.

[0203] In some embodiments, if the current training is not the first time, the current training number is equal to the training number at the last moment plus 1. If the current training is the first time, the current training number is equal to 1.

[0204] S107: Detect whether the current number of training times is greater than a training times threshold.

[0205] In some embodiments, the training times threshold may be set to 100 times, 200 times, 500 times, etc. The present application does not limit the training times threshold.

[0206] S108: When it is determined that the current number of training times is greater than the training times threshold, the first Actor network is used as a navigation model.

[0207] In some embodiments, see Figure 1 As shown, after detecting whether the current number of training times is greater than the training times threshold, the method further includes:

[0208] S109: When it is determined that the current number of training times is less than or equal to the training times threshold, use the first deep learning model as the initial deep learning model; obtain the state data of the first robot at the next moment and the radar data at the next moment as the state data at the current moment and the radar data at the current moment.

[0209] In some embodiments, when it is determined that the current number of training times is less than or equal to the training times threshold, a control instruction is generated according to the "predicted navigation information at the current moment", the next moment is taken as the current moment, and the first robot is controlled by the control instruction to move according to the linear velocity and angular velocity in the "predicted navigation information at the current moment", move to the new current position, and obtain new state data and new radar data. Then, steps S102 to S107 are performed again until the current number of training times is greater than the training times threshold.

[0210] This application proposes a navigation method, see Figure 3 As shown, when implemented specifically, the method includes the following contents.

[0211] S201: Obtaining the noisy navigation information, noisy radar data, and current position of the second robot at the current moment.

[0212] In some embodiments, the second robot is a real robot, specifically a wheeled robot, and is equipped with sensors such as a depth camera, an odometer, a laser radar, an accelerometer, a gyroscope, and a GPS positioning device.

[0213] S202: Acquire a second low-precision global map and a navigation model; the navigation model is trained by any of the above navigation model training methods.

[0214] In some embodiments, the second low-precision global map can be constructed by using depth camera data and / or odometer data. The second low-precision global map is less difficult to construct.

[0215] In some embodiments, the navigation model is migrated to a real scene for deployment and application to provide predicted navigation information for the second robot.

[0216] S203: Selecting a second global sampling point and a second target end point from the second low-precision global map.

[0217] In some embodiments, the endpoint of the second robot serves as the second target endpoint.

[0218] In some embodiments, selecting a second global sampling point and a second target end point from the second low-precision global map specifically includes:

[0219] S1: Selecting a second target destination from the second low-precision global map;

[0220] S2: performing path planning on the second low-precision global map according to the second target destination to obtain a second global path;

[0221] S3: Perform sampling according to a preset sampling interval, and select two second global sampling points from the second global path.

[0222] In some embodiments, the second global path is the shortest path connecting the current position and the second target end point. The second global path is obtained through the current position and noisy navigation information, and has low accuracy.

[0223] In some embodiments, the sampling interval can be set to 0.5 meters. With the current position as the sampling starting point, two sampling points are selected from the second global path forward according to the sampling interval to obtain two second global sampling points. Alternatively, the current position can be projected into the second global path, with the projection point as the sampling starting point, and two sampling points are selected from the second global path forward according to the sampling interval to obtain two second global sampling points.

[0224] S204: Determine the noise-containing distance information between the current position and the second target end point.

[0225] In some embodiments, the noisy distance information includes: a noisy unit vector of the second target endpoint relative to the current position, and a noisy Euclidean distance from the second target endpoint to the current position. Since the current position itself has errors, the calculated unit vector and Euclidean distance are both noisy data.

[0226] S205: Inputting the noisy navigation information, the noisy radar data, the second global sampling point, and the noisy distance information into the navigation model to obtain the predicted navigation information at the current moment.

[0227] In some embodiments, the navigation model includes: a first convolutional neural network model, a second convolutional neural network model, a first multilayer perceptron, a second multilayer perceptron, and a gated recurrent unit.

[0228] In some embodiments, the noisy navigation information, the noisy radar data, the second global sampling point, and the noisy distance information are input into the navigation model to obtain the predicted navigation information at the current moment, specifically including:

[0229] S1: Obtain the predicted navigation information of the previous moment and the first hidden state of the previous moment;

[0230] S2: splicing the noisy distance information and the noisy navigation information to obtain a fifth splicing result;

[0231] S3: inputting the fifth concatenation result into the first multilayer perceptron to obtain a third calculation result;

[0232] S4: inputting the second global sampling point into the first convolutional neural network model for feature extraction to obtain a sixth eigenvalue;

[0233] S5: Inputting the noisy radar data into the second convolutional neural network model for feature extraction to obtain a seventh eigenvalue;

[0234] S6: splicing the third calculation result, the sixth feature, the seventh feature value, and the predicted navigation information at the previous moment to obtain a sixth splicing result;

[0235] S7: Input the sixth concatenation result and the first hidden state at the previous moment into the gated recurrent unit to obtain the first hidden state at the current moment;

[0236] S8: Input the first hidden state at the current moment into the second multilayer perceptron to obtain the predicted navigation information at the current moment.

[0237] In some embodiments, the noisy navigation information includes: a unit vector (sinβ, cosβ) corresponding to a noisy measured yaw angle (denoted as β), a noisy measured linear velocity, and a noisy measured angular velocity. The noisy navigation information is collected by a gyroscope and an accelerometer.

[0238] In some embodiments, the predicted navigation information includes: an angular velocity prediction value and a linear velocity prediction value.

[0239] S206: Generate a control instruction according to the predicted navigation information at the current moment; and use the control instruction to perform navigation control on the motion behavior of the second robot at the next moment.

[0240] In some embodiments, control instructions are used to perform navigation control on the movement behavior of the second robot at the next moment, and the angular velocity and linear velocity of the second robot at the next moment are controlled to be equal to the predicted angular velocity value and linear velocity predicted value in the predicted navigation information. The robot moves to a new position, and the next moment is taken as the current moment. Steps S201 to S205 are then repeated until the second robot reaches the second target endpoint, and the navigation ends.

[0241] In some embodiments, the control of the linear velocity and angular velocity can be achieved using an underlying velocity tracking controller.

[0242] In some embodiments, in order to improve navigation accuracy, it is necessary to ensure that the training conditions of the navigation model in the simulation environment are the same as the application conditions of the navigation model in the real scene, for example, to ensure that the execution frequency and input data format of the two are the same.

[0243] Based on the above embodiments, the present application uses a reinforcement learning algorithm to construct a target loss function using hidden states, uses the target loss function to pre-construct a navigation model, and uses the navigation model in actual scenarios to achieve navigation control of the robot; further, the navigation model can be applied to a variety of complex terrains, and has good performance in terms of navigation time and number of collisions; and the navigation model does not require the input of a high-precision map but only a low-precision map, which effectively reduces the difficulty of map construction.

[0244] Based on the above-mentioned navigation model training method, one or more embodiments of the present application also provide a navigation model training device. Figure 4 It is a structural diagram of an embodiment of a navigation model training device provided by the present application. The navigation model training device provided by the present application may include: a first acquisition module 401, a first calculation module 402, a construction module 403, a processing module 404, an update module 405, a detection module 406, and a generation module 407.

[0245] The first acquisition module 401 is used to acquire a first high-precision global map and an initial deep learning model; and acquire the current state data and radar data of the first robot.

[0246] A first calculation module 402, configured to calculate a target reward value at a current moment according to the state data at the current moment and the first high-precision global map;

[0247] A construction module 403 is used to determine a first input data set and a second input data set according to the state data at the current moment, the radar data at the current moment, and the first high-precision global map;

[0248] A processing module 404 is used to process the first input data set and the second input data set using the initial deep learning model to obtain a target loss function;

[0249] An adjustment module 405 is used to adjust the weight of the initial deep learning model using the target loss function and the target reward value to obtain a first deep learning model; wherein the first deep learning model includes a first Actor network and a first Critic network;

[0250] An updating module 406 is used to update the current training times;

[0251] A detection module 407 is used to detect whether the current number of training times is greater than a training times threshold;

[0252] The generating module 408 is configured to use the first Actor network as a navigation model when it is determined that the current number of training times is greater than a training times threshold.

[0253] In some embodiments, the first calculation module 402 is specifically used to: use the first high-precision global map to obtain a first low-precision global map; select a first target endpoint of the first robot from the first low-precision global map; select a first global sampling point from the first low-precision global map according to the first target endpoint; calculate the target endpoint reward value according to the first global sampling point and the current position; calculate a dense reward value according to the first global sampling point, the current position, the velocity vector, and the maximum forward speed; calculate a collision reward value according to the collision force information of the first robot; perform a weighted sum operation on the target endpoint reward value, the dense reward value, and the collision reward value to obtain the target reward value at the current moment.

[0254] In some embodiments, construction module 403 is specifically used to: extract a circular area with the first robot as the center from the first high-precision global map as a local map; determine the noise-free distance information and the noisy distance information between the current position and the first target end point; combine the noisy distance information, the noisy navigation information, the first global sampling point, and the noisy radar data as a first input data set; combine the noise-free distance information, the noise-free navigation information, the first global sampling point, the noise-free radar data, and the local map as a second input data set.

[0255] In some embodiments, the processing module 404 is specifically used to: input the first input data set into the initial Actor network to obtain the first hidden state of the initial Actor network at the current moment; input the second input data set into the initial Critic network to obtain the second hidden state of the initial Critic network at the current moment; and construct the target loss function of the initial deep learning model using the first hidden state at the current moment and the second hidden state at the current moment.

[0256] Based on the above navigation method, one or more embodiments of the present application further provide a navigation device. Figure 5 It is a structural diagram of an embodiment of a navigation device provided by the present application. The navigation device provided by the present application may include: a second acquisition module 501, a third acquisition module 502, a selection module 503, a second calculation module 504, a prediction module 505, and a control module 506.

[0257] The second acquisition module 501 is used to acquire the noisy navigation information, the noisy radar data, and the current position of the second robot at the current moment;

[0258] A third acquisition module 502 is used to acquire a second low-precision global map and a navigation model; the navigation model is trained by any navigation model training method;

[0259] A selection module 503 is used to select a second global sampling point and a second target end point from the second low-precision global map;

[0260] A second calculation module 504 is used to determine the noise-containing distance information between the current position and the second target end point;

[0261] The prediction module 505 is used to input the noisy navigation information, the noisy radar data, the second global sampling point, and the noisy distance information into the navigation model to obtain the predicted navigation information at the current moment;

[0262] The control module 506 is used to generate a control instruction according to the predicted navigation information at the current moment; and use the control instruction to perform navigation control on the motion behavior of the second robot at the next moment.

[0263] In some embodiments, the selection module 503 is specifically used to: select a second target destination from the second low-precision global map; perform path planning on the second low-precision global map according to the second target destination to obtain a second global path; perform sampling according to a preset sampling interval, and select two second global sampling points from the second global path.

[0264] It should be noted that the above-mentioned device may also include other implementation modes according to the description of the method embodiment. The specific implementation mode may refer to the description of the relevant method embodiment, which will not be described one by one here.

[0265] The present application also provides an electronic device, comprising at least one processor and a memory for storing processor executable instructions, wherein the processor can implement any of the above method embodiments when executing the instructions. For example, when the instructions are executed by the processor, the following steps are implemented: obtaining a first high-precision global map and an initial deep learning model; obtaining the state data of the first robot at the current moment and the radar data at the current moment; calculating the target reward value at the current moment according to the state data at the current moment and the first high-precision global map; determining a first input data set and a second input data set according to the state data at the current moment, the radar data at the current moment and the first high-precision global map; processing the first input data set and the second input data set using the initial deep learning model to obtain a target loss function; adjusting the weight of the initial deep learning model using the target loss function and the target reward value to obtain a first deep learning model; wherein the first deep learning model includes a first Actor network and a first Critic network; updating the current training times; detecting whether the current training times are greater than the training times threshold; and using the first Actor network as a navigation model when it is determined that the current training times are greater than the training times threshold.

[0266] The present application also provides an electronic device, comprising at least one processor and a memory for storing processor executable instructions, wherein the processor can implement any one of the above method embodiments when executing the instructions. For example, when the instructions are executed by the processor, the following steps are implemented: obtaining the noisy navigation information, noisy radar data, and current position of the second robot at the current moment; obtaining a second low-precision global map and a navigation model; the navigation model is trained by any navigation model training method; selecting a second global sampling point and a second target end point from the second low-precision global map; determining the noisy distance information between the current position and the second target end point; inputting the noisy navigation information, noisy radar data, the second global sampling point, and the noisy distance information into the navigation model to obtain the predicted navigation information at the current moment; generating control instructions based on the predicted navigation information at the current moment; and using the control instructions to perform navigation control on the motion behavior of the second robot at the next moment.

[0267] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0268] It should be noted that the electronic device described above may also include other implementations according to the description of the method or device embodiment. The specific implementation methods may refer to the description of the relevant method embodiment, which will not be described one by one here.

[0269] The present application also provides a storage medium, which is a computer-readable storage medium, and the storage medium stores computer program instructions, which, when executed, implement the following: obtaining a first high-precision global map and an initial deep learning model; obtaining the current state data and radar data of the first robot; calculating the target reward value at the current moment according to the current state data and the first high-precision global map; determining a first input data set and a second input data set according to the current state data, the current radar data and the first high-precision global map; processing the first input data set and the second input data set using the initial deep learning model to obtain a target loss function; adjusting the weight of the initial deep learning model using the target loss function and the target reward value to obtain a first deep learning model; wherein the first deep learning model includes a first Actor network and a first Critic network; updating the current training times; detecting whether the current training times is greater than a training times threshold; and using the first Actor network as a navigation model when it is determined that the current training times is greater than the training times threshold.

[0270] The present application also provides a storage medium, which is a computer-readable storage medium, and the storage medium stores computer program instructions, which, when executed, implement the following: obtaining the noisy navigation information, noisy radar data, and current position of the second robot at the current moment; obtaining a second low-precision global map and a navigation model; the navigation model is trained by any navigation model training method; selecting a second global sampling point and a second target end point from the second low-precision global map; determining the noisy distance information between the current position and the second target end point; inputting the noisy navigation information, noisy radar data, the second global sampling point, and the noisy distance information into the navigation model to obtain the predicted navigation information at the current moment; generating control instructions based on the predicted navigation information at the current moment; and using the control instructions to perform navigation control on the movement behavior of the second robot at the next moment.

[0271] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0272] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0273] In summary, although the present application has been disclosed as above with preferred embodiments, the above preferred embodiments are not intended to limit the present application. Ordinary technicians in this field can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined in the claims.

Claims

1. A navigation model training method, characterized in that: The method comprises: Obtain a first high-precision global map and an initial deep learning model; obtain the current state data and radar data of the first robot; Calculating a target reward value at the current moment according to the current state data and the first high-precision global map; Determine a first input data set and a second input data set according to the state data at the current moment, the radar data at the current moment, and the first high-precision global map; Processing the first input data set and the second input data set using the initial deep learning model to obtain a target loss function; Using the target loss function and the target reward value to adjust the weight of the initial deep learning model, a first deep learning model is obtained; wherein the first deep learning model includes a first Actor network and a first Critic network; Update the current number of training times; Check whether the current number of training times is greater than the training times threshold; When it is determined that the current number of training times is greater than the training times threshold, the first Actor network is used as the navigation model.

2. The method according to claim 1, characterized in that The first high-precision global map includes a target obstacle scene; the target obstacle scene includes at least one of the following: a threshold whose height is less than a first threshold, a channel whose width is less than a second threshold, a channel closed at one end, a scene of multiple obstacles densely stacked, and a scene of multiple obstacles discretely distributed.

3. The method according to claim 1, characterized in that The state data includes: current position, noise-free navigation information, noise-containing navigation information, velocity vector, maximum forward speed, and collision force information of the first robot; The radar data includes: noisy radar data and noise-free radar data.

4. The method according to claim 3, characterized in that Calculating the target reward value at the current moment according to the current state data and the first high-precision global map, including: Obtaining a first low-precision global map using the first high-precision global map; Selecting a first target destination of the first robot from the first low-precision global map; According to the first target endpoint, selecting a first global sampling point from the first low-precision global map; Calculating a target endpoint reward value according to the first global sampling point and the current position; Calculate a dense reward value according to the first global sampling point, the current position, the velocity vector, and the maximum forward speed; Calculating a collision reward value according to the collision force information of the first robot; Perform a weighted sum operation on the target endpoint reward value, dense reward value, and collision reward value to obtain the target reward value at the current moment.

5. The method according to claim 4, characterized in that According to the first target endpoint, selecting a first global sampling point from the first low-precision global map includes: According to the first target destination, path planning is performed on the first low-precision global map to obtain a first global path; Sampling is performed according to a preset sampling interval, and two first global sampling points are selected from the first global path.

6. The method according to claim 4, characterized in that Determining a first input data set and a second input data set according to the current state data, the current radar data, and the first high-precision global map, includes: From the first high-precision global map, extract a circular area with the first robot as the center as a local map; Determine noise-free distance information and noise-containing distance information between the current position and the first target end point; combining the noisy distance information, the noisy navigation information, the first global sampling point, and the noisy radar data as a first input data set; The noise-free distance information, the noise-free navigation information, the first global sampling point, the noise-free radar data, and the local map are combined as a second input data set.

7. The method according to claim 6, characterized in that The initial deep learning model includes an initial Actor network and an initial Critic network; The initial Actor network includes: a first convolutional neural network model, a second convolutional neural network model, a first multi-layer perceptron, a second multi-layer perceptron, and a gated recurrent unit; The initial critic network includes: a third convolutional neural network model, a fourth convolutional neural network model, a fifth convolutional neural network model, a third multilayer perceptron, a fourth multilayer perceptron, and a fifth multilayer perceptron.

8. The method according to claim 7, characterized in that Processing the first input data set and the second input data set using the initial deep learning model to obtain a target loss function includes: Inputting the first input data set into the initial Actor network to obtain a first hidden state of the initial Actor network at a current moment; Inputting the second input data set into the initial critic network to obtain the second hidden state of the initial critic network at the current moment; The target loss function of the initial deep learning model is constructed using the first hidden state at the current moment and the second hidden state at the current moment.

9. The method according to claim 8, characterized in that Inputting the first input data set into the initial Actor network to obtain a first hidden state of the initial Actor network at a current moment includes: Obtain the predicted navigation information of the previous moment and the first hidden state of the previous moment; Splicing the noisy distance information and the noisy navigation information to obtain a first splicing result; Inputting the first concatenation result into the first multilayer perceptron to obtain a first calculation result; Inputting the first global sampling point into the first convolutional neural network model for feature extraction to obtain a first eigenvalue; Inputting the noisy radar data into the second convolutional neural network model for feature extraction to obtain a second eigenvalue; splicing the first calculation result, the first eigenvalue, the second eigenvalue, and the predicted navigation information at the previous moment to obtain a second splicing result; The second splicing result and the first hidden state at the previous moment are input into the gated recurrent unit to obtain the first hidden state of the initial Actor network at the current moment.

10. The method according to claim 9, characterized in that After obtaining the first hidden state of the initial Actor network at the current moment, the method further includes: The first hidden state at the current moment is input into the second multilayer perceptron to obtain the predicted navigation information at the current moment.

11. The method according to claim 8, characterized in that Inputting the second input data set into the initial Critic network to obtain the second hidden state of the initial Critic network at the current moment includes: splicing the noise-free distance information and the noise-free navigation information to obtain a third splicing result; Inputting the third concatenation result into the third multilayer perceptron to obtain a second calculation result; Inputting the local map into the third convolutional neural network model for feature extraction to obtain a third eigenvalue; Inputting the first global sampling point into the fourth convolutional neural network model for feature extraction to obtain a fourth eigenvalue; Inputting the noise-free radar data into the fifth convolutional neural network model for feature extraction to obtain a fifth eigenvalue; The second calculation result, the third eigenvalue, the fourth eigenvalue, and the fifth eigenvalue are converted into one-dimensional vectors and then concatenated to obtain a fourth concatenation result; The fourth concatenation result is input into the fourth multilayer perceptron to obtain the second hidden state of the initial Critic network at the current moment.

12. The method according to claim 11, characterized in that The method further comprises: The second hidden state at the current moment is input into the fifth multilayer perceptron to obtain the state value function of the initial Critic network.

13. The method according to claim 1, characterized in that After detecting whether the current number of training times is greater than the training times threshold, the method further includes: When it is determined that the current number of training times is less than or equal to a training times threshold, using the first deep learning model as an initial deep learning model; The state data of the first robot at the next moment and the radar data at the next moment are obtained as the state data at the current moment and the radar data at the current moment.

14. A navigation method, characterized in that: The method comprises: Obtaining the noisy navigation information, the noisy radar data, and the current position of the second robot at the current moment; Acquire a second low-precision global map and a navigation model; the navigation model is trained by the method according to any one of claims 1 to 13; Selecting a second global sampling point and a second target endpoint from the second low-precision global map; Determining noisy distance information between the current position and the second target end point; Inputting noisy navigation information, noisy radar data, a second global sampling point, and noisy distance information into the navigation model to obtain predicted navigation information at the current moment; A control instruction is generated according to the predicted navigation information at the current moment; and the control instruction is used to perform navigation control on the motion behavior of the second robot at the next moment.

15. The method according to claim 14, characterized in that Selecting a second global sampling point and a second target end point from the second low-precision global map includes: Selecting a second target destination from the second low-precision global map; According to the second target destination, performing path planning on the second low-precision global map to obtain a second global path; Sampling is performed according to a preset sampling interval, and two second global sampling points are selected from the second global path.

16. A navigation model training device, characterized in that: The device comprises: A first acquisition module is used to acquire a first high-precision global map and an initial deep learning model; and to acquire the current state data and radar data of the first robot. A first calculation module, used for calculating a target reward value at a current moment according to the state data at the current moment and the first high-precision global map; A construction module, configured to determine a first input data set and a second input data set according to the state data at the current moment, the radar data at the current moment, and the first high-precision global map; A processing module, configured to process the first input data set and the second input data set using the initial deep learning model to obtain a target loss function; An adjustment module, configured to adjust the weight of the initial deep learning model using the target loss function and the target reward value to obtain a first deep learning model; wherein the first deep learning model includes a first Actor network and a first Critic network; Update module, used to update the current training times; A detection module is used to detect whether the current number of training times is greater than a training times threshold; The generation module is used to use the first Actor network as a navigation model when it is determined that the current training number is greater than a training number threshold.

17. A navigation device, characterized in that: The device comprises: A second acquisition module is used to acquire the noisy navigation information, the noisy radar data, and the current position of the second robot at the current moment; A third acquisition module, used to acquire a second low-precision global map and a navigation model; the navigation model is trained by the method according to any one of claims 1 to 13; A selection module, used for selecting a second global sampling point and a second target end point from the second low-precision global map; A second calculation module, used to determine the noise-containing distance information between the current position and the second target end point; A prediction module, used for inputting the noisy navigation information, the noisy radar data, the second global sampling point, and the noisy distance information into the navigation model to obtain the predicted navigation information at the current moment; The control module is used to generate control instructions according to the predicted navigation information at the current moment; and use the control instructions to perform navigation control on the movement behavior of the second robot at the next moment.

18. An electronic device, comprising a processor and a memory for storing processor-executable instructions, wherein the processor implements the steps of any one of claims 1 to 13 or any one of claims 14 to 15 when executing the instructions.

19. A storage medium, characterized in that: Computer instructions are stored thereon, and when the instructions are executed by a processor, the steps of the method described in any one of claims 1 to 13 or any one of claims 14 to 15 are performed.

Images