Autonomous navigation method, device and medium for unmanned vehicle in unknown environment

Abstract

The present application discloses a method, device and medium for autonomous navigation of an unmanned vehicle in an unknown environment, and relates to the field of unmanned driving technology. The three-dimensional point cloud data at the current moment is processed to obtain the position and surrounding environment information of the unmanned vehicle at the current moment. The position and surrounding environment information of the target point at the current moment are used as input, and the speed of the unmanned vehicle at the current moment is determined by using a trained reinforcement learning model. The speed of the unmanned vehicle includes linear velocity and angular velocity. The unmanned vehicle is controlled to move according to the speed of the unmanned vehicle at the current moment until the unmanned vehicle reaches the target point. The present application introduces a trained reinforcement learning model, which can determine the speed of the unmanned vehicle based on sample experience, without manual adjustment of parameters, and can solve the problem of a large number of parameter adjustments required for autonomous navigation of the unmanned vehicle in an unknown environment, thereby improving the adaptability of the unmanned vehicle in an unknown environment and avoiding a large amount of manpower and material burden on the development process.

Description

Technical Field

[0001] The present application relates to the field of unmanned driving technology, and in particular to an autonomous navigation method, device and medium for an unmanned vehicle in an unknown environment. Background Art

[0002] Unmanned vehicles refer to vehicles that integrate advanced sensors and use artificial intelligence technology to achieve autonomous driving functions. With the development of science and technology, the field of unmanned driving has developed rapidly and the technology has become increasingly mature. However, there is no mature solution for the scenario where unmanned vehicles rely solely on target points for autonomous navigation in unknown environments.

[0003] At present, the autonomous navigation of unmanned vehicles mainly relies on high-definition maps for global path planning. However, when map information is unavailable or there is a large deviation, the execution of this method will be hindered and lead to serious navigation errors, which will have a significant impact on the autonomous navigation performance of unmanned vehicles.

[0004] Most of the existing unmanned driving technologies require global map information, but some unmanned driving technologies have considered autonomous navigation solutions in unknown environments. However, these autonomous navigation solutions usually require a lot of parameter adjustments, especially when switching to a new unmanned vehicle or facing a new environment. It is necessary to frequently adjust the parameters manually based on the actual motion state of the unmanned vehicle, rather than optimizing based on sample experience. This not only greatly limits the adaptability of unmanned vehicles in unknown environments, but also brings a lot of manpower and material resources to the development process.

[0005] Based on this, the self-optimizing autonomous navigation technology of unmanned vehicles in unknown environments has become a difficulty and hot spot in current research. Summary of the invention

[0006] The purpose of this application is to provide a method, device and medium for autonomous navigation of an unmanned vehicle in an unknown environment, which can solve the problem that a large number of parameter adjustments are required when the unmanned vehicle autonomously navigates in an unknown environment, improve the adaptability of the unmanned vehicle in an unknown environment, and avoid bringing a large amount of manpower and material resources to the development process.

[0007] To achieve the above objectives, this application provides the following solutions.

[0008] In a first aspect, the present application provides an autonomous navigation method for an unmanned vehicle in an unknown environment, and the autonomous navigation method for an unmanned vehicle in an unknown environment includes the following steps.

[0009] Acquire the three-dimensional point cloud data at the current moment; the three-dimensional point cloud data is data obtained by the laser radar sensor installed on the unmanned vehicle scanning the surrounding environment of the unmanned vehicle.

[0010] The three-dimensional point cloud data at the current moment is processed to obtain the position and surrounding environment information of the unmanned vehicle at the current moment; the surrounding environment information includes the minimum distance from the unmanned vehicle to the obstacle in front and the local occupancy grid map, and the local occupancy grid map is an occupancy grid map of a local area with the center point of the unmanned vehicle as the center point and a preset length as the side length.

[0011] The target point posture, the current posture of the unmanned vehicle and the surrounding environment information are used as input, and the trained reinforcement learning model is used to determine the speed of the unmanned vehicle at the current moment; the speed of the unmanned vehicle includes linear velocity and angular velocity.

[0012] Control the unmanned vehicle to move according to the current speed of the unmanned vehicle, and determine whether the unmanned vehicle reaches the target point. If not, return to the step of "obtaining the three-dimensional point cloud data at the current moment".

[0013] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned method for autonomous navigation of an unmanned vehicle in an unknown environment.

[0014] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for autonomous navigation of an unmanned vehicle in an unknown environment.

[0015] According to the specific embodiments provided in this application, this application discloses the following technical effects.

[0016] The present application provides a method, device and medium for autonomous navigation of an unmanned vehicle in an unknown environment. The three-dimensional point cloud data at the current moment is processed to obtain the unmanned vehicle's position and surrounding environment information at the current moment. The surrounding environment information includes the minimum distance from the unmanned vehicle to the obstacle in front and the local occupancy grid map. The target point position and the unmanned vehicle's position and surrounding environment information at the current moment are used as input, and a trained reinforcement learning model is used to determine the unmanned vehicle's speed at the current moment, and the unmanned vehicle is controlled to move at the unmanned vehicle's speed at the current moment until the unmanned vehicle reaches the target point. The present application introduces a trained reinforcement learning model, which can determine the speed of the unmanned vehicle based on sample experience, without the need for manual adjustment of parameters. This can solve the problem of a large number of parameter adjustments required for autonomous navigation of the unmanned vehicle in an unknown environment, improve the adaptability of the unmanned vehicle in an unknown environment, and avoid bringing a large amount of manpower and material resources to the development process. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A flowchart of an autonomous navigation method for an unmanned vehicle in an unknown environment provided in Example 1 of the present application.

[0019] Figure 2 A schematic diagram of the angle partitioning process when the number of angle partitions is 9 provided in Example 1 of the present application.

[0020] Figure 3 A schematic diagram showing the local environment provided in Example 1 of the present application.

[0021] Figure 4 A schematic diagram of the network framework of the trained reinforcement learning model provided in Example 1 of the present application.

[0022] Figure 5 A schematic diagram of the framework of the training method provided in Example 1 of the present application.

[0023] Figure 6 A schematic diagram of a simple indoor environment provided in Example 1 of the present application.

[0024] Figure 7 A schematic diagram of the structure of a computer device provided in Example 2 of the present application.

[0025] Figure numerals: 1-wall; 2-table; 3-chair; 4-unmanned vehicle. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] Example 1.

[0028] like Figure 1 As shown, this embodiment provides an autonomous navigation method for an unmanned vehicle in an unknown environment, which is executed by a computer device. The autonomous navigation method for an unmanned vehicle in an unknown environment includes the following steps.

[0029] Step S1, obtaining three-dimensional point cloud data at the current moment; the three-dimensional point cloud data is data obtained by a laser radar sensor installed on the unmanned vehicle scanning the surrounding environment of the unmanned vehicle.

[0030] Step S2, processing the three-dimensional point cloud data at the current moment to obtain the unmanned vehicle posture and surrounding environment information at the current moment; the surrounding environment information includes the minimum distance from the unmanned vehicle to the obstacle in front and the local occupancy grid map, and the local occupancy grid map is an occupancy grid map of a local area with the center point of the unmanned vehicle as the center point and a preset length as the side length.

[0031] Step S3, taking the target point posture and the current posture of the unmanned vehicle and the surrounding environment information as input, using the trained reinforcement learning model to determine the speed of the unmanned vehicle at the current moment; the speed of the unmanned vehicle includes linear velocity and angular velocity.

[0032] Step S4, control the unmanned vehicle to move according to the current speed of the unmanned vehicle, and determine whether the unmanned vehicle reaches the target point. If not, return to the step of "obtaining the three-dimensional point cloud data at the current moment".

[0033] By implementing the above-mentioned steps S1 to S4, this embodiment provides a self-optimization method for unmanned vehicles to perform autonomous navigation through self-learning in unknown environments, aiming to solve the problems of insufficient adaptability and robustness and a large amount of manpower and material burden on the development process due to the need for a large number of parameter adjustments when the unmanned vehicle performs autonomous navigation in an unknown environment. It can significantly improve the navigation efficiency and accuracy of the unmanned vehicle in various unknown environments, and has strong practicality and broad application prospects.

[0034] In S1, this embodiment installs a laser radar sensor on the unmanned vehicle, which can be installed at the center of the roof of the unmanned vehicle to provide the laser radar sensor with a 360° unobstructed field of view, so that it can capture all-round environmental information around the vehicle. The laser radar sensor is used to scan the surrounding environment of the unmanned vehicle in an unknown environment to obtain the three-dimensional point cloud data at the current moment.

[0035] In S2, this embodiment uses the three-dimensional point cloud data at the current moment as input, and uses the SLAM (Simultaneous localization and mapping) algorithm to obtain the unmanned vehicle's posture at the current moment. The unmanned vehicle's posture includes the position and posture of the unmanned vehicle. The position is represented by three-dimensional coordinates, and the posture is represented by pitch angle, heading angle, and roll angle.

[0036] In S2, this embodiment provides a comprehensive processing method for environmental information based on a two-layer perception mechanism, that is, an environmental perception method based on three-dimensional point cloud data, which is used to improve the navigation performance of unmanned vehicles in unknown environments. Specifically, the three-dimensional point cloud data at the current moment is processed to obtain the surrounding environment information at the current moment. The surrounding environment information includes the minimum distance from the unmanned vehicle to the obstacle ahead and the local occupancy grid map. The local occupancy grid map is an occupancy grid map of a local area with the center point of the unmanned vehicle as the center point and a preset length as the side length.

[0037] (1) The minimum distance between the unmanned vehicle and the obstacle ahead.

[0038] This embodiment processes the three-dimensional point cloud data directly read from the laser radar sensor to accurately determine the distance from the nearest obstacle in front of the unmanned vehicle to the unmanned vehicle, that is, to determine the minimum distance from the unmanned vehicle to the obstacle in front.

[0039] (1.1) Initialize an angle partition array, which includes multiple angle partitions, which are used to classify and process the obstacle distance information in different directions in the three-dimensional point cloud data (i.e., the distance from the obstacle to the unmanned vehicle). Each angle partition is extracted from the scanning range of the lidar sensor, and these angle partitions are processed to ensure that the area of ​​each angle partition evenly covers the specific angle range. All angle partitions need to completely cover the 180° area in front of the unmanned vehicle, and in order to avoid potential data parsing errors at the partition boundary, each angle partition is slightly expanded by 0.03 radians at its starting angle and ending angle so that two adjacent angle partitions partially overlap. Through this angle partitioning method, the obstacle distance information in each direction can be more accurately identified and processed, thereby improving the environmental perception ability and navigation accuracy of the unmanned vehicle. When the number of angle partitions is 9, the angle partitioning process is as follows Figure 2 As shown, Figure 2 middle, Represents obstacle distance information.

[0040] The mathematical form of angle partitioning is shown in equation (1).

[0041] (1).

[0042] In formula (1), Angle range for each angle partition; is the number of angle partitions.

[0043] (1.2) Process the three-dimensional point cloud data to calculate the distance information of the nearest obstacle in each angle partition. The processing steps are as follows.

[0044] (a) Convert the incoming 3D point cloud data into a , , A list of coordinates.

[0045] (b) Initialize a distance array whose length is the same as the number of angle partitions Similarly, each element is initialized to a larger value, which is usually selected as the maximum detection distance of the lidar sensor, indicating the farthest measured distance in this direction.

[0046] (c) For each point in the 3D point cloud data, ignore For points with coordinates less than 0.5 meters, since these points may be from the ground, 0.5 meters can also be replaced by other values. The specific value is based on the installation position of the lidar sensor and is determined by experiments. Then calculate the polar coordinate angle of each remaining point , the polar coordinate angle represents the angle between the point and The angle between the axes is defined as shown in the following formula (2).

[0047] (2).

[0048] (d) Based on the calculated polar coordinate angle , determine which angle partition the point is in. If the distance between the unmanned vehicle and the point is less than the minimum distance recorded in the angle partition, update the minimum distance recorded in the angle partition, and the distance is the Euclidean distance. After traversing all points, select the minimum value of the minimum distance recorded in all angle partitions as the minimum distance from the unmanned vehicle to the obstacle in front.

[0049] The above process is the first-layer environmental information processing method of the two-layer perception mechanism. This method can optimize the long-distance navigation capability of the unmanned vehicle. Through the lidar sensor installed on the unmanned vehicle, the closest distance information between the obstacle and the unmanned vehicle is captured and provided in real time. This information is recorded in the form of a one-dimensional array and directly input into the reinforcement learning model. Among them, the data structure for recording the closest distance information is designed in a simple form, with the aim of quickly and effectively processing a wide range of environmental data, thereby achieving accurate detection and classification of long-distance obstacles. The closest distance information is directly input into the model to improve the model's recognition of long-distance obstacles, thereby enhancing the overall performance of the unmanned vehicle's autonomous navigation.

[0050] (2) Local occupancy grid map.

[0051] The second-layer environmental information processing method of the dual-layer perception mechanism is mainly responsible for the deep processing and precise positioning of environmental information, achieving precise positioning relative to the starting point and generating a local occupancy grid map based on it.

[0052] (2.1) Data acquisition.

[0053] Use the lidar sensor installed on the unmanned vehicle to continuously scan the environment around the unmanned vehicle at a fixed frequency to obtain high-density three-dimensional point cloud data. The three-dimensional point cloud data includes the spatial coordinates and reflection intensity of multiple points. These data will be used for subsequent feature extraction and pose estimation.

[0054] (2.2) Positioning calculation.

[0055] From the acquired 3D point cloud data, edge detection and plane segmentation techniques are used to identify key feature points. Feature points are divided into two categories: edge points and plane points. Edge points are identified by calculating the local curvature of each point in the 3D point cloud data. Points whose local curvature is higher than the set local curvature threshold are marked as edge points. Plane points are identified by calculating the local planarity of each point in the 3D point cloud data. Specifically, all points in the neighborhood of each point are used as input, and the RANSAC (Random Sample Consensus) algorithm is used to identify the optimal plane model that best matches each point. The planarity of each point is evaluated by calculating the vertical distance from each point to its corresponding optimal plane model. Specifically, the vertical distance is used as the local planarity of the point. When the local planarity of a point exceeds the set local planarity threshold, the point is marked as a plane point.

[0056] The mathematical form of feature point extraction is shown in formula (3).

[0057] (3).

[0058] In formula (3), For feature points A collection of; It is three-dimensional point cloud data; For feature points The local curvature of is the set local curvature threshold; For feature points The local planarity of is the local planarity threshold.

[0059] After extracting the feature points, the pose estimation is performed using the iterative closest point (ICP) algorithm based on the feature points to determine the transformation matrix of the three-dimensional point cloud data at the current moment compared to the three-dimensional point cloud data at the previous moment. This algorithm not only considers the minimum distance between points, but also the distance from points to features. This algorithm constructs an optimization problem to minimize the cost function shown in the following formula (4).

[0060] (4).

[0061] In formula (4), is the optimized transformation matrix; is the transformation matrix; is the feature point in the three-dimensional point cloud data at the current moment; is the feature point in the 3D point cloud data at the previous moment; is a set of matching feature point pairs; and Both are weight parameters, used to adjust the importance of plane features and edge features in the overall optimization; For feature points The local planarity of For feature points The local curvature of .

[0062] The estimated relative pose transformation (i.e., transformation matrix) between each frame and the previous frame is accumulated to construct a complete path from the initial position to the current position. During this process, the accumulated pose information is used to update the relative coordinates of the unmanned vehicle to ensure the continuity and accuracy of the navigation system.

[0063] (2.3) Initialize grid information.

[0064] In the constructor, the physical size, resolution and related parameters for generating the local occupancy grid map are initialized. They are usually defined according to the actual size of the unmanned vehicle. The physical size is the size of the local area with the center point of the unmanned vehicle as the center point and the preset length as the side length. The preset length is generally larger than the actual size of the unmanned vehicle, but smaller than the maximum detection distance of the lidar sensor. The resolution can be determined according to user needs. The related parameters refer to the height threshold when the grid is generated (including the preset minimum and maximum values). Only points with z coordinates between the preset minimum and maximum values ​​are used to generate the local occupancy grid map. The minimum value is determined based on the ground height, and the maximum value is determined based on the height of the unmanned vehicle. This is because points on the ground and points higher than the unmanned vehicle will not affect the movement of the unmanned vehicle. The above parameters define the level of detail and coverage of the grid map. The constructor also initializes the grid map array to store the occupancy status of each grid.

[0065] (2.4) Generate a local occupancy grid map.

[0066] Nonlinear interpolation technology is used to perform time synchronization processing on the asynchronous 3D point cloud data and the posture of the unmanned vehicle. The posture interpolation is based on quaternion and exponential mapping theory to ensure the smooth transition of continuous postures. The mathematical form is shown in the following equation (5).

[0067] (5).

[0068] In formula (5), For the current moment The corresponding transformation matrix is, It is the timestamp of the 3D point cloud data at the current moment, that is, the timestamp of the unmanned vehicle’s position at the current moment; is the first transformation matrix of the last moment compared to the moment before that; is the second transformation matrix between the current moment and the previous moment; To obtain the timestamp of the first transformation matrix; To obtain the timestamp of the second transformation matrix.

[0069] Calculate the current 3D point cloud data and the current time The product of the corresponding transformation matrices gives , which is the precise three-dimensional point cloud data obtained by the above-mentioned pose interpolation. The precise three-dimensional point cloud data is further converted into the vehicle body coordinate system. The mathematical form is shown in the following formula (6).

[0070] (6).

[0071] In formula (6), It is local point cloud data; and They are the rotation matrix and translation vector from the local coordinate system (i.e. the coordinate system of the three-dimensional point cloud data obtained by the lidar sensor) to the vehicle coordinate system (i.e. the coordinate system of the unmanned vehicle).

[0072] This embodiment can also perform denoising and outlier filtering of precise 3D point cloud data to improve the accuracy and reliability of the data. The converted local point cloud data can be used to dynamically update the local occupancy grid map. During the map update process, the map generation algorithm also takes into account the dynamic changes of the environment, such as pedestrians and other moving obstacles.

[0073] After the above processing, the local occupancy grid map can be effectively generated from the 3D point cloud data, laying the foundation for the subsequent autonomous navigation of unmanned vehicles in unknown environments. The fused local environment is represented as follows: Figure 3 As shown, Figure 3 In the equation, s is the distance between the unmanned vehicle and the target point. is the angle between the front of the unmanned vehicle and the target point.

[0074] The above process is the second-layer environmental information processing method of the dual-layer perception mechanism. This method collects the surrounding environment data of the unmanned vehicle in real time and processes this data to generate a high-resolution local occupancy grid map. Each grid element represents whether the corresponding position in the space is occupied, thereby providing data support for the precise positioning of obstacles. Subsequently, the local occupancy grid map is input into the reinforcement learning model in the form of a matrix. The matrix includes the occupancy state of the grid, where the value of each element represents the probability of the corresponding grid being occupied. By processing the matrix of the input local occupancy grid map, the model optimizes obstacle avoidance decisions within a short distance by learning the local obstacle pattern, ensuring that safety is maximized while maintaining efficient navigation.

[0075] In S3, the target point posture and the current unmanned vehicle posture and surrounding environment information are used as inputs, and the trained reinforcement learning model is used to determine the speed of the unmanned vehicle at the current moment, and the speed of the unmanned vehicle includes linear velocity and angular velocity. Among them, the trained reinforcement learning model can use the soft policy gradient (Soft Actor Critic, SAC) algorithm. This embodiment describes an innovative autonomous navigation method, which uses the SAC algorithm to optimize the action of the unmanned vehicle in the continuous action space, breaking through the limitation that the traditional autonomous navigation method needs to adjust parameters extensively for specific scenes. Through deep reinforcement learning technology, it automatically learns and optimizes the behavior strategy, and achieves high adaptability and responsiveness to complex environmental factors. Compared with the prior art, the autonomous navigation method of this embodiment not only reduces the dependence on manual intervention, but also improves universality and scalability. By conducting large-scale training in a high-fidelity simulation environment, this method can significantly improve the navigation accuracy and efficiency of the unmanned vehicle in the real world, and can be adjusted according to the simulation environment without limiting the terrain conditions. In addition, this method is highly flexible, and can adjust the algorithm and expand the function according to different application requirements, providing a reliable infrastructure for the future development of autonomous navigation technology.

[0076] In S3, the trained reinforcement learning model maps the unmanned vehicle's position, surrounding environment information, and target point position to the unmanned vehicle's speed to achieve autonomous navigation of the unmanned vehicle.

[0077] Specifically, Figure 4 As shown, the trained reinforcement learning model includes a first feature extraction branch, a second feature extraction branch and an action determination branch. The first feature extraction branch is used to extract features from the local occupancy grid map at the current moment to obtain the first feature. The second feature extraction branch is used to extract features from the target point posture, the unmanned vehicle posture at the current moment, and the minimum distance from the unmanned vehicle to the obstacle in front to obtain the second feature. The action determination branch is used to process the first feature and the second feature to obtain the speed of the unmanned vehicle at the current moment.

[0078] In this embodiment, the first feature extraction branch includes a plurality of convolutional layers, a flattening layer and a plurality of fully connected layers connected in sequence, the number of the convolutional layers may be 3 and the number of the fully connected layers may be 2.

[0079] (1) 3 convolutional layers.

[0080] In this embodiment, a local occupancy grid map is processed by a convolutional neural network. Specifically, three convolutional layers are used to extract features from the input local occupancy grid map. The first convolutional layer uses 32 5 5 convolution kernels with a stride of 3 are used to capture preliminary spatial features. The second and third convolutional layers use 64 3 The convolution kernel of 3 and the step size of 2 further refine the feature representation. The mathematical form of the convolution layer is shown in the following formula (7).

[0081] (7).

[0082] In formula (7), For the The output feature map of the convolutional layer, that is, The input feature map of the convolutional layer; is an activation function, which is used to increase nonlinear processing capabilities; and Respectively The weights and biases of the convolution kernels of the convolutional layers; Represents the convolution operation; For the The input feature map of the convolutional layer.

[0083] (2) Flatten the layer.

[0084] After being processed by three convolutional layers, the output feature map is converted into a one-dimensional array through flattening for subsequent processing by the fully connected layer. The mathematical form of the flattening layer is shown in the following formula (8).

[0085] (8).

[0086] In formula (8), is the flattened one-dimensional vector, for The index of the element in; It is the feature map output by 3 convolutional layers; The dimension of the feature map output by the three convolutional layers; is rounded down; mod is the modulus operation. Flattening operations are arranged in row order This allows the subsequent fully connected layers to receive data from the convolutional layer and perform further processing.

[0087] In this embodiment, the second feature extraction branch includes a concatenation layer and a fully connected layer connected in sequence.

[0088] In this embodiment, the action determination branch includes a concatenation layer, multiple fully connected layers and a normalization layer connected in sequence, and the number of fully connected layers may be 3.

[0089] This embodiment uses six fully connected layers to integrate the data of the laser radar sensor and the target point pose to generate the action distribution of the unmanned vehicle, which includes multiple values ​​of linear velocity and angular velocity and the probability of each value. The mathematical form of the first fully connected layer in the action determination branch is shown in the following formula (9).

[0090] (9).

[0091] In formula (9), is the output feature; Represent the weight matrix and bias vector of the fully connected layer respectively; represents feature connection, As the first feature, The second feature.

[0092] The normalization layer samples from a Gaussian distribution parameterized by the mean and standard deviation generated by the model, and includes a mean layer, a standard deviation layer, an activation function layer, and a logarithmic probability density calculation layer.

[0093] The average layer is used to calculate the average value of the action distribution, and its mathematical form is shown in the following formula (10).

[0094] (10).

[0095] In formula (10), is the average value; Calculate the function for the mean value; The action distribution output by the last fully connected layer of the action determination branch.

[0096] The standard deviation layer is used to produce the logarithm of the standard deviation, which provides a measure of the dispersion or variability of the action distribution, and its mathematical form is shown in Equation (11) below.

[0097] (11).

[0098] In formula (11), The final standard deviation The logarithm of is a function that converts Restricted to a certain range, express The standard deviation of and the maximum standard deviation The range of composition, if Less than ,but = ,like Greater than ,but = .

[0099] The activation function layer constrains the final action value within the maximum speed range of the unmanned vehicle to ensure that the output is an allowed action. Its mathematical form is shown in the following equation (12).

[0100] (12).

[0101] In formula (12), is the final action, i.e. the speed of the unmanned vehicle at the current moment; tanh is the hyperbolic tangent function, is the maximum action value of the unmanned vehicle.

[0102] The logarithmic probability density calculation layer calculates the final action The logarithmic probability density of ensures that the final action taken is not only selected with a high probability according to the current strategy, but also takes into account the adjustment of the action value and the accurate representation of the probability density. Its mathematical form is shown in the following formula (13).

[0103] (13).

[0104] In formula (13), is the logarithmic probability density of the final action under a given strategy, is the mapping from state space to action space, that is, in a given state Take every possible action probability; The natural logarithm of the probability density under the probability distribution defined for the strategy; Compensatory adjustment introduced to change the output action distribution for the tanh function.

[0105] like Figure 5 As shown in the figure, a specific unmanned vehicle model and a training environment model are used as input. The lidar sensor on the unmanned vehicle model continuously interacts with the training environment to generate a positioning module, which is used to determine the position and posture of the unmanned vehicle, and extract the minimum distance information from the unmanned vehicle to the obstacle in front and the local occupancy grid map information. Combined with the target point position and posture, they are input into the reinforcement learning model together. After continuous trial and error training, the trained reinforcement learning model will output linear velocity and angular velocity according to real-time perception information, control the movement of the unmanned vehicle, and finally reach the target point. As the training progresses, the navigation strategy will be continuously optimized to select a better driving path.

[0106] Before using the target point posture and the current posture of the unmanned vehicle and the surrounding environment information as input and using the trained reinforcement learning model to determine the current speed of the unmanned vehicle, the autonomous navigation method of the unmanned vehicle in an unknown environment of this embodiment also includes the following steps.

[0107] (1) Obtain the sample state at the current moment. The sample state includes the target point posture and the unmanned vehicle posture and surrounding environment information at the current moment.

[0108] (2) Taking the sample state at the current moment as input, the sample action at the current moment is determined using the initial reinforcement learning model. The initial reinforcement learning model includes an action network, a first evaluation network, and a second evaluation network. The sample action includes the speed of the unmanned vehicle.

[0109] Determining the sample action of the sample at the current moment by using the initial reinforcement learning model specifically includes: determining the sample action of the sample at the current moment by using the action network of the initial reinforcement learning model.

[0110] This embodiment uses two evaluation networks and , which are respectively , Parameterization, each evaluation network performs a forward pass to output the estimated Q value of each sample state corresponding to the sample action. The Q value represents the expected future reward obtained by taking an action based on the state. The final Q value used for policy update is the minimum of the estimated Q values ​​calculated by the two evaluation networks. This can reduce the positive bias in policy update caused by overestimation of Q value.

[0111] (3) Control the unmanned vehicle to move according to the sample action at the current moment of the sample, determine the sample state, reward, and final value at the next moment of the sample, and combine the sample state and sample action at the current moment of the sample and the sample state, reward, and final value at the next moment of the sample into an experience.

[0112] The sample state at the next moment is obtained by interacting with the environment, the reward is calculated by the reward function, and the final value is the final Q value.

[0113] (4) Determine whether the unmanned vehicle has rolled over, skidded, or collided, or whether the unmanned vehicle has reached the target point, and obtain a judgment result.

[0114] (5) If the judgment result is no, the next moment of the sample is taken as the current moment of the sample, and the process returns to the step of "taking the sample state at the current moment of the sample as input and using the initial reinforcement learning model to determine the sample action at the current moment of the sample".

[0115] (6) If the judgment result is yes, all the experience obtained is used to train the initial reinforcement learning model to obtain the trained model, and then determine whether the iteration termination condition is met.

[0116] This embodiment applies entropy maximization to refine the target, thereby guiding subsequent network training. The mathematical form is shown in the following formula (14).

[0117] (14).

[0118] In formula (14), is the sample status; It is the experience replay buffer area, which stores all the experience gained; is the sample action; For action strategy; For the final value; It is a parameter that controls the effect of entropy on rewards. Network parameters The action network is trained by performing gradient ascent to improve the policy by leveraging the performance of the current policy and its exploration behavior.

[0119] The iteration termination condition can be reaching the target point multiple times in a row.

[0120] (7) If so, stop training and use the action network in the trained model as the trained reinforcement learning model.

[0121] (8) If not, continue training, use the trained model as the initial reinforcement learning model, and return to the step of “obtaining the sample state at the current moment”.

[0122] To reduce the risk of premature convergence to local optimum, an adaptive exploration noise module is integrated in this embodiment. This module aims to ensure comprehensive exploration of the solution by fine-tuning the balance between exploration and exploitation during training. The mathematical form is shown in the following equation (15).

[0123] (15).

[0124] In formula (15), is the exploration noise corresponding to the sample at the current moment; is the initial exploration noise. As the training progresses, the initial exploration noise is adjusted according to the predetermined minimum exploration noise value. and the total number of steps of noise attenuation This decrease continues until the exploration noise is below a predetermined minimum exploration noise value, which usually depends on the control accuracy of the unmanned vehicle. This adaptive strategy ensures that the unmanned vehicle has a good exploration mechanism when it first arrives in an unknown environment, which helps to discover different strategies. As training progresses, the exploration noise gradually decreases, and the focus will be on acquiring navigation strategies from historical experience, thereby achieving an effective balance between exploring new actions and exploiting known reward actions in complex environments.

[0125] Then the unmanned vehicle is controlled to move according to the sample action at the current moment of the sample, which specifically includes the following steps.

[0126] (1) Calculate the exploration noise corresponding to the sample at the current moment.

[0127] Use formula (15) to calculate the exploration noise corresponding to the sample at the current moment.

[0128] (2) Determine whether the exploration noise corresponding to the sample at the current moment is less than the minimum exploration noise.

[0129] (3) If not, then add the exploration noise corresponding to the sample at the current moment to the sample action at the current moment to obtain the noisy action of the sample at the current moment, and control the unmanned vehicle to move according to the noisy action of the sample at the current moment.

[0130] The calculation formula of the noise-added action of the sample at the current moment is shown in the following formula (16).

[0131] (16).

[0132] In formula (16), is the noise-added action of the sample at the current moment; is the sample action at the current moment of the sample; is the added normal distribution noise, The dimension of action; is a function that converts Limit to the action value of the unmanned vehicle between, is the minimum action value, is the maximum action value.

[0133] (4) If yes, then no exploration noise is added and the unmanned vehicle is directly controlled to move according to the sample action at the current moment.

[0134] This embodiment designs a reward function that takes into account the smoothness, safety and dynamic constraints of the unmanned vehicle's driving, which is a key component in evaluating the navigation and obstacle avoidance performance of the unmanned vehicle. The reward function accurately evaluates the actions taken by the unmanned vehicle in a specific environment based on the current state and the changes in the environmental state caused by the actions. Through this method, the unmanned vehicle can not only effectively navigate and avoid obstacles, but also adjust its behavior in real time to optimize the overall performance. Its mathematical form is shown in the following formula (17).

[0135] (17).

[0136] In formula (17), is the reward of the sample at the next moment; The current time of the sample The sample status; The current time of the sample Sample actions of Reward for reaching the target point. You can reach the target point by To characterize, is the distance from the unmanned vehicle to the target point, is the expansion coefficient of the target point, is the horizontal distance from the lidar sensor to the front of the unmanned vehicle; For collision rewards; Reward for rollover or sideslip; Rewards for approaching the target point; Reward for steering stability; Attract rewards for target points; Rewards for exploration; Reward for dynamic process.

[0137] The final reward depends on four parts. If the distance between the unmanned vehicle and the target point is less than the predetermined distance threshold , it is considered to have successfully reached the target point, and a positive reward is given If it is judged as a collision, a negative reward will be given If it is judged as a rollover or side slip, a negative reward will be given. If none of them, calculate the instant reward + + + + .

[0138] This embodiment provides a collision detection module that takes into account the kinematics of the unmanned vehicle. The collision detection module identifies and avoids obstacles in real time based on the local occupancy grid map obtained by S2. It uses polar coordinates to detect and calculate the minimum distance from these obstacles to the unmanned vehicle, so that the unmanned vehicle can move safely in a dynamic environment.

[0139] The grid coordinates of the detected obstacle in the local occupied grid map are converted from the grid map coordinate system to the vehicle body coordinate system. The mathematical form is shown in the following equation (18).

[0140] (18).

[0141] In formula (18), and is the coordinate of the obstacle in the vehicle coordinate system; and is the coordinate of the obstacle in the grid map coordinate system; is the current heading angle of the unmanned vehicle.

[0142] The scanning distance is gradually increased until the preset maximum scanning distance is reached or an obstacle is detected. Then, the obstacle coordinates are recorded and the minimum distance from the unmanned vehicle to the obstacle is calculated, or the distance from each obstacle to the unmanned vehicle is directly calculated to obtain the minimum distance from the unmanned vehicle to the obstacle.

[0143] Considering that the shortest stopping distance of an unmanned vehicle in an emergency braking situation depends on the current speed of the unmanned vehicle and the maximum braking acceleration it can achieve, the mathematical form of the shortest stopping distance of the unmanned vehicle is shown in the following formula (19).

[0144] (19).

[0145] In formula (19), is the shortest stopping distance; is the speed of the driverless car; It is the delay time from when the wire control system receives the speed command to when it implements it; is the maximum braking acceleration of the unmanned vehicle.

[0146] Consider the friction coefficient between the tire and the road The road surface conditions (such as dry, slippery, snowy or icy) may vary. To adjust, as shown in the following formula (20).

[0147] (20).

[0148] In formula (20), is the adjusted friction coefficient.

[0149] Based on equation (19) and equation (20), we can get equation (21).

[0150] (twenty one).

[0151] The calculation formula for the safety distance is shown in formula (22).

[0152] (twenty two).

[0153] In formula (22), For safe distance; is the minimum obstacle distance critical threshold.

[0154] Evaluate whether the minimum distance from the autonomous vehicle to the obstacle is less than the safe distance, and if so, mark the collision.

[0155] In order to ensure that the unmanned vehicle maintains high performance in complex off-road environments, this embodiment has specially developed a module that takes vehicle dynamics into consideration. This module introduces dynamic constraints (i.e., rollover constraints and sideslip constraints) that are strongly coupled with vehicle speed and acceleration in response to the risks of rollover and sideslip under off-road conditions. In order to improve the safety and stability of the unmanned vehicle on uneven roads, this embodiment integrates a dynamic penalty mechanism for undulating roads. This mechanism effectively optimizes the control strategy of the unmanned vehicle by accurately calculating and applying dynamic penalties, thereby ensuring excellent performance under various extreme terrain and environmental conditions, significantly improving the adaptability of the unmanned vehicle in off-road scenarios, and enhancing its operational safety and overall stability in complex environments.

[0156] In unmanned driving systems, integrating side slip constraints and rollover constraints based on vehicle dynamics is key to ensuring the safety of unmanned vehicles under various driving conditions. These constraints are designed to prevent unmanned vehicles from engaging in dangerous dynamic behaviors such as side slips and rollovers.

[0157] (1) Rollover restraint.

[0158] Rollover constraint involves preventing the unmanned vehicle from flipping over during sharp turns or lateral maneuvers. Rollover usually occurs when the lateral dynamics and high center of gravity position of the unmanned vehicle cause excessive force on the outer tires. Its mathematical form is shown in the following equation (23).

[0159] (twenty three).

[0160] In formula (23), is the lateral acceleration of the unmanned vehicle; is the wheelbase of the unmanned vehicle; is the acceleration due to gravity; is the height from the center of gravity of the unmanned vehicle to the ground. The above constraints ensure that the moment of the center of gravity under a given lateral acceleration will not cause the unmanned vehicle to roll over.

[0161] (2) Side slip restraint.

[0162] Side slip refers to the slip phenomenon caused by the lateral force between the tire and the road exceeding the tire grip when the unmanned vehicle turns at high speed or makes sudden maneuvers. The purpose of side slip constraint is to limit the lateral acceleration of the unmanned vehicle to prevent the vehicle from losing control due to excessive side slip. Its mathematical form is shown in the following equation (24).

[0163] (twenty four).

[0164] In formula (24), is the lateral acceleration of the unmanned vehicle; is the tire friction coefficient; is the acceleration due to gravity; is the inclination angle of the unmanned vehicle relative to the horizontal plane.

[0165] For an Ackerman steering vehicle, the lateral acceleration can be estimated directly by calculating the curvature of the path, and its mathematical form is shown in Equation (25).

[0166] (25).

[0167] (26).

[0168] In the above formula, is the curvature of the path; and is the first derivative of position with respect to time, representing the velocity component; and is the second derivative of position with respect to time, representing the acceleration component. These derivatives can be calculated numerically from the position data.

[0169] For a differential drive vehicle, the lateral acceleration of the vehicle can be directly calculated from the angular velocity and speed of the vehicle, and its mathematical form is shown in Equation (27).

[0170] (27).

[0171] In formula (27), is the angular velocity of the driverless car.

[0172] In the simulation scenario, the inclination angle of the ground is determined according to the actual inclination angle of the vehicle, and the center of gravity height of the vehicle is determined in the simulation according to the standard center of gravity height when the vehicle is loaded and unloaded.

[0173] This part aims to monitor the lateral acceleration of the unmanned vehicle in real time when performing off-road navigation tasks, and compare it with the predefined sideslip and rollover constraints. If the lateral acceleration is detected to exceed these constraints, negative rewards will be automatically implemented and the training will be terminated. This negative reward can instantly adjust the driving behavior of the unmanned vehicle, reduce the risk of sideslip and rollover, and thus optimize the dynamic stability and safety of the vehicle. In this way, the unmanned vehicle can respond more accurately to complex and changing off-road environments.

[0174] In this embodiment, determining whether the unmanned vehicle has rolled over, skidded, or collided, or whether the unmanned vehicle has reached the target point specifically includes the following steps.

[0175] (1) Determine whether the unmanned vehicle meets the rollover constraint. If not, the unmanned vehicle rolls over. The rollover constraint is: ;in, is the lateral acceleration of the unmanned vehicle; is the wheelbase of the unmanned vehicle; is the acceleration due to gravity; is the height from the center of gravity of the unmanned vehicle to the ground.

[0176] (2) Determine whether the unmanned vehicle satisfies the sideslip constraint. If not, the unmanned vehicle will side-slip. The sideslip constraint is: ;in, is the lateral acceleration of the unmanned vehicle; is the tire friction coefficient; is the acceleration due to gravity; is the inclination angle of the unmanned vehicle relative to the horizontal plane.

[0177] (3) Based on the local occupancy grid map at the next moment of the sample, calculate the minimum distance between the unmanned vehicle and the obstacle; based on the speed of the unmanned vehicle at the next moment of the sample and the maximum braking acceleration of the unmanned vehicle, calculate the safe distance of the unmanned vehicle during emergency braking; determine whether the minimum distance is less than the safe distance. If not, the unmanned vehicle will collide.

[0178] (4) Calculate the distance from the unmanned vehicle to the target point at the next moment of the sample, and determine whether the distance is less than the distance threshold. If so, the unmanned vehicle reaches the target point.

[0179] If none of these three conditions are met, the immediate reward is calculated based on the current state of the unmanned vehicle and the environmental variables to effectively guide the navigation strategy. The calculation method of the immediate reward is as follows.

[0180] Represents the reward for approaching the target point, which is determined based on the change in the distance between consecutive frames. This reward encourages the unmanned vehicle to approach the target point effectively. The calculation formula for the reward for approaching the target point is shown in the following formula (28).

[0181] (28).

[0182] In formula (28), is the distance from the unmanned vehicle to the target point at the current moment of the sample; is the distance from the unmanned vehicle to the target point at the next moment of the sample.

[0183] represents the steering stability reward involving the dynamics of the unmanned vehicle. This reward is designed to prevent frequent and high-intensity steering under high-speed conditions and promote smoother and more predictable navigation. The calculation formula of the steering stability reward is shown in the following formula (29).

[0184] (29).

[0185] In formula (29), is the linear velocity of the sample at the next moment; is the angular velocity of the sample at the next moment.

[0186] The target point attraction reward assigned to the target point attraction area is intended to motivate the unmanned vehicle to drive to the target point more accurately when approaching the target point. The calculation formula of the target point attraction reward is shown in the following formula (30).

[0187] (30).

[0188] The exploration reward represents weak exploration awareness and is applicable when far away from the target point. It encourages active exploration. The calculation formula of the exploration reward is shown in the following formula (31).

[0189] (31).

[0190] Dynamic Process Rewards The integration of the angle between the unmanned vehicle and the target point and the minimum distance between the unmanned vehicle and the obstacle can encourage effective autonomous navigation strategies while minimizing unnecessary turns and maintaining a safe distance from obstacles. The calculation formula for the dynamic process reward is shown in Equation (32).

[0191] (32).

[0192] In formula (32), is the angle between the unmanned vehicle and the target point; is the minimum distance from the unmanned vehicle to the obstacle.

[0193] (33).

[0194] In the calculation formulas of the above five rewards, if the if condition is not met, 0 is taken.

[0195] This embodiment proposes an autonomous navigation training method based on the SAC algorithm, which integrates obstacle distance data, the real-time status information of the unmanned vehicle and its local occupancy grid map through a deep reinforcement learning framework, and can learn and optimize its navigation decisions in real time when facing a complex and changeable driving environment, thereby greatly improving the environmental adaptability and operating efficiency of the unmanned vehicle. The detailed training process is shown in Table 1 below.

[0196] Table 1 Training process

[0197]

[0198] In order to better illustrate the advantages of this embodiment, an example is given below.

[0199] In order to adapt to various types of unmanned vehicles and different mission scenarios, the vehicle system dynamics and simulation training scenarios are input into the algorithm framework as independent inputs. This strategy allows customized training for each specific vehicle model and mission scenario, thereby optimizing the navigation performance of the unmanned vehicle. In order to fully verify and optimize, the reward strategy of this embodiment takes into account the possible impact of diverse environments such as urban traffic to complex off-road on the autonomous navigation of unmanned vehicles, ensuring that the system can adapt to the needs of different navigation tasks. This approach not only enhances the adaptability of unmanned vehicles to the environment, but also significantly improves their performance and reliability under various operating conditions.

[0200] Taking a simple indoor environment as an example, when an unmanned vehicle wants to perform autonomous navigation in an indoor environment, but currently only has a vehicle model and cannot perform real vehicle debugging, the self-optimization method for autonomous navigation in an unknown environment of this embodiment can be used. Figure 6 Training is first performed in the simulation environment shown and then transplanted to a specific unmanned vehicle for autonomous navigation. Figure 6 In the figure, the simple indoor environment includes a wall 1, a table 2 and a chair 3, and the unmanned vehicle 4 moves in the simple indoor environment.

[0201] The training process is as follows: At the beginning of the training, a target point will be randomly released. The position of the target point will gradually move away from the unmanned vehicle as the training runs to achieve better navigation results. The surrounding environment is represented by three-dimensional point cloud data. The environment consists of two parts, namely the distance information of the nearest obstacle (i.e., the minimum distance between the unmanned vehicle and the obstacle in front) and the local occupancy grid map information. Combining the surrounding environment information and the vehicle status (i.e., the position of the unmanned vehicle), it is possible to interact with the simulation environment and train the model. In this process, considering the safety of the vehicle is the most important point in the vehicle driving process. Because this environment is an indoor environment and does not involve undulating road scenes, therefore, during the training process, only the collision safety of the unmanned vehicle needs to be considered. The unmanned vehicle model constantly interacts with the environment and combines the surrounding environment information and vehicle status information to determine in real time whether a collision occurs. If a collision occurs, the collision will be marked, indicating that the current training cycle is over. After that, the network, environment, and vehicle model will be reinitialized to start a new training cycle. If no collision is detected and the target point is not reached, the corresponding reward will be calculated and distributed in real time according to the current state of the vehicle and the specific conditions of the surrounding environment. This reward mechanism is based on real-time evaluation of the vehicle's operating performance and the interaction with the environment. It aims to promote the vehicle to move towards the target point more effectively while avoiding collisions, improve the autonomous navigation performance and safety of the unmanned vehicle, and adjust the behavior of the unmanned vehicle through real-time feedback to achieve the best operating effect. After repeated training, the model eventually learns to drive to the target point autonomously, and as the training continues, it can continuously optimize its navigation strategy to complete the navigation task more safely and efficiently.

[0202] The trained model was saved as a .npy file and was then successfully migrated to a real-world autonomous vehicle that was the same model as the one used in the training environment. To ensure the operational performance of the autonomous vehicle, the relevant sensors were connected to ensure that the sensor data could be read and processed normally.

[0203] The results show that by using the trained model, the unmanned vehicle can achieve autonomous navigation in unknown environments without complex manual parameter adjustments. This technology ensures that the unmanned vehicle can reach the designated target point safely, smoothly and quickly. More importantly, as the training progresses, the model can continuously optimize its performance, further improving the efficiency and safety of the unmanned vehicle in reaching the target point, and is used to enhance the navigation ability of the unmanned vehicle in complex environments, while reducing the need for manual intervention and significantly improving the level of automation of operations.

[0204] The embodiment of the present application also provides an application scenario, which applies the above-mentioned method for autonomous navigation of an unmanned vehicle in an unknown environment. Specifically, the method for autonomous navigation of an unmanned vehicle in an unknown environment provided in this embodiment can be applied in the scenario of autonomous navigation of an unmanned vehicle. The scenario of autonomous navigation of an unmanned vehicle includes a data acquisition link and a data processing link. The data acquisition link is used to acquire three-dimensional point cloud data, and the data processing link is used to process the three-dimensional point cloud data, control the movement of the unmanned vehicle in an unknown environment, until the unmanned vehicle reaches the target point, thereby completing the autonomous navigation of the unmanned vehicle in an unknown environment. The method for autonomous navigation of an unmanned vehicle in an unknown environment provided in this embodiment belongs to the data processing link.

[0205] Example 2.

[0206] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data to be processed. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for autonomous navigation of an unmanned vehicle in an unknown environment is implemented.

[0207] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0208] In an exemplary embodiment, a computer device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the autonomous navigation method of the unmanned vehicle in an unknown environment described in Example 1.

[0209] Example 3.

[0210] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for autonomous navigation of an unmanned vehicle in an unknown environment described in Embodiment 1 is implemented.

[0211] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0212] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. An autonomous navigation method for an unmanned vehicle in an unknown environment, characterized in that: The autonomous navigation method of the unmanned vehicle in an unknown environment includes: Acquire three-dimensional point cloud data at the current moment; the three-dimensional point cloud data is data obtained by a laser radar sensor installed on the unmanned vehicle scanning the surrounding environment of the unmanned vehicle; Processing the three-dimensional point cloud data at the current moment to obtain the position and surrounding environment information of the unmanned vehicle at the current moment; the surrounding environment information includes the minimum distance from the unmanned vehicle to the obstacle in front and a local occupancy grid map, wherein the local occupancy grid map is an occupancy grid map of a local area with the center point of the unmanned vehicle as the center point and a preset length as the side length; Taking the target point position and the current position of the unmanned vehicle and the surrounding environment information as input, the trained reinforcement learning model is used to determine the speed of the unmanned vehicle at the current moment; the speed of the unmanned vehicle includes linear speed and angular speed; Control the unmanned vehicle to move at the current speed and determine whether the unmanned vehicle has reached the target point. If not, return to the step of "obtaining the three-dimensional point cloud data at the current moment"; The acquisition of the local occupancy grid map includes a positioning calculation step and a local occupancy grid map generation step; In the positioning calculation step, the mathematical form of feature point extraction is: ； in, For feature points A collection of; It is three-dimensional point cloud data; For feature points The local curvature of is the set local curvature threshold; For feature points The local planarity of is the set local planarity threshold; In the step of generating the local occupancy grid map, nonlinear interpolation technology is used to synchronize the asynchronous 3D point cloud data and the posture of the unmanned vehicle. The posture interpolation is based on quaternion and exponential mapping theory to ensure the smooth transition of continuous postures. The mathematical form is: ； in, For the current moment The corresponding transformation matrix is, It is the timestamp of the 3D point cloud data at the current moment, that is, the timestamp of the unmanned vehicle’s position at the current moment; is the first transformation matrix of the last moment compared to the moment before that; is the second transformation matrix between the current moment and the previous moment; To obtain the timestamp of the first transformation matrix; To obtain the timestamp of the second transformation matrix; Calculate the current 3D point cloud data and the current time The product of the corresponding transformation matrices gives , which is the precise three-dimensional point cloud data obtained by the above-mentioned pose interpolation. The precise three-dimensional point cloud data is further converted into the vehicle body coordinate system. The mathematical form is: ； in, It is local point cloud data; and are the rotation matrix and translation vector from the local coordinate system to the vehicle body coordinate system respectively.

2. The autonomous navigation method of an unmanned vehicle in an unknown environment according to claim 1, characterized in that: The trained reinforcement learning model includes a first feature extraction branch, a second feature extraction branch and an action determination branch; The first feature extraction branch is used to extract features from the local occupancy grid map at the current moment to obtain a first feature; the first feature extraction branch includes a plurality of convolutional layers, a flattening layer and a plurality of fully connected layers connected in sequence; The second feature extraction branch is used to extract features from the target point posture, the unmanned vehicle posture at the current moment, and the minimum distance from the unmanned vehicle to the obstacle in front, to obtain a second feature; the second feature extraction branch includes a concatenation layer and a fully connected layer connected in sequence; The action determination branch is used to process the first feature and the second feature to obtain the speed of the unmanned vehicle at the current moment; the action determination branch includes a splicing layer, multiple fully connected layers and a normalization layer connected in sequence.

3. The autonomous navigation method of an unmanned vehicle in an unknown environment according to claim 1, characterized in that: Before using the target point position and the current position and surrounding environment information of the unmanned vehicle as inputs and using the trained reinforcement learning model to determine the current speed of the unmanned vehicle, the autonomous navigation method of the unmanned vehicle in an unknown environment also includes: Obtaining the sample state at the current moment of the sample; the sample state includes the target point posture and the unmanned vehicle posture and surrounding environment information at the current moment of the sample; Taking the sample state of the sample at the current moment as input, an initial reinforcement learning model is used to determine the sample action of the sample at the current moment; the initial reinforcement learning model includes an action network, a first evaluation network and a second evaluation network; the sample action includes the speed of the unmanned vehicle; Control the unmanned vehicle to move according to the sample action at the current moment of the sample, determine the sample state, reward and final value at the next moment of the sample; combine the sample state and sample action at the current moment of the sample and the sample state, reward and final value at the next moment of the sample into an experience; Determine whether the unmanned vehicle has rolled over, skidded, or collided, or whether the unmanned vehicle has reached the target point, and obtain a judgment result; If the judgment result is no, the next moment of the sample is taken as the current moment of the sample, and the process returns to the step of "taking the sample state at the current moment of the sample as input and using the initial reinforcement learning model to determine the sample action at the current moment of the sample"; If the judgment result is yes, then all the obtained experiences are used to train the initial reinforcement learning model to obtain a trained model, and determine whether the iteration termination condition is met; If yes, stop training and use the action network in the trained model as the trained reinforcement learning model; If not, continue training, use the trained model as the initial reinforcement learning model, and return to the step of "obtaining the sample state at the current moment of the sample".

4. The autonomous navigation method of an unmanned vehicle in an unknown environment according to claim 3, characterized in that: Control the unmanned vehicle to move according to the sample action at the current moment of the sample, including: Calculate the exploration noise corresponding to the sample at the current moment; Determine whether the exploration noise corresponding to the sample at the current moment is less than the minimum exploration noise; If not, then add the exploration noise corresponding to the current moment of the sample to the sample action at the current moment of the sample to obtain the noise-added action at the current moment of the sample, and control the unmanned vehicle to move according to the noise-added action at the current moment of the sample; If so, the unmanned vehicle is controlled to move according to the sample action at the current moment of the sample.

5. The autonomous navigation method of an unmanned vehicle in an unknown environment according to claim 3, characterized in that: The reward calculation formula is: ； in, is the reward of the sample at the next moment; The current time of the sample The sample status; The current time of the sample Sample actions of Rewards for reaching the target point; For collision rewards; Reward for rollover or sideslip; Rewards for approaching the target point; Reward for steering stability; Attract rewards for target points; Rewards for exploration; Reward for dynamic process.

6. The autonomous navigation method of an unmanned vehicle in an unknown environment according to claim 5, characterized in that: The calculation formula for the reward for approaching the target point is: ； in, is the distance from the unmanned vehicle to the target point at the current moment of the sample; is the distance from the unmanned vehicle to the target point at the next moment of the sample; The calculation formula for the steering stability bonus is: ； in, is the linear velocity of the sample at the next moment; is the angular velocity of the sample at the next moment; The calculation formula for the target point attraction reward is: ； The calculation formula for exploration rewards is: ； The calculation formula for dynamic process rewards is: ； in, is the angle between the unmanned vehicle and the target point; is the minimum distance from the unmanned vehicle to the obstacle.

7. The autonomous navigation method of an unmanned vehicle in an unknown environment according to claim 3, characterized in that: Determine whether the unmanned vehicle has rolled over, skidded, or collided, or whether the unmanned vehicle has reached the target point, including: Determine whether the unmanned vehicle meets the rollover constraint. If not, the unmanned vehicle rolls over. The rollover constraint is: ;in, is the lateral acceleration of the unmanned vehicle; is the wheelbase of the unmanned vehicle; is the acceleration due to gravity; The height from the center of gravity of the unmanned vehicle to the ground; Determine whether the unmanned vehicle satisfies the side slip constraint. If not, the unmanned vehicle will side slip. The side slip constraint is: ;in, is the lateral acceleration of the unmanned vehicle; is the tire friction coefficient; is the acceleration due to gravity; is the inclination angle of the unmanned vehicle relative to the horizontal plane; Based on the local occupancy grid map at the next moment of the sample, the minimum distance from the unmanned vehicle to the obstacle is calculated; based on the speed of the unmanned vehicle at the next moment of the sample and the maximum braking acceleration of the unmanned vehicle, the safe distance of the unmanned vehicle during emergency braking is calculated; it is determined whether the minimum distance is less than the safe distance, if not, the unmanned vehicle collides; Calculate the distance from the unmanned vehicle to the target point at the next moment of the sample, and determine whether the distance is less than the distance threshold. If so, the unmanned vehicle reaches the target point.

8. The autonomous navigation method of an unmanned vehicle in an unknown environment according to claim 4, characterized in that: The calculation formula of the exploration noise corresponding to the current moment of the sample is: ； in, is the exploration noise corresponding to the sample at the current moment; To explore the minimum value of noise; is the initial exploration noise; is the total number of steps of noise attenuation; The calculation formula of the noise-added action of the sample at the current moment is: ； in, is the noise-added action of the sample at the current moment; is a function that converts Limit to the action value of the unmanned vehicle between; is the sample action at the current moment of the sample; is the added normal distribution noise, is the standard deviation, The dimension of action; is the minimum action value; is the maximum action value.

9. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the autonomous navigation method of an unmanned vehicle in an unknown environment as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the autonomous navigation method of the unmanned vehicle in an unknown environment described in any one of claims 1-8 is implemented.

Images