Trolley obstacle avoidance method based on deep reinforcement learning

Abstract

The invention provides a dolly obstacle avoidance method based on deep reinforcement learning. The method comprises the following steps: acquiring image data of an environment state through a camera on a dolly; inputting the image data into a trained first neural network model, and outputting corresponding 3D point cloud data; inputting the 3D point cloud data into a trained second neural network model, and outputting an action selection probability value of the trolley; and controlling the trolley to execute corresponding actions according to the action selection probability value. According to the vehicle obstacle avoidance method based on deep reinforcement learning, the obstacle avoidance logic in a complex environment is simplified, the flexibility and robustness of the system are greatly improved, and an efficient solution is provided for reliable application of the unmanned driving technology in a dynamic road scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of machine learning, in particular to a small vehicle obstacle avoidance method based on deep reinforcement learning. BACKGROUND

[0002] The safe operation of unmanned vehicles highly depends on efficient and reliable obstacle avoidance technology. Traditional obstacle avoidance methods are mainly based on a pre-defined rule system, combined with environmental data collected by sensors such as laser radars and cameras for obstacle detection and path planning.

[0003] The above method can meet the basic needs in structured roads or relatively simple scenes, but its limitations are increasingly prominent in complex and variable unstructured environments, such as densely populated areas, construction sections, or sudden obstacles in dynamic scenarios. The existing technology lacks adaptability to the environment, making it difficult to respond to unforeseen obstacle shapes or sudden conditions in real time. The flexibility of path planning is also limited by the rigid framework of pre-set rules. In addition, the detection performance of sensors fluctuates under extreme weather or special lighting conditions, further restricting the reliability and generalization ability of traditional methods in real road environments. As the application scenarios of autonomous driving continue to expand, how to achieve more intelligent and flexible obstacle avoidance decisions in highly dynamic and unstructured real-world environments has become a technical problem to be solved. SUMMARY

[0004] The present application provides a small vehicle obstacle avoidance method based on deep reinforcement learning to solve the above technical problems.

[0005] The small vehicle obstacle avoidance method based on deep reinforcement learning provided by the present application includes the following steps: Obtain image data of the environmental state through the camera on the small vehicle; Input the image data into the trained first neural network model to output corresponding 3D point cloud data; Input the 3D point cloud data into the trained second neural network model to output the action selection probability value of the small vehicle; Control the small vehicle to perform the corresponding action according to the action selection probability value.

[0006] In one embodiment, the training method of the first neural network model includes the following steps: Construct a training data set including RGB images and three-dimensional point cloud true value correspondence, and perform image normalization and point cloud standardization processing; Use a cascading training strategy to sequentially train the feature extraction module, the voxel generation module, and the point cloud decoding module; Activate the geometric optimization module when the chamfer distance of the validation set is below the threshold value, and optimize the point cloud surface continuity through the differentiable Poisson reconstruction layer; An end-to-end fine-tuning is performed by using a composite loss function comprising a chamfer distance, an earth mover distance, and a normal vector consistency constraint.

[0007] In an embodiment, the cascaded training strategy comprises: The weights of a fixed ResNet-50 backbone network are fixed, only the feature pyramid network (FPN) layers are trained, and a VGG16 perception loss is used for supervision. A three-dimensional convolution network is used to generate a 64x64x64 probability voxel grid, and an Adam optimizer is used with an initial learning rate of 1e-4. A point cloud decoder is constructed based on a tree graph convolution network (Tree-GCN), and the output point density is gradually increased in stages.

[0008] In an embodiment, the optimization of point cloud surface continuity comprises the following steps: An implicit surface function gradient field VS is calculated, and a Poisson equation is used to calculate the gradient field VS. A continuous surface is reconstructed. A moving average method is used to update the normal vector field, and the update formula is V new = 0.9V prev + 0.1VS.

[0009] In an embodiment, the training method of the first neural network model further comprises an adversarial training step, which comprises the following steps: A discriminator network based on the PointNet++ architecture is constructed, and a mixed set of generated point clouds and real point clouds is input. The generator and the discriminator are alternately updated at a ratio of 3:1, and the adversarial loss function is: L adv =E[logD(P real )]+E[log(1-D(P fake ))]。

[0010] In an embodiment, the training method of the second neural network model comprises the following steps: The second neural network model is built using Tensorflow, the input is 3D point cloud data, and the output is an action selection probability value. The pygame module in the Python library is used to simulate a straight road with obstacles, an agent role in reinforcement learning is used, a car is used as an agent to interact with the simulated road environment with obstacles, and interaction data is collected. The DQN algorithm is used to train the second neural network model and optimize the model parameters.

[0011] In an embodiment, the interaction data generated when the car interacts with the simulated road environment is placed in an experience pool. The interaction data comprises: a car action a, a current state s of an environment, a next state s of the environment, and a reward r given by the environment. t+1

[0012] In an embodiment, the method for optimizing model parameters comprises the following steps: randomly sampling a small batch of interaction data from an experience pool; calculating a target Q value using a target network, wherein: wherein θ - is a target network parameter, and γ is a discount factor; calculating a mean square error loss of a current Q value and the target Q value, and updating a main network parameter θ by back propagation; periodically copying the main network parameter θ to the target network parameter θ - ; adopting an epsilon-greedy strategy to balance exploration and utilization, and gradually decaying an exploration rate epsilon.

[0013] In an embodiment, the experience pool adopts a FIFO mechanism to manage data, and when the experience pool is filled, the earliest stored data is deleted and the latest interaction data is stored.

[0014] In an embodiment, the action selection probability value comprises a steering, acceleration and deceleration instruction, and the car performs obstacle avoidance driving according to the highest probability action.

[0015] Advantages: The car automatically avoids obstacles through deep reinforcement learning. Compared with the traditional obstacle avoidance method based on predefined rules and fixed sensor input, the deep reinforcement learning technology enables the car to autonomously learn the optimal strategy in a dynamic environment. The deep neural network efficiently extracts environmental features through a multi-layer convolution and full connection structure, and optimizes action selection in combination with the DQN algorithm, so that the car can make accurate steering decisions according to the real-time perceived state. The experience pool mechanism ensures the diversity and timeliness of the training data, and optimizes the model parameters by continuously updating the interaction data, effectively avoiding overfitting. This method not only simplifies the obstacle avoidance logic in complex environments, but also greatly improves the flexibility and robustness of the system, providing an efficient solution for the reliable application of unmanned driving technology in simple road scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the contents of the embodiments of the present application and these drawings.​

[0017] Figure 1 is a step flow chart of the car obstacle avoidance method based on deep reinforcement learning provided by the embodiment of the present application; Figure 2 is a feature extraction and decision structure diagram of the environment state in the present application; Figure 3 is a storage structure diagram of experience data in the experience pool in the present application; Figure 4 is a process diagram of the deep neural network model guiding the car and the environment interaction in the present application. DETAILED DESCRIPTION

[0018] The present application will be further described below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings, not all the structures.

[0019] In the description of the present application, unless otherwise explicitly specified and limited, the terms "connected", "connected", "fixed" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrated; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the internal communication of two elements or the interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0020] In the present application, unless otherwise explicitly specified and limited, the first feature "on" or "under" the second feature can include that the first and second features are in direct contact, or that the first and second features are not in direct contact but are in contact through another feature between them. Moreover, the first feature "on", "above" and "above" the second feature includes that the first feature is directly above and obliquely above the second feature, or only indicates that the horizontal height of the first feature is higher than that of the second feature. The first feature "below", "below" and "below" the second feature includes that the first feature is directly below and obliquely below the second feature, or only indicates that the horizontal height of the first feature is less than that of the second feature.

[0021] In the description of the present embodiment, the terms "up", "down", "left", "right" and other orientation or position relationship are based on the orientation or position relationship shown in the drawings, which is only for the convenience of description and simplification of operation, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first" and "second" are only used to distinguish in the description, and have no special meaning.

[0022] As Figure 1 shown, the embodiment provides a dolly obstacle avoidance method based on deep reinforcement learning, comprising the following steps: S1, obtaining image data of the environment state through the camera on the dolly; S2, inputting the image data into the trained first neural network model to output corresponding 3D point cloud data; S3, inputting the 3D point cloud data into the trained second neural network model to output the action selection probability value of the dolly; wherein the second neural network model is a deep neural network model; S4, controlling the dolly to perform the corresponding action according to the action selection probability value.

[0023] Specifically, the action selection probability value includes steering, acceleration and deceleration instructions, and the dolly performs obstacle avoidance driving according to the highest probability action.

[0024] The scheme provided by the embodiment can realize the obstacle avoidance function of the dolly in a dynamic and complex environment. The 3D point cloud contains the geometric structure of the environment, is suitable for spatial reasoning, improves the robustness of the system, and does not need to rely on 3D sensors to obtain point clouds. Only a normal camera needs to be installed on the dolly, and the cost is low.

[0025] In one embodiment, the training method of the first neural network model comprises the following steps: S21, constructing a training data set comprising RGB images and three-dimensional point cloud true values, performing image normalization and point cloud standardization processing; S22, using a cascade training strategy to sequentially train the feature extraction module, the voxel generation module and the point cloud decoding module; S23, activating the geometric optimization module when the chamfer distance of the validation set is lower than the threshold, and optimizing the surface continuity of the point cloud through the differentiable Poisson reconstruction layer; S24, using a composite loss function containing chamfer distance, earth moving distance and normal vector consistency constraint for end-to-end fine-tuning; Wherein, the composite loss function is represented as: L total =0.6L CD +0.3L EMD +0.1L normal Wherein, L CD chamfer distance, L EMD earth moving distance, L normal normal vector consistency loss. Chamfer distance L CD is calculated by bidirectional nearest neighbor search, normal vector consistency constraint L normal is realized by local plane fitting.

[0026] S25, a discriminator network based on the PointNet++ architecture is constructed, and a mixed set of generated point clouds and real point clouds is input; S26, the generator and the discriminator are alternately updated at a ratio of 3:1, and the adversarial loss function is: adv =E[logD(P real )]+E[log(1-D(P fake ))]。

[0027] Wherein, the adversarial loss function is the core component of the generative adversarial network, and its essence is to let the two neural networks compete with each other, and the generator is responsible for generating false data, and the discriminator is responsible for judging whether the data is real or generated. Specifically, the generator inputs an image and outputs the predicted point cloud P fake , the discriminator simultaneously accepts the real point cloud P real and the generated point cloud P fake , and outputs a probability value 0-1, the adversarial loss function is composed of the recognition accuracy of the discriminator for real data logD(P real ) and the recognition error of the discriminator for generated data log(1-D(P fake )), the goal of the generator is to minimize the loss to deceive the discriminator, and the goal of the discriminator is to maximize the loss to improve the discrimination ability, and the two are optimized through the adversarial mechanism. The performance of the model.

[0028] Specifically, in the adversarial training phase, automatic mixed precision AMP is enabled, and the gradient scaling factor is set to 1024.

[0029] In step S21, the preprocessing of the data includes the following steps: S211, color jitter enhancement is performed on the RGB image, and the parameters include brightness adjustment range ±0.2 and contrast adjustment range ±0.3; S212, rigid transformation enhancement is performed on the three-dimensional point cloud, including rotating around the Z axis by ±15° and translating by ±0.1 meters; S213, the output point cloud density is dynamically adjusted to 512-2048 points by the farthest point sampling algorithm.

[0030] In one embodiment, the cascade training strategy includes: S221, fix the ResNet-50 backbone network weight, only train the feature pyramid network FPN layer, and use VGG16 perception loss supervision; S222, a three-dimensional convolutional network is used to generate a 64x64x64 probability voxel grid, and the optimizer is Adam and the initial learning rate is 1e-4; S223, a point cloud decoder is constructed based on a tree graph convolutional network Tree-GCN, and the output point density is gradually increased in stages.

[0031] In an embodiment, the optimization of the point cloud surface continuity comprises the following steps: S231, calculating the implicit surface function gradient field ∇S by the Poisson equation to reconstruct a continuous surface; wherein f is an implicit surface function, denotes the second-order spatial derivative of the function f, VS is the input gradient field, denotes the local variation of the input gradient field.

[0032] S232, updating the normal vector field using the moving average method, and the update formula is V new = 0.9V prev + 0.1VS. Wherein V new is the updated normal vector field, V prev is the normal vector field of the last iteration.

[0033] The working principle of the geometry optimization module is as follows: first, the discrete point cloud is converted into an implicit function f(p) to define the object surface (f(p)=0), the interior (f(p)<0) and the exterior (f(p)>0); then, by using the normal vector field of the point cloud, the Poisson equation is solved by the Poisson reconstruction method to generate a continuous surface; finally, the process is embedded in a neural network in a differentiable form, and by inputting a rough point cloud and predicting a normal vector, an optimized smooth surface is output, and the surface smoothness and the fitting degree to the original point cloud are simultaneously constrained by a loss function, so that end-to-end training and optimization are realized.

[0034] In an embodiment, the training method of the second neural network model comprises the following steps: S31, using Tensorflow to build the second neural network model, and the input is 3D point cloud data and the output is action selection probability value; S32, using the pygame module in the Python library to simulate a straight road with obstacles, using the agent role in reinforcement learning, and using the car as the agent to interact with the simulated road environment with obstacles to collect interaction data; Specifically, the interaction data generated when the car interacts with the simulated road environment is put into the experience pool. The interaction data includes: car action a, current state s of the environment, next state s t+1 of the environment and the reward r given by the environment.

[0035] Further, in the above steps, Prioritized Experience Replay is used, and experience data is sampled according to the priority of TD error δ i , wherein the priority . Wherein, .

[0036] Specifically, the experience pool adopts a FIFO mechanism to manage data, and when the experience pool is filled, the earliest stored data is discarded and the latest interaction data is stored.

[0037] S33, using the DQN algorithm to train the second neural network model, and optimizing the model parameters.

[0038] In an embodiment, the method of optimizing the model parameters comprises the following steps: S331, randomly sampling a small batch of interaction data from the experience pool; S332, calculating the target Q value using the target network, wherein: wherein, θ - is the target network parameter, and gamma is the discount factor; S333, calculating the mean square error loss of the current Q value and the target Q value, and updating the main network parameter theta through back propagation; S334, periodically copying the main network parameter theta to the target network parameter theta - ; S335, using the epsilon-greedy strategy to balance exploration and utilization, and gradually decaying the exploration rate epsilon.

[0039] The car obstacle avoidance method based on deep reinforcement learning provided by the present application has significant advantages compared to traditional obstacle avoidance techniques. Through the combination of deep neural networks and DQN algorithms, the system can autonomously learn the optimal obstacle avoidance strategy from environmental interactions, eliminating the dependence on predefined rules and effectively improving the adaptability in dynamic and complex environments. The method uses the experience pool mechanism to store and update interaction data, enabling the model to continuously optimize decision parameters, ensuring learning efficiency and enhancing generalization performance. Compared to traditional methods, the present application not only simplifies the complexity of system design, but also significantly improves the accuracy of obstacle avoidance decisions and environmental adaptability, providing a more intelligent and reliable solution for the safe operation of unmanned vehicles in unstructured environments.

[0040] Obviously, the above embodiments of the present application are only examples for clarity, and are not intended to limit the embodiments of the present application. For those skilled in the art, various obvious changes, readjustments and substitutions can be made without departing from the scope of the present application. Here, it is not necessary or possible to exhaust all embodiments. Any modification, equivalent substitution and improvement made within the spirit and principles of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. A method for obstacle avoidance in a vehicle based on deep reinforcement learning, characterized in that, Includes the following steps: The vehicle acquires image data of the environmental conditions using its onboard camera. The image data is input into the trained first neural network model, and the corresponding 3D point cloud data is output. The 3D point cloud data is input into the trained second neural network model, which outputs the probability value of the car's action selection. Based on the selected probability value for the action, the robot is controlled to perform the corresponding action.

2. The obstacle avoidance method for a car based on deep reinforcement learning according to claim 1, characterized in that, The training method for the first neural network model includes the following steps: Construct a training dataset that includes RGB images and ground truth values ​​of 3D point clouds, and perform image normalization and point cloud standardization processing; The feature extraction module, voxel generation module, and point cloud decoding module are trained sequentially using a cascaded training strategy. When the chamfer distance on the validation set is below a threshold, the geometry optimization module is activated to optimize the surface continuity of the point cloud through a differentiable Poisson reconstruction layer. End-to-end fine-tuning is performed using a composite loss function that includes chamfer distance, Earth movement distance, and normal vector consistency constraints.

3. The obstacle avoidance method for a car based on deep reinforcement learning according to claim 2, characterized in that, Cascaded training strategies include: The weights of the ResNet-50 backbone network are fixed, and only the feature pyramid network FPN layer is trained, with VGG16 perceptual loss used for supervision. A 3D convolutional network is used to generate a 64×64×64 probability voxel grid. The optimizer is Adam and the initial learning rate is 1e-4. A point cloud decoder is constructed based on the Tree-GCN convolutional network, and the output point density is gradually increased in stages.

4. The obstacle avoidance method for a car based on deep reinforcement learning according to claim 2, characterized in that, Optimizing point cloud surface continuity includes the following steps: Calculate the implicit surface function gradient field VS using the Poisson equation. Reconstructing a continuous surface; The normal vector field is updated using the moving average method, and the update formula is V. new =0.9V prev +0.1VS.

5. The obstacle avoidance method for a car based on deep reinforcement learning according to claim 2, characterized in that, The training method for the first neural network model further includes an adversarial training step, which comprises the following steps: Construct a discriminator network based on the PointNet++ architecture, and input a mixture of generated point clouds and real point clouds; The generator and discriminator are updated alternately in a 3:1 ratio, and the adversarial loss function is: L adv =E[logD(P real )]+E[log(1-D(P fake ))).

6. The obstacle avoidance method for a car based on deep reinforcement learning according to claim 1, characterized in that, The training method for the second neural network model includes the following steps: The second neural network model is built using Tensorflow, with 3D point cloud data as input and action selection probability values ​​as output. The pygame module in the Python library is used to simulate a straight road with obstacles. The car is used as an agent in reinforcement learning to interact with the simulated road environment with obstacles and to collect interaction data. The second neural network model was trained using the DQN algorithm to optimize the model parameters.

7. The obstacle avoidance method for a vehicle based on deep reinforcement learning according to claim 6, characterized in that, The interaction data generated when the car interacts with the simulated road environment is placed into the experience pool; The interactive data includes: the car's action 'a', the current state 's' of the environment, and the next state 's' of the environment. t+1 And the reward r given by the environment.

8. The obstacle avoidance method for a car based on deep reinforcement learning according to claim 7, characterized in that, The methods for optimizing model parameters include the following steps: A small batch of interaction data is randomly sampled from the experience pool; The target Q-value is calculated using the target network, where: Where, θ - Here, γ represents the target network parameters, and γ is the discount factor. Calculate the mean squared error loss between the current Q value and the target Q value, and update the main network parameters θ through backpropagation; Periodically copy the master network parameters θ to the target network parameters θ - ; A greedy strategy is adopted to balance exploration and exploitation, and the exploration rate ϵ is gradually reduced.

9. The obstacle avoidance method for a car based on deep reinforcement learning according to claim 7, characterized in that, The experience pool uses a FIFO mechanism to manage data. When the experience pool is full, the earliest data stored is removed and the latest interaction data is stored.

10. The obstacle avoidance method for a car based on deep reinforcement learning according to claim 1, characterized in that, The action selection probability value includes steering, acceleration, and deceleration commands, and the vehicle performs obstacle avoidance driving based on the action with the highest probability.

Images