Unmanned vehicle dynamic obstacle avoidance method and system based on near-end strategy optimization

Abstract

The invention discloses an unmanned vehicle dynamic obstacle avoidance method and system based on near-end strategy optimization, and the method comprises the steps: firstly constructing a differential drive unmanned vehicle dynamic model, and defining an action space; then designing a composite observation space, and constructing a mixed reward function; and then a control strategy is trained based on a near-end strategy optimization PPO algorithm in combination with a course learning mechanism. According to the designed composite observation space, multi-modal environment representation is provided, the time sequence motion trend of the dynamic obstacle is captured by adopting the frame stacking technology, and the decision continuity is improved. The dynamic obstacle avoidance performance of the unmanned vehicle is improved through staged course learning and a mixed reward mechanism, so that the unmanned vehicle can realize dynamic obstacle avoidance.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent driving technology, and specifically relates to a dynamic obstacle avoidance method and system for unmanned vehicles based on proximal policy optimization (PPO), which is suitable for real-time decision-making and strategy optimization in complex scenarios for intelligent bodies such as unmanned vehicles and mobile robots. Background Art

[0002] Autonomous driving technology, a disruptive innovation merging artificial intelligence and the transportation industry, is reshaping the core of future mobility. Its significance lies not only in freeing humans from the burden of driving but also in building a next-generation transportation system characterized by zero accidents, high efficiency, and low energy consumption through precise decision-making and coordinated control enabled by machine intelligence. According to statistics, global economic losses from traffic accidents reach $1.8 trillion annually. Autonomous vehicles, through communication and coordinated planning, can optimize road resource utilization and alleviate urban congestion. Therefore, the maturity and widespread adoption of autonomous driving technology will profoundly impact sustainable social development from the perspectives of safety, economy, and environment.

[0003] Dynamic obstacle avoidance, a core capability of autonomous driving systems, directly determines the vehicle's survivability and reliability in complex traffic scenarios. Unlike static path planning, dynamic obstacle avoidance requires real-time perception of the surrounding dynamic environment, generating an avoidance strategy that balances safety, comfort, and traffic regulations within a millisecond time window. This process, along with the fusion of high-dimensional sensor data, game-based decision-making in uncertain environments, and the precise execution of control commands, is far more technically challenging than cruise control on structured roads. For example, when an autonomous vehicle encounters a pedestrian suddenly crossing in rainy or snowy weather, the algorithm must simultaneously account for multiple interfering factors, including low-visibility perception errors, changes in road friction coefficient, and the pedestrian's ambiguous movement intentions. Any decision delay or path deviation could lead to catastrophic consequences.

[0004] Traditional obstacle avoidance algorithms, such as artificial potential field and dynamic window methods, while stable in structured scenarios, exhibit limitations in dynamic and complex environments. For example, APF often causes autonomous vehicles to become trapped within U-shaped obstacles due to local minima, while DWA suffers from a significant decrease in computational efficiency in densely populated obstacle scenarios, making it difficult to meet real-time requirements. Furthermore, these algorithms rely heavily on static environment assumptions, making them difficult to handle with high-speed dynamic obstacles. Furthermore, they require manual tuning to optimize potential field coefficients or evaluation function weights, limiting their generalization capabilities.

[0005] Although dynamic obstacle avoidance algorithms based on deep reinforcement learning (DRL) have shown potential in complex environments, they still have the following major shortcomings: First, the generalization ability of dynamic environments is limited. The algorithm usually performs well in a specific training environment, but it is difficult to generalize to unseen dynamic scenarios; second, the reward function design is difficult. Dynamic obstacle avoidance requires a balance between immediate obstacle avoidance and long-term path planning, but manually designed reward functions are prone to introduce biases, causing the strategy to fall into local optimality; third: sample utilization efficiency is low. DRL usually requires a large amount of interaction data to learn effective strategies. In real scenarios, collecting this data can be costly and time-consuming.

[0006] In summary, to address the adaptability bottleneck of traditional methods in dynamic and complex environments, this paper proposes a dynamic obstacle avoidance method for unmanned vehicles based on proximal policy optimization (PPO). Summary of the Invention

[0007] In response to the shortcomings of the existing technology, the present invention proposes a dynamic obstacle avoidance method and system for unmanned vehicles based on proximal policy optimization (PPO). The dynamic obstacle avoidance performance of the unmanned vehicle is improved through phased curriculum learning and a hybrid reward mechanism, enabling the unmanned vehicle to achieve dynamic obstacle avoidance.

[0008] In order to solve the above technical problems, the technical solution of the present invention is:

[0009] In a first aspect, an embodiment of the present application provides a method for dynamic obstacle avoidance of an unmanned vehicle based on proximal strategy optimization, comprising the following steps:

[0010] First, a dynamic model of the differential drive unmanned vehicle is constructed and the action space is defined.

[0011] Then, a composite observation space is designed to obtain lidar ranging data, calculate the relative position information of the target point and the distance and azimuth of the nearest obstacle, and retain the temporal state characteristics through frame stacking technology.

[0012] Construct a hybrid reward function that integrates sparse event and dense behavior rewards to balance obstacle avoidance safety and path efficiency.

[0013] The control strategy is trained based on the proximal policy optimization (PPO) algorithm combined with a curriculum learning mechanism. The curriculum learning mechanism divides the training environment into multiple stages of gradually increasing difficulty, and performs control strategy training in stages.

[0014] After training is completed, the PPO policy network (Actor network) with the best average reward is derived, and the optimal policy is applied to the unmanned vehicle to generate dynamic obstacle avoidance control instructions, realizing real-time obstacle avoidance control in a dynamic obstacle environment.

[0015] In one possible implementation, the hybrid reward function is obtained by fusing a sparse event reward with a dense behavior reward, wherein the sparse event reward includes a collision penalty and a goal achievement reward; and the dense behavior reward includes an obstacle approach penalty, a time penalty, a forward reward, and a motion smoothness penalty.

[0016] In one possible implementation, in the initial stage of the course learning mechanism, there are no obstacles in the training environment, and only basic obstacle avoidance and target tracking capabilities are trained; then static and dynamic obstacles are gradually introduced, and the average reward of the current course learning stage (that is, the statistical mean of the cumulative reward function of a single round during interaction with the environment) is judged to meet the current course stage threshold. If so, the environment complexity is increased to the next stage, otherwise the current stage environment settings are maintained.

[0017] In one possible implementation, the training environment definition includes:

[0018] Establish a differential motion model for the car. Add static and dynamic obstacles according to the requirements of different stages. Determine the car's initial position, target position, and initial position of each obstacle. Each dynamic obstacle has a random initial speed and boundary rebound logic. Add a reinforcement learning interface to support environment reset, action execution, and reward feedback functions.

[0019] In one possible implementation, the curriculum learning mechanism is divided into the following three training environment stages:

[0020] Phase 1: Obstacle-free environment, training basic target tracking capabilities;

[0021] Phase 2: Introducing random static obstacles to train static obstacle avoidance capabilities;

[0022] Phase 3: Combine static and dynamic obstacles to simulate complex scenarios through the random initial velocity and movement direction of dynamic obstacles.

[0023] In one possible implementation, in the training environment at each stage, the current strategy generates actions, updates the posture of the unmanned vehicle, detects collisions and goal achievement states, and stores trajectory data.

[0024] Then, the stored trajectory data is randomly sampled to train the Actor-Critic network, the target value function is calculated through the generalized advantage estimation (GAE), and the weight parameters of the Actor network and Critic are updated using the clipping objective function.

[0025] Finally, the strategy is evaluated and the average reward of the current stage is calculated.

[0026] In a second aspect, an embodiment of the present application provides a dynamic obstacle avoidance system for an unmanned vehicle based on proximal strategy optimization, comprising the following modules:

[0027] Dynamics model construction module: used to build the dynamics model of the differential drive unmanned vehicle and define the action space.

[0028] Composite observation space design module: used to design a composite observation space, obtain lidar ranging data, calculate the relative position information of the target point and the distance and azimuth of the nearest obstacle, and preserve the temporal state characteristics through frame stacking technology.

[0029] Hybrid reward function construction module: used to construct a hybrid reward function, integrating sparse event and dense behavior rewards, and balancing obstacle avoidance safety and path efficiency.

[0030] Control strategy training module: The control strategy is trained based on the proximal policy optimization (PPO) algorithm combined with a curriculum learning mechanism. The curriculum learning mechanism divides the training environment into multiple stages of gradually increasing difficulty, and conducts control strategy training in stages.

[0031] Inference module: After the control strategy training module is completed, the PPO strategy network (Actor network) with the optimal average reward is derived. The optimal strategy is applied to the unmanned vehicle, and dynamic obstacle avoidance control instructions are generated to achieve real-time obstacle avoidance control in a dynamic obstacle environment.

[0032] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory;

[0033] The memory is used to store computer programs;

[0034] The processor is used to implement any unmanned vehicle dynamic obstacle avoidance method described in this application when executing the program stored in the memory.

[0035] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it implements any unmanned vehicle dynamic obstacle avoidance method described in the present application.

[0036] In a fifth aspect, an embodiment of the present application provides a computer program product containing instructions, which, when run on a computer, enables the computer to execute any unmanned vehicle dynamic obstacle avoidance method described in the present application.

[0037] The present invention has the following characteristics and beneficial effects:

[0038] 1. The composite observation space designed by this invention integrates lidar ranging, information about the nearest obstacle, and the relative position of the target, providing a multimodal environmental representation. By adopting frame stacking technology, it captures the temporal motion trends of dynamic obstacles and improves decision consistency.

[0039] 2. This invention designs a state representation and reward mechanism suitable for obstacle avoidance tasks. It constructs a hybrid reward function that combines sparse event rewards (such as reaching the target and collision penalties) with dense behavior rewards (such as safe distance, time consumption, and motion smoothness), effectively balancing the safety and path efficiency of the obstacle avoidance strategy.

[0040] 3. Implement a PPO-based dynamic obstacle avoidance strategy training process. Based on an actor-critic architecture network model, this system combines frame stacking, advantage estimation, and strategy clipping to improve training stability and strategy expression capabilities. It also introduces a curriculum learning mechanism, setting environmental complexity in stages (no obstacles, static obstacles, and a combination of static and dynamic obstacles) to enhance the model's generalization capabilities in complex scenarios.

[0041] 4. By training and testing the proposed model in a custom simulation environment, analyzing the average reward curve during training, and testing the obstacle avoidance success rate in a random environment, it is verified that the PPO algorithm adopted in this invention has a higher obstacle avoidance success rate and fewer steps than the DDPG algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 This is an example of the vehicle observation space provided by the present invention.

[0044] Figure 2 It is a customized dynamic obstacle avoidance environment rendering interface provided by the present invention.

[0045] Figure 3 It is a schematic diagram of the collision detection coordinate transformation of the present invention.

[0046] Figure 4 This is the Actor-Critic network structure provided by the present invention.

[0047] Figure 5 This is a hyperparameter setting diagram based on PPO training provided by the present invention.

[0048] Figure 6 and Figure 7 It is the specific average reward and average number of steps changes in the training process of the PPO algorithm model and DDPG algorithm model provided by the present invention.

[0049] Figure 8This is a successful obstacle avoidance driving process of the unmanned vehicle dynamic obstacle avoidance method based on proximal policy optimization (PPO) provided by the present invention. DETAILED DESCRIPTION

[0050] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0051] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0052] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0053] The present invention provides a dynamic obstacle avoidance method for an unmanned vehicle based on proximal policy optimization (PPO), which is implemented according to the following steps.

[0054] Step 1: Build a dynamic model of a differential drive unmanned vehicle and define the linear speed and angular velocity action space.

[0055] Step 2: Design a composite observation space, obtain lidar ranging data, calculate the relative position information of the target point and the distance and azimuth of the nearest obstacle, and retain the temporal state characteristics through frame stacking technology.

[0056] Step 3: Construct a hybrid reward function that integrates sparse event and dense behavior rewards to balance obstacle avoidance safety and path efficiency.

[0057] Step 4: Based on the proximal strategy optimization PPO algorithm combined with the curriculum learning mechanism to train the control strategy. The curriculum learning mechanism divides the training environment into multiple stages with gradually increasing difficulty. In the initial stage, there are no obstacles in the environment, and only basic obstacle avoidance and target tracking capabilities are trained; then static obstacles and dynamic obstacles are gradually introduced, and the average reward of the current course learning stage is judged. (i.e., the cumulative mixed reward function in a single round during the interaction with the environment Whether the statistical mean of the value of (In one embodiment, a course stage threshold is set ), if it is satisfied, the environment complexity is increased to the next stage, otherwise the current stage environment settings are maintained.

[0058] Step 5: In the current training environment, execute the current strategy generation action ( , ), update the unmanned vehicle’s position and detect collision and goal achievement status, and store trajectory data.

[0059] Step 6: Randomly sample the stored trajectory data to train the Actor-Critic network, calculate the target value function through the generalized advantage estimation (GAE), and use the clipping objective function to update the weight parameters of the Actor network and Critic.

[0060] Step 7: Evaluate the strategy and calculate the average reward for the current stage (defined as the statistical mean of the cumulative rewards in a single round during training) Return to step 4.

[0061] Step 8. After training is completed (the average reward curve converges), derive the PPO policy network (Actor network) with the optimal average reward. Apply the optimal policy to the unmanned vehicle, generate dynamic obstacle avoidance control instructions, and achieve real-time obstacle avoidance control in a dynamic obstacle environment.

[0062] In one possible implementation, step 1 is specifically implemented as follows:

[0063] Carry out dynamic modeling of the vehicle, and model the basic information of the vehicle and the physical quantities of the vehicle.

[0064] 1-1: Establish the mapping relationship between the linear velocity and angular velocity of the left and right wheels. The expression is as follows:

[0065] (1)

[0066] in, and are the linear velocities of the right and left wheels respectively, is the wheel spacing;

[0067] 1-2: Establish the operation model of the car:

[0068] (2)

[0069] in, Indicates the real-time position (position and orientation) of the car, is the linear velocity of the car, represents the angular velocity of the car;

[0070] 1-3: Define the action space of the car:

[0071] (3)

[0072] in, and It is the upper limit of the linear velocity and angular velocity that the car can reach due to power constraints.

[0073] In one possible implementation, a composite observation space (e.g. Figure 1 ), and retain the timing state characteristics through frame stacking technology, as follows:

[0074] 2-1: Calculate the ranging data vectors r0, r1, ... r in N directions evenly distributed around the laser radar ranging simulation module N-1 ; The specific calculation definition is as follows:

[0075] Through the laser radar ranging simulation module, the length of each ranging direction is emitted Calculate the distance between the intersection of the ray and the obstacle in each direction and solve the equation as follows:

[0076] (4)

[0077] The obstacle is modeled as a circle with the center of the obstacle being , the radius is , the starting point of the laser beam is , the direction vector is By repeating the calculation, the distance measurement values in all distance measurement directions are calculated. In actual application, the distance measurement data can be directly obtained by using the unmanned vehicle-mounted laser radar.

[0078] 2-2: Based on the acquired ranging data, get the minimum distance to the nearest obstacle and the cosine and sine of its relative azimuth , ;

[0079] 2-3: Based on the position of the unmanned vehicle and the target point, calculate the relative distance d of the target point and the cosine and sine values of the angle between the target point and the positive direction of the unmanned vehicle. , ;

[0080] 2-4: Based on the above observation data, the car's composite observation space is defined as follows, with a total of N+6 dimensions:

[0081] (5)

[0082] 2-5: Use frame stacking technology to stitch four consecutive frames of observation data into -dimensional input vector.

[0083] In one possible implementation, constructing a hybrid reward function includes:

[0084] 3-1: Constructing sparse event rewards:

[0085] 3-1-1: is the collision penalty, expressed as:

[0086] (6)

[0087] in, is a constant term for the collision penalty (e.g., in one embodiment );

[0088] 3-1-2: is the goal achievement reward, expressed as:

[0089] (7)

[0090] in, is a constant term for the goal achievement reward (e.g., in one embodiment );

[0091] 3-2: Constructing dense behavior rewards:

[0092] 3-2-1: is the obstacle approach penalty, expressed as:

[0093] (8)

[0094] in, is the safety distance threshold, is the minimum distance between the car and the obstacle, Reward and penalty coefficient for dense behavior;

[0095] 3-2-2: is the time penalty, and the expression is:

[0096] (9)

[0097] in, is the time penalty constant term;

[0098] 3-2-3: is the forward reward, the expression is:

[0099] (10)

[0100] in, and are the distances between the car and the target point at the current moment and the previous moment respectively, is the maximum distance, is the forward reward coefficient;

[0101] 3-2-4: is the motion smoothness penalty, expressed as:

[0102] (11)

[0103] in, are the linear velocity at the current moment, the linear velocity at the previous moment, and the angular velocity at the current moment, respectively. is the maximum linear speed, is the maximum angular velocity, is the linear velocity smoothness penalty coefficient, is the angular velocity smoothness penalty coefficient.

[0104] 3-3: The specific composition of the hybrid reward function is:

[0105] (12)

[0106] In one possible implementation, the course learning mechanism is mainly divided into the following three training environment stages:

[0107] 4-1-1: Phase 1: Obstacle-free environment, training basic target tracking ability;

[0108] 4-1-2: Phase 2: Introducing random static obstacles to train static obstacle avoidance capabilities;

[0109] 4-1-3: Phase 3: Combine static and dynamic obstacles to simulate complex scenarios through the random initial velocity and movement direction of dynamic obstacles.

[0110] In one possible implementation, define the two-dimensional environment required for the unmanned vehicle dynamic obstacle avoidance training (e.g. Figure 2 ),include:

[0111] 4-2-1: Establish the differential motion model of the car (obtained by discretizing the car's motion model), that is, the iterative formula of the car's position and posture is as follows:

[0112] (13)

[0113] in, , , , are the position, orientation, linear velocity, and angular velocity of the car at step t, respectively. is the time difference between the previous and next steps;

[0114] 4-2-2: According to the requirements of the three stages in 4-1 above, add static and dynamic obstacles, determine the initial position of the car, the target position, and the initial position of each obstacle. Each dynamic obstacle has a random initial speed and boundary rebound logic;

[0115] 4-2-3: Add reinforcement learning interface to support environment reset, action execution and reward feedback functions.

[0116] In one possible implementation, a specific method for detecting a collision with an unmanned vehicle includes:

[0117] 5-1-1.: Model the car as a rectangle and the obstacle as a circle. The width and height of the rectangle are W and H respectively. The center coordinates are (x, y). The angle between the long axis and the positive direction of the x-axis is , the center of the circle is , with a radius of R;

[0118] 5-1-2: Align the rectangle to the coordinate axis through coordinate transformation (such as Figure 3 ), where the center coordinates The transformation formula is:

[0119] (14)

[0120] Among them, after the coordinate transformation, the rectangular area is:

[0121] (15)

[0122] 5-1-3: Find the closest projection point from the center of the circle to the rectangle:

[0123] (16)

[0124] 5-1-4: Calculate the Euclidean distance between the projection point and the center of the circle. Based on the relationship between this distance and the radius R, if the distance is less than the radius R, a collision is determined to have occurred; otherwise, no collision has occurred.

[0125] In one possible implementation, training an Actor-Critic network based on a proximal policy optimization (PPO) algorithm includes:

[0126] 6-1: Actor-Critic network architecture (such as Figure 4 ):

[0127] 6-1-1: Input layer: Receives composite observation space data, including time-series stacked lidar ranging data, target relative position and nearest obstacle information, with dimensions of (N is the number of radar directions);

[0128] 6-1-2: Hidden layer: A double hidden layer fully connected neural network is used, with 256 neurons in each layer and ReLU activation function;

[0129] 6-1-3: Output layer: Actor network outputs continuous action space , linear velocity and angular velocity respectively. Critic network output state value estimation .

[0130] 6-2: Strategy optimization process:

[0131] 6-2-1: Sampling trajectory data using the old strategy ;

[0132] 6-2-2: Obtaining the state value estimate output by the Critic network Calculate the advantage function estimate. The calculation formula of the generalized advantage function estimate (GAE) is:

[0133] (17)

[0134] Among them, TD error , is the attenuation factor;

[0135] 6-2-3: Multiple rounds of gradient updates (multiple iterative updates on the same batch of empirical data, the number of rounds is usually 3-10) Actor network and Critic network, the clipping objective function is defined as follows:

[0136] (18)

[0137] in, is the strategy ratio, is the advantage function estimate, is the clipping threshold;

[0138] 6-2-4: Regularly synchronize old policies.

[0139] 6-3: The specific hyperparameter settings of the PPO algorithm in the embodiment are as follows Figure 5 shown.

[0140] Experimental data:

[0141] Using the DDPG algorithm as the benchmark algorithm, a model was trained in the same simulation environment provided by the embodiment of the present invention and compared with the PPO algorithm used in the present invention. The specific average reward and average number of steps in the training process of the PPO algorithm model and the DDPG algorithm model are shown in the following figure: Figure 6 and Figure 7 As shown in FIG, the results show that the PPO algorithm used in the present invention has higher training stability and faster convergence speed.

[0142] The trained PPO algorithm model and DDPG algorithm model were tested in 100 random environments, with 10 static and 10 dynamic obstacles respectively. The test results are shown in Table 1:

[0143] Table 1 Algorithm performance comparison

[0144] algorithm Obstacle avoidance success rate (%) Average number of steps PPO 97 284.7 DDPG 94 305.6

[0145] The test results are shown in Table 1. The results show that the PPO algorithm used in the proposed unmanned vehicle dynamic obstacle avoidance method based on the proximal optimization strategy (PPO) has a higher obstacle avoidance success rate and fewer steps than the baseline algorithm. At the same time, the driving process of the car successfully avoiding an obstacle under the PPO algorithm is as follows: Figure 8 As shown in the figure, it can be observed that the car will choose to detour when an obstacle approaches, with obvious active obstacle avoidance behavior, and can eventually reach the destination, realizing real-time obstacle avoidance control in a dynamic obstacle environment.

[0146] The present application embodiment includes a dynamic obstacle avoidance system for an unmanned vehicle based on proximal strategy optimization, including the following modules:

[0147] Dynamics model building module: used to build the dynamics model of the differential drive unmanned vehicle and define the action space.

[0148] Composite observation space design module: used to design a composite observation space, obtain lidar ranging data, calculate the relative position information of the target point and the distance and azimuth of the nearest obstacle, and preserve the temporal state characteristics through frame stacking technology.

[0149] Hybrid reward function construction module: used to construct a hybrid reward function, integrating sparse event and dense behavior rewards, and balancing obstacle avoidance safety and path efficiency.

[0150] The hybrid reward function construction module constructs a hybrid reward function by fusing sparse event rewards with dense behavior rewards. The sparse event rewards include collision penalties and goal achievement rewards; the dense behavior rewards include obstacle approach penalties, time penalties, forward rewards, and motion smoothness penalties.

[0151] Control strategy training module: The control strategy is trained based on the proximal policy optimization (PPO) algorithm combined with a curriculum learning mechanism. The curriculum learning mechanism divides the training environment into multiple stages of gradually increasing difficulty, and conducts control strategy training in stages.

[0152] In the initial stage of the control strategy training module's course learning mechanism, there are no obstacles in the training environment, and only basic obstacle avoidance and target tracking capabilities are trained; then static and dynamic obstacles are gradually introduced, and the average reward of the current course learning stage (that is, the statistical mean of the cumulative reward function of a single round during the interaction with the environment) is judged to see whether it meets the current course stage threshold. If so, the environment complexity is increased to the next stage, otherwise the current stage environment settings are maintained.

[0153] The training environment definition of the control strategy training module includes:

[0154] Establish a differential motion model for the car. Add static and dynamic obstacles according to the requirements of different stages. Determine the car's initial position, target position, and initial position of each obstacle. Each dynamic obstacle has a random initial speed and boundary rebound logic. Add a reinforcement learning interface to support environment reset, action execution, and reward feedback functions.

[0155] In one possible implementation, the control strategy training module course learning mechanism is divided into the following three training environment stages:

[0156] Phase 1: Obstacle-free environment, training basic target tracking capabilities;

[0157] Phase 2: Introducing random static obstacles to train static obstacle avoidance capabilities;

[0158] Phase 3: Combine static and dynamic obstacles to simulate complex scenarios through the random initial velocity and movement direction of dynamic obstacles.

[0159] In one possible implementation, the control strategy training module executes the current strategy generation action ( , ), update the unmanned vehicle’s position and detect collision and goal achievement status, and store trajectory data.

[0160] Then, the stored trajectory data is randomly sampled to train the Actor-Critic network, the target value function is calculated through the generalized advantage estimation (GAE), and the weight parameters of the Actor network and Critic are updated using the clipping objective function.

[0161] Finally, the strategy is evaluated and the average reward of the current stage is calculated.

[0162] Inference module: After the control strategy training module is completed, the PPO strategy network (Actor network) with the optimal average reward is derived. The optimal strategy is applied to the unmanned vehicle, and dynamic obstacle avoidance control instructions are generated to achieve real-time obstacle avoidance control in a dynamic obstacle environment.

[0163] The embodiment of the present application further provides an electronic device, comprising a processor and a memory;

[0164] The memory is used to store computer programs;

[0165] The processor is configured to implement any of the methods described in this application when executing a program stored in the memory.

[0166] In a possible implementation, the electronic device of the embodiment of the present application further includes a communication interface and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus.

[0167] The communication bus mentioned in the electronic device mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0168] The communication interface is used for communication between the above electronic device and other devices.

[0169] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0170] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0171] In another embodiment provided in the present application, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the methods described in the present application is implemented.

[0172] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the methods described in the present application.

[0173] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0174] Each embodiment in this specification is described in a related manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referenced to each other.

[0175] The above description is only a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application are included in the scope of protection of the present application.

Claims

1. A dynamic obstacle avoidance method for unmanned vehicles based on proximal strategy optimization, characterized in that: The steps include: First, a dynamic model of a differential drive unmanned vehicle is constructed to define the action space; Then, a composite observation space is designed to obtain LiDAR ranging data, calculate the relative position information of the target point and the distance and azimuth of the nearest obstacle, and preserve the temporal state characteristics through frame stacking technology; Construct a hybrid reward function that combines sparse event and dense behavior rewards to balance obstacle avoidance safety and path efficiency; The control strategy is trained based on the proximal policy optimization (PPO) algorithm combined with a curriculum learning mechanism. The curriculum learning mechanism divides the training environment into multiple stages with increasing difficulty, and the control strategy training is carried out in stages. After training is completed, the PPO strategy network with the best average reward is derived, and the optimal strategy is applied to the unmanned vehicle to generate dynamic obstacle avoidance control instructions and realize real-time obstacle avoidance control in a dynamic obstacle environment.

2. The unmanned vehicle dynamic obstacle avoidance method based on proximal strategy optimization according to claim 1 is characterized in that: The hybrid reward function is obtained by fusing sparse event rewards with dense behavior rewards, where the sparse event rewards include collision penalties and goal achievement rewards; the dense behavior rewards include obstacle approach penalties, time penalties, forward rewards, and motion smoothness penalties.

3. The unmanned vehicle dynamic obstacle avoidance method based on proximal strategy optimization according to claim 1 is characterized in that: In the initial stage of the course learning mechanism, there are no obstacles in the training environment, and only basic obstacle avoidance and target tracking capabilities are trained; then static and dynamic obstacles are gradually introduced, and the average reward of the current course learning stage is judged to see whether it meets the threshold of the current course stage. If so, the environment complexity is increased to the next stage, otherwise the current stage environment settings are maintained.

4. The unmanned vehicle dynamic obstacle avoidance method based on proximal strategy optimization according to claim 3 is characterized in that: Training environment definition, including: Establish a differential motion model for the car. Add static and dynamic obstacles according to the requirements of different stages. Determine the car's initial position, target position, and initial position of each obstacle. Each dynamic obstacle has a random initial speed and boundary rebound logic. Add a reinforcement learning interface to support environment reset, action execution, and reward feedback functions.

5. The unmanned vehicle dynamic obstacle avoidance method based on proximal strategy optimization according to claim 3 or 4, characterized in that: The course learning mechanism is divided into the following three training environment stages: Phase 1: Obstacle-free environment, training basic target tracking capabilities; Phase 2: Introducing random static obstacles to train static obstacle avoidance capabilities; Phase 3: Combine static and dynamic obstacles to simulate complex scenarios through the random initial velocity and movement direction of dynamic obstacles.

6. The unmanned vehicle dynamic obstacle avoidance method based on proximal strategy optimization according to claim 3 is characterized in that: In the training environment at each stage, the current strategy generates actions, updates the autonomous vehicle's position, detects collisions and goal achievement, and stores trajectory data. Then, the stored trajectory data is randomly sampled to train the Actor-Critic network, the target value function is calculated through generalized advantage estimation, and the weight parameters of the Actor network and Critic are updated using the clipping objective function; Finally, the strategy is evaluated and the average reward of the current stage is calculated.

7. A dynamic obstacle avoidance system for unmanned vehicles based on proximal strategy optimization, characterized in that: Includes the following modules: Dynamics model building module: used to build the dynamics model of the differential drive unmanned vehicle and define the action space; Composite observation space design module: used to design composite observation spaces, obtain lidar ranging data, calculate the relative position information of target points and the distance and azimuth of the nearest obstacle, and preserve the temporal state characteristics through frame stacking technology; Hybrid reward function construction module: used to construct a hybrid reward function, integrating sparse event and dense behavior rewards, and balancing obstacle avoidance safety and path efficiency; Control strategy training module: This module trains control strategies based on the Proximal Policy Optimization (PPO) algorithm combined with a curriculum learning mechanism. The curriculum learning mechanism divides the training environment into multiple stages of increasing difficulty, and conducts control strategy training in stages. Inference module: After the control strategy training module is completed, the PPO strategy network with the best average reward is derived, and the optimal strategy is applied to the unmanned vehicle to generate dynamic obstacle avoidance control instructions to achieve real-time obstacle avoidance control in a dynamic obstacle environment.

8. An electronic device, characterized in that: including processor and memory; The memory is used to store computer programs; The processor is used to implement the unmanned vehicle dynamic obstacle avoidance method as described in any one of claims 1 to 6 when executing the program stored in the memory.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the unmanned vehicle dynamic obstacle avoidance method as described in any one of claims 1 to 6 is implemented.

Images