A vehicle driving control method, device, electronic device and storage medium

Abstract

This disclosure provides a vehicle driving control method, device, electronic device, and storage medium. The method includes: acquiring a set of boundary points of the current drivable area and the current vehicle state during actual vehicle driving; performing global pooling on the set of boundary points of the current drivable area to obtain global features of the current drivable area; concatenating the current vehicle state and the obtained global features to obtain a current fused feature vector; inputting the current fused feature vector into a reinforcement learning agent to obtain the acceleration output by the reinforcement learning agent; and controlling the vehicle driving according to the output acceleration. Accordingly, this disclosure enables adaptive control of vehicle driving based on a reinforcement learning agent, allowing the vehicle to adjust its driving speed according to the spatial features of the drivable area and its own state, improving the adaptability and stability of vehicle driving control in unstructured road scenarios, and enhancing user comfort.

Description

Technical Field

[0001] This disclosure relates to the field of driver assistance technology, and more particularly to a vehicle driving control method, device, electronic device, and storage medium. Background Technology

[0002] With advancements in driver assistance technology, vehicles have become increasingly adept at driving on structured roads. However, they still face significant challenges in unstructured scenarios such as rural roads, industrial parks, and parking lots. These scenarios are characterized by narrow roads, a lack of clear road markings, irregular boundaries, and dynamic, ever-changing environments, placing extremely high demands on vehicles' deceleration and traffic-passing capabilities.

[0003] Existing vehicle deceleration strategies mostly employ no deceleration, fixed threshold deceleration, or simple graded deceleration modes. These methods have several drawbacks: First, they lack adaptability; the system struggles to dynamically adjust vehicle speed based on specific road conditions and obstacle distribution details, easily leading to low traffic efficiency or safety hazards. Second, the control process lacks human-like simulation; speed changes often exhibit abrupt, step-like changes, failing to adjust smoothly and continuously like a human driver, severely impacting passenger comfort. Third, they have weak generalization capabilities; the solutions heavily rely on manual parameter calibration, requiring recalibration for different vehicle models or new scenarios, resulting in high development costs. Summary of the Invention

[0004] In view of the above, in a first aspect, this disclosure provides a vehicle driving control method, the method comprising: During actual vehicle operation, acquire the set of boundary points of the current drivable area and the current vehicle status; Global pooling is performed on the set of boundary points of the current drivable area of ​​the vehicle to obtain the global features of the current drivable area of ​​the vehicle. The current vehicle state and the obtained global features are concatenated to obtain the current fused feature vector; The current fused feature vector is input into the reinforcement learning agent to obtain the acceleration output by the reinforcement learning agent; The vehicle's movement is controlled based on the output acceleration.

[0005] Optionally, obtaining the set of boundary points of the current drivable area of ​​the vehicle includes: Obtain environmental data of the road where the vehicle is currently located; The environmental data is processed to obtain the initial set of boundary points of the current drivable area of ​​the vehicle; The initial boundary point set is transformed to the vehicle coordinate system, and the initial boundary point set is resampled to obtain a boundary point set with a preset number of boundary points. The preset number is consistent with the number of boundary points in the set of boundary points of the drivable area used by the reinforcement learning agent during training.

[0006] Optionally, before performing the resampling operation on the initial boundary point set, the method further includes: The initial set of boundary points transformed to the vehicle coordinate system is normalized to obtain the normalized set of boundary points. The normalized set of boundary points is clustered using a spatial clustering algorithm to obtain at least one cluster. Calculate the distance between each boundary point that does not belong to the cluster and the cluster, and remove boundary points whose distance is greater than a preset threshold to form a denoised boundary point set.

[0007] Optionally, the reinforcement learning agent comprises a policy network and a value network, and the reinforcement learning agent is trained in the following manner: In the simulation environment, vehicle status information and a set of boundary points of the vehicle's drivable area are generated, and then a fused feature vector is constructed based on the generated vehicle status information and the set of boundary points of the vehicle's drivable area. The fused feature vector is input into the policy network and the value network to obtain the acceleration output by the policy network and the current state value output by the value network. The output acceleration is executed in the simulation environment to obtain the vehicle's driving state and reward value at the next moment; and the vehicle's driving state at the next moment is input into the value network to obtain the state value at the next moment. The loss is calculated based on the current state value, the reward value, and the next state value, and the network parameters of the policy network and the value network are updated based on the loss until the training of the two networks converges.

[0008] Optionally, the step of calculating the loss based on the current state value, the reward value, and the next time-step state value, and updating the network parameters of the policy network and the value network based on the loss, includes: Calculate the target value based on the reward value and the state value at the next moment; A value network loss function is constructed based on the mean square error between the target value and the current state value, and the parameters of the value network are updated based on the value network loss function. The policy network loss function is constructed based on the output of the value network after the parameter update and the output acceleration, and the parameters of the policy network are updated based on the policy network loss function.

[0009] Optionally, the step of generating vehicle state information and a set of boundary points of the vehicle's drivable area in the simulation environment, and then constructing a fused feature vector based on the generated vehicle state information and the set of boundary points of the vehicle's drivable area, includes: The road environment and vehicle status information for vehicle driving are randomly generated in the simulation environment. Based on the road environment, determine the set of boundary points of the vehicle-accessible area; Global pooling is performed on the boundary point set of the vehicle drivable area to obtain the global features of the vehicle drivable area; wherein, the global features are used to represent the overall geometric shape and spatial distribution features of the vehicle drivable area. The global features and the vehicle state information are concatenated to obtain a fused feature vector.

[0010] Optionally, the step of executing the output acceleration in the simulation environment to obtain the vehicle's driving state and reward value at the next moment includes: The output acceleration is executed in the simulation environment to obtain the vehicle's driving state at the next moment; Based on the state evaluation function, calculate the reward value corresponding to the vehicle's driving state at the next moment; The state evaluation function includes a preset dimension and a preset weight corresponding to the preset dimension. The preset dimension includes at least one of a safety dimension, an efficiency dimension, and a comfort dimension.

[0011] Secondly, this disclosure provides a vehicle driving control device, the device comprising: The acquisition unit is used to acquire the set of boundary points of the current drivable area and the current vehicle status during actual vehicle driving. The pooling unit is used to perform global pooling on the set of boundary points of the current drivable area of ​​the vehicle to obtain the global features of the current drivable area of ​​the vehicle. The splicing unit is used to splice the current vehicle state and the obtained global features to obtain the current fused feature vector; The input unit is used to input the current fused feature vector into the reinforcement learning agent to obtain the acceleration output by the reinforcement learning agent. The control unit is used to control the vehicle's movement based on the output acceleration.

[0012] Thirdly, this disclosure also provides an electronic device, including a communication interface, a processor, a memory, and a bus, wherein the communication interface, the processor, and the memory are interconnected via the bus; The memory stores machine-readable instructions, and the processor executes the above method by invoking the machine-readable instructions.

[0013] Fourthly, this disclosure also provides a machine-readable storage medium storing machine-readable instructions that, when called and executed by a processor, implement the above-described method.

[0014] Therefore, in the technical solution of this disclosure, the vehicle acquires the boundary point set of the drivable area and vehicle state information during actual driving. Global features are obtained by global pooling of the boundary point set of the drivable area, and then concatenated with the vehicle state to form a fused feature vector. This realizes the mapping of the complex geometric information of unstructured roads and the current vehicle state into a representation that can be processed by a reinforcement learning agent. Furthermore, this disclosure inputs the fused feature vector into the reinforcement learning agent to output an acceleration matching the current road conditions. The vehicle's driving is controlled based on this acceleration, enabling the vehicle to adaptively adjust its speed according to the spatial characteristics of the drivable area and the current vehicle state.

[0015] Through the above methods, this disclosure enables adaptive control of vehicle driving based on reinforcement learning agents. This allows the vehicle to adjust its speed according to the spatial characteristics of the drivable area and its own state, improving the adaptability and stability of vehicle driving control in unstructured road scenarios and effectively avoiding passenger discomfort caused by sudden speed changes. Furthermore, this control method can flexibly adapt to various unstructured road scenarios without requiring recalibration of rules and parameters for different vehicle models or new environments, thus reducing system deployment costs. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a vehicle driving control method as an exemplary embodiment; Figure 2 This is a schematic diagram of a real-vehicle inference system architecture, as illustrated in an exemplary embodiment. Figure 3 This is a schematic diagram illustrating an offline training architecture for a reinforcement learning agent, as shown in an exemplary embodiment. Figure 4 This is an exemplary embodiment illustrating the hardware structure of an electronic device; Figure 5 This is a block diagram illustrating a vehicle driving control device as an exemplary embodiment. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solutions in this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this disclosure.

[0019] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this disclosure in other embodiments. In some other embodiments, the methods may include more or fewer steps than those described in this disclosure. Furthermore, a single step described in this disclosure may be broken down into multiple steps in other embodiments; and multiple steps described in this disclosure may be combined into a single step in other embodiments.

[0020] With advancements in driver assistance technology, vehicles have become increasingly adept at driving on structured roads. However, they still face significant challenges in unstructured scenarios such as rural roads, industrial parks, and parking lots. These scenarios are characterized by narrow roads, a lack of clear road markings, irregular boundaries, and dynamic, ever-changing environments, placing extremely high demands on vehicles' deceleration and traffic-passing capabilities.

[0021] Existing vehicle deceleration strategies mostly employ no deceleration, fixed threshold deceleration, or simple graded deceleration modes. These methods have several drawbacks: First, they lack adaptability; the system struggles to dynamically adjust vehicle speed based on specific road conditions and obstacle distribution details, easily leading to low traffic efficiency or safety hazards. Second, the control process lacks human-like simulation; speed changes often exhibit abrupt, step-like changes, failing to adjust smoothly and continuously like a human driver, severely impacting passenger comfort. Third, they have weak generalization capabilities; the solutions heavily rely on manual parameter calibration, requiring recalibration for different vehicle models or new scenarios, resulting in high development costs.

[0022] In view of this, the present disclosure aims to propose a technical solution for smoothly and adaptively adjusting vehicle speed on unstructured roads.

[0023] The technical solution first acquires the set of boundary points of the current drivable area and the current vehicle status during actual vehicle driving; Secondly, global pooling is performed on the set of boundary points of the current drivable area of ​​the vehicle to obtain the global features of the current drivable area of ​​the vehicle. Then, the current vehicle state and the obtained global features are concatenated to obtain the current fused feature vector. Further, the current fused feature vector is input into the reinforcement learning agent to obtain the acceleration output by the reinforcement learning agent. Finally, the vehicle is controlled to drive according to the output acceleration.

[0024] For example, during actual vehicle operation, the assisted driving control system (hereinafter referred to as the "system") acquires data from an onboard LiDAR scanner scanning the surrounding road environment, uses a semantic segmentation network to identify the edges of the road surface and obstacles, and extracts a set of boundary points representing the left and right edges of the road and obstacles currently being driven by the vehicle. Simultaneously, it collects real-time data such as current vehicle speed, vehicle position, orientation angle, and vehicle acceleration through an inertial measurement unit, wheel speed sensors, and a global navigation satellite system, collectively forming the current vehicle state information. Then, the system inputs the acquired set of boundary points representing the drivable areas into a feature extraction network. A multilayer perceptron maps the coordinates of each point into a high-dimensional feature vector. Subsequently, a global max-pooling operation is performed to extract a fixed-length vector representing key information such as the narrowest point width of the passage, the maximum curvature, and the distance to the nearest obstacle from the features of all points. This vector serves as the global feature of the current drivable area. Furthermore, the system directly concatenates the global feature vector and the vehicle state information vector along the feature dimension to form the current fused feature vector. Then, the system inputs this current fused feature vector into a trained and converged reinforcement learning agent, which performs multiple nonlinear transformations and outputs the acceleration at the current moment. Finally, the system receives the acceleration output by the reinforcement learning agent, converts it into specific motor torque and steering angle requests, and drives the vehicle through the drive-by-wire chassis system to perform corresponding acceleration, deceleration, and steering maneuvers, allowing the vehicle to smoothly pass through narrow roads along the center of the drivable area.

[0025] Therefore, in the technical solution of this disclosure, the vehicle acquires the boundary point set of the drivable area and vehicle state information during actual driving. Global features are obtained by global pooling of the boundary point set of the drivable area, and then concatenated with the vehicle state to form a fused feature vector. This realizes the mapping of the complex geometric information of unstructured roads and the current vehicle state into a representation that can be processed by a reinforcement learning agent. Furthermore, this disclosure inputs the fused feature vector into the reinforcement learning agent to output an acceleration matching the current road conditions. The vehicle's driving is controlled based on this acceleration, enabling the vehicle to adaptively adjust its speed according to the spatial characteristics of the drivable area and the current vehicle state.

[0026] Through the above methods, this disclosure enables adaptive control of vehicle driving based on reinforcement learning agents. This allows the vehicle to adjust its speed according to the spatial characteristics of the drivable area and its own state, improving the adaptability and stability of vehicle driving control in unstructured road scenarios and effectively avoiding passenger discomfort caused by sudden speed changes. Furthermore, this control method can flexibly adapt to various unstructured road scenarios without requiring recalibration of rules and parameters for different vehicle models or new environments, thus reducing system deployment costs.

[0027] The present disclosure will now be described through specific embodiments and in conjunction with specific application scenarios.

[0028] Please see Figure 1 , Figure 1 This is a flowchart illustrating an exemplary embodiment of a vehicle driving control method, which may perform the following steps: Step 102: During actual vehicle operation, obtain the set of boundary points of the current drivable area and the current vehicle status.

[0029] For example, during actual vehicle operation, the system acquires the vehicle-mounted LiDAR scan of the road environment around the vehicle, uses a semantic segmentation network to identify the road surface and obstacle edges, and extracts a set of boundary points representing the left and right edges of the road and obstacles currently being driven by the vehicle. At the same time, the system collects the current vehicle speed, vehicle position, orientation angle, vehicle acceleration, etc. in real time through the inertial measurement unit, wheel speed sensor and global navigation satellite system, which together form the current vehicle status information.

[0030] The drivable area boundary point set can flexibly adapt to the irregular geometry of unstructured roads (such as construction sections and narrow rural roads), while vehicle state information reflects the vehicle's current dynamic pose. Together, they constitute the basic data source for reinforcement learning agents to perceive external environmental constraints and vehicle motion states. Semantic segmentation networks are deep learning models used to classify each point in the original point cloud into semantic categories such as road surface, obstacles, or background to extract effective boundaries.

[0031] Step 104: Perform global pooling on the set of boundary points of the current drivable area of ​​the vehicle to obtain the global features of the current drivable area of ​​the vehicle.

[0032] For example, the system inputs a set of boundary points of several drivable areas into the feature extraction network. The coordinates of each point are mapped into a high-dimensional feature vector through a multilayer perceptron. Then, a global max pooling operation is performed to extract a fixed-length vector from the features of all points, representing key information such as the width of the narrowest point of the channel, the maximum curvature, and the distance to the nearest obstacle, as the global feature of the current drivable area of ​​the vehicle.

[0033] Global pooling, with its permutation invariance, effectively eliminates inconsistencies in the dimensionality of the input point set caused by fluctuations in sensor sampling rates or changes in the number of obstacles. This ensures that the output global feature dimension remains constant regardless of the number of road boundary points, thereby improving generalization ability and robustness to various complex road conditions. Besides global max pooling, global average pooling can also be performed; this disclosure does not limit its application.

[0034] Step 106: Concatenate the current vehicle state and the obtained global features to obtain the current fused feature vector.

[0035] For example, the system directly concatenates the global feature vector (including the most extreme geometric information of the road boundary, such as minimum passage width, maximum curvature, etc.) with the vehicle state information vector (including current vehicle speed, vehicle position, orientation angle, vehicle acceleration, etc.) in the feature dimension to form a high-dimensional current fused feature vector.

[0036] Among them, the splicing operation realizes the deep coupling between the geometric constraints of the environment and the dynamic state of the vehicle itself. This enables the reinforcement learning agent to not only perceive the road topology (such as width and curvature) when making decisions, but also to associate the current motion state of the vehicle (such as the distance of the vehicle relative to the boundary) in real time. This integrated "environment-state" feature representation eliminates the semantic gap between environmental perception and decision control in traditional rule-based methods. It provides a complete and high-dimensional state basis for the reinforcement learning agent to output the optimal acceleration command that takes into account safety, traffic efficiency and ride comfort.

[0037] Step 108: Input the current fused feature vector into the reinforcement learning agent to obtain the acceleration output by the reinforcement learning agent.

[0038] For example, the system inputs the current fused feature vector into a reinforcement learning agent trained and converged based on the PPO (Proximal Policy Optimization) algorithm. This reinforcement learning agent performs deep analysis of the current fused feature vector through a multi-layer nonlinear neural network, simulating the thought process of a human driver, and finally outputs the acceleration at the current moment.

[0039] Among them, the reinforcement learning agent is trained by reinforcement learning algorithm in a large-scale simulation environment. It implicitly contains a reward mechanism that takes into account safety, efficiency and comfort. Therefore, it can directly map acceleration strategies that conform to human driving experience based on fused features without explicit rule code.

[0040] Step 110: Control the vehicle to move according to the output acceleration.

[0041] For example, the system receives the acceleration output by the reinforcement learning agent, converts it into specific motor torque requests and steering angle requests, and drives the vehicle to perform corresponding acceleration, deceleration and steering actions through the drive-by-wire chassis system, so that the vehicle can smoothly pass through the narrow road along the center of the drivable area.

[0042] This step completes the closed loop from digital decision-making to physical execution. Through high-frequency cyclic execution (such as once every 100 milliseconds), the vehicle can respond in real time to dynamic changes in road boundaries (such as the sudden appearance of pedestrians or narrowing road conditions), achieving adaptive assisted driving control without human intervention.

[0043] In this embodiment, global features are extracted by performing global pooling on the boundary point set of the current drivable area of ​​the vehicle during actual driving. These features are then concatenated with the current vehicle state information to form a fused feature vector. With the help of this fused feature vector and the collaboration of the converged reinforcement learning agent, the vehicle autonomously outputs acceleration that conforms to human driving habits based on real-time perceived spatial constraints and vehicle state, without preset rules or manual calibration. The vehicle's electronic control system then converts this acceleration into physical acceleration and deceleration actions, ultimately achieving adaptive driving control that balances safety, efficiency, and comfort in unstructured narrow passages.

[0044] In one embodiment shown, obtaining the current vehicle drivable area boundary point set includes: obtaining environmental data of the road where the current vehicle is located; performing perception processing on the environmental data to obtain an initial boundary point set of the current vehicle drivable area; converting the initial boundary point set to the vehicle coordinate system and performing a resampling operation on the initial boundary point set to obtain a preset number of boundary point sets; wherein the preset number is consistent with the number of boundary points in the vehicle drivable area boundary point set used by the reinforcement learning agent during training.

[0045] For example, during actual vehicle operation, the system acquires the road environment around the vehicle from the onboard LiDAR scan, uses a semantic segmentation network to identify the edges of the road surface and obstacles, and extracts a set of boundary points representing the left and right edges of the road and obstacles currently being driven by the vehicle. Subsequently, the system uses an extrinsic calibration matrix to transform these points from the sensor coordinate system to the vehicle coordinate system with the rear axle center as the origin. Then, a resampling algorithm is used to extract a preset number of boundary points at equal intervals along the boundary curve of the drivable area. If the original number of points is insufficient, linear interpolation is used to supplement them; if there are too many, downsampling is used to remove them, finally generating a standard boundary point set with fixed dimensions.

[0046] The coordinate system transformation ensures that the coordinate representation of the boundary point set is always based on the vehicle's own perspective. Resampling involves using interpolation or downsampling algorithms to uniformly adjust the variable number of initial boundary points into a fixed sequence of key points. Resampling addresses the issue of variable boundary point numbers caused by fluctuations in sensor detection distance or changes in road curvature. It also ensures that the input data distribution during the inference phase is strictly consistent with the data distribution during the reinforcement learning training phase, avoiding a decrease in model generalization performance due to differences in input dimensions and ensuring the accuracy and stability of the acceleration output by the reinforcement learning agent.

[0047] In one embodiment shown, before resampling the initial boundary point set, the method further includes: normalizing the initial boundary point set transformed to the vehicle coordinate system to obtain a normalized boundary point set; clustering the normalized boundary point set using a spatial clustering algorithm to obtain at least one cluster; calculating the distance between each boundary point that does not belong to the cluster and the cluster, and removing boundary points whose distance is greater than a preset threshold to form a denoised boundary point set.

[0048] For example, during actual vehicle operation, the system obtains an initial set of 217 boundary points for the drivable area in the vehicle coordinate system. Then, the system performs normalization on the transformed initial boundary point set, scaling each boundary point to a different scale. The system then uses a clustering algorithm to cluster the processed boundary points, identifying three main clusters covering 189 boundary points; the remaining 28 boundary points are marked as non-clustered points. To filter out abnormal boundary points, the system calculates the minimum Euclidean distance from each of these 28 boundary points to its nearest cluster, finding that 22 of them have a distance greater than 0.2 m, and therefore these 22 boundary points are removed; finally, 195 points are retained to form the denoised boundary point set for subsequent resampling and global pooling processing.

[0049] The normalization process involves normalizing each boundary point. Based on the vehicle's center coordinates With maximum sensing distance Perform linear scaling so that its coordinates fall within the interval [-1, 1], expressed as: In the formula, The value is determined based on the actual capabilities of the vehicle, such as 5 m. The purpose of this normalization process is to unify the spatial scale of the input point cloud, improve the robustness of subsequent spatial clustering algorithms in recognizing the distribution density and geometric structure of boundary points, and avoid misjudgment of the main boundary due to the imbalance of local point sets, such as sparseness or density.

[0050] Please see Figure 2 , Figure 2 This is a schematic diagram illustrating a real-vehicle inference system architecture, as shown in an exemplary embodiment. Figure 2As shown, the system first integrates multi-sensor data (such as LiDAR, cameras, millimeter-wave radar, etc.) through the perception fusion module to construct a unified environmental representation of the vehicle's drivable area and dynamic obstacles. Then, it enters a real-time data preprocessing stage, completing operations such as coordinate system transformation, point cloud resampling, and state normalization to ensure that the input features conform to the training distribution of the reinforcement learning agent. Next, the trained reinforcement learning agent outputs acceleration based on the current fused feature vector. This acceleration is then converted into specific throttle, braking, and steering control signals by other post-processing modules and the vehicle execution unit, driving the vehicle's actions. Simultaneously, the system uses a vehicle state feedback loop to transmit the actual vehicle speed, position, and attitude information after execution back to the perception fusion layer, forming a continuously iterative online decision-making-execution-feedback loop, thereby achieving end-to-end, adaptive assisted driving control.

[0051] In one embodiment, the reinforcement learning agent comprises a policy network and a value network. The reinforcement learning agent is trained as follows: Vehicle state information and a set of boundary points of the drivable area are generated in a simulation environment; a fused feature vector is then constructed based on the generated vehicle state information and the set of boundary points of the drivable area; the fused feature vector is input into the policy network and the value network to obtain the acceleration output by the policy network and the current state value output by the value network; the output acceleration is executed in the simulation environment to obtain the vehicle's driving state and reward value at the next moment; the vehicle's driving state at the next moment is input into the value network to obtain the state value at the next moment; a loss is calculated based on the current state value, the reward value, and the state value at the next moment, and the network parameters of the policy network and the value network are updated based on the loss until the training of the two networks converges.

[0052] For example, the system first randomly generates diverse road environments and initial vehicle states in the simulation environment; then, based on the generated road environment, it determines the set of boundary points of the drivable area of ​​the vehicle; next, it performs global pooling on the boundary point set to extract global features that can characterize the overall geometric shape and spatial distribution of the area; finally, it concatenates the extracted global features with the vehicle state information to obtain a fused feature vector.

[0053] Secondly, the system simultaneously inputs the fused feature vectors into a deep neural network structure that shares underlying features. This structure includes a common feature extraction layer (such as a fully connected layer or a convolutional layer), which then branches into two independent output heads: a policy network head and a value network head, where the policy network head outputs an acceleration of -1.85 m / s². 2This indicates a suggestion to decelerate. The value network head outputs a current state value of 12.45, indicating that the current vehicle state is expected to yield a high long-term return. Subsequently, the system accelerates to -1.85 m / s². 2 The simulation environment performs integral calculations based on the current vehicle dynamics model (considering mass, friction, and road adhesion coefficient) to extrapolate to the next time step (after a control step of 0.1 s). During this process, the simulation environment detects that the vehicle successfully avoids the abrupt obstacle boundary on the left without a collision. Therefore, a reward value of 2.5 is calculated according to the preset reward function (including a safe driving reward of +3.0 and a slight deceleration comfort penalty of -0.5). Simultaneously, the simulation environment outputs the vehicle's driving state at the next time step (including both the vehicle state and the set of boundary points of the vehicle's drivable area), including the vehicle's longitudinal speed decreasing to 3.32 km / h, and corresponding updates to other attitude parameters and the set of boundary points of the vehicle's drivable area at the next time step.

[0054] Then, the system constructs a new fusion feature vector based on the vehicle state information (vehicle speed, vehicle position, orientation angle, vehicle acceleration, etc.) and the boundary point set of the vehicle's drivable area at the next moment. The new fusion feature vector is input into the value network. After forward propagation calculation, the value network evaluates the expected long-term return in the new state and outputs the state value of the vehicle at the next moment as 13.10.

[0055] Based on this, the system utilizes the current state value (12.45), the next state value (13.10), and the reward value (2.5) obtained in the previous step to construct an objective function reflecting the prediction error and the quality of the strategy. Specifically, the system uses the reward value (2.5) and the next state value (13.10) to construct a target value estimate, and calculates the deviation between this target value estimate and the current state value (12.45), thus constructing the loss basis for the value network. Simultaneously, based on the deviation between this target value estimate and the current state value (12.45), the system derives an advantage estimate, and combines this with the decision information output by the policy network based on the current state to construct the loss basis for the policy network. The decision information output by the policy network based on the current state manifests as a specific acceleration in deterministic policies and as a probability distribution characteristic of the acceleration in stochastic policies.

[0056] Subsequently, the system employs a gradient optimization algorithm to calculate the gradients of the value network loss and the policy network loss relative to their respective network parameters. The weight parameters of the two networks are then synchronously iteratively updated along the inverse direction of the gradients to reduce value prediction errors and improve the long-term cumulative return expectation of the policy in the simulation environment.

[0057] Finally, the system enters an iterative training loop. At the beginning of each iteration, driving scenarios and vehicle state information with different geometric features are randomly generated again in the simulation environment, resulting in different sets of boundary points for the vehicle's drivable area. Subsequently, the system performs global pooling on the boundary point sets to extract global features, and concatenates them with the vehicle state information to form the current fused feature vector. Next, the system inputs this vector into the value network to obtain the current state value output by the value network; and inputs this vector into the policy network to obtain the acceleration output by the policy network. This acceleration is then executed in the simulation environment and extrapolated to the next time step to obtain the reward value. The system also calculates the state value for the next time step based on the newly formed fused feature vector. Furthermore, the system calculates the temporal difference error between the current state value, the reward value, and the state value for the next time step. Based on the calculated error, the system backpropagates the gradient to synchronously update the weight parameters of the policy network and the value network until the average cumulative reward of the reinforcement learning agent is stable above a preset threshold and the standard deviation of the fluctuation is below a set range over multiple consecutive evaluation periods. At this point, the reinforcement learning agent is considered to have converged, and the parameters of the policy network and the value network at this time are saved for subsequent deployment in a real vehicle.

[0058] The drivable area refers to the spatial region that a vehicle can safely pass through under the current road conditions without a collision. The boundary point set of the drivable area is a two-dimensional discrete point sequence representing the geometric contour of the drivable area, typically established as a coordinate system with the vehicle center as the origin and the vehicle's direction of travel as the x-axis. Vehicle state information is real-time data representing the vehicle's current motion and spatial relationship. In this embodiment, the road environment and vehicle state information are generated synchronously, providing input conditions with a consistent spatiotemporal reference for subsequent boundary point set extraction. Global features are used to represent the overall geometric shape and spatial distribution characteristics of the drivable area, including maximum passage width, average boundary curvature, etc. Spatial distribution features include the nearest point in the most dangerous direction, the maximum value of the boundary abrupt change angle, etc.

[0059] The policy network head uses a feedforward neural network with two hidden layers to map the extracted high-dimensional features into an action space distribution, outputting a continuous scalar value as the expected longitudinal acceleration command. The value network head uses a similar network structure to evaluate the expected value of the cumulative reward in the future for the state represented by the current fused feature vector, outputting a scalar as the current state value. These two outputs will be used for subsequent action execution control and network parameter update calculations based on temporal difference errors, respectively.

[0060] The acceleration output by the execution is input into the vehicle dynamics model in the simulation environment by the acceleration output by the policy network. The vehicle kinematic state is updated by integration, thereby obtaining the driving state at the next moment. The driving state is a multi-dimensional state variable that includes vehicle state information such as vehicle speed, vehicle position, orientation angle, and vehicle acceleration at the next moment, as well as the boundary point set of the vehicle's drivable area at the next moment.

[0061] The reward value can be an instantaneous feedback signal calculated by weighting safety, efficiency, and comfort dimensions. The next-moment state value is the scalar value output by the value network after the vehicle's next-moment driving state is processed through the same procedure to form a new fused feature vector. In this embodiment, the loss calculation process embodies the core idea of ​​temporal difference learning, that is, by summing the reward value generated after the action is executed and the next-moment state value, the evaluation result of the current state value is corrected, thereby providing a guiding signal for the update of network parameters. Since the network parameter update mechanism has universal algorithmic adaptability, by minimizing the predefined loss function, the advantage estimate is used to guide the policy network to converge to the high-reward action space, and drive the value network to approximate the true state value function, thus being compatible with various reinforcement learning algorithms, including but not limited to DDPG (Deep Deterministic Policy Gradient), SAC (Soft Actor Critic), and PPO (Proximal Policy Optimization), ensuring stable policy convergence and value evaluation under different algorithm architectures.

[0062] The core of calculating temporal difference error lies in constructing a deviation signal between "expectation" and "reality": the system subtracts the sum of the reward value obtained "after executing the action" and the value of the next state from the current state value; if the result is greater than 0, it indicates that the currently executed acceleration strategy is better than expected, and the loss function will guide the network to increase the output probability of this strategy in similar states; conversely, if the result is less than 0, it indicates that the currently executed acceleration strategy is worse than expected, and the network parameters will reduce the output probability of this strategy in similar states. Through a closed-loop feedback mechanism based on the difference in state values ​​before and after execution, the policy network gradually learns to output acceleration that balances safety, efficiency, and comfort under complex boundary constraints, while the value network continuously approximates the true state value distribution, thus jointly achieving the convergence and optimization of the driving strategy.

[0063] In one embodiment, the step of calculating the loss based on the current state value, the reward value, and the next-time state value, and updating the network parameters of the policy network and the value network based on the loss, includes: calculating a target value based on the reward value and the next-time state value; constructing a value network loss function based on the mean square error between the target value and the current state value, and updating the parameters of the value network based on the value network loss function; constructing a policy network loss function based on the output of the value network after parameter updates and the output acceleration, and updating the parameters of the policy network based on the policy network loss function.

[0064] For example, in a simulation environment, when a reinforcement learning agent (taking the PPO algorithm as an example) executes the acceleration output by the policy network... After obtaining the mean of the normal distribution of the output acceleration, the system receives a reward value. And observe the vehicle's driving status at the next moment. The system will record the driving status. Input the value network to obtain the value of the next time step. Subsequently, the system uses a discount factor. According to the formula The target value is calculated and will serve as the supervised truth value for constructing the value network loss function. The system then uses the calculated target value... The current state value output by the value network before acceleration execution Substitute into the mean square error loss function The gradient is calculated through backpropagation to update the value network parameters. After the value network update is completed, the system enters the policy network optimization phase: the system takes acceleration... The old policy probability output by the old policy network parameters during execution In subsequent optimization iterations, the latest policy network parameters are used before each iteration. Calculate the probability of the new strategy Dividing the two yields the probability ratio. At the same time, the system combines reward values. Current state value Value of the state at the next moment According to the formula Calculate the advantage estimate. Finally, the system calculates the advantage estimate. With probability ratio Construct the objective function And by updating the policy network parameters through gradient ascent, the objective function is improved. Maximize. After the update, the value network for the same driving state... The state value output again will be closer The policy network applies to the same driving state. The acceleration output again It is closer to the acceleration that will bring the greatest expected return.

[0065] Among them, reward value Indicates "by time" Execution acceleration The reward that is triggered, rather than the reward at any moment. "Received reward signal". State value. Indicates "at time" State "Expected return estimate". Both share the same subscript. This is because they are all anchored to "moment". The decision event is a "decision event" even though the actual timing of its physical occurrence may differ. Furthermore, the policy network objective function introduces a pruning function. Probability ratio Pruning is performed to limit the policy update magnitude in a single iteration, thereby improving the stability of the training process and preventing policy collapse.

[0066] It should be noted that this embodiment is also applicable to other reinforcement learning algorithms. Taking the DDPG algorithm as an example, its mechanism of constructing mean squared error based on temporal difference objective to update the value network is similar to PPO, but the policy network update method is different: DDPG does not rely on probability ratio and pruning, but uses deterministic policy gradient to directly optimize the policy based on the derivative of the value function with respect to acceleration in order to maximize the expected reward.

[0067] Taking the SAC algorithm as an example, its logic of iteratively updating the value network using the target value is similar to that of PPO, but the optimization target and value definition are different: SAC introduces a maximum entropy regularization term in policy update, which aims to maximize the weighted sum of "expected return and policy entropy" to enhance exploration ability; and its value calculation explicitly includes entropy estimation, usually using two independent value networks to work together, which is significantly different from the mechanism of PPO based on a single value network and advantage estimation.

[0068] In some embodiments, vehicle dynamics constraints (maximum deceleration, maximum acceleration, maximum rate of change of acceleration, etc.) need to be embedded during the training process of the reinforcement learning agent to ensure that the acceleration output by the reinforcement learning agent conforms to the actual handling capability of the vehicle.

[0069] In one embodiment shown, the step of generating vehicle state information and a set of boundary points of the vehicle's drivable area in a simulation environment, and then constructing a fused feature vector based on the generated vehicle state information and the set of boundary points of the vehicle's drivable area, includes: randomly generating the road environment and vehicle state information of the vehicle in the simulation environment; determining the set of boundary points of the vehicle's drivable area according to the road environment; performing global pooling on the set of boundary points of the vehicle's drivable area to obtain global features of the vehicle's drivable area; wherein, the global features are used to represent the overall geometric shape and spatial distribution features of the vehicle's drivable area; and concatenating the global features and the vehicle state information to obtain a fused feature vector.

[0070] For example, the system randomly generates a sharp corner passage with a radius of curvature of 1.8 m in the simulation environment, and sets the vehicle to enter at a speed of 6 km / h and a distance of 0.45 m from the left curb. Based on this road environment and vehicle state, the simulation engine outputs a set of boundary points of the vehicle's drivable area containing 97 points. Then, the system performs max pooling on the boundary point set of the vehicle's drivable area to extract the maximum offset in the x-direction (0.82 m), the maximum offset in the y-direction (1.35 m), the minimum distance from the vehicle center (0.38 m), and the maximum local curvature (0.41 m). The extreme values ​​of dimensions such as rad / m constitute an 8-dimensional global feature. This 8-dimensional global feature is then concatenated with the 4-dimensional vehicle state information (vehicle speed, vehicle position, orientation angle, vehicle acceleration, etc.) to obtain a 12-dimensional fused feature vector (where the first 4 dimensions are vehicle state information, including vehicle speed, vehicle position, orientation angle, and vehicle acceleration; the last 8 dimensions are the geometric features of the drivable area boundary after global pooling, including the maximum offset in the x-direction, the maximum offset in the y-direction, the minimum distance from the vehicle center, the maximum local curvature, etc.).

[0071] The randomly generated simulation environment includes one or more scenarios such as straight roads, sharp corners, temporary obstacles blocking the road, and static obstacles blocking the road. The system typically generates different simulation environments based on different road environment geometric parameters, including scene, width (1.2-2.0 times the vehicle size), radius of curvature (1-3m), and obstacle ratio (0.2-0.5 times the traffic width). This disclosure can extract global features by performing dimension-wise extreme value extraction on each coordinate dimension of the boundary point set using max pooling, or by performing mean statistics on each dimension of the boundary point set using average pooling, followed by nonlinear mapping to enhance discriminability, thus obtaining global features.

[0072] In one embodiment shown, the step of executing the output acceleration in the simulation environment to obtain the vehicle's driving state and reward value at the next moment includes: executing the output acceleration in the simulation environment to obtain the vehicle's driving state at the next moment; calculating the reward value corresponding to the vehicle's driving state at the next moment according to a state evaluation function; wherein the state evaluation function includes a preset dimension and a preset weight corresponding to the preset dimension, and the preset dimension includes at least one of a safety dimension, an efficiency dimension, and a comfort dimension.

[0073] For example, the system receives an acceleration of -1.5 m / s² from the policy network output. 2 and advance a control cycle in the simulation environment ( The system updates the vehicle's position, speed, and orientation angle for the next moment; then, it calls a multi-dimensional weighted reward function to calculate the total reward, setting the formula as follows: If the shortest distance between the vehicle and the boundary of the drivable area at the current moment... If the depth is 0.8m, then a safety bonus will be awarded. If the current vehicle speed The optimal vehicle speed for this scenario is 5.4 km / h. If the speed is 6.0 km / h, then the efficiency bonus is... If the acceleration was predicted at the previous moment... -1.2 m / s 2 Current actual acceleration -1.5 m / s 2 Then comfort reward Finally, the system performs a weighted calculation of the rewards from each dimension to obtain the total reward value. .

[0074] It should be noted that, It refers to the "relative error between the actual speed and the optimal speed": the smaller the error, the closer the actual speed is to the optimal speed, and the higher the reward. At that time, the error was 0. (Maximum Reward). When the error is greater than or equal to 1 (speed deviation exceeds 100%), the reward drops to 0, but... To ensure that rewards do not become negative, the penalty for efficiency is not too large, which would overshadow the guiding effect of safety rewards on driving behavior. The guidance strategy network should not slow down excessively for safety, but should maintain a traffic efficiency close to that of human drivers, so that the car does not drive too slowly and affect the experience.

[0075] The safety dimension quantifies the proximity of the vehicle to the boundary using an exponential decay function. A large negative penalty (e.g., -10) can be directly applied upon collision, using strong penalties to encourage the policy network to prioritize collision avoidance. The efficiency dimension measures traffic efficiency through normalized speed deviation, encouraging vehicles to drive close to the optimal speed (the optimal speed is pre-set according to different scenarios, typically the speed of experienced drivers in the corresponding scenario). The comfort dimension is quantified based on the rate of change of longitudinal acceleration (i.e., jerk), with more drastic changes resulting in lower scores, thus guiding vehicles to balance traffic efficiency and passenger comfort while ensuring safety. The weights of each dimension in the multi-dimensional weighted reward function are ranked as follows: Safety > Efficiency > Comfort. The primary goal of assisted driving is to avoid collisions, hence the highest weight for safety; secondly, traffic efficiency; and finally, passenger comfort. This weight allocation aligns with the design logic of assisted driving.

[0076] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating an offline training architecture for a reinforcement learning agent, as shown in an exemplary embodiment. Figure 3 As shown, the scene generator constructs diverse driving scenarios, including roads with different curvatures, obstacles, and vehicle state information, and generates a set of boundary points for the vehicle's drivable area based on these scenarios. The vehicle dynamics model simulates the real vehicle's response to control commands based on physical laws, ensuring that the action execution results conform to actual motion characteristics. The set of boundary points for the vehicle's drivable area and the vehicle state information are standardized by the data preprocessing module and then fed into the reinforcement learning agent. The agent consists of an encoding network, a policy network, and a value network. The encoding network constructs a fusion feature vector based on the vehicle state information and the set of boundary points for the vehicle's drivable area, and inputs the fusion feature vector into the policy network and the value network (in practice, they can be input separately or first into the policy network). The policy network outputs acceleration (positive or negative) based on the fusion feature vector, while the value network evaluates the long-term reward potential of the current state. During the training loop, the acceleration output by the reinforcement learning agent is fed into the environment module, which generates the state for the next moment and submits it to the reward calculation module to evaluate the quality of the current decision. Finally, the internal parameters of the reinforcement learning agent are updated in reverse through the PPO optimization policy algorithm, achieving continuous policy iteration and performance improvement.

[0077] Regarding the transformation of the boundary point set of the vehicle's drivable area into a unified global feature representation by the coding network, this embodiment provides two methods: Implementation Method 1: Lightweight Encoding Based on PointNet. A point embedding layer maps the coordinates of individual boundary points to low-dimensional features. Then, global max pooling is used to aggregate the features of all points, obtaining global features without capturing local neighborhood relationships. The calculation formula is as follows: ;in This is the embedding layer weight matrix. For bias vectors, This represents a global max-pooling operation. This method offers fast inference speed and is suitable for scenarios with high real-time requirements.

[0078] Implementation Method Two: Lightweight Convolutional Encoding Based on PointCNN. This method captures the spatial topological relationships and local detail features of a single point and its neighbors through lightweight point cloud convolution operations, followed by global max pooling aggregation. The calculation formula is as follows: ,in For the first The method utilizes the neighborhood point set of each point. This approach is more adaptable to complex boundary scenarios (such as sharp bends and irregular obstacles).

[0079] Both the policy network and value network in the reinforcement learning agent adopt a two-layer fully connected architecture based on fused feature vectors, and specific activation functions and distribution modeling methods are designed for their respective output targets to meet the requirements of vehicle control safety, continuity and PPO algorithm training stability.

[0080] Specifically, the policy network is used to generate accelerations that conform to vehicle dynamics constraints based on the current state. Its structure consists of three layers: the first layer is a fully connected layer to fuse feature vectors. As input, through the weight matrix and bias vector After linear transformation, the intermediate features are processed by the ReLU activation function. The second fully connected layer fuses feature vectors. As input, through the weight matrix and bias vector After linear transformation, the ReLU activation function is applied to output higher-order decision features. The third layer is a Gaussian-distributed output layer, based on... Calculate the mean separately With variance The mean is derived from We obtain the variance from This ensures that the variance is non-negative; finally, the policy network outputs a Gaussian distribution. And the predicted acceleration is obtained by sampling from this distribution. ,in This ensures that all output actions are within physically feasible limits, achieving safe and smooth longitudinal control, by maximizing the vehicle's deceleration.

[0081] The value network is used to evaluate the expected long-term cumulative discounted reward of the current state (i.e., the state value), providing a benchmark for calculating the advantage function of the PPO algorithm. Its structure also consists of three layers: the first fully connected layer receives the fused feature vector. Through the weight matrix and bias vector After linear transformation, the intermediate features are processed by the ReLU activation function. The second fully connected layer receives... Through the weight matrix and bias vector After linear transformation, the ReLU activation function is applied to output higher-order features. The third layer is the state value output layer, which uses a linear activation function to directly regress the scalar value, i.e. ,in and These are the output layer weights and biases, respectively. This design avoids the distortion of value estimation caused by nonlinear activation, enabling the network to more accurately reflect the expected level of future returns in the current state, thereby improving the efficiency and stability of policy gradient updates.

[0082] The policy network and the value network mentioned above share the same input features. However, each parameter is optimized independently to form the strategy-value architecture required by the PPO algorithm. They work together and iterate during offline training to gradually approach the optimal driving strategy.

[0083] Secondly, corresponding to the embodiments of the above-described vehicle driving control method, this disclosure also provides an embodiment of a vehicle driving control device.

[0084] Please see Figure 4 , Figure 4 This is an exemplary embodiment illustrating the hardware structure of an electronic device. At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, memory 408, and non-volatile memory 410, and may also include other necessary hardware. One or more embodiments of this disclosure can be implemented in software, for example, the processor 402 reads the corresponding computer program from the non-volatile memory 410 into memory 408 and then runs it. Of course, besides software implementation, one or more embodiments of this disclosure do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0085] Please see Figure 5 , Figure 5This is a block diagram illustrating an exemplary embodiment of a vehicle driving control device 500. This vehicle driving control device 500 can be applied to, for example... Figure 4 The illustrated electronic device is used to implement the technical solution of this disclosure. The device includes: The acquisition unit 502 is used to acquire the set of boundary points of the current drivable area of ​​the vehicle and the current vehicle status during actual vehicle driving. Pooling unit 504 is used to perform global pooling on the set of boundary points of the current vehicle drivable area to obtain the global features of the current vehicle drivable area. The splicing unit 506 is used to splice the current vehicle state and the obtained global features to obtain the current fused feature vector; The input unit 508 is used to input the current fused feature vector into the reinforcement learning agent to obtain the acceleration output by the reinforcement learning agent; Control unit 510 is used to control the vehicle's movement based on the output acceleration.

[0086] In some embodiments, obtaining the set of boundary points of the current drivable area of ​​the vehicle includes: Obtain environmental data of the road where the vehicle is currently located; The environmental data is processed to obtain the initial set of boundary points of the current drivable area of ​​the vehicle; The initial boundary point set is transformed to the vehicle coordinate system, and the initial boundary point set is resampled to obtain a boundary point set with a preset number of boundary points. The preset number is consistent with the number of boundary points in the set of boundary points of the drivable area used by the reinforcement learning agent during training.

[0087] In some embodiments, prior to the resampling operation on the initial set of boundary points, the apparatus further includes: Normalization unit 512 is used to normalize the initial boundary point set transformed to the vehicle coordinate system to obtain the normalized boundary point set. Clustering unit 514 is used to perform clustering processing on the normalized boundary point set using a spatial clustering algorithm to obtain at least one cluster. The calculation unit 516 is used to calculate the distance between each boundary point that does not belong to the cluster and the cluster, and to remove boundary points whose distance is greater than a preset threshold to form a denoised boundary point set.

[0088] In some embodiments, the reinforcement learning agent comprises a policy network and a value network, and the reinforcement learning agent is trained in the following manner: In the simulation environment, vehicle status information and a set of boundary points of the vehicle's drivable area are generated, and then a fused feature vector is constructed based on the generated vehicle status information and the set of boundary points of the vehicle's drivable area. The fused feature vector is input into the policy network and the value network to obtain the acceleration output by the policy network and the current state value output by the value network. The output acceleration is executed in the simulation environment to obtain the vehicle's driving state and reward value at the next moment; and the vehicle's driving state at the next moment is input into the value network to obtain the state value at the next moment. The loss is calculated based on the current state value, the reward value, and the next state value, and the network parameters of the policy network and the value network are updated based on the loss until the training of the two networks converges.

[0089] In some embodiments, calculating the loss based on the current state value, the reward value, and the next time-step state value, and updating the network parameters of the policy network and the value network based on the loss, includes: Calculate the target value based on the reward value and the state value at the next moment; A value network loss function is constructed based on the mean square error between the target value and the current state value, and the parameters of the value network are updated based on the value network loss function. The policy network loss function is constructed based on the output of the value network after the parameter update and the output acceleration, and the parameters of the policy network are updated based on the policy network loss function.

[0090] In some embodiments, generating vehicle state information and a set of boundary points of the vehicle's drivable area in a simulation environment, and then constructing a fused feature vector based on the generated vehicle state information and the set of boundary points of the vehicle's drivable area, includes: The road environment and vehicle status information for vehicle driving are randomly generated in the simulation environment. Based on the road environment, determine the set of boundary points of the vehicle-accessible area; Global pooling is performed on the boundary point set of the vehicle drivable area to obtain the global features of the vehicle drivable area; wherein, the global features are used to represent the overall geometric shape and spatial distribution features of the vehicle drivable area. The global features and the vehicle state information are concatenated to obtain a fused feature vector.

[0091] In some embodiments, the step of executing the output acceleration in the simulation environment to obtain the vehicle's driving state and reward value at the next moment includes: The output acceleration is executed in the simulation environment to obtain the vehicle's driving state at the next moment; Based on the state evaluation function, calculate the reward value corresponding to the vehicle's driving state at the next moment; The state evaluation function includes a preset dimension and a preset weight corresponding to the preset dimension. The preset dimension includes at least one of a safety dimension, an efficiency dimension, and a comfort dimension.

[0092] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0093] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0094] The systems, devices, modules, or units described in the above embodiments can be implemented by a computer or entity, or by a product with a certain function. A typical implementation device is a computer, which can be a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0095] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0096] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0097] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0098] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0099] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0100] The foregoing has described specific embodiments of this disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0101] The terminology used in one or more embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this disclosure. The singular forms “a,” “the,” and “the” as used in one or more embodiments of this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0102] It should be understood that although the terms first, second, third, etc., may be used to describe various information in one or more embodiments of this disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of one or more embodiments of this disclosure, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0103] The above description is merely a preferred embodiment of one or more embodiments of this disclosure and is not intended to limit the scope of one or more embodiments of this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this disclosure should be included within the scope of protection of one or more embodiments of this disclosure.

Claims

1. A vehicle travel control method characterized by comprising: The method includes: During actual vehicle operation, acquire the set of boundary points of the current drivable area and the current vehicle status; Global pooling is performed on the set of boundary points of the current drivable area of ​​the vehicle to obtain the global features of the current drivable area of ​​the vehicle. The current vehicle state and the obtained global features are concatenated to obtain the current fused feature vector; The current fused feature vector is input into the reinforcement learning agent to obtain the acceleration output by the reinforcement learning agent; The vehicle's movement is controlled based on the output acceleration.

2. The method of claim 1, wherein, The step of obtaining the set of boundary points of the current drivable area of ​​the vehicle includes: Obtain environmental data of the road where the vehicle is currently located; The environmental data is processed to obtain the initial set of boundary points of the current drivable area of ​​the vehicle; The initial boundary point set is transformed to the vehicle coordinate system, and the initial boundary point set is resampled to obtain a boundary point set with a preset number of boundary points. The preset number is consistent with the number of boundary points in the set of boundary points of the drivable area used by the reinforcement learning agent during training.

3. The method according to claim 2, characterized in that, Before performing the resampling operation on the initial boundary point set, the method further includes: The initial set of boundary points transformed to the vehicle coordinate system is normalized to obtain the normalized set of boundary points. The normalized set of boundary points is clustered using a spatial clustering algorithm to obtain at least one cluster. Calculate the distance between each boundary point that does not belong to the cluster and the cluster, and remove boundary points whose distance is greater than a preset threshold to form a denoised boundary point set.

4. The method according to any one of claims 1 to 3, characterized in that, The reinforcement learning agent comprises a policy network and a value network, and is trained in the following manner: In the simulation environment, vehicle status information and a set of boundary points of the vehicle's drivable area are generated, and then a fused feature vector is constructed based on the generated vehicle status information and the set of boundary points of the vehicle's drivable area. The fused feature vector is input into the policy network and the value network to obtain the acceleration output by the policy network and the current state value output by the value network. The output acceleration is executed in the simulation environment to obtain the vehicle's driving state and reward value at the next moment; and the vehicle's driving state at the next moment is input into the value network to obtain the state value at the next moment. The loss is calculated based on the current state value, the reward value, and the next state value, and the network parameters of the policy network and the value network are updated based on the loss until the training of the two networks converges.

5. The method of claim 4, wherein, The step of calculating the loss based on the current state value, the reward value, and the next time-instance state value, and updating the network parameters of the policy network and the value network based on the loss, includes: Calculate the target value based on the reward value and the state value at the next moment; A value network loss function is constructed based on the mean square error between the target value and the current state value, and the parameters of the value network are updated based on the value network loss function. The policy network loss function is constructed based on the output of the value network after the parameter update and the output acceleration, and the parameters of the policy network are updated based on the policy network loss function.

6. The method of claim 4, wherein, The process of generating vehicle state information and a set of boundary points of the vehicle's drivable area in a simulation environment, and then constructing a fused feature vector based on the generated vehicle state information and the set of boundary points of the vehicle's drivable area, includes: The road environment and vehicle status information for vehicle driving are randomly generated in the simulation environment. Based on the road environment, determine the set of boundary points of the vehicle-accessible area; Global pooling is performed on the boundary point set of the vehicle drivable area to obtain the global features of the vehicle drivable area; wherein, the global features are used to represent the overall geometric shape and spatial distribution features of the vehicle drivable area. The global features and the vehicle state information are concatenated to obtain a fused feature vector.

7. The method of claim 4, wherein, The process of executing the output acceleration in the simulation environment to obtain the vehicle's driving state and reward value at the next moment includes: The output acceleration is executed in the simulation environment to obtain the vehicle's driving state at the next moment; Based on the state evaluation function, calculate the reward value corresponding to the vehicle's driving state at the next moment; The state evaluation function includes a preset dimension and a preset weight corresponding to the preset dimension. The preset dimension includes at least one of a safety dimension, an efficiency dimension, and a comfort dimension.

8. A vehicle travel control device characterized by comprising: The device includes: The acquisition unit is used to acquire the set of boundary points of the current drivable area and the current vehicle status during actual vehicle driving. The pooling unit is used to perform global pooling on the set of boundary points of the current drivable area of ​​the vehicle to obtain the global features of the current drivable area of ​​the vehicle. The splicing unit is used to splice the current vehicle state and the obtained global features to obtain the current fused feature vector; The input unit is used to input the current fused feature vector into the reinforcement learning agent to obtain the acceleration output by the reinforcement learning agent. The control unit is used to control the vehicle's movement based on the output acceleration.

9. An electronic device, comprising: It includes a communication interface, a processor, a memory, and a bus, wherein the communication interface, the processor, and the memory are interconnected via the bus; The memory stores machine-readable instructions, and the processor executes the method according to any one of claims 1 to 7 by invoking the machine-readable instructions.

10. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-readable instructions, which, when invoked and executed by a processor, implement the method described in any one of claims 1 to 7.

Images