Block chain assisted PPO-based automatic driving decision-making method

Through the aggregation of federated learning models combined with DAG blockchain and DDPG algorithm, and the autonomous driving decision model combining conditional imitation learning and PPO algorithm, the challenges of the autonomous driving decision model in different regions are solved, and the user privacy security challenges are improved. The generalization and training efficiency of the model are achieved, ensuring user data privacy and system security.

CN119067200AActive Publication Date: 2024-12-03CHONGQING UNIV OF POSTS & TELECOMM

Patent Information

Application Number
CN202411127492.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-12-03
Estimated Expiration
2044-08-16

AI Technical Summary

Technical Problem

The existing autonomous driving decision-making model has challenges in data heterogeneity and user privacy security in different regions. Traditional federated learning algorithms cannot effectively solve the problem of non-independent and homogeneous data distribution, and reinforcement learning training is slow and resource consumption is large, making it difficult to be safe. and efficiency trade-off.

Method used

The two-layer network model based on DAG blockchain is used to combine the DDPG algorithm for federated learning model aggregation, and the policy network is pre-trained through conditional imitation learning, and the PPO algorithm is used to train the autonomous driving decision model, and the model parameters and reward functions are dynamically adjusted to improve the Decision-making accuracy and security, while protecting user data privacy through blockchain technology.

Benefits of technology

It improves the generalization and robustness of the autonomous driving decision model, improves training efficiency and system security, ensures user data privacy and security, realizes low-load CAV collaborative training and prevents single point of failure.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067200A_ABST
    Figure CN119067200A_ABST
Patent Text Reader

Abstract

The invention relates to a block chain assisted PPO-based automatic driving decision-making method, and belongs to the technical field of mobile communication. An intelligent connected automobile CAV automatic driving decision-making model needs massive training data, but sharing of CAV data causes privacy disclosure and data security problems. In order to solve the problem, a double-layer vehicle networking architecture combining a directed acyclic graph (DAG) block chain and a federated learning technology is firstly designed, and an asynchronous federated learning scheme based on a depth deterministic policy gradient (DDPG) algorithm is designed based on the architecture; secondly, in a local training stage, proposing a strategy network pre-training algorithm based on conditional imitation learning CIL; and finally, proposing an automatic driving decision model based on a near-end strategy optimization PPO, and designing a personalized reward function in a complex road scene. According to the scheme, low-load CAV cooperative training is realized while the CAV automatic driving decision performance is improved, and the privacy security of CAV data is ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of mobile communication technology and relates to a blockchain-assisted PPO-based autonomous driving decision-making method. Background Art

[0002] Connected and Automated Vehicles (CAV), as one of the key applications of the sixth generation of mobile communication technology (6G), is greatly improving people's travel experience. CAV uses on-board sensors and communication modules to perceive road information and interact with road side units (RSU) to help drivers or autonomous driving systems make optimal driving decisions. Artificial intelligence is considered to be a key technology for realizing autonomous driving. Reinforcement learning, as an evolutionary technology of artificial intelligence, plays an important role in realizing autonomous driving. Through the training model of CAV perception data, real-time mapping from data to decision-making is achieved, and intelligent applications such as pedestrian detection, collision warning, and traffic sign recognition are realized. However, decision models trained using data from the same region are often difficult to achieve good generalization, because there are limitations on perception data in the same region. The ideal solution is to collect large-scale road data from different regions to centrally train a model with strong generalization, but the centralized training method cannot meet the requirements of user privacy and data security. In addition, the potential single point failure of vehicle service providers and the problem of non-independent and identically distributed data between different regions also limit the rapid application of autonomous driving.

[0003] Federated learning, as a distributed learning paradigm, allows multiple participants to train their own models locally, and then aggregate the global model only by exchanging model parameters, rather than directly exchanging data, thereby protecting user privacy and security. In addition, blockchain technology, as a distributed database technology, stores and manages data in a distributed manner, thereby realizing decentralized data management and transaction processing. Combining blockchain and federated learning can achieve data security while preventing single point failure problems of vehicle service providers. However, road data from different regions often show heterogeneity, which affects the quality of global model fusion. The traditional FedAvg algorithm simply uses weighted average to aggregate model parameters, which cannot solve the problem of non-independent and identically distributed data and does not consider the impact of client model quality, which will lead to a decline in global model quality.

[0004] Reinforcement Learning (RL) is a machine learning method. Different from traditional supervised learning and unsupervised learning, reinforcement learning learns through trial and error in the interaction between the agent and the environment to maximize the expected cumulative reward. However, traditional reinforcement learning training is slow and requires a lot of resources, and how to balance efficiency and comfort while ensuring safety needs to be solved urgently.

[0005] Based on the above problems, the present invention designs an autonomous driving decision model assisted by DAG blockchain. Under this architecture, the accuracy of the global model will be improved by dynamically adjusting the federated fusion parameters, and the privacy and security of user data will be protected by relying on the decentralized characteristics of blockchain technology. In addition, the autonomous driving policy model is trained based on the Proximal Policy Optimization (PPO) method, and pre-training is used to reduce the time and resources required for the exploration phase in reinforcement learning. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a blockchain-assisted PPO-based autonomous driving decision-making method.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] In the first aspect, the present invention provides a secure and high-quality data sharing solution based on blockchain according to the data sharing scenario of Internet of Vehicles and the privacy protection requirements of vehicle data. The execution process of the solution is as follows:

[0009] S1: Two-layer network model based on DAG blockchain;

[0010] S2: Federated learning model aggregation solution based on DDPG algorithm;

[0011] S3: Policy network pre-training scheme based on conditional imitation learning;

[0012] S4: Autonomous driving decision model based on PPO.

[0013] On the second aspect, in S1, an embodiment of the present invention establishes a two-layer network model based on the DAG blockchain: 1) DAG layer, including RSU responsible for model aggregation, and the DAG-structured blockchain maintained by it; 2) CAV layer, including CAV equipped with multiple sensors for autonomous driving decision-making.

[0014] RSU is an edge node equipped with a server deployed near the road, with powerful communication, storage and computing capabilities. In the example of the present invention, RSU not only participates in the blockchain network as a node, but also needs to train a reinforcement learning model that can derive the optimal aggregation weight. Due to the heterogeneity of BEV image data caused by different road environments, and RSUs in different regions cannot directly exchange data, when aggregating each node model, weighted aggregation should be performed in combination with the model aggregation weight given by the DDPG algorithm.

[0015] CAV uses a proximal policy optimization algorithm for horizontal and vertical decision-making, and is equipped with an expert system to generate demonstration data locally. In the early stages of training, CAV collects expert demonstration data and uses a conditional imitation learning algorithm to pre-train the policy network. Due to the differences in the locations of different CAVs, the collected demonstration data often presents heterogeneity, which leads to a decrease in global model performance.

[0016] In the third aspect, the embodiment of the present invention uses a federated learning model aggregation scheme based on the DDPG algorithm in S2, and uses a weighted aggregation federated learning algorithm. The algorithm uses the weights and model accuracy of each node as the state space St, changes the model aggregation weight by taking actions At, and finally updates the DDPG algorithm network parameters through model accuracy evaluation rewards.

[0017] Fourthly, in S3, an embodiment of the present invention proposes a method for pre-training a policy network using conditional imitation learning. This method learns to perform tasks by observing and imitating the behavior of experts, and adds information such as expert intentions, goals, and prior knowledge to traditional imitation learning, which together with the observation vector determine the action. CAV obtains road information data through on-board cameras and other sensors, and the MobileNet convolutional neural network processes it into a latent space to help CAV process information faster and make decisions. In the process of pre-training the policy network, the vehicle policy models of different regions are shared through the two-layer network based on the DAG blockchain proposed in S1 to improve model performance and improve training efficiency.

[0018] In the fifth aspect, in S4, an embodiment of the present invention proposes an autonomous driving decision model based on the PPO algorithm in complex road scenarios. In this model, the autonomous driving system learns the optimal driving strategy through interaction with the environment, maximizes the cumulative reward through iterative optimization strategy, and defines a multi-task objective reward function from the perspectives of safety, efficiency and location. The algorithm limits the change range of the strategy parameters in each update to keep the strategy updated within a small range, and uses the importance sampling ratio to control the update direction to prevent the gradient update from being too large while keeping the update direction close to the current strategy.

[0019] The beneficial effects of the present invention are:

[0020] (1) Improving autonomous driving decision-making performance

[0021] Through federated learning, driving data from vehicles in different regions can be aggregated to train a more generalized autonomous driving decision model, thereby improving the robustness and adaptability of the model.

[0022] The PPO algorithm can effectively process continuous action space and maximize cumulative rewards through strategy optimization, thereby improving the efficiency and safety of autonomous driving decisions.

[0023] The personalized reward function designed for complex road scenarios can better guide the autonomous driving decision model to learn and make decisions, thereby improving the accuracy and safety of decisions.

[0024] (2) Protecting user data privacy and security

[0025] DAG blockchain uses distributed ledger technology to store data in multiple nodes, avoiding the risk of data leakage caused by centralized storage.

[0026] Federated learning only exchanges model parameters and does not share original data, effectively protecting user data privacy.

[0027] Data can be encrypted to further protect user data security.

[0028] (3) Improve training efficiency

[0029] Federated learning allows vehicles to perform model training locally, avoiding delays and bandwidth consumption caused by data transmission and improving training efficiency.

[0030] Pre-training through conditional imitation learning can reduce the exploration phase in reinforcement learning and thus speed up training.

[0031] The DDPG algorithm can effectively process continuous action space and model it through deep neural networks, which improves training efficiency.

[0032] (4) Realizing low-load CAV collaborative training

[0033] Federated learning allows vehicles to perform model training locally, avoiding the high load problem caused by centralized training.

[0034] The distributed architecture of DAG blockchain can effectively disperse the network load and realize low-load CAV collaborative training.

[0035] (5) Improve system security

[0036] Blockchain technology is tamper-proof and traceable, and can effectively prevent malicious attacks and data tampering.

[0037] Federated learning can avoid the single point failure problem caused by centralized storage and improve system security.

[0038] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0040] Figure 1 This is the architecture diagram of the Internet of Vehicles based on the DAG blockchain;

[0041] Figure 2 This is a flowchart of weighted federated learning based on DAG blockchain;

[0042] Figure 3 Schematic diagram for optimizing driving decision training based on PPO;

[0043] Figure 4 Flowchart for blockchain-assisted PPO-based autonomous driving algorithm execution. DETAILED DESCRIPTION

[0044] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0045] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0046] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0047] Figure 1 FIG. 1 shows a possible structural diagram of a communication system involved in an embodiment of the present invention. Figure 1 As shown in Figure 1, the network considers a two-layer structure, including a DAG layer and a CAV layer. The DAG layer is a blockchain based on a DAG structure, which is jointly maintained by the RSUs near the road. Each transaction exists as an independent node. These nodes are connected by directed edges to form a directed acyclic graph structure, which is used to record global model updates. The CAV layer consists of CAVs equipped with multiple sensors, which have environmental perception capabilities and can communicate with RSUs through wireless links.

[0048] 1. Transaction verification process based on DAG blockchain

[0049] In the DAG-based blockchain, each transaction is generated by the CAV participating in the collaborative training and consists of a transaction header and a transaction body. The transaction header contains the version number, timestamp, random number, transaction hash value, the number of forward transactions and the hash values ​​of all transactions pointing to ensure the validity and consistency of the transaction; the transaction body contains the strategy model and model aggregation weight uploaded by the CAV after local training, which are transmitted and shared through the DAG blockchain.

[0050] Figure 1The middle DAG layer shows that the DAG structure is mainly composed of Genesis transactions, Confirm transactions, Unconfirm transactions, Tips, and New Tips. Genesis transactions are the creation transactions in the DAG structure. All subsequent transactions can be traced back to Genesis transactions, which marks the starting point of the entire DAG chain and the generation of the initial model. Confirm transactions are transactions that are pointed to by enough other transactions in the DAG structure and are therefore considered credible. Unconfirm transactions are transitional transactions that have not yet been pointed to by enough other transactions. These transactions are waiting for enough new transactions to point to them so that they can eventually become Confirm transactions. As new transactions are continuously added and confirmed, the status of Unconfirm transactions will gradually change to Confirm transactions, thereby being added to the list of credible transactions. Tips are unconfirmed transactions that have been propagated in the edge network but have not yet been included in the DAG structure. These transactions need to be verified and confirmed by subsequent new transactions.

[0051] 2. Weighted federation aggregation process based on DDPG algorithm

[0052] Weighted federated learning based on the DDPG algorithm mainly includes two processes: first, RSU calculates its model aggregation weight based on the accuracy of the CAV model; second, RSU aggregates the CAV model weight into a global model. Figure 2 As shown in the weighted federated aggregation based on the DDPG algorithm, the model aggregation weight is dynamically adjusted according to the quality of the CAV model, and the resulting global model can achieve higher accuracy. The global model aggregation process is divided into the following steps:

[0053] (1) Get Tips: RSU sends a request to obtain all Tips in the current edge network;

[0054] (2) Calculate the aggregation weight: RSU obtains the model accuracy of each node and forms the current state S t , randomly take action A t Change the fusion weight and get the reward R t Then enter the next state S t+1 , store this experience. When enough experience is obtained, randomly select an experience from the experience database, input the state and action into the Q network, and update the parameters of the Q network and μ network. The current optimal model aggregation weight can be obtained through the μ network.

[0055] (3) Model aggregation: RSU weights and aggregates all models in Tips to obtain a global model, and sends it to the corresponding CAV;

[0056] (4) Local training: Based on the global model sent by the RSU, the CAV updates the model parameters through the Dagger algorithm and records the model accuracy;

[0057] (5) Model on-chain: CAV packages the local trained policy model and model accuracy into a New Tip and sends it to the corresponding RSU. The RSU adds the version number, timestamp, random number, and transaction hash value, and uploads it to the DAG blockchain, which becomes a Tip and is broadcast in the edge network.

[0058] 3. Policy network pre-training process based on conditional imitation learning

[0059] Conditional Imitation Learning (CIL) is a method of achieving tasks by observing the behavior of a demonstrator and imitating these behaviors under specific conditions. Expert intentions, goals, and prior knowledge information are added to the ordinary imitation learning algorithm. Before inputting into the fully connected layer, the images collected by the on-board camera and the real vehicle information need to be encoded. Due to the limited computing resources of CAV, the encoding module adopts the MobileNet neural network architecture, which can minimize the number of parameters and computational complexity of the model while maintaining good performance, so that the model can run efficiently in a resource-constrained environment. The Dagger algorithm is used in CAV to train the local model:

[0060] (1) Data collection phase: Experts first collect data in the simulation scenario and record observation-action pairs;

[0061] (2) Model training phase: An initial model is trained using the initial expert dataset. The model predicts actions from states, with the goal of imitating the expert’s behavior as closely as possible.

[0062] (3) Data augmentation: Use the local model to perform actions in the environment to generate new data, and merge the newly generated data with the expert data to form a richer augmented dataset.

[0063] (4) Model update: Use the enhanced dataset to train the local model again until the model converges.

[0064] 4. Autonomous driving decision model based on PPO algorithm

[0065] The design of the autonomous driving decision model based on the PPO algorithm is as follows Figure 3 As shown in the figure, it mainly consists of two parts: the Actor network and the Critic network. The Actor network selects the action strategy in a given state. By comparing the ratio between the current strategy and the old strategy and the advantage function, the objective function is jointly constructed to update the network parameters. The Critic network is responsible for evaluating the quality of the action. The input is the interaction data obtained by executing the current strategy in the environment, and the network parameters are updated through the advantage function. Among them, the agent's perception state, action and reward are defined as follows:

[0066] (1) Status:

[0067] The BEV image is rendered by C bird olive images of size W×H. The vehicle state s(t) observed by the BEV image includes the target vehicle position, speed, acceleration, wheel steering angle, lane number, and road and weather conditions. Therefore, the vehicle state at time t is:

[0068] s(t)={(x t ,y t ),(vx t ,vy t ),a t ,θ t ,l t} (1)

[0069] where x t and t represents the horizontal and vertical coordinates of the vehicle at time t, vx t and vy t represents the vertical speed and horizontal speed of the vehicle at time t, a t Represents the acceleration of the vehicle in the direction of travel at time t. t is the turning angle, which consists of the inner wheel steering angle and the outer wheel steering angle. t Indicates the lane number of the target vehicle. Since only two-lane lane change scenarios are considered, there are only two possibilities: 0 and 1.

[0070] (2) Action

[0071] During driving, the action space mainly includes the acceleration and steering angle of the vehicle. Therefore, the action space a(t) of the vehicle at time t is expressed as:

[0072] a(t)={a a (t),θ(t)} (2)

[0073] where a a (t) represents the acceleration of the vehicle in the direction of travel at time t. To ensure the safety and comfort of passengers, the acceleration range is limited to [-4, 2] m / s 2 θ(t) represents the steering angle, where the inner wheel steering angle θ it ∈[-36,36], outer wheel steering angle θ ot ∈[-33.5,33.5].

[0074] (3) Reward Function

[0075] In deep reinforcement learning, a task-specific reward function is used to guide the adjustment of network parameters and help the agent make correct decisions. This paper designs a reward function for the rural two-lane road scenario from the perspectives of vehicle position, safety, and efficiency. At the same time, the weather factor is taken into account to calculate the maximum safe driving speed of the car without skidding in the current weather, and punish the agent that exceeds the speed limit.

[0076] In terms of vehicle position, the vehicle should be as close to the target position as possible without deviating from the center of the road. The reward function is designed as follows:

[0077] r loca =k l1 r θ +k l2 r d +k l3 r f (3)

[0078] where r θ Indicates the penalty for deviation from the center of the road. When there is an obstacle in the current lane, the value is 0, allowing the vehicle to deviate from the center of the current lane to avoid collision with the obstacle. When there is no obstacle, the value is proportional to the difference between the vehicle deflection angle and the actual road deflection angle, which is determined by the constant k. l1 Make proportional adjustments. d represents the destination difference penalty, which encourages the vehicle to get as close to the target location as possible. In the event of a collision, the farther away from the destination, the greater the penalty, and the ratio is determined by k. l2 Control. f represents the task completion reward. A high reward will be given when the vehicle reaches the target location. If a collision occurs before the target location or exceeds the drivable range, a corresponding penalty will be imposed. The proportion is determined by k l3 control.

[0079] In terms of vehicle safety, in addition to the road speed limit, the impact of weather on the maximum vehicle speed is also considered. The weather conditions are quantified as a constant value c between 0 and 1. w The smaller the quantitative value, the better the weather and the greater the ground friction. The larger the quantitative value, the worse the weather and the smaller the ground friction, which makes it easy to slip. Therefore, you should slow down your driving speed. This can be described by the formula

[0080]

[0081] In the above formula, v max Indicates the speed limit of the road. The more the target speed exceeds the maximum speed limit, the greater the penalty. w Indicates that the weather condition is c w In the case of , the penalty for turning speed is expressed as:

[0082]

[0083] Where c1 represents a constant which is proportional to the friction and wheelbase and inversely proportional to the mass of the bicycle.

[0084] In terms of lane-changing efficiency, if there is an obstacle in the current lane, the target vehicle should not collide with the obstacle while changing lanes as quickly as possible. Assume that the vehicle moves in a straight line at a uniform speed when changing lanes, and the maximum angle with the original lane is θ c , then the minimum speed required for the vehicle to change lanes is:

[0085]

[0086] In the above formula, r1 represents the turning radius of the inner lane, d represents the distance between the two lanes, and t c represents the earliest time to start lane changing. When the vehicle's lane changing speed is less than the minimum lane changing speed, the penalty function can be expressed as:

[0087] r ef =k e1 (v min -v t )||{change line} (7)

[0088] The final reward function can be expressed as a weighted sum of three reward functions, namely:

[0089] R=k1r loca +k2r sa +k3r ef (8)

[0090] Where k1, k2, and k3 represent the weights of the position, safety, and efficiency reward functions, respectively.

[0091] 5. System Process

[0092] Figure 4 The figure shows a flowchart of a blockchain-assisted PPO-based autonomous driving decision-making method. The specific execution steps are as follows:

[0093] Step 401: Algorithm initialization;

[0094] Step 402: The vehicle service provider issues a task through the RSU;

[0095] Step 403: Initialize the model and generate Genesis transaction;

[0096] Step 404: The CAV applies to be registered as a training vehicle;

[0097] Step 405: RSU processes the request information;

[0098] Step 406: CAV obtains SDK certificate;

[0099] Step 407: CAV initiates a request to obtain Tips from RSU;

[0100] Step 408: RSU initializes state space and network parameters;

[0101] Step 409: RSU obtains Tips;

[0102] Step 410: RSU updates the DDPG model state;

[0103] Step 411: RSU stores experience;

[0104] Step 412: RSU performs experience playback and inputs the state and action into the Q network;

[0105] Step 413: Critic network update;

[0106] Step 414: Actor network update;

[0107] Step 415: RSU updates Q network and μ network parameters;

[0108] Step 416: The RSU obtains the weight of each local model;

[0109] Step 417: The RSU weights and aggregates the local model according to the weights in step 416;

[0110] Step 418: RSU sends the global model to CAV;

[0111] Step 419: CAV collects data;

[0112] Step 420: The CAV performs pre-training and makes a decision;

[0113] Step 421: CAV calculates the loss function;

[0114] Step 422: CAV updates CIL network parameters;

[0115] Step 423: CAV calculates the strategy model accuracy;

[0116] Step 424: CAV generates Tip according to the local model;

[0117] Step 425: CAV uploads the Tip to the edge network;

[0118] Step 426: Tip is verified by other Tips;

[0119] Step 427: After the Tip is verified, it becomes an Unconfirm transaction. If it is not verified, return to step 425;

[0120] Steps 428-429: After the Unconfirm transaction is verified by enough transactions, it will become a Confirm transaction;

[0121] Step 430: If the stop condition is met, the CAV executes step 431, if not, it executes step 407;

[0122] Step 431: CAV obtains a global model;

[0123] Step 432: CAV execution strategy;

[0124] Step 433: The CAV collects trajectory data;

[0125] Step 434: CAV calculates advantage estimate;

[0126] Step 435: CAV optimization objective function;

[0127] Step 436: CAV update strategy;

[0128] Step 437: If the stop condition is reached, execute step 438, otherwise return to step 432;

[0129] Step 438: The algorithm ends.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A blockchain-assisted PPO-based autonomous driving decision-making method, characterized by: The method comprises the following steps: S1: A two-layer network model based on the directed acyclic graph (DAG) blockchain, including: DAG layer: includes the Road Side Unit (RSU) responsible for model aggregation and the blockchain based on the DAG structure, which is used to record global model updates; Connected and Automated Vehicles (CAV) layer: includes CAVs equipped with multiple sensors for autonomous driving decision making; S2: Federated learning model aggregation solution based on DDPG algorithm, including: RSU calculates its model aggregation weight based on the CAV model accuracy; RSU weights and aggregates the CAV models into a global model; S3: Policy network pre-training scheme based on conditional imitation learning, including: Collect expert data in the simulation scenario and record the set of observation-action pairs; Pre-train the policy network using conditional imitation learning; S4: Autonomous driving decision model based on proximal policy optimization (PPO), including: Define a multi-task objective reward function and design the reward function from the three aspects of vehicle position, safety and efficiency; Use the PPO algorithm to train the autonomous driving policy model.

2. According to claim 1, a blockchain-assisted PPO-based autonomous driving decision-making method is characterized by: In S1, the RSU in the DAG layer is responsible for maintaining the DAG blockchain, adding the version number and timestamp information to the transaction header in the New Tip, which becomes a Tip and is broadcast on the edge network.

3. According to claim 1, a blockchain-assisted PPO-based autonomous driving decision-making method is characterized by: In S2, the RSU dynamically adjusts the model aggregation weight according to the quality of the CAV model, and the obtained global model can achieve higher accuracy.

4. According to claim 1, a blockchain-assisted PPO-based autonomous driving decision-making method is characterized by: In S3, in conditional imitation learning, the decision-making behavior of the CAV depends not only on the observed expert behavior, but also on the current environmental state or specific conditions, allowing the CAV to adjust its decision-making behavior according to different environmental conditions, thereby improving the generalization and robustness of the model.

5. The blockchain-assisted PPO-based autonomous driving decision-making method according to claim 1, characterized in that: In S4, the PPO algorithm limits the change range of the policy parameters in each update to keep the policy update within a certain range, and uses the importance sampling ratio to control the update direction to prevent the gradient update from being too large while keeping the update direction close to the current policy.

Citation Information

Patent Citations

  • Distributed intelligent auxiliary automatic driving method based on block chain

    CN116588138A

  • Method of neural architecture search using continuous action reinforcement learning

    WO2022068934A1

Cited By

  • Emergency vehicle priority passing control method based on parameterized reinforcement learning

    CN119832755A