Personalized intelligent car following method based on double-layer block chain in Internet of Vehicles

Abstract

The invention relates to a personalized intelligent car following method based on a double-layer block chain in the Internet of Vehicles, and belongs to the technical field of mobile communication. At present, deep reinforcement learning DRL is used, so that CAV can continuously interact with the environment to learn and optimize strategies, and the adaptability and flexibility of system decision making are improved. However, the complexity of the current traffic environment makes a challenge to make an accurate response to the change of a bicycle independent learning environment. Federal learning can aggregate a plurality of learning models in different environments, and the adaptability of CAV to complex environments is improved. According to the invention, a group learning DRL architecture based on a double-layer block chain in an Internet of Vehicles scene is established, a CAV selection strategy based on a reputation value is provided based on CAV mobility, and then a personalized car following model based on a deterministic strategy gradient algorithm is provided. According to the method, personalized safe and intelligent car-following driving can be realized, and application of artificial intelligence in a car networking scene can be promoted by utilizing an edge network and CAV user data resources.

Description

Technical Field

[0001] The present invention belongs to the field of mobile communication technology and relates to a personalized intelligent following method based on a double-layer blockchain in a connected vehicle network. Background Art

[0002] With the development of intelligent transportation systems and Internet of Vehicles technology, autonomous driving and intelligent driving assistance systems have become hot research directions in the modern automotive industry. Internet of Vehicles (IoV) connects CAVs, infrastructure and networks to achieve real-time information sharing and intelligent scheduling, improving traffic efficiency and driving safety. However, traditional Internet of Vehicles systems face challenges in data privacy, security and communication efficiency when dealing with personalized driving needs and CAV following control.

[0003] Federated Learning (FL) is a distributed machine learning method that aims to jointly train models on multiple devices or nodes without uploading the data to a single server. Since data does not need to leave the local device, federated learning significantly improves data privacy and security. In addition, in federated learning, only model parameters are transmitted instead of original parameters, reducing the demand for communication bandwidth. This method is suitable for scenarios with high requirements for data privacy and security, but it is vulnerable to reverse inference attacks on model parameters, which reduces the security of the transmission process.

[0004] In this context, blockchain technology, with its characteristics of decentralization, immutability and transparency, can be combined with federated learning to provide system-level tamper-proof and traceability. The blockchain-based Internet of Vehicles system can achieve automated management through smart contracts and ensure the authenticity and security of data through a consensus mechanism, thereby effectively solving the single point of failure and data leakage problems of traditional centralized systems.

[0005] Deep reinforcement learning (DRL) is an important branch of machine learning. Through interaction with the environment, intelligent agents can autonomously learn optimal strategies. It is widely used in robot control, game battles, and intelligent driving. Applying DRL to CAV following control can achieve adaptive adjustment to complex dynamic environments and improve driving safety and comfort. However, the local data and computing resources of a single CAV are limited, making it difficult to fully train a high-quality reinforcement learning model.

[0006] In order to solve the above problems, the present invention designs a personalized intelligent following method based on a double-layer blockchain. This method combines blockchain and DRL technology, realizes collaborative learning and model sharing between CAVs through a double-layer blockchain network, and uses group wisdom to improve the following control performance of individual CAVs while protecting the privacy of CAV users. Specifically, the task publisher registers and publishes training tasks in the blockchain network, the CAV performs preliminary model training locally, and updates and shares the model through the blockchain network, thereby achieving efficient personalized following control while ensuring data privacy and security. The present invention not only improves the intelligence and adaptability of the Internet of Vehicles system, but also provides a safe and reliable solution, which is expected to be widely used in future intelligent transportation systems. Summary of the invention

[0007] In view of this, the purpose of the present invention is to provide a personalized intelligent following method based on a double-layer blockchain in the Internet of Vehicles, which ensures the security and privacy of Internet of Vehicles data through the distributed ledger of blockchain technology; at the same time, group reinforcement learning is used to train a personalized following model in a distributed environment to achieve efficient collaboration and following between CAVs.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] In the first aspect, the present invention provides a personalized intelligent following method based on a double-layer blockchain according to the personalized following intelligent driving scenario in the Internet of Vehicles and the CAV user data privacy protection requirements. The execution process of the method is as follows:

[0010] S1: Group learning DRL architecture based on two-layer blockchain;

[0011] S2: CAV selection strategy based on reputation value;

[0012] S3: Personalized car-following model based on DDPG;

[0013] Secondly, in S1, an embodiment of the present invention establishes a three-layer system model, which includes: 1) a blockchain layer, which is used by task publishers to register and publish learning content and related requirements, and record training results; 2) a learning layer, where CAV performs local model training; 3) a CAV layer, which displays CAV following scenarios in different situations.

[0014] Road Side Units (RSUs) and Base Stations (BSs) are key communication devices in the Internet of Vehicles, responsible for short-distance road-side communications and wide-area cellular network communications, respectively. They work together to provide efficient, safe and reliable Internet of Vehicles communication services, and promote the development and application of intelligent transportation systems. In the example of the present invention, RSU and CAV are divided into multiple partitions based on geographic location and BS coverage, and each partition collaborates and processes consensus tasks in parallel. Each partition independently maintains a blockchain to record the training process of the personalized following model. Model sharing is carried out in the form of federated learning. CAV only uploads training model parameters instead of local original data, thereby ensuring the data privacy of CAV users.

[0015] A two-layer blockchain structure is adopted at the blockchain layer, including the upper DAG-based main chain and the lower PBFT-based partitioned blockchain. Among them, the subchain deployed in each partition of the lower layer is independent and is responsible for coordinating the CAV in the partition to synchronously complete the training task. The subchain adopts PBFT consensus and is maintained by multiple RSUs in the partition, which is responsible for transaction verification (including identity verification and model verification) and model aggregation process in the partition. Partitioning can reduce the number of parallel transaction processing and effectively alleviate the blockchain throughput limit; the upper main chain adopts an asynchronous consensus mechanism based on DAG. Each transaction in the main chain contains a partition training model, providing a secure aggregation method between different partitions, realizing asynchronous cross-region verification of the model, and the DAG chain is jointly maintained by the BS of all partitions. Unlike the traditional blockchain chain structure, DAG allows multiple transactions to be added and verified at the same time due to its parallel structure and acyclicity, thereby realizing asynchronous processing.

[0016] The RSUs and BSs in a partition are connected via DSRC to form an edge computing network within the partition. The BSs between partitions are connected via a wired network to form an inter-region edge computing network. CAVs transmit data to BSs and RSUs via wireless links, and the entire process of model upload and aggregation will be recorded in the partition blockchain.

[0017] First, the task publisher registers and publishes the training task in the blockchain network, and CAV will perform personalized car-following model training based on DDPG. Secondly, the cloud sends the initial model to each partition blockchain. Finally, the BS of each partition will generate a genesis block for the task and the initial model, and add it to the corresponding partition PBFT subchain. All nodes train local models based on the genesis block at startup, process subsequent transactions through the consensus mechanism, and upload blocks.

[0018] On the third aspect, the embodiment of the present invention provides a method for uploading a model to a blockchain in S1. According to the local model obtained in S3, the selected CAV uploads its local model to the corresponding RSUs for aggregation. The RSUs sends a request to the blockchain partition network where it is located. Each sub-chain will execute the PBFT consensus and record the partition aggregation parameters. Then the RSUs package these parameters into transactions and upload them to the BSs. The RSU will complete the block header information of the transaction, including version number, timestamp, random number, hash value, etc. Finally, it is uploaded to the main chain DAG by the BSs. Each transaction in the main chain network contains a partition-trained model, which provides a secure aggregation method between partitions and realizes asynchronous cross-region verification.

[0019] In the fourth aspect, the embodiment of the present invention provides a CAV training task termination method in S1. The task publisher will request the BS to update the current model in real time and analyze the performance of the current model. When it has met the requirements, it will issue task termination information to each partition BS and RSUs, and the CAV training task will end. According to the car reputation value recorded in the RSUs, it will be rewarded or punished. In addition, the training task can also be carried out for a long time to enable it to continuously adapt to the dynamic changes of the environment.

[0020] In a fifth aspect, the embodiment of the present invention provides a CAV selection strategy based on reputation value in S2. In the continuous interaction between CAV and the environment, CAV will consider the performance of CAV from multiple angles, select high-quality CAVs to participate in aggregation, and the selected CAV will send its trained and updated model to the corresponding RSU for aggregation. The CAV reputation value consists of four parts: familiarity, freshness, time-varying, and importance.

[0021] Therefore, for the four parts of this reputation value:

[0022] 1. Familiarity: Since DRL cannot provide specific model accuracy, the accuracy is replaced by the average reward. The larger the average reward, the greater the familiarity;

[0023] 2. Freshness: Since the training process in each partition will be performed for multiple rounds, the reputation of the CAV in the most recent round of training is more important than in the past;

[0024] 3. Time-varying: The change of communication quality between CAV and RSU, and the calculation of the average transmission rate of CAV within a round of training time;

[0025] 4. Importance: Since some CAVs are close to the RSU, the models trained by their perceived information have high similarity, and the most important CAVs in the RSU coverage area are selected.

[0026] The selected CAV packages the trained local new model into a transaction and sends it to RSUs to obtain the updated partition model, and then aggregates the partition model to obtain the global model. Among them, the transaction header includes the hash value of all selected transactions, and the transaction body includes the trained model.

[0027] In the sixth aspect, in S3 of the embodiment of the present invention, CAV initiates an identity request to the blockchain network through its built-in communication module, uses the public key infrastructure PKI to verify the identity of CAV, and the public key and private key pair of CAV is used for signature verification. First, CAV generates a request message containing a unique identifier and a timestamp, and signs the message with its private key. After receiving the request, the authentication node in the blockchain network verifies the authenticity of the signature using the public key of CAV. Once the verification is passed, CAV will be granted access rights. After obtaining the current initial model, CAV loads it into the local computing module. CAV performs personalized car-following model training based on DDPG to achieve high-performance car-following that is close to the driver's driving style. First, CAV collects current environmental information and records it as the current state; the CAV policy network will generate corresponding action decisions based on the current environment, execute actions and reward and evaluate the action decisions; CAV stores relevant information about environmental interactions, actions, and rewards in the experience replay memory; when training is required, the experience data will be sampled in batches from the experience replay memory, and the network parameters will be updated using the experience data.

[0028] During the training process, we first construct the driving information of the target CAV and surrounding CAVs required for the CAV decision-making model, and then design the DDPG reward function by comprehensively considering factors such as driving style, driving efficiency, collision avoidance, and comfort to solve the problem of lack of safety assurance in the decision-making process.

[0029] The beneficial effects of the present invention are:

[0030] (1) Security and privacy protection

[0031] Two-layer blockchain architecture: Lower-layer partition blockchain: Based on PBFT consensus, it ensures data security and consistency within the partition and prevents malicious node attacks. Upper-layer DAG main chain: Based on DAG consensus, it realizes asynchronous cross-region verification, ensures the security of model aggregation, and prevents model tampering.

[0032] CAV only uploads training model parameters instead of original data to protect user privacy and prevent data leakage.

[0033] No central server is required, reducing the risk of single point failure and improving system reliability.

[0034] (2) Intelligence and personalization

[0035] Personalized car-following model: Based on the DDPG algorithm, a personalized car-following model is trained according to different driving styles to improve driving safety and comfort and meet the needs of different drivers.

[0036] Swarm intelligence: CAVs collaborate and learn from each other, share training models, and use swarm intelligence to improve individual following control performance and quickly adapt to complex traffic environments.

[0037] (3) Efficiency and reliability

[0038] CAV only uploads training model parameters instead of raw data, reducing communication bandwidth requirements and improving network efficiency. RSUs and BSs form an edge computing network to achieve efficient model training and aggregation, reduce the burden on central servers, and improve system response speed.

[0039] Each partition independently maintains a blockchain, reducing the number of transactions processed in parallel, alleviating blockchain throughput limitations, and improving system efficiency. Each partition can collaboratively process consensus tasks in parallel to improve system throughput.

[0040] (4) Dynamic adaptability

[0041] The model can be trained over a long period of time and continuously adapt to environmental changes, such as traffic flow, road conditions, etc., to improve the adaptability of the system.

[0042] CAV selection strategy based on reputation: select high-quality CAVs to participate in training, such as familiarity, freshness, time variability, importance and other indicators, to improve training efficiency and model accuracy, and improve system adaptability.

[0043] (5) Safe driving

[0044] Encourage safe driving behaviors, such as maintaining a safe distance and avoiding collisions, to reduce the risk of collisions and improve driving safety.

[0045] It is encouraged to maintain an appropriate following distance, such as aggressive, moderate, conservative, etc., to improve driving safety.

[0046] (6) Traffic efficiency

[0047] Encourage efficient driving behaviors, such as shortening the following distance and increasing the headway time, to improve traffic efficiency.

[0048] Improve the smoothness and stability of traffic flow, reduce traffic congestion and improve traffic efficiency.

[0049] (7) Passenger comfort

[0050] Encourage smooth driving, such as controlling acceleration, jerkiness, etc., to improve passenger comfort.

[0051] (8) System Scalability

[0052] The system adopts a layered architecture and is easy to expand, such as adding partitions, increasing the number of CAVs, etc., to improve system scalability.

[0053] The present invention realizes a safe, efficient, intelligent and personalized intelligent following method for Internet of Vehicles by combining blockchain, federated learning and deep reinforcement learning technologies. It has significant technical advantages and application value, and is expected to promote the development and application of intelligent transportation systems.

[0054] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0056] Figure 1 This is a personalized smart car-following model diagram based on a double-layer blockchain;

[0057] Figure 2 Flowchart for selecting CAV based on reputation value;

[0058] Figure 3 This is a personalized intelligent car-following framework diagram based on DDPG;

[0059] Figure 4 Execution flow chart of personalized intelligent car-following method based on double-layer blockchain. DETAILED DESCRIPTION

[0060] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0061] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0062] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0063] Figure 1 The figure shows a personalized intelligent car-following model diagram based on a double-layer blockchain in the Internet of Vehicles involved in the embodiment of the present invention. Figure 1 As shown in the figure, the system model has three layers: the blockchain layer, which is used by the task publisher to register and publish learning content and related requirements, and record training results; the learning layer, where CAV performs local model training; and the CAV layer, which displays CAV following scenarios in different situations.

[0064] Each partition independently maintains a blockchain to record the training process of the personalized car-following model. Model sharing is carried out in the form of federated learning. CAV only uploads the training model parameters instead of local original data, thereby ensuring the data privacy of CAV users. The blockchain layer adopts a two-layer architecture of PBFT and DAG to achieve safe and efficient federated learning and trusted data migration. Among them, the upper layer is composed of the main blockchain based on DAG consensus, and the lower layer is composed of the partitioned blockchain based on PBFT consensus. CAV obtains a local model based on local data training. The selected CAV transmits the local model to the corresponding RSUs for aggregation. RSUs upload the partition model to the corresponding partition blockchain and execute PBFT consensus to record the partition model parameters. Then RSUs package these parameters into transactions and upload them to BSs. RSU will complete the block header information of the transaction, including version number, timestamp, random number, hash value, etc. Finally, it is uploaded to the main chain DAG by BSs. Each transaction in the main chain contains a partition model, which provides a secure aggregation method between partitions and realizes asynchronous cross-region verification.

[0065] 1. CAV selection strategy based on reputation value

[0066] During the driving process, CAV will continuously interact with the environment. The data transmission rate between CAV and RSUs will be affected by its mobility. The selection process is as follows: Figure 2 As shown. The core of the CAV selection strategy is to consider CAV performance from multiple angles and select high-quality CAVs to participate in aggregation to improve the speed and model accuracy of group reinforcement learning. The algorithm is divided into two parts: direct attribute evaluation and indirect attribute evaluation. In the direct attribute evaluation stage, RSUs will calculate the familiarity, freshness, and time-varying properties of CAVs to obtain the CAV reputation value and sort them. In the indirect attribute evaluation stage, considering that the models trained by the perception information of CAVs with closer positions have high similarity, the most important CAVs in each partition are selected to join the group learning to improve training efficiency.

[0067] In the direct attribute evaluation phase, the reputation value of CAVi at time t can be calculated as follows:

[0068]

[0069] Among them, ψ is the freshness adjustment weight, and its value is less than 1. Reputation value, average reward and average transmission rate of CAVi at time t respectively. It can be calculated by the following formula:

[0070]

[0071] Where N is the number of episodes experienced in a round of training, that is, the average reward of the last N episodes is calculated, T represents the current episode, and Represents the reward value obtained by CAVi in the mth round.

[0072] The channel transmission quality between CAV and RSU needs to be considered. Since the transmission rate between CAV and RSU changes with the movement of CAV, the average transmission rate within one round of training time is used for calculation. According to Shannon's formula, the instantaneous transmission rate between CAV node i and aggregation node RSU is as follows:

[0073]

[0074] Where B is the bandwidth of the communication channel, p i is the transmission power, and h i is the channel gain, N 0 is the additive white Gaussian noise power.

[0075] Channel gain refers to the total gain from the transmitter to the receiver, including the effects of antenna gain and free space propagation. Since channel gain is greatly affected by transmission distance, the ITU-R M.2135 path loss line-of-sight model is considered here, which is applicable to various scenarios, including cities, suburbs, and highways. LOS as follows:

[0076] PL LOS (d i )=22log 10 (d i )+28+20log 10 (f) (4)

[0077] where d i is the distance between CAV and aggregation node RSU (in meters), f is the frequency of the signal (in Hertz), and the channel gain h can be calculated i as follows:

[0078]

[0079] Due to d i As the CAV moves, it will continue to change. According to the linear motion model, it is assumed that the CAV train The CAV maintains uniform linear motion. This scenario can be equivalent to a plane coordinate system with the aggregation node as the origin. The CAV's following movement direction is parallel to the road direction. At this time, the relationship between the distance from the CAV to the aggregation node and the time is as follows:

[0080]

[0081] Among them, (x i0 ,y i0 ) represents the initial coordinate of CAVi, then the function of transmission rate changing with time can be expressed as:

[0082]

[0083] Finally, CAVi is trained in one round of time T train The average transmission rate can be expressed as:

[0084]

[0085]

[0086] After the above process, the average transmission rate of the CAV node in one round of training can be obtained. In the direct attribute evaluation stage, the RSU first obtains the average reward of the CAV and the reputation value of the previous moment, and then obtains the location and movement information of the CAV through its global perception ability, and learns the relevant parameters of the CAV communication capability through interaction with the CAV, calculates the average transmission rate of each CAV within its coverage, and finally obtains the reputation value of the CAV at the current moment.

[0087] In the indirect attribute evaluation stage, since some CAVs are close to each other, the models trained using perception information have high similarity. Therefore, the most important CAVs in each partition are selected to join the aggregation, which can further reduce the model transmission overhead. In this stage, a homogeneous area is introduced, which represents a range with a radius of c centered on CAVi. The value of c is determined according to the volume of traffic. The larger the volume, the smaller the value, and the smaller the volume, the larger the value. The CAVs within this range have a high similarity with the models trained by CAVi, so these CAVs no longer participate in model aggregation. The relative positions between CAVs change as they move. Suppose CAVi moves at a speed v for a period of time. i Maintaining uniform linear motion, the aggregation node RSU calculates the average distance matrix D of CAV at time t, as shown below:

[0088]

[0089] in represents the average distance between any two CAVi and j, which can be expressed as:

[0090]

[0091]

[0092] In the direct attribute evaluation stage, the reputation value of each CAV can be obtained. The aggregation node RSU sorts the reputation values ​​of all CAVs from high to low, and selects the top-ranked CAVs to enter the indirect attribute evaluation stage. In this stage, the CAV with the highest reputation value is first selected, and then the CAVs in the homogeneous area are excluded based on the homogeneous area and the distance matrix. Next, the CAV with the highest reputation value is selected from outside the homogeneous area, and a new homogeneous area is divided with this CAV as the center. RSU repeats this process until there are no more CAVs to be selected.

[0093] 2. Personalized car-following model based on DDPG

[0094] The model training mainly includes two steps: first, CAV shares the aggregate model through group learning; second, CAV uses DDPG to train a personalized car-following model based on local data. Group learning combines edge computing and blockchain technology, and realizes model sharing and distributed learning by aggregating on RSUs.

[0095] The CAV following DRL algorithm allows CAV to use artificial intelligence to fit the optimal strategy based on the exploration and experience between CAV and the environment when establishing a decision-making method. The personalized following process of CAV is modeled as an infinite state Markov decision process. The present invention uses the Deep Deterministic Policy Gradient (DDPG) algorithm to solve this high-dimensional continuous control problem. DDPG is a reinforcement learning algorithm based on the Actor-Critic architecture, which combines deep learning and policy gradient and is suitable for control tasks in continuous action space.

[0096] CAV obtains maximum rewards by interacting with the environment. The reward function of different driving styles can be used as a training signal to encourage or discourage the behavior of following. CAV uses the perception data during driving as the environment. The state space, action space and personalized reward function of CAV are defined as follows:

[0097] (1) State space:

[0098] At time t, the state perceived by the following CAVi is as follows:

[0099] s t =(v i ,v p ,Δv,d i,p ) (13)

[0100] Among them, v i and v p are the speeds of the following car CAVi and the front car CAVp, Δv is its relative speed, d i,p is the relative longitudinal distance between the two workshops.

[0101] (2) Action Space:

[0102] During the car-following process, the car-following CAVi mainly controls the acceleration of the CAV to ensure safe car-following driving. Therefore, the action space of CAVi at time t is:

[0103] a t ={a i (t)}(14)

[0104] Considering the comfort of passengers, the acceleration range is [-4,2]m / s2 .

[0105] (3) Reward Function

[0106] CAV responds to changes in the environment through its perception, decision-making and control capabilities. Different drivers have different relative following distances and following speeds under different ideal conditions, which is a relatively complex subjective judgment behavior. When the following distance is different from the driver's expected distance, the driver will choose to adjust the driving speed appropriately. When the front car cuts out of the current lane or other CAVs cut into the current lane, the following distance will increase or decrease. If the following distance is close at this time, the driver must take emergency braking measures to avoid collision and achieve safe driving; if the following distance is too far at this time, the driver will shorten the following distance to achieve efficient following and improve traffic efficiency. In this process, the driver's comfort cannot be ignored. Based on the above situation, three different styles of following reward functions are designed based on radical, moderate and conservative styles, which include driving safety reward function, following distance reward function, driving efficiency reward function and comfort reward function.

[0107] ① Driving safety reward function

[0108] Taking into account the safety of the car following process, you can set:

[0109] J safety = -δ[v i (t) 2 +a]||{Collsion} (15)

[0110] Among them, δ and a are weight parameters, and ||{Collsion} means that its value is 1 when a collision occurs and 0 in other cases.

[0111] ②Following distance reward function

[0112] Considering that in real life some drivers may like to maintain a larger following distance to ensure safety, while some drivers may prefer a smaller following distance to quickly pass through traffic flow, the following distance reward function is as follows:

[0113]

[0114] Where x is the current following distance, d ideal For the preferred following distance, 48m, 69m and 90m are selected to represent the preferred following distances of the three styles: aggressive, moderate and conservative.

[0115] ③ Driving efficiency reward function

[0116] Traffic efficiency is evaluated using the headway time value TH, which is defined as the time between the following vehicle CAVi and the leading vehicle CAVp reaching a specified point. A smaller TH value at a safe following distance represents a greater road capacity and a more aggressive driving style of the driver. The driving efficiency reward function is as follows:

[0117]

[0118] For the aggressive style, the reward function has a high reward when the TH value is small; for the moderate style, the reward function is sensitive to the TH value to a certain extent, but does not particularly emphasize smaller TH values.

[0119] ④ Comfort reward function

[0120] Considering the jerk value, which is the derivative of acceleration with respect to time, the smaller the jerk value, the higher the riding comfort. The normalization method is used to adjust the comfort value to [0,1]. The comfort reward function is as follows:

[0121]

[0122] Where jerk is the jerk value and 3600 is the square of the maximum allowable jerk value.

[0123] For the car-following task, the total reward function based on moderate, conservative, or aggressive driving styles can be expressed as a linear combination of reward functions:

[0124] J style =ω 1 J safety +ω 2 J ideal +ω 3 J effciency +ω 4 J comfort (19)

[0125] Among them J style is the total reward function corresponding to the three styles. 1 ,ω 2 ,ω 3 ,ω 4 is the weight coefficient of each reward function.

[0126] DDPG consists of a policy network Actor and an evaluation network Critic. Figure 3 As shown in the figure, two independent neural networks represent the Actor and Critic in DDPG. At time t, ActorNetwork transforms the state s t =s t =(v i ,v p ,Δv,di,p ) as input and outputs a continuous action a(t). The input of CriticNetwork is a state-action pair (s t ,a t ), whose output is a scalar Q value Q(s t ,a t ). To solve the instability problem of a single sub-network during the learning phase, DDPG divides Actor and Critic into two sub-networks with the same topology: OnlineNet and TargetNet. The former uses the latest network parameters and updates the Target network after a certain period of time, which improves the stability of the algorithm. The parameters of the Actor and Critic networks are determined by the gradient And minimize the loss function L to update.

[0127] The execution strategy is output by the Actor unit in the figure, and its loss function is:

[0128]

[0129] The Critic unit evaluates the output strategy of the Actor, and its corresponding loss function is expressed as follows:

[0130]

[0131] y k =r k +γQ'(s k+1 ,μ'(s k+1 |θ μ' )|θ Q' ) (twenty two)

[0132] Among them, θ μ ,θ μ' are the parameters of Online and Target in Actor, θ Q ,θ Q' are the network parameters of Online and Target in Critic, r k is the reward value at the current moment, n is the number of samples sampled each time, and γ is the discount factor.

[0133] As a deterministic strategy, DDPG performs well in learning efficiency, but lacks in exploration ability. By adding OU noise to the output actions of the Actor's Online network, the behavior of the Agent's actions has a certain degree of randomness, thereby enhancing its exploration ability.

[0134] 3. System Process

[0135] like Figure 4As shown in the figure, it is a flowchart of the execution of the personalized intelligent car-following method based on blockchain. The specific steps are as follows:

[0136] S401: System initialization;

[0137] S402: The task publisher publishes the car-following training task and the initial task, and uploads the initial model;

[0138] S403: CAV initiates identity authentication and obtains an initial model;

[0139] S404: The CAV generates a corresponding action decision according to the current environment state;

[0140] S405: The policy network executes the action and evaluates the reward based on the action decision generated in S404;

[0141] S406: CAV storage experience (s t ,a t ,r t ,s t+1 );

[0142] S407: After completing S404-S406, CAV performs the next round of model training and updates the policy model;

[0143] S408: The RSU of each partition calculates the reputation value of the CAV in the area and makes a selection;

[0144] S409: If CAV is selected, execute S410, otherwise execute S404;

[0145] S410: Aggregate the policy models of the selected CAVs to obtain partition models of each partition;

[0146] S411: RSUs send requests to the blockchain partition network and execute PBFT consensus, recording the partition aggregation parameters;

[0147] S412: RSUs packages the partition aggregation model parameters into a transaction and sends it to the partition BSs;

[0148] S413: Each partition BSs uploads the transaction to the main chain DAG to achieve asynchronous cross-zone verification;

[0149] S414: The task publisher checks whether the model meets the requirements. If so, S415 is executed; otherwise, S416 is executed.

[0150] S415: DAG publishes the updated global model to BSs and RSUs in each partition of the blockchain network;

[0151] S416: The algorithm ends.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A personalized intelligent following method based on double-layer blockchain in the Internet of Vehicles. Features: The method comprises the following steps: S1: Establish a group learning DRL architecture based on a two-layer blockchain; S2: CAV selection based on reputation value; S3: Establish a personalized car-following model based on DDPG.

2. According to claim 1, a personalized intelligent following method based on a double-layer blockchain in a connected vehicle network, Features: In S1, a three-layer system model is established, including a blockchain layer, a learning layer, and a connected and autonomous vehicles (CAV) layer; the blockchain layer adopts a double-layer chain architecture, the upper main chain adopts a main chain based on a directed acyclic graph (DAG) consensus, and the lower layer adopts a partitioned blockchain based on a PBFT consensus; based on the geographical location and base station (BS); the coverage range divides this scenario into multiple partitions, each partition consists of a BS and multiple road side units (RSUs), and each partition collaborates and processes consensus tasks in parallel; Model sharing is carried out in the form of federated learning; the CAV layer is the car following scenario in different situations; The RSUs and BS in the partition are connected via DSRC to form an edge computing network within the partition; The BSs between partitions are connected through a wired network to form a cross-region edge computing network; CAVs transmit data with BSs and RSUs through wireless links, and the entire process of model uploading and aggregation will be recorded in the partition blockchain; First, the task publisher registers and publishes the training task in the blockchain network, and CAV will perform personalized car-following model training based on DDPG. Second, the cloud sends the initial model to each partition blockchain. Finally, the BS of each partition will generate a genesis block for the task and the initial model, and add it to the corresponding partition PBFT subchain. All nodes train local models based on the genesis block at startup, process subsequent transactions through the consensus mechanism, and upload blocks. The method of uploading the model to the blockchain is as follows: CAV uploads its local model to the corresponding RSUs for aggregation based on the local model obtained in S3. RSUs sends a request to the blockchain partition network where it is located. Each subchain will execute PBFT consensus and record the partition aggregation parameters. Then RSUs package these parameters into transactions and upload them to BSs. RSU will complete the block header information of the transaction, including version number, timestamp, random number and hash value. Finally, BSs uploads it to the main chain DAG. Each transaction in the main chain network contains a partition-trained model, which provides a secure aggregation method between partitions and realizes asynchronous cross-region verification. The CAV training task termination method is as follows: the task publisher will request the BS for the current model update in real time and analyze the performance of the current model. When it has met the requirements, it will issue task termination information to each partition BS and RSUs, and the CAV training task will end; based on the car reputation value recorded in the RSUs, it will be rewarded or punished.

3. According to claim 1, a personalized intelligent following method based on a double-layer blockchain in a connected vehicle network, Features: In S2, during the continuous interaction between CAV and the environment, the performance of CAV will be considered from multiple perspectives, and high-quality CAVs will be selected to participate in aggregation. The selected CAV will send its trained and updated model to the corresponding RSU for aggregation; the CAV reputation value consists of four parts: familiarity, freshness, time-varying, and importance; Familiarity: replaced by average reward. The larger the average reward, the greater the familiarity. Freshness: The reputation of a CAV in the most recent round of training is more important than in the past; Time-varying: The communication quality between CAV and RSU changes, and the average transmission rate of CAV within a round of training time is calculated; Importance: Select the most important CAV in the RSU coverage area; The selected CAV packages the trained local new model into a transaction and sends it to the RSUs to obtain the updated partition model, and then aggregates the partition model to obtain the global model; among them, the transaction header includes the hash value of all selected transactions, and the transaction body includes the trained model.

4. According to claim 1, a personalized intelligent following method based on a double-layer blockchain in a connected vehicle network, Features: In S3, CAV initiates an identity request to the blockchain network through its built-in communication module, and uses the public key infrastructure PKI to verify the identity of CAV. The public key and private key pair of CAV are used for signature verification. First, CAV generates a request message containing a unique identifier and a timestamp, and signs the message with its private key. After receiving the request, the authentication node in the blockchain network verifies the authenticity of the signature using the public key of CAV. If the verification is successful, CAV is granted access rights. After obtaining the current initial model, CAV loads it into the local computing module. CAV performs personalized car-following model training based on DDPG to achieve high-performance car-following that is close to the driver's driving style. First, CAV collects current environmental information and records it as the current state. CAV policy network generates corresponding action decisions based on the current environment, executes actions and evaluates the action decisions with rewards. CAV stores relevant information about environmental interactions, actions, and rewards in the experience replay memory. When training is required, experience data is sampled in batches from the experience replay memory and the network parameters are updated using the experience data; During the training process, we first construct the driving information of the target CAV and surrounding CAVs required for the CAV decision-making model, and then design the DDPG reward function by comprehensively considering their driving style, driving efficiency, collision avoidance, and comfort factors to solve the problem of lack of safety assurance in the decision-making process.

5. According to claim 1, a personalized intelligent following method based on a double-layer blockchain in a connected vehicle network, Features: In S2, in the direct attribute evaluation stage, CAV i The reputation value at time t is calculated as follows: Among them, ψ is the freshness adjustment weight, and its value is less than 1. CAV i Reputation value, average reward and average transmission rate at time t; Calculated by the following formula: Where N is the number of rounds experienced in a round of training, that is, the average reward of the last N rounds is calculated, T represents the current round, and r i|t＝m Representing CAV i The reward value obtained in the mth round; The channel transmission quality between CAV and RSU needs to be considered; the transmission rate between CAV and RSU will change with the movement of CAV, and the average transmission rate within one round of training time is used for calculation; according to Shannon's formula, the instantaneous transmission rate between CAV node i and aggregation node RSU is as follows: Where B is the bandwidth of the communication channel, p i is the transmission power, and h i is the channel gain, N 0 is the additive white Gaussian noise power; Channel gain refers to the total gain from the transmitter to the receiver, including the effects of antenna gain and free space propagation. Considering the ITU-R M.2135 path loss line-of-sight model, it is applicable to various scenarios, including cities, suburbs, and highways. LOS as follows: PL LOS (d i )=22log 10 (d i )+28+20log 10 (f) (4) where d i is the distance between CAV and aggregation node RSU, in meters, f is the frequency of the signal, in Hertz, and the channel gain h is calculated. i as follows: d i As the movement of CAV keeps changing, according to the linear motion model, suppose CAV train The CAV maintains uniform linear motion. This scenario is equivalent to a plane coordinate system with the aggregation node as the origin. The CAV's following movement direction is parallel to the road direction. At this time, the relationship between the distance from the CAV to the aggregation node and the time is as follows: Among them, (x i0 ,y i0 ) represents the initial coordinate of CAVi, then the function of transmission rate changing with time is expressed as: Finally, CAVi is trained in one round of time T train The average transmission rate is expressed as: The average transmission rate of the CAV node in a round of training is obtained; in the direct attribute evaluation stage, the RSU first obtains the average reward of the CAV and the reputation value of the previous moment, and then obtains the location and movement information of the CAV through its global perception ability, and learns the relevant parameters of the CAV communication capability through interaction with the CAV, calculates the average transmission rate of each CAV within its coverage, and finally obtains the reputation value of the CAV at the current moment; In the indirect attribute evaluation stage, the most important CAV in each partition is selected to be aggregated to reduce the model transmission overhead; in this stage, homogeneous regions are introduced, which represent CAV i is the range with a center radius of c. The value of c is determined by the traffic volume. The larger the traffic volume, the smaller the value. The smaller the traffic volume, the larger the value. CAVs within this range do not participate in model aggregation. The relative positions of CAVs change as they move. Let CAV i At a speed v over a period of time i Maintaining uniform linear motion, the aggregation node RSU calculates the average distance matrix D of CAV at time t, as shown below: in Represents any two CAVs i The average distance between and j is expressed as: In the direct attribute evaluation stage, the reputation value of each CAV is obtained. The aggregation node RSU sorts the reputation values ​​of all CAVs from high to low, and selects the top-ranked CAVs to enter the indirect attribute evaluation stage. In this stage, the CAV with the highest reputation value is first selected, and then the CAVs in the homogeneous area are excluded based on the homogeneous area and the distance matrix. Then, the CAV with the highest reputation value is selected from outside the homogeneous area, and a new homogeneous area is divided with this CAV as the center. RSU repeats this process until there are no more CAVs to be selected.

6. According to claim 5, a personalized intelligent following method based on a double-layer blockchain in a connected vehicle network, Features: In S3, model training includes two steps: first, CAV shares the aggregate model through group learning; Second, CAV uses DDPG to train a personalized car-following model based on local data; Group learning combines edge computing and blockchain technologies to achieve model sharing and distributed learning by aggregating on RSUs; The CAV car-following DRL algorithm allows CAV to use artificial intelligence to fit the optimal strategy based on the exploration and experience between CAV and the environment when establishing a decision-making method; the personalized car-following process of CAV is modeled as an infinite-state Markov decision process, and the Deep Deterministic Policy Gradient (DDPG) algorithm is used to solve this high-dimensional continuous control problem; DDPG is a reinforcement learning algorithm based on the Actor-Critic architecture, which combines deep learning and policy gradient and is suitable for control tasks in continuous action space; CAV obtains the maximum reward by interacting with the environment. The reward function of different driving styles is used as a training signal to encourage or discourage the behavior of following the car. CAV uses the perception data during driving as the environment. The state space, action space and personalized reward function of CAV are defined as follows: (1) State space At time t, the state perceived by the following CAVi is as follows: s t =(v i ,v p ,Δv,d i,p ) (13) Among them, v i and v p are the speeds of the following car CAVi and the front car CAVp, Δv is its relative speed, d i,p is the relative longitudinal distance between the two workshops; (2) Action Space During the car-following process, the car-following CAVi controls the acceleration of the CAV. The action space of CAVi at time t is: a t ={a i (t)}(14) Considering the comfort of passengers, the acceleration range is [-4,2]m / s 2 ; (3) Reward Function Design three different styles of following reward functions: radical, moderate, and conservative, including driving safety reward function, following distance reward function, driving efficiency reward function, and comfort reward function; ① Driving safety reward function Taking into account the safety of the car-following process, the following settings are: J safety =-δ[v i (t) 2 +a]||{Collision} (15) Among them, δ and a are weight parameters, ||{Collsion} means that its value is 1 when a collision occurs and 0 in other cases; ②Following distance reward function The following distance reward function is as follows: Where x is the current following distance, d ideal For the preferred following distance, 48m, 69m, and 90m are selected to represent the preferred following distances of the three styles: aggressive, moderate, and conservative. ③ Driving efficiency reward function The headway time value TH is used to evaluate traffic efficiency. TH is defined as the time between the following vehicle CAVi and the front vehicle CAVp reaching the specified point. The driving efficiency reward function is as follows: For the aggressive style, the reward function has a high reward when the TH value is small; for the moderate style, the reward function is sensitive to the TH value; ④ Comfort reward function The normalization method is used to adjust the comfort value to [0,1]. The comfort reward function is as follows: Where jerk is the jerk value, and 3600 is the square of the maximum allowable jerk value; For the car-following task, the total reward function based on moderate, conservative, or aggressive driving style is expressed as a linear combination of reward functions: J style =ω 1 J safety +oh 2 J ideal +oh 3 J effciency +oh 4 J comfort (19) Among them J style is the total reward function corresponding to the three styles; ω 1 ,ω 2 ,ω 3 ,ω 4 is the weight coefficient of each reward function; DDPG consists of a policy network Actor and an evaluation network Critic; two independent neural networks represent the Actor and Critic in DDPG respectively; at time t, ActorNetwork transforms the state s t =s t =(v i ,v p ,Δv,d i,p ) as input and outputs a continuous action a(t); the input of CriticNetwork is a state-action pair (s t ,a t ), whose output is a scalar Q value Q(s t ,a t ) ; DDPG divides Actor and Critic into two sub-networks with the same topology: OnlineNet and TargetNet. The former uses the latest network parameters and updates the Target network after a certain period of time; the parameters of the Actor and Critic networks are determined by the gradient And minimize the loss function L update; The execution strategy is output by the Actor unit, and its loss function is: The Critic unit evaluates the output strategy of the Actor, and its corresponding loss function is expressed as follows: y k =r k +γQ'(s k+1 ,m'(s k+1 |θ μ’ )|θ Q’ ) (22) Among them, θ μ ,θ μ’ are the parameters of Online and Target in Actor, θ Q ,θ Q’ are the network parameters of Online and Target in Critic, r k is the reward value at the current moment, n is the number of samples sampled each time, and γ is the discount factor.

Images