An individualized intelligent following method based on double-layer blockchains in a vehicle networking

Abstract

The application relates to a personalized intelligent following method based on a double-layer blockchain in a vehicle networking, and belongs to the technical field of mobile communication. At present, deep reinforcement learning (DRL) can enable a connected autonomous vehicle (CAV) to learn and optimize a strategy by continuously interacting with an environment, so as to improve the adaptability and flexibility of system decision-making. However, the complexity of the current traffic environment challenges the ability of a single vehicle to accurately respond to changes in the environment. Federated learning can aggregate multiple learning models in different environments, thereby improving the adaptability of CAVs to complex environments. The application establishes a group learning DRL architecture based on a double-layer blockchain in a vehicle networking scenario, proposes a CAV selection strategy based on reputation values based on the mobility of CAVs, and further proposes a personalized following model based on a deterministic policy gradient algorithm. The method can realize personalized safe and intelligent following driving, can utilize edge network and CAV user data resources, and can promote the application of artificial intelligence in a vehicle networking scenario.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of mobile communication, and relates to a personalized intelligent following method based on a double-layer blockchain in a vehicle Internet of Things. BACKGROUND

[0002] With the development of intelligent transportation systems and vehicle Internet of Things technologies, autonomous driving and intelligent driving assistance systems have become a hot research direction in the modern automobile industry. The vehicle Internet of Things (Internet of Vehicles, IoV) connects CAVs, infrastructure and networks to realize real-time sharing and intelligent scheduling of information, improving traffic efficiency and driving safety. However, traditional vehicle Internet of Things systems face challenges in handling personalized driving needs and CAV following control, such as data privacy, security and communication efficiency.

[0003] Federated learning (FL) is a distributed machine learning method that aims to train models on multiple devices or nodes without uploading data to a server. Since data does not need to leave the local device, federated learning significantly improves data privacy and security. In addition, in federated learning, only model parameters are transmitted instead of raw parameters, reducing the demand for communication bandwidth. This method is suitable for scenarios with high requirements for data privacy and security, but it is vulnerable to reverse inference attacks on model parameters, reducing the security of the transmission process.

[0004] In this context, blockchain technology, with its characteristics of decentralization, tamper resistance and transparency, can be combined with federated learning to provide system-level tamper resistance and traceability. A vehicle Internet of Things system based on blockchain can achieve automated management through smart contracts and ensure data authenticity and security through consensus mechanisms, effectively solving the single-point failure and data leakage problems of traditional centralized systems.

[0005] Deep reinforcement learning (DRL) is an important branch of machine learning, in which an agent can autonomously learn the optimal policy through interaction with the environment, and is widely used in robot control, game play and intelligent driving. Applying DRL to CAV following control can achieve adaptive adjustment of complex dynamic environments and improve driving safety and comfort. However, the local data and computing resources of a single CAV are limited, making it difficult to fully train high-quality reinforcement learning models.

[0006] To solve the above problems, the application designs a personalized intelligent following method based on double-layer blockchain. This method combines blockchain and DRL technology, realizes the collaborative learning and model sharing between CAVs through double-layer blockchain network, and uses group wisdom to improve the following control performance of individual CAV while protecting the privacy of CAV users. Specifically, the task publisher registers and publishes training tasks in the blockchain network, the CAV performs preliminary model training locally, and updates and shares the model through the blockchain network, so as to realize efficient personalized following control under the premise of ensuring data privacy and security. The application not only improves the intelligent and adaptive ability of the Internet of Vehicles system, but also provides a safe and reliable solution, which is expected to be widely used in future intelligent transportation systems. SUMMARY

[0007] Therefore, the application aims to provide a personalized intelligent following method based on double-layer blockchain in the Internet of Vehicles, which ensures the safety and privacy of Internet of Vehicles data through the distributed ledger of blockchain technology, and at the same time, trains personalized following models in a distributed environment using group reinforcement learning to realize efficient collaboration and following between CAVs.

[0008] To achieve the above purpose, the application provides the following technical solutions:

[0009] In the first aspect, the application provides a personalized intelligent following method based on double-layer blockchain according to the personalized following intelligent driving scene in the Internet of Vehicles and the data privacy protection requirements of CAV users, and the execution process of the method is as follows:

[0010] S1: Group learning DRL architecture based on double-layer blockchain;

[0011] S2: CAV selection strategy based on reputation value;

[0012] S3: Personalized following model based on DDPG;

[0013] In the second aspect, the application establishes a three-layer system model in S1, which includes: 1) blockchain layer, used for task publisher registration and publication of learning content and related requirements, and recording of training results; 2) learning layer, CAV performs local model training; 3) CAV layer, showing the following scene of CAV in different situations.

[0014] Road Side Units (RSUs) and Base Stations (BSs) are the key communication devices in the Internet of Vehicles, responsible for short-range roadside communication and wide-area cellular network communication, respectively. They work together to provide efficient, secure, and reliable Internet of Vehicles communication services and promote the development and application of intelligent transportation systems. In the present invention, RSUs and CAVs are divided into multiple partitions based on geographic location and BS coverage, and each partition cooperates and processes consensus tasks in parallel. Each partition independently maintains a blockchain to record the personalized following model training process, and model sharing is performed in the form of federated learning. CAVs only upload training model parameters rather than local raw data, thereby ensuring the data privacy of CAV users.

[0015] A double-layer blockchain structure is adopted at the blockchain layer, including an upper layer main chain based on DAG and a lower layer partitioned blockchain based on PBFT. Among them, each sub-chain deployed in the lower layer partition is independent and responsible for coordinating CAVs in the partition to complete the training task. PBFT consensus is used in the sub-chain, which is maintained by multiple RSUs in the partition and is responsible for transaction verification (including identity verification and model verification) and model aggregation process within the partition. Through partitioning, the number of parallel transaction processing can be reduced, effectively alleviating the throughput limitation of the blockchain. The upper layer main chain adopts an asynchronous consensus mechanism based on DAG. Each transaction in the main chain contains a model trained by a partition, providing a secure aggregation method between different partitions and realizing asynchronous cross-zone verification of the model. The DAG chain is maintained by all BSs in the partition. Unlike the traditional chain structure of the blockchain, DAG allows multiple transactions to be added and verified simultaneously due to its parallel structure and acyclic nature, thereby realizing asynchronous processing.

[0016] RSUs and BSs within a partition are connected through DSRC to form an edge computing network within the partition. BSs between partitions are connected through a wired network to form a cross-zone edge computing network. CAVs, BSs, and RSUs are connected through wireless links for data transmission, and the entire process of model uploading and aggregation is recorded in the partitioned blockchain.

[0017] First, the task publisher registers and publishes the training task in the blockchain network, and the CAVs will train personalized following models based on DDPG. Second, the cloud server distributes the initial model to each partitioned blockchain. Finally, each BS in the partition generates a genesis block for the task and the initial model and adds it to the corresponding PBFT sub-chain. All nodes train local models based on the genesis block at startup and process subsequent transactions through the consensus mechanism and upload the blocks.

[0018] Thirdly, in S1, this embodiment of the invention provides a method for uploading a model to the blockchain. Based on the local model obtained in S3, the selected CAV uploads its local model to the corresponding RSUs for aggregation. The RSUs send requests to the blockchain partition network where they reside. Each sub-chain executes PBFT consensus and records the partition aggregation parameters. Then, the RSUs package these parameters into a transaction and upload it to the BSs. The RSUs complete the block header information of the transaction, including version number, timestamp, random number, hash value, etc. Finally, the BSs uploads it to the main chain DAG. Each transaction in the main chain network contains a partition-trained model, providing a secure aggregation method between partitions and enabling asynchronous cross-partition verification.

[0019] Fourthly, in embodiment S1 of this invention, a method for terminating a CAV training task is provided. The task issuer requests the current model update status from the BS in real time and analyzes the performance of the current model. When the requirements are met, task termination information is issued to each partition BS and RSUs, and the CAV training task ends. Based on the car reputation values ​​recorded in the RSUs, rewards or penalties are applied. Furthermore, the training task can be carried out over a long period, allowing it to continuously adapt to dynamic changes in the environment.

[0020] Fifthly, in S2, this embodiment of the invention provides a reputation-based CAV selection strategy. As the CAV continuously interacts with the environment, its performance is considered from multiple perspectives, and high-quality CAVs are selected for aggregation. The selected CAVs send their trained and updated models to their corresponding RSUs for aggregation. The CAV reputation value consists of four parts: familiarity, novelty, time-varying nature, and importance.

[0021] Therefore, the reputation value has four parts:

[0022] 1. Familiarity: Since DRL cannot provide specific model accuracy, this accuracy is replaced by average reward. The higher the average reward, the greater the familiarity.

[0023] 2. Freshness: Since the training process within each partition will be performed in multiple rounds, the reputation value of CAV in the most recent training round is more important than in the past.

[0024] 3. Time-varying nature: The communication quality variation between CAV and RSU, calculating the average transmission rate of CAV during one training round;

[0025] 4. Importance: Since some CAVs are close to the RSU location, the models trained by their perceived information have high similarity. Select the most important CAVs within the RSU coverage area.

[0026] The selected CAV packs the trained local new model into a transaction and sends it to the RSU to obtain an updated partition model, and then aggregates the partition models to obtain a global model. The transaction header includes the hash value of all transactions selected by it, and the transaction body includes the model obtained by training.

[0027] In the sixth aspect, in S3, the CAV initiates an identity request to the blockchain network through its built-in communication module, verifies the identity of the CAV using the public key infrastructure (PKI), and uses the public and private key pair of the CAV for signature verification. First, the CAV generates a request message containing a unique identifier and a timestamp, and signs the message using its private key. After receiving the request, the authentication node in the blockchain network verifies the authenticity of the signature using the public key of the CAV. Once the verification is passed, the CAV will be granted access. After obtaining the current initial model, the CAV loads it into the local computing module. The CAV performs DDPG-based personalized following model training to achieve high-performance following close to the driving style of the driver. First, the CAV collects the current environmental information and records it as the current state; the CAV policy network generates corresponding action decisions based on the current environment, executes the actions and evaluates the action decisions; the CAV stores the relevant information of environmental interaction, action and reward in the experience replay memory; when training is needed, the experience data is sampled in batches from the experience replay memory, and the network parameters are updated using the experience data.

[0028] During the training process, first, the driving information of the target CAV and the surrounding CAVs required for the CAV decision model is constructed, and the driving style, driving efficiency, collision avoidance, comfort and other factors are considered to design the reward function of DDPG, so as to solve the problem of lack of safety guarantee in the decision process.

[0029] The beneficial effects of the present application are:

[0030] (1) Safety and privacy protection

[0031] Double-layer blockchain architecture: lower partition blockchain: based on PBFT consensus, ensuring data safety and consistency within the partition, preventing malicious node attacks. Upper DAG main chain: based on DAG consensus, realizing asynchronous cross-zone verification, ensuring the safety of model aggregation, and preventing model tampering.

[0032] The CAV only uploads the training model parameters instead of the original data, protecting user privacy and preventing data leakage.

[0033] No central server is needed, reducing the risk of single point of failure and improving system reliability.

[0034] (2) Intelligence and personalization

[0035] Personalized Following Model: Based on the DDPG algorithm, a personalized following model is trained according to different driving styles to improve driving safety and comfort and meet the needs of different drivers.

[0036] Collective Intelligence: CAVs learn and share training models with each other to improve individual following control performance and quickly adapt to complex traffic environments.

[0037] (3) Efficiency and Reliability

[0038] CAVs only upload training model parameters instead of raw data, reducing communication bandwidth requirements and improving network efficiency. RSUs and BSs form an edge computing network to achieve efficient model training and aggregation, reducing the burden on the central server and improving system response speed.

[0039] Each partition maintains a separate blockchain, reducing the number of parallel transaction processing and alleviating the throughput limitations of the blockchain, improving system efficiency. Each partition can cooperatively process consensus tasks in parallel to improve system throughput.

[0040] (4) Dynamic Adaptability

[0041] The model can be trained for a long time to adapt to changes in the environment, such as traffic flow and road conditions, improving system adaptability.

[0042] Reputation-based CAV selection strategy: Select high-quality CAVs for training, such as familiarity, freshness, time-varying, importance, etc., to improve training efficiency and model accuracy and improve system adaptability.

[0043] (5) Safe Driving

[0044] Encourage safe driving behaviors such as maintaining a safe distance and avoiding collisions to reduce the risk of collisions and improve driving safety.

[0045] Encourage maintaining appropriate following distances, such as aggressive, moderate, and conservative, to improve driving safety.

[0046] (6) Traffic Efficiency

[0047] Encourage efficient driving behaviors such as shortening following distances and improving headway to improve traffic efficiency.

[0048] Improve traffic flow smoothness and stability, reduce traffic congestion, and improve traffic efficiency.

[0049] (7) Passenger Comfort

[0050] Encourage smooth driving, such as controlling acceleration and jerk, to improve passenger comfort.

[0051] (8) System Scalability

[0052] The system adopts a layered architecture, is easy to expand, for example, increase the partition, increase the number of CAV, etc.

[0053] The application realizes a safe, efficient, intelligent and personalized intelligent following method for Internet of Vehicles by combining blockchain, federated learning and deep reinforcement learning technology, has significant technical advantages and application value, and is expected to promote the development and application of intelligent transportation system.

[0054] Other advantages, objects, and features of the application will be set forth in part by the description that follows, and in part will become apparent to those skilled in the art upon examination of the following specification or can be learned by practice of the application. The objects and other advantages of the application can be realized and attained by the methods and compositions particularly pointed out in the specification. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to make the objects, technical solutions and advantages of the application clearer, the preferred detailed description of the application will be combined with the drawings to describe the application, and the drawings are as follows:

[0056] Figure 1 It is a personalized intelligent following model diagram based on double-layer blockchain;

[0057] Figure 2 It is a CAV selection flowchart based on reputation value;

[0058] Figure 3 It is a personalized intelligent following framework diagram based on DDPG;

[0059] Figure 4 It is a personalized intelligent following method execution flowchart based on double-layer blockchain. DETAILED DESCRIPTION

[0060] The embodiments of the application are described below through specific concrete examples, and those skilled in the art can easily understand other advantages and effects of the application from the content disclosed in the specification. The application can also be implemented or applied by different specific embodiments, and the details in the specification can be modified or changed based on different views and applications without departing from the spirit of the application. It should be noted that the diagrams provided in the following examples only illustrate the basic concept of the application in a schematic manner, and the following examples and features in the examples can be combined with each other without conflict.

[0061] The drawings are only used for exemplary illustration, and the representation is only a schematic diagram, not a physical diagram, and cannot be understood as a limitation on the present application; in order to better illustrate the embodiments of the present application, some components of the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0062] The same or similar reference numerals in the drawings of the embodiments of the present application correspond to the same or similar components; in the description of the present application, it should be understood that if the terms "upper", "lower", "left", "right", "front", "back" and the like indicate the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the terms describing the positional relationship in the drawings are only used for exemplary illustration, and cannot be understood as a limitation on the present application, for those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0063] Figure 1 A personalized intelligent following model based on a double-layer blockchain in a vehicle Internet of Things is shown. As shown in the figure, the system model has three layers: a blockchain layer, used for task publisher registration and publishing of learning content and related requirements, recording of training results; a learning layer, in which CAVs perform local model training; and a CAV layer, showing the following scenarios of CAVs in different situations. Figure 1

[0064] Each partition independently maintains a blockchain to record the personalized following model training process, and model sharing is performed in the form of federated learning, and CAVs only upload training model parameters rather than local raw data, thereby ensuring the data privacy of CAV users. The blockchain layer adopts a PBFT and DAG double-layer architecture to achieve safe and efficient federated learning and data trusted migration. The upper layer is composed of a main blockchain based on DAG consensus, and the lower layer is composed of a partitioned blockchain based on PBFT consensus. CAVs train local models based on local data, and selected CAVs transmit the local models to corresponding RSUs for aggregation, RSUs upload the partitioned models to the corresponding partitioned blockchains and perform PBFT consensus, and record the partitioned model parameters. Then the RSUs package these parameters into transactions and upload them to the BSs, and the RSUs complete the block header information of the transactions, including version number, timestamp, random number, hash value, etc. Finally, the BSs upload them to the main chain DAG. Each transaction in the main chain contains a partitioned model, providing a secure aggregation method between partitions and realizing asynchronous cross-zone verification.

[0065] 1. CAV selection strategy based on reputation value

[0066] ​The CAVs will interact with the environment during the driving process, and the data transmission rate between the CAVs and the RSUs will be affected by the mobility of the CAVs, and the selection process is as shown in Figure 2 The core of the CAV selection strategy is to consider the performance of the CAVs from multiple perspectives and select high-quality CAVs to participate in aggregation to improve the speed and model accuracy of group reinforcement learning. The algorithm is divided into direct attribute evaluation and indirect attribute evaluation. In the direct attribute evaluation stage, the RSUs will calculate the familiarity, freshness and time-varying of the CAVs to obtain the reputation value of the CAVs and sort them. In the indirect attribute evaluation stage, considering that the CAVs with close positions have high similarity in the models trained by their perception information, the most important CAVs in each partition are selected to join the group learning to improve the training efficiency.

[0067] In the direct attribute evaluation stage, the reputation value of CAVi at time t can be calculated by the following formula:

[0068]

[0069] where ψ is the freshness adjustment weight, and its value is less than 1, The reputation value, average reward and average transmission rate of CAVi at time t are calculated by the following formula:

[0070]

[0071] where N is the number of episodes experienced in a round of training, i.e. the average reward of the last N episodes is calculated, T represents the current episode, and represents the reward value obtained by CAVi in the mth episode.

[0072] The channel transmission quality between the CAV and the RSU needs to be considered. Since the transmission rate between the CAV and the RSU will change with the movement of the CAV, the average transmission rate during a round of training is used for calculation. According to the Shannon formula, the instantaneous transmission rate between CAV node i and aggregation node RSU can be obtained as follows:

[0073]

[0074] where B is the bandwidth of the communication channel, p i is the transmission power, and h i is the channel gain, and N0 is the additive white Gaussian noise power.

[0075] ​Channel gain refers to the total gain from the transmitting end to the receiving end, including the antenna gain and the influence of free space propagation. Since the channel gain is greatly affected by the transmission distance, the ITU-R M.2135 path loss line-of-sight model is considered here, which is suitable for various scenarios, including urban, suburban and highway. PL LOS As follows:

[0076] PL LOS (d i )=22log 10 (d i )+28+20log 10 (f) (4)

[0077] Where d i is the distance between CAV and aggregation node RSU (unit: meter), f is the frequency of the signal (unit: hertz), and the channel gain h i can be calculated as follows:

[0078]

[0079] Since d i will change as the CAV moves, according to the straight line motion model, it is assumed that the CAV maintains a uniform straight line motion within a round of training time T train , this scenario can be equivalent to a plane coordinate system with the aggregation node as the origin, and the following relationship between the distance and time of CAV to the aggregation node is obtained:

[0080]

[0081] Where (x i0 , y i0 ) represents the initial coordinates of CAVi, and the function of the transmission rate changing with time can be expressed as:

[0082]

[0083] Finally, the average transmission rate of CAVi within a round of training time T train can be expressed as:

[0084]

[0085]

[0086] After the above process, the average transmission rate of CAVs in a round of training can be obtained. In the direct attribute evaluation stage, the RSU first obtains the average reward of the CAV and the reputation value at the previous moment, and then obtains the position and movement information of the CAV through its global perception ability, and obtains the relevant parameters of the communication ability of the CAV through interaction with the CAV, calculates the average transmission rate of each CAV in its coverage range, and finally obtains the reputation value of the CAV at the current moment.

[0087] In the indirect attribute evaluation stage, since some CAVs are relatively close in position, the model trained by using the perception information has high similarity, so the most important CAV in each partition is selected to join the aggregation, which can further reduce the model transmission overhead. In this stage, a homogenization region is introduced, which represents a range with CAVi as the center and c as the radius. The value of c is determined according to the traffic flow, and the smaller the traffic flow, the smaller the value of c, and the larger the traffic flow, the larger the value of c. The CAVs in this range have high similarity with the model trained by CAVi, so these CAVs no longer participate in model aggregation. The relative positions between CAVs change with their movement. Assuming that CAVi moves at a speed of v i Maintaining a uniform linear motion, the RSU calculates the average distance matrix D of the CAV at time t, as follows:

[0088]

[0089] wherein represents the average distance between any two CAVs i and j, which can be represented as:

[0090]

[0091]

[0092] In the direct attribute evaluation stage, the reputation value of each CAV can be obtained, and the RSU sorts all the reputation values of the CAVs from high to low, and selects the CAVs with high ranking to enter the indirect attribute evaluation stage. In this stage, the CAV with the highest reputation value is first selected, and then the CAVs located in the homogenization region are excluded according to the homogenization region and the distance matrix. Then, the CAV with the highest reputation value is selected from outside the homogenization region, and a new homogenization region is divided with this CAV as the center. The RSU repeatedly performs this process until there are no more CAVs to be selected.

[0093] 2. Individualized following model based on DDPG

[0094] The model training mainly contains two steps: first, CAVs share the aggregated model through swarm learning; second, CAVs train the individualized following model based on local data using DDPG. Swarm learning combines edge computing and blockchain technology together, and realizes model sharing and distributed learning through aggregation on RSUs.

[0095] The CAV following DRL algorithm allows CAVs to use artificial intelligence to fit the optimal strategy based on the exploration and experience between CAVs and the environment when establishing a decision-making method. The individualized following process of CAVs is modeled as an infinite state Markov decision process, and the deep deterministic policy gradient algorithm (DDPG) is used to solve this high-dimensional continuous control problem. DDPG is a reinforcement learning algorithm based on the Actor-Critic architecture, which combines deep learning and policy gradient and is suitable for control tasks in continuous action space.

[0096] CAVs obtain the maximum reward by interacting with the environment, and the reward function of different driving styles can be used as a kind of training signal to encourage or prevent behaviors in the following action. The CAVs take the perception data in driving as the environment, and the state space, action space and individualized reward function of the CAVs are defined as follows:

[0097] (1) State space:

[0098] At time t, the state perceived by the following CAVi is as follows:

[0099] s t =(v i ,v p ,Δv,d i,p ) (13)

[0100] Where v i and v p are the speeds of the following CAVi and the preceding CAVp, Δv is the relative speed, and d i,p is the relative longitudinal distance between the two vehicles.

[0101] (2) Action space:

[0102] In the following process, the following CAVi mainly controls the acceleration of the CAV to ensure safe following driving. Therefore, the action space of CAVi at time t is:

[0103] a t ={a i (t)}(14)

[0104] Considering the comfort of passengers, the acceleration selection range is [-4, 2] m / s2 .

[0105] (3) Reward function

[0106] CAVs respond to changes in the environment through their perception decisions and control capabilities. Different drivers have different ideal state relative following distance and following speed, which is a more complex subjective judgment behavior. When the following distance is different from the expected distance of the driver, the driver will choose to adjust the driving speed appropriately. When the current car cuts out the current lane or other CAVs cut into the current lane, the following distance will increase or decrease. If the following distance is too close at this time, the driver must take emergency braking measures to avoid collision to achieve safe driving; if the following distance is too far at this time, the driver will shorten the following distance to achieve efficient following and improve traffic efficiency. In this process, the comfort of the driver cannot be ignored. Based on the above situation, three different styles of following reward functions based on aggressive, moderate and conservative are designed, which include driving safety reward function, following distance reward function, driving efficiency reward function and comfort reward function.

[0107] ① Driving safety reward function

[0108] Following considers the safety in the following process, which can be set as:

[0109] J safety = - δ [v i (t) 2 + a]||{Collision} (15)

[0110] Where δ and a are weight parameters, and ||{Collision} represents 1 when collision occurs, and 0 in other cases.

[0111] ② Following distance reward function

[0112] Considering that some drivers in real life may prefer to maintain a larger following distance to ensure safety, while some drivers may prefer a smaller following distance to quickly pass through the traffic flow, the following distance reward function is as follows:

[0113]

[0114] Where x is the current following distance, d ideal is the preferred following distance, and 48m, 69m, 90m are selected to represent the preferred following distance of aggressive, moderate and conservative styles.

[0115] ③ Driving efficiency reward function

[0116] The traffic efficiency is evaluated using the headway value TH, which is defined as the time elapsed between the follower CAVi and the leader CAVp reaching a given point. A smaller TH value at a safe following distance represents a greater road capacity and a more aggressive driving style. The driving efficiency reward function is given by:

[0117]

[0118] For the aggressive style, the reward function has a high reward for small TH values. For the moderate style, the reward function has some sensitivity to TH values, but does not emphasize small TH values.

[0119] (4) Comfort reward function

[0120] The jerk value, which is the derivative of acceleration with respect to time, is considered. A smaller jerk value indicates a higher comfort level for the passengers. The jerk value is normalized to [0, 1] and the comfort reward function is given by:

[0121]

[0122] where jerk is the jerk value and 3600 is the square of the maximum allowable jerk value.

[0123] For the following task, the total reward function based on the moderate, conservative, or aggressive driving style can be represented as a linear combination of the reward functions:

[0124] J style = ω1J safety + ω2J ideal + ω3J effciency + ω4J comfort (19)

[0125] where J style is the total reward function corresponding to the three styles. ω1, ω2, ω3, and ω4 are the weight coefficients of each reward function.

[0126] DDPG consists of a policy network Actor and a critic network Critic. As shown in Figure 3 , two independent neural networks represent the Actor and Critic in DDPG. At time t, the Actor Network takes the state s t = s t = (v i , v p , Δv, d i,p ) as input and outputs a continuous action a(t). The input of the Critic Network is a state-action pair (s t , a tIts output is a scalar Q-value Q(s) t ,a t To address the instability of a single subnetwork during the learning phase, DDPG divides the Actor and Critic networks into two subnetworks with identical topologies: OnlineNet and TargetNet. OnlineNet uses the latest network parameters, while TargetNet is updated periodically, thus improving the algorithm's stability. The parameters of the Actor and Critic networks are determined by gradients. Update by minimizing the loss function L.

[0127] The execution strategy is output by the Actor unit in the diagram, and its loss function is:

[0128]

[0129] The Critic unit evaluates the Actor's output strategy, and its corresponding loss function is expressed as follows:

[0130]

[0131] y k =r k +γQ'(s k+1 ,μ'(s k+1 |θ μ' )|θ Q' ) (twenty two)

[0132] Where, θ μ θ μ' θ represents the parameters of Online and Target in the Actor, respectively. Q θ Q' These are the network parameters for Online and Target in Critic, respectively. k γ is the reward value at the current moment, n is the number of samples taken each time, and γ is the discount factor.

[0133] As a deterministic strategy, DDPG performs well in learning efficiency, but it lacks in exploration ability. By adding OU noise to the output actions of the Actor's Online network, the behavior of the Agent is made more random, thereby enhancing its exploration ability.

[0134] 3. System Flow

[0135] like Figure 4 The diagram shows the execution flowchart of the blockchain-based personalized intelligent car-following method. The specific steps are as follows:

[0136] S401: System initialization;

[0137] S402: The task publisher publishes the following-the-vehicle training task and the initial task, and uploads the initial model;

[0138] S403: The CAV initiates identity verification and obtains the initial model;

[0139] S404: The CAV generates a corresponding action decision according to the current environment state;

[0140] S405: The policy network executes the action decision generated in S404 and evaluates the action decision;

[0141] S406: The CAV stores the experience (s t ,a t ,r t ,s t+1 ) in the experience database;

[0142] S407: After S404-S406 are completed, the CAV performs the next round of model training and updates the policy model;

[0143] S408: The RSU of each subzone calculates the reputation value of the CAV in the subzone and performs selection;

[0144] S409: If the CAV is selected, S410 is executed, otherwise S404 is executed;

[0145] S410: The policy model of the selected CAV is aggregated to obtain a subzone model of each subzone;

[0146] S411: The RSU sends a request to the blockchain subzone network and performs PBFT consensus to record the subzone aggregation parameters;

[0147] S412: The RSU packs the subzone aggregation model parameters into a transaction and uploads them to the subzone BSs;

[0148] S413: Each subzone BS uploads the transaction to the main chain DAG to realize asynchronous cross-zone verification;

[0149] S414: The task publisher checks whether the model meets the requirements, and if so, S415 is executed, otherwise S416 is executed;

[0150] S415: The DAG publishes the updated global model to the BSs and RSUs of each subzone of the blockchain network;

[0151] S416: The algorithm ends.

[0152] Finally, it is to be explained that the above embodiments are only used to illustrate the technical solutions of the present application but not to limit the present application. Although the present application is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can be modified or equivalently replaced without departing from the purpose and scope of the technical solutions, and all should be covered in the scope of the claims of the present application.

Claims

1.A personalized intelligent car-following method based on double-layer blockchains in a vehicle Internet of Things, characterized in that: The method comprises the following steps: S1: establishing a group learning DRL architecture based on a double-layer blockchain; S2: CAV selection based on reputation value; the CAV considers the performance of the CAV from multiple angles in the continuous interaction between the CAV and the environment, selects high-quality CAVs to participate in aggregation, and sends the updated model of the CAV to the corresponding RSU for aggregation; the reputation value of the CAV is composed of four parts: familiarity, freshness, time variation and importance; Familiarity: replaced by average reward, the greater the average reward, the greater the familiarity; Freshness: the reputation value of the CAV in the last round of training is more important than in the past; Time variation: the communication quality change between the CAV and the RSU, the average transmission rate of the CAV in a round of training is calculated; Importance: selecting the most important CAV in the RSU coverage area; The selected CAV packs the trained local new model into a transaction and sends it to the RSU to obtain an updated partition model, and then aggregates the partition model to obtain a global model; wherein the transaction header includes the hash value of all selected transactions, and the transaction body includes the trained model; S3: establishing a personalized following model based on DDPG; the model training comprises two steps: first, the CAV shares the aggregated model through group learning; second, the CAV trains a personalized following model based on local data using DDPG; group learning combines edge computing and blockchain technology together, and realizes model sharing and distributed learning by aggregating on the RSU; The CAV following DRL algorithm allows the CAV to establish a decision method based on the exploration and experience between the CAV and the environment, and uses artificial intelligence to fit the optimal strategy; the personalized following process of the CAV is modeled as an infinite state Markov decision process, and the deep deterministic policy gradient algorithm DDPG is used to solve this high-dimensional continuous control problem; DDPG is a reinforcement learning algorithm based on the Actor-Critic architecture, which combines deep learning and policy gradient and is suitable for control tasks in continuous action space; The CAV obtains the maximum reward by interacting with the environment, and the reward function of different driving styles is used as a training signal to encourage or prevent behavior in the following action; the CAV takes the perception data in driving as the environment, and the state space, action space and personalized reward function of the CAV are defined as follows: (1) State space At time t, the state perceived by the following CAV i is as follows: s t = (v i ,v p ,Δv,d i,p ) (13) where v i and v p are the speeds of the following CAV i and the preceding CAV p, respectively, Δv is their relative speed, and d i,p is the relative longitudinal distance between the two vehicles. (2) Action space In the following process, the following CAV i controls the acceleration of the CAV, and the action space of the CAV i at time t is: a t = {a i (t)} (14) The acceleration selection range is [-4, 2] m / s in consideration of passenger comfort 2 ; (3) Reward function Design a following reward function based on three different styles of aggressive, moderate and conservative, which includes a driving safety reward function, a following distance reward function, a driving efficiency reward function and a comfort reward function; ① Driving safety reward function The following considers the safety in the following process, and is set as: J safety = -δ[v i (t) 2 +a]||{Collsion} (15) Wherein δ and a are weight parameters, and ||{Collision} represents 1 when collision occurs, and 0 in other cases; ② Following distance reward function The following distance reward function is as follows: where x is the current following distance, d ideal The preferred following distance is 48 m, 69 m, and 90 m, which represent the aggressive, moderate, and conservative styles, respectively. ③Driving efficiency reward function The driving efficiency is evaluated using the headway value TH, which is defined as the time elapsed between the CAV i and the preceding vehicle CAVp to reach a given point. The driving efficiency reward function is as follows: For the aggressive style, the reward function has a high reward when the TH value is small; for the moderate style, the reward function has a certain sensitivity to the TH value; ④Comfort reward function The comfort value is adjusted to [0, 1] using a normalization method. The comfort reward function is as follows: where jerk is the jerk value, and 3600 is the square of the maximum allowable jerk value; For the following task, the total reward function based on the moderate, conservative, or aggressive driving style is represented as a linear combination of the reward functions: J style = ω1J safety + ω2J ideal + ω3J effciency + ω4J comfort (19) where J style is the total reward function corresponding to the three styles; ω1, ω2, ω3, ω4 are the weight coefficients of each reward function; DDPG consists of a policy network Actor and a critic network Critic; two independent neural networks represent Actor and Critic in DDPG respectively; at time t, Actor Network takes state s t =(v i ,v p ,Δv,d i,p ) as input and outputs continuous action a(t); the input of Critic Network is a state-action pair (s t ,a t ), and its output is a scalar Q value Q(s t ,a t ); DDPG divides Actor and Critic into two sub-networks with the same topology structure: OnlineNet and Target Net, the former adopts the latest network parameters, and the Target network is updated after a certain time; the parameters of Actor and Critic networks are updated by gradient and minimizing loss function L; The execution strategy is output by the Actor unit, and its loss function is as follows: The Critic unit evaluates the output strategy of the Actor, and its corresponding loss function is represented as follows: y k =r k +γQ'(s k+1 ,m'(s k+1 |θ μ’ )|θ Q’ ) (22) where θ μ , θ μ’ are the parameters of Online and Target in Actor, respectively, θ Q , θ Q’ are the network parameters of Online and Target in Critic, respectively, r k is the reward value at the current time, n is the number of samples per sampling, and γ is the discount factor. 2.The personalized intelligent car-following method based on double-layer blockchain in V2X according to claim 1, wherein: In S1, a three-layer system model is established, including a blockchain layer, a learning layer, and a smart connected vehicle CAV layer. The blockchain layer adopts a double-layer chain architecture, the upper main chain adopts a main chain based on directed acyclic graph DAG consensus, and the lower layer adopts a partitioned blockchain based on PBFT consensus. Based on geographic location and base station BS, the coverage range is divided into multiple partitions, each partition is composed of a BS and multiple roadside units RSUs, and each partition cooperates and processes consensus tasks in parallel; Model sharing is in the form of federated learning; the CAV layer is a car following scene in different situations; The RSUs and BSs in each partition are connected through DSRC to form an edge computing network within the partition; The BSs between partitions are connected through a wired network to form a cross-zone edge computing network; the CAVs transmit data to the BSs and RSUs through wireless links, and the entire process of model uploading and aggregation is recorded in the partitioned blockchain; First, the task publisher registers and publishes the training task in the blockchain network, and the CAVs will train personalized car following models based on DDPG; second, the cloud end will distribute the initial model to each partition blockchain; finally, the BS of each partition will generate a genesis block for the task and the initial model, and add it to the corresponding PBFT sub-chain. All nodes train local models based on the genesis block at startup, and process subsequent transactions through the consensus mechanism and upload blocks; The method of uploading the model to the blockchain is as follows: the CAV uploads its local model to the corresponding RSUs based on the local model obtained in S3, the selected CAV uploads its local model to the corresponding RSUs for aggregation, the RSUs request the blockchain partition network, each sub-chain executes PBFT consensus, and records the partition aggregation parameters; then the RSUs package these parameters into transactions and upload them to the BS, the RSU completes the block header information of the transaction, including version number, timestamp, random number, and hash value; finally, the BS uploads it to the main chain DAG; each transaction in the main chain network contains a model trained by a partition, providing a secure aggregation method between partitions for asynchronous cross-zone verification. The CAV training task termination method is that the task publisher requests the current model update in real time from the BS, analyzes the performance of the current model, and publishes the task termination information to each partition BS and RSU when the performance of the current model reaches the requirement, and the CAV training task ends; and the reputation value of the CAV recorded in the RSU is used to reward or punish the CAV. 3.The method of claim 1, wherein the method is characterized in that: In S3, the CAV initiates an identity request to the blockchain network through the built-in communication module, verifies the identity of the CAV using a public key infrastructure (PKI), and uses the public key and private key pair of the CAV for signature verification; first, the CAV generates a request message containing a unique identifier and a timestamp, and signs the message using its private key; the authentication node in the blockchain network verifies the authenticity of the signature using the public key of the CAV after receiving the request; if the verification is passed, the CAV is granted access; after obtaining the current initial model, the CAV loads it into the local computing module; The CAV performs personalized following model training based on DDPG to achieve high-performance following close to the driving style of the driver; first, the CAV collects the current environmental information and records it as the current state; the CAV policy network generates corresponding action decisions based on the current environment, executes the actions and evaluates the action decisions; the CAV stores the relevant information of environmental interaction, action and reward in the experience replay memory; When training is needed, the experience data is sampled in batches from the experience replay memory, and the network parameters are updated using the experience data; During the training process, first, the driving information of the target CAV and the surrounding CAVs required for the CAV decision model is constructed, and the driving style, driving efficiency, collision avoidance and comfort are considered to design the reward function of DDPG, so as to solve the problem of lack of safety guarantee in the decision-making process. 4.The personalized intelligent car-following method based on double-layer blockchain in V2X according to claim 1, wherein: In S2, in the direct attribute evaluation stage, the reputation value of CAV i at time t is calculated as follows: where ψ is a freshness adjustment weight, whose value is less than 1, respectively, are the reputation value, average reward and average transmission rate of CAV i at time t; is calculated from the following equation: where N is the number of rounds experienced in a round of training, i.e. the average reward is calculated over the last N rounds, T represents the current round, and r i|t＝m represents the reward value obtained by CAV i in the mth round. The channel transmission quality between CAV and RSU needs to be considered; the transmission rate between CAV and RSU will change with the movement of CAV, and the average transmission rate within one round of training time is used for calculation; according to Shannon formula, the instantaneous transmission rate between CAV node i and aggregation node RSU is as follows: where B is the bandwidth of the communication channel, p i is the transmission power, and h i is the channel gain, and N0is the additive white Gaussian noise power; Channel gain refers to the total gain from the transmitting end to the receiving end, including the antenna gain and the influence of free space propagation, considering the ITU-R M.2135 path loss line-of-sight model, which is applicable to various scenarios, including urban, suburban and highway, PL LOS As follows: where d i is the distance between CAV and RSU, in meters, f is the frequency of the signal, in Hertz, and h i is calculated as follows: d i As the movement of CAV is constantly changing, according to the straight motion model, it is assumed that the CAV keeps uniform straight motion in a round of training time T train , the scene is equivalent to a plane coordinate system with the aggregation node as the origin, and the following relationship between the distance from the CAV to the aggregation node and the time is obtained when the following movement direction of the CAV is parallel to the road direction: where (x i0 ,y i0 ) denotes the initial coordinates of CAV i, the function of the transmission rate over time is expressed as: Finally, CAV i in a round of training time T train The average transmission rate within is expressed as: The average transmission rate of the CAV node in a round of training is obtained; in the direct attribute evaluation stage, the RSU first obtains the average reward of the CAV and the reputation value at the previous time, then obtains the position and movement information of the CAV through its global perception ability, and obtains the relevant parameters of the CAV communication ability through interaction with the CAV, calculates the average transmission rate of each CAV in its coverage range, and finally obtains the reputation value of the CAV at the current time; In the indirect attribute evaluation stage, the most important CAV in each partition is selected to join the aggregation to reduce the model transmission overhead. In this stage, the homogenization area is introduced, which represents the range with CAV i as the center and c as the radius. The value of c is determined according to the traffic flow. The smaller the traffic flow, the smaller the value of c. The larger the traffic flow, the larger the value of c. The CAVs in this range do not participate in model aggregation. The relative positions between CAVs change with their movement. Let CAV i keep a constant linear motion with speed v i The average distance matrix D of the CAVs is calculated by the aggregation node RSU at time t while keeping a constant linear motion, as follows: wherein represents the average distance between any two CAVs i and j, expressed as: In the direct attribute evaluation stage, the reputation values of each CAV are obtained, the reputation values of all CAVs are sorted from high to low by the aggregation node RSU, and the CAVs with high reputation values are selected to enter the indirect attribute evaluation stage; in this stage, the CAV with the highest reputation value is first selected, and then the CAVs located in the homogenization area are excluded according to the homogenization area and distance matrix; then, the CAV with the highest reputation value is selected from outside the homogenization area, and a new homogenization area is divided around this CAV; the RSU repeatedly performs this process until there are no more CAVs to be selected.

Images