A Blockchain-Based Distributed Intelligent Assisted Automated Driving Method

Abstract

This invention relates to a blockchain-based distributed intelligent assisted autonomous driving method, belonging to the field of mobile communication technology. First, a two-layer network model based on a directed acyclic graph (DAG) blockchain is established, and an asynchronous federated learning scheme is designed under this model. Second, knowledge distillation technology is employed during the local training phase to achieve low-load model interaction in asynchronous federated learning. Then, from the perspectives of high quality, high safety, and high efficiency, an autonomous driving decision-making model based on proportional-integral-derivative (PID) control is established, and CAV acceleration control is optimized based on the deep deterministic policy gradient (DDPG) algorithm. Finally, a mobile perception asynchronous federated distillation learning (MAFDL) algorithm based on DAG blockchain is proposed. This scheme effectively assists CAV intelligent driving decision-making while ensuring safe and low-load CAV collaborative training.

Description

Technical Field

[0001] This invention belongs to the field of mobile communication technology and relates to a blockchain-based distributed intelligent assisted autonomous driving method. Background Technology

[0002] Connected and automated vehicles (CAVs), as a key vertical application of sixth-generation (6G) mobile communication, are improving people's transportation. CAVs utilize onboard sensors and communication modules to enhance their perception capabilities for motion planning and automatic control. Machine learning is considered a key technology for achieving autonomous driving, relying on CAV perception data to train models to support intelligent driving, such as pedestrian detection, distance judgment, and traffic sign recognition. However, the perception data of a single CAV is limited and cannot cope with more complex traffic environments. Therefore, vehicle service providers collect large-scale CAV perception data from the bottom up through roadside units (RSUs) to centrally train a model with strong generalization ability to support CAV autonomous driving in complex traffic environments. However, this centralized training method cannot meet the requirements of user privacy and data security in 6G technology. Furthermore, the potential single point of failure of vehicle service providers threatens the application of autonomous driving.

[0003] Federated learning, as a distributed learning paradigm, trains high-quality models using interactive models rather than raw data, thus protecting user privacy. Furthermore, blockchain technology enables RSUs to jointly maintain a traceable and immutable distributed ledger, ensuring data security while preventing single points of failure for vehicle service providers. The distributed nature of blockchain and federated learning is seamlessly integrated to achieve a CAV collaborative training paradigm with strong privacy protection and high data security. However, the heterogeneity of perception data from CAVs operating in different road environments weakens the generalization ability of the global model. This effect is even more pronounced when training the model while the CAV is in motion. Moreover, the rapidly changing speeds of CAVs during travel cause motion blur in their perception data, further reducing the robustness of the global model.

[0004] Reinforcement learning (RL) adjusts its decision-making and control through interaction between the Continuous Ability Observer (CAV) and the environment to obtain better reward outcomes, thus achieving intelligent control that accelerates CAV. Furthermore, proportional-integral-derivative (PID) controllers, based on gain coefficient adjustment, can achieve smooth control of CAV speed. Therefore, how to interweave PID controllers to formulate the relationship between reward decisions and environmental changes to achieve intelligent and smooth control that accelerates CAV urgently needs to be addressed. Simultaneously, to reduce communication load and achieve low-latency CAV federated learning, federated learning schemes that reduce the size of the interaction model are worth exploring. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a blockchain-based distributed intelligent assisted autonomous driving method.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] In a first aspect, embodiments of the present invention, based on the characteristics of network scenarios, realize a distributed intelligent process based on DAG blockchain, and simultaneously optimize CAV acceleration control based on PID control and DDPG algorithm. The method includes the following steps:

[0008] S1: A two-layer network model based on DAG blockchain;

[0009] S2: A low-load model interaction scheme based on asynchronous federated distillation learning;

[0010] S3: PID-based autonomous driving decision-making model;

[0011] S4: A CAV acceleration control optimization method based on the DDPG algorithm.

[0012] Secondly, in step S1 of this embodiment of the invention, a two-layer network model based on DAG blockchain is established: 1) DAG layer, including RSU and the blockchain based on DAG structure maintained by it; 2) CAV layer, including heterogeneous vehicles trained in a cooperative distributed manner.

[0013] RSUs are equipped with micro edge servers, possessing abundant computing and communication resources to provide high-performance computing and low-latency services for CAVs. The CAV layer consists of CAVs equipped with onboard sensors, which use sensor perception data for collaborative training and driving decisions. The heterogeneity of perception data due to CAVs operating in diverse road environments leads to reduced global model accuracy during collaborative training while in motion. Furthermore, rapid changes in CAV speed cause motion blur in the perception data, resulting in poor generalization ability of the local model. Therefore, reinforcement learning algorithms are needed to implement intelligent driving decisions, with PID algorithms used to optimize acceleration and smooth speed changes. Simultaneously, federated distillation learning is used to transmit smaller mentee models during model interaction and aggregation phases to reduce communication load, while larger mentor models are used during model training to transfer knowledge to the mentee models and improve their performance.

[0014] Thirdly, in step S2 of this embodiment, a low-load model interaction scheme based on asynchronous federated distillation learning is proposed. Due to the mobility of CAVs, the training method where the RSU is responsible for synchronous model aggregation faces bottlenecks. The asynchronous federated distillation learning method based on the DAG blockchain, on the one hand, avoids considering the model performance impact caused by CAV switching RSUs, and on the other hand, reduces the communication load by using models with smaller interaction structures. In this invention, for each CAV, it requests the RSU and obtains all Tips from the DAG to test the model accuracy. The two models with the highest accuracy are selected and aggregated to generate a local mentor model. The CAV then performs knowledge distillation training on this model with the local mentor model to generate a NewTip, which is uploaded to the RSU. The RSU receives the New Tip, generates a Tip, and uploads it to the DAG chain.

[0015] Fourthly, in step S3 of this embodiment, an autonomous driving decision-making model based on PID control is proposed. The rapid changes in vehicle speed in traditional CAV acceleration control algorithms cause unstable following distances, affecting road safety, and on the other hand, sensor jitter leads to motion blur in the perceived data. In this invention, the CAV observes the state from the road environment, adjusts the proportional gain, integral gain, and derivative gain based on the state, accelerates through PID control, and then observes the state and reward. During this process, the CAV maximizes the reward. Therefore, the reward design is based on high quality, high safety, and high efficiency considerations.

[0016] Fifthly, in step S4 of this embodiment, a CAV acceleration control optimization method based on the DDPG algorithm is proposed. It mainly includes two steps: experience replay and gradient descent, and two networks: Actor and Critic. Each network consists of a target network and an online network. In this invention, experience replay assists the target network in outputting the correct label. This label guides the gradient descent update of the online network. The online network outputs optimized CAV decisions while simultaneously assisting the target network in soft updates. The actions, rewards, and states output after network training are stored as experience.

[0017] The beneficial effects of this invention are as follows: This invention designs a vehicle-to-everything (V2X) architecture based on a directed acyclic graph (DAG) blockchain. Under this architecture, asynchronous federated distillation learning will improve the accuracy of the global model while reducing communication load. Furthermore, vehicle speed control based on the deep deterministic policy gradient (DDPG) algorithm will ensure high-quality perception data while achieving intelligent driving.

[0018] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0019] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0020] Figure 1 This is a diagram of a vehicle-to-everything (V2X) architecture based on DAG blockchain.

[0021] Figure 2 This is a flowchart of the asynchronous federated distillation learning process based on DAG blockchain;

[0022] Figure 3 This is a schematic diagram of CAV acceleration control based on the DDPG algorithm;

[0023] Figure 4 This is a flowchart of the MADFL algorithm execution based on DAG blockchain. Detailed Implementation

[0024] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0025] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0026] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0027] Figure 1 A possible structural schematic diagram of a communication system according to an embodiment of the present invention is shown. For example... Figure 1 As shown, the network considers a two-layer structure, including a DAG layer and a CAV layer. The DAG layer is a blockchain based on the DAG structure, where each transaction is stored in the form of a tangle. The blockchain is jointly maintained by RSUs located near the road and is used to record model updates. The CAV layer consists of CAVs equipped with multiple sensors, possessing good sensing capabilities and computing resources, and communicates with the RSUs via a wireless link.

[0028] 1. Transaction verification process based on DAG blockchain

[0029] In a DAG-based blockchain, every transaction except the genesis transaction is generated by CAVs participating in collaborative training. The DAG chain is maintained by RSU, which packages NewTip sent by CAVs into Tips and uploads them to the chain. Each transaction consists of a transaction header and a transaction body. The transaction header includes a version number, timestamp, random number, transaction hash value, the number of forward transactions, and the hash values ​​of all pointing transactions. The transaction body contains the local mentee model uploaded by CAV after local training. In the DAG, each transaction points to two forward transactions, indicating that the local mentee model in the transaction was obtained by CAV by aggregating the models from the two forward transactions and then training them locally. It also signifies that the legitimacy of the two forward transactions has been verified.

[0030] Figure 1 The middle DAG layer demonstrates that the DAG structure mainly consists of Genesis transactions, Confirm transactions, Unconfirm transactions, Tips, and NewTips. Genesis transactions, or genesis transactions, contain the initial model generated by the vehicle service provider and uploaded to the chain via RSUs. In subsequent verification processes, all transactions can be traced back to the Genesis transaction. Confirm transactions are those linked to by a sufficient number of transactions, signifying their trustworthiness. Unconfirm transactions are transitional transactions that haven't yet been linked to by enough transactions; as new transactions link to Unconfirm transactions, they eventually become Confirm transactions. Tips are unconfirmed transactions already propagating on the edge network but awaiting inclusion in the DAG structure. NewTips are newly created transactions that haven't yet propagated on the edge network. As new transactions are continuously uploaded to the chain, the DAG chain grows. Newly added transactions await verification by subsequent new transactions; subsequently, Tips become Unconfirm transactions, and Unconfirm transactions become Confirm transactions. During this process, the model accuracy in each transaction continuously increases until convergence.

[0031] 2. Asynchronous Federated Distillation Learning Process Based on DAG Blockchain

[0032] Asynchronous federated distillation learning based on DAG mainly includes two processes: first, CAV requests Tips from RSU to aggregate the model; second, CAV completes knowledge distillation training based on the local mentor model and the local mentor model. Figure 2 As shown, in federated distillation learning, the richer network structure of the local mentor model achieves higher accuracy, while the smaller network size of the local mentor model enables low-load transmission. For each CAV, its local training consists of the following steps:

[0033] (1) Get Tips: CAV first establishes a connection with RSU and then sends a request to get all the Tips in the current edge network.

[0034] (2) Validation Tips: CAV collects real-time sensing data based on multiple sensors and uses the local sensing dataset to validate the accuracy of the models in the Tips before sorting them in descending order. After selecting the first two Tips, the models in the Tips are aggregated to obtain the local Mendee model.

[0035] (3) Task loss calculation: CAV obtains the task loss of the mentee model and the mentor model based on the real labels of the perception dataset and the predicted labels of the local mentee model and mentor model, respectively.

[0036] (4) Knowledge distillation training: CAV uses the predicted labels of the two models to calculate the distillation loss of the local mentee and mentor models respectively through KL (Kullback-Leibler) divergence. Then, the normalized loss of the sum of the task loss and the distillation loss is calculated, and the local mentee and mentor models are updated once respectively based on the normalized loss using the stochastic gradient descent (SGD) algorithm.

[0037] (5) Model on-chain: CAV generates a New Tip, in which the transaction header includes two hash values ​​pointing to the transaction and the transaction body includes the local Mendee model trained by knowledge distillation. After the New Tip is sent to RSU, RSU adds the version number, timestamp, random number and transaction hash value to the transaction header to make it a Tip and broadcast it in the edge network.

[0038] 3. Automated driving decision-making model based on PID control

[0039] Autonomous driving decision-making models are based on reinforcement learning. They primarily involve agents interacting with the environment through actions to obtain states and rewards, maximizing these rewards in the process. In the Internet of Vehicles (IoV), each CAV (Consumer-Agent Vehicle) is an agent, and real-time road changes constitute the environment. Furthermore, the actions, observations, and rewards of a CAV are defined as follows:

[0040] (1) Action: CAV mainly adjusts non-negative parameters. The proportional gain, integral gain, and derivative gain are used to control acceleration via PID control. In PID control, the error is the difference between the actual CAV speed and the reference speed, where the reference speed is obtained by the onboard sensors sensing the road environment and outputting it through a local mentor model.

[0041] (2) Observation: After the CAV control outputs acceleration through PID control, the road environment will change. Based on the vehicle sensor, it mainly observes the speed, walking distance, position, distance to the vehicle in front, and reference speed.

[0042] (3) Rewards: CAVs will receive rewards after making motion observations. The rewards are mainly based on quality, safety, and efficiency. To ensure high quality, CAVs will be penalized for accelerating too much, causing motion blur in the sensor data. To ensure high safety, CAVs will be penalized for having a large difference between their actual speed and the reference speed, causing driving safety issues. To ensure high efficiency, CAVs will be penalized for being too close or too far from the vehicle in front, affecting traffic efficiency.

[0043] 4. Optimization of CAV acceleration control based on DDPG algorithm

[0044] The DDPG algorithm mainly consists of two parts: the Actor network and the Critic network. The Actor network generates a set of actions based on the current state, while the Critic network estimates the current state and the estimated actions. Its training process is mainly divided into two parts: experience replay and gradient descent. Experience replay is used to store experience to assist in updates, while gradient descent is used to update both the Actor and Critic networks. Figure 3 As shown, the DDPG algorithm mainly consists of the following steps:

[0045] (1) Stochastic strategy: CAV makes the decision a stochastic process by adding random noise to the output of the Actor network and sampling the action from the stochastic process.

[0046] (2) Storing experience: CAV uses a cache to store experience (the process from one decision to the next) in the experience replay area.

[0047] (3) Sampling: CAV randomly samples a set of experiences from the experience replay area as updates for the Actor and Critic networks.

[0048] (4) Critic network update: The output of the targetQ network in the Critic network guides the SGD update of the onlineQ network, while the onlineQ network updates the targetQ network through a soft update method.

[0049] (5) Actor network update: The results of the onlineQ network are used to guide the update of the online policy network SGD, while the target policy network update is based on the soft update method.

[0050] 5. System Flow

[0051] Figure 4The diagram shows the execution flowchart of the Mobility-Aware Asynchronous Federated Distillation Learning (MAFDL) algorithm based on DAG blockchain. The specific steps are as follows:

[0052] Step 501: Algorithm initialization;

[0053] Step 502: The vehicle service provider issues the task through the RSU;

[0054] Step 503: Initialize the model and generate Genesis transactions;

[0055] Step 504: CAV applies for registration as a training vehicle;

[0056] Step 505: RSU processes the request information;

[0057] Step 506: CAV obtains SDK certificate;

[0058] Step 507: Send a request to the RSU to obtain tips;

[0059] Step 508: RSU sends all Tips to the requesting CAV;

[0060] Step 509: CAV test tips accuracy;

[0061] Step 510: CAV selects the two models with the highest accuracy for aggregation;

[0062] Step 511: Calculate the task loss of the local Mentee model;

[0063] Step 512: Calculate the task loss of the local mentor model;

[0064] Step 513: Calculate distillation loss;

[0065] Step 514: Calculate the normalization loss and update the local mentee and mentor models;

[0066] Steps 515-522: CAV driving control optimization;

[0067] Step 515: CAV perceives the road environment;

[0068] Step 516: CAV decision based on PID control acceleration;

[0069] Step 517: Store experience;

[0070] Step 518: Sampling in the experience playback area;

[0071] Step 519: Critic network update;

[0072] Step 520: Actor network update;

[0073] Step 521: If the reward is maximized, proceed to step 522; otherwise, repeat step 515.

[0074] Step 522: CAV generates NewTip based on the local mentee model;

[0075] Step 523: NewTip is sent to RSU;

[0076] Step 524: Add version timestamp and other information to generate a Tip;

[0077] Steps 525-526: Tip verification;

[0078] Step 525: Propagate the tip on the edge network;

[0079] Step 526: If the Tip is validated by other Tips, proceed to step 527; otherwise, repeat step 525.

[0080] Steps 527-528: Unconfirm transaction verification;

[0081] Step 527: Tip becomes an Unconfirmed transaction;

[0082] Step 528: If the Unconfirm transaction is verified by enough transactions, proceed to step 529; otherwise, repeat step 527.

[0083] Step 529: An Unconfirmed transaction becomes a Confirmed transaction;

[0084] Step 530: If the stopping condition is met, proceed to step 531; otherwise, repeat step 507.

[0085] Step 531: Algorithm ends.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A blockchain-based distributed intelligent assisted autonomous driving method, characterized in that: The method includes the following steps: S1: A two-layer network of Directed Acyclic Graph (DAG) and Connected Vehicle (CAV) is adopted to realize federated learning based on DAG blockchain to assist CAV autonomous driving decision-making. The DAG chain is maintained by the Roadside Unit (RSU). The vehicle service provider publishes learning tasks through the RSU. The CAV obtains transactions of the DAG chain by requesting the RSU. The CAV perceives road data based on multiple sensors and uses it for both federated learning and driving decision-making. S2: Propose an asynchronous federated learning scheme based on DAG; based on the verification mechanism of DAG transactions, CAV requests nearby RSUs to obtain all Tips and test the accuracy, selects the two models with the highest accuracy, aggregates and trains them, and then points the newly generated New Tip transaction header to these two Tips; Low-load model interaction is achieved through federated distillation learning; CAV holds a local mentee model and a local mentor model. During model training, the task loss, distillation loss, and normalization loss of each model are calculated separately, and the model is updated based on the normalization loss; CAV only generates New Tips based on the local mentee model and uploads them to RSU, while the local mentor model is always kept locally. S3: Design an autonomous driving decision-making model based on the proportional-integral-derivative PID algorithm; wherein, the CAV observes the road environment to obtain the state and evaluates the reward. The reward design considers three aspects: high quality, high safety and high efficiency. The gain coefficient output in the PID algorithm is adjusted by the state and reward to accelerate the decision-making and change the road environment. The CAV continuously maximizes the reward in this process. S4: Optimize CAV acceleration control based on the deep deterministic policy gradient (DDPG) algorithm; where experience replay provides historical experience for the updates of the Actor and Critic networks, the Actor and Critic networks are updated alternately based on gradient descent, and the CAV makes the optimal acceleration decision to maximize the reward.

2. The distributed intelligent assisted autonomous driving method based on blockchain according to claim 1, characterized in that: The DAG consists of Genesis transactions, Confirm transactions, Unconfirm transactions, Tips, and New Tips; among them, Genesis transactions are the genesis transactions that contain the initial model generated by the vehicle service provider and uploaded to the chain through RSUs, and all transactions in the subsequent verification process can be traced back to Genesis transactions; Confirm transactions are those that are pointed to by a sufficient number of transactions, indicating that they are trustworthy; Unconfirmed transactions are transitional transactions that have not yet been pointed to by enough transactions. As new transactions continuously point to Unconfirmed transactions, they will eventually become Confirmed transactions. Tips are unconfirmed transactions that have already spread on the edge network but are waiting to be included in the DAG structure. New Tips are newly created transactions that have not yet spread on the edge network. As new transactions are continuously added to the chain, the DAG chain continues to grow. Newly added transactions await verification by subsequent new transactions. Then, Tips become Unconfirmed transactions, and Unconfirmed transactions become Confirmed transactions. In this process, the model accuracy in each transaction will continuously increase until convergence.

3. The blockchain-based distributed intelligent assisted autonomous driving method according to claim 1, characterized in that: In S2, for each CAV, local training consists of the following steps: (1) Obtain Tips: CAV first establishes a connection with RSU and then sends a request to obtain all Tips in the current edge network; (2) Validation Tips: CAV collects sensing data in real time based on multiple sensors, and then sorts the models in the Tips in descending order after validating the accuracy of the models in the Tips using the local sensing dataset; after selecting the first two Tips, the models in the Tips are aggregated to obtain the local Mendee model; (3) Task loss calculation: CAV obtains the task loss of the mentee model and the mentor model based on the real labels of the perception dataset and the predicted labels of the local mentee model and mentor model, respectively; (4) Knowledge distillation training: CAV uses the predicted labels of the two models to calculate the distillation loss of the local mentee and mentor models respectively through KL divergence; then calculates the normalized loss of the sum of task loss and distillation loss, and updates the local mentee and mentor models once respectively based on the normalized loss through the stochastic gradient descent (SGD) algorithm. (5) Model on-chain: CAV generates a New Tip, in which the transaction header includes two hash values ​​pointing to the transaction and the transaction body includes the local Mendee model trained by knowledge distillation; after the New Tip is sent to RSU, RSU adds version number, timestamp, random number and transaction hash value to the transaction header to become a Tip and broadcast it in the edge network.

4. The distributed intelligent assisted autonomous driving method based on blockchain according to claim 1, characterized in that: The CAV's actions, observations, and rewards are defined as follows: (1) Action: The CAV adjusts the non-negative parameters, namely proportional gain, integral gain and derivative gain, to control the acceleration through PID control; the error in PID control is the difference between the actual speed of the CAV and the reference speed, where the reference speed is obtained by the on-board sensor sensing the road environment and outputting through the local mentor model; (2) Observation: After the CAV control outputs acceleration through PID control, the road environment will change. The observation of speed, walking distance, position, distance to the vehicle in front and reference speed is based on the on-board sensor. (3) Rewards: CAVs will receive rewards for making motion observations, including quality, safety and efficiency; To ensure high quality, penalties will be imposed for excessive acceleration of the CAV, which may cause motion blur in the sensor data; to ensure high safety, penalties will be imposed for a large difference between the actual speed and the reference speed of the CAV, which may cause driving safety issues; and to ensure high efficiency, penalties will be imposed for the CAV being too close or too far from the vehicle in front, which may affect traffic efficiency.

5. The blockchain-based distributed intelligent assisted autonomous driving method according to claim 1, characterized in that: The DDPG algorithm consists of two parts: Actor and Critic. The Actor network generates a set of actions based on the current state, and the Critic network is used to estimate the current state and the value of the actions. Its training process is divided into two parts: experience replay and gradient descent. Experience replay is used to store experience to assist in updates, and gradient descent is used to update the Actor and Critic networks. The DDPG algorithm consists of the following steps: (1) Stochastic strategy: CAV makes the decision a stochastic process by adding random noise to the output of the Actor network and sampling the action from the stochastic process; (2) Storing experience: CAV uses a cache to store the experience of the process from one decision to the next in the experience replay area; (3) Sampling: CAV randomly samples a set of experiences from the experience replay area as updates for the Actor and Critic networks; (4) Critic network update: The output of the target Q network in the Critic network guides the SGD update of the online Q network, while the online Q network updates the target Q network through a soft update method; (5) Actor network update: The results of the online Q network are used to guide the online policy network SGD update, while the target policy network update is based on the soft update method.

Images