Scene-driven high-generalization federal reinforcement learning automatic driving system and method

Abstract

The invention discloses a scene-driven high-generalization federal reinforcement learning automatic driving system and a scene-driven high-generalization federal reinforcement learning automatic driving method, which are used for simulating learning experts to learn implicit expert preferences from expert demonstration and realizing multi-angle guidance of reinforcement learning student feature extraction, reward function modeling, loss function construction and group optimization in combination with dynamic driving suggestions. The problem of reinforcement learning target alignment is solved in a targeted manner; learning experts are simulated, the relation among multiple learning tasks is enhanced through cross attention, and the aerial view reconstruction process under the embedded guidance of scene depth perception and scene semantic perception is achieved. Through cross-scene optimization, specific experience sharing among different expert data distributions is realized, and high-universality imitation learning experts are trained; motion planning, reward feedback and dynamic driving suggestions are output through a proposed reward function decoder to serve as training and optimization bases of reinforcement learning students, and learning of driving rules is achieved. And through combination of the time sequence processing module and knowledge distillation, the representation capability of original image input is improved.

Description

Technical Field

[0001] The present invention belongs to the field of vehicle engineering and transportation engineering, and relates to a scenario-driven, highly generalized federated reinforcement learning autonomous driving system and method. Background Art

[0002] Urban driving is an unresolved problem in the field of autonomous driving due to its diverse road conditions, complex traffic conditions, and frequent interaction scenarios. In urban driving environments, there are frequent interactions between traffic participants, road infrastructure, and traffic rules, which further tests the safety of autonomous driving algorithms. Although rule-based methods have been successful in simple driving scenarios, the limitations of manual rules restrict their application in complex traffic situations.

[0003] In contrast, data-driven methods use a large amount of driving data to help intelligent agents learn driving strategies, without being bound by rules and logic written in specific situations. This type of method achieves feature expression in high-dimensional state space through the fitting ability of deep neural networks combined with supervised or unsupervised training methods. However, the sample coverage problem faced by data-driven methods and the lack of training on out-of-distribution data of this type of method can lead to serious safety consequences. At the same time, the lack of transparency in the internal working principles of end-to-end methods increases the difficulty of framework interpretation and subsequent optimization, causing a crisis of trust. In addition, data-driven autonomous driving systems rely on incremental learning in different scenarios and lack the ability to integrate information across scenarios and synchronize optimization capabilities. The scenario-driven optimization problem caused by indicator conflicts and experience coverage in the incremental optimization process of different scenarios will lead to catastrophic forgetting of the algorithm in continuous learning, reducing the system versatility and reliability in complex driving scenarios. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a scenario-driven highly generalized federated reinforcement learning autonomous driving system and method. By imitating learning experts to learn implicit expert preferences from expert demonstrations, combined with the proposed dynamic driving suggestions, the reinforcement learning student feature extraction, reward function modeling, loss function construction, and group optimization multi-angle guidance are realized, and the reinforcement learning target alignment problem is solved in a targeted manner. High reliability control performance is achieved by relying only on a monocular camera in a complex traffic environment; the proposed imitation learning expert enhances the connection between multiple learning tasks through cross-attention, and realizes the reconstruction process of the bird's-eye view under the guidance of scene depth perception and scene semantic perception embedding. In view of the catastrophic forgetting problem in continuous learning, the specific experience sharing between different expert data distributions is realized through cross-scenario optimization, and highly universal imitation learning experts are trained; through the proposed reward function decoder, the federated teacher-student architecture outputs motion planning, reward feedback and dynamic driving suggestions as the basis for training and optimization of reinforcement learning students, so as to realize their learning of driving rules. In view of the high-dimensional environment representation problem, the federated teacher-student architecture combines the query-based knowledge distillation with the temporal processing module to improve the representation ability of reinforcement learning for raw image input.

[0005] The technical solution of the scenario-driven, highly generalized federated reinforcement learning autonomous driving system proposed in the present invention includes three parts: a reinforcement learning student modeling part, a training imitation learning expert part, and a distillation reinforcement learning student part.

[0006] For the reinforcement learning student modeling part, the feedback-based target alignment module is first built, and then the timing processing module is built.

[0007] The target alignment module generates binary feedback by comparing the weighted cumulative rewards of two trajectories and establishes a preference model based on the true value of the reward and the expert demonstration, as follows:

[0008]

[0009] where y(τ0,τ1)∈{(1,0),(0,1)} represents the binary feedback of the preference model for trajectory segments τ0 and τ1 of length H, i∈{0,1} represents a state-action pair The true value of the reward for i∈{0,1}, Denotes the trajectory segment τ i , driving behavior uncertainty weight of i∈{0,1}, Denotes the trajectory segment τ i , the error weight of i∈{0,1}, Represents the trajectory segment τ of the agent at time t i , lateral movement under i∈{0,1}, Represents the trajectory segment τ of the agent at time t i , i∈{0,1}, △d(t) represents the lateral error of the agent from the expected path at the time t, △θ(t) represents the heading angle deviation of the agent from the expected path at the time t, and the true reward value Set to:

[0010]

[0011] Where v(t) represents the speed of the agent at the time t, v desire (t) represents the expected speed of the agent at the time t, v max =25km / h indicates the maximum speed. represents speed-related environmental feedback, and They represent the environmental feedback related to the action and the terminal state respectively, and their values ​​are shown in Table 1. The output of the reward function model is described as a probability in the present invention according to the Bradley-Terry (BT) selection model:

[0012]

[0013] Where P reward [τ0>τ1] indicates the probability that the reward function model with parameter φ prefers trajectory segment τ0 more than τ1 in binary preference.

[0014] Table 1

[0015]

[0016] The time series processing module first processes the additional input including the unique hot encoding of the vehicle speed and the vehicle driving command through a multi-layer perceptron (MLP), then stacks the features output by the MLP processing with the features extracted by the backbone network, and initializes a set of learnable historical queries As the original input of the timing processing module. temporal The stacked feature order is passed through the temporal cross attention layer, self-attention layer, and feedforward neural network (FFN) to output the current query The present invention repeats this process three times, and in the new round of cross-attention, the current query is used as the historical query. It is worth noting that in the initial stage of the interaction, the first frame image and the first set of additional inputs will be reused. As the interaction proceeds, the input used changes and overlaps with the time frame. The proposed transformer-based temporal processing module uses 3 consecutive frames of images and 3 sets of additional inputs as module inputs starting from the third frame. Specifically:

[0017] In the first frame, the first frame image and the first set of additional input are used three times; the second frame uses the first frame image and the first set of additional input twice, and uses the second frame image and the second set of additional input; the third frame uses the first, second, and third frame images and the first, second, and third sets of additional input; the fourth frame uses the second, third, and fourth frame images and the second, third, and fourth sets of additional input.

[0018] For the training of imitation learning experts, we first use scene depth perception and scene semantic perception embedding to guide the bird's-eye view reconstruction process to output the bird's-eye view query through the semantic-depth heuristic decoder, and use the speed decoder, motion decoder, bird's-eye view decoder, object decoder, and reward function decoder to output speed prediction, motion prediction, bird's-eye view prediction, object detection, and reward feedback prediction respectively. Finally, through the specific experience sharing module, we train a highly applicable imitation learning expert.

[0019] The bird's-eye view reconstruction adds two additional tasks, depth estimation and semantic segmentation, to the bird's-eye view prediction process. Depth information and semantic information are used as labels combined with the attention mechanism to achieve depth-semantic heuristic BEV reconstruction. The specific bird's-eye view prediction process is as follows: First, the input of the model is a monocular RGB image. Where h×w represents the resolution of the image used, c=3 represents a 3-channel RGB image, and the present invention uses RegNety-32 as the backbone network and combines it with the Feature Pyramid Networks to obtain a multi-scale image feature map f image .f image It is then passed to three self-attention based encoders to obtain deep embedding, visual embedding, and semantic embedding, as follows Figure 1 After obtaining these embeddings, the present invention adopts a deep-semantic heuristic module based on a cross-attention mechanism to promote the interaction between different prediction tasks.

[0020] The deep-semantic inspired decoder includes two self-attention layers and three cross-attention layers. The present invention initializes a set of learnable bird's-eye view queries As the original input of the deep-semantic inspired decoder, N = 320 represents the number of queries and C = 256 represents the number of channels. The present invention first passes the bird's-eye view query through the self-attention layer to generate the internal connection of the query. Then the query is passed to the cross-attention layer, and the interactive bird's-eye view query is output as the basis for the aggregate embedding. This process can be defined as:

[0021] Q=w′ q q,K=w′ k f image ,V=w v ′f image

[0022]

[0023] Q′=w″(AV)

[0024] Where Q, K and V represent query, key and value respectively, w′ q , w′ k , w v ′ and w″ represent learnable parameters, A represents the attention map, h represents the number of multi-head attention, Q′ represents the output query, and the three cross-attention mechanisms adopt the same definition process. In order to further promote the interaction between different tasks, the present invention repeats the entire depth-semantic heuristic module three times, outputs the fully interactive bird's-eye view query, and splits the bird's-eye view query into motion query Q m and prediction query Q p Two parts, Q m The proposed timing processing module is used to process and output the processed current motion query Q current .

[0025] The speed decoder outputs a set of weights: The weighted four-mode speed [0, 7, 14 or 25 km / h] is used to obtain the predicted speed. The process is defined as:

[0026] λ v =MLP(FFN(crossaten(Q v =Q current ,K v V v =q v )))

[0027] Among them, Q current Indicates the current motion query, Q v , K v and V v Represents the query, key, and value of the speed decoder, respectively. represents a set of learnable velocity queries and crossaten represents the cross attention mechanism.

[0028] The motion decoder outputs a set of navigation point queries And output a set of navigation point predictions through the gated recurrent unit (GRU), where T = 10 represents the navigation point prediction time step, and the process is defined as:

[0029] Q waypoint =FFN(selfatten(crossaten(Q w =Q current ,K w V w=q w )))

[0030]

[0031] x t+1 =x t +△x t

[0032] Among them, Q w , K w and V w Represent the query, key, and value of the motion decoder, respectively. represents a set of learnable navigation point queries, △x t represents the expected coordinate change at time t, x t represents the coordinates of the agent at time t, x tt represents the target point of the agent at time t, x t+1 represents the coordinates of the agent at time t+1, and selfatten represents the self-attention mechanism.

[0033] The bird's-eye view decoder consists of a convolutional layer and an upsampling layer to output a bird's-eye view reconstruction result. where c b Indicates the number of categories.

[0034] The object decoder outputs a position heatmap prediction ∈ [0, 1] (64×64) , Direction pattern prediction And regression graph prediction. The regression graph prediction includes car size prediction Position offset prediction And direction offset prediction

[0035] The reward function decoder constructs a transformer-based reward function model to further process the query. First, an initial set of learnable reward queries Then we use a motion criss-cross attention layer and a prediction criss-cross attention layer to sequentially aggregate q r , the output Q of the timing processing module current With the prediction query Q p The output query is then passed to the self-attention layer for internal interaction aggregation. The aggregated query is passed to the feed-forward neural network (FFN) and outputs the reward value through the multi-layer perceptron (MLP) layer.

[0036] The specific experience sharing module trains the respective teacher models on different expert demonstration distributions, and aggregates the reward function decoder and semantic-depth heuristic decoder parameters according to the aggregation interval during the training process, thereby achieving specific experience sharing without transmitting samples.

[0037] For the distillation reinforcement learning student part, the reinforcement learning student is mainly trained according to the loss function distillation, and parameter aggregation is performed after completing local training. The loss functions include gradient descent loss, exploration loss, distillation loss, and suggestion loss.

[0038] The distillation loss The mean square error (MSE) loss is used to describe the distance between the feature matrix output by the reinforcement learning student and the imitation learning expert.

[0039]

[0040] where λ kd represents the weight of the distillation loss, Q student represents the BEV query of the reinforcement learning teacher, Q expert Represents a BEV query for an imitation learning expert.

[0041] The proposed loss

[0042]

[0043] Among them, λ sug represents the weight of the proposed loss, Indicates that at time TN z +1 to the indicator function at the end time T, KL(π θ (·|s RL )B sug ) indicates that in state input s RL The strategy π θ With the recommended strategy B sug The KL divergence between sug Represents the dynamic driving suggestions from the imitation learning expert to the reinforcement learning student, including longitudinal driving suggestions and lateral driving suggestions.

[0044] The present invention proposes longitudinal driving suggestion B longitudinal_sug Defined as:

[0045]

[0046] Among them, B go Indicates dynamic acceleration suggestions when the vehicle is blocked, B stop Indicates the dynamic deceleration suggestion before the vehicle collides or runs a red light, μ go =-0.04 indicates the recommended adjustment parameters for dynamic acceleration, μ stop =-0.025 indicates the adjustment parameter recommended by dynamic deceleration.

[0047] The present invention uses uniform distribution as the lateral driving suggestion B before the vehicle deviates from the lanehorizontal_sug , that is, B horizontal_sug =(1,1).

[0048] The parameter aggregation process:

[0049]

[0050] Among them, θ global represents the global student model parameters, θ student represents the local student model parameters, and n=6 represents the number of local student models.

[0051] The present invention also proposes a technical solution for a scenario-driven, highly generalized federated reinforcement learning autonomous driving method, which is performed in the following steps:

[0052] Step 1: Perform reinforcement learning student modeling, first perform feedback-based goal alignment, and then build a timing processing module.

[0053] Step 2: Perform imitation learning expert training. First, use scene depth perception and scene semantic perception embedding to guide the bird's-eye view reconstruction process. Then, use the semantic-depth heuristic decoder to output the bird's-eye view query. Use the speed decoder, motion decoder, bird's-eye view decoder, object decoder, and reward function decoder to output speed prediction, motion prediction, bird's-eye view prediction, object detection, and reward feedback prediction, respectively. Finally, train a high-applicability imitation learning expert through specific experience sharing.

[0054] Step 3: Distill the reinforcement learning student, mainly train the reinforcement learning student according to the loss function distillation, and perform parameter aggregation after completing the local training. The loss functions include gradient descent loss, exploration loss, distillation loss, and suggestion loss.

[0055] Preferably, in step 1, the target alignment process generates binary feedback by comparing the weighted cumulative rewards of the two trajectories, and establishes a preference model based on the true value of the reward and the expert demonstration:

[0056]

[0057]

[0058] where y(τ0,τ1)∈{(1,0),(0,1)} represents the binary feedback of the preference model for trajectory segments τ0 and τ1 of length H, i∈{0,1} represents a state-action pair The true value of the reward for i∈{0,1}, Denotes the trajectory segment τ i , driving behavior uncertainty weight of i∈{0,1}, Denotes the trajectory segment τi , the error weight of i∈{0,1}, Represents the trajectory segment τ of the agent at time t i ,i∈{0,1}, the lateral movement under Represents the trajectory segment τ of the agent at time t i , i∈{0,1}, longitudinal action under, △d(t) represents the lateral error of the agent from the expected path under the sample at time t, △θ(t) represents the heading angle deviation of the agent from the expected path under the sample at time t, and the reward true value set up:

[0059]

[0060] Where v(t) represents the speed of the agent at the time t, v desire (t) represents the expected speed of the agent under the sample at time t, represents the speed-related environmental feedback, v max =25km / h indicates the maximum speed. and They represent the environmental feedback related to the action and the terminal state respectively, and their values ​​are shown in Table 1. The output of the reward function model is described as a probability in the present invention according to the Bradley-Terry (BT) selection model:

[0061]

[0062] Where P reward [τ0>τ1] indicates the probability that the reward function model with parameter φ prefers trajectory segment τ0 more than τ1 in binary preference.

[0063] Table 1

[0064]

[0065]

[0066] Preferably, in step 1, the time series processing module first processes the additional input including the unique hot encoding of the vehicle speed and the vehicle driving command through a multi-layer perceptron (MLP), and then stacks the features output after the MLP processing with the features extracted by the backbone network. And initialize a set of learnable historical queries As the original input of the timing processing module. temporal The stacked features are sequentially passed through the time-series cross attention layer, the self-attention layer, and the feed-forward neural network (FFN) to output the current query. The present invention repeats this process three times, and in a new round of cross attention, the current query is used as a historical query. It is worth noting that in the initial stage of the interaction, the first frame image and the first set of additional inputs will be reused. As the interaction proceeds, the proposed transformer-based temporal processing module uses three consecutive frames of images and three sets of additional inputs as module inputs starting from the third frame.

[0067] Preferably, in step 2, the bird's-eye view reconstruction adds two additional tasks, depth estimation and semantic segmentation, to the bird's-eye view prediction process. Depth information and semantic information are used as labels combined with the attention mechanism to achieve depth-semantics-inspired BEV reconstruction. The specific bird's-eye view prediction process is as follows: first, the input of the model is a monocular RGB image. Where h×w represents the resolution of the image used, c=3 represents a 3-channel RGB image, RegNety-32 is used as the backbone network, and Feature Pyramid Networks is combined to obtain multi-scale image feature maps f image .f image It is then passed to three self-attention based encoders to obtain deep embedding, visual embedding, and semantic embedding, as follows Figure 2 After obtaining these embeddings, the present invention adopts a deep-semantic heuristic module based on a cross-attention mechanism to promote the interaction between different prediction tasks.

[0068] Preferably, in step 2, the deep-semantic heuristic decoder includes 2 self-attention layers and 3 cross-attention layers. The present invention initializes a set of learnable bird's-eye view queries As the original input of the inspiration module, N = 320 represents the number of queries and C = 256 represents the number of channels. The present invention first passes the bird's-eye view query through the self-attention layer to generate the internal connection of the query. Then the query is passed to the cross-attention layer as the basis for aggregate embedding. This process can be defined as:

[0069] Q=w′ q q,K=w′ k f image ,V=w v ′f image

[0070]

[0071] Q′=w″(AV)

[0072] Where Q, K and V represent query, key and value respectively, w′ q , w′ k , w v′ and w″ represent learnable parameters, A represents the attention map, h represents the number of multi-head attention, Q′ represents the output query, and the three cross-attention mechanisms adopt the same definition process. In order to further promote the interaction between different tasks, the present invention repeats the entire depth-semantic heuristic module three times, outputs the fully interactive bird's-eye view query, and splits the bird's-eye view query into motion query Q m and prediction query Q p Two parts, Q m The proposed timing processing module is used to process and output the processed current motion query Q current .

[0073] Preferably, in step 2, the speed decoder outputs a set of weights: The weighted four-mode speed [0, 7, 14 or 25 km / h] is used to obtain the predicted speed. The process is defined as:

[0074] λ v =MLP(FFN(crossaten(Q v =Q current ,K v V v =q v )))

[0075] Among them, Q current Indicates the current motion query, Q v , K v and V v Represents the query, key, and value of the speed decoder, respectively. represents a set of learnable velocity queries and crossaten represents the cross attention mechanism.

[0076] Preferably, in step 2, the motion decoder outputs a set of navigation point queries And output a set of navigation point predictions through the gated recurrent unit (GRU), where T = 10 represents the navigation point prediction time step, and the process is defined as:

[0077] Q waypoint =FFN(selfatten(crossaten(Q w =Q current ,K w V w =q w )))

[0078]

[0079] x t+1 =x t +△x t

[0080] Among them, Q w , K w and V w Represent the query, key, and value of the motion decoder, respectively. represents a set of learnable navigation point queries, △x t represents the expected coordinate change at time t, x t represents the coordinates of the agent at time t, x tt represents the target point of the agent at time t, x t+1 represents the coordinates of the agent at time t+1, and selfatten represents the self-attention mechanism.

[0081] Preferably, in step 2, the bird's-eye view decoder is composed of a convolutional layer and an upsampling layer to output a bird's-eye view reconstruction result. where c b Indicates the number of categories.

[0082] Preferably, in step 2, the object decoder outputs a position heat map prediction ∈ [0, 1] (64×64) , Direction pattern prediction And regression graph prediction. The regression graph prediction includes car size prediction Position offset prediction And direction offset prediction

[0083] Preferably, in step 2, the reward function decoder constructs a transformer-based reward function model to further process the query. First, an initial set of learnable reward queries is Then we use a motion criss-cross attention layer and a prediction criss-cross attention layer to sequentially aggregate q r , the output Q of the timing processing module current With the prediction query Q p The output query is then passed to the self-attention layer for internal interaction aggregation. The aggregated query is passed to the feed-forward neural network (FFN) and outputs the reward value through the multi-layer perceptron (MLP) layer.

[0084] Preferably, in step 2, the specific experience sharing is to train respective teacher models on different expert demonstration distributions, and during the training process, the reward function decoder and the semantic-depth heuristic decoder parameters are aggregated according to the aggregation interval, so as to achieve specific experience sharing without transmitting samples.

[0085] Preferably, in step 3, the distillation loss uses mean square error (MSE) loss to describe the distance between the feature matrix output by the reinforcement learning student and the imitation learning expert.

[0086]

[0087] where λ kd represents the weight of the distillation loss, Q student represents the BEV query of the reinforcement learning teacher, Q expert Represents a BEV query for an imitation learning expert.

[0088] Preferably, in step 3, the proposed loss is:

[0089]

[0090] Among them, λ sug represents the weight of the proposed loss, Indicates that at time TN z +1 to the indicator function at the end time T, KL(π θ (·|s RL )B sug ) indicates that in state input s RL The strategy π θ With the recommended strategy B sug The KL divergence between sug represents the dynamic driving suggestions of the imitation learning expert to the reinforcement learning student, including longitudinal driving suggestions and lateral driving suggestions. The present invention defines the longitudinal driving suggestions as:

[0091]

[0092] Among them, B go Indicates dynamic acceleration suggestions when the vehicle is blocked, B stop Indicates the dynamic deceleration suggestion before the vehicle collides or runs a red light, μ go =-0.04 indicates the recommended adjustment parameters for dynamic acceleration, μ stop =-0.025 indicates the adjustment parameter recommended by dynamic deceleration.

[0093] The present invention uses uniform distribution B horizontal_sug =(1,1) as a lateral driving suggestion before the vehicle leaves the lane.

[0094] Preferably, in step 3, the parameter aggregation process:

[0095]

[0096] Among them, θ global represents the global student model parameters, θ student represents the local student model parameters, and n=6 represents the number of local student models.

[0097] Beneficial effects of the present invention:

[0098] (1) By imitating the learning expert to learn implicit expert preferences from expert demonstrations, combined with the proposed dynamic driving suggestions, the reinforcement learning student feature extraction, reward function modeling, loss function construction, and group optimization multi-angle guidance are realized, which specifically solves the reinforcement learning target alignment problem and achieves high-reliability control performance in complex traffic environments relying only on a monocular camera;

[0099] (2) The proposed imitation learning expert enhances the connection between multiple learning tasks through cross-attention, and realizes the reconstruction process of the bird's-eye view under the guidance of scene depth perception and scene semantic perception embedding. In order to solve the problem of catastrophic forgetting in continuous learning, the imitation learning expert realizes the sharing of specific experience between different expert data distributions through cross-scene optimization, and trains highly universal imitation learning experts;

[0100] (3) Through the proposed reward function decoder, the imitation learning expert outputs motion planning, reward feedback, and dynamic driving suggestions as the basis for the training and optimization of the reinforcement learning student, so as to achieve its learning of driving rules. In view of the difficulty of representing high-dimensional environments, the time series processing module is combined with query-based knowledge distillation to improve the representation ability of reinforcement learning for raw image input. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] Figure 1 Schematic diagram of the temporal processing module based on the cross-attention mechanism proposed in the present invention;

[0102] Figure 2 Schematic diagram of the imitation learning teacher used in the present invention;

[0103] Figure 3 Schematic diagram of knowledge distillation of the teacher-student architecture proposed in this invention. DETAILED DESCRIPTION

[0104] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings, but the content of the present invention is not limited thereto.

[0105] The present invention provides a scenario-driven, highly generalized federated reinforcement learning autonomous driving system. Figure 1 As shown, highly generalized autonomous driving can be achieved. The specific implementation process includes the following steps:

[0106] (1) To model a reinforcement learning student, we first perform feedback-based goal alignment and then build a timing processing module.

[0107] The target alignment process generates binary feedback by comparing the weighted cumulative rewards of two trajectories and establishing a preference model based on the true value of the reward and the expert demonstration:

[0108]

[0109] where y(τ0,τ1)∈{(1,0),(0,1)} represents the binary feedback of the preference model for trajectory segments τ0 and τ1 of length H, i∈{0,1} represents a state-action pair The true value of the reward for i∈{0,1}, Denotes the trajectory segment τ i , driving behavior uncertainty weight of i∈{0,1}, Denotes the trajectory segment τ i , the error weight of i∈{0,1}, Represents the trajectory segment τ of the agent at time t i , lateral movement under i∈{0,1}, Represents the trajectory segment τ of the agent at time t i , i∈{0,1}, △d(t) represents the lateral error of the agent from the expected path at the time t, △θ(t) represents the heading angle deviation of the agent from the expected path at the time t, and the true reward value Set to:

[0110]

[0111]

[0112] Where v(t) represents the speed of the agent at the time t, v desire (t) represents the expected speed of the agent under the sample at time t, represents the speed-related environmental feedback, v max =25km / h indicates the maximum speed. and They represent the environmental feedback related to the action and the terminal state respectively, and their values ​​are shown in Table 1. The output of the reward function model is described as a probability in the present invention according to the Bradley-Terry (BT) selection model:

[0113]

[0114] Where P reward [τ0>τ1] indicates the probability that the reward function model with parameter φ prefers trajectory segment τ0 more than τ1 in binary preference.

[0115] Table 1

[0116]

[0117] The timing processing module, such as Figure 1As shown in the figure, the additional input including the unique hot encoding of vehicle speed and vehicle driving command is first processed by a multi-layer perceptron (MLP), and then the features output by the MLP processing are stacked with the features extracted by the backbone network. And a set of learnable historical queries is initialized As the original input of the timing processing module. temporal The stacked features are sequentially passed through the time-series cross attention layer, the self-attention layer, and the feed-forward neural network (FFN) to output the current query. The present invention repeats this process three times, and in a new round of cross attention, the current query is used as a historical query. It is worth noting that in the initial stage of the interaction, the first frame image and the first set of additional inputs will be reused. As the interaction proceeds, the proposed transformer-based temporal processing module uses three consecutive frames of images and three sets of additional inputs as module inputs starting from the third frame.

[0118] (2) Training imitation learning experts, such as Figure 2 As shown, scene depth perception and scene semantic perception embedding are first used to guide the bird's-eye view reconstruction process, the bird's-eye view query is output through the semantic-depth heuristic decoder, and the speed decoder, motion decoder, bird's-eye view decoder, object decoder, and reward function decoder are used to output speed prediction, motion prediction, bird's-eye view prediction, object detection, and reward feedback prediction respectively. Finally, through specific experience sharing, highly applicable imitation learning experts are trained.

[0119] The bird's-eye view reconstruction adds two additional tasks, depth estimation and semantic segmentation, to the bird's-eye view prediction process. Depth information and semantic information are used as labels combined with the attention mechanism to achieve depth-semantic heuristic BEV reconstruction. The specific bird's-eye view prediction process is as follows: First, the input of the model is a monocular RGB image. Where h×w represents the resolution of the image used, c=3 represents a 3-channel RGB image, and we use RegNety-32 as the backbone network and combine it with the Feature Pyramid Networks to obtain multi-scale image feature maps f image .f image It is then passed to three self-attention based encoders to obtain deep embedding, visual embedding, and semantic embedding, as follows Figure 2 After obtaining these embeddings, the present invention adopts a deep-semantic heuristic module based on a cross-attention mechanism to promote the interaction between different prediction tasks.

[0120] The deep-semantic inspired decoder includes two self-attention layers and three cross-attention layers. The present invention initializes a set of learnable bird's-eye view queries As the original input of the inspiration module, N = 320 represents the number of queries and C = 256 represents the number of channels. The present invention first passes the bird's-eye view query through the self-attention layer to generate the internal connection of the query. Then the query is passed to the cross-attention layer as the basis for aggregate embedding. This process can be defined as:

[0121] Q=w′ q q,K=w′ k f image ,V=w v ′f image

[0122]

[0123] Q′=w″(AV)

[0124] Where Q, K and V represent query, key and value respectively, w′ q , w′ k , w v ′ and w″ represent learnable parameters, A represents the attention map, h represents the number of multi-head attention, Q′ represents the output query, and the three cross-attention mechanisms adopt the same definition process. In order to further promote the interaction between different tasks, the present invention repeats the entire depth-semantic heuristic module three times, outputs the fully interactive bird's-eye view query, and splits the bird's-eye view query into motion query Q m and prediction query Q p Two parts, Q m The proposed timing processing module is used to process and output the processed current motion query Q current .

[0125] The velocity decoder outputs a set of weights The weighted four-modal speed [0, 7, 14 or 25 km / h] is used to obtain the predicted speed. The process is defined as:

[0126] λ v =MLP(FFN(crossaten(Q v =Q current ,K v V v =q v )))

[0127] Among them, Q current Indicates the current motion query, Q v , K v and V v Represents the query, key, and value of the speed decoder, respectively. Represents a set of learnable velocity queries.

[0128] The motion decoder outputs a set of navigation point queries And output a set of navigation point predictions through the gated recurrent unit (GRU), where T = 10 represents the navigation point prediction time step, and the process is defined as:

[0129] Q waypoint =FFN(selfatten(crossaten(Q w =Q current ,K w V w =q w )))

[0130]

[0131] x t+1 =x t +△x t

[0132] Among them, Q w , K w and V w Represent the query, key, and value of the motion decoder, respectively. represents a set of learnable navigation point queries, △x t represents the expected coordinate change at time t, x t represents the coordinates of the agent at time t, x tt represents the target point of the agent at time t, x t+1 Represents the coordinates of the agent at time t+1.

[0133] The bird's-eye view decoder consists of a convolutional layer and an upsampling layer to output a bird's-eye view reconstruction result. where c b Indicates the number of categories.

[0134] The object decoder outputs a position heatmap prediction ∈ [0, 1] (64×64) , Direction pattern prediction And regression graph prediction. The regression graph prediction includes car size prediction Position offset prediction And direction offset prediction

[0135] The reward function decoder constructs a transformer-based reward function model to further process the query. First, an initial set of learnable reward queries Then we use a motion criss-cross attention layer and a prediction criss-cross attention layer to sequentially aggregate q r , the output Q of the timing processing module current With the prediction query Q pThe output query is then passed to the self-attention layer for internal interaction aggregation. The aggregated query is passed to the feed-forward neural network (FFN) and outputs the reward value through the multi-layer perceptron (MLP) layer.

[0136] The specific experience sharing trains the respective teacher models on different expert demonstration distributions, and during the training process, aggregates the reward function decoder and the semantic-depth heuristic decoder parameters according to the aggregation interval, thereby achieving specific experience sharing without transmitting samples.

[0137] (3) Distillation reinforcement learning students, such as Figure 3 As shown, the reinforcement learning student is mainly trained according to the loss function distillation, and parameter aggregation is performed after completing the local training. The loss functions include gradient descent loss, exploration loss, distillation loss, and suggestion loss.

[0138] The distillation loss uses the mean square error (MSE) loss to describe the distance between the feature matrix output by the reinforcement learning student and the imitation learning expert.

[0139]

[0140] where λ kd represents the weight of the distillation loss, Q student represents the BEV query of the reinforcement learning teacher, Q expert Represents a BEV query for an imitation learning expert.

[0141] The proposed loss:

[0142]

[0143] Among them, λ sug represents the weight of the proposed loss, Indicates that at time TN z +1 to the indicator function at the end time T, KL(π θ (·|s RL )B sug ) indicates that in state input s RL The strategy π θ With the recommended strategy B sug The KL divergence between sug represents the dynamic driving suggestions of the imitation learning expert to the reinforcement learning student, including longitudinal driving suggestions and lateral driving suggestions. The present invention defines the longitudinal driving suggestions as:

[0144]

[0145] Among them, B go Indicates dynamic acceleration suggestions when the vehicle is blocked, B stopIndicates the dynamic deceleration suggestion before the vehicle collides or runs a red light, μ go =-0.04 indicates the recommended adjustment parameters for dynamic acceleration, μ stop =-0.025 represents the adjustment parameter of dynamic deceleration suggestion. The present invention uses uniform distribution B horizontal_sug =(1,1) as a lateral driving suggestion before the vehicle leaves the lane.

[0146] The parameter aggregation process:

[0147]

[0148] Among them, θ global represents the global student model parameters, θ student represents the local student model parameters, and n=6 represents the number of local student models.

[0149] In summary, the present invention proposes a scenario-driven highly generalized federated reinforcement learning autonomous driving system and method. By imitating learning experts to learn implicit expert preferences from expert demonstrations, combined with the proposed dynamic driving suggestions, the reinforcement learning student feature extraction, reward function modeling, loss function construction, and group optimization multi-angle guidance are realized, and the reinforcement learning target alignment problem is solved in a targeted manner. High-reliability control performance is achieved by relying only on a monocular camera in a complex traffic environment; the proposed imitation learning expert enhances the connection between multiple learning tasks through cross-attention, and realizes the reconstruction process of a bird's-eye view under the guidance of scene depth perception and scene semantic perception embedding. In view of the catastrophic forgetting problem in continuous learning, the specific experience sharing between different expert data distributions is realized through cross-scenario optimization, and highly universal imitation learning experts are trained; through the proposed reward function decoder, the federated teacher-student architecture outputs motion planning, reward feedback, and dynamic driving suggestions as the basis for training and optimization of reinforcement learning students, so as to realize their learning of driving rules. In view of the difficulty of high-dimensional environment representation, the federated teacher-student architecture combines the query-based knowledge distillation with the temporal processing module to improve the representation ability of reinforcement learning for raw image input.

[0150] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. All equivalent methods or changes that do not deviate from the technical creation of the present invention should be included in the scope of protection of the present invention.

Claims

1. A scenario-driven, highly generalizable federated reinforcement learning autonomous driving system, characterized by: include: Reinforcement learning student modeling part, training imitation learning expert part, and distillation reinforcement learning student part; The reinforcement learning student modeling part first constructs a feedback-based target alignment module and then constructs a timing processing module; The target alignment module generates binary feedback by comparing the weighted cumulative rewards of two trajectories, and establishes a preference model based on the true value of the reward and the expert demonstration; the time series processing module first processes the additional input including the unique hot encoding of the vehicle speed and the vehicle driving command through a multi-layer perceptron (MLP), then stacks the features output by the MLP processing with the features extracted by the backbone network, and initializes a set of learnable historical queries As the original input of the timing processing module, q temporal The stacked feature order is passed through the temporal cross attention layer, self-attention layer, and feedforward neural network (FFN) to output the current query The training of the imitation learning expert first uses scene depth perception and scene semantic perception embedding to guide the bird's-eye view reconstruction process, outputs the bird's-eye view query through a semantic-depth heuristic decoder, and uses a speed decoder, a motion decoder, a bird's-eye view decoder, an object decoder, and a reward function decoder to output speed prediction, motion prediction, bird's-eye view prediction, object detection, and reward feedback prediction respectively, and finally trains a high-applicability imitation learning expert through a specific experience sharing module; The distillation reinforcement learning student part trains the reinforcement learning student according to the loss function distillation, and performs parameter aggregation after completing the local training, wherein the loss function includes gradient descent loss, exploration loss, distillation loss, and suggestion loss.

2. The scenario-driven highly generalized federated reinforcement learning autonomous driving system according to claim 1, characterized in that: The specific implementation of the target alignment module includes the following: where y(τ0,τ1)∈{(1,0),(0,1)} represents the binary feedback of the preference model for trajectory segments τ0 and τ1 of length H, Representing state-action pairs The true value of the reward, Denotes the trajectory segment τ i The driving behavior uncertainty weight, Denotes the trajectory segment τ i The error weight of , where i∈{0,1}, represents the lateral action of the agent under the sample at time t, Represents the trajectory segment τ of the agent at time t i , i∈{0,1}, △d(t) represents the lateral error of the agent from the expected path at the time t, △θ(t) represents the heading angle deviation of the agent from the expected path at the time t, and the true reward value Set to: Where v(t) represents the speed of the agent at the time t, v desire (t) represents the expected speed of the agent under the sample at time t, represents the speed-related environmental feedback, v max Indicates the maximum speed, and Represent the action-related and terminal state-related environmental feedback respectively. According to the Bradley-Terry selection model, the output of the reward function model is described in the form of probability: Where P reward [τ0>τ1] indicates the probability that the reward function model with parameter φ prefers trajectory segment τ0 more than τ1 in binary preference.

3. The scenario-driven highly generalized federated reinforcement learning autonomous driving system according to claim 2, characterized in that: and The values ​​of are shown in Table 1: Table 1 4. The scenario-driven highly generalized federated reinforcement learning autonomous driving system according to claim 2, characterized in that: The timing processing module uses the current query as a historical query in a new round of cross-attention. In the initial stage of the interaction, the first frame image and the first set of additional inputs are reused. As the interaction proceeds, the timing processing module uses three consecutive frames of images and three sets of additional inputs as module inputs starting from the third frame.

5. The scenario-driven highly generalized federated reinforcement learning autonomous driving system according to claim 1, characterized in that: The bird's-eye view reconstruction is to add two additional tasks, depth estimation and semantic segmentation, to the bird's-eye view prediction process, using depth information and semantic information as labels combined with an attention mechanism to achieve depth-semantic heuristic BEV reconstruction; The deep-semantic inspired decoder, consisting of 2 self-attention layers and 3 cross-attention layers, initializes a set of learnable bird’s-eye view queries As the original input, N represents the number of queries and C represents the number of channels; first let the bird's-eye view query pass through the self-attention layer to generate the internal connection of the query; then pass the query to the cross-attention layer, and output the interactive bird's-eye view query as the basis for aggregate embedding. This process can be defined as: Q=w′ q q,K=w′ k f image ,V=w v ′f image Q′=w″(AV) Where Q, K and V represent query, key and value respectively, w q ′,w′ k , w v ′ and w″ represent learnable parameters, A represents the attention map, h represents the number of multi-head attention, Q′ represents the output query, and the three cross-attention mechanisms are the same. In order to further promote the interaction between different tasks, the present invention repeats the entire depth-semantic heuristic module 3 times, outputs the bird's-eye view query after full interaction, and splits the bird's-eye view query into motion query Q m and prediction query Q p Two parts, Q m The proposed timing processing module is used to process and output the processed current motion query Q current ; The speed decoder outputs a set of weights: The weighted four-modal velocity of is used to obtain the predicted velocity, and the process is defined as: λ v =MLP(FFN(crossaten(Q v =Q current ,K v 5 v =q v ))) Among them, Q current Indicates the current motion query, Q v , K v and V v Represents the query, key, and value of the speed decoder, respectively. represents a set of learnable velocity queries; The motion decoder outputs a set of navigation point queries And output a set of navigation point predictions through the gated recurrent unit (GRU), where T = 10 represents the navigation point prediction time step, and the process is defined as: Q waypoint =FFN(selfatten(crossaten(Q w =Q current ,K w 5 w =q w ))) x t+1 =x t +△x t Among them, Q w , K w and V w Represent the query, key, and value of the motion decoder, respectively. represents a set of learnable navigation point queries, △x t represents the expected coordinate change at time t, x t represents the coordinates of the agent at time t, x tt represents the target point of the agent at time t, x t+1 Represents the coordinates of the agent at time t+1; The bird's-eye view decoder consists of a convolutional layer and an upsampling layer to output a bird's-eye view reconstruction result. where c b Indicates the number of categories; The object decoder outputs a position heatmap prediction ∈ [0, 1] (64×64) , Direction pattern prediction and regression graph prediction; where the regression graph prediction includes car size prediction Position offset prediction And direction offset prediction The reward function decoder constructs a transformer-based reward function model to further process the query; first, an initial set of learnable reward queries Then we use a motion criss-cross attention layer and a prediction criss-cross attention layer to sequentially aggregate q r , the output Q of the timing processing module current With the prediction query Q p ,Then the output query is passed to the self-attention layer for internal interaction aggregation, and the aggregated query is passed to the feed-forward neural network (FFN) and outputs the reward value through the multi-layer perceptron (MLP) layer; The specific experience sharing module trains respective teacher models on different expert demonstration distributions, and during the training process, aggregates reward function decoder and semantic-depth heuristic decoder parameters according to the aggregation interval, thereby achieving specific experience sharing without transmitting samples.

6. The scenario-driven highly generalized federated reinforcement learning autonomous driving system according to claim 5, characterized in that: The bird's-eye view reconstruction process is as follows: First, the input of the model is a monocular RGB image Where h×w represents the resolution of the image used, c=3 represents a 3-channel RGB image, RegNety-32 is used as the backbone network, and the feature pyramid network is combined to obtain the multi-scale image feature map f image ;f image It is then passed to three self-attention-based encoders to obtain deep embedding, visual embedding, and semantic embedding, followed by a deep-semantic heuristic module based on a cross-attention mechanism to promote interaction between different prediction tasks.

7. The scenario-driven highly generalized federated reinforcement learning autonomous driving system according to claim 1, characterized in that: The distillation loss uses the mean square error (MSE) loss to describe the distance between the feature matrix output by the reinforcement learning student and the imitation learning expert. where λ kd represents the weight of the distillation loss, Q student represents the BEV query of the reinforcement learning teacher, Q expert Represents a BEV query for an imitation learning expert.

8. The scenario-driven highly generalized federated reinforcement learning autonomous driving system according to claim 1, characterized in that: The proposed loss Among them, λ sug represents the weight of the proposed loss, Indicates that at time TN z +1 to the indicator function at the end time T, KL(π θ (·|s RL )B sug ) indicates that in state input s RL The strategy π θ With the recommended strategy B sug The KL divergence between sug Represents dynamic driving advice from an imitation learning expert to a reinforcement learning student.

9. The scenario-driven highly generalized federated reinforcement learning autonomous driving system according to claim 1, characterized in that: The parameter aggregation process is as follows: Among them, θ global represents the global student model parameters, θ student represents the local student model parameters, and n represents the number of local student models.

10. A scenario-driven, highly generalizable federated reinforcement learning autonomous driving method, characterized in that: The steps include: Step 1: Perform reinforcement learning student modeling; first perform feedback-based goal alignment, and then build a timing processing module; Step 2: Perform imitation learning expert training; first, use scene depth perception and scene semantic perception embedding to guide the bird's-eye view reconstruction process, then use the semantic-depth heuristic decoder to output the bird's-eye view query, and use the speed decoder, motion decoder, bird's-eye view decoder, object decoder, and reward function decoder to output speed prediction, motion prediction, bird's-eye view prediction, object detection, and reward feedback prediction respectively. Finally, through specific experience sharing, train high-applicability imitation learning experts; Step 3: Distill the reinforcement learning student; train the reinforcement learning student according to the loss function distillation, and perform parameter aggregation after completing local training; the loss function includes gradient descent loss, exploration loss, distillation loss, and suggestion loss; In step 1, the target alignment process generates binary feedback by comparing the weighted cumulative rewards of the two trajectories, and establishes a preference model based on the true value of the reward and the expert demonstration: where y(τ0,τ1)∈{(1,0),(0,1)} represents the binary feedback of the preference model for trajectory segments τ0 and τ1 of length H, i∈{0,1} represents a state-action pair The true value of the reward for i∈{0,1}, Denotes the trajectory segment τ i , driving behavior uncertainty weight of i∈{0,1}, Denotes the trajectory segment τ i , the error weight of i∈{0,1}, Represents the trajectory segment τ of the agent at time t i , lateral movement under i∈{0,1}, Represents the trajectory segment τ of the agent at time t i , i∈{0,1}, △d(t) represents the lateral error of the agent from the expected path at the time t, △θ(t) represents the heading angle deviation of the agent from the expected path at the time t, and the true reward value Set to: Where v(t) represents the speed of the agent at the time t, v desire (t) represents the expected speed of the agent under the sample at time t, represents the speed-related environmental feedback, v max Indicates the maximum speed, and Represent the action-related and terminal state-related environmental feedback respectively. According to the Bradley-Terry (BT) selection model, the output of the reward function model is described in the form of probability: Where P reward [τ0>τ1] indicates the probability that the reward function model with parameter φ prefers the trajectory segment τ0 more than τ1 in the binary preference; The time series processing module in step 1: first, the additional input including the one-hot encoding of vehicle speed and vehicle driving command is processed by a multi-layer perceptron (MLP), then the features output by the MLP processing are stacked with the features extracted by the backbone network, and a set of learnable historical queries is initialized As the original input of the timing processing module, q temporal The stacked features are sequentially passed through the time-series cross attention layer, the self-attention layer, and the feed-forward neural network (FFN) to output the current query. This process is repeated three times, and in the next round of cross-attention, the current query is used as the historical query; in the initial stage of the interaction, the first frame image and the first set of additional inputs will be reused. As the interaction progresses, the temporal processing module uses three consecutive frames of images and three sets of additional inputs as module inputs starting from the third frame; In step 2, the bird's-eye view reconstruction is to add two additional tasks, depth estimation and semantic segmentation, to the bird's-eye view prediction process. The depth information and semantic information are used as labels combined with the attention mechanism to achieve depth-semantic heuristic BEV reconstruction. The specific bird's-eye view prediction process is as follows: First, the input of the model is a monocular RGB image Where h×w represents the resolution of the image used, c represents the number of channels of the RGB image, RegNety-32 is used as the backbone network, and the feature pyramid is combined to obtain the multi-scale image feature map f image , f image It is then passed to three self-attention-based encoders to obtain deep embedding, visual embedding, and semantic embedding. After obtaining these embeddings, a deep-semantic heuristic module based on the cross-attention mechanism is used to promote the interaction between different prediction tasks; In step 2, the deep-semantic inspired decoder, which contains 2 self-attention layers and 3 cross-attention layers, initializes a set of learnable bird's-eye view queries As the raw input of this module, where N represents the number of queries and C represents the number of channels, the bird’s-eye view query is first passed through the self-attention layer to generate the internal connections of the query, and then the query is passed to the cross-attention layer as the basis for the aggregate embedding. This process can be defined as: Q=w′ q q,K=w′ k f image ,V=w v ′f image Q′=w″(AV) Where Q, K and V represent query, key and value respectively, w′ q , w′ k , w v ′ and w″ represent learnable parameters, A represents the attention map, h represents the number of multi-head attention, Q′ represents the output query, and the three cross-attention mechanisms use the same definition process; in order to further promote the interaction between different tasks, the entire depth-semantic heuristic module is repeated 3 times, and the bird's-eye view query after full interaction is output, and the bird's-eye view query is split into motion query Q m and prediction query Q p Two parts, Q m The proposed timing processing module is used to process and output the processed current motion query Q current ; In step 2, the speed decoder outputs a set of weights: The weighted four-modal velocity of is used to obtain the predicted velocity, and the process is defined as: λ v =MLP(FFN(crossaten(Q v =Q current ,K v 5 v =q v ))) Among them, Q current Indicates the current motion query, Q v , K v and V v Represents the query, key, and value of the speed decoder, respectively. represents a set of learnable velocity queries; In step 2, the motion decoder outputs a set of navigation point queries And output a set of navigation point predictions through the gated recurrent unit (GRU), where T represents the navigation point prediction time step, and the process is defined as: Q waypoint =FFN(selfaten(crossaten(Q w =Q current ,K w 5 w =q w ))) x t+1 =x t +△x t Among them, Q w , K w and V w Represent the query, key, and value of the motion decoder, respectively. represents a set of learnable navigation point queries, △x t represents the expected coordinate change at time t, x t represents the coordinates of the agent at time t, x tt represents the target point of the agent at time t, x t+1 Represents the coordinates of the agent at time t+1; In step 2, the bird's-eye view decoder consists of a convolutional layer and an upsampling layer to output the bird's-eye view reconstruction result. where c b Indicates the number of categories; In step 2, the object decoder outputs a position heat map prediction ∈ [0, 1] (64×64) , Direction pattern prediction and regression graph predictions, where the regression graph predictions include car size predictions Position offset prediction And direction offset prediction In step 2, the reward function decoder constructs a transformer-based reward function model to further process the query. First, an initial set of learnable reward queries is Then we use a motion criss-cross attention layer and a prediction criss-cross attention layer to sequentially aggregate q r , the output Q of the timing processing module current With the prediction query Q p ; The output query is then passed to the self-attention layer for internal interaction aggregation; the aggregated query is passed to the feed-forward neural network (FFN) and outputs the reward value through the multi-layer perceptron (MLP) layer; In step 2, the specific experience sharing is to train the respective teacher models on different expert demonstration distributions, and to aggregate the reward function decoder and the semantic-depth heuristic decoder parameters according to the aggregation interval during the training process, so as to achieve specific experience sharing without transmitting samples; In step 3, the distillation loss uses mean square error (MSE) loss to describe the distance between the feature matrices output by the reinforcement learning student and the imitation learning expert: where λ kd represents the weight of the distillation loss, Q student represents the BEV query of the reinforcement learning teacher, Q expert represents the BEV query of the imitation learning expert; The proposed loss: Among them, λ sug represents the weight of the proposed loss, Indicates that at time TN z +1 to the indicator function at the end time T, KL(π θ (·|s RL )B sug ) indicates that in state input s RL The strategy π θ With the recommended strategy B sug The KL divergence between sug represents the dynamic driving suggestions from the imitation learning expert to the reinforcement learning student, including longitudinal driving suggestions and lateral driving suggestions; Define longitudinal driving advice as: Among them, B go Indicates dynamic acceleration suggestions when the vehicle is blocked, B stop Indicates the dynamic deceleration suggestion before the vehicle collides or runs a red light, μ go =-0.04 indicates the recommended adjustment parameters for dynamic acceleration, μ stop =-0.025 indicates the adjustment parameter for dynamic deceleration suggestion; Use uniform distribution B horizontal_sug =(1,1) as a lateral driving suggestion before the vehicle leaves the lane; In step 3, the parameter aggregation process: Among them, θ global represents the global student model parameters, θ student represents the local student model parameters, and n=6 represents the number of local student models.

Images