Scenario-driven highly generalized federated reinforcement learning autonomous driving system and method

Abstract

The application discloses a scene-driven high-generalization federated reinforcement learning automatic driving system and method, an imitation learning expert learns implicit expert preferences from expert demonstrations, combines dynamic driving suggestions to realize feature extraction of a reinforcement learning student, modeling of a reward function, construction of a loss function, multi-angle guidance of group optimization, and solves the problem of alignment of a reinforcement learning target; the imitation learning expert enhances the connection between multiple learning tasks through cross attention, realizes the bird's eye view reconstruction process under the guidance of scene depth perception and scene semantic perception embedding; through cross-scene optimization, the specific experience sharing between different expert data distributions is realized, and a high-universality imitation learning expert is trained; the reward function decoder outputs motion planning, reward feedback and dynamic driving suggestions as the basis for training and optimization of the reinforcement learning student, and the learning of driving rules is realized; through the time sequence processing module combined with knowledge distillation, the representation ability of the original image input is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of vehicle engineering and transportation engineering, and relates to a scene-driven high-generalization federated reinforcement learning automatic driving system and method. BACKGROUND

[0002] Urban driving is a difficult problem in the field of automatic driving due to its diversified road conditions, complex traffic conditions, and frequent interaction scenarios. In the urban driving environment, there is frequent interaction among traffic participants, road infrastructure, and traffic rules, which further tests the safety of automatic driving algorithms. Although rule-based methods have been successful in simple driving scenarios, the limitations of hand-crafted rules limit their application in complex traffic situations.

[0003] On the contrary, data-driven methods use a large amount of driving data to help agents learn driving strategies, without being bound by rules and logic written for specific situations. This type of method uses the fitting ability of deep neural networks combined with supervised or unsupervised training methods to achieve feature representation in high-dimensional state space. However, the sample coverage problem faced by data-driven methods results in a lack of training on out-of-distribution data, which can cause serious safety consequences. At the same time, the lack of transparency in the internal working principle of end-to-end methods increases the difficulty of framework explanation and subsequent optimization, causing a crisis of trust among people. Moreover, data-driven automatic driving systems rely on incremental learning in different scenarios, lacking the ability to integrate information across scenarios and the ability to optimize simultaneously. The scene-driven optimization problem caused by index conflict and experience coverage in the incremental optimization process of different scenarios can lead to catastrophic forgetting in continuous learning, reducing the system's generality and reliability in complex driving scenarios. SUMMARY

[0004] To solve the above problems, the application provides a scene-driven high generalization federated reinforcement learning automatic driving system and method, which learns implicit expert preferences from expert demonstrations through imitation learning experts, combines the proposed dynamic driving recommendations, realizes feature extraction of reinforcement learning students, reward function modeling, loss function construction, and multi-angle guidance of group optimization, and solves the reinforcement learning target alignment problem, and realizes high reliability control performance in complex traffic environment only relying on a monocular camera; the proposed imitation learning expert enhances the connection between multiple learning tasks through cross attention, realizes the bird's eye view reconstruction process under the guidance of scene depth perception and scene semantic perception embedding. In view of the problem of catastrophic forgetting in continuous learning, the specific experience sharing between different expert data distributions is realized through cross-scene optimization, and a high universality imitation learning expert is trained; through the proposed reward function decoder, the federated teacher-student architecture outputs motion planning, reward feedback and dynamic driving recommendations as the basis for reinforcement learning student training and optimization, and realizes the learning of driving rules. In view of the high-dimensional environment representation problem, the federated teacher-student architecture improves the representation ability of reinforcement learning for original image input through the time sequence processing module combined with query-based knowledge distillation.

[0005] The technical scheme of the scene-driven high generalization federated reinforcement learning automatic driving system provided by the application comprises three parts: reinforcement learning student modeling part, training imitation learning expert part, and distilling reinforcement learning student part.

[0006] For the reinforcement learning student modeling part, a feedback-based target alignment module is first constructed, and then a time sequence processing module is constructed.

[0007] The target alignment module produces binary feedback by comparing the weighted cumulative rewards of two trajectories, establishes a preference model based on reward true value and expert demonstration, and specifically as follows:

[0008]

[0009] Wherein y(τ0,τ1)∈{(1,0),(0,1)} represents the binary feedback of the preference model for the trajectory segments τ0 and τ1 with a length of H, i∈{0,1} represents the state-action pair i∈{0,1} reward true value, represents the driving behavior uncertainty weight of the trajectory segment τ i ,i∈{0,1}, represents the error weight of the trajectory segment τ i ,i∈{0,1}, represents the lateral action of the agent under the trajectory segment τ i ,i∈{0,1} at t time, denotes the trajectory segment of the agent at time t i denotes the longitudinal action under the condition of i element of {0, 1}, denotes the lateral error of the agent from the desired path at time t, denotes the heading angle deviation of the agent from the desired path at time t, and denotes the reward true value is set as:

[0010]

[0011] wherein v(t) denotes the speed of the agent at time t, v desire (t) denotes the desired speed of the agent at time t, and v max = 25 km / h denotes the maximum speed. denotes the speed-related environmental feedback, and denote the action-related and termination state-related environmental feedbacks respectively, and the values are shown in Table 1. The output of the reward function model is described in the form of probability according to the Bradley-Terry (BT) selection model:

[0012]

[0013] wherein P reward [τ0>τ1] denotes the probability that the reward function model with the parameter of φ prefers the trajectory segment τ0 to τ1 in the binary preference.

[0014] Table 1

[0015]

[0016] The time sequence processing module first processes the additional inputs including the one-hot encoding of the vehicle speed and the vehicle driving command through a multi-layer perceptron (MLP), then stacks the features outputted after the MLP processing with the backbone network extracted features, and initializes a set of learnable historical queries as the original input of the time sequence processing module. q temporal After the stacked features sequentially pass through the time sequence cross-attention layer, the self-attention layer and the feed-forward neural network (FFN), the current query The present application repeats this process for 3 times, and in the new round of cross-attention, the current query is used as the historical query. It is worth noting that in the initial stage of interaction, the first frame of image and the first set of additional inputs will be reused. As the interaction proceeds, the used inputs change and cover with the time frame rolling, and the proposed transformer-based time sequence processing module uses 3 consecutive frames of images and 3 sets of additional inputs as the module input from the third frame, specifically:

[0017] The first frame uses the first frame image three times and the first set of additional inputs; the second frame uses the first frame image twice and the first set of additional inputs, and uses the second frame image and the second set of additional inputs; the third frame uses the first, second and third frame images and the first, second and third sets of additional inputs; and the fourth frame uses the second, third and fourth frame images and the second, third and fourth sets of additional inputs.

[0018] For training the imitation learning expert part, firstly, the scene depth perception and scene semantic perception embedding are used to guide the bird's eye view reconstruction process to output the bird's eye view query through a semantic-depth heuristic decoder, and a speed decoder, a motion decoder, a bird's eye view decoder, an object decoder and a reward function decoder are used to respectively output speed prediction, motion prediction, bird's eye view prediction, object detection and reward feedback prediction, and finally a specific experience sharing module is used to train a high adaptability imitation learning expert.

[0019] The bird's eye view reconstruction additionally adds depth estimation and semantic segmentation tasks in the bird's eye view prediction process. Depth information and semantic information are used as labels combined with an attention mechanism to realize depth-semantic heuristic BEV reconstruction. The specific bird's eye view prediction process is that the input of the model is a monocular RGB image where h x w represents the resolution of the used image, c = 3 represents a 3-channel RGB image, the application adopts RegNety-32 as the backbone network, and combines a feature pyramid (Feature Pyramid Networks) to obtain a multi-scale image feature map f image . image Subsequently, it is passed to three self-attention-based encoders to obtain depth embedding, visual embedding and semantic embedding, as shown in the process Figure 1 After obtaining these embeddings, the application uses a depth-semantic heuristic module based on a cross-attention mechanism to promote the interaction between different prediction tasks.

[0020] The depth-semantic heuristic decoder includes two self-attention layers and three cross-attention layers, and the application initializes a set of learnable bird's eye view queries as the original input of the depth-semantic heuristic decoder, where N = 320 represents the number of queries, and C = 256 represents the number of channels. The application firstly lets the bird's eye view query pass through the self-attention layer to produce the internal contact of the query. Then the query is passed to the cross-attention layer to output the interactive bird's eye view query as the basis for aggregated embedding. This process can be defined as:

[0021] Q = w' q q, K = w' k f image , V = w v 'f image

[0022]

[0023] Q' = w"(AV)

[0024] where Q, K and V represent query, key and value respectively, w' q , w' k , w v and w" represent learnable parameters, A represents attention map, h represents the number of multi-head attention, Q' represents output query, the three cross attention mechanisms adopt the same definition process. In order to further promote the interaction between different tasks, the whole deep-semantics heuristic module is repeated 3 times, the output is the bird's eye view query after sufficient interaction, and the bird's eye view query is split into motion query Q m and prediction query Q p Two parts, Q m are processed by the proposed timing processing module, and the processed current motion query Q current .

[0025] The speed decoder outputs a set of weighted four-modal speed [0, 7, 14 or 25 km / h] with weight to obtain the predicted speed, and the process is defined as:

[0026] λ v = MLP(FFN(crossaten(Q v = Q current , K v V v = q v ))

[0027] where Q current represents the current motion query, Q v , K v and V v represent the query, key and value of the speed decoder respectively, represents a set of learnable speed queries, and crossaten represents cross attention mechanism.

[0028] The motion decoder outputs a set of navigation point queries and outputs a set of navigation point predictions through a gated recurrent unit (GRU), where T = 10 represents the navigation point prediction step, and the process is defined as:

[0029] Q waypoint = FFN(selfatten(crossaten(Q w = Q current , K w V w= q w )))

[0030]

[0031] x t+1 = x t +△x t

[0032] where Q w , K w and V w represent the query, key, and value of the motion decoder, represent a set of learnable navigation point queries,△x t represent the expected coordinate change at time t, x t represent the agent coordinate at time t, x tt represent the agent target point at time t, x t+1 represent the agent coordinate at time t+1, selfatten represents the self-attention mechanism.

[0033] The bird's eye view decoder is composed of convolutional layers and up-sampling layers to output the bird's eye view reconstruction result where c b represents the number of categories.

[0034] The object decoder outputs the position heat map prediction ∈ [0, 1] (64×64) , the direction map prediction and the regression map prediction. The regression map prediction includes the car size prediction the position offset prediction and the direction offset prediction

[0035] The reward function decoder constructs a transformer-based reward function model for further processing of the query. First, a set of learnable reward queries is initialized Then, a motion cross-attention layer and a prediction cross-attention layer are used to sequentially aggregate q r , the output of the time processing module Q current and the prediction query Q p . Subsequently, the output query is passed to the self-attention layer for internal interaction aggregation. The aggregated query is passed to the feed-forward neural network (FFN) and output through the multi-layer perceptron (MLP) layer to obtain the reward value.

[0036] The specificity experience sharing module trains respective teacher models on different expert demonstration distributions. During the training process, the reward function decoder and the semantic-depth heuristic decoder parameters are aggregated according to the aggregation interval. Thus, specificity experience sharing is achieved without transmitting samples.

[0037] For the distillation reinforcement learning student part, the reinforcement learning student is mainly trained by distillation according to the loss function, and the parameters are aggregated after local training is completed. The loss function includes gradient descent loss, exploration loss, distillation loss, and proposal loss.

[0038] distillation loss The mean squared error (MSE) loss is used to describe the distance between the feature matrices output by the reinforcement learning student and the imitation learning expert.

[0039]

[0040] Where λ kd Q represents the weight of distillation loss. student The BEV query for reinforcement learning teachers, Q expert This indicates a BEV query that imitates learning experts.

[0041] The suggested loss

[0042]

[0043] Where, λ sug This indicates the weight of the suggested loss. Indicates time TN z The indicator function from +1 to the termination time T, KL(π) θ (·|s RL B sug ) indicates the state input s RL The following strategy π θ Recommended Strategy B sug KL divergence between B sug This refers to the dynamic driving suggestions from imitation learning experts for reinforcement learning students, including longitudinal driving suggestions and lateral driving suggestions.

[0044] This invention will provide longitudinal driving suggestion B longitudinal_sug Defined as:

[0045]

[0046] Among them, B go B indicates dynamic acceleration suggestions when the vehicle is blocked. stop This indicates dynamic deceleration recommendations before a vehicle is involved in a collision or runs a red light. go = -0.04 indicates the recommended adjustment parameter for dynamic acceleration, μ stop =-0.025 indicates the adjustment parameter recommended for dynamic deceleration.

[0047] This invention uses a uniform distribution as lateral driving advice B before a vehicle deviates from its lane.horizontal_sug i.e. B horizontal_sug = (1, 1).

[0048] The parameter aggregation process:

[0049]

[0050] where θ global represents global student model parameters, θ student represents local student model parameters, and n = 6 represents the number of local student models.

[0051] The application also proposes a technical scheme of a scenario-driven high-generalization federated reinforcement learning automatic driving method, which is executed according to the following steps:

[0052] Step 1: Reinforcement learning student modeling is performed, first, a feedback-based goal alignment is performed, and then a time series processing module is constructed.

[0053] Step 2: Imitation learning expert training is performed, first, a scene depth perception and scene semantic perception embedding is adopted to guide the bird's eye view reconstruction process, then a semantic-depth heuristic decoder is used to output a bird's eye view query, and a speed decoder, a motion decoder, a bird's eye view decoder, an object decoder, and a reward function decoder are used to respectively output speed prediction, motion prediction, bird's eye view prediction, object detection, and reward feedback prediction, finally, a specific experience sharing is performed to train a high-applicability imitation learning expert.

[0054] Step 3: Distillation of reinforcement learning students, mainly according to a loss function, a reinforcement learning student is trained, and after local training is completed, parameter aggregation is performed, the loss function includes a gradient descent loss, an exploration loss, a distillation loss, and a suggestion loss.

[0055] Preferably, in step 1, the goal alignment process produces binary feedback by comparing the weighted cumulative rewards of two trajectories, and a preference model based on reward true value and expert demonstration is established:

[0056]

[0057]

[0058] where y(τ0,τ1)∈{(1,0),(0,1)} represents the binary feedback of the preference model for trajectory segments τ0 and τ1 with a length of H, i∈{0,1} represents the reward true value of a state-action pair i∈{0,1}, represents the driving behavior uncertainty weight of trajectory segment τ i ,i∈{0,1}, represents the driving behavior uncertainty weight of trajectory segment τi error weights for i∈{0,1}, denotes the lateral action of the agent at time t for trajectory segment τ i error weights for i∈{0,1}, denotes the longitudinal action of the agent at time t for trajectory segment τ i error weights for i∈{0,1}, denotes the lateral error of the agent from the desired path at time t for sample s, denotes the heading angle deviation of the agent from the desired path at time t for sample s, reward ground truth is set as:

[0059]

[0060] where v(t) denotes the speed of the agent at time t for sample s, v desire (t) denotes the desired speed of the agent at time t for sample s, denotes the speed-related environment feedback, v max = 25 km / h denotes the maximum speed. and denote the action-related and termination state-related environment feedbacks, respectively, with values as shown in Table 1. The output of the reward function model is described in the form of probability according to the Bradley-Terry (BT) choice model:

[0061]

[0062] where P reward [τ0>τ1] denotes the probability that the reward function model with parameter φ prefers trajectory segment τ0 over τ1 in binary preference.

[0063] Table 1

[0064]

[0065]

[0066] Preferably, in step 1, the time series processing module first processes the additional inputs including the one-hot encoding of the vehicle speed and the vehicle driving command through a multi-layer perceptron (MLP), and then stacks the features outputted after the MLP processing with the backbone network extracted features. And initialize a set of learnable historical queries as the original input of the time series processing module. q temporal After the stacked features sequentially pass through the time series cross-attention layer, the self-attention layer and the feed-forward neural network (FFN), the current query The present application repeats this process 3 times, and in the new round of cross-attention, the current query is used as the historical query. It is worth noting that in the initial stage of interaction, the first frame of image and the first set of additional inputs will be reused. As the interaction proceeds, the proposed transformer-based timing processing module uses 3 consecutive frames of images and 3 sets of additional inputs as module inputs from the third frame.

[0067] Preferably, in step 2, the aerial view reconstruction additionally increases two tasks of depth estimation and semantic segmentation in the aerial view prediction process. The depth information and semantic information are used as labels combined with attention mechanism to realize the depth-semantic heuristic BEV reconstruction. The specific aerial view prediction process is that the input of the model is a monocular RGB image where h x w represents the resolution of the used image, c = 3 represents a 3-channel RGB image, RegNety-32 is used as the backbone network, and a feature pyramid (Feature Pyramid Networks) is combined to obtain multi-scale image feature maps f image . f image Subsequently, it is passed to 3 self-attention-based encoders to obtain depth embedding, visual embedding, and semantic embedding, as shown in the process Figure 2 After obtaining these embeddings, the present application uses a depth-semantic heuristic module based on cross-attention mechanism to promote the interaction between different prediction tasks.

[0068] Preferably, in step 2, the depth-semantic heuristic decoder includes 2 self-attention layers and 3 cross-attention layers, and the present application initializes a set of learnable aerial view queries as the original input of the heuristic module, where N = 320 represents the number of queries, and C = 256 represents the number of channels. The present application first lets the aerial view query pass through the self-attention layer to produce the internal contact of the query. Then the query is passed to the cross-attention layer as the basis for aggregated embedding. This process can be defined as:

[0069] Q = w' q q, K = w' k f image , V = w v 'f image

[0070]

[0071] Q' = w"(AV)

[0072] where Q, K, and V represent the query, the key, and the value, respectively, w' q , w' k , w vand w" represent learnable parameters, A represents attention map, h represents the number of multi-head attention, Q' represents output query, and the three cross attention mechanisms adopt the same definition process. In order to further promote the interaction between different tasks, the whole deep-semantics heuristic module is repeated 3 times, the aerial view query after sufficient interaction is output, and the aerial view query is split into motion query Q m and prediction query Q p Two parts, Q m are processed by the proposed timing processing module, and the processed current motion query Q current .

[0073] Preferably, in step 2, the speed decoder outputs a set of weighted four-modal speed [0, 7, 14 or 25 km / h] with weight to obtain the predicted speed, and the process is defined as:

[0074] λ v = MLP (FFN (crossaten (Q v = Q current , K v V v = q v ))

[0075] Wherein, Q current represents the current motion query, Q v , K v and V v respectively represent the query, key and value of the speed decoder, represents a set of learnable speed queries, and crossaten represents cross attention mechanism.

[0076] Preferably, in step 2, the motion decoder outputs a set of navigation point queries and outputs a set of navigation point predictions through a gated recurrent unit (GRU), wherein T = 10 represents the navigation point prediction step, and the process is defined as:

[0077] Q waypoint = FFN (selfatten (crossaten (Q w = Q current , K w V w = q w ))

[0078]

[0079] x t+1 = x t +△x t

[0080] where Q w , K w and V w represent the query, key and value of the motion decoder respectively, represent a set of learnable navigation point queries,△x t represents the expected coordinate change at time t, x t represents the agent coordinate at time t, x tt represents the agent target point at time t, x t+1 represents the agent coordinate at time t+1, selfatten represents the self-attention mechanism.

[0081] Preferably, in step 2, the aerial view decoder is composed of a convolutional layer and an up-sampling layer to output the aerial view reconstruction result where c b represents the number of categories.

[0082] Preferably, in step 2, the object decoder outputs the position heat map prediction ∈ [0, 1] (64×64) , the direction map prediction and the regression map prediction. The regression map prediction includes the car size prediction the position offset prediction and the direction offset prediction

[0083] Preferably, in step 2, the reward function decoder constructs a transformer-based reward function model for further processing of the query. First, a set of learnable reward queries is initialized Then, a motion cross-attention layer and a prediction cross-attention layer are used to sequentially aggregate q r , the output of the time sequence processing module Q current and the prediction query Q p . Subsequently, the output query is passed to a self-attention layer for internal interaction aggregation. The aggregated query is passed to a feed-forward neural network (FFN) and output through a multi-layer perceptron (MLP) layer to obtain the reward value.

[0084] Preferably, in step 2, the specific experience sharing trains respective teacher models on different expert demonstration distributions, and aggregates the parameters of the reward function decoder and the semantic-depth heuristic decoder according to the aggregation interval during the training process. Thus, specific experience sharing is achieved without transmitting samples.

[0085] Preferably, in step 3, the distillation loss adopts a mean square error (MSE) loss to describe the distance between the feature matrices output by the reinforcement learning student and the imitation learning expert

[0086]

[0087] Where λ kd Q represents the weight of distillation loss. student The BEV query for reinforcement learning teachers, Q expert This indicates a BEV query that imitates learning experts.

[0088] Preferably, in step 3, the suggested loss is:

[0089]

[0090] Where, λ sug This indicates the weight of the suggested loss. Indicates time TN z The indicator function from +1 to the termination time T, KL(π) θ (·|s RL B sug ) indicates the state input s RL The following strategy π θ Recommended Strategy B sug KL divergence between B sug This invention represents the dynamic driving suggestions from imitation learning experts to reinforcement learning students, including longitudinal driving suggestions and lateral driving suggestions. The longitudinal driving suggestion is defined as follows:

[0091]

[0092] Among them, B go B indicates dynamic acceleration suggestions when the vehicle is blocked. stop This indicates dynamic deceleration recommendations before a vehicle is involved in a collision or runs a red light. go = -0.04 indicates the recommended adjustment parameter for dynamic acceleration, μ stop =-0.025 indicates the adjustment parameter recommended for dynamic deceleration.

[0093] This invention uses uniformly distributed B horizontal_sug = (1,1) serves as a lateral driving suggestion before the vehicle deviates from the lane.

[0094] Preferably, in step 3, the parameter aggregation process is as follows:

[0095]

[0096] Where, θ global θ represents the global student model parameters. student This represents the parameters of the local student model, and n=6 represents the number of local student models.

[0097] The beneficial effects of this invention are:

[0098] (1) By imitating learning experts from expert demonstrations to learn implicit expert preferences, combined with the proposed dynamic driving suggestions, the reinforcement learning student feature extraction, reward function modeling, loss function construction, and multi-angle guidance of group optimization are realized, and the reinforcement learning goal alignment problem is solved, and high reliability control performance is realized in complex traffic environment only relying on monocular camera;

[0099] (2) The proposed imitation learning expert enhances the connection between multiple learning tasks through cross-attention, realizes the bird's eye view reconstruction process under the guidance of scene depth perception and scene semantic perception embedding. In order to solve the problem of catastrophic forgetting in continuous learning, the imitation learning expert realizes the sharing of specific experience between different expert data distributions through cross-scene optimization, and trains a high universality imitation learning expert;

[0100] (3) Through the proposed reward function decoder, the imitation learning expert outputs motion planning, reward feedback and dynamic driving suggestions as the basis for reinforcement learning student training and optimization, and realizes its learning of driving rules. In order to solve the problem of high-dimensional environment representation, through the time sequence processing module combined with query-based knowledge distillation, the representation ability of reinforcement learning for original image input is improved. BRIEF DESCRIPTION OF DRAWINGS

[0101] Figure 1 The time sequence processing module based on cross-attention mechanism proposed in the application is shown in the figure;

[0102] Figure 2 The imitation learning teacher used in the application is shown in the figure;

[0103] Figure 3 The teacher-student architecture knowledge distillation proposed in the application is shown in the figure. DETAILED DESCRIPTION

[0104] The technical solutions of the application will be described in detail below with reference to the accompanying drawings, but the content of the application is not limited thereto.

[0105] The application provides a scene-driven high generalization federated reinforcement learning automatic driving system, as shown in Figure 1 The specific implementation process includes the following steps:

[0106] (1) Reinforcement learning student modeling is performed, first based on feedback goal alignment, and then constructing a time sequence processing module.

[0107] The goal alignment process produces binary feedback by comparing the weighted cumulative rewards of two trajectories, and establishes a preference model based on reward true value and expert demonstration:

[0108]

[0109] where y (τ0, τ1) e {(1, 0), (0, 1)} represents the binary feedback of the preference model for trajectory segments τ0and τ1of length H, i e {0, 1} represents a state-action pair i e {0, 1} represents the reward truth value, represents the driving behavior uncertainty weight of trajectory segment τ i i e {0, 1}, represents the error weight of trajectory segment τ i i e {0, 1}, represents the lateral action of the agent under trajectory segment τ i i e {0, 1} at time t, represents the longitudinal action of the agent under trajectory segment τ i i e {0, 1} at time t, △d(t) represents the lateral error of the agent under the sample at time t from the expected path, △θ(t) represents the deviation of the heading angle of the agent under the sample at time t from the expected path, and the reward truth value is set as:

[0110]

[0111]

[0112] where v(t) represents the speed of the agent under the sample at time t, v desire (t) represents the expected speed of the agent under the sample at time t, represents the speed-related environmental feedback, v max = 25 km / h represents the maximum speed. and respectively represent the action-related and termination state-related environmental feedbacks, and the values are shown in Table 1. According to the Bradley-Terry (BT) selection model, the output of the reward function model is described in the form of probability by the present application:

[0113]

[0114] where P reward [τ0>τ1] represents the probability that the preference of the trajectory segment τ0to the trajectory segment τ1by the reward function model with parameter φ is greater than the preference in the binary preference.

[0115] Table 1

[0116]

[0117] The timing processing module, such as Figure 1As shown, first, the additional inputs including vehicle speed and vehicle driving commands are processed by a multi-layer perceptron (MLP), and then the features outputted by the MLP are stacked with the backbone network extracted features. And a set of learnable history queries as the original input of the time series processing module. temporal After the stacked features sequentially pass through the time cross-attention layer, the self-attention layer and the feed-forward neural network (FFN), the current query The present application repeats this process 3 times, and in the new round of cross-attention, the current query is used as the history query. It is worth noting that in the initial stage of interaction, the first frame of image and the first set of additional inputs will be reused. The proposed transformer-based time series processing module uses 3 consecutive frames of images and 3 sets of additional inputs as module inputs from the third frame.

[0118] (2) Train the imitation learning expert, such as Figure 2 As shown, first, the scene depth perception and scene semantic perception embedding are used to guide the bird's eye view reconstruction process, the bird's eye view query is outputted by the semantic-depth heuristic decoder, and the speed decoder, the motion decoder, the bird's eye view decoder, the object decoder, and the reward function decoder are used to output the speed prediction, the motion prediction, the bird's eye view prediction, the object detection, and the reward feedback prediction respectively, and finally the high adaptability imitation learning expert is trained through specific experience sharing.

[0119] The bird's eye view reconstruction additionally adds depth estimation and semantic segmentation tasks in the bird's eye view prediction process. Depth information and semantic information are used as labels combined with attention mechanism to realize depth-semantic heuristic BEV reconstruction. The specific bird's eye view prediction process is that the input of the model is a monocular RGB image Where h x w represents the resolution of the used image, c = 3 represents a 3-channel RGB image, we use RegNety-32 as the backbone network, and combine it with the feature pyramid (Feature Pyramid Networks) to obtain multi-scale image feature maps f image . f image Subsequently, it is passed to 3 self-attention-based encoders to obtain depth embedding, visual embedding and semantic embedding, as shown in the process Figure 2 After obtaining these embeddings, the present application uses a depth-semantic heuristic module based on cross-attention mechanism to promote the interaction between different prediction tasks.

[0120] The depth-semantic heuristic decoder includes 2 self-attention layers and 3 cross-attention layers, and the present application initializes a set of learnable bird's eye view queries As the original input of the inspiration module, where N = 320 represents the number of queries, and C = 256 represents the number of channels. The invention first lets the aerial view query pass through the self-attention layer to generate the internal contact of the query. Then the query is passed to the cross-attention layer as the basis for aggregated embedding. This process can be defined as:

[0121] Q = w' q q, K = w' k f image , V = w v 'f image

[0122]

[0123] Q' = w"(AV)

[0124] where Q, K and V represent the query, key and value respectively, w' q , w' k , w v ' and w" represent learnable parameters, A represents the attention map, h represents the number of multi-head attention, Q' represents the output query, and the three cross-attention mechanisms use the same definition process. In order to further promote the interaction between different tasks, the invention repeats the entire deep-semantics inspiration module 3 times, outputs the aerial view query after sufficient interaction, and splits the aerial view query into motion query Q m and prediction query Q p Two parts, Q m are processed by the proposed timing processing module, and the processed current motion query Q current is output.

[0125] The speed decoder outputs a set of weights to weight the four modal speeds [0, 7, 14 or 25 km / h] to obtain the predicted speed, and the process is defined as:

[0126] λ v = MLP(FFN(crossaten(Q v = Q current , K v V v = q v )))

[0127] where Q current represents the current motion query, Q v , K v and V v represent the query, key and value of the speed decoder respectively, represent a set of learnable speed queries.

[0128] The motion decoder outputs a set of navigation point queries and outputs a set of navigation point predictions through a gated recurrent unit (GRU), where T = 10 represents the navigation point prediction time step, and the process is defined as:

[0129] Q waypoint = FFN (selfatten (crossatten (Q w = Q current , K w V w = q w )))

[0130]

[0131] x t+1 = x t +△x t

[0132] where Q w , K w and V w represent the query, key, and value of the motion decoder, respectively, represents a set of learnable navigation point queries,△x t represents the expected coordinate change at time t, x t represents the agent coordinate at time t, x tt represents the agent target point at time t, x t+1 represents the agent coordinate at time t+1.

[0133] The overhead view decoder is composed of convolutional layers and up-sampling layers to output an overhead view reconstruction result where c b represents the number of categories.

[0134] The object decoder outputs position heat map prediction ∈ [0, 1] (64×64) , direction map prediction and regression map prediction. The regression map prediction includes car size prediction position offset prediction and direction offset prediction

[0135] The reward function decoder constructs a transformer-based reward function model for further processing of the query. First, a set of learnable reward queries is initialized Then, a motion cross-attention layer and a prediction cross-attention layer are used to sequentially aggregate q r , the output of the time series processing module Q current and the prediction query Q pThe output query is then passed to a self-attention layer for internal interaction aggregation. The aggregated query is passed to a feed-forward neural network (FFN) and outputted with a reward value through a multi-layer perceptron (MLP) layer.

[0136] The specific experience sharing trains respective teacher models on different expert demonstration distributions, and aggregates reward function decoder and semantic-depth heuristic decoder parameters according to an aggregation interval during the training process. Thus, specific experience sharing is realized without sample transmission.

[0137] (3) Distillation reinforcement learning students, such as Figure 3 As shown, the reinforcement learning students are mainly trained according to a loss function distillation, and parameter aggregation is performed after local training is completed, and the loss function includes gradient descent loss, exploration loss, distillation loss, and suggestion loss.

[0138] The distillation loss uses a mean square error (MSE) loss to describe the distance between the feature matrices outputted by the reinforcement learning student and the imitation learning expert

[0139]

[0140] where λ kd represents the weight of the distillation loss, Q student represents the BEV query of the reinforcement learning teacher, Q expert represents the BEV query of the imitation learning expert.

[0141] The suggestion loss is:

[0142]

[0143] where λ sug represents the weight of the suggestion loss, represents an indicator function from time T-N z +1 to the end time T, KL(π θ (·|s RL )B sug represents the KL divergence between the policy π RL and the suggestion policy B sug under the state input s θ , and B sug represents the dynamic driving suggestion of the imitation learning expert to the reinforcement learning student, including longitudinal driving suggestion and lateral driving suggestion. The longitudinal driving suggestion is defined as:

[0144]

[0145] where B go represents a dynamic acceleration suggestion when the vehicle is blocked, and B stoprepresents dynamic deceleration suggestion before the vehicle collision or red light running behavior, μ go represents the adjustment parameter of dynamic acceleration suggestion, μ stop represents the adjustment parameter of dynamic deceleration suggestion. The present application uses uniform distribution B horizontal_sug =(1,1) as the lateral driving suggestion before the vehicle deviates from the lane.

[0146] The parameter aggregation process is:

[0147]

[0148] wherein θ global represents global student model parameters, θ student represents local student model parameters, and n=6 represents the number of local student models.

[0149] In summary, the present application proposes a scenario-driven high generalization federated reinforcement learning automatic driving system and method. Through the imitation learning expert, the implicit expert preferences are learned from the expert demonstration, and the dynamic driving suggestion is combined to realize the feature extraction of the reinforcement learning student, the reward function modeling, the loss function construction, and the multi-angle guidance of group optimization, to solve the reinforcement learning target alignment problem. In the complex traffic environment, only a monocular camera is used to realize high reliability control performance. The proposed imitation learning expert enhances the connection between multiple learning tasks through cross-attention, realizes the bird's eye view reconstruction process under the guidance of scene depth perception and scene semantic perception embedding. To solve the problem of catastrophic forgetting in continuous learning, cross-scene optimization is used to share specific experience between different expert data distributions, and a high universality imitation learning expert is trained. Through the proposed reward function decoder, the federated teacher-student architecture outputs motion planning, reward feedback, and dynamic driving suggestion as the basis for reinforcement learning student training and optimization, to realize its learning of driving rules. To solve the problem of high-dimensional environment representation, the federated teacher-student architecture uses the time series processing module combined with query-based knowledge distillation to improve the representation ability of the reinforcement learning for the original image input.

[0150] The above series of detailed descriptions are only specific descriptions of the feasible embodiments of the present application, and are not intended to limit the protection scope of the present application. Any equivalent means or changes made without departing from the technology of the present application shall be included in the protection scope of the present application.

Claims

1. A scenario-driven, highly generalizable federated reinforcement learning autonomous driving system, characterized in that, include: The reinforcement learning student modeling part, the training imitation learning expert part, and the distillation reinforcement learning student part; The reinforcement learning student modeling part first constructs a feedback-based target alignment module, and then constructs a temporal processing module; The target alignment module generates binary feedback by comparing the weighted cumulative rewards of two trajectory segments, establishing a preference model based on the reward truth value and expert demonstration. The temporal processing module first processes additional inputs, including one-hot encodings of vehicle speed and driving commands, through a multilayer perceptron. Then, it stacks the features output from the multilayer perceptron with features extracted from the backbone network and initializes a set of learnable historical queries. As the raw input to the timing processing module, q temporal The stacked features are processed sequentially through a temporal cross-attention layer, a self-attention layer, and a feedforward neural network (FFN) to output the current query. The training of the imitation learning expert first uses scene depth perception and scene semantic perception embedding to guide the bird's-eye view reconstruction process. The bird's-eye view query is output through a semantic-depth heuristic decoder, and speed decoder, motion decoder, bird's-eye view decoder, object decoder and reward function decoder are used to output speed prediction, motion prediction, bird's-eye view prediction, object detection and reward feedback prediction respectively. Finally, a highly applicable imitation learning expert is trained through a specific experience sharing module. The bird's-eye view reconstruction adds two additional tasks, depth estimation and semantic segmentation, to the bird's-eye view prediction process. It uses depth information and semantic information as labels and combines them with an attention mechanism to achieve depth-semantic heuristic bird's-eye view reconstruction. The deep-semantic heuristic decoder comprises two self-attention layers and three cross-attention layers, initializing a set of learnable bird's-eye view queries. As the initial input, where N represents the number of queries and C represents the number of channels; firstly, the bird's-eye view queries are passed through a self-attention layer to generate internal relationships between queries; then, the queries are passed to a cross-attention layer, outputting the interacted bird's-eye view queries, which serve as the basis for aggregation embedding. This process can be defined as: Q = w' q q,K = w' k f image ,V = w' v f image Q′=w″(AV) Where Q, K, and V represent the query, key, and value, respectively, w q ′,w′ k w v ' and w' represent learnable parameters, A represents the attention map, h represents the number of multi-head attention, and Q' represents the output query. The three cross-attention mechanisms are the same. To further promote interaction between different tasks, the entire deep-semantic heuristic module is repeated three times, outputting a fully interactive bird's-eye view query, and the bird's-eye view query is split into motion queries Q'. m And predictive query Q p Two parts, Q m The proposed timing processing module is used to process the data, and the processed current motion query Q is output. current ; The speed decoder outputs a set of weights. The process of obtaining the predicted velocity by weighted four-modal velocity is defined as follows: λ v =MLP(FFN(crossaten(Q v =Q current ,K v 5 v =q v ))) Among them, Q current Q represents the current motion query. v K v and V v These represent the query, key, and value of the speed decoder, respectively. Represents a set of learnable speed queries; The motion decoder outputs a set of navigation point queries. A set of navigation point predictions is output through a gated recurrent unit (GRU), where T = 10 represents the navigation point prediction time step, and the process is defined as follows: Q waypoint =FFN(selfatten(crossaten(Q w =Q current ,K w 5 w =q w ))) x t+1 =x t +Δx t Among them, Q w K w and V w These represent the query, key, and value of the motion decoder, respectively. Describes a set of learnable waypoint queries, Δx t Let x represent the expected change in coordinates at time t. t Let x represent the agent's coordinates at time t. tt Let x represent the agent's target point at time t. t+1 This represents the agent's coordinates at time t+1; The bird's-eye view decoder consists of convolutional layers and upsampling layers to output the bird's-eye view reconstruction result. Where c b Indicates the number of categories; The object decoder outputs a location heatmap prediction ∈ [0,1]. (64×64) , And regression plot prediction; among which as well as The reward function decoder constructs a transformer-based reward function model for further processing of queries; firstly, it initializes a set of learnable reward queries. Then, a motion cross-attention layer and a prediction cross-attention layer are sequentially aggregated using q. r The output Q of the timing processing module current With predictive query Q p The output query is then passed to the self-attention layer for internal interaction aggregation. The aggregated query is then passed to the feedforward neural network (FFN) and outputs the reward value through the multilayer perceptron (MLP) layer. The specific experience sharing module trains its respective teacher models on different expert demonstration distributions. During the training process, it aggregates the parameters of the reward function decoder and the semantic-deep heuristic decoder according to the aggregation interval, thereby achieving specific experience sharing without transmitting samples. The distillation reinforcement learning student part trains reinforcement learning students according to the loss function, and performs parameter aggregation after local training is completed. The loss function includes gradient descent loss, exploration loss, distillation loss, and proposal loss.

2. The scene-driven, highly generalizable federated reinforcement learning autonomous driving system according to claim 1, characterized in that, The specific implementation of the target alignment module includes the following: Where y(τ0,τ1)∈{(1,0),(0,1)} represents the binary feedback of the preference model for trajectory segments τ0 and τ1 of length H. Representing state-action pairs The reward is really worth it. Represents trajectory segment τ i Uncertainty weights of driving behavior Represents trajectory segment τ i The error weights, where i∈{0,1}, Represents the trajectory segment τ of the agent at time t. i The lateral movement below, Represents the trajectory segment τ of the agent at time t. i The vertical action of the agent in the range i∈{0,1} is represented by Δd(t), which is the lateral error of the agent from the desired path in the sample at time t. Δθ(t) is the heading angle deviation of the agent from the desired path in the sample at time t. The reward is the true value. Set as: Where v(t) represents the velocity of the agent at time t, v desire (t) represents the agent's expected velocity at time t for the sample. This indicates environmental feedback related to speed, v max Indicates the maximum speed. and Representing the environmental feedback related to the action and the termination state, respectively, the output of the reward function model is described in probabilistic form according to the Bradley-Terry selection model: Where P reward [τ0>τ1] represents the probability that, in a binary preference model, the reward function model with parameter φ has a greater preference for trajectory segment τ0 than for τ1.

3. The scene-driven, highly generalizable federated reinforcement learning autonomous driving system according to claim 2, characterized in that, In the new round of cross-attention, the timing processing module uses the current query as a historical query. In the initial stage of the interaction, the first frame image and the first set of additional inputs are reused. As the interaction progresses, the timing processing module uses three consecutive frames of images and three sets of additional inputs as module inputs starting from the third frame.

4. The scene-driven, highly generalizable federated reinforcement learning autonomous driving system according to claim 1, characterized in that, The bird's-eye view reconstruction process specifically involves: firstly, the input to the model is a monocular RGB image. Where h×w represents the resolution of the image used, c=3 represents a 3-channel RGB image, RegNety-32 is used as the backbone network, and a feature pyramid network is combined to obtain multi-scale image feature maps f. image ;f image The data is then passed to three self-attention-based encoders to obtain deep embeddings, visual embeddings, and semantic embeddings. A deep-semantic heuristic module based on a cross-attention mechanism is then used to facilitate interaction between different prediction tasks.

5. The scene-driven, highly generalizable federated reinforcement learning autonomous driving system according to claim 1, characterized in that, The distillation loss is described using mean squared error (MSE) loss to represent the distance between the feature matrices output by the reinforcement learning student and the imitation learning expert. Where λ kd Q represents the weight of distillation loss. student The BEV query for reinforcement learning teachers, Q expert This indicates a BEV query that imitates learning experts.

6. The scene-driven, highly generalizable federated reinforcement learning autonomous driving system according to claim 1, characterized in that, The suggested loss Where, λ sug This indicates the weight of the suggested loss. Indicates time TN z The indicator function from +1 to the termination time T, KL(π) θ (·|s RL B sug ) indicates the state input s RL The following strategy π θ Recommended Strategy B sug KL divergence between B sug This refers to the dynamic driving recommendations from imitation learning experts for students learning reinforcement learning.

7. The scene-driven, highly generalizable federated reinforcement learning autonomous driving system according to claim 1, characterized in that, The parameter aggregation process is as follows: Where, θ global θ represents the global student model parameters. student This represents the parameters of the local student model, and n represents the number of local student models.

8. A scenario-driven, highly generalizable federated reinforcement learning method for autonomous driving, characterized in that, Includes the following steps: Step 1: Perform reinforcement learning student modeling; first, perform feedback-based target alignment, and then build a temporal processing module; Step 2: Train an imitation learning expert; First, use scene depth perception and scene semantic perception embedding to guide the bird's-eye view reconstruction process. Then, output the bird's-eye view query through a semantic-depth heuristic decoder. Then, use a velocity decoder, motion decoder, bird's-eye view decoder, object decoder, and reward function decoder to output velocity prediction, motion prediction, bird's-eye view prediction, object detection, and reward feedback prediction, respectively. Finally, train a highly applicable imitation learning expert through specific experience sharing. Step 3: Distill the reinforcement learning student; Distill the reinforcement learning student according to the loss function, and aggregate the parameters after completing the local training; the loss function includes gradient descent loss, exploration loss, distillation loss, and proposal loss; In step 1, the target alignment process generates binary feedback by comparing the weighted cumulative rewards of the two trajectories, and establishes a preference model based on the reward truth value and expert demonstration: Where y(τ0,τ1)∈{(1,0),(0,1)} represents the binary feedback of the preference model for trajectory segments τ0 and τ1 of length H. Representing state-action pairs The reward is really worth it. Represents trajectory segment τ i Uncertainty weights for driving behavior, i∈{0,1} Represents trajectory segment τ i Error weights for i∈{0,1} Represents the trajectory segment τ of the agent at time t. i The horizontal movement below, i∈{0,1}, Represents the trajectory segment τ of the agent at time t. i The vertical action of the agent in the range i∈{0,1} is represented by Δd(t), which is the lateral error of the agent from the desired path in the sample at time t. Δθ(t) is the heading angle deviation of the agent from the desired path in the sample at time t. The reward is the true value. Set as: Where v(t) represents the velocity of the agent at time t, v desire (t) represents the agent's expected velocity at time t for the sample. This indicates environmental feedback related to speed, v max Indicates the maximum speed. and Let represent the environmental feedback related to the action and the termination state, respectively. Based on the Bradley-Terry (BT) selection model, the output of the reward function model is described in the form of probabilities: Where P reward [τ0>τ1] represents the probability that, in a binary preference model, the reward function model with parameter φ has a greater preference for trajectory segment τ0 than for τ1. The temporal processing module in step 1 first processes the additional inputs, including the one-hot encoding of vehicle speed and driving commands, using a multilayer perceptron. Then, it stacks the features output from the multilayer perceptron with the features extracted by the backbone network and initializes a set of learnable historical queries. As the raw input to the timing processing module, q temporal The stacked features are sequentially passed through a temporal cross-attention layer, a self-attention layer, and a feedforward neural network (FFN) before the current query is output. This process is repeated 3 times, and in the new round of cross-attention, the current query is used as a historical query; at the initial stage of the interaction, the first frame image and the first set of additional inputs will be reused. As the interaction progresses, the timing processing module uses 3 consecutive frames of images and 3 sets of additional inputs as module inputs starting from the third frame. In step 2, the bird's-eye view reconstruction adds two additional tasks—depth estimation and semantic segmentation—to the bird's-eye view prediction process. Depth and semantic information are used as labels combined with an attention mechanism to achieve depth-semantic heuristic BEV reconstruction. Specifically, the bird's-eye view prediction process is as follows: First, the model's input is a monocular RGB image. Where h×w represents the resolution of the image used, c represents the number of channels in the RGB image, RegNety-32 is used as the backbone network, and feature pyramids are combined to obtain multi-scale image feature maps f. image f image The embeddings are then passed to three self-attention-based encoders to obtain deep embeddings, visual embeddings, and semantic embeddings. After obtaining these embeddings, a deep-semantic heuristic module based on a cross-attention mechanism is used to promote the interaction between different prediction tasks. In step 2, the deep-semantic heuristic decoder, comprising two self-attention layers and three cross-attention layers, initializes a set of learnable bird's-eye view queries. As the initial input to this module, where N represents the number of queries and C represents the number of channels, the bird's-eye view queries are first passed through a self-attention layer to generate internal relationships between the queries. Then, the queries are passed to a cross-attention layer as the basis for aggregation and embedding. This process can be defined as follows: Q=w′ q q,K=w′ k f image ,V=w′ v f image Q′=w″(AV) Where Q, K, and V represent the query, key, and value, respectively, w q ′,w′ k w v ' and w' represent learnable parameters, A represents the attention map, h represents the number of multi-head attention, and Q' represents the output query. The three cross-attention mechanisms adopt the same definition process. To further promote interaction between different tasks, the entire deep-semantic heuristic module is repeated three times to output a fully interactive bird's-eye view query, and the bird's-eye view query is decomposed into a motion query Q. m And predictive query Q p Two parts, Q m The proposed timing processing module is used to process the data, and the processed current motion query Q is output. current ; In step 2, the speed decoder outputs a set of weights. The process of obtaining the predicted velocity by weighted four-modal velocity is defined as follows: λ v =MLP(FFN(crossaten(Q v =Q current ,K v 5 v =q v ))) Among them, Q current Q represents the current motion query. v K v and V v These represent the query, key, and value of the speed decoder, respectively. Represents a set of learnable speed queries; In step 2, the motion decoder outputs a set of navigation point queries. A set of navigation point predictions is output through a gated recurrent unit (GRU), where T represents the navigation point prediction time step, and the process is defined as follows: Q waypoint =FFN(selfatten(crossaten(Q w =Q current ,K w 5 w =q w ))) x t+1 =x t +Δx t Among them, Q w K w and V w These represent the query, key, and value of the motion decoder, respectively. Describes a set of learnable waypoint queries, Δx t Let x represent the expected change in coordinates at time t. t Let x represent the agent's coordinates at time t. tt Let x represent the agent's target point at time t. t+1 This represents the agent's coordinates at time t+1; In step 2, the bird's-eye view decoder consists of convolutional layers and upsampling layers to output the bird's-eye view reconstruction result. Where c b Indicates the number of categories; In step 2, the object decoder outputs a location heatmap prediction ∈ [0,1]. (64×64) , And regression plot prediction, among which as well as In step 2, the reward function decoder constructs a transformer-based reward function model to further process the query. First, it initializes a set of learnable reward queries. Then, a motion cross-attention layer and a prediction cross-attention layer are sequentially aggregated using q. r The output Q of the timing processing module current With predictive query Q p The output query is then passed to the self-attention layer for internal interaction aggregation; the aggregated query is then passed to the feedforward neural network (FFN) and outputs a reward value through the multilayer perceptron (MLP) layer. In step 2, the specific experience sharing involves training the respective teacher models on different expert demonstration distributions. During the training process, the parameters of the reward function decoder and the semantic-deep heuristic decoder are aggregated according to the aggregation interval, thereby achieving specific experience sharing without transmitting samples. In step 3, the distillation loss is described using mean squared error (MSE) loss to represent the distance between the feature matrices output by the reinforcement learning student and the imitation learning expert. Where λ kd Q represents the weight of distillation loss. student The BEV query for reinforcement learning teachers, Q expert This indicates a BEV query that imitates learning experts. The suggested loss: Where, λ sug This indicates the weight of the suggested loss. Indicates time TN z The indicator function from +1 to the termination time T, KL(π) θ (·|s RL B sug ) indicates the state input s RL The following strategy π θ Recommended Strategy B sug KL divergence between B sug This refers to the dynamic driving suggestions from imitation learning experts for reinforcement learning students, including longitudinal driving suggestions and lateral driving suggestions; Longitudinal driving recommendations are defined as: Among them, B go B indicates dynamic acceleration suggestions when the vehicle is blocked. stop This indicates dynamic deceleration recommendations before a vehicle is involved in a collision or runs a red light. go = -0.04 indicates the recommended adjustment parameter for dynamic acceleration, μ stop =-0.025 indicates the recommended adjustment parameter for dynamic deceleration; Using uniform distribution B horizontal_sug = (1,1) serves as a lateral driving suggestion before the vehicle deviates from its lane; In step 3, the parameter aggregation process is as follows: Where, θ global θ represents the global student model parameters. student This represents the parameters of the local student model, and n=6 represents the number of local student models.

Images