Action-smoothing reinforcement learning policy network for end-to-end autonomous driving

Abstract

The present disclosure relates to an action-smoothing reinforcement learning policy network suitable for end-to-end autonomous driving. It includes: obtaining observation data collected by sensors; inputting the observation data into the Fourier filter layer in the policy network, which can filter observation noise, extract important frequencies in the observation input, and obtain filtered features; inputting the filtered features at the current moment into the subsequent sub-network in the policy network to obtain vehicle control actions, which are used to control the vehicle for autonomous driving; the sub-network uses the Jacobi regularization method to control the smoothness of the policy function. According to the embodiment of the present application, the problem of action oscillation of reinforcement learning for end-to-end autonomous driving can be effectively solved, and the action smoothness of the policy network output can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and in particular to an action-smoothing reinforcement learning strategy network suitable for end-to-end autonomous driving. Background Art

[0002] Reinforcement learning (RL) has become an important method for solving optimal control tasks in real physical environments. Because neural networks can model complex nonlinear functions, they are often used to carry out control strategies for fitting reinforcement learning. However, strategies trained with reinforcement learning often encounter the problem of motion oscillation. Even if there are slight differences in the state observations at adjacent moments, the actions at adjacent moments will show significant differences. Although this problem is often ignored in the simulation and training stages, it can cause serious problems in real-world applications, such as actuator wear and safety risks. The problem of motion oscillation is also prevalent in various industrial application scenarios such as autonomous driving.

[0003] Current reinforcement learning approaches fail to understand the root causes of reinforcement learning action oscillation, resulting in poor smoothness and decision-making performance in the policy network output. Furthermore, existing action smoothing techniques, when constraining policy network parameters, suffer from slow feedforward inference speeds, making them unsuitable for real-time decision-making tasks in autonomous driving. Summary of the Invention

[0004] In view of this, the present disclosure proposes an action-smoothing reinforcement learning strategy network suitable for end-to-end autonomous driving.

[0005] According to one aspect of the present disclosure, a method for executing an action-smoothing reinforcement learning policy network suitable for end-to-end autonomous driving is provided. The method comprises:

[0006] Obtain observation data collected by sensors;

[0007] Input the observed data into the Fourier filter layer in the strategy network to obtain the filtered features;

[0008] The filtered features at the current moment are input into the sub-network in the policy network to obtain the vehicle control action, which is used to control the vehicle for autonomous driving; the Jacobi regularization method is used in the sub-network to control the smoothness of the policy function.

[0009] In one possible implementation, the observation data is input into the Fourier filter layer in the policy network to obtain filtered features, including:

[0010] The Fourier filter layer calculates the first frequency feature matrix based on the observed data using a two-dimensional discrete Fourier transform;

[0011] Selecting the upper half or the lower half of the first frequency characteristic matrix to obtain a second frequency characteristic matrix;

[0012] Perform filtering and feature extraction based on the second frequency feature matrix to restore a complete third frequency feature matrix;

[0013] Based on the third frequency feature matrix, a two-dimensional discrete inverse Fourier transform is used to obtain the filtered features.

[0014] In one possible implementation, filtering and feature extraction are performed based on the second frequency feature matrix to restore a complete third frequency feature matrix, including:

[0015] Perform element-by-element multiplication on the trainable filter matrix and the second frequency feature matrix to obtain a product matrix;

[0016] The product matrix is ​​conjugate-symmetrized and restored to obtain the third frequency characteristic matrix.

[0017] In one possible implementation, based on the third frequency feature matrix, a two-dimensional discrete inverse Fourier transform is used to obtain filtered features, including:

[0018] The third frequency feature matrix is ​​processed using a two-dimensional discrete inverse Fourier transform to obtain a time domain feature matrix;

[0019] The time domain feature matrix is ​​divided to obtain the filtered features at the current moment.

[0020] In one possible implementation, the policy network is a trained policy network, and the method further includes:

[0021] Obtain observation signals of training sample states;

[0022] Input the observation signal of the training sample state into the initial strategy network to obtain the control action corresponding to the observation signal of the training sample state;

[0023] Based on the observation signal of the training sample state and the control action corresponding to the observation signal of the training sample state, the value function estimate under the current observation and action is obtained using the value evaluation network. The value function estimate represents the estimate of the expected return of executing the corresponding control action based on the observation signal of the training sample state;

[0024] The initial policy network is trained based on the loss function to obtain a trained policy network, where the loss function includes the expected value estimated by the negative value function.

[0025] In one possible implementation, the loss function further includes a Jacobi regularization term, where the Jacobi regularization term is the product of the first learnable parameter and the Jacobi norm.

[0026] In one possible implementation, the Jacobi norm is an approximation of the local Lipschitz constant of the policy function, which describes the smoothness of the policy function in a preset kick-off neighborhood in Euclidean space.

[0027] In a possible implementation, the loss function further includes a filtering term, where the filtering term is the product of the second learnable parameter and the Frobenius norm of the filter matrix.

[0028] According to another aspect of the present disclosure, a motion smoothing reinforcement learning strategy network system suitable for end-to-end autonomous driving is provided. The system includes:

[0029] An observation data acquisition unit, used to acquire observation data collected by sensors;

[0030] The Fourier filter unit is used to input the observation data into the Fourier filter layer in the strategy network to obtain the filtered features;

[0031] The sub-network processing unit is used to input the filtered features at the current moment into the sub-network in the policy network to obtain the vehicle control action, which is used to control the vehicle for autonomous driving; the Jacobi regularization method is used in the sub-network to control the smoothness of the policy function.

[0032] In a possible implementation, the Fourier filter unit is configured to:

[0033] The Fourier filter layer calculates the first frequency feature matrix based on the observed data using a two-dimensional discrete Fourier transform;

[0034] Selecting the upper half or the lower half of the first frequency characteristic matrix to obtain a second frequency characteristic matrix;

[0035] Perform filtering and feature extraction based on the second frequency feature matrix to restore a complete third frequency feature matrix;

[0036] Based on the third frequency feature matrix, a two-dimensional discrete inverse Fourier transform is used to obtain the filtered features.

[0037] In one possible implementation, filtering and feature extraction are performed based on the second frequency feature matrix to restore a complete third frequency feature matrix, including:

[0038] Perform element-by-element multiplication on the trainable filter matrix and the second frequency feature matrix to obtain a product matrix;

[0039] The product matrix is ​​conjugate-symmetrized and restored to obtain the third frequency characteristic matrix.

[0040] In one possible implementation, based on the third frequency feature matrix, a two-dimensional discrete inverse Fourier transform is used to obtain filtered features, including:

[0041] The third frequency feature matrix is ​​processed using a two-dimensional discrete inverse Fourier transform to obtain a time domain feature matrix;

[0042] The time domain feature matrix is ​​divided to obtain the filtered features at the current moment.

[0043] In one possible implementation, the policy network is a trained policy network, and the system further includes:

[0044] A sample state observation acquisition unit, used to obtain an observation signal of the training sample state;

[0045] A policy network processing unit, configured to input the observation signal of the training sample state into the initial policy network to obtain a control action corresponding to the observation signal of the training sample state;

[0046] a value evaluation network processing unit for obtaining a value function estimate under the current observation and action based on the observation signal of the training sample state and the control action corresponding to the observation signal of the training sample state using the value evaluation network, wherein the value function estimate represents an estimate of the expected reward of executing the corresponding control action based on the observation signal of the training sample state;

[0047] The training unit is used to train the initial policy network based on the loss function to obtain a trained policy network, where the loss function includes an expected value estimated by a negative value function.

[0048] In one possible implementation, the loss function further includes a Jacobi regularization term, where the Jacobi regularization term is the product of the first learnable parameter and the Jacobi norm.

[0049] In one possible implementation, the Jacobi norm is an approximation of the local Lipschitz constant of the policy function, which describes the smoothness of the policy function in a preset kick-off neighborhood in Euclidean space.

[0050] In a possible implementation, the loss function further includes a filtering term, where the filtering term is the product of the second learnable parameter and the Frobenius norm of the filter matrix.

[0051] According to another aspect of the present application, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0052] According to another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above method is implemented.

[0053] According to another aspect of the present application, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0054] According to an embodiment of the present application, by inputting the observation data collected by the sensor into the Fourier filter layer in the policy network, the observation noise can be filtered, the important frequencies in the observation input can be extracted, and the filtered features can be obtained; the filtered features at the current moment are input into the subsequent sub-network in the policy network to obtain the vehicle control action for controlling the vehicle to perform autonomous driving, and the end-to-end autonomous driving task can be completed. The Jacobi regularization method is used in the sub-network to control the smoothness of the policy function, which can effectively solve the action oscillation problem of reinforcement learning for end-to-end autonomous driving, and improve the action smoothness and decision-making performance of the policy network output. At the same time, by using the Jacobi regularization method in the sub-network to control the smoothness of the policy function, matrix calculation is not required during the forward propagation process, the inference speed is faster, and it can be suitable for high real-time autonomous driving tasks.

[0055] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0057] Figure 1 A schematic diagram illustrating an application scenario according to an embodiment of the present application.

[0058] Figure 2 A decision control flow chart of an execution method of an action-smoothing reinforcement learning strategy network suitable for end-to-end autonomous driving according to an embodiment of the present application is shown.

[0059] Figure 3 A schematic structural diagram of FlipNet according to an embodiment of the present application is shown.

[0060] Figure 4 A training flowchart of an action-smoothing reinforcement learning strategy network suitable for end-to-end autonomous driving according to an embodiment of the present application is shown.

[0061] Figure 5A schematic diagram of the process of Fourier filter layer processing according to an embodiment of the present application is shown.

[0062] Figure 6 A schematic diagram illustrating an experimental scenario according to an embodiment of the present application is shown.

[0063] Figure 7 A schematic diagram showing the experimental results (vehicle trajectory) according to an embodiment of the present application.

[0064] Figure 8 A schematic diagram showing an experimental effect (control action) according to an embodiment of the present application.

[0065] Figure 9 A structural diagram of an action smoothing reinforcement learning strategy network system suitable for end-to-end autonomous driving according to one embodiment of the present application is shown.

[0066] Figure 10 1 is a block diagram of an execution device 1900 of an action-smoothing reinforcement learning policy network suitable for end-to-end autonomous driving, according to an exemplary embodiment. DETAILED DESCRIPTION

[0067] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0068] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0069] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0070] Reinforcement learning (RL) has become an important method for solving optimal control tasks in real physical environments. Because neural networks can model complex nonlinear functions, they are often used to carry out control strategies for fitting reinforcement learning. However, strategies trained with reinforcement learning often encounter the problem of motion oscillation. Even if there are slight differences in the state observations at adjacent moments, the actions at adjacent moments will show significant differences. Although this problem is often ignored in the simulation and training stages, it can cause serious problems in real-world applications, such as actuator wear and safety risks. The problem of motion oscillation is also prevalent in various industrial application scenarios such as autonomous driving.

[0071] Current reinforcement learning approaches fail to understand the root causes of reinforcement learning action oscillation, resulting in poor smoothness and decision-making performance in the policy network output. Furthermore, existing action smoothing techniques, when constraining policy network parameters, suffer from slow feedforward inference speeds, making them unsuitable for real-time decision-making tasks in autonomous driving.

[0072] In view of this, the present application provides an action smoothing reinforcement learning policy network suitable for end-to-end autonomous driving. The present application obtains filtered features by inputting the observation data collected by the sensor into the Fourier filter layer in the policy network; the filtered features at the current moment are input into the sub-network in the policy network to obtain the vehicle control action for controlling the vehicle to perform autonomous driving, which can complete the end-to-end autonomous driving task. The Jacobi regularization method is used in the sub-network to control the smoothness of the policy function, which can solve the problem of non-smooth fitting of the policy network, effectively solve the problem of action oscillation, and improve the action smoothness and decision-making performance of the policy network output. At the same time, the Jacobi norm is used to make it unnecessary to perform matrix calculations during the forward propagation process, and the inference speed is faster, which can be applied to high-real-time autonomous driving tasks.

[0073] Figure 1 A schematic diagram showing an application scenario according to an embodiment of the present application is shown. Figure 1 As shown, the action smoothing reinforcement learning policy network system suitable for end-to-end autonomous driving of the present application can be applied to the scenario of end-to-end autonomous driving. The action smoothing reinforcement learning policy network system suitable for end-to-end autonomous driving may include a policy network (Actor) and a value evaluation network (Critic). The policy network may be the FlipNet of an embodiment of the present application. In the process of performing autonomous driving tasks, the policy network may output vehicle control actions based on the observation data collected by sensors to control the vehicle to achieve autonomous driving.

[0074] The motion smoothing reinforcement learning strategy network system suitable for end-to-end autonomous driving in the embodiment of the present application can be deployed in a terminal device or a server. The present application does not limit the type of the terminal device or the server.

[0075] Figure 2 FIG2 shows a decision control flow chart of an execution method of an action smoothing reinforcement learning strategy network for end-to-end autonomous driving according to an embodiment of the present application. Figure 2 As shown, the method may include:

[0076] Step S201: Acquire observation data collected by sensors.

[0077] The observation data may be used to represent observations of the environmental state around the vehicle (such as image data, etc.), which may include observation data at the current moment, and may also include observation data at one or more moments before the current moment.

[0078] Step S202: input the observed data into the Fourier filter layer in the strategy network to obtain filtered features.

[0079] The following first introduces the structure of the policy network in this application.

[0080] Figure 3 FIG. 1 shows a schematic diagram of the structure of FlipNet according to an embodiment of the present application. Figure 3 As shown in FIG, FlipNet is a strategy network proposed in this application. The strategy network may include a Fourier filter layer and a sub-network f.

[0081] The Fourier filter layer can be used to filter the total N observation data at the current time (time t) and N-1 times before time t (time t to (t-N+1)) (as shown in the figure t 、o t-1 、o t-N+1 , which can be obtained based on observation data at multiple moments) to obtain multiple filtered features (as shown in the figure ).like Figure 3 As shown, the Fourier filter layer may include a Fourier transform module (FFT) and an inverse Fourier transform module (IFFT), which are used to perform fast Fourier transform (Fast Fourier Transform) and inverse fast Fourier transform (Inverse Fast Fourier Transform), respectively. Half of the frequency feature matrix (frequency feature matrix obtained by FFT) X half The product of the two can be used to input the inverse Fourier transform module.

[0082] The subnetwork can be used to process the filtered features and output the decision action a at time t t , the decision action is the vehicle control action output by the policy network at time t.

[0083] Among them, the policy network is the trained policy network. Figure 4 For an introduction to the training method of the policy network, see Figure 4 , shows a training flow chart of an action smoothing reinforcement learning strategy network suitable for end-to-end autonomous driving according to an embodiment of the present application. Figure 4 As shown, the training process includes:

[0084] Step S401: Obtain observation signals of training sample states.

[0085] The observation signal of the training sample state may include observations of the environmental state around the vehicle at multiple moments.

[0086] Step S402: input the observation signal of the training sample state into the initial strategy network to obtain the control action corresponding to the observation signal of the training sample state.

[0087] The structure of the initial policy network can be found in Figure 3 .

[0088] The observation signal of the training sample state can be used as multiple historical observation data N represents the number of historical observation data, and D represents the dimension of historical observation data.

[0089] In order to alleviate the motion vibration problem caused by observation noise, a Fourier filter layer based on Fourier transform is proposed in this application.

[0090] Figure 5 FIG. 1 is a flow chart showing the process of Fourier filter layer processing according to an embodiment of the present application. Figure 5 As shown, the Fourier filter layer can be used to transform multiple historical observation data (o t ,…,o t-N+1 ) into a matrix Since zero padding does not change the main frequency components of the signal in Fourier transform, but only increases the spectral resolution, when the length of the historical observation data is less than N, the missing part can be filled with 0.

[0091] You can use the two-dimensional discrete Fourier transform (DFT) to perform Fourier transform (FFT) and calculate the frequency characteristic matrix One calculation method can be found in formula (1):

[0092]

[0093] Among them, X u,v It can represent the element value of the uth row and vth column in the frequency feature matrix X, x n,d It can represent the d-th dimension feature value of the n-th historical observation data. u,v The size of can represent the signal strength of the historical observation data under the frequency combination (u, v), where u and v are frequency indices rather than actual frequency values.

[0094] Since the historical observation data only contains real numbers, the resulting frequency feature matrix X has conjugate symmetry, that is, This means that only half of the matrix X can be used to represent the complete information contained in the signal. In this application, the upper half or lower half of the frequency characteristic matrix is ​​selected to obtain half of the frequency characteristic matrix, and the half matrix of X is recorded as Based on X half Subsequent processing can reduce the amount of calculation.

[0095] In this application, a trainable filter matrix is ​​introduced By adding X half After Hadamard multiplication with H (i.e., element-by-element multiplication), the conjugate symmetrization of the product matrix is ​​restored (i.e., the conjugate symmetry of the product matrix is ​​restored) to obtain a restored frequency characteristic matrix The calculation process can be found in formula (2):

[0096]

[0097] Where symmetrize(·) represents the conjugate symmetry restoration operation, X half ⊙H can represent X half The product matrix obtained by element-wise multiplication with H.

[0098] Since the filter matrix H is a complex matrix, by introducing the filter matrix into the Fourier filter layer, not only can the X half The frequency amplitude can also be used to extract its features. The modulus length of the elements in the filter matrix H can be used to determine whether the frequency is suppressed or enhanced. In order to enable the policy network to have the observation noise filtering function, the modulus length of the elements in H of this application is a small value. As a result, the policy network can automatically extract more valuable frequencies and filter out irrelevant frequencies that may contain noise to solve the problem of action oscillation, making the policy network more robust.

[0099] Finally, the Fourier filter layer can perform an inverse Fourier transform (IFFT) through a two-dimensional discrete inverse Fourier transform (IDFT) to convert the recovered frequency feature matrix Converted into a time domain feature matrix, the calculation process can be seen in formula (3):

[0100]

[0101] in, It can represent the element value of the nth row and dth column in the time domain feature matrix, is a real matrix, we can Perform segmentation to obtain filtered features

[0102] The filtered features at the current moment can be As the input of the sub-network f. The sub-network can be regarded as a mapping function for fitting the policy function π: The policy function can transform the state space Mapping to action space The state space can correspond to the filtered features, and the action space can correspond to the control actions. Thus, the sub-network can be used to map the filtered features to the control actions.

[0103] In this application, in order to enhance the smoothness of the policy network and further solve the problem of action oscillation, the Jacobi regularization method is used to control the smoothness of the policy function (continuously differentiable neural network, i.e., sub-network). The Jacobi norm is introduced in the Jacobi regularization method. The Jacobi norm is an approximation of the local Lipschitz constant of the policy function. For the continuously differentiable sub-network f: Jacobian norm is an approximation of the local Lipschitz constant of the policy function f in the infinitesimal neighborhood of x, that is

[0104] K(x) is the local Lipschitz constant of f near x, and its calculation method can be seen in formula (4):

[0105]

[0106] The local Lipschitz constant describes the preset kick-off neighborhood of the strategy function f in Euclidean space. The degree of smoothness in . represents the open ball neighborhood in the Euclidean space with x as the center and radius ρ, that is ‖·‖ represents the 2-norm. x1 and x2 are the kick-off neighborhoods. For the points in , f(x1) and f(x2) can represent the values ​​of points x1 and x2 in the policy function f respectively.

[0107] Step S403 , based on the observation signal of the training sample state and the control action corresponding to the observation signal of the training sample state, the value function estimation under the current observation and action is obtained using the value evaluation network.

[0108] The value evaluation network is the Critic network in deep learning (see Figure 1 ), the Critic network can be used to fit the value function Q(s,a). This value function can be used to estimate the value function based on the observed signal of the training sample state and the control action corresponding to the observed signal of the training sample state. The value function estimate can be used to estimate the expected reward of executing the corresponding control action based on the observed signal of the training sample state. The value function estimate can be used to evaluate the value of the actions taken by the policy network (i.e., the output control action).

[0109] Step S404: Train the initial policy network based on the loss function to obtain a trained policy network.

[0110] The loss function may include the expected value of the negative value function estimate. One way to calculate the loss function can be seen in formula (5):

[0111]

[0112] in, The loss function can be expressed as Indicates that the experience replay pool The expected value of the sampled state s (corresponding to the state of the input policy network). represents the value function estimate, and π(·) is the policy function.

[0113] The parameters in the initial policy network can be updated by minimizing the loss function, that is, by maximizing the expected return predicted by the Critic network to obtain the trained policy network.

[0114] Optionally, the loss function also includes a Jacobian regularization term, which is the product of the first learnable parameter and the Jacobian norm. The calculation method of the optimized loss function can be seen in formula (6):

[0115]

[0116] Among them, λ k is the first learnable parameter, represents the Jacobian norm, that is, the norm of the Jacobian matrix, which is an approximation of the local Lipschitz constant. k and Adaptive updates can be performed during the training process, which can flexibly enhance the smoothness of the strategy and make the output of the strategy network smoother.

[0117] Because the loss function including the Jacobian regularization term is applied to the training of the entire policy network, severe performance loss can be avoided. The Jacobian regularization term does not require manual initialization, which does not impair the exploration capabilities of reinforcement learning. Furthermore, compared to existing action smoothing methods such as LipsNet, the introduction of the Jacobian regularization term eliminates the need to calculate the Jacobian matrix during forward propagation, resulting in faster inference speed.

[0118] Optionally, the loss function further includes a filtering term, which is the product of the second learnable parameter and the Frobenius norm of the filter matrix. The calculation method of the optimized loss function can be seen in formula (7):

[0119]

[0120] Among them, λ h is the second learnable parameter, which can be adaptively updated during the training process, ‖H‖ F is the Frobenius norm of the filter matrix H.

[0121] The loss function can be minimized based on the gradient descent method To update the parameters in the policy network, we can obtain the trained policy network.

[0122] By introducing both Jacobian regularization and filtering terms into the loss function, the problems of non-smooth policy network fitting and the presence of environmental observation noise can be solved simultaneously during the training of the policy network, so that the action smoothness and decision-making performance of the trained policy network can be optimized.

[0123] After completing the training of the policy network, you can return to executing steps S201 to S203 to use the trained policy network to perform the autonomous driving task.

[0124] Return to step S202. In step S202, you can:

[0125] The Fourier filter layer uses a two-dimensional discrete Fourier transform to calculate the first frequency feature matrix based on the observed data; selects the upper half or the lower half of the first frequency feature matrix to obtain the second frequency feature matrix; performs filtering and feature extraction based on the second frequency feature matrix to restore the complete third frequency feature matrix; and uses a two-dimensional discrete inverse Fourier transform based on the third frequency feature matrix to obtain the filtered features.

[0126] This process can be seen in the implementation of step S402, where the observation data can be o t,o t-1 ,…,o t-N+1 , the first frequency feature matrix can be X, and the second frequency feature matrix can be X half , the third frequency feature matrix can be The filtered features can be

[0127] In the process of filtering and extracting features based on the second frequency feature matrix and recovering the complete third frequency feature matrix, the following can be done:

[0128] Based on the trainable filter matrix and the second frequency characteristic matrix, element-by-element multiplication is performed to obtain a product matrix; the product matrix is ​​conjugate-symmetric restored to obtain a third frequency characteristic matrix.

[0129] The process can be referred to the above formula (2). The trainable filter matrix can be H, and the product matrix is ​​X half ⊙H.

[0130] In the process of obtaining filtered features using a two-dimensional discrete inverse Fourier transform based on the third frequency feature matrix, it is possible to:

[0131] The third frequency feature matrix is ​​processed using a two-dimensional discrete inverse Fourier transform to obtain a time domain feature matrix; the time domain feature matrix is ​​segmented to obtain the filtered features at the current moment.

[0132] The time domain feature matrix can be The method for calculating the time domain feature matrix can be found in the above formula (3).

[0133] Step S203: Input the filtered features at the current moment into the sub-network in the strategy network to obtain the vehicle control action.

[0134] Among them, vehicle control actions can be used to control the vehicle for automatic driving, which may include acceleration, braking, and steering angles.

[0135] According to an embodiment of the present application, by inputting the observation data collected by the sensor into the Fourier filter layer in the policy network, the observation noise can be filtered, the important frequencies in the observation input can be extracted, and the filtered features can be obtained; the filtered features at the current moment are input into the subsequent sub-network in the policy network to obtain the vehicle control action for controlling the vehicle to perform autonomous driving, and the end-to-end autonomous driving task can be completed. The Jacobi regularization method is used in the sub-network to control the smoothness of the policy function, which can effectively solve the action oscillation problem of reinforcement learning for end-to-end autonomous driving, and improve the action smoothness and decision-making performance of the policy network output. At the same time, by using the Jacobi regularization method in the sub-network to control the smoothness of the policy function, matrix calculation is not required during the forward propagation process, the inference speed is faster, and it can be suitable for high real-time autonomous driving tasks.

[0136] Vehicle trajectory tracking is a crucial task in autonomous driving. To validate the effectiveness of our proposed FlipNet network in the real world, we conducted experiments on a minivan platform. The vehicle moves via two differential wheels, with the goal of tracking a reference trajectory and velocity while avoiding surrounding obstacles.

[0137] Figure 6 Schematic diagram showing an experimental scenario according to an embodiment of the present application. Figure 6 As shown, the present application constructs an experimental scenario of turning left at an intersection, so that a reinforcement learning robot (RL robot, an example of which is shown in the self-car in the figure) can successfully avoid an obstacle (Obstacle, an example of which is shown in the surrounding car in the figure) by slightly shifting to the left and tracking a reference trajectory (Reference trajectory).

[0138] Figure 7 Schematic diagram showing the experimental effect (vehicle trajectory) according to an embodiment of the present application. Figure 7 As shown, the vehicle state (such as angular velocity, linear velocity, direction) and motion trajectory generated by MLP (existing technology, multi-layer perceptron) and FlipNet (this application) are respectively shown.

[0139] Figure 8 Schematic diagram showing the experimental effect (control action) according to an embodiment of the present application. Figure 8 As shown in Figure 3, the control commands (longitudinal acceleration and yaw acceleration) generated by the MLP and FlipNet models under the same conditions are shown.

[0140] from Figure 7 and Figure 8 As can be seen from the figure, the control actions generated by FlipNet are smoother than those of MLP. These smooth actions lead to smoother vehicle states (speed, heading angle, etc.) and motion trajectories.

[0141] Figure 9 The structure diagram of the action smoothing reinforcement learning strategy network system suitable for end-to-end autonomous driving according to one embodiment of the present application is shown. Figure 9 As shown, the system includes:

[0142] Observation data acquisition unit 901, used to acquire observation data collected by sensors;

[0143] Fourier filter unit 902, used to input the observation data into the Fourier filter layer in the strategy network to obtain filtered features;

[0144] The sub-network processing unit 903 is used to input the filtered features at the current moment into the sub-network in the strategy network to obtain the vehicle control action, which is used to control the vehicle to perform automatic driving; the Jacobi regularization method is used in the sub-network to control the smoothness of the strategy function.

[0145] In a possible implementation, the Fourier filter unit 902 is configured to:

[0146] The Fourier filter layer calculates the first frequency feature matrix based on the observed data using a two-dimensional discrete Fourier transform;

[0147] Selecting the upper half or the lower half of the first frequency characteristic matrix to obtain a second frequency characteristic matrix;

[0148] Perform filtering and feature extraction based on the second frequency feature matrix to restore a complete third frequency feature matrix;

[0149] Based on the third frequency feature matrix, a two-dimensional discrete inverse Fourier transform is used to obtain the filtered features.

[0150] In one possible implementation, filtering and feature extraction are performed based on the second frequency feature matrix to restore a complete third frequency feature matrix, including:

[0151] Perform element-by-element multiplication on the trainable filter matrix and the second frequency feature matrix to obtain a product matrix;

[0152] The product matrix is ​​conjugate-symmetrized and restored to obtain the third frequency characteristic matrix.

[0153] In one possible implementation, based on the third frequency feature matrix, a two-dimensional discrete inverse Fourier transform is used to obtain filtered features, including:

[0154] The third frequency feature matrix is ​​processed using a two-dimensional discrete inverse Fourier transform to obtain a time domain feature matrix;

[0155] The time domain feature matrix is ​​divided to obtain the filtered features at the current moment.

[0156] In one possible implementation, the policy network is a trained policy network, and the system further includes:

[0157] A sample state observation acquisition unit, used to obtain an observation signal of the training sample state;

[0158] A policy network processing unit, configured to input the observation signal of the training sample state into the initial policy network to obtain a control action corresponding to the observation signal of the training sample state;

[0159] a value evaluation network processing unit for obtaining a value function estimate under the current observation and action based on the observation signal of the training sample state and the control action corresponding to the observation signal of the training sample state using the value evaluation network, wherein the value function estimate represents an estimate of the expected reward of executing the corresponding control action based on the observation signal of the training sample state;

[0160] The training unit is used to train the initial policy network based on the loss function to obtain a trained policy network, where the loss function includes an expected value estimated by a negative value function.

[0161] In one possible implementation, the loss function further includes a Jacobi regularization term, where the Jacobi regularization term is the product of the first learnable parameter and the Jacobi norm.

[0162] In one possible implementation, the Jacobi norm is an approximation of the local Lipschitz constant of the policy function, which describes the smoothness of the policy function in a preset kick-off neighborhood in Euclidean space.

[0163] In a possible implementation, the loss function further includes a filtering term, where the filtering term is the product of the second learnable parameter and the Frobenius norm of the filter matrix.

[0164] According to an embodiment of the present application, by inputting the observation data collected by the sensor into the Fourier filter layer in the policy network, the observation noise can be filtered, the important frequencies in the observation input can be extracted, and the filtered features can be obtained; the filtered features at the current moment are input into the subsequent sub-network in the policy network to obtain the vehicle control action for controlling the vehicle to perform autonomous driving, and the end-to-end autonomous driving task can be completed. The Jacobi regularization method is used in the sub-network to control the smoothness of the policy function, which can effectively solve the action oscillation problem of reinforcement learning for end-to-end autonomous driving, and improve the action smoothness and decision-making performance of the policy network output. At the same time, by using the Jacobi regularization method in the sub-network to control the smoothness of the policy function, matrix calculation is not required during the forward propagation process, the inference speed is faster, and it can be suitable for high real-time autonomous driving tasks.

[0165] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0166] According to another aspect of the present application, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0167] According to another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above method is implemented.

[0168] According to another aspect of the present application, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0169] Figure 10 1 is a block diagram of an execution device 1900 for an action smoothing reinforcement learning strategy network suitable for end-to-end autonomous driving according to an exemplary embodiment. The device can be suitable for end-to-end autonomous driving. For example, the device 1900 can be provided as a server or terminal device. Figure 10 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described method.

[0170] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , MacOS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0171] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the apparatus 1900 to perform the above-described method.

[0172] The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0173] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0174] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0175] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0176] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0177] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0178] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0179] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0180] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for executing a motion-smoothing reinforcement learning policy network suitable for end-to-end autonomous driving, characterized in that: The method comprises: Obtain observation data collected by sensors; Inputting the observed data into the Fourier filter layer in the strategy network to obtain filtered features; Inputting the filtered features at the current moment into a sub-network in the policy network to obtain a vehicle control action, which is used to control the vehicle to perform autonomous driving; the sub-network uses a Jacobi regularization method to control the smoothness of the policy function; The policy network is a trained policy network, and the method further includes: Obtain observation signals of training sample states; Inputting the observation signal of the training sample state into the initial strategy network to obtain the control action corresponding to the observation signal of the training sample state; Based on the observation signal of the training sample state and the control action corresponding to the observation signal of the training sample state, using a value evaluation network to obtain a value function estimate under the current observation and action, wherein the value function estimate represents an estimate of the expected reward of executing the corresponding control action based on the observation signal of the training sample state; Train the initial policy network based on the loss function to obtain the trained policy network; The loss function includes the expected value of the negative value function estimate, and the loss function also includes a Jacobi regularization term, where the Jacobi regularization term is the product of the first learnable parameter and the Jacobi norm, and the Jacobi norm is an approximation of the local Lipschitz constant of the strategy function, and the local Lipschitz constant describes the smoothness of the strategy function in a preset kick-off neighborhood in Euclidean space.

2. The method according to claim 1, characterized in that Inputting the observation data into the Fourier filter layer in the strategy network to obtain filtered features includes: The Fourier filter layer calculates a first frequency feature matrix based on the observation data using a two-dimensional discrete Fourier transform; Selecting the upper half or the lower half of the first frequency characteristic matrix to obtain a second frequency characteristic matrix; Perform filtering and feature extraction based on the second frequency feature matrix to restore a complete third frequency feature matrix; Based on the third frequency feature matrix, a two-dimensional discrete inverse Fourier transform is used to obtain filtered features.

3. The method according to claim 2, characterized in that The filtering and feature extraction based on the second frequency feature matrix to restore a complete third frequency feature matrix includes: Perform element-by-element multiplication on the trainable filter matrix and the second frequency feature matrix to obtain a product matrix; The product matrix is ​​conjugate-symmetric restored to obtain the third frequency characteristic matrix.

4. The method according to claim 2, characterized in that The method of obtaining filtered features by using a two-dimensional discrete inverse Fourier transform based on the third frequency feature matrix includes: Processing the third frequency feature matrix using a two-dimensional discrete inverse Fourier transform to obtain a time domain feature matrix; The time domain feature matrix is ​​segmented to obtain the filtered features at the current moment.

5. The method according to claim 1, wherein The loss function also includes a filtering term, which is the product of the second learnable parameter and the Frobenius norm of the filter matrix.

6. A motion smoothing reinforcement learning strategy network system suitable for end-to-end autonomous driving, characterized by: The system comprises: An observation data acquisition unit, used to acquire observation data collected by sensors; A Fourier filter unit, configured to input the observation data into a Fourier filter layer in the strategy network to obtain filtered features; a subnetwork processing unit configured to input the filtered features at the current moment into a subnetwork within the policy network to obtain a vehicle control action, wherein the vehicle control action is used to control the vehicle to perform autonomous driving; and wherein the Jacobi regularization method is used in the subnetwork to control the smoothness of the policy function; The policy network is a trained policy network, and the method further includes: A sample state observation acquisition unit, used to obtain an observation signal of the training sample state; a policy network processing unit, configured to input the observation signal of the training sample state into the initial policy network to obtain a control action corresponding to the observation signal of the training sample state; a value evaluation network processing unit configured to obtain, based on the observation signal of the training sample state and the control action corresponding to the observation signal of the training sample state, a value function estimate under the current observation and action using a value evaluation network, wherein the value function estimate represents an estimate of the expected reward of executing the corresponding control action based on the observation signal of the training sample state; A training unit is used to train the initial policy network based on the loss function to obtain a trained policy network; The loss function includes the expected value of the negative value function estimate, and the loss function also includes a Jacobi regularization term, where the Jacobi regularization term is the product of the first learnable parameter and the Jacobi norm, and the Jacobi norm is an approximation of the local Lipschitz constant of the strategy function, and the local Lipschitz constant describes the smoothness of the strategy function in a preset kick-off neighborhood in Euclidean space.

7. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method according to any one of claims 1 to 5 when executing the instructions stored in the memory.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

9. A computer program product comprising computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, characterized in that: When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the method according to any one of claims 1 to 5.

Images