Action smooth reinforcement learning strategy network suitable for end-to-end automatic driving

Abstract

The invention relates to an action smoothing type reinforcement learning strategy network suitable for end-to-end automatic driving. Comprises: obtaining observation data collected by a sensor; the observation data are input into a Fourier filtering layer in the strategy network, observation noise can be filtered, the important frequency in the observation input is extracted, and the filtered features are obtained; inputting the filtered features at the current moment into a subsequent sub-network in the strategy network to obtain a vehicle control action which is used for controlling the vehicle to perform automatic driving; and the sub-network uses a Jacobian regularization method to control the smoothness of the strategy function. According to the embodiment of the invention, the motion oscillation problem of reinforcement learning for end-to-end automatic driving can be effectively solved, and the motion smoothness of strategy network output is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and in particular to an action smoothing reinforcement learning strategy network suitable for end-to-end autonomous driving. Background Art

[0002] Reinforcement learning (RL) has become an important method for solving optimal control tasks in real physical environments. Since neural networks can model complex nonlinear functions, they are often used to carry out control strategies for fitting reinforcement learning. However, strategies trained by reinforcement learning often encounter the problem of motion oscillation, which causes the actions at adjacent moments to show significant differences despite slight differences in state observations at adjacent moments. Although this problem is often overlooked in the simulation and training stages, it can cause serious problems in real-world applications, such as actuator wear and safety risks. The problem of motion oscillation is also prevalent in various industrial application scenarios such as autonomous driving.

[0003] Current reinforcement learning schemes do not recognize the root cause of reinforcement learning action oscillation, resulting in poor action smoothness or decision-making performance of the policy network output. In addition, existing action smoothing techniques will lead to slow feedforward reasoning when constraining policy network parameters, making them unsuitable for real-time decision-making tasks in autonomous driving. Summary of the invention

[0004] In view of this, the present disclosure proposes an action smoothing reinforcement learning strategy network suitable for end-to-end autonomous driving.

[0005] According to one aspect of the present disclosure, a method for executing an action smoothing reinforcement learning strategy network suitable for end-to-end autonomous driving is provided. The method comprises:

[0006] Obtain observation data collected by sensors;

[0007] Input the observed data into the Fourier filter layer in the strategy network to obtain the filtered features;

[0008] The filtered features at the current moment are input into the sub-network in the strategy network to obtain the vehicle control action, which is used to control the vehicle for autonomous driving; the Jacobi regularization method is used in the sub-network to control the smoothness of the strategy function.

[0009] In one possible implementation, the observation data is input into the Fourier filter layer in the strategy network to obtain filtered features, including:

[0010] The Fourier filter layer calculates the first frequency feature matrix using a two-dimensional discrete Fourier transform based on the observed data;

[0011] Selecting the upper half or the lower half of the first frequency characteristic matrix to obtain a second frequency characteristic matrix;

[0012] Filtering and feature extraction are performed based on the second frequency feature matrix to restore a complete third frequency feature matrix;

[0013] Based on the third frequency feature matrix, a two-dimensional discrete inverse Fourier transform is used to obtain the filtered features.

[0014] In a possible implementation, filtering and feature extraction are performed based on the second frequency feature matrix to restore a complete third frequency feature matrix, including:

[0015] Performing element-by-element multiplication on the trainable filter matrix and the second frequency feature matrix to obtain a product matrix;

[0016] The product matrix is ​​conjugate-symmetric restored to obtain the third frequency characteristic matrix.

[0017] In a possible implementation, based on the third frequency feature matrix, a two-dimensional discrete inverse Fourier transform is used to obtain filtered features, including:

[0018] The third frequency feature matrix is ​​processed using a two-dimensional discrete inverse Fourier transform to obtain a time domain feature matrix;

[0019] The time domain feature matrix is ​​split to obtain the filtered features at the current moment.

[0020] In a possible implementation, the policy network is a trained policy network, and the method further includes:

[0021] Obtain observation signals of training sample states;

[0022] Input the observation signal of the training sample state into the initial strategy network to obtain the control action corresponding to the observation signal of the training sample state;

[0023] Based on the observation signal of the training sample state and the control action corresponding to the observation signal of the training sample state, the value function estimation under the current observation and action is obtained by using the value evaluation network. The value function estimation represents the estimation of the expected return of executing the corresponding control action based on the observation signal of the training sample state;

[0024] The initial policy network is trained based on the loss function to obtain a trained policy network, where the loss function includes an expected value estimated by a negative value function.

[0025] In a possible implementation, the loss function further includes a Jacobi regularization term, where the Jacobi regularization term is the product of the first learnable parameter and the Jacobi norm.

[0026] In one possible implementation, the Jacobian norm is an approximation of the local Lipschitz constant of the policy function, and the local Lipschitz constant describes the smoothness of the policy function in a preset kick-off neighborhood in the Euclidean space.

[0027] In a possible implementation, the loss function further includes a filtering term, where the filtering term is the product of the second learnable parameter and the Frobenius norm of the filter matrix.

[0028] According to another aspect of the present disclosure, an action smoothing reinforcement learning strategy network system suitable for end-to-end autonomous driving is provided. The system includes:

[0029] An observation data acquisition unit, used to acquire observation data collected by the sensor;

[0030] The Fourier filter unit is used to input the observed data into the Fourier filter layer in the strategy network to obtain the filtered features;

[0031] The sub-network processing unit is used to input the filtered features at the current moment into the sub-network in the strategy network to obtain the vehicle control action, and the vehicle control action is used to control the vehicle to perform automatic driving; the Jacobi regularization method is used in the sub-network to control the smoothness of the strategy function.

[0032] In a possible implementation, the Fourier filter unit is used to:

[0033] The Fourier filter layer calculates the first frequency feature matrix using a two-dimensional discrete Fourier transform based on the observed data;

[0034] Selecting the upper half or the lower half of the first frequency characteristic matrix to obtain a second frequency characteristic matrix;

[0035] Filtering and feature extraction are performed based on the second frequency feature matrix to restore a complete third frequency feature matrix;

[0036] Based on the third frequency feature matrix, a two-dimensional discrete inverse Fourier transform is used to obtain the filtered features.

[0037] In a possible implementation, filtering and feature extraction are performed based on the second frequency feature matrix to restore a complete third frequency feature matrix, including:

[0038] Performing element-by-element multiplication on the trainable filter matrix and the second frequency feature matrix to obtain a product matrix;

[0039] The product matrix is ​​conjugate-symmetric restored to obtain the third frequency characteristic matrix.

[0040] In a possible implementation, based on the third frequency feature matrix, a two-dimensional discrete inverse Fourier transform is used to obtain filtered features, including:

[0041] The third frequency feature matrix is ​​processed using a two-dimensional discrete inverse Fourier transform to obtain a time domain feature matrix;

[0042] The time domain feature matrix is ​​split to obtain the filtered features at the current moment.

[0043] In a possible implementation, the policy network is a trained policy network, and the system further includes:

[0044] A sample state observation acquisition unit, used to acquire an observation signal of a training sample state;

[0045] A strategy network processing unit, used to input the observation signal of the training sample state into the initial strategy network to obtain the control action corresponding to the observation signal of the training sample state;

[0046] A value evaluation network processing unit is used to obtain a value function estimate under current observation and action based on the observation signal of the training sample state and the control action corresponding to the observation signal of the training sample state by using the value evaluation network, wherein the value function estimate represents an estimate of the expected return of executing the corresponding control action based on the observation signal of the training sample state;

[0047] The training unit is used to train the initial policy network based on the loss function to obtain the trained policy network, and the loss function includes the expected value of the negative value function estimate.

[0048] In a possible implementation, the loss function further includes a Jacobi regularization term, where the Jacobi regularization term is the product of the first learnable parameter and the Jacobi norm.

[0049] In one possible implementation, the Jacobian norm is an approximation of the local Lipschitz constant of the policy function, and the local Lipschitz constant describes the smoothness of the policy function in a preset kick-off neighborhood in the Euclidean space.

[0050] In a possible implementation, the loss function further includes a filtering term, where the filtering term is the product of the second learnable parameter and the Frobenius norm of the filter matrix.

[0051] According to another aspect of the present application, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0052] According to another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.

[0053] According to another aspect of the present application, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0054] According to an embodiment of the present application, by inputting the observation data collected by the sensor into the Fourier filter layer in the strategy network, the observation noise can be filtered, the important frequencies in the observation input can be extracted, and the filtered features can be obtained; the filtered features at the current moment are input into the subsequent sub-network in the strategy network to obtain the vehicle control action for controlling the vehicle to perform autonomous driving, and the end-to-end autonomous driving task can be completed. The Jacobi regularization method is used in the sub-network to control the smoothness of the policy function, which can effectively solve the problem of action oscillation used in reinforcement learning for end-to-end autonomous driving, and improve the action smoothness and decision-making performance of the policy network output. At the same time, by using the Jacobi regularization method in the sub-network to control the smoothness of the policy function, matrix calculation is not required during the forward propagation process, and the inference speed is faster, which can be suitable for high real-time autonomous driving tasks.

[0055] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0057] Figure 1 A schematic diagram showing an application scenario according to an embodiment of the present application.

[0058] Figure 2 A decision control flow chart of an execution method of an action smoothing reinforcement learning strategy network suitable for end-to-end autonomous driving according to an embodiment of the present application is shown.

[0059] Figure 3 A schematic diagram of the structure of FlipNet according to an embodiment of the present application is shown.

[0060] Figure 4 A training flowchart of an action smoothing reinforcement learning strategy network suitable for end-to-end autonomous driving according to an embodiment of the present application is shown.

[0061] Figure 5A schematic diagram of the process of Fourier filter layer processing according to an embodiment of the present application is shown.

[0062] Figure 6 A schematic diagram showing an experimental scenario according to an embodiment of the present application.

[0063] Figure 7 A schematic diagram showing the experimental effect (vehicle trajectory) according to an embodiment of the present application.

[0064] Figure 8 A schematic diagram showing an experimental effect (control action) according to an embodiment of the present application.

[0065] Fig. 9 A structural diagram of an action smoothing reinforcement learning strategy network system suitable for end-to-end autonomous driving according to an embodiment of the present application is shown.

[0066] Fig.10 It is a block diagram of an execution device 1900 of an action smoothing reinforcement learning strategy network suitable for end-to-end autonomous driving according to an exemplary embodiment. DETAILED DESCRIPTION

[0067] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0068] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0069] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0070] Reinforcement learning (RL) has become an important method for solving optimal control tasks in real physical environments. Since neural networks can model complex nonlinear functions, they are often used to carry out control strategies for fitting reinforcement learning. However, strategies trained by reinforcement learning often encounter the problem of motion oscillation, which causes the actions at adjacent moments to show significant differences despite slight differences in state observations at adjacent moments. Although this problem is often overlooked in the simulation and training stages, it can cause serious problems in real-world applications, such as actuator wear and safety risks. The problem of motion oscillation is also prevalent in various industrial application scenarios such as autonomous driving.

[0071] Current reinforcement learning schemes do not recognize the root cause of reinforcement learning action oscillation, resulting in poor action smoothness or decision-making performance of the policy network output. In addition, existing action smoothing techniques will lead to slow feedforward reasoning when constraining policy network parameters, making them unsuitable for real-time decision-making tasks in autonomous driving.

[0072] In view of this, the present application provides an action smoothing reinforcement learning policy network suitable for end-to-end autonomous driving. The present application obtains filtered features by inputting the observation data collected by the sensor into the Fourier filter layer in the policy network; the filtered features at the current moment are input into the sub-network in the policy network to obtain the vehicle control action for controlling the vehicle to perform autonomous driving, which can complete the end-to-end autonomous driving task, wherein the Jacobi regularization method is used in the sub-network to control the smoothness of the policy function, which can solve the problem of non-smooth fitting of the policy network, effectively solve the problem of action oscillation, and improve the action smoothness and decision-making performance of the policy network output. At the same time, the Jacobi norm makes it unnecessary to perform matrix calculations during the forward propagation process, and the reasoning speed is faster, which can be applied to high real-time autonomous driving tasks.

[0073] Figure 1 A schematic diagram showing an application scenario according to an embodiment of the present application is shown. Figure 1 As shown, the action smoothing type reinforcement learning strategy network system suitable for end-to-end autonomous driving of the present application can be applied to the scenario of end-to-end autonomous driving. The action smoothing type reinforcement learning strategy network system suitable for end-to-end autonomous driving may include a strategy network (Actor) and a value evaluation network (Critic). The strategy network may be the FlipNet of an embodiment of the present application. In the process of performing autonomous driving tasks, the strategy network may output vehicle control actions based on the observation data collected by sensors to control the vehicle to achieve autonomous driving.

[0074] The action smoothing reinforcement learning strategy network system suitable for end-to-end autonomous driving in the embodiment of the present application can be deployed in a terminal device or a server. The present application does not limit the type of the terminal device or the server.

[0075] Figure 2 The following is a decision control flow chart of an execution method of an action smoothing reinforcement learning strategy network suitable for end-to-end autonomous driving according to an embodiment of the present application. Figure 2 As shown, the method may include:

[0076] Step S201, obtaining observation data collected by the sensor.

[0077] The observation data may be used to represent observations of the environmental state around the vehicle (such as image data, etc.), which may include observation data at the current moment, and may also include observation data at one or more moments before the current moment.

[0078] Step S202, input the observed data into the Fourier filter layer in the strategy network to obtain filtered features.

[0079] The following first introduces the structure of the policy network in this application.

[0080] Figure 3 FIG. 4 is a schematic diagram showing the structure of FlipNet according to an embodiment of the present application. Figure 3 As shown, FlipNet is a strategy network proposed in this application. The strategy network may include a Fourier filter layer and a sub-network f.

[0081] The Fourier filter layer can be used to filter the total N observation data at the current time (time t) and N-1 times before time t (time t to (t-N+1)) (as shown in Figure o t , o t-1 , o t-N+1 , which can be obtained based on observation data at multiple times) to obtain multiple filtered features (as shown in the figure ).like Figure 3 As shown, the Fourier filter layer may include a Fourier transform module (FFT) and an inverse Fourier transform module (IFFT), which are used to perform a fast Fourier transform (FastFourier Transform) and an inverse fast Fourier transform (Inverse Fast Fourier Transform), respectively. Half of the frequency feature matrix (frequency feature, the frequency feature matrix is ​​obtained by FFT) X half The product of the trainable filter matrix H can be used to input the inverse Fourier transform module.

[0082] The subnetwork can be used to process the filtered features and output the decision action a at time t t , the decision action is the vehicle control action output by the strategy network at time t.

[0083] Among them, the policy network is the trained policy network. Figure 4 For an introduction to the training method of the policy network, see Figure 4 , shows a training flow chart of an action smoothing reinforcement learning strategy network suitable for end-to-end autonomous driving according to an embodiment of the present application. Figure 4 As shown, the training process includes:

[0084] Step S401, obtaining an observation signal of a training sample state.

[0085] The observation signal of the training sample state may include observations of the environment state around the vehicle at multiple moments.

[0086] Step S402: input the observation signal of the training sample state into the initial strategy network to obtain the control action corresponding to the observation signal of the training sample state.

[0087] The structure of the initial policy network can be found in Figure 3 .

[0088] The observation signal of the training sample state can be used as multiple historical observation data N represents the number of historical observation data, and D represents the dimension of historical observation data.

[0089] In order to alleviate the motion vibration problem caused by observation noise, a Fourier filter layer based on Fourier transform is proposed in this application.

[0090] Figure 5 FIG. 2 is a schematic diagram showing a process flow of Fourier filter layer processing according to an embodiment of the present application. Figure 5 As shown, multiple historical observation data (o t ,…,o t-N+1 ) are concatenated into a matrix Since zero padding does not change the main frequency components of the signal in Fourier transform, but only increases the spectral resolution, when the length of the historical observation data is less than N, the missing part can be filled with 0.

[0091] The frequency feature matrix can be calculated by using the two-dimensional discrete Fourier transform (DFT) to perform Fourier transform (FFT) One calculation method can be found in formula (1):

[0092]

[0093] Among them, X u,v It can represent the element value of the uth row and vth column in the frequency feature matrix X, x n,d It can represent the d-th dimension feature value of the n-th historical observation data. u,v The size of can represent the signal strength of the historical observation data under the frequency combination (u, v), where u and v are frequency indices rather than actual frequency values.

[0094] Since the historical observation data only contains real numbers, the resulting frequency feature matrix X has conjugate symmetry, that is, This means that only half of the matrix X can be used to represent the complete information contained in the signal. In this application, the upper half or the lower half of the frequency characteristic matrix is ​​selected to obtain half of the frequency characteristic matrix, and the half matrix of X is recorded as X-based half Subsequent processing can reduce the amount of calculation.

[0095] In this application, a trainable filter matrix is ​​introduced By adding X half After Hadamard multiplication with H (i.e., element-by-element multiplication), the product matrix is ​​restored by conjugate symmetrization (i.e., restoring the conjugate symmetry of the product matrix) to obtain a restored frequency characteristic matrix The calculation process can be seen in formula (2):

[0096]

[0097] where symmetrize(·) represents the conjugate symmetry restoration operation, X half ⊙H can represent the half The product matrix obtained by element-wise multiplication with H.

[0098] Since the filter matrix H is a complex matrix, by introducing the filter matrix in the Fourier filter layer, not only can X be changed half The frequency amplitude can also be used to extract its features. The modulus length of the elements in the filter matrix H can be used to determine whether the frequency is suppressed or enhanced. In order to enable the policy network to have the observation noise filtering function, the modulus length of the elements in H of the present application is a smaller value. Thus, the policy network can automatically extract more valuable frequencies and filter out irrelevant frequencies that may have noise to solve the problem of action oscillation, making the policy network more robust.

[0099] Finally, the Fourier filter layer can perform an inverse Fourier transform (IFFT) through a two-dimensional discrete inverse Fourier transform (IDFT) to restore the frequency feature matrix Converted into a time domain feature matrix, the calculation process can be seen in formula (3):

[0100]

[0101] in, It can represent the element value of the nth row and dth column in the time domain feature matrix. is a real matrix, we can Perform segmentation to obtain filtered features

[0102] The filtered features at the current moment can be As the input of the sub-network f. The sub-network can be regarded as a mapping function used to fit the policy function π: The policy function can transform the state space Mapping to action space The state space may correspond to the filtered features, and the action space may correspond to the control actions. Thus, the subnetwork may be used to map the filtered features to obtain the control actions.

[0103] In this application, in order to enhance the smoothness of the policy network and further solve the problem of action oscillation, the Jacobi regularization method is used to control the smoothness of the policy function (continuously differentiable neural network, i.e., sub-network). The Jacobi norm is introduced in the Jacobi regularization method, which is an approximation of the local Lipschitz constant of the policy function. For a continuously differentiable sub-network f: Jacobian norm is an approximation of the local Lipschitz constant of the policy function f in the infinitesimal neighborhood of x, that is

[0104] K(x) is the local Lipschitz constant of f near x, and its calculation method can be seen in formula (4):

[0105]

[0106] The local Lipschitz constant describes the preset kick-off neighborhood of the strategy function f in Euclidean space. The degree of smoothness in . represents the open ball neighborhood in the Euclidean space with x as the center and radius ρ, that is, ‖·‖ represents the 2-norm. x1 and x2 are the kick-off neighborhoods. For the points in , f(x1) and f(x2) can represent the values ​​of points x1 and x2 in the strategy function f respectively.

[0107] Step S403, based on the observation signal of the training sample state and the control action corresponding to the observation signal of the training sample state, the value function estimation under the current observation and action is obtained by using the value evaluation network.

[0108] The value evaluation network is the Critic network in deep learning (see Figure 1 ), the Critic network can be used to fit the value function Q(s,a), through which the value function can be estimated based on the observation signal of the training sample state and the control action corresponding to the observation signal of the training sample state. The value function estimate can represent the estimate of the expected return of executing the corresponding control action based on the observation signal of the training sample state. The value function estimate can be used to evaluate the value of the action taken by the policy network (i.e., the output control action).

[0109] Step S404: Train the initial policy network based on the loss function to obtain a trained policy network.

[0110] The loss function may include the expected value of the negative value function estimate. A method for calculating the loss function can be seen in formula (5):

[0111]

[0112] in, The loss function can be expressed as Indicates that the experience replay pool The expected value of the sampled state s (corresponding to the state of the input policy network) in . represents the value function estimate, and π(·) is the policy function.

[0113] The trained policy network can be obtained by minimizing the loss function, that is, by maximizing the expected return predicted by the Critic network to update the parameters in the initial policy network.

[0114] Optionally, the loss function also includes a Jacobian regularization term, which is the product of the first learnable parameter and the Jacobian norm. The calculation method of the optimized loss function can be seen in formula (6):

[0115]

[0116] Among them, λ k is the first learnable parameter, represents the Jacobian norm, that is, the norm of the Jacobian matrix, which is an approximation of the local Lipschitz constant. k and Adaptive updates can be performed during the training process, which can flexibly enhance the smoothness of the strategy and make the output of the strategy network smoother.

[0117] Since the loss function including the Jacobian regularization term is applied to the training of the entire policy network, serious performance loss can be avoided. The Jacobian regularization term does not need to be manually set to an initial value, and does not impair the exploration ability of reinforcement learning. Compared with existing action smoothing methods such as LipsNet, the introduction of the Jacobian regularization term does not require the calculation of the Jacobian matrix during the forward propagation process, and the inference speed is faster.

[0118] Optionally, the loss function further includes a filtering term, which is the product of the second learnable parameter and the Frobenius norm of the filter matrix. The calculation method of the optimized loss function can be seen in formula (7):

[0119]

[0120] Among them, λ h is the second learnable parameter, which can be adaptively updated during the training process, ‖H‖ F is the Frobenius norm of the filter matrix H.

[0121] The loss function can be minimized based on the gradient descent method To update the parameters in the policy network, obtain the trained policy network.

[0122] By introducing both Jacobian regularization and filtering terms into the loss function, the problems of non-smooth policy network fitting and environmental observation noise can be solved simultaneously during the training of the policy network, so that the action smoothness and decision-making performance of the trained policy network can be optimized.

[0123] After completing the training of the policy network, you can return to executing steps S201 to S203 to use the trained policy network to perform the autonomous driving task.

[0124] The process may return to step S202. In step S202, the following may be performed:

[0125] The Fourier filter layer calculates the first frequency feature matrix based on the observed data using a two-dimensional discrete Fourier transform; selects the upper or lower half of the first frequency feature matrix to obtain a second frequency feature matrix; performs filtering and feature extraction based on the second frequency feature matrix to restore a complete third frequency feature matrix; and uses a two-dimensional discrete inverse Fourier transform based on the third frequency feature matrix to obtain filtered features.

[0126] This process can be seen in the implementation of step S402, where the observation data can be o t,o t-1 ,…,o t-N+1 , the first frequency feature matrix can be X, and the second frequency feature matrix can be X half , the third frequency feature matrix can be The features after filtering can be

[0127] In the process of filtering and extracting features based on the second frequency feature matrix and restoring the complete third frequency feature matrix, the following can be done:

[0128] Based on the trainable filter matrix and the second frequency characteristic matrix, element-by-element multiplication is performed to obtain a product matrix; the product matrix is ​​conjugate symmetric restored to obtain a third frequency characteristic matrix.

[0129] The process can be referred to in the above formula (2). The trainable filter matrix can be H, and the product matrix is ​​X half ⊙H.

[0130] In the process of obtaining the filtered features by using the two-dimensional discrete inverse Fourier transform based on the third frequency feature matrix, it is possible to:

[0131] The third frequency feature matrix is ​​processed using a two-dimensional discrete inverse Fourier transform to obtain a time domain feature matrix; the time domain feature matrix is ​​segmented to obtain the filtered features at the current moment.

[0132] The time domain feature matrix can be The method of calculating the time domain feature matrix can be found in the above formula (3).

[0133] Step S203, inputting the filtered features at the current moment into the sub-network in the strategy network to obtain the vehicle control action.

[0134] Among them, vehicle control actions can be used to control the vehicle for autonomous driving, which may include acceleration, braking, and steering angles, etc.

[0135] According to an embodiment of the present application, by inputting the observation data collected by the sensor into the Fourier filter layer in the strategy network, the observation noise can be filtered, the important frequencies in the observation input can be extracted, and the filtered features can be obtained; the filtered features at the current moment are input into the subsequent sub-network in the strategy network to obtain the vehicle control action for controlling the vehicle to perform autonomous driving, and the end-to-end autonomous driving task can be completed. The Jacobi regularization method is used in the sub-network to control the smoothness of the policy function, which can effectively solve the problem of action oscillation used in reinforcement learning for end-to-end autonomous driving, and improve the action smoothness and decision-making performance of the policy network output. At the same time, by using the Jacobi regularization method in the sub-network to control the smoothness of the policy function, matrix calculation is not required during the forward propagation process, and the inference speed is faster, which can be suitable for high real-time autonomous driving tasks.

[0136] Vehicle trajectory tracking is an important task in autonomous driving. In order to verify the effect of the FlipNet network in this application in the real world, this application conducted experiments on a micro-car platform. The vehicle moves through two differential wheels, and the goal is to track the reference trajectory and speed while avoiding surrounding obstacles.

[0137] Figure 6 FIG. 1 is a schematic diagram showing an experimental scenario according to an embodiment of the present application. Figure 6 As shown, the present application constructs an experimental scenario of turning left at an intersection, so that a reinforcement learning robot (RL robot, an example of which is shown in the self-car in the figure) successfully avoids an obstacle (Obstacle, an example of which is shown in the surrounding car in the figure) by slightly shifting to the left and tracking a reference trajectory (Referencetrajectory).

[0138] Figure 7 FIG. 1 is a schematic diagram showing the experimental effect (vehicle trajectory) according to an embodiment of the present application. Figure 7 As shown, the vehicle state (such as angular velocity, linear velocity, direction) and motion trajectory generated by MLP (existing technology, multi-layer perceptron) and FlipNet (this application) are respectively shown.

[0139] Figure 8 FIG. 1 is a schematic diagram showing the experimental effect (control action) according to an embodiment of the present application. Figure 8 As shown, the control commands (longitudinal acceleration and yaw acceleration) generated by the MLP and FlipNet models under the same conditions are shown.

[0140] from Figure 7 and Figure 8 It can be seen that the control actions generated by FlipNet are smoother than those of MLP. These smooth actions lead to smoother vehicle states (speed, heading angle, etc.) and motion trajectories.

[0141] Fig. 9 FIG. 1 is a structural diagram of an action smoothing reinforcement learning strategy network system suitable for end-to-end autonomous driving according to an embodiment of the present application. Fig. 9 As shown, the system includes:

[0142] An observation data acquisition unit 901 is used to acquire observation data collected by a sensor;

[0143] Fourier filter unit 902, used to input the observation data into the Fourier filter layer in the strategy network to obtain filtered features;

[0144] The sub-network processing unit 903 is used to input the filtered features at the current moment into the sub-network in the strategy network to obtain the vehicle control action, which is used to control the vehicle to perform automatic driving; the Jacobi regularization method is used in the sub-network to control the smoothness of the strategy function.

[0145] In a possible implementation, the Fourier filter unit 902 is configured to:

[0146] The Fourier filter layer calculates the first frequency feature matrix using a two-dimensional discrete Fourier transform based on the observed data;

[0147] Selecting the upper half or the lower half of the first frequency characteristic matrix to obtain a second frequency characteristic matrix;

[0148] Filtering and feature extraction are performed based on the second frequency feature matrix to restore a complete third frequency feature matrix;

[0149] Based on the third frequency feature matrix, a two-dimensional discrete inverse Fourier transform is used to obtain the filtered features.

[0150] In a possible implementation, filtering and feature extraction are performed based on the second frequency feature matrix to restore a complete third frequency feature matrix, including:

[0151] Performing element-by-element multiplication on the trainable filter matrix and the second frequency feature matrix to obtain a product matrix;

[0152] The product matrix is ​​conjugate-symmetric restored to obtain the third frequency characteristic matrix.

[0153] In a possible implementation, based on the third frequency feature matrix, a two-dimensional discrete inverse Fourier transform is used to obtain filtered features, including:

[0154] The third frequency feature matrix is ​​processed using a two-dimensional discrete inverse Fourier transform to obtain a time domain feature matrix;

[0155] The time domain feature matrix is ​​split to obtain the filtered features at the current moment.

[0156] In a possible implementation, the policy network is a trained policy network, and the system further includes:

[0157] A sample state observation acquisition unit, used to acquire an observation signal of a training sample state;

[0158] A strategy network processing unit, used to input the observation signal of the training sample state into the initial strategy network to obtain the control action corresponding to the observation signal of the training sample state;

[0159] A value evaluation network processing unit is used to obtain a value function estimate under current observation and action based on the observation signal of the training sample state and the control action corresponding to the observation signal of the training sample state by using the value evaluation network, wherein the value function estimate represents an estimate of the expected return of executing the corresponding control action based on the observation signal of the training sample state;

[0160] The training unit is used to train the initial policy network based on the loss function to obtain the trained policy network, and the loss function includes the expected value of the negative value function estimate.

[0161] In a possible implementation, the loss function further includes a Jacobi regularization term, where the Jacobi regularization term is the product of the first learnable parameter and the Jacobi norm.

[0162] In one possible implementation, the Jacobian norm is an approximation of the local Lipschitz constant of the policy function, and the local Lipschitz constant describes the smoothness of the policy function in a preset kick-off neighborhood in the Euclidean space.

[0163] In a possible implementation, the loss function further includes a filtering term, where the filtering term is the product of the second learnable parameter and the Frobenius norm of the filter matrix.

[0164] According to an embodiment of the present application, by inputting the observation data collected by the sensor into the Fourier filter layer in the strategy network, the observation noise can be filtered, the important frequencies in the observation input can be extracted, and the filtered features can be obtained; the filtered features at the current moment are input into the subsequent sub-network in the strategy network to obtain the vehicle control action for controlling the vehicle to perform autonomous driving, and the end-to-end autonomous driving task can be completed. The Jacobi regularization method is used in the sub-network to control the smoothness of the policy function, which can effectively solve the problem of action oscillation used in reinforcement learning for end-to-end autonomous driving, and improve the action smoothness and decision-making performance of the policy network output. At the same time, by using the Jacobi regularization method in the sub-network to control the smoothness of the policy function, matrix calculation is not required during the forward propagation process, and the inference speed is faster, which can be suitable for high real-time autonomous driving tasks.

[0165] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0166] According to another aspect of the present application, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0167] According to another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.

[0168] According to another aspect of the present application, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0169] Fig.10 1 is a block diagram of an execution device 1900 of an action smoothing reinforcement learning strategy network suitable for end-to-end autonomous driving according to an exemplary embodiment. The device can be suitable for end-to-end autonomous driving. For example, the device 1900 can be provided as a server or a terminal device. Fig.10 , the apparatus 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions, such as an application, that can be executed by the processing component 1922. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0170] The device 1900 may also include a power supply component 1926 configured to perform power management of the device 1900, a wired or wireless network interface 1950 configured to connect the device 1900 to a network, and an input / output interface 1958 (I / O interface). The device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2000. TM , MacOS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0171] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the device 1900 to perform the above method.

[0172] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0173] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0174] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0175] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0176] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0177] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0178] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0179] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0180] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for executing a motion smoothing reinforcement learning policy network suitable for end-to-end autonomous driving, characterized in that: The method comprises: Obtain observation data collected by sensors; Inputting the observed data into the Fourier filter layer in the strategy network to obtain filtered features; The filtered features at the current moment are input into a sub-network in the strategy network to obtain a vehicle control action, wherein the vehicle control action is used to control the vehicle to perform automatic driving; the sub-network uses a Jacobi regularization method to control the smoothness of the strategy function.

2. The method according to claim 1, characterized in that The step of inputting the observed data into the Fourier filter layer in the strategy network to obtain filtered features includes: The Fourier filter layer calculates a first frequency feature matrix based on the observed data using a two-dimensional discrete Fourier transform; Selecting the upper half or the lower half of the first frequency characteristic matrix to obtain a second frequency characteristic matrix; Filtering and feature extraction are performed based on the second frequency feature matrix to restore a complete third frequency feature matrix; Based on the third frequency feature matrix, a two-dimensional discrete inverse Fourier transform is used to obtain filtered features.

3. The method according to claim 2, characterized in that The filtering and feature extraction based on the second frequency feature matrix to restore a complete third frequency feature matrix includes: Performing element-by-element multiplication on the trainable filter matrix and the second frequency feature matrix to obtain a product matrix; The product matrix is ​​conjugate-symmetric restored to obtain the third frequency characteristic matrix.

4. The method according to claim 2, characterized in that: The method of obtaining filtered features by using a two-dimensional discrete inverse Fourier transform based on the third frequency feature matrix includes: Processing the third frequency feature matrix using a two-dimensional discrete inverse Fourier transform to obtain a time domain feature matrix; The time domain feature matrix is ​​segmented to obtain the filtered features at the current moment.

5. The method according to claim 1, characterized in that The policy network is a trained policy network, and the method further includes: Obtain observation signals of training sample states; Inputting the observation signal of the training sample state into the initial strategy network to obtain the control action corresponding to the observation signal of the training sample state; Based on the observation signal of the training sample state and the control action corresponding to the observation signal of the training sample state, using the value evaluation network to obtain a value function estimate under the current observation and action, wherein the value function estimate represents an estimate of the expected reward of executing the corresponding control action based on the observation signal of the training sample state; The initial policy network is trained based on a loss function to obtain a trained policy network, wherein the loss function includes an expected value estimated by a negative value function.

6. The method according to claim 5, characterized in that The loss function also includes a Jacobian regularization term, which is the product of the first learnable parameter and the Jacobian norm.

7. The method according to claim 1 or 6, characterized in that: The Jacobian norm is an approximation of the local Lipschitz constant of the strategy function, which describes the smoothness of the strategy function in a preset kick-off neighborhood in the Euclidean space.

8. The method according to claim 5 or 6, characterized in that: The loss function also includes a filtering term, which is the product of the second learnable parameter and the Frobenius norm of the filter matrix.

9. An action smoothing reinforcement learning strategy network system suitable for end-to-end autonomous driving, characterized in that: The system comprises: An observation data acquisition unit, used to acquire observation data collected by the sensor; A Fourier filter unit, used for inputting the observed data into the Fourier filter layer in the strategy network to obtain filtered features; The sub-network processing unit is used to input the filtered features at the current moment into the sub-network in the strategy network to obtain a vehicle control action, wherein the vehicle control action is used to control the vehicle to perform automatic driving; the Jacobi regularization method is used in the sub-network to control the smoothness of the strategy function.

10. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the method described in any one of claims 1 to 8 when executing the instructions stored in the memory.

11. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising computer readable code, or a non-volatile computer readable storage medium carrying computer readable code, characterized in that: When the computer readable code runs in a processor of an electronic device, the processor in the electronic device executes the method according to any one of claims 1 to 8.

Images