A control system and method for autonomous driving

Abstract

The present invention discloses a control system for autonomous driving, which includes a multi-step control signal prediction module, a trajectory prediction module, and a fusion module. Among them, the multi-step control signal prediction module is used to predict the first control signal for each time step within a specified future time period, the trajectory prediction module is used to predict the driving trajectory within a specified future time period, and obtain the corresponding second control signal according to the driving trajectory. The fusion module then fuses the first and second control signals to obtain a third control signal as the final control signal.

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving, and particularly relates to a control system and method for autonomous driving. Background Art

[0002] As a hot technology in the field of artificial intelligence, in recent years, autonomous driving technology has developed rapidly and has been popularized in mass-produced vehicles.

[0003] Currently, most autonomous driving is achieved based on vision solutions, mainly including indirect perception methods, direct perception methods, and end-to-end control methods. Among them, the end-to-end control method refers to extracting high-dimensional effective features from signals collected by vehicle body vision sensors and the like, and then predicting trajectory signals or control signals based on the high-dimensional effective features. Compared with other autonomous driving methods, the end-to-end control method can avoid cascading errors and complex artificial rule specifications, is closer to human driving habits, and has a lower development cost, so it is widely applied.

[0004] Current end-to-end control methods can be divided into two categories: The first category is to first predict a trajectory and then convert the trajectory into a control signal through a downstream controller, and the second category is to directly output a control signal. For the first category of control methods, limited by the accuracy of the downstream controller, the obtained control signal often makes it difficult for the vehicle to actually drive out a trajectory that perfectly fits the predicted trajectory, while the second category of control methods may cause situations such as collisions due to untimely braking because they only focus on the current moment. Summary of the Invention

[0005] In view of some or all of the problems in the prior art, on the one hand, the present invention provides a control system for autonomous driving, including:

[0006] A multi-step control signal prediction module, which is used to form a first control signal, and the formation of the first control signal includes: predicting the first control signal at each time step within a specified future time period;

[0007] A trajectory prediction module, which is used to form a second control signal, and the formation of the second control signal includes: predicting the driving trajectory within a specified future time period and obtaining the corresponding second control signal according to the driving trajectory; and

[0008] A fusion module, which is used to fuse the first and second control signals to obtain a third control signal.

[0009] Further, the multi-step signal prediction module includes:

[0010] A timing module, which is used to calculate the environmental and vehicle own information at each time step; and

[0011] An attention module for predicting control signals at each time step.

[0012] Furthermore, the multi-step signal prediction module and / or the trajectory prediction module are implemented based on a gated recurrent neural network.

[0013] Furthermore, the first control signal, the second control signal, and the third control signal include at least one of the following signals: a steering wheel control signal, an accelerator control signal, and a brake control signal.

[0014] Furthermore, the control system further includes an information encoding module for encoding the system input signal into a feature vector and sending it to the multi-step control signal prediction module and the trajectory prediction module.

[0015] Based on the control system as described above, another aspect of the present invention further provides a control method for autonomous driving, including:

[0016] Through the trajectory prediction module, predicting a trajectory within a specified future time period based on a 2D feature map and a vehicle state feature vector, and forming a second control signal according to the trajectory;

[0017] Through the multi-step signal prediction module, predicting the first control signal at each time step within a specified future time period based on a 2D feature map and a vehicle state feature vector;

[0018] Fusing the first and second control signals through a fusion module to obtain a third control signal; and

[0019] Controlling the vehicle for autonomous driving according to the third control signal.

[0020] Furthermore, the formation of the 2D feature map includes:

[0021] Converting an input RGB image into a 2D feature map through a convolutional neural network.

[0022] Furthermore, the formation of the vehicle state feature vector includes:

[0023] Concatenating the input current vehicle speed and navigation information, and converting the current vehicle speed and navigation information into a vehicle state feature vector through a multi-layer perceptron.

[0024] Furthermore, the prediction of the trajectory includes:

[0025] Performing global average pooling on the 2D feature map and concatenating it with the vehicle state feature vector to obtain concatenated information;

[0026] Inputting the concatenated information into a multi-layer perceptron;

[0027] Feed the output of the multi-layer perceptron into a gated recurrent neural network to predict the coordinates at each time step within the specified future time period in an autoregressive manner; and

[0028] Combine the coordinates at each time step into a trajectory.

[0029] Further, the forming of the second control signal according to the trajectory includes:

[0030] Feed the trajectory into a lateral and longitudinal PID controller to obtain a second control signal.

[0031] Further, the prediction of the first control signal includes:

[0032] Through a timing module, based on the 2D feature map and the vehicle state feature vector, obtain a first latent variable, where the first latent variable includes the environment and vehicle self-information at the next time step;

[0033] Input the first latent variable and the second latent variable at the corresponding time step in the trajectory prediction module into a multi-layer perceptron to obtain an attention matrix, where the second latent variable includes the environment and vehicle self-information at the next time step;

[0034] According to the attention matrix, re-aggregate the 2D feature map and aggregate it with the first latent variable to obtain a representation vector; and

[0035] Based on the representation vector, form the first control signal at the next time step through a multi-layer perceptron.

[0036] Further, the forming of the third control signal includes:

[0037] According to the current environment information and the vehicle's own state, determine the respective fusion ratios of the first control signal and the second control signal, and according to the fusion ratios, fuse the first control signal and the second control signal to obtain a third control signal, where the sum of the fusion ratios of the first and second control signals is equal to 100%.

[0038] Further, determining the fusion ratio according to the current environment information and the vehicle's own state includes:

[0039] If the current vehicle is in a turning state, the fusion ratio of the first control signal is higher than that of the second control signal; and

[0040] If the current vehicle is in a straight state, the fusion ratio of the first control signal is lower than that of the second control signal.

[0041] A control system and method for autonomous driving provided by the present invention combines trajectory prediction and direct control prediction. On the one hand, by fusing the two control signals, the accuracy of the final control signal can be effectively improved, and thus the actual driving trajectory obtained according to the final control signal is closer to the predicted trajectory. On the other hand, in the present invention, the direct control prediction is a multi-step control prediction achieved through trajectory guidance. Therefore, it can solve the problem that only the current moment is concerned in the existing direct control prediction model. In addition, the fusion scheme of the two control signals can also be adjusted according to the actual situation to achieve a better control effect. It has been verified that based on the control system and method, good driving effects can be achieved with only a single monocular camera as the input, greatly reducing collisions and violations. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] To further clarify the above and other advantages and features of the embodiments of the present invention, a more specific description of the embodiments of the present invention will be presented with reference to the accompanying drawings. It can be understood that these drawings only depict typical embodiments of the present invention and thus will not be considered as limiting its scope. In the drawings, for clarity, the same or corresponding components will be denoted by the same or similar reference numerals.

[0043] Figure 1 A schematic structural diagram of a control system for autonomous driving showing an embodiment of the present invention; and

[0044] Figure 2 A schematic flowchart of a control method for autonomous driving showing an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0045] In the following description, the present invention is described with reference to the embodiments. However, those skilled in the art will recognize that the embodiments can be implemented without one or more of the specific details or in combination with other alternative and / or additional methods. In other cases, well-known operations are not shown or described in detail to avoid obscuring the inventive points of the present invention. Similarly, for the purpose of explanation, specific configurations are set forth in order to provide a comprehensive understanding of the embodiments of the present invention. However, the present invention is not limited to these specific details.

[0046] In this specification, the reference to "an embodiment" or "the embodiment" means that the specific features, structures, or characteristics described in connection with the embodiment are included in at least one embodiment of the present invention. The phrase "in an embodiment" appearing throughout this specification does not necessarily refer to the same embodiment.

[0047] It should be noted that the embodiments of the present invention describe the method steps in a specific order. However, this is only for explaining the specific embodiment and does not limit the sequence of the steps. On the contrary, in different embodiments of the present invention, the sequence of the steps can be adjusted according to actual needs.

[0048] In the existing end-to-end control methods, the trajectory prediction model subsequently requires a controller to convert the trajectory into a control signal. Currently, relatively simple controllers, such as PID controllers, are mostly used to implement this function. However, simple controllers perform poorly when the trajectory changes significantly and require a large amount of parameter tuning. The model that directly predicts the control signal only focuses on the current moment, resulting in the phenomenon of collisions due to untimely braking, and the frames are independent of each other, resulting in discontinuous control and large jitters. In view of the deficiencies of the two models, on the one hand, this application uses the information of trajectory prediction combined with a timing module to implement multi-step control signal prediction to alleviate the timeliness problem of the model that directly predicts the control signal. On the other hand, a fusion scheme is formulated in combination with the current state to fuse the control signals output by the two models, so that the trajectory traveled according to the final control signal is closer to the predicted trajectory.

[0049] The following further describes the solution of the present invention in conjunction with the accompanying drawings of the embodiments.

[0050] Figure 1 The structural schematic diagram of a control system for autonomous driving showing an embodiment of the present invention is presented.

[0051] As Figure 1 shown, a control system for autonomous driving includes a multi-step control signal prediction module 101, a trajectory prediction module 102, and a fusion module 103. Among them, the multi-step control signal prediction module 101 is used to predict the first control signal for each time step within a specified future time period, the trajectory prediction module 102 is used to predict the driving trajectory within a specified future time period and obtain the corresponding second control signal according to the driving trajectory, and the fusion module 103 is used to fuse the first and second control signals to obtain a third control signal as the final control signal.

[0052] In the embodiments of the present invention, the term "control signal" mainly includes a steering wheel control signal, an accelerator control signal, and a brake control signal. Therefore, the first control signal, the second control signal, and the third control signal as described above include at least one of the following signals: a steering wheel control signal, an accelerator control signal, and a brake control signal. According to the control signal, the driving direction and driving speed of the vehicle can be controlled. Specifically, the rotation angle of the steering wheel is controlled by the steering wheel control signal, thereby controlling the driving direction of the vehicle, and the depression degrees of the accelerator and the brake are respectively controlled by the accelerator control signal and the brake control signal, thereby controlling the driving speed of the vehicle.

[0053] In one embodiment of the present invention, the multi-step control signal prediction module 101 and the trajectory prediction module 102 perform trajectory and / or control signal prediction based on the 2D feature map and the vehicle state feature vector. Among them, the 2D feature map and the vehicle state feature vector are generated by the information encoding module 104. Specifically, the information encoding module 104 may include, for example, a convolutional neural network and a multi-layer perceptron. Among them, the convolutional neural network is used to convert the input RGB image into a 2D feature map, and the multi-layer perceptron can convert the concatenated input current vehicle speed and navigation information into a vehicle state feature vector. Among them, the RGB image can be obtained, for example, only by a single monocular camera. The navigation information includes at least one navigation point coordinate and a discrete navigation instruction, and thus information such as vehicle position and direction can be obtained based on the navigation information.

[0054] In one embodiment of the present invention, the multi-step signal prediction module 101 includes a timing module and an attention module. Among them, the timing module is implemented based on a gated recurrent neural network (GRU). At the future time step t, the input of the timing module is the current, the representation vector at the t-th time step, and the currently predicted control signal. The timing module infers the interaction and change process between the environment and the ego vehicle based on the representation vector and the control signal, and finally outputs a first hidden variable containing the environment and vehicle own information at the t+1 moment. The attention module is guided by the trajectory information obtained by the trajectory prediction module. Specifically, it aggregates the second hidden variable at the corresponding time step in the trajectory prediction module with the first hidden variable, and then predicts a 2D attention matrix through a multi-layer perceptron to be used for re-aggregating the 2D feature map obtained from the RGB image, and then aggregating it with the first hidden variable into a representation vector at the t+1 time step. The representation vector corresponding to each time step can obtain the corresponding first control signal through a shared multi-layer perceptron. In the present invention, the term "representation vector" refers to the feature vector obtained by converting the environment and vehicle own information at the corresponding time step.

[0055] In one embodiment of the present invention, the trajectory prediction module 102 is implemented based on a gated recurrent neural network (GRU). Specifically, the trajectory prediction module 102 can concatenate the 2D feature map after global average pooling and the vehicle state feature vector, and send the concatenated result through a multi-layer perceptron and then into a gated recurrent neural network (GRU) to predict the coordinates of each time step within a specified future time period in an autoregressive manner, and finally perform multi-step coordinate merging to obtain the predicted driving trajectory. In one embodiment of the present invention, the trajectory prediction module 102 can further send the driving trajectory into a lateral and longitudinal PID controller to obtain the corresponding second control signal.

[0056] Based on the control system described above, Figure 2 The flowchart shows a control method for autonomous driving according to an embodiment of the present invention. As Figure 2 shown, a control method for autonomous driving includes:

[0057] First, in step 101, feature extraction. Through the information encoding module 104, information encoding is performed on the input RGB image, current vehicle speed, navigation information, etc., and then features for trajectory and / or control signal prediction are extracted. In an embodiment of the present invention, the input RGB image is converted into a 2D feature map through a convolutional neural network. At the same time, the input current vehicle speed and navigation information are concatenated and input into a multi-layer perceptron to obtain a vehicle state feature vector; it should be understood that in other embodiments of the present invention, other networks may be used to replace the convolutional neural network; in addition, the input RGB image may be obtained by a single monocular camera, or may be an image obtained by one or more other sensors, and / or an image obtained after a certain image fusion process; the navigation information may include, for example, navigation point coordinates and discrete navigation instructions;

[0058] Next, in step 102, trajectory prediction. Through the trajectory prediction module, based on the 2D feature map and the vehicle state feature vector, the driving trajectory within a specified future time period is predicted. In an embodiment of the present invention, the trajectory prediction includes:

[0059] Performing global average pooling on the 2D feature map and concatenating it with the vehicle state feature vector to obtain concatenated information;

[0060] Inputting the concatenated information into a multi-layer perceptron;

[0061] Sending the output of the multi-layer perceptron to a gated recurrent neural network to predict the coordinates of each time step within the specified future time period in an autoregressive manner. In an embodiment of the present invention, the duration of the specified future time period is 2s, and one coordinate is predicted every 0.5s. Therefore, a total of 4 coordinate values are output. It should be understood that in other embodiments of the present invention, the duration of the specified future time period may be longer or shorter, and the selection of time steps may also be different; and

[0062] Combining the coordinates of each time step into a driving trajectory;

[0063] Next, in step 103, forming a second control signal. According to the driving trajectory, a second control signal is converted. In an embodiment of the present invention, the driving trajectory is sent into a lateral and longitudinal PID controller to obtain a corresponding second control signal;

[0064] Meanwhile, at step 104, a first control signal is formed. Through the multi-step signal prediction module, the first control signal for each time step within a specified future time period is predicted based on the 2D feature map and the vehicle state feature vector. In an embodiment of the present invention, the formation of the first control signal includes:

[0065] Through the timing module, based on the 2D feature map and the vehicle state feature vector, a first latent variable is obtained. The first latent variable includes the environment and vehicle self-information of the next time step. Specifically, at the initial time step, the inputs to the timing module are the 2D feature map obtained through the information encoding module and the vehicle state feature vector. At the future t-th time step, the inputs to the timing module include, at the previous time step, i.e., the (t - 1)-th time step, the characterization vector and the predicted first control signal obtained through the multi-step signal prediction module, where the characterization vector refers to the feature vector corresponding to the environment and vehicle self-information predicted for the t-th time step.

[0066] Aggregate the first latent variable with the second latent variable of the corresponding time step in the trajectory prediction module and input it into a multi-layer perceptron to obtain an attention matrix, where the second latent variable refers to the environment and vehicle self-information of the next time step obtained through the trajectory prediction module.

[0067] According to the attention matrix, re-aggregate the 2D feature map and aggregate it with the first latent variable to obtain a characterization vector; and

[0068] Based on the characterization vector, form the first control signal of the next time step through a multi-layer perceptron.

[0069] In an embodiment of the present invention, the first control signal and the characterization vector of each step predicted through the multi-step signal prediction module are supervised by the true value of the expert model, so that the multi-step signal prediction module has a certain temporal correlation reasoning ability within a relatively short time range, such as what kind of control prediction should be made currently to make the future environment and the state of the vehicle itself similar to the expert model; and

[0070] Finally, at step 105, the control signals are fused. The first and second control signals are fused by a fusion module to obtain a third control signal, and the vehicle's autonomous driving is controlled according to the third control signal. After obtaining the second and first control signals from the trajectory prediction module and the multi-step signal prediction module respectively, the module that is more advantageous currently can be determined according to the current state, and when combining the two, the proportion of the advantageous module can be made larger. That is to say, according to the current environmental information and the vehicle's own state, the respective fusion ratios of the first control signal and the second control signal can be determined, and according to the fusion ratios, the first control signal and the second control signal are fused to obtain a third control signal, where the sum of the fusion ratios of the first and second control signals is equal to 100%. According to experiments and prior knowledge, the multi-step signal prediction module is more advantageous when the vehicle is turning, while the trajectory prediction module is more advantageous when going straight. Based on this, in an embodiment of the present invention, it is judged whether the vehicle is turning according to the current vehicle steering wheel angle. If the current vehicle is in a turning state, the fusion ratio of the first control signal is made higher than that of the second control signal. For example, the fusion ratio of the first control signal is 70%, while the fusion ratio of the second control signal is 30%; and if the current vehicle is in a straight state, the fusion ratio of the first control signal is made lower than that of the second control signal.

[0071] The control system and method combine trajectory prediction and direct control prediction in a unified framework, and propose a trajectory-guided multi-step control prediction scheme, which alleviates the problem of only focusing on the current moment in the direct control prediction model, and combines the results of the two modules to achieve the effect of complementing each other's advantages. It has been verified that in a simulated driving environment with only a single monocular camera as the input, the control system and method can still achieve the best driving effect, greatly reducing collisions and violations. The effectiveness of the control system and method has been verified through a large number of test experiments in the autonomous driving simulator Carla, and the first driving score has been obtained on the official autonomous driving leaderboard of Carla.

[0072] Although the embodiments of the present invention have been described above, it should be understood that they are presented only as examples and not as limitations. It will be obvious to those skilled in the relevant art that various combinations, modifications and changes can be made without departing from the spirit and scope of the present invention. Therefore, the width and scope of the present invention disclosed herein should not be limited by the above-disclosed exemplary embodiments, but should be defined only by the appended claims and their equivalents.

Claims

1. A control system for autonomous driving, characterized in that, Comprising: A multi-step control signal prediction module configured to predict a first control signal for each time step within a specified future time period, the multi-step control signal prediction module comprising: A timing module configured to calculate a first latent variable, wherein the first latent variable includes environmental and vehicle self-information for the next time step; and An attention module configured to predict a first control signal based on the environmental and vehicle self-information, including aggregating a second latent variable corresponding to the time step in the trajectory prediction module with the first latent variable, predicting an attention matrix through a multi-layer perceptron, re-aggregating a 2D feature map based on the attention matrix, and aggregating with the first latent variable to obtain a representation vector, wherein the representation vector corresponding to each time step obtains a corresponding first control signal through a shared multi-layer perceptron; A trajectory prediction module configured to predict a driving trajectory within a specified future time period and obtain a corresponding second control signal according to the driving trajectory; and A fusion module configured to fuse the first control signal and the second control signal to obtain a third control signal.

2. The control system according to claim 1, wherein The multi-step control signal prediction module and / or the trajectory prediction module are implemented based on a gated recurrent neural network.

3. The control system according to claim 1, characterized in that, The first control signal, the second control signal, and the third control signal include at least one of the following signals: a steering wheel control signal, an accelerator control signal, and a brake control signal.

4. The control system according to claim 1, wherein, It further includes an information encoding module configured to encode a system input signal into a feature vector and send it to the multi-step control signal prediction module and the trajectory prediction module, wherein the system input signal includes an RGB image and current vehicle speed and navigation information.

5. A control method for autonomous driving, characterized in that, Including steps: Through the trajectory prediction module, predicting a driving trajectory within a specified future time period based on a 2D feature map and a vehicle state feature vector, and forming a second control signal according to the driving trajectory; Through the multi-step control signal prediction module, predicting a first control signal for each time step within a specified future time period based on a 2D feature map and a vehicle state feature vector, including: Through the timing module, obtaining a first latent variable based on the 2D feature map and the vehicle state feature vector, the first latent variable including environmental and vehicle self-information for the next time step; Inputting the first latent variable and the second latent variable corresponding to the time step in the trajectory prediction module into a multi-layer perceptron to obtain an attention matrix, the second latent variable including environmental and vehicle self-information for the next time step; Re-aggregating the 2D feature map according to the attention matrix and aggregating with the first latent variable to obtain a representation vector; and Based on the representation vector, forming a first control signal for the next time step through a multi-layer perceptron; Fusing the first and second control signals through the fusion module to obtain a third control signal; and Controlling vehicle autonomous driving according to the third control signal.

6. The control method according to claim 5, wherein The formation of the 2D feature map includes: Converting an input RGB image into a 2D feature map through a convolutional neural network.

7. The control method according to claim 5, characterized in that, The formation of the vehicle state feature vector includes: Concatenate the current vehicle speed and navigation information of the input, and convert the current vehicle speed and navigation information into a vehicle state feature vector through a multi-layer perceptron.

8. The control method according to claim 5, characterized in that, The prediction of the driving trajectory includes: Perform global average pooling on the 2D feature map and concatenate it with the vehicle state feature vector to obtain concatenated information; Input the concatenated information into a multi-layer perceptron; Send the output of the multi-layer perceptron to a gated recurrent neural network to predict the coordinates at each time step within the specified future time period in an autoregressive manner; and Merge the coordinates at each time step into a driving trajectory.

9. The control method according to claim 5, characterized in that, The formation of the second control signal according to the driving trajectory includes: Send the driving trajectory into a lateral and longitudinal PID controller to obtain a second control signal.

10. The control method according to claim 5, characterized in that, The formation of the third control signal includes: Determine the respective fusion ratios of the first control signal and the second control signal according to the current environmental information and the vehicle's own state, and fuse the first control signal and the second control signal according to the fusion ratios to obtain a third control signal, where the sum of the fusion ratios of the first and second control signals is equal to 100%.

11. The control method according to claim 10, wherein Determining the fusion ratio according to the current environmental information and the vehicle's own state includes: If the current vehicle is in a turning state, the fusion ratio of the first control signal is higher than that of the second control signal; and If the current vehicle is in a straight state, the fusion ratio of the first control signal is lower than that of the second control signal.

Images