System and corresponding method for obtaining vehicle action prediction

By using multiple recursive neural network branches and command adjustment switches in the autonomous driving system, combined with external command sequences, the problem of lack of memory and advanced control in the CNN architecture in the prior art is solved, and more accurate vehicle action prediction and advanced control are achieved. .

CN113382908BActive Publication Date: 2024-11-26MARELLI EURO SPA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202080011839.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-02-27
Filing Date
2020-02-20
Publication Date
2024-11-26
Estimated Expiration
2040-02-20

AI Technical Summary

Technical Problem

Existing deep learning-based autonomous driving methods, especially the 'CNN-only' architecture in the behavioral reflection paradigm, lacks 'memory' of the scene dynamics observed by vehicles, resulting in decision-making only relying on the current frame and poor loss performance. Especially when multimodal target distribution, advanced control is lacking.

Method used

Multiple recursive neural network branches are adopted, each branch corresponds to a control instruction. The corresponding branch is selected through the instruction adjustment switch to process visual features and vehicle position, generate vehicle action predictions, and use externally provided command sequences for advanced control.

Benefits of technology

By considering the instruction sequence of passenger targets, more accurate prediction and advanced control of vehicle actions are achieved, and the intelligence and prediction accuracy of network behavior are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113382908B_ABST
    Figure CN113382908B_ABST
Patent Text Reader

Abstract

An action (a) for obtaining a vehicle (V) t ), comprising: a camera for acquiring an image sequence (F) of the scene dynamics seen by the vehicle (V); t ), in particular, a sequence of images (F) of scene dynamics in front of the vehicle (V) t ); a convolutional neural network visual encoder (50) configured to obtain an image sequence (F) of the scene dynamics seen by the vehicle (V) at each time step (t). t ) in each acquired image (F t )’s corresponding visual feature vector (v t ); one or more sensors (40) configured to acquire at the same time step (s t ) of the vehicle position (s t ); a recurrent neural network (65; 70), in particular an LSTM network, configured to receive the visual feature vector (v) at the time step (t); t ) and vehicle position (s t ), and generates the action (a t ) prediction; the system (20) is configured to receive as input a control instruction set (C) representing the operation of the vehicle (V); the recurrent neural network (70) includes a plurality of recurrent neural network branches (70 1 , 70 2 , 70 3 , 70 4 ), each recursive neural network branch corresponds to a control instruction (c) in the control instruction set (C). i ); the system (20) includes a command adjustment switch (60) configured to receive a control command (c i ) when selecting the corresponding branch (70) in the recursive neural network (70) 1 , 70 2 , 70 3 , 70 4 ); the system (20) is then configured to operate the corresponding branch (70 1 , 70 2 , 70 3 , 70 4 ) to process the visual feature vector (v) at the time step (t) t ) and vehicle position (s t ) to obtain the action (a) of the vehicle (V) t ) of the said prediction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This specification relates to a technique for obtaining a vehicle motion prediction, and in particular to a technique for obtaining a road vehicle motion prediction, including:

[0002] a camera for acquiring an image sequence of the scene dynamics seen by the vehicle, in particular an image sequence of the scene dynamics in front of the vehicle,

[0003] a convolutional neural network visual encoder configured to acquire a corresponding visual feature vector for each acquired image in the sequence of images of scene dynamics seen by the vehicle at each time step,

[0004] One or more sensors configured to obtain the vehicle position (s t ),

[0005] A recurrent neural network, in particular an LSTM network, is configured to receive the visual feature vector and the vehicle position at the time step and generate a vehicle action prediction. Background Art

[0006] Most deep learning-based autonomous driving methods can be classified into two major paradigms: mediated perception methods and behavioral reflection (or, end-to-end) methods. The former consists of different, distinct recognition components, such as pedestrian detectors, road segmentation algorithms, traffic light / sign detectors, etc. The corresponding detection results are then combined into an intermediate overall scene representation, which is the knowledge input of the (usually rule-based) decision-maker system in order to plan the vehicle's next actions.

[0007] On the other hand, behavioral reflection methods are an emerging paradigm that consists of training deep networks to map raw sensor data directly to the vehicle's action decisions. Networks that take as input raw sensor data (e.g., images) and output vehicle actions are also represented as end-to-end trainable. Modern behavioral reflection methods use convolutional neural networks (CNNs) to extract visual information from frames captured by the vehicle's onboard camera, for example using a simple CNN trained for a regression task: the output neuron predicts the steering angle. One problem with this "CNN-only" architecture is that each decision depends only on the current frame. There is no "memory" about the observed scene dynamics, as previous frames are not represented at all.

[0008] For example, from the "End-to-end learning of driving models from large-scale video datasets" by Huazhe Xu, Yang Gao, Fisher Yu, and Trevor Darrell published on pages 3530-3538 in CVPR 2017, it can be seen that dynamic representation is introduced into the network using LSTM (Long Short Term Memory) network.

[0009] Specifically, Figure 1 As shown, the system 10 is indeed a neural network in nature, consisting of two main sub-networks, a dilated fully convolutional neural network visual encoder (FCN for short) 50, which represents the (static) visual information extracted from each frame independently of other frames. Given a frame F input at time step t t , FCN 50 uses the feature vector v t Indicates frame F t Specifically, the feature vector v t corresponds to the neuron activation vector of the last layer of FCN 50. Then, the feature vector v t With the current vehicle position s t (represented using a two-dimensional vector) and input to the LSTM network 65. This second sub-network predicts the most likely action a considering its previous hidden state values t .

[0010] Continuous prediction can be formulated using a regression task and a mean squared error loss. However, it has been widely observed across different works that this loss performs poorly, for example, when the target distribution is multimodal. Therefore, the regression problem is formulated in this scheme as a classification task using discrete bins representing the range of target values. In more detail, the steering angle α t The possible range of values ​​of ([-90,90] degrees) is discretized into N = 181 bins. Similarly, the vehicle speed m t The possible range of values ​​of is discretized into N = 181 bins. Therefore, the number of output neurons of the network is 2N = 362, each neuron corresponds to a "bin-class", and the loss function used (see equations 1 and 2 below) is the standard cross entropy between the predicted class value and the true class value. Equation 1 represents the steering angle loss H(p α ,q α ), q α (x) is the network prediction, and p α (x) is the training ground truth. Similarly, Equation 2 refers to the speed loss H(p m ,qm ).

[0011] H(p α ,q α )= -Σp α (F t ,s t )log q α (F t ,s t ) (1)

[0012] H(p m ,q m )= -Σp m (F t ,s t )log q m (F t ,s t ) (2)

[0013] The final loss is the equally weighted sum of the two cross entropy losses.

[0014] This approach, besides suffering from poor performance, is based solely on sensors and thus lacks advanced control over the network's behavior. Summary of the invention

[0015] One or more embodiments are directed to overcoming limitations inherent in solutions that can be implemented from the prior art.

[0016] According to one or more embodiments, this object is achieved by a method having the features stated in claim 1. One or more embodiments may relate to a corresponding system.

[0017] The claims form an integral part of the technical teaching provided herein in relation to the embodiments.

[0018] According to the approach described herein, the system is configured to receive as input a set of control instructions representing a vehicle maneuver,

[0019] The recursive neural network includes a plurality of recursive neural network branches, each of which corresponds to a control instruction in the control instruction set.

[0020] The system includes an instruction adjustment switch, wherein the instruction adjustment switch is configured to select a corresponding branch of the recursive neural network upon receiving a control instruction,

[0021] The system is then configured to operate the selected corresponding branch to process the visual feature vector and the vehicle position at the time step to obtain the prediction of the vehicle action.

[0022] The approach described herein is also directed to a corresponding method for predicting vehicle motion. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Embodiments will now be described, by way of non-limiting example only, with reference to the accompanying drawings, in which:

[0024] Figure 1 Already discussed in the previous article.

[0025] Figure 2 The application environment of the solution described in this article is shown.

[0026] Figure 3 A block schematic diagram representing the system described herein.

[0027] Figure 4 The system described herein is shown in greater detail. DETAILED DESCRIPTION

[0028] The following description shows various specific details intended to provide a deeper understanding of the embodiments. These embodiments may be implemented without one or more of the specific details, or they may be implemented using other methods, components, materials, etc. In other cases, well-known structures, materials, or operations are not shown or described in detail, and aspects of the embodiments would not be unclear.

[0029] In the framework of this specification, references to "an embodiment" or "an embodiment" are intended to indicate that the specific configuration, structure or feature described in connection with the embodiment is included in at least one embodiment. Similarly, phrases such as "in an embodiment" or "in an embodiment" that may be present in various aspects of this specification do not necessarily refer to one embodiment and the same embodiment. In addition, specific configurations, structures or features may be combined in one or more embodiments in an appropriate manner.

[0030] The references used herein are for convenience only and do not thereby limit the scope or extent of the embodiments.

[0031] In short, this paper describes a system and method for obtaining vehicle action predictions, which is a deep learning-based autonomous driving method based on an "end-to-end" trained network. The scheme described in this paper jointly models the scene dynamics observed by the vehicle while moving and a command regulation decision strategy that takes into account high-level commands, which represent, for example, the goals of the passengers (e.g., desired destinations). The scene dynamics are modeled using recurrent neural networks (particularly using LSTMs). However, the scheme described in this paper relates to a system and method that also takes into account the goals of the passengers while modeling the scene dynamics, rather than having a network that is simply a function of sensor data. The scheme described in this paper provides high-level control of the network behavior, using externally provided "commands" as additional inputs to the network to regulate the network behavior.

[0032] For example, refer to Figure 2 , which schematically shows a road map with streets, assuming that a passenger, such as a vehicle V, in particular a road vehicle or a land vehicle, wants to go from point A to point B. The navigator will generate the following sequence of instructions corresponding to the path P: turn right at the next intersection (c4 as described in more detail below), turn left at the next intersection (c2), drive along the road (c1). The network function is now described by a function of sensors and instructions, and the network function can be controlled from the outside. The instruction (e.g., left) requires the network to plan a short-term policy, i.e., a vehicle maneuver (corresponding to a sequence of actions) that enables the vehicle to drive to the next intersection and then turn left.

[0033] According to the scheme described in this article, Figure 3 The method for obtaining the action a of the vehicle V is shown. t The prediction system 20.

[0034] The dilated fully convolutional neural network visual encoder is represented by 50, which receives the image frame F t , the image frame F t represents the observed scene dynamics captured by a front camera (not shown) mounted on the vehicle V at time step t and in the image frame F t Based on the extraction of the corresponding visual representation v t , such as foreground pixels and background pixels. The dilated FCN 50 mentioned is a CNN extracted, for example, from the well-known AlexNet, as discussed in the article by Xu et al., where the last dense layer has been replaced with a 1×1 convolutional filter. It is pre-trained on ImageNet and then fine-tuned along with the rest of the network. The dilated FCN 50 is configured to represent each frame F independently of other frames. t Extracted visual information v t, i.e., static visual information. More specifically, given the input frame F at time step t t , the dilated FCN 50 uses the feature vector or visual vector v obtained by the activation of neurons in its last layer t Represents such a frame F t .

[0035] The expanded FCN 50 transforms the CNN feature vector v t The command adjustment switch block 60 also receives the current position of the vehicle s t . Current vehicle position s t =(x t ,y t ), where x t ,y t are the coordinates of the vehicle V at time step t and are obtained using an ego-motion sensor represented by the ego-motion sensor block 40. An ego-motion sensor is a sensor (e.g., an IMU sensor) used to measure the motion of the vehicle relative to an arbitrary fixed reference frame. The trajectories output by these sensors are synchronized with the camera frame in such a way that the position is obtained at each time step t.

[0036] Then, the CNN feature v in instruction block 60 t The current position of the vehicle t Concatenation, using a two-dimensional vector (x t ,y t ) is expressed as a joint representation (s t ,v t ). The LSTM network 70 includes a plurality of LSTM branches, for example, four LSTM branches 701, 702, 703, and 704. The instruction block 60 receives the control instruction c t When configured to switch to the multiple LSTM branches 701, 702, 703, 704 corresponding to the control instruction c t Corresponding to an LSTM branch.

[0037] Specifically, the control instruction c t Acts as a switch between branches 701, 702, 703, 704. During the "forward pass", according to the input instruction c t , only one of these branches is activated. Therefore, when processing the joint representation (s t ,v t ), only the sub-strategies corresponding to the current values ​​of branches 701, 702, 703, and 704 are involved. Preferably, the joint representation (s t ,v t ) is input to each LSTM branch, but only the selected branch processes the input.

[0038] Control command c t From instruction c i The predefined set C of i is indexed from 1 to |C|. For example, the predefined set C can be C={c1,c2,c3,c4}, where c1, c2, c3, c4 are respectively shown as: continue, left, straight, right in the example of this article. These control instructions c t It may originate from, for example, a navigator of vehicle V.

[0039] The system 20 outputs (ie, learns) a mapping function f(Ft,s t ,c t )→a t , where a t is the predicted vehicle action at time step t, i.e., as the acquired image Ft (specifically composed of the corresponding CNN feature v t The current position of the vehicle t and instruction c t A mapping of the predicted vehicle actions to a function.

[0040] Since continuous output is used, the predicted vehicle action a t is defined as a pair of steering angle and velocity magnitude: a t =(α t , m t ), where α t is the steering angle in radians, m t is the vehicle speed value.

[0041] It is emphasized that control instructions are used as input to the system 20 or network to select a short-term strategy, while actions are the instantaneous outputs of the system 10, i.e., constitute such a short-term strategy.

[0042] Each LSTM branch 70 i Considering its previous hidden state value h t-1 And predict the most likely action a t , action a t indicates visual dynamics. It is important to note that although Figure 3 shows a single hidden state h t-1 , in fact, each LSTM in each branch computes its own hidden state h t-1 .

[0043] In the figure, it is shown a part of the system 20 in a more detailed manner.

[0044] Each input frame F tResized to a resolution of 360×640 pixels and represented with three RGB channels. FCN 50 includes a first convolutional layer CV1 (96 11×11 filters), followed by a MaxPool layer MP with 3×3 filters, a second convolutional layer CV2 (27x27x256) with a stride of 1 and a 5x5 filter, a third convolutional layer CV4 [13x13x256] with a stride of 1 and a pad of 1 and 384 3x3 filters, a fourth convolutional layer CV4 [13x13x256] with a stride of 1 and a pad of 1 and 384 3x3 filters, a fifth convolutional layer CV5 with a stride of 1 and a pad of 1 and 256 3x3 filters, followed by a sixth convolutional layer CV6 with 40961x1 filters and a seventh convolutional layer CV7 with 4096 1x1 filters. Each LSTM branch 70 i Each includes two stacked LSTM layers, each including 64 neurons. The final output layer 71 in each branch consists of 362 neurons, in particular a FC Softmax layer 64x362. After the sixth convolutional layer CV6 and the seventh convolutional layer CV7, a dropout layer with a dropout coefficient equal to 0.5 can be used for regularization.

[0045] Therefore, in short, the action a for obtaining the vehicle (ie, road vehicle or land vehicle) V t The prediction system 20, as just described, comprises: a sequence of images F for acquiring the dynamics of the scene seen by the vehicle V t A camera, in particular a camera in front of the vehicle V, that is, it obtains images of the scene dynamics in front of the vehicle V; a convolutional neural network visual encoder 50, preferably an expanded FCN, configured to obtain the image sequence F of the scene dynamics seen by the vehicle V at each time step t t Each acquired image F at different time t t The corresponding visual feature vector v t , for example, a classification of pixels representing an image according to categories such as foreground, background, and others; one or more sensors 40, for example, an ego-motion sensor acquired by using an image frame acquired by a camera, configured to acquire the position s of the vehicle at the same time step t ; A recurrent neural network 70, in particular an LSTM network 70, configured to receive the visual feature vector v at the time step t t and the vehicle position s t And considering the previous hidden state h t-1 And the action a of the vehicle (V) is generated twherein the system 20 is configured to receive a control instruction set C as input, the control instruction set C represents the manipulation of the vehicle V, in particular corresponding to the action sequence, the recursive neural network 70 includes a plurality of recursive neural network branches 701, 702, 703, 704, each recursive neural network branch corresponds to a control instruction c in the control instruction set C i The system 20 includes a command adjustment switch 60. The command adjustment switch 60 receives a control command c i When the system 20 is configured to select the corresponding branch 701, 702, 703, 704 in the recurrent neural network 70, the system 20 is then configured to operate the selected corresponding branch 701, 702, 703, 704 selected by the switch 60 to process the input (i.e., the visual feature vector v at the time step t) t and the vehicle position s t ) and obtain the action a of vehicle V t (In particular, as the image Ft acquired at the same given time step t, the vehicle position s t and control instructions c t As indicated, preferably, the system 20 maps the visual feature vector v at the time step t to t and the vehicle position s t Provided to each corresponding branch 701, 702, 703, 704, and then only the selected branch processes the input, which shows that compared with the visual feature vector v t and the vehicle's position s t A simpler implementation would be to apply the input only to the selected branch.

[0046] Preferably, the described system 20 is comprised in an automated driving system, for example providing a prediction of actions to be performed by the vehicle to travel along a certain path P.

[0047] To better understand this article Figure 3 and Figure 4 In order to understand the performance of system 20, the loss function associated with system 20 is briefly discussed here.

[0048] In the following equation 3, it shows the loss function Loss(Ft,s t ,c t ).

[0049]

[0050] The first term is the sum of the cross entropy losses of the steering angle α, and the second term is the sum of the cross entropy losses of the speed m on the command nimne|C|. The indicated prediction q α ,q m The function of pα , p m is the training ground truth. The number of branches corresponds to the number of instructions (|C|), and thus to the number of loss components in each term. Each branch 70 i Responsible for the corresponding instruction c i Therefore, for the ground-truth instruction c t One backpropagation pass of the associated sample should help backpropagation only in branch 70 i The error in c t =c i In equation 3, this is given by the indicator function 1(c i ,c t ) means that if and only if c t =ci, the indicator function 1(c i ,c t ) is equal to 1. For efficiency reasons, the control instruction c i Encoded as a one-hot vector.

[0051] The advantages of the method and system just disclosed are therefore apparent.

[0052] The described methods and systems allow predicting actions to improve LSTM operation by conditioning the network using instructions.

[0053] A known solution using FCN and LMTS is a reflexive system, where the LMTS provides a steering angle or other parameter as a reaction to the output of the FCN encoder. The method and system described using a command regulation network, where each LMTS is trained for a specific vehicle maneuver, can operate by taking into account the passenger's final destination, which is represented as a sequence of commands as input to the network, the latter being provided by the vehicle navigator.

[0054] Furthermore, the approach described in this paper is to apply dynamic inputs from image sequences rather than static inputs to the instruction conditioning network of the LSTM branches.

[0055] Of course, without prejudice to the principle of the embodiment, the details of the structure and the embodiment may be widely varied with respect to what has been described and shown herein merely by way of example, without thereby departing from the scope of protection of the embodiments of the application as defined in the subsequent claims.

[0056] Of course, the neural network of the system, ie as described herein, may be implemented by one or more processors or microprocessors or any processing system, in particular by any processing system arranged in a vehicle capable of supporting the neural network.

Claims

1. An action (a) for acquiring a vehicle (V) t ) of the predicted system (10; 20), include: A camera is used to acquire a dynamic image sequence (F) of a scene seen by the vehicle (V) t ), A convolutional neural network visual encoder (50) configured to acquire the image sequence (F) of the scene dynamics seen by the vehicle (V) at each time step (t) t ) in each acquired image (F t )’s corresponding visual feature vector (v t ), One or more sensors (40) configured to obtain the vehicle position (s t ), A recurrent neural network (65; 70) configured to receive the visual feature vector (v) at each time step (t) t ) and the vehicle position (s t ), and considering the previous hidden state (h t-1 ) and generate the action (a) of the vehicle (V) t ) predictions, Features: The system (20) is configured to receive as input a set of control instructions (C) representing a maneuver of a vehicle (V), the maneuver of the vehicle (V) corresponding to a sequence of actions; The recursive neural network (70) comprises a plurality of recursive neural network branches (70 1 , 70 2 , 70 3 , 70 4 ), each recursive neural network branch corresponds to a control instruction (c) in the control instruction set (C). i ), The system (20) comprises a command adjustment switch (60), wherein the command adjustment switch (60) is configured to, upon receiving the control command (c i ) when selecting the corresponding branch (70) in the recursive neural network (70) 1 , 70 2 , 70 3 , 70 4 ), The system (20) is then configured to operate the selected corresponding branch (70 1 , 70 2 , 70 3 , 70 4 ) to process the visual feature vector (v) at each time step (t) t ) and the vehicle position (s t ) to obtain the action (a) of the vehicle (V) t ) of the said prediction.

2. The system according to claim 1, It is characterized in that The recurrent neural network includes an LSTM network.

3. The system according to claim 1, It is characterized in that The convolutional neural network visual encoder (50) is an expanded fully convolutional neural network visual encoder (50).

4. The system according to claim 1, It is characterized in that The system (20) is configured to operate the branch (70 1 , 70 2 , 70 3 , 70 4 ) to obtain the image (F) of the vehicle (V) as acquired at the same given time step (t) t ), vehicle position (s t ) and the mapping action (a t ) of the said prediction.

5. The system according to claim 1, It is characterized in that The maneuver is included in a navigation path of the vehicle.

6. The system according to claim 1, It is characterized in that The action includes steering angle and vehicle speed.

7. A system according to any one of the preceding claims, It is characterized in that The system is included in an autonomous driving system of the vehicle.

8. A method for predicting the motion of a vehicle (a t ) method, include: Get the image sequence (F) of the scene dynamics seen by the vehicle (V) t ), By applying a convolutional neural network visual encoder (50) to the corresponding acquired image (F t ) to obtain the visual feature vector (v) at each time step (t) t ), Get the vehicle position at the same time step (s t ), The visual feature vector (v) at each time step (t) t ) and the vehicle position (s t ) is provided to a recurrent neural network, Characterized in that the method comprises: determining a set (C) of control instructions representing a maneuver of a vehicle (V), the maneuver of the vehicle (V) corresponding to a sequence of actions; Providing a plurality of branches (70) in the recursive neural network (70) 1 , 70 2 , 70 3 , 70 4 ), each branch corresponds to a control instruction (c) in the control instruction set (C). i ), When the control command (c i ) is issued, selecting the corresponding control instruction (c i ) branch (70 1 , 70 2 , 70 3 , 70 4 ) and the visual feature vector (v t ) and the vehicle position (s t ) is provided to the branch (70 1 , 70 2 , 70 3 , 70 4 ), Operation of the branch (70 1 , 70 2 , 70 3 , 70 4 ) to perform the above mentioned action of the vehicle (a t ) of the said prediction.

9. The method according to claim 8, It is characterized in that The operation of the branch (70 1 , 70 2 , 70 3 , 70 4 ) to obtain the image (F) of the vehicle (V) as that acquired at the same given time step (t) t ), the vehicle position (s t ) and the mapping action (a t ) of the said prediction.

10. The method according to claim 8, It is characterized in that The maneuver is included in a navigation path of the vehicle.

11. The method according to claim 8, It is characterized in that The action includes steering angle and vehicle speed.