System for obtaining predictions of vehicle actions and corresponding method - Patents.com

Abstract

Vehicle (V) Actions (a t ) a system (10;20) for obtaining a prediction of A sequence of images (F) of a dynamic scene observed by the vehicle (V), in particular in front of the vehicle (V). t ) and a camera to capture The sequence of images (F) of the dynamic scene observed by the vehicle (V). t ) for each acquired image (F t ), the corresponding visual feature vector (v t a convolutional neural network visual encoder (50) configured to obtain The position (s) of the vehicle (V) at the same time step (t) t one or more sensors (40) configured to acquire At the same time step (t), the visual feature vector (v t ) and the position of the vehicle (s t ) and predicts the action of the vehicle (V) t a recurrent neural network, in particular an LSTM network (65; 70), configured to generate a The system (20) is configured to receive as input a set of control commands (C) representing the operation of the vehicle (V); The recurrent neural network (70) has a plurality of recurrent neural network branches (701, 702, 703, 704), each branch being a control command (c i ) and The system (20) receives control commands (c i a command condition switch (60) configured to select the corresponding branch (701, 702, 703, 704) of the recurrent neural network (70) upon receiving a command condition switch (60); Thus, the system (20) operates on the selected corresponding branch (701, 702, 703, 704) to obtain the visual feature vector (v t ) and the position of the vehicle (s t ) to obtain the prediction (a) of the action of the vehicle (V). t ) is configured to retrieve the system.

Description

[Technical field]

[0001] The present description relates to a technique for obtaining predictions of actions of a vehicle, in particular a road vehicle, the invention comprising: a camera for acquiring a sequence of images of the scene dynamics observed by the vehicle, in particular in front of the vehicle; a convolutional neural network visual encoder configured to obtain, for each captured image at each time step in the sequence of images of the scene dynamics observed by the vehicle, a corresponding visual feature vector; At the same time step, the vehicle position (s t one or more sensors configured to acquire a The method includes a recurrent neural network, in particular an LSTM network, configured to receive the visual feature vector and the vehicle's position at the time step and generate a prediction of the vehicle's action. [Background technology]

[0002] Many deep learning based autonomous driving methods can be categorized into two main paradigms: mediated perception approaches and behavior-reflective methods (or end-to-end). The former consists of different separate perception components such as pedestrian detectors, lane segmentation, traffic light / sign detectors, etc. The corresponding detection results are combined into one intermediate global scene representation, which is the knowledge input to a decision-making system (usually rule-based) for planning the vehicle's next actions.

[0003] On the other hand, the behavioral reflex approach is an emerging paradigm that consists of training deep networks to directly map raw sensor data to vehicle action decisions. Networks that take raw sensor data (e.g., images) as input and output vehicle actions are also sometimes represented as end-to-end trainable. Current behavioral reflex approaches use convolutional neural networks (CNNs) to extract visual information from frames captured by vehicle-mounted cameras, e.g., simple CNNs trained for regression tasks, and output neurons predict steering angles. One challenge with this "CNN-only" architecture is that every decision depends only on the current frame. Since past frames are never represented, there is no "memory" of the dynamics of the observed scene.

[0004] For example, Huazhe Xu, Yang Gao, Fisher Yu, and Trevor Darrell, in their paper "End-to-end learning of driving models from large-scale video datasets," pp. 3530-3538 of CVPR 2017, describe the introduction of dynamic representations into the network using a Long Short Term Memory (LSTM) network.

[0005] Specifically, as shown in FIG. 1, system 10, which is in fact essentially a neural network, consists of two main sub-networks: a dilate fully convolutional neural network visual encoder, or FCN 50 for short, which represents the visual information (static) extracted from each frame, independent of other frames. FCN 50 is a neural network that encodes the visual information (static) extracted from each frame at time step t. t Given an input, a feature vector v t Using frame F t Specifically, the feature vector vt corresponds to the vector of neuron activations in the last layer of the FCN50. Then, the feature vector v t is the current vehicle position s t (represented using a two-dimensional vector) and fed into an LSTM network 65. This second sub-network selects the most likely action a given its previous hidden state value. t Predict.

[0006] A continuous prediction can be formed using a regression task and a mean squared error loss. However, it has been widely observed in various studies that this loss performs poorly, for example, when the target distribution is multimodal. Therefore, in such solutions, the regression problem is cast into a classification task with discrete bins representing the range of target values. More specifically, the steering angle α t The range of possible values ​​of ([-90,90] degrees) is discretized into N = 181 bins. Similarly, the vehicle speed m t The range of possible values ​​of p is discretized into N=181 bins. As a result, the number of output neurons of the network is 2N=362, where each neuron corresponds to a "bin class", and the fitted loss function (see Equations 1 and 2 below) is the standard cross entropy between the predicted and true class values. In Equation 1, the steering angle loss H(p α ,q α ) when expressing q α (x) is the network prediction, and p α (x) is the training answer, and similarly, Equation 2 expresses the velocity loss H(p m ,q m )

number

number

[0007] The final loss is an equally weighted sum of the two cross-entropy losses.

[0008] In addition to poor loss, such solutions lack high level control of network operation since they are based solely on sensors. Summary of the Invention

[0009] An aim of one or more embodiments is to overcome the limitations inherent in solutions achievable in the prior art.

[0010] According to one or more embodiments, this object is achieved by a method having the features defined in claim 1. One or more of the embodiments may represent a corresponding system.

[0011] The claims form an integral part of the technical teachings described herein with respect to the various embodiments.

[0012] According to the solution described herein, the system is configured to receive as input a set of control commands representative of a vehicle steering, the recurrent neural network has a plurality of recurrent neural network branches, each branch corresponding to a control command in the set of control commands; the system comprising a command condition switch configured to, upon receiving a control command, select a corresponding branch of the recurrent neural network; Thus, the system is configured to operate on the selected corresponding branch to process the visual feature vector and the vehicle position at the time step to obtain the prediction of the vehicle's action.

[0013] The solutions described herein are also directed to corresponding methods for predicting vehicle actions. [Brief description of the drawings]

[0014] In the following, embodiments will be described purely by way of non-limiting example with reference to the accompanying drawings, in which: [Figure 1] This has already been explained above. [Diagram 2] 1 illustrates the context of application of the solutions described herein. [Diagram 3] 1 depicts a block schematic diagram of the system described herein. [Figure 4] 4 shows the system of FIG. 3 in more detail. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] The following description sets forth various specific details for the purpose of a thorough understanding of the embodiments. The embodiments may be implemented without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring various aspects of the embodiments.

[0016] In the framework of this specification, a reference to "an embodiment" or "one embodiment" is meant to indicate that a particular configuration, structure, or feature described with respect to an embodiment is included in at least one embodiment. Similarly, phrases such as "in an embodiment" or "in one embodiment" that may appear anywhere in this specification do not necessarily refer to one and the same embodiment. Furthermore, particular arrangements, structures, or features may be combined as appropriate in one or more embodiments.

[0017] Reference signs used in this specification are for convenience only and do not define the scope of protection or the scope of the embodiments.

[0018] Herein, a system and method for obtaining predictions of vehicle actions based on a deep learning-based method of autonomous driving based on a network trained "end-to-end" is briefly described. The solution described here performs joint modeling of scene dynamics observed by the vehicle during its movement and a command conditioning strategy that considers high-level commands, e.g., representing the occupant's goals (i.e., desired destination). The scene dynamics is modeled using a recurrent neural network (specifically, using LSTM). However, the solution described here considers the system and method as not having a resulting network that is a function of sensor data only, but also modeling the scene dynamics while considering the occupant's goals. The solution described here provides high-level control of the network operation and utilizes "commands" provided externally as additional inputs to the network to condition the network operation.

[0019] As an example, referring to FIG. 2, a road map including streets is shown diagrammatically, and a passenger, e.g. a vehicle V, specifically a road or land vehicle, wants to go from point A to point B. The navigator generates the following sequence of commands corresponding to the path P: turn right at the next intersection (c4, explained in more detail below), turn left at the next intersection (c2), and follow the road (c1). Here, the network function is described by a function of sensors and commands, which may be controlled externally. The command, e.g. left, requests the network to plan a short-term policy, i.e. a vehicle maneuver corresponding to a sequence of actions that allows the vehicle to drive to the next intersection and then turn left.

[0020] The solution described herein allows the action a of a vehicle V to t A system 20 for obtaining a prediction of is shown in FIG.

[0021] A dilate fully convolutional neural network visual encoder is denoted at 50, which receives an image frame F representing the observed dynamics of a scene captured by a front camera (not shown) mounted on a vehicle V at time step t. t Receive such an image frame F t Based on the result, a corresponding visual representation v, such as foreground and background pixels, is obtained. t As mentioned above, the dilate FCN50 is a CNN derived from, for example, the well-known AlexNet, with the last dense layer replaced by a 1×1 convolutional filter, for example as described by Xu et al. It is pre-trained on ImageNet and then fine-tuned on the rest of the network. The dilate FCN50 extracts a vector of each frame F independently of other frames. t Visual information extracted from v t , i.e., static visual information. More specifically, at time step t, an input frame F t Given a feature vector or visual vector v obtained by the neuron activations in its last layer, the dilate FCN50 t Using such a frame F t Represents.

[0022] Dilate FCN50 is a method to estimate the CNN feature vector v t to the command condition switch block 60, which also outputs the current position s t The current vehicle position is also received. t =(x t ,y t ), and x t , y tare the coordinates of the vehicle V at time step t and are obtained using ego-motion sensors represented by the ego-motion sensor block 40. Ego-motion sensors are sensors (e.g. IMU sensors) that measure the vehicle motion relative to an arbitrary fixed reference frame. The trajectories output by these sensors are synchronized with the camera frame to obtain the position at each time step t.

[0023] Next, in command block 60, we use the CNN feature v t is the current position of the vehicle t and a two-dimensional vector (x t ,y t ) to express the joint expression (s t ,v t ) The LSTM network 70 includes multiple LSTM branches 701, 702, 703, 704, four in this example. The command block 60 is represented by the control command c t Upon receiving the control command c from multiple LSTM branches 701, 702, 703, and 704, t The method is configured to switch to one LSTM branch corresponding to

[0024] Specifically, the control command c t acts as a switch between branches 701, 702, 703, and 704. During the "forward pass", the input command c t Depending on the,representation, only one of these branches is activated.,As a result, the joint representation (s t ,v t When processing the joint expression (s), only the sub-policy among the branches 701, 702, 703, 704 that corresponds to the current value is involved. t ,v t ) is input to each LSTM branch, but only the selected branch processes such an input.

[0025] Control command c t is the command iThe control commands c are derived from a predefined set C of i, where i is an index from 1 to |C|, e.g., C={c1, c2, c3, c4}, where c1, c2, c3, c4 are denoted as continue, left, straight, and right, respectively, in the examples herein. t may be issued, for example, by the navigator of the vehicle V.

[0026] The system 20 is a map function f(F t ,s t ,c t ) →a t Output, i.e., learn, where a t is the predicted vehicle action at time step t, i.e., the captured image F t , which is specifically represented by the corresponding CNN feature vt, and the current position s t and command c t 4 is a map of predicted vehicle actions as a function of .

[0027] Since continuous output is used, the predicted vehicle action a t is a pair of steering angle and velocity magnitude, a t =(α t ,m t ), where α t is the steering angle in radians, and m t is the vehicle speed value.

[0028] It should be emphasized that while control commands are used as inputs to the system 20 or network to select short-term policies, actions are the instantaneous outputs of the system 10, i.e., they constitute such short-term policies.

[0029] Each LSTM branch 70 i is the previous hidden state value h t-1 Taking into account the most likely action a t It is important to note that Figure 3 shows the prediction of a single hidden state, h t-1However, in reality, each LSTM in each branch has its own hidden state h t-1 Calculate.

[0030] In the drawing, a portion of the system 20 is shown in greater detail.

[0031] For each input frame F t is sized to a resolution of 360 × 640 pixels and is represented by three RGB channels. FCN50 includes a first convolutional layer CV1 with 96 11 × 11 filters, then a MaxPool layer MP with 3 × 3 filters, a second convolutional layer CV2 with 5 × 5 filters at stride 1 27 × 27 × 256, a third convolutional layer with 384 3 × 3 filters at stride 1 and padding 1, a fourth convolutional layer CV4 with 384 3 × 3 filters at stride 1 and padding 1, a fifth convolutional layer CV5 with 256 3 × 3 filters at stride 1 and padding 1 13 × 13 × 256, then a sixth convolutional layer CV6, and a seventh convolutional layer CV7 with 4096 1 × 1 filters. Each LSTM branch 70 i has two stacked LSTM layers containing 64 neurons each. The final output layer 71 in each branch consists of 362 neurons, specifically, a FC Softmax layer 64 × 362. After the sixth convolutional layer CV6 and the seventh convolutional layer CV7, a dropout layer with a dropout factor equal to 0.5 may be applied for regularization.

[0032] So, in summary, the prediction of the action of the vehicle V just described, i.e. a road vehicle or land vehicle, t The system 20 for acquiring a sequence of images F of the scene dynamics observed by a vehicle V, in particular in front of the vehicle. t a camera for acquiring said sequence of images F of the scene dynamics observed by the vehicle V, i.e. a camera for capturing images in front of the vehicle V, and a convolutional neural network visual encoder 50, preferably a dirate FCN, for acquiring said sequence of images F of the scene dynamics observed by the vehicle V. tEach captured image F at different times t in t For each time step t, the corresponding visual feature vector v t and one or more sensors 40, e.g., an ego-motion sensor, configured to obtain the vehicle position s at the same time step by utilizing image frames obtained by a camera. t and a recurrent neural network, specifically an LSTM network 70, configured to acquire the visual feature vector v at the same time step t. t and vehicle position t Receive the previous hidden state h t-1 Taking into account the above, the prediction of the vehicle's (V) action a t and a recurrent neural network 70 configured to generate a control command c in the set of control commands C, the control command c corresponding to a sequence of actions, as input, representing the steering of a vehicle V, the recurrent neural network 70 having a number of recurrent neural network branches 701, 702, 703, 704, each branch corresponding to a control command c in the set of control commands C. i In response to the control command c i The system 20 comprises a command condition switch 60 configured to, upon receiving a command, select a corresponding branch 701, 702, 703, 704 of the recurrent neural network 70, and thus operates the corresponding branch 701, 702, 703, 704 selected by the switch 60 to generate an input, i.e., the visual feature vector v at the time step t. t and vehicle position t The above prediction of the action of vehicle V is made by processing t Specifically, the image F t , vehicle position st , and control command c t As mentioned above, preferably, the system 20 is configured to obtain the visual feature vector v at the time step t as a map of t and vehicle position t , respectively, to the corresponding branches 701, 702, 703, 704, and only the selected branch processes the input, which is the input visual feature vector v t and vehicle position t This represents a simpler implementation than applying only to the selected branch.

[0033] The illustrated system 20 is preferably included in an autonomous driving system, for example to provide predictions of actions to be performed by a vehicle following a particular route P.

[0034] To better understand the performance of the system 20 of FIGS. 3 and 4, a brief description of the loss function associated with such a system 20 will be provided.

[0035] In the following formula 3, the loss function (F t ,s t ,c t ) is shown.

number

[0036] The first term is the sum of the cross-entropy loss of the steering angle α, and the second term is the sum of the cross-entropy loss of the velocity m over the command nimne |C|. As above, the prediction q α、 q m and p α、 p m The function of is the training ground truth. The number of branches corresponds to the number of commands (|C|), i.e., the number of loss elements in each term. i is the command i Therefore, the training command ct One backpropagation pass for the sample associated with branch 70 i should only contribute to backpropagating errors in c t =c i In Equation 3, this is the indicator function 1(c i ,c t ), and indicator function 1 is c t =c i is equal to 1 if and only if. For efficiency reasons, the control command c i is encoded as a one-hot vector.

[0037] Thus, the advantages of the methods and systems disclosed herein are clear.

[0038] The above method and system improves LSTM operation using a command condition network to enable prediction of actions.

[0039] Known solutions using FCN and LMTS are recursive systems, where the LMTS provides steering angles or other parameters as a response to the output of the FCN encoder. The above methods and systems using command condition networks, where each LMTS is trained for a specific vehicle maneuver, are operable given the passenger's final destination (expressed as an input as a series of commands to the network, the latter being provided by the vehicle's navigator).

[0040] The solution described herein also applies dynamic inputs originating from a sequence of images, instead of static inputs, to a command-condition network of LSTM branches.

[0041] Of course, without losing sight of the principles of the embodiments, the details of construction and the details of the embodiments may vary widely from those described and illustrated herein, solely by way of example, without thereby departing from the scope of the present embodiments as defined in the following claims.

[0042] Of course, the system, i.e., the neural network of the system described herein, may be implemented by one or more processors or microprocessors, or any processing system, particularly any processing system located in a vehicle capable of supporting such a neural network.

Claims

1. 1. A system for obtaining a prediction of a vehicle's action, comprising: a camera for acquiring a sequence of images of the scene dynamics in front of the vehicle as observed by the vehicle; a convolutional neural network visual encoder configured to obtain, for each captured image at each time step in the sequence of images of the scene dynamics observed by the vehicle, a corresponding visual feature vector; one or more sensors configured to acquire a position of the vehicle at the same time step; a recurrent neural network configured to receive the visual feature vector and the vehicle position concatenated into a joint representation at the same time step and generate the prediction of the action of the vehicle taking into account a previous hidden state; the system is configured to receive as input a set of control commands representative of a steering of the vehicle; the recurrent neural network has a plurality of recurrent neural network branches, each branch corresponding to a control command in the set of control commands; the system comprising a command condition switch configured to, upon receiving a control command, select the corresponding branch of the recurrent neural network; Thus, the system is configured to operate on the selected corresponding branch and process the concatenated visual feature vector and the position of the vehicle at the time step to obtain the prediction of the action of the vehicle.

2. The system of claim 1 , wherein the recurrent neural network comprises a LSTM network.

3. 2. The system of claim 1 , wherein the convolutional neural network visual encoder is a dilated fully convolutional neural network visual encoder.

4. 2. The system of claim 1, wherein the system is configured to operate the corresponding branch to obtain the prediction of the action of the vehicle as a map function of the captured images, the position of the vehicle and a control command at the same particular time step.

5. The system of claim 1 , wherein the maneuver is included in a navigational path for the vehicle that is provided by a navigation system.

6. The system of claim 1 , wherein the actions include steering angle and vehicle speed.

7. The system according to claim 1 , wherein the system is included in a system for autonomous driving of the vehicle.

8. 1. A method for predicting an action of a vehicle by a processor or microprocessor, comprising: acquiring, by said processor or microprocessor, a series of images of scene dynamics in front of said vehicle as observed by said vehicle; applying, by the processor or microprocessor, a convolutional neural network to the corresponding captured image to obtain a visual feature vector at each time step; obtaining, by said processor or said microprocessor, a position of said vehicle at the same time step; and feeding, by the processor or microprocessor, the visual feature vector and the vehicle position concatenated into a joint representation at the same time step to a recurrent neural network, the method further comprising: determining, by the processor or microprocessor, a set of control commands representative of a steering of the vehicle; providing, by the processor or microprocessor, a plurality of branches of the recurrent neural network, each branch corresponding to a control command in the set of control commands; selecting, by the processor or the microprocessor, the corresponding branch of the recurrent neural network when a control command is issued, and supplying the concatenated visual feature vector and the position of the vehicle at the time step to the corresponding branch; and manipulating, by the processor or microprocessor, the corresponding branch to obtain the prediction of the action of the vehicle.

9. 9. The method of claim 8, wherein the step of operating the corresponding branch by the processor or microprocessor to obtain the prediction of the action of the vehicle includes the step of operating the corresponding branch by the processor or microprocessor to obtain the prediction of the action of the vehicle as a map function of the captured images, the position of the vehicle and control commands at the same particular time step.

10. The method of claim 8 , wherein the maneuver is included in a navigational path for the vehicle that is provided by a navigation system.

11. The method of claim 8 , wherein the actions include steering angle and vehicle speed.

Images