Apparatus for predicting travel route of vehicle, method for predicting travel route of vehicle, and system for predicting travel route of vehicle

Abstract

The invention provides a driving route prediction apparatus of a vehicle, a driving route prediction method thereof, and a system. Provided are a travel route prediction device, method, and system capable of predicting a travel route using a model trained on the basis of actual travel data. The driving route prediction apparatus for a vehicle includes a first feature extraction module configured to extract a first feature vector from vehicle driving information, a second feature extraction module configured to extract a second feature vector from driving image information, a fusion module configured to fuse the first feature vector and the second feature vector to generate fused data; and a driving route prediction module configured to output route prediction information of the vehicle based on the fused data.

Description

Technical Field

[0001] The present invention relates to prediction of a driving route in a vehicle, and in particular to a driving route prediction device, system and method capable of predicting a driving route using a model trained based on actual driving data. Background Art

[0002] Path prediction technology for an ego vehicle (i.e., a vehicle that includes sensors to perceive the vehicle's surroundings, typically applied to autonomous vehicles) is a technology that detects the driver's driving intention by utilizing sensor values ​​(such as the vehicle's acceleration / angular velocity), steering angle, lane recognition information from a camera module, and information from a linkage system, and is used to predict the position of the ego vehicle at a future point. Path prediction technology provides important information when determining collision risk information about a target object.

[0003] Conventional ego vehicle route prediction techniques predict future routes based on the Constant Turn Rate and Acceleration (CTRA) model and the Constant Turn Rate and Velocity (CTRV) model. As a result, conventional ego or autonomous vehicle route prediction techniques have limitations in terms of accuracy in predicting long-term future routes in the short term and in route prediction accuracy for nonlinear and complex driving.

[0004] In various real-world road driving situations, the ego vehicle's driving path depends largely on the external driving network (e.g., lane recognition information, traffic signals, pedestrian crossings, road traffic signs, unidentified structures, etc.). However, the problem with conventional ego vehicle path prediction techniques is that they cannot effectively consider such external driving networks.

[0005] In particular, even though conventional self-driving vehicle route prediction technology utilizes camera recognition technology to recognize external information. This technology also performs route prediction by determining the driver's driving intention through a rule-based arbitration model. Therefore, the conventional self-driving or autonomous vehicle route prediction technology has a problem in that the accuracy of route prediction is limited for unset conditions.

[0006] The background technology is the technical information that the inventor has in order to obtain the present invention or obtained in the process of obtaining the present invention. The background technology cannot necessarily be regarded as the known technology disclosed to the public before submitting the present invention. Summary of the invention

[0007] Therefore, the present invention is made in view of the above problems. The object of the present invention is to provide a driving route prediction device for a vehicle, which can predict the driving route using a model trained based on actual driving data. A further object of the present invention is to provide a driving route prediction method and system thereof.

[0008] Another object of the present invention is to provide a vehicle driving route prediction device that can effectively predict long-term complex driving routes. A further object of the present invention is to provide a driving route prediction method and system thereof.

[0009] Another object of the present invention is to provide a driving route prediction device for a vehicle, which is capable of predicting a driving route using a model generated by learning an actual driving route based on actual road driving data. A further object of the present invention is to provide a driving route prediction method and a system thereof.

[0010] Another object of the present invention is to provide a driving route prediction device for a vehicle, which can effectively predict the driving route by reflecting various conditions in actual road driving situations. A further object of the present invention is to provide a driving route prediction method and system thereof.

[0011] Another object of the present invention is to provide a driving route prediction device for a vehicle, which is capable of predicting a driving route using a model generated by combining and learning driving image information and actual driving motion data through an AI-based end-to-end (E2E) model.

[0012] The technical problems solved by the present invention are not limited to the above problems. According to the following description, a person skilled in the art to which the present invention belongs should be able to clearly understand other problems solved by the present invention.

[0013] As a technical means to achieve the above technical tasks, a driving route prediction device for a vehicle that can use a model trained based on actual driving data to predict the driving route, a driving route prediction method thereof, and a system including the device can be provided.

[0014] According to one aspect of the present invention, the above and other purposes can be achieved by providing a driving route prediction device for a vehicle. The driving route prediction device includes: a first feature extraction module configured to extract a first feature vector from vehicle driving information; a second feature extraction module configured to extract a second feature vector from driving image information; a fusion module configured to fuse the first feature vector and the second feature vector to generate fusion data; and a driving route prediction module configured to output vehicle route prediction information based on the fusion data.

[0015] The first feature extraction module may normalize each element included in the vehicle driving information and input each normalized element into a dense layer to extract a first feature vector of a one-dimensional (1D) array.

[0016] The first feature extraction module may perform maximum-minimum normalization on lane information, accelerator pedal control amount, brake pedal control amount, vehicle speed, and wheel speed among elements included in the vehicle driving information.

[0017] The first feature extraction module may perform Gaussian distribution normalization on a vehicle dynamics-based element among elements included in the vehicle driving information.

[0018] Vehicle dynamics-based factors may include steering angle, yaw rate, longitudinal acceleration, and lateral acceleration.

[0019] The second feature extraction module may extract a second feature vector of a two-dimensional (2D) array from the driving image information using a convolutional neural network (CNN), and may input the second feature vector of the 2D array into a dense layer to extract a second feature vector of a 1D array.

[0020] The fusion module may concatenate the first feature vector and the second feature vector to generate a concatenated feature vector, and may embed the concatenated feature vector into a preset prediction time stamp size to generate fused data.

[0021] The fusion module may include a neural network that uses the concatenated feature vector as an input and outputs an operation result based on a rectified linear unit (ReLU) function.

[0022] The driving route prediction module can input the fused data into a recurrent neural network (RNN) and can input the route prediction information output from the RNN into a dense layer to output the route prediction information in a 1D array.

[0023] The driving route prediction module can predict the driving route at each preset prediction time point. The output includes the vehicle speed v x , yaw rate Route prediction information of X-direction moving distance dx, Y-direction moving distance dy and heading moving angle dθ.

[0024] The objective function θ of Equation 3 for training that optimizes the travel route prediction result can be used * To train the driving route prediction module.

[0025] According to another aspect of the present invention, a method for predicting a driving route of a vehicle is provided. The method comprises: extracting a first feature vector from vehicle driving information; extracting a second feature vector from driving image information; fusing the first feature vector and the second feature vector to generate fused data; and outputting vehicle route prediction information based on the fused data.

[0026] Extracting the first feature vector may include normalizing each element included in the vehicle driving information, and may include inputting each normalized element into a dense layer to extract the first feature vector of the 1D array.

[0027] Extracting the first feature vector may include performing maximum-minimum normalization or Gaussian distribution normalization on the elements included in the vehicle travel information according to the type of the elements.

[0028] Extracting the second feature vector may include extracting the second feature vector of the 2D array from the driving image information using a CNN, and may include inputting the second feature vector of the 2D array into a dense layer to extract the second feature vector of the 1D array.

[0029] Generating the fused data may include concatenating the first feature vector and the second feature vector to generate a concatenated feature vector, and may include embedding the concatenated feature vector into a preset prediction timestamp size to generate the fused data.

[0030] The output of the route prediction information may include inputting the fused data into the RNN, and inputting the route prediction information output from the RNN into a dense layer to output the route prediction information of a 1D array.

[0031] Output route prediction information may include outputting the route prediction information at each preset prediction timestamp time point. The output includes the vehicle speed v x , yaw rate Route prediction information of X-direction moving distance dx, Y-direction moving distance dy and heading moving angle dθ.

[0032] The output of the route prediction information can be performed by a driving route prediction module, which uses the objective function θ of equation 3 for training to optimize the driving route prediction result. * Come to train.

[0033] According to another aspect of the present invention, a driving route prediction system for a vehicle is provided. The driving route prediction system includes: a vehicle driving information provider configured to provide vehicle driving information indicating a driving state of the vehicle; a driving image information provider configured to provide driving image information related to the front and surrounding environment of the driving vehicle; and a driving route prediction device according to an embodiment of the present invention.

[0034] In addition to means for solving the above-mentioned problems, specific details of various examples according to the present invention are included in the following description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The above and other objects, features and other advantages of the present invention will be more clearly understood through the following detailed description taken in conjunction with the accompanying drawings, in which:

[0036] Figure 1 is a diagram showing a configuration of a travel route prediction system for a vehicle according to an embodiment of the present invention;

[0037] Figure 2 is a diagram showing a configuration of a travel route prediction device according to an embodiment of the present invention;

[0038] Figure 3 is a diagram for describing a first feature extraction module of a driving route prediction device according to an embodiment of the present invention;

[0039] Figure 4 is a diagram for describing a second feature extraction module of a driving route prediction device according to an embodiment of the present invention;

[0040] Figure 5 is a diagram for describing a driving route prediction module of a driving route prediction device according to an embodiment of the present invention;

[0041] Figure 6 is a flowchart for describing a driving route prediction method according to an embodiment of the present invention; and

[0042] Fig. 7A and 7B 1 is a diagram for comparing a predicted route according to a conventional travel route prediction technique with a predicted route according to a travel route prediction method of the present invention. DETAILED DESCRIPTION

[0043] The advantages and features of the present invention and the methods for achieving these advantages and features should become apparent with reference to the embodiments described in detail below in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. These embodiments make the present invention complete and are provided to fully inform the scope of the present invention to those of ordinary skill in the art to which the present invention belongs. In addition, the present invention is defined by the scope of the claims.

[0044] The shapes, sizes, proportions, angles, quantities, etc. disclosed in the drawings used to describe the embodiments of the present invention are illustrative. Therefore, the present invention is not limited to the elements shown. Throughout the specification, the same reference numerals refer to the same elements. In addition, when describing the present invention, when it has been determined that the detailed description of the relevant known technology may unnecessarily obscure the subject matter of the present invention, its detailed description is omitted. When "including", "having", "comprising", etc. are used in this specification, other components may also be present unless "only" is used. When an element is represented by the singular, unless otherwise explicitly stated, the case of including the plural number is also included.

[0045] When interpreting an element, even if there is no separate explicit description, the element should be interpreted as including a range of error.

[0046] When a time relationship is described with "after", "subsequently", "next", "before", etc., non-sequential cases may be included unless "immediately" or "directly" is used.

[0047] Although "first", "second", etc. are used to describe various elements, these elements are not limited by these terms. These terms are only used to distinguish one element from another element. Therefore, within the spirit and scope of the present invention, the first element mentioned below may be the second element.

[0048] When describing the elements of the present invention, terms such as first, second, A, B, (a) and (b) may be used. These terms are only used to distinguish the element from other elements. The nature, order, sequence or quantity of the corresponding components are not limited by the terms. When an element is described as "coupled to", "combined in" or "connected to" another element, the element may be directly coupled or connected to another element. However, it should be understood that, unless otherwise specifically stated, other elements may be "inserted" between each element that is indirectly connected or connectable to each other.

[0049] "At least one" should be understood to include all combinations of one or more related elements. For example, "at least one of the first, second or third element" may not only refer to the first, second or third element, but also include all combinations of two or more of the first, second and third elements.

[0050] The various features of the various embodiments of this specification may be partially or completely combined or associated with each other, and various interlocks and drives are possible. In addition, the various embodiments may be implemented independently of each other, or may be implemented interdependently. In addition, when a component, device, element, etc. of the present invention is described as having a purpose or performing an operation, function, etc., the component, device or element should be considered as "configured to" meet the purpose or perform the operation or function in this article.

[0051] Hereinafter, embodiments of the present invention are examined through the following drawings and embodiments. Since the proportions of components shown in the drawings are different from the actual ones for the convenience of description, the components are not limited to the proportions shown in the drawings.

[0052] Hereinafter, a traveling route prediction apparatus for a vehicle, a traveling route prediction method thereof, and a system including the same according to an embodiment of the present invention are described with reference to the accompanying drawings.

[0053] Figure 1 is a diagram showing a configuration of a travel route prediction system 1 for a vehicle according to an embodiment of the present invention.

[0054] refer to Figure 1 , a driving route prediction system 1 (hereinafter referred to as a system) according to an embodiment of the present invention is a system that can be installed in an automatic driving vehicle. The system 1 may include a vehicle driving information provider 100, a driving image information provider 200, and a driving route prediction device 300. The configuration of the system 1 is not limited thereto.

[0055] The vehicle driving information provider 100 according to the embodiment may provide various types of information indicating a driving state of a vehicle (hereinafter referred to as vehicle driving information).

[0056] For example, the vehicle driving information may include: acceleration (X direction and Y direction); angular velocity (roll angular rate, pitch angular rate and yaw angular rate); steering angle; accelerator pedal control amount; brake pedal control amount; driving speed / wheel speed; lane information; linkage system operation information (e.g., rear wheel steering information), etc. However, the types of vehicle driving information are not limited thereto.

[0057] In order to provide vehicle driving information, the vehicle driving information provider 100 may include: an acceleration sensor capable of measuring lateral and / or longitudinal acceleration; an angular velocity sensor capable of measuring angular velocity of a roll angular rate, a pitch angular rate, and / or a yaw angular rate; a steering angle sensor; an accelerator pedal sensor (APS); a brake pedal sensor (BPS); a vehicle speed sensor; a lane recognition sensor, etc. However, the configuration of the vehicle driving information provider 100 is not limited thereto.

[0058] The driving image information provider 200 according to the embodiment can provide image information (hereinafter referred to as driving image information) related to the front and surrounding environment (driving environment) of the driving vehicle. For example, the driving image information may include surrounding objects of the driving vehicle, such as surrounding vehicles, pedestrians, and road facilities (e.g., traffic lights, road signs, traffic signs, etc.).

[0059] To provide the traveling image information, the traveling image information provider 200 may include a light detection and ranging (LiDAR) sensor, a camera, a radio detection and ranging (RaDAR) sensor, an ultrasonic sensor, etc. However, the configuration of the traveling image information provider 200 is not limited thereto.

[0060] The driving route prediction device 300 according to the embodiment can predict the driving route of the vehicle based on the vehicle driving information from the vehicle driving information provider 100 and the driving image information from the driving image information provider 200. The device 300 can also predict output route prediction information (or driving route prediction information).

[0061] According to an embodiment, the driving route prediction apparatus 300 may extract feature vectors from the vehicle driving information and the driving image information, respectively, and may generate fused data by fusing the extracted feature vectors.

[0062] In addition, the driving route prediction device 300 according to the embodiment includes a neural network model using the fusion data as input. Accordingly, the output of the neural network model can be output as route prediction information.

[0063] According to an embodiment, the driving route prediction device 300 may concatenate feature vectors and generate fused data through an embedding process with a preset prediction timestamp size (eg, 0 to 4 seconds, intervals of 0.1 seconds).

[0064] According to an embodiment, the driving route prediction device 300 may output route prediction information at each preset prediction timestamp time point. For example, the route prediction information may include the vehicle speed v x , yaw rate The X-direction moving distance dx, the Y-direction moving distance dy and the heading moving angle dθ.

[0065] According to an embodiment, the driving route prediction device 300 may output route prediction information in an array form, for example

[0066] In this way, the driving route prediction device 300 according to the embodiment can use the actual road driving data as input to perform training. The device 300 can also use a neural network model to predict the driving route of the vehicle, which is an end-to-end (E2E) model that directly processes input data as output data.

[0067] Therefore, compared with the conventional constant turning rate and acceleration (CTRA) model or constant turning rate and speed (CTRV) model for predicting a short-term driving route, the driving route prediction device 300 according to an embodiment of the present invention can effectively predict a long-term complex driving route.

[0068] For example, short term may refer to a time period of 1 second or less, and long term may refer to a time period of more than 1 second. In addition, a complex driving route may refer to a driving route in which acceleration / deceleration and yaw rate are irregular or nonlinear.

[0069] Furthermore, the neural network model of the driving route prediction apparatus 300 according to the embodiment of the present invention can predict the driving route while being trained using driving image information as input in addition to vehicle driving information.

[0070] The driving image information according to the embodiment may include driving environments that are not considered in advance. Accordingly, the driving route prediction device 300 may predict the driving route by learning various driving environments that are not considered in advance.

[0071] Therefore, in addition to predicting a route for a previously considered driving environment, the driving route prediction device 300 according to the embodiment of the present invention can also effectively predict a driving route for various driving environments that have not been considered in advance.

[0072] Figure 2 is a diagram showing a configuration of a travel route prediction device 300 according to an embodiment of the present invention.

[0073] refer to Figure 1 and Figure 2 , the driving route prediction device 300 according to the embodiment of the present invention may include a first feature extraction module 310, a second feature extraction module 320, a fusion module 330 and a driving route prediction module 340. The configuration of the driving route prediction device 300 is not limited thereto.

[0074] According to the embodiment, the first feature extraction module 310 (or the first encoder or the driving information encoder) can extract a feature vector (hereinafter referred to as the first feature vector) from the vehicle driving information. The module 310 can also output the extracted first feature vector to the fusion module 330.

[0075] Figure 3 is a diagram for describing a first feature extraction module 310 of a driving route prediction apparatus 300 according to an embodiment of the present invention.

[0076] like Figure 3 As shown, the first feature extraction module 310 is a dense layer-based encoder and may include a normalizer 311 and a dense layer 312 .

[0077] The first feature extraction module 310 may normalize each element included in the vehicle driving information using a normalizer 311 , and may input each normalized element to a dense layer 312 to extract a one-dimensional (1D) feature vector.

[0078] For example, the elements included in the vehicle driving information may include longitudinal acceleration, lateral acceleration, angular velocity (roll angular rate, pitch angular rate, and yaw angular rate), steering angle, accelerator pedal control amount, brake pedal control amount, vehicle speed, wheel speed, lane information, linkage system operation information (e.g., rear wheel steering information), etc. The elements included in the vehicle driving information are not limited thereto.

[0079] According to an embodiment, the first feature extraction module 310 (or the normalizer 311) may perform maximum-minimum normalization using maximum and minimum values ​​as parameters of lane information, accelerator pedal control amount, brake pedal control amount, vehicle speed, and wheel speed in the elements.

[0080] In addition, the first feature extraction module 310 (or the normalizer 311) can perform Gaussian distribution normalization using the mean and variance as parameters of the elements based on the vehicle dynamics model in the elements. The elements based on the vehicle dynamics model can be steering angle, yaw rate, longitudinal acceleration or lateral acceleration.

[0081] Accordingly, the first feature extraction module 310 (or the normalizer 311 ) may perform Gaussian distribution normalization on the steering angle, the yaw rate, the longitudinal acceleration, or the lateral acceleration.

[0082] The dense layer 312 is a neural network that is configured to output a result through an operation (Sigmoid function) as shown in the following equation 1 when an input (x) is given, and is trained to update the weight (w) parameter in a direction in which the error between the given data and the inferred value becomes smaller.

[0083] [Equation 1]

[0084] dense output = sigmoid(∑(w j x i +b j ))

[0085] Here, dense output represents dense output, w represents dense layer weight, b represents dense layer bias, and i>j.

[0086] In addition, the dense layer 312 is a layer configured to perform the role of compressing the dimension of a given input into an output vector and extracting core information. A detailed description of the general structure and operation related to the dense layer 312 is omitted.

[0087] According to an embodiment, the dense layer 312 may use a maximum-minimum normalization factor and a Gaussian distribution normalization factor as learning parameters.

[0088] According to the embodiment, the second feature extraction module 320 (or the second encoder or the image information encoder) can extract a feature vector (hereinafter referred to as a second feature vector) from the driving image information, and output the extracted second feature vector to the fusion module 330 .

[0089] Figure 4 is a diagram for describing the second feature extraction module 320 of the driving route prediction apparatus 300 according to an embodiment of the present invention.

[0090] like Figure 4 As shown, the second feature extraction module 320 is a decoder based on a convolutional neural network (CNN) 321 and a dense layer 322 , and may include the CNN 321 and the dense layer 322 .

[0091] The second feature extraction module 320 may extract a second feature vector of a two-dimensional (2D) array from the driving image information using a CNN 321. The module 320 may also input the second feature vector of the 2D array to a dense layer 322 to extract a 1D second feature vector.

[0092] For example, according to an embodiment, CNN 321 performs an encoder function. CNN 321 may be an encoder component extracted from a CNN-based encoder-decoder structure in which an input image is restored without change by attaching symmetric CNNs on both sides.

[0093] According to an embodiment, CNN 321 is a neural network for extracting key features from driving image information. A detailed description of the general structure and operation of CNN 321 is omitted.

[0094] Therefore, the second feature extraction module 320 is an encoder based on a neural network, which is trained based on driving image information input in real time while the vehicle is driving. Therefore, the second feature extraction module 320 can learn various driving environments that have not been considered in advance.

[0095] The fusion module 330 according to the embodiment may fuse the first feature vector from the first feature extraction module 310 and the second feature vector from the second feature extraction module 320 to generate fusion data. The fusion module 330 may output the generated fusion data to the driving route prediction module 340.

[0096] According to an embodiment, the fusion module 330 may include a cascading module 331 and an embedding module 332 .

[0097] The concatenation module 331 may receive input of the first feature vector output from the first feature extraction module 310 and the second feature vector output from the second feature extraction module 320. The module 331 may concatenate the first feature vector and the second feature vector and then output the concatenated vector to the embedding module 332.

[0098] Hereinafter, a feature vector in which the first feature vector and the second feature vector are cascaded with each other is referred to as a "cascaded feature vector (or cascaded feature vector)".

[0099] The embedding module 332 may embed the concatenated feature vector output from the concatenation module 331 into a preset prediction timestamp size to generate fused data. The module 332 may then output the generated fused data to the driving route prediction module 340.

[0100] For example, the embedding module 332 may embed 41 pieces at intervals of 0.1 seconds to 4 seconds. However, the present invention is not limited thereto.

[0101] The embedding module 332 may include a neural network configured to output a result when an input (x) is given through an operation (rectified linear unit (ReLU) function) as shown in the following equation 2. Hereinafter, the neural network included in the embedding module 332 is referred to as an embedding layer.

[0102] [Equation 2]

[0103] embedding output = relu(∑(w j x i +b j ))

[0104] Here, embedding output represents the embedding output, x represents the input feature, w represents the embedding layer weight, b represents the embedding layer bias, and i>j.

[0105] Accordingly, the embedding module 332 can be trained to update the weight (w) parameter in the direction in which the error between the given data and the inferred value becomes smaller. The embedding model is also used to compress the dimension of the input (x) into an output vector and fuse the input information.

[0106] The fusion module 330 according to the embodiment may concatenate the first feature vector and the second feature vector using the concatenation module 331. Then, the module 330 may embed the concatenated feature vector into a preset prediction timestamp size using the embedding module 332, thereby generating fused data.

[0107] This embodiment shows that the fusion module 330 is configured separately from the feature extraction modules 310 and 320 and the driving route prediction module 340. However, the present invention is not limited thereto.

[0108] For example, the cascade module 331 of the fusion module 330 can be combined with the first feature extraction module 310 and the second feature extraction module 320 into one module A. The embedding module 332 of the fusion module 330 can be combined with the driving route prediction module 340 into one module B.

[0109] As another example, the entire fusion module 330 may be included in the driving route prediction module 340. As yet another example, the cascade module 331 may be provided separately from the feature extraction modules 310, 320 and the driving route prediction module 340. The embedding module 332 may be combined with the driving route prediction module 340 into one module.

[0110] According to an embodiment, the driving route prediction module 340 (or a neural network based decoder) may receive input of the fused data output from the fusion module 330. The module 340 may generate and output route prediction information based on the received fused data.

[0111] Figure 5 is a diagram for describing a driving route prediction module 340 of a driving route prediction apparatus 300 according to an embodiment of the present invention.

[0112] like Figure 5 As shown, the driving route prediction module 340 can output a result output by inputting the fused data into a neural network trained based on the input as route prediction information.

[0113] The driving route prediction module 340 may be a decoder based on a recurrent neural network (RNN) and a dense layer. The driving route prediction module 340 may include an RNN 341 and a dense layer 342. The RNN 341 may be a long short-term memory (LSTM).

[0114] The route prediction module 340 may be represented as an LSTM-based decoder (or LSTM decoder).

[0115] The RNN 341 may output route prediction information to the dense layer 342 based on the input fused data.

[0116] The dense layer 342 may arrange and output the route prediction information output from the RNN 341 so that the route prediction information forms an array

[0117] The driving route prediction module 340 according to the embodiment may input route prediction information, and output the route prediction information to the dense layer 342 by inputting fused data to the RNN 341 to output 1D route prediction information.

[0118] According to an embodiment, the driving route prediction module 340 may output route prediction information at each preset prediction timestamp time point.

[0119] For example, the driving route prediction module 340 may output route prediction information at every 41 time points at intervals of 0.1 seconds to 4 seconds.

[0120] In addition, route prediction information at each time point can be used To train the driving route prediction module 340.

[0121] The driving route prediction module 340 may use an objective function θ for training that optimizes the vehicle's driving route prediction result. * To train. The objective function θ * It can be defined as the following equation 3.

[0122] [Equation 3]

[0123]

[0124] Here, S t represents the predicted route at each time point, I represents the input image, and θ represents the overall neural network learning parameters.

[0125] Figure 6 is a flowchart for describing a driving route prediction method according to an embodiment of the present invention.

[0126] Figure 6 The driving route prediction method shown can be referred to Figure 1-5 The driving route prediction system 1 described is executed.

[0127] In the following, reference is made to Figure 1-6 The operation of the driving route prediction device 300 when performing driving route prediction will be mainly described.

[0128] First, the driving route prediction apparatus 300 may receive vehicle driving information and driving image information (step S600 ).

[0129] According to an embodiment, the driving route prediction apparatus 300 may receive the vehicle driving information output from the vehicle driving information provider 100 and the driving image information output from the driving image information provider 200 .

[0130] After step S600 , the driving route prediction apparatus 300 may extract a first feature vector from the vehicle driving information (step S610 ), and extract a second feature vector from the driving image information (step S620 ).

[0131] According to an embodiment, the driving route prediction device 300 may use the normalizer 311 to normalize each element included in the vehicle driving information (step S611). Then, the device 300 may input each normalized element into the dense layer 312 to extract the first feature vector of the 1D array (S612).

[0132] The dense layer 312 of the driving route prediction device 300 can be trained based on the input vehicle driving information, and extract the first feature vector based on the trained model.

[0133] The travel route prediction device 300 may perform maximum-minimum normalization using maximum and minimum values ​​as parameters of lane information, accelerator pedal control amount, brake pedal control amount, vehicle speed, and wheel speed among elements of vehicle travel information.

[0134] In addition, the driving route prediction device 300 may perform Gaussian distribution normalization using mean and variance as parameters of vehicle dynamics-based elements among the elements of vehicle driving information. The vehicle dynamics-based elements may be steering angle, yaw rate, longitudinal acceleration, or lateral acceleration.

[0135] According to an embodiment, the driving route prediction device 300 can extract the second feature vector of the 2D array from the driving image information using the CNN 321 (step S621). The device 300 can also input the second feature vector of the 2D array into the dense layer 322 (step S622) to extract the second feature vector of the 1D array.

[0136] The CNN 321 of the driving route prediction apparatus 300 may be trained based on input driving image information, and extract a second feature vector of the 2D array based on the trained model.

[0137] The dense layer 322 of the driving route prediction device 300 may be trained based on the second feature vector of the input 2D array, and extract the 1D second feature vector based on the trained model.

[0138] After steps S610 and S620 , the driving route prediction apparatus 300 may generate fusion data based on the first feature vector and the second feature vector (step S630 ).

[0139] According to an embodiment, the driving route prediction device 300 can use the cascade module 331 to cascade the first feature vector and the second feature vector (step S631). The device 300 can also input the cascaded feature vector (or cascaded feature vector) into the embedding module 332 (step S632) to generate fused data embedded in a preset prediction timestamp size.

[0140] The embedding module 332 of the driving route prediction device 300 may include a neural network (embedding layer) configured to output a result through a ReLU function.

[0141] The embedding module 332 of the driving route prediction device 300 may be trained based on the input fusion data, and may generate fusion data based on the trained model.

[0142] After step S630 , the driving route prediction device 300 may generate route prediction information based on the fused data (step S640 ).

[0143] According to an embodiment, the driving route prediction device 300 may use the route prediction information (including the vehicle speed v) output by inputting the fusion data to the RNN 341. x , yaw rate The X-direction moving distance dx, the Y-direction moving distance dy and the heading moving angle dθ) are used as the input of the dense layer 342. Thus, a 1D array form such as path prediction information.

[0144] For example, the RNN 341 of the driving route prediction device 300 may be an LSTM neural network.

[0145] The RNN 341 of the driving route prediction device 300 may output route prediction information at each preset prediction timestamp time point.

[0146] The RNN 341 of the driving route prediction device 300 may use an objective function θ for training that optimizes the driving route prediction result of the vehicle. * Come to train.

[0147] Fig. 7A and 7B 1 is a diagram for comparing a predicted route according to a conventional travel route prediction technique with a predicted route according to a travel route prediction method of the present invention.

[0148] Fig. 7A 1 is a diagram showing a predicted route (route A) when traveling straight according to a conventional traveling route prediction technique and a predicted route (route B) when traveling straight according to a traveling route prediction method of the present invention.

[0149] also, Figure 7B: is a diagram showing a predicted route when turning left according to a conventional travel route prediction technique (route C) and a predicted path when turning left according to a travel route prediction method of the present invention (route D).

[0150] exist Fig. 7A and 7B In , the prediction time period is 4 seconds, and the constant speed / acceleration constant turning speed model is used as the traditional driving route prediction technology.

[0151] Reference Fig. 7A , the prediction according to the conventional driving route prediction technology and the prediction according to the driving route prediction method of the present invention are performed at two time points t1 and t2. When the ego vehicle passes through the intersection in a straight line, it can be seen that there is no significant difference in the route prediction time point (or the number of route predictions) and the predicted route between the conventional technology and the technology of the present invention with the same prediction time period of 4 seconds.

[0152] However, reference Figure 7B , it can be seen that the route prediction according to the conventional technology is performed at 3 time points (t1 to t3), while the route prediction according to the technology of the present invention is performed at 7 time points (t1 to t7) with the same prediction time period.

[0153] When the ego vehicle turns (left turn) through the intersection, it can be seen that with the same prediction time period of 4 seconds, in the route prediction according to the technology of the present invention, the route prediction is performed at more time points (or the number of route predictions is greater).

[0154] Furthermore, it can be seen that the path predicted according to the technique of the present invention (route D) is formed at a greater distance from other vehicles and pedestrians than the path predicted according to the conventional technique (route C).

[0155] Therefore, in the case where the ego vehicle turns left through an intersection, it can be seen that the route prediction according to the technology of the present invention is much more effective than the route prediction according to the conventional technology.

[0156] According to an embodiment of the present invention, a driving route prediction device, system and method for a vehicle can be provided, which can predict a driving route using a model trained based on actual driving data.

[0157] The driving route prediction technology according to the embodiment of the present invention can use a neural network model to predict the driving route of a vehicle, which is an end-to-end (E2E) model that uses actual road driving data as input for training and directly processes the input data as output data.

[0158] Therefore, when the driving route prediction technology according to an embodiment of the present invention is used, a long-term complex driving route can be effectively predicted compared to the traditional constant turning rate and acceleration (CTRA) model or constant turning rate and speed (CTRV) model for predicting a short-term driving route.

[0159] Furthermore, the travel route prediction technology according to the embodiment of the present invention can perform training using travel image information as input in addition to vehicle travel information and predict a travel route.

[0160] The driving image information according to the embodiment may include driving environments that are not considered in advance. Therefore, the driving route prediction technology according to the embodiment of the present invention can predict the driving route by learning various driving environments that are not considered in advance.

[0161] Therefore, using the driving route prediction technology according to the embodiment of the present invention, in addition to predicting the route of the driving environment previously considered, it is also possible to effectively predict the driving routes of various driving environments that have not been considered in advance.

[0162] The effects of the present invention are not limited to the above effects. A person skilled in the art should be able to clearly understand other effects not mentioned through the above description.

[0163] Since the above-mentioned problems to be solved, means for solving the problems and effects are not used to define the essential features of the claims, the scope of the claims is not limited by the matters described in the summary of the invention.

[0164] Although the embodiments of the present invention have been described in more detail with reference to the accompanying drawings, the present invention is not limited to these embodiments, and various modifications and implementations can be made without departing from the technical spirit of the present invention. Therefore, the embodiments disclosed herein are not intended to limit the technical spirit of the present invention, but to describe the spirit of the present invention. The scope of the technical spirit of the present invention is not limited by these embodiments. Therefore, it should be understood that the above-mentioned embodiments are illustrative in all aspects and not restrictive. The scope of protection of the present invention should be interpreted according to the scope of the claims, and all technical ideas within the scope equivalent thereto should be interpreted as included within the scope of the present invention.

Claims

1. A driving route prediction device for a vehicle, the driving route prediction device include: A first feature extraction module is configured to extract a first feature vector from vehicle driving information; A second feature extraction module is configured to extract a second feature vector from the driving image information; a fusion module, configured to fuse the first feature vector and the second feature vector to generate fused data; as well as The driving route prediction module is configured to output route prediction information of the vehicle based on the fused data.

2. The driving route prediction device according to claim 1, in, The first feature extraction module normalizes each element included in the vehicle driving information, and inputs each normalized element into a dense layer to extract a first feature vector of a one-dimensional 1D array.

3. The driving route prediction device according to claim 2, in, The first feature extraction module performs maximum-minimum normalization on lane information, accelerator pedal control amount, brake pedal control amount, vehicle speed, and wheel speed among the elements included in the vehicle driving information.

4. The driving route prediction device according to claim 2, in, The first feature extraction module performs Gaussian distribution normalization on elements based on a vehicle dynamics model among the elements included in the vehicle driving information.

5. The driving route prediction device according to claim 4, in, The elements based on the vehicle dynamics model include steering angle, yaw rate, longitudinal acceleration and lateral acceleration.

6. The driving route prediction device according to claim 1, in, The second feature extraction module extracts a second feature vector of a two-dimensional 2D array from the driving image information using a convolutional neural network (CNN), and inputs the second feature vector of the 2D array into a dense layer to extract a second feature vector of a 1D array.

7. The driving route prediction device according to claim 1, in, The fusion module cascades the first feature vector and the second feature vector to generate a cascade feature vector, and embeds the cascade feature vector into a preset prediction timestamp size to generate the fused data.

8. The driving route prediction device according to claim 7, in, The fusion module includes a neural network that uses the cascaded feature vector as input and outputs an operation result based on a rectified linear unit (ReLU) function.

9. The driving route prediction device according to claim 1, in, The driving route prediction module inputs the fused data into a recurrent neural network (RNN) and inputs the route prediction information output from the RNN into a dense layer to output the route prediction information in a 1D array.

10. The driving route prediction device according to claim 1, in, The driving route prediction module is based on the prediction time stamp at each preset time point. The output includes the vehicle speed v x , yaw rate Route prediction information of X-direction moving distance dx, Y-direction moving distance dy and heading moving angle dθ.

11. The driving route prediction device according to claim 1, in, The objective function θ for training is used according to the following equation * To train the driving route prediction module, the objective function θ * Optimize driving route prediction results: Among them, S 2 represents the predicted route at each time point, I represents the input image, and θ represents the overall neural network learning parameters.

12. A method for predicting a driving route of a vehicle, the driving route prediction method The following steps are involved: Extracting a first feature vector from vehicle driving information; extracting a second feature vector from the driving image information; fusing the first feature vector and the second feature vector to generate fused data; and Output vehicle route prediction information based on the fused data.

13. The driving route prediction method according to claim 12, in, The step of extracting the first eigenvector comprises: normalizing each element included in the vehicle driving information; and Each normalized feature is input into a dense layer to extract the first eigenvector of a 1D array.

14. The driving route prediction method according to claim 13, in, The step of extracting the first feature vector includes performing maximum-minimum normalization or Gaussian distribution normalization on the elements included in the vehicle driving information according to the type of the elements.

15. The driving route prediction method according to claim 12, in, The steps of extracting the second eigenvector include: Extracting a second feature vector of a two-dimensional 2D array from the driving image information using a convolutional neural network (CNN); and The second eigenvector of the 2D array is input to a dense layer to extract the second eigenvector of the 1D array.

16. The driving route prediction method according to claim 12, in, The steps to generate fused data include: concatenating the first feature vector and the second feature vector to generate a concatenated feature vector; and The concatenated feature vector is embedded into a preset prediction timestamp size to generate the fused data.

17. The driving route prediction method according to claim 12, in, The steps for outputting route prediction information include: Inputting the fused data into a recurrent neural network (RNN); and The route prediction information output from the RNN is input to the dense layer to output the route prediction information in a 1D array.

18. The driving route prediction method according to claim 12, in, The step of outputting route prediction information includes: outputting route prediction information at each preset prediction time stamp time point The output includes the vehicle speed v x , yaw rate Route prediction information of X-direction moving distance dx, Y-direction moving distance dy and heading moving angle dθ.

19. The driving route prediction method according to claim 12, in, The step of outputting route prediction information is performed by a driving route prediction module, which uses an objective function θ for training according to the following equation: * To train, the objective function θ * Optimize driving route prediction results: Among them, S t represents the predicted route at each time point, I represents the input image, and θ represents the overall neural network learning parameters.

20. A vehicle driving route prediction system, the driving route prediction system include: a vehicle driving information provider configured to provide vehicle driving information indicating a driving state of a vehicle; A driving image information provider configured to provide driving image information related to the front and surrounding environment of the driving vehicle; and Driving route prediction equipment, Wherein, the driving route prediction device comprises: A first feature extraction module is configured to extract a first feature vector from the vehicle driving information; A second feature extraction module is configured to extract a second feature vector from the driving image information; a fusion module configured to fuse the first feature vector and the second feature vector to generate fused data; and The driving route prediction module is configured to output route prediction information of the vehicle based on the fused data.

Images