Method and apparatus for determining the road path of a moving vehicle using an artificial neural network, and navigation apparatus including the same.

The vehicle path determination device uses an artificial neural network to accurately predict lanes from driving video, improving navigation and autonomous driving stability by calibrating the neural network for precise lane information extraction.

JP7954323B2Active Publication Date: 2026-11-19FORTY TWO DOT INC
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2026004306
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-09-08
Filing Date
2026-01-14
Publication Date
2026-11-19
Estimated Expiration
2041-09-07

AI Technical Summary

Technical Problem

Existing navigation systems struggle to accurately determine the lane a vehicle is traveling on, especially at intersections, leading to potential accidents and instability in autonomous driving, as they rely on GPS-based positioning without precise lane information.

Method used

A vehicle path determination device using an artificial neural network that processes driving video from cameras to predict lane information and calibrate the neural network model for improved accuracy, incorporating confidence information and exception classes to enhance reliability.

Benefits of technology

Provides more accurate lane prediction and guidance, enabling stable lane changes and autonomous driving by automatically extracting lane information from driving images, reducing the need for manual labeling and enhancing map creation for autonomous systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007954323000002
    Figure 0007954323000002
  • Figure 0007954323000003
    Figure 0007954323000003
  • Figure 0007954323000004
    Figure 0007954323000004
Patent Text Reader

Abstract

To provide a method for determining a driving lane of a vehicle.SOLUTION: An apparatus for determining a lane of a vehicle using an artificial neural network according to an embodiment includes an image information collecting module configured to acquire driving image information of the vehicle from at least one camera module installed in the vehicle, and an image information processing module configured to receive the driving image information as input information. A pre-learned lane prediction artificial neural network module configured to generate first data distribution degree by calculating a data distribution degree of the output information; an output information distribution calculation module configured to collect reference information on actual driving lane prediction information of the vehicle and calculate a data distribution degree of the reference information; A reference information distribution calculation module for generating a second data distribution, and a reliability calibration module for updating a parameter of the artificial neural network module based on the second data distribution such that a difference between the first data distribution and the second data distribution is reduced.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a lane determination method and apparatus for a traveling vehicle using an artificial neural network, and a navigation device including the same. More specifically, the present invention relates to a technology for providing lane prediction information indicating which lane a currently traveling vehicle is traveling in and reliability information thereof by utilizing an artificial neural network based on information related to a traveling image of the vehicle.

Background Art

[0002] In general, when a user inputs a destination, a navigation device used in a vehicle searches for a travel route to the destination using map information, and guides the vehicle along the travel route selected by the user. In addition, the navigation device provides various types of information such as the travel route to the destination, features located around the travel route, and road congestion level visually or audibly, thereby providing guidance that allows the user to arrive at the destination.

[0003] In order to guide accurate travel information to the driver of the vehicle using the navigation device, it is impossible to provide accurate guidance to the driver of the vehicle without knowing the position where the current vehicle is traveling, specifically, the lane prediction information indicating which lane the current vehicle is traveling in. In particular, when a turning point is included in the travel route, such as at an intersection, an IC (Inter Change) where a general road and an expressway meet, and a JC (Junction) where an expressway and another expressway meet, by notifying the driver of the vehicle of information on whether to keep the current lane or change lanes at an appropriate time, accidents caused by sudden lane changes can be prevented in advance.

[0004] However, in the case of the navigation device according to the related art, since the current vehicle position is determined on a GPS basis, there is a problem that it is difficult to accurately acquire the lane prediction information for the currently traveling vehicle. Therefore, the actual situation is that the vehicle guidance using the navigation device merely provides guidance only on travel information.

[0005] Furthermore, for autonomous driving to be stable, accurate information about the road currently being traveled on is essential; otherwise, it is impossible to stably perform actions such as changing lanes or entering intersections. Currently, there is no technology that can accurately determine information about the road currently being traveled on, which is why the stability of autonomous driving is not yet guaranteed.

[0006] Furthermore, in order to create maps for autonomous driving, it is necessary to include information such as the total number of lanes on each road and which lane the currently moving vehicle is in. However, with conventional technology, it is not possible to automatically extract lane information by looking at the road image, and there was a problem that the creator had to manually label it. [Overview of the project] [Problems that the invention aims to solve]

[0007] Therefore, the method and apparatus for determining the road path of a moving vehicle using an artificial neural network, and the navigation device including the same, according to one embodiment, are inventions devised to solve the problems described above, and their purpose is to provide more accurate road path prediction information for a currently moving vehicle using an artificial neural network.

[0008] More specifically, the objective is to improve the reliability of the artificial neural network module that performs road prediction by using an artificial neural network module to predict the road path based on input information about the vehicle driving image, and then performing calibration on the artificial neural network module based on the predicted information and reference information, thereby adjusting the reliability of the actual artificial neural network model to resemble the actual results.

[0009] Furthermore, by providing information on vehicles currently in motion and their reliability, the aim is to provide drivers with more accurate information on the current road conditions, while program developers can utilize road forecast information and its reliability to provide various guidance services for navigation systems.

[0010] Furthermore, the vehicle lane determination device for a moving vehicle using an artificial neural network according to one embodiment aims to automatically extract lane information based on driving video and easily create a map that includes lane information. [Means for solving the problem]

[0011] A vehicle path determination device using an artificial neural network according to one embodiment may include: a video information collection module that acquires driving video information of the vehicle from at least one camera module installed on the vehicle; a pre-trained vehicle path prediction artificial neural network module that takes the driving video information as input information and outputs vehicle path prediction information and confidence information for the vehicle path prediction information as output information; an output information distribution calculation module that calculates the data distribution of the output information and generates a first data distribution; a reference information distribution calculation module that collects reference information for the actual driving path prediction information of the vehicle, calculates the data distribution of the reference information and generates a second data distribution; and a confidence calibration module that updates the parameters of the artificial neural network module so that the difference between the first data distribution and the second data distribution decreases based on the second data distribution.

[0012] The method is characterized by updating the parameters of the artificial neural network module by classifying the logic vectors for all classes of the artificial neural network module into a single scalar parameter.

[0013] The confidence calibration module can perform training on the single scalar parameter using the same calibration validation set as the validation set of the artificial neural network module.

[0014] The confidence calibration module can learn the single scalar parameter so that the NLL (Negative Log Likelihood) value in the calibration validation set is minimized.

[0015] The aforementioned lane prediction information may include lane information indicating which lane the vehicle is currently traveling in, based on the first or last lane of the road on which the vehicle is traveling.

[0016] The artificial neural network module includes a first output layer and a second output layer, the first output layer outputs first lane information, which is information about which lane the vehicle is currently traveling in relative to one lane of the road the vehicle is traveling on, and a confidence value for the first lane information, and the second output layer can output second lane information, which is information about which lane the vehicle is currently traveling in relative to the last lane of the road the vehicle is traveling on, and a confidence value for the second lane information.

[0017] A method for determining the road path of a moving vehicle using another artificial neural network according to one embodiment is a road path determination method using a pre-trained road path prediction artificial neural network module that takes driving video information as input information and outputs road path prediction information of the vehicle and confidence information for the road path prediction information, and may include: a video information collection step of acquiring driving video information of the vehicle from at least one camera module installed on the vehicle; an output information distribution calculation step of calculating the data distribution of the output information and generating a first data distribution; a reference information distribution calculation step of collecting reference information for the actual driving road path prediction information of the vehicle, calculating the data distribution of the reference information and generating a second data distribution; and a step of updating the parameters of the artificial neural network module so that the difference between the first data distribution and the second data distribution decreases based on the second data distribution.

[0018] A navigation device according to one embodiment may include: a video information collection module that acquires driving video information of the vehicle from at least one camera module installed in the vehicle; a pre-trained artificial neural network module that takes the driving video information as input information and outputs the vehicle's predicted road path information and confidence information for the predicted road path information as output information; an output information distribution calculation module that calculates the data distribution of the output information and generates a first data distribution; a reference information distribution calculation module that collects reference information for the actual driving predicted road path information of the vehicle, calculates the data distribution of the reference information and generates a second data distribution; a confidence calibration module that updates the parameters of the artificial neural network module so that the difference between the first data distribution and the second data distribution decreases based on the second data distribution; and a display module that displays both the output information output from the artificial neural network module and the vehicle's driving video externally. [Effects of the Invention]

[0019] According to one embodiment, a method and apparatus for determining the road path of a moving vehicle using an artificial neural network, and a navigation device including the same, perform calibration by updating the parameters of the artificial neural network module based on the results inferred by the artificial neural network module and reference information, thereby producing the effect that the reliability information output from the artificial neural network module approaches the actual probability of the road path prediction information.

[0020] Furthermore, a vehicle lane determination method and apparatus using an artificial neural network according to one embodiment, and a navigation system including the same, can provide more reliable predictive information about the lane, thus enabling more stable lane changes and driving during autonomous driving.

[0021] Furthermore, a navigation system including a vehicle road judgment device using an artificial neural network according to one embodiment can provide the driver with more accurate guidance in areas where accidents are likely to occur, such as lane changes and turning areas, based on more reliable road prediction information, thereby enabling the driver to drive more stably.

[0022] Furthermore, the vehicle lane determination method and apparatus for a moving vehicle using an artificial neural network according to one embodiment can accurately extract lane information of the road the vehicle is traveling on based on the vehicle's driving image. This has the effect of easily producing maps that include lane information without having to manually input lane information during map production. [Brief explanation of the drawing]

[0023] [Figure 1] This is a block diagram showing some of the components of a vehicle path determination device using an artificial neural network according to one embodiment. [Figure 2] This figure shows the input information and output information of a road path prediction artificial neural network module according to one embodiment. [Figure 3] This figure shows the structure of an artificial neural network module applied to the present invention. [Figure 4]This is a schematic diagram showing the calculation process at nodes of an artificial neural network module. [Figure 5] This is a diagram for explaining the process in which input information input to a CNN is converted into output information according to an embodiment. [Figure 6] This is a diagram showing the relationship between an artificial neural network module and each constituent element according to an embodiment. [Figure 7] (a) is a general softmax function program code, and (b) is a diagram describing code obtained by calibrating the softmax function according to an embodiment. [Figure 8] This is a diagram showing a comparison of results calculated when a program is executed using the code of Figure 7(b). [Figure 9] This is a diagram for explaining a further embodiment of the output layer of the present invention. [Figure 10] This is a diagram showing a state in which information output by an artificial neural network is displayed on a display module according to an embodiment. [Figure 11] This is a diagram for explaining a method of calculating final lane prediction information based on lane prediction information output by an artificial neural network module and reliability information corresponding thereto according to another embodiment. [Figure 12] This is a diagram showing input information and output information input to an artificial neural network module according to a further embodiment. [Figure 13] This is a diagram showing an example for a point where lane division is not clear according to an embodiment of the present invention. [Figure 14] This is a diagram showing a comparison of results output from the lane prediction artificial neural network module when there is no exception class information and when there is exception class information. [Figure 15] This is a diagram showing a comparison of results output from the lane prediction artificial neural network module when there is no exception class information and when there is exception class information. [Figure 16] This is a diagram showing a comparison of results output from the lane prediction artificial neural network module when there is no exception class information and when there is exception class information. [Figure 17]This figure illustrates how a road feature vector is extracted in a road path prediction artificial neural network module according to one embodiment. [Figure 18] This figure shows an artificial neural network module for determining lane changes according to one embodiment. [Figure 19] A block diagram showing some of the components of a navigation device according to one embodiment. [Modes for carrying out the invention]

[0024] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In assigning reference numerals to the components in each drawing, it should be noted that the same components will have the same reference numeral whenever possible, even if they are shown in other drawings. Furthermore, in describing embodiments of the present invention, if a detailed description of a relevant published configuration or function is deemed to hinder understanding of the embodiments of the present invention, such detailed description will be omitted. In addition, while embodiments of the present invention will be described below, the technical idea of ​​the present invention is not limited or restricted and may be modified and implemented in various ways by those skilled in the art.

[0025] Furthermore, the terms used herein are for illustrative purposes only and are not intended to limit and / or restrict the disclosed invention. Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0026] In this specification, terms such as “includes,” “equipment,” or “have” are intended to indicate the presence of features, figures, stages, operations, components, parts, or combinations thereof described in the specification, and do not preemptively exclude the possibility of the presence or addition of one or more other features, figures, stages, operations, components, parts, or combinations thereof.

[0027] Furthermore, throughout the specification, if any part is “connected” to another part, this includes not only “directly connected” parts but also “indirectly connected” parts with further elements in between, and the ordinal terms such as “first,” “second,” etc. used herein may be used to describe various components, but the above-mentioned components are not limited to these terms.

[0028] In the following, embodiments of the present invention will be described in detail with reference to the attached drawings, so that those with ordinary skill in the art to which the present invention pertains can easily implement them. Furthermore, in order to clearly illustrate the present invention with the drawings, parts unrelated to the description will be omitted.

[0029] Figure 1 is a block diagram showing some components of a road path determination device 100 using an artificial neural network according to one embodiment of the present invention, and Figure 2 is a diagram showing input information input to a road path prediction artificial neural network module 120 according to one embodiment and output information output to it. Hereinafter, for the sake of convenience of explanation, the road path determination device 100 using an artificial neural network will be referred to as the road path determination device 100.

[0030] Referring to Figure 1, the roadway determination device 100 includes a video information acquisition module 110, a roadway prediction artificial neural network module 120, an output information distribution calculation module 130, a reference information distribution calculation module 140, a reliability calibration module 150, a feature extraction module 160, and a roadway change confirmation artificial neural network module 170, among others.

[0031] The video information collection module 110 collects video information of the vehicle's front and sides, captured via at least one camera mounted on the vehicle, and transmits the collected video to the road prediction artificial neural network module 120.

[0032] The vehicle path prediction artificial neural network module 120 takes vehicle driving video information 10 collected by the video information collection module 110 as input information and outputs vehicle path prediction information 20 inferred based on which lane the current vehicle is traveling in, and confidence information 30 for the inferred vehicle path as output information. The vehicle path prediction artificial neural network module 120 includes a learning session 121 that performs learning based on the input information and output information, and an inference session 122 that infers output information based on the input information.

[0033] The learning session 121 of the road path prediction artificial neural network module 120 is a session that performs learning based on input information 10 and output information 30,40. The inference session 122 uses the learned road path prediction artificial neural network module 120 to analyze the real-time input driving video information 10 and can output both road path prediction information 20 that infers which road the vehicle is currently traveling on, and confidence information 30 for that information.

[0034] For example, when an image is input to the road path prediction artificial neural network module 120, the input image is analyzed and the road path for the currently moving vehicle is inferred. If it is determined that the road path is the first road path, the first road path is output as road path prediction information 20, and at the same time, confidence information 30 (for example, 0.8) for determining it to be the first road path is output as output information.

[0035] Furthermore, the road prediction information 20 output from the road prediction artificial neural network module 120 can be output based on information regarding which lane a vehicle is in relative to a specific lane. For example, road information can be output regarding which lane a vehicle is currently traveling in relative to one lane of the road, or conversely, road information can be output regarding which lane a vehicle is currently traveling in relative to the last lane of the road.

[0036] In providing lane information, when information is provided as in the present invention, based on a specific lane, regarding which lane a vehicle is in, the driver can understand which lane their vehicle is in relative to the reference lane adjacent to the vehicle. This has the effect of allowing the driver to prepare for and change lanes more stably along the driving route.

[0037] Figure 3 shows the structure of the artificial neural network module applied to the present invention, and Figure 4 is a schematic diagram showing the calculation process at the nodes of the artificial neural network module.

[0038] Referring to Figure 3, the multilayer neural network model of the vehicle path prediction artificial neural network module 120 consists of an input layer 60, a hidden layer 70, and an output layer 80.

[0039] The input layer 60 consists of nodes corresponding to each input factor, and the number of nodes is the same as the number of input factors. The hidden layer 70 processes the linear combination of factor values ​​transmitted from the input layer 60 into a nonlinear function such as a sigmoid function and transmits it to the output layer or other hidden layers. In Figure 3, due to space limitations, the road path prediction artificial neural network module 120 is shown as having one hidden layer 70, but it is not limited to this, and the road path prediction artificial neural network module 120 may include multiple hidden layers 70. The output layer 80 generates output nodes corresponding to output factors, with the number of output nodes equal to the number of classes in the classification model. In this invention, the output information consists of road path prediction information 20 regarding which lane the vehicle is currently traveling in and confidence information 30 for that information, and the output layer 80 is shown as a single layer as shown in Figure 3. However, this invention is not limited to this, and it may consist of multiple output layers 80 depending on the application environment of the road path prediction artificial neural network module 120.

[0040] As shown in Figure 4, when input information 10 is input to the trajectory prediction artificial neural network module 120, calculations are actually performed at each node. The calculation process of the trajectory prediction artificial neural network module 120 is mathematically designed to mimic the processes that occur in neurons that make up the human nervous system.

[0041] A node responds to stimuli above a certain magnitude, and the magnitude of this response, excluding bias, is roughly proportional to the product of the input value and the node's parameters (or weights). Generally, a node receives several inputs and has as many parameters as there are inputs. Therefore, by adjusting these parameters, different weights can be assigned to various inputs. Finally, all the multiplied values ​​are added together, and this sum is fed into the activation function of the output layer 80. The result of the activation function corresponds to the node's output, and this output value is ultimately used for classification and regression analysis.

[0042] Each layer of the neural network model consists of at least one node, and the activation or deactivation of each node is determined by the input value. The input information becomes the input to the first layer (input layer), and then the output of each layer becomes the input to the next layer. All coefficients change slightly during the learning process, but ultimately they reflect what kind of input each node considers important. The "training" of the artificial neural network model mentioned above refers to the process of updating these coefficients.

[0043] The structure of a typical CNN network applied to the vehicle path prediction artificial neural network module 120 of the present invention will be described below with reference to Figure 5. Figure 5 is a diagram illustrating the process by which input information input to a CNN is converted into output information according to one embodiment of the present invention.

[0044] A CNN (Convolutional Neural Network) is a type of feedforward deep neural network (DNN) used to analyze visual images. It consists of one or more convolutional layers, a pooling layer, and a fully connected layer. In Korean, it is called a convolutional neural network. CNNs have a suitable structure for learning 2D data and can be trained via the backpropagation algorithm.

[0045] The CNN process can obtain global features that represent the overall image from local features by repeatedly performing feature extraction, influence minimization, and classification tasks.

[0046] Applying this to the present invention and explaining it with reference to Figure 5, when the video collected by the video acquisition module 110 is input as input information 10 to the road path prediction artificial neural network module 120, the road path prediction artificial neural network module 120 applies a filter to the input image 11 to generate a first feature map 12. Applying a filter means performing a convolution operation on the image using a filter having specific coefficients.

[0047] The trajectory prediction artificial neural network module 120 generates a first feature map 12 using a specific filter. Once the first feature map 12 is generated, subsampling is applied to the first feature map 12 to reduce the image size. Through the subsampling process, a second feature map 13 containing local features is generated. The trajectory prediction artificial neural network module 120 then repeatedly applies a filter with different features to the second feature map 13 and performs subsampling again. By continuing to repeat this process, a final feature map 14 containing global features can be generated. The final feature map 14 thus obtained is connected as an input to the fully connected network 17, and output information can be generated via the output layer. In the present invention, the trajectory prediction artificial neural network module 120 generates a final feature map from the input image 11 using the method described above. The final feature map is connected as an input to the fully connected network 17, and through the activation function of the output layer 80, the final output information, which is trajectory prediction information 20 and confidence information 30 therefor, can be output.

[0048] As mentioned above, in the case of CNNs, repeatedly applying filters with the same coefficients to the entire image dramatically reduces the number of variables, which has the effect of obtaining invariance that is unrelated to topological changes. This effect can improve accuracy when determining roadways based on input images, as in the present invention.

[0049] Furthermore, although this specification has described the neural network applied to the trajectory prediction artificial neural network module 120 of the present invention based on CNN, the neural network structure applied to the trajectory prediction artificial neural network module 120 of the present invention is not limited to CNN, and various artificial neural network models published such as Google Mobile Net v2, VGGNet16, and ResNet50, which are useful for image detection, can be applied.

[0050] Figure 6 is a diagram illustrating the relationship between the artificial neural network module and its components according to one embodiment of the present invention, specifically illustrating the process by which the parameters of the artificial neural network module are updated through learning.

[0051] Referring to Figure 6, the road path prediction information 20 and the confidence level information 30 therefor, output by the road path prediction artificial neural network module 120, can be input from the output information distribution calculation module 130.

[0052] The output information distribution calculation module 130 is a module that specifically calculates the data distribution for the road prediction information 20 and the confidence information 30 therefor, which are output from the road prediction artificial neural network module 120.

[0053] For example, if the trajectory prediction artificial neural network module 120 outputs trajectory prediction information 20 and confidence information 30 based on 100 input images, the output information distribution calculation module 130 receives the trajectory prediction information 20 and confidence information 30 output by the trajectory prediction artificial neural network module 120, analyzes the distribution of this information based on the received information, and generates a first data distribution (X). The first data distribution (X) represents information about the distribution of the output information inferred and output for each image input to the trajectory prediction artificial neural network module 120.

[0054] The reference information distribution calculation module 140 can collect reference information for the actual road path prediction information of a vehicle, calculate the data distribution of the reference information, and generate a second data distribution (Y).

[0055] Reference information, referred to as Ground Truth information, serves as the basis for updating the parameters of the artificial neural network module. In this invention, the reference information may include actual road information for the image input to the road path prediction artificial neural network module 120.

[0056] The confidence calibration module 150 may perform calibration by changing various parameter values ​​of the artificial neural network model so that the output value of the artificial neural network model reflects the actual confidence. For example, if the model's output for input information A and output information B is 0.9, calibration can be performed to change various parameter values ​​of the artificial neural network model so that it means there is a 90% probability that B is true.

[0057] The reliability calibration module 150 can calibrate the model of the road path prediction artificial neural network module 120 by receiving a first data distribution degree (X) from the output information distribution calculation module 130 and a second data distribution degree (Y) from the reference information distribution calculation module 140, and updating the parameters of the road path prediction artificial neural network module 120 so that the difference between the first data distribution degree (X) and the second data distribution degree (Y) decreases based on the second data distribution degree (Y).

[0058] In relation to road path prediction, one embodiment can be described as follows: For 100 input images, the road path prediction information inferred by the road path prediction artificial neural network module 120 shows that 20% are predicted as the first road, 50% as the second road, and 30% as the third road. If, according to the reference information, the proportions for the first road are 30%, the proportions for the second road are 45%, and the proportions for the third road are 25%, then the confidence calibration module 150 can update the parameters of the road path prediction artificial neural network module 120 to match the actual reference information proportions.

[0059] According to this, the reliability information 30 output from the road path prediction artificial neural network module 120 will approach the actual probability for the road path prediction information 20.

[0060] Furthermore, as one embodiment, the confidence calibration module 150 may perform calibration as a method of softening the softmax function corresponding to the output layer. Specifically, in the case of a multiple classification problem with K labels, calibration can be performed by converting the logic vector Z using a single scalar parameter T to the following formula (1).

[0061] Formula (1)

number

[0062] q^ represents the calibrated probability, and the above formula means that in a multiple classification problem with K labels, calibration is performed by dividing the logic vector by a single scalar parameter T.

[0063] In equation (3), when the single scalar parameter T is 1, it becomes the original formula for calculating confidence using Softmax, where the larger T is, the closer the final q is to 1 / K, and the closer T is to 0, the closer q is to 1.

[0064] Furthermore, while the optimal value of a single scalar parameter T is determined through learning, learning can be performed on the single scalar parameter T using a calibration validation set, which is a further validation set having the same characteristics as the validation set of the road path prediction artificial neural network module 120.

[0065] Specifically, the confidence calibration module 150 may use a calibration validation set to train a single scalar parameter T so that the Negative Log Likelihood (NLL) value in the calibration validation set is minimized. When calibration is performed in this way, the single scalar parameter T can be calibrated without changing the maximum value of the softmax output. Therefore, this method has the advantage of being applicable to conventionally trained artificial neural network modules because it only affects the calibration of the conventional model and does not affect the accuracy.

[0066] Figures 7 and 8 illustrate the effect of calibration performed by the confidence calibration module. Figure 7(a) is the program code for a typical softmax function, Figure 7(b) is the code for the softmax function after prior calibration, and Figure 8 is a graph showing the probability (P) values ​​calculated when applying T=0 (Figure 8(a)), T=2 (Figure 8(b)), and T=10 (Figure 8(c)) to a single scalar parameter T using Figure 7(b).

[0067] When T is increased sequentially from 0, it is observed that the difference between each probability value decreases, as shown in Figure 10. However, since the order can change, this does not affect accuracy. Therefore, when calibration is performed using the reliability calibration module of the present invention, the reliability of the road path prediction artificial neural network module 120 is improved without affecting accuracy.

[0068] Figure 9 is a diagram illustrating a further embodiment of the output layer of the present invention, and shows how the information output by the artificial neural network of Figure 10 is displayed on the display module of a navigation device.

[0069] Referring to Figure 9, the input information received by the road path prediction artificial neural network module 120 of the present invention is the same as in Figure 3 as the driving video information 10, but the output layer includes a first output layer 81 and a second output layer 82, which are not a single layer.

[0070] The first output layer 81 and the second output layer 82 act as activation functions, and typically, a softmax function or a cross-entropy function may be used. For the sake of explanation, the softmax function will be used as the basis for the following explanation.

[0071] As shown in Figure 9, when the roadway prediction artificial neural network module 120 of the present invention uses a first softmax function and a second softmax function in the output layer 80, the first softmax function outputs first roadway prediction information 21 regarding which roadway the currently traveling vehicle is operating in relative to the first lane, and first roadway reliability information 31 therefor. The second softmax function outputs second roadway prediction information 22 regarding which roadway the currently traveling vehicle is operating in relative to the last lane, and second roadway reliability information 32 therefor.

[0072] Therefore, as shown in Figure 10, the output information of the first softmax function can be such that the first lane prediction information 21, which is determined to be the second lane relative to the first lane, and the first lane reliability information 31 for that lane are output as 2 lanes and 0.999, respectively, and the output information of the second softmax function can be such that the second lane prediction information 22, which is determined to be the second lane relative to the last lane, and the reliability information 32 for that lane are output as 2 lanes and 0.994, respectively.

[0073] Generally, there is no major problem if the front camera installed on the vehicle can capture all the lanes of the road. However, depending on the camera's capabilities, the front camera may not capture all the lanes. Therefore, in such cases, using a single softmax function would not allow for the extraction of information about how many lanes there are, resulting in an inability to provide accurate lane information. However, as in the present invention, classifying using two softmax functions on a single network allows for the acquisition of information about all lanes, resulting in the provision of more accurate lane information.

[0074] Furthermore, when lane information can be extracted from a driving image as in the present invention, it has the effect of easily creating maps that include lane information. In other words, in the conventional technology, there was a problem that lane information could not be automatically extracted by looking at a road image, and the creator had to manually label it. However, the vehicle lane determination method and apparatus for a driving vehicle using an artificial neural network according to one embodiment can automatically extract lane information based on driving video as described above, so it has the effect of making it easier to create maps that include lane information. Therefore, it has the effect of making it possible to create maps for autonomous driving, which require accurate knowledge of lane information, more accurately than with the conventional technology.

[0075] Figure 11 illustrates a method for calculating final road prediction information based on road prediction information output by a road prediction artificial neural network module and confidence information therefor, according to a further embodiment of the present invention.

[0076] Specifically, the blue line in Figure 11(a) is a graph that displays predicted road path information in real time, while Figure 11(b) is a graph that outputs confidence information for the predicted road path information in time.

[0077] Referring to Figure 11, the confidence level for the road path prediction information is 1 from 0 to t1 hours, and the road path prediction information is output as the second road path. Because the confidence level is 1, the blue line shown in Figure 11A for the road path shows almost no change.

[0078] However, as shown in Figure 11B, it can be seen that there is a change with a lot of confidence information between t1 and t2. This is a development that occurs because the lanes are changed. Therefore, it can be seen that the blue line shown in Figure 11A also changes continuously between two lanes and one lane. In such cases, the present invention can provide results by removing values ​​below a certain confidence level, selecting only meaningful confidence levels to derive the green line shown in Figure 11(a), and then deriving the red line through a final correction process. If confidence information does not exist, the process of correcting the result values ​​for the confidence level that changes in real time and calculating the final data must be done manually, but in the case of the present invention, it is possible to automatically derive results like the red line based on pre-set criteria using confidence information.

[0079] Figure 12 shows input and output information that are input to the artificial neural network module according to a further embodiment.

[0080] The roadway prediction artificial neural network module 120 shown in Figure 12 has the same basic structure as the roadway prediction artificial neural network module 120 described with reference to Figure 9. However, while the input information for driving video information in Figure 9 is information received from one camera, in Figure 12, first driving video information 10 and second driving video information 50 acquired using two cameras can be input to the roadway prediction artificial neural network module 120. In this case, the first output layer 81 outputs first roadway prediction information 21 and first roadway confidence information 31 learned based on the first driving video information 10, and the second output layer 82 outputs second roadway prediction information 22 and second roadway confidence information 32 learned based on the second driving video information 50, where the second roadway refers to the last lane of the road.

[0081] Furthermore, the road prediction information 20 of the road prediction artificial neural network module 120 according to the embodiment of the present invention may be configured to include not only class information for roads, but also exceptional class information.

[0082] Exception class information is class information classified for points on a road where the lane division is not clear, and refers to points where it is difficult to accurately infer lane information from the driving video information 10 alone. Figure 13 is a diagram showing an example of a point classified as class 0 according to an embodiment of the present invention, and as shown in Figure 13, this includes (a) lane change points, (b) points where obstacles are present, (c) intersection points, (d) lane merging points, (e) lane widening points, and (f) construction points.

[0083] Therefore, the trajectory prediction artificial neural network module 120 according to one embodiment can learn and infer by utilizing the exception class. Specifically, the learning session of the trajectory prediction artificial neural network module 120 performs map learning based on information about an exception class that is labeled separately, and when learning is complete, trajectory information can be output as output information. The exception class may be called the miscellaneous class or the zero (0) class due to its characteristics.

[0084] Locations classified as exceptional are those where it is difficult to accurately extract information about the road surface using only the driving video information 10, and due to the characteristics of the road surface, there are far too many such locations. However, the amount of data available for training is relatively small, so training at each location where the road surface classification is unclear is not only difficult but also inefficient.

[0085] However, in the case of a road path prediction artificial neural network module 120 according to one embodiment, if points where road sections are not clearly defined are classified into separately defined exception classes and then learning is performed based on the remaining classes, learning is performed only on data that has relatively accurate information, which has the advantage of improving the reliability of the road path prediction artificial neural network module 120. A specific embodiment of this will be described with reference to Figures 14 to 16.

[0086] Figures 14 to 16 show a comparison of the output results of the road path prediction artificial neural network module 120 without exception class information and the output results of the neural network module 120 with exception class information.

[0087] Specifically, Figure 14(a) shows the output information of the road prediction artificial neural network module 120 without exception class information at the lane widening point, and Figure 14(b) shows the output information of the road prediction artificial neural network module 120 with exception class information at the lane widening point.

[0088] Referring to Figure 14(a), since the expansion of the new lane occurs at the leftmost point, if the roadway prediction artificial neural network module 120 infers the roadway without including the exception class, the first roadway prediction information 21 and the corresponding first roadway reliability information 31 cannot accurately determine the roadway information, as shown in Figure 14(a).

[0089] However, as shown in Figure 14(b), the road prediction artificial neural network module 120, which includes exception class information, classifies lane widening points into exception classes and infers roads. As a result, as shown in Figure 14(b), the first road prediction information 21 and the first road reliability information 31 are output as a separate class value of 0 at lane widening points. This has the effect of preventing the inferred roads from being confused.

[0090] Figure 15(a) shows the output information of the road prediction artificial neural network module 120 without exception class information at the location of an obstacle, and Figure 15(b) shows the output information of the road prediction artificial neural network module 120 with exception class information at the location of an obstacle.

[0091] Similar to Figure 14, Figures 15 and 16 show that when inferring road paths using exception class information, the accuracy of road path inference decreases at locations where road path divisions cannot be clearly defined. However, for locations where road path divisions are unclear, classifying them into exception classes and then predicting the road path based on the remaining data allows for more accurate inference of road path information.

[0092] Furthermore, in another embodiment of the present invention, the road prediction information 20 of the road prediction artificial neural network module 120 can be configured to include not only class information for roads but also a plurality of exception class information.

[0093] Multiple exception class information can be defined as class information obtained by classifying the above-described exception class information into multiple classes according to certain criteria. For example, lane change points can be classified into the first exception class, lane merging points into the second exception class, obstacle locations into the third exception class, lane widening points into the fourth exception class, intersection points into the fifth exception class, and construction sites into the sixth exception class. After learning is performed for each class, the road prediction information 20 of the road prediction artificial neural network module 120 can be configured to further include the first to sixth exception class information described above. Therefore, the road prediction artificial neural network module 120 according to the embodiment of the present invention not only outputs road information but can also output separate information for locations that are difficult to classify as roads, thus providing more diverse and accurate information when creating maps that include road information or when performing road guidance.

[0094] Figure 17 illustrates a method for extracting road feature vectors from a road path prediction artificial neural network module according to one embodiment, and Figure 18 shows a road path change determination artificial neural network module according to one embodiment.

[0095] The road path prediction artificial neural network module 120 in Figure 17 is identical to the road path prediction artificial neural network module 120 described above, and the feature extraction module 162 is a module that outputs road path feature vectors 16 to the driving video information 10 input to the road path prediction artificial neural network module 120. Specifically, the feature extraction module 162 is a module that outputs road path feature vectors 16 to the input driving video information 10 using a pre-trained image classification artificial neural network including a ConvNet, with vectors output from the fully connected layer 17 and the layer 15 prior to the output layer.

[0096] According to an embodiment of the present invention, the feature extraction module 160 can be configured to output a road feature vector 61 for the input driving video information 10, which is an n-dimensional vector obtained by applying mean pooling (avg_pooling) to the result of extracting features in the last stage of the convolutional layer of a pre-trained image classification artificial neural network. As shown in Figure 18, the road change confirmation artificial neural network module 170 according to an embodiment of the present invention consists of a circulating artificial neural network in the form of an RNN (Recurrent Neural Network) or LSTM (Long-Short Term Memory) that takes road feature vectors as input data (xt) and road change confirmation information (yt) as output data.

[0097] Here, the input data consists of lane feature vectors 16 (xt, xt+1, xt+2...) from the lane change start time (t) after the lane change has been confirmed by the lane prediction artificial neural network module 120. In an inference session of the lane change confirmation artificial neural network module 170 according to one embodiment of the present invention, the lane change confirmation information (yt, yt+1, yt+2...), which is output data output via a hidden layer (ht, ht+1, ht+2...) using the information of the lane feature vectors 16 sequentially input from xt as input data, can be configured to confirm the lane prediction information, which is output information of the lane prediction artificial neural network module 120, as confirmed lane information if the confidence value of the lane change confirmation information 18 is greater than or equal to a specific value.

[0098] In relation to the learning session of the lane change confirmation artificial neural network module 170, a loss function can be constructed that updates the parameters of the lane change confirmation artificial neural network module 170 in a direction that reduces the difference between the output data and the comparison data, by using the lane feature vector 16 as input data, the lane change confirmation information 18 as output data, and the data regarding whether or not the actual lane was changed as comparison data (ground truth).

[0099] For example, if the predicted lane information at time (t-1) before the lane change is for the first lane, and the predicted lane information at the start time of the lane change (t) is changed to the second lane, and the lane change is confirmed, then the lane feature vectors (xt, xt+1, xt+2...) after the start time of the lane change are input as input data to the lane change confirmation artificial neural network module 170.

[0100] Subsequently, the lane change confirmation artificial neural network module 170 sequentially outputs lane change confirmation information along with its confidence level (yt='Lane change, 0.4', yt+1='Lane change, 0.8'...), and at the time when lane change confirmation information with a certain confidence level or higher is output, the lane prediction information output from the lane prediction artificial neural network module 120 is confirmed and output as confirmed lane information ("second lane").

[0101] According to the artificial neural network module 170 for determining lane changes in one embodiment of the present invention, when the lane change process caused by a user's lane change causes the lane prediction information, which is output information from the artificial neural network module 120, to be repeatedly converted and output to different lanes, the effect is that the actual time of the lane change can be determined. As a result, when operating a lane-based navigation service or an autonomous driving service, the time of the lane change can be determined, resulting in the effect of enabling a precise navigation service or a precise autonomous driving service.

[0102] Figure 19 is a block diagram showing some of the components of a navigation device 200 according to one embodiment of the present invention.

[0103] Referring to Figure 19, the navigation device 200 includes a video information acquisition module 210, a road path prediction artificial neural network module 220, an output information distribution calculation module 230, a reference information distribution calculation module 240, a reliability calibration module 250, a feature extraction module 260, a road path change confirmation artificial neural network module 270, and a display module 280. In Figure 19, all components except the display module 280 perform the same functions as those described in Figure 1, so a detailed explanation of them is omitted.

[0104] The display module 280 can not only display the road prediction information 20 and the confidence level information 30 output by the road prediction artificial neural network module 120, but can also display processed and realized information along with a service that guides the vehicle's driving path.

[0105] Therefore, the display module 280 may include various display panels such as a liquid crystal display (LCD) panel, a light-emitting diode (LED) panel, or an organic light-emitting diode (OLED) panel. On the other hand, if the display includes a GUI (Graphical User Interface), i.e., a software-based device such as a touch pad, the display module may also function as an input unit that receives user input.

[0106] According to one embodiment, a vehicle road judgment device using an artificial neural network and a navigation device including the same perform calibration based on a method of updating the parameters of the artificial neural network module based on the results inferred by the artificial neural network module and reference information, thereby producing the effect that the reliability information output from the artificial neural network module approaches the actual probability for road prediction information.

[0107] Furthermore, the vehicle lane determination device for a moving vehicle using an artificial neural network according to one embodiment can provide more reliable predictive information about the road, thus enabling more stable lane changes during autonomous driving.

[0108] Furthermore, a navigation system including a vehicle lane determination device using an artificial neural network according to one embodiment can provide the driver with more accurate guidance in areas where accidents are likely to occur, such as lane changes and turning areas, based on more reliable lane prediction information, thereby enabling the driver to drive more stably.

[0109] On the other hand, the components, units, modules, etc. described herein as "~parts" may be common or individual, but may be individually implemented as interoperable logic devices. The descriptions of different features of modules, units, etc. are intended to highlight different functional embodiments and do not necessarily mean that they must be implemented by individual hardware or software components. Rather, the functions relating to one or more modules or units may be performed by individual hardware or software components, or integrated within common or individual hardware or software components.

[0110] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any form of a programming language, including compiled or interpreted languages, a priori or procedural languages, and can be developed into any form, including independent programs, modules, components, subroutines, or other units appropriate for use in a computer environment.

[0111] Additionally, the logical flow and structural block diagrams described in this patent document describe corresponding functions supported by the disclosed structural means and corresponding actions and / or specific methods supported by the stages, and can also be used to construct corresponding software structures, algorithms and their equivalents.

[0112] The processes and logical flows described herein are executable by one or more programmable processors that perform their functions by operating on input data and generating output, with one or more computer programs running on the input data.

[0113] The description provided presents the best mode of the present invention and provides examples to illustrate the invention and to enable those skilled in the art to manufacture and make the invention available. This specification is not intended to limit the invention to the specific terms presented herein.

[0114] Although preferred embodiments of the present invention have been described above with reference to them, a person skilled in the art or a person with ordinary knowledge of the art will understand that the present invention can be modified and altered in various ways without departing from the spirit and technical domain of the invention as described in the claims below. Therefore, the technical scope of the present invention will not be limited to what is described in the detailed description of the specification, but will be determined by the claims.

Claims

1. A method for determining the roadway a vehicle is traveling on, The operation of acquiring video information of the vehicle in motion, The operation involves inputting the vehicle's driving video information into the first artificial neural network to generate the vehicle's road path prediction information, An operation to detect a change in the vehicle's lane based on the vehicle's predicted lane information, Based on the detection of the vehicle changing lanes, the operation involves generating lane change confirmation information using a second artificial neural network to indicate whether or not the vehicle has changed lanes, Includes, The first artificial neural network is an artificial neural network that has been pre-trained using the vehicle's driving video information as input information and the vehicle's predicted road path information as output information. The method is characterized in that the second artificial neural network is a pre-trained artificial neural network that sequentially receives lane feature vectors extracted chronologically from the vehicle's driving video information after the vehicle's lane change is detected, and outputs the lane change confirmation information.

2. The method according to claim 1, wherein the second artificial neural network includes a recurrent neural network (RNN) or long-short term memory (LSTM).

3. The method according to claim 1, wherein the operation for generating the vehicle's road path prediction information includes inputting the vehicle's driving video information to the first artificial neural network to generate the vehicle's road path prediction information and confidence information for the vehicle's road path prediction information.

4. The method according to claim 1, wherein the operation to generate lane change confirmation information indicating whether or not the vehicle has changed lanes includes the operation to generate the lane change confirmation information and confidence information for the lane change confirmation information using the second artificial neural network.

5. The method according to claim 1, wherein the parameters of the second artificial neural network are updated to reduce the difference between the output data of the second artificial neural network, which is obtained based on processing the driving video data using the second artificial neural network, and the ground truth data corresponding to the driving video data.

6. An electronic device, At least one processor, Memory for storing instructions, Includes, When the instruction is executed by the at least one processor, it causes the electronic device to perform a plurality of operations. The aforementioned multiple operations are, The operation of acquiring vehicle driving video information, The operation involves inputting the vehicle's driving video information into the first artificial neural network to generate the vehicle's road path prediction information, An operation to detect a change in the vehicle's lane based on the vehicle's predicted lane information, Based on the detection of the vehicle changing lanes, the operation involves generating lane change confirmation information using a second artificial neural network to indicate whether or not the vehicle has changed lanes, Includes, The first artificial neural network is an artificial neural network that has been pre-trained using the vehicle's driving video information as input information and the vehicle's predicted road path information as output information. The second artificial neural network is an electronic device that is a pre-trained artificial neural network that sequentially receives lane feature vectors extracted chronologically from the vehicle's driving video information after the vehicle's lane change is detected, and outputs the lane change confirmation information.

Citation Information

Patent Citations

  • Travel compartment line recognition apparatus

    JP2016081361A

  • Danger prediction method

    JP2017162438A

  • Lane information processing device, lane identification system, lane information processing method and lane information processing program

    JP2019194751A

  • Vehicle position determining method and vehicle position determining device

    JP2020064046A