Method and device for determining roadway for moving vehicle using artificial neural network, and navigation device including the same

Abstract

To provide a method for determining a roadway on which a vehicle is traveling. [Solution] According to one embodiment, a roadway judgment device for a moving vehicle using an artificial neural network includes: an image information collection module that acquires driving image information of the vehicle from at least one camera module installed in the vehicle; a pre-trained roadway prediction artificial neural network module that receives the driving image information as input information and outputs roadway prediction information for the vehicle and reliability information for the roadway prediction information; an output information distribution calculation module that calculates a data distribution rate of the output information to generate a first data distribution rate; a reference information distribution calculation module that collects reference information for actual driving road prediction information of the vehicle, calculates a data distribution rate of the reference information, and generates a second data distribution rate; and a reliability calibration module that updates parameters of the artificial neural network module based on the second data distribution rate so as to reduce a difference between the first data distribution rate and the second data distribution rate.

Description

[Technical Field]

[0001] The present invention relates to a method and device for determining the lane of a moving vehicle using an artificial neural network, and a navigation device including the same. More specifically, the present invention relates to a technology that uses an artificial neural network based on information about a moving vehicle's driving image to provide lane prediction information on which lane a currently moving vehicle is driving on and reliability information for the lane prediction information. [Background technology]

[0002] Generally, a navigation device used in a vehicle searches for a route to the destination using map information when a user inputs a destination, and guides the vehicle along the route selected by the user. The navigation device also provides various visual or audio information, such as the route to the destination, features located around the route, and road congestion levels, to guide the user to the destination.

[0003] In order to provide accurate driving information to a vehicle driver using a navigation device, it is necessary to know the current vehicle's driving position, specifically, the lane in which the vehicle is driving, in order to provide accurate guidance to the vehicle driver. In particular, when a turning point, such as an intersection or an IC (Interchange) where a general road meets an expressway, or a JC (Junction) where two expressways meet, is included in the driving route, it is possible to prevent accidents caused by sudden lane changes by providing the vehicle driver with information on whether to stay in the lane or change lanes at an appropriate time.

[0004] However, in the case of a navigation device according to the prior art, since the current vehicle position is determined based on GPS, it is difficult to accurately obtain predicted road information for the vehicle currently traveling. Therefore, the vehicle guidance using the navigation device is merely a guide for driving information.

[0005] In addition, to ensure stable autonomous driving, accurate information about the road the vehicle is currently traveling on is necessary to ensure stable lane changes, intersections, etc. Currently, there is no technology that can accurately determine information about the road the vehicle is currently traveling on, so the stability of autonomous driving cannot be ensured.

[0006] In addition, to create a map for autonomous driving, the map must include information about how many lanes each road has in total, and which lane out of the total lanes the vehicle currently traveling in is in. However, with conventional technology, it was not possible to automatically extract information about lanes from a road image, and the creator had to manually label the lanes. Summary of the Invention [Problem to be solved by the invention]

[0007] Therefore, the method and device for determining the roadway of a moving vehicle using an artificial neural network and the navigation device including the same according to one embodiment are inventions devised to solve the above-described problems, and their purpose is to provide more accurate roadway prediction information for a currently moving vehicle using an artificial neural network.

[0008] More specifically, the purpose is to predict the roadway using an artificial neural network module based on information about the input vehicle driving image, and then calibrate the artificial neural network module based on the prediction information and reference information, thereby adjusting the reliability of the actual artificial neural network model so that it is similar to the actual result, thereby increasing the reliability of the artificial neural network module that performs roadway prediction.

[0009] Furthermore, by providing information on the vehicle currently in motion and its reliability information, the driver of the vehicle is provided with more accurate information on the current road, and from the perspective of the program developer, the purpose is to provide various guidance services for the navigation device by utilizing road prediction information and its reliability information.

[0010] In addition, one embodiment of a roadway judgment device for a moving vehicle using an artificial neural network has the purpose of automatically extracting information about lanes based on driving images and easily creating a map containing lane information. [Means for solving the problem]

[0011] According to one embodiment, a roadway judgment device for a moving vehicle using an artificial neural network may include: an image information collection module that acquires driving image information of the vehicle from at least one camera module installed in the vehicle; a pre-trained roadway prediction artificial neural network module that receives the driving image information as input information and outputs roadway prediction information for the vehicle and reliability information for the roadway prediction information; an output information distribution calculation module that calculates a data distribution rate of the output information to generate a first data distribution rate; a reference information distribution calculation module that collects reference information for actual driving road prediction information of the vehicle and calculates a data distribution rate of the reference information to generate a second data distribution rate; and a reliability calibration module that updates parameters of the artificial neural network module based on the second data distribution rate so as to reduce the difference between the first data distribution rate and the second data distribution rate.

[0012] The parameters of the artificial neural network module are updated using a method of classifying a logit vector into a single scalar parameter for all classes of the artificial neural network module.

[0013] The confidence calibration module can be trained on the single scalar parameter using a calibration validation set that is the same as the validation set of the artificial neural network module.

[0014] The reliability calibration module can perform training on the single scalar parameter so that a negative log likelihood (NLL) value in the calibration validation set is minimized.

[0015] The roadway prediction information may include roadway information regarding the lane number of the roadway on which the vehicle is currently traveling, based on the first lane or the last lane of the road on which the vehicle is traveling.

[0016] The artificial neural network module includes a first output layer and a second output layer, and the first output layer outputs first lane information, which is information regarding the lane number of the lane on which the vehicle is currently traveling based on the first lane of the road on which the vehicle is traveling, and a reliability value for the first lane information. The second output layer outputs second lane information, which is information regarding the lane number of the lane on which the vehicle is currently traveling based on the last lane of the road on which the vehicle is traveling, and a reliability value for the second lane information.

[0017] According to another embodiment, a method for determining a roadway for a moving vehicle using an artificial neural network uses a pre-trained roadway prediction artificial neural network module, and inputs driving video information, and outputs vehicle roadway prediction information and reliability information for the roadway prediction information. The method may include: acquiring video information of the vehicle from at least one camera module installed on the vehicle; calculating an output information distribution of the output information to generate a first data distribution; collecting reference information for prediction information of the actual driving road of the vehicle, calculating a data distribution of the reference information, and generating a second data distribution; and updating parameters of the artificial neural network module based on the second data distribution so that a difference between the first data distribution and the second data distribution decreases.

[0018] A navigation device according to one embodiment may include: an image information collection module that acquires driving image information of the vehicle from at least one camera module installed in the vehicle; a pre-trained road prediction artificial neural network module that receives the driving image information as input information and outputs road prediction information for the vehicle and reliability information for the road prediction information; an output information distribution calculation module that calculates a data distribution rate of the output information to generate a first data distribution rate; a reference information distribution calculation module that collects reference information for prediction information of the actual driving road of the vehicle, calculates a data distribution rate of the reference information, and generates a second data distribution rate; a reliability calibration module that updates parameters of the artificial neural network module based on the second data distribution rate so that a difference between the first data distribution rate and the second data distribution rate is reduced; and a display module that externally displays the output information output from the artificial neural network module and the driving image of the vehicle together. [Effects of the Invention]

[0019] In one embodiment, a method and device for determining a roadway for a moving vehicle using an artificial neural network and a navigation device including the same perform calibration by updating the parameters of the artificial neural network module based on the results of inference by the artificial neural network module and reference information, thereby achieving the effect that the reliability information output from the artificial neural network module approaches the actual probability of the roadway prediction information.

[0020] In addition, the method and device for determining the roadway of a moving vehicle using an artificial neural network according to one embodiment, and the navigation device including the same, can provide more reliable roadway prediction information, thereby enabling the vehicle to change lanes more stably during autonomous driving.

[0021] In addition, a navigation device including a roadway judgment device for a moving vehicle using an artificial neural network according to one embodiment can provide the driver with more accurate guidance in areas where accidents are more likely to occur, such as when changing lanes or entering turning areas, based on more reliable roadway prediction information, thereby enabling the driver to drive stably.

[0022] In addition, the method and apparatus for determining a roadway for a moving vehicle using an artificial neural network according to one embodiment can accurately extract lane information of the road on which the vehicle is traveling based on an image of the vehicle traveling, which has the effect of making it possible to easily create a map including lane information without manually inputting lane information when creating the map. [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 1 is a block diagram showing some components of a roadway judgment device using an artificial neural network according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating input information input to and output information output from a roadway prediction artificial neural network module according to an embodiment. [Figure 3] FIG. 1 is a diagram showing the structure of an artificial neural network module applied in the present invention. [Figure 4]FIG. 1 is a schematic diagram showing the calculation process at a node of an artificial neural network module. [Figure 5] 1 is a diagram illustrating a process in which input information input to a CNN is converted into output information according to an embodiment. FIG. [Figure 6] FIG. 2 illustrates an artificial neural network module and the relationships between its components, according to one embodiment. [Figure 7] FIG. 10A is a diagram showing program code for a general softmax function, and FIG. 10B is a diagram showing code for calibrating a softmax function according to an embodiment. [Figure 8] This is a diagram showing a comparison of the results calculated when the program is executed using the code in FIG. 7(b). [Figure 9] FIG. 10 is a diagram illustrating a further embodiment of the output layer of the present invention. [Figure 10] FIG. 10 illustrates how information output by an artificial neural network may be displayed on a display module according to one embodiment. [Figure 11] 10 is a diagram illustrating a method for calculating final roadway prediction information based on roadway prediction information output by an artificial neural network module and its reliability information according to another embodiment. FIG. [Figure 12] FIG. 10 illustrates input and output information input to an artificial neural network module according to a further embodiment. [Figure 13] 10A and 10B are diagrams illustrating an example of a point where lane division is unclear according to an embodiment of the present invention; [Figure 14] FIG. 10 is a diagram showing a comparison of the results output from the roadway prediction artificial neural network module when there is and when there is exception class information. [Figure 15] FIG. 10 is a diagram showing a comparison of the results output from the roadway prediction artificial neural network module when there is and when there is exception class information. [Figure 16] FIG. 10 is a diagram showing a comparison of the results output from the roadway prediction artificial neural network module when there is and when there is exception class information. [Figure 17]FIG. 10 is a diagram illustrating a method for extracting a road feature vector in a road prediction artificial neural network module according to an embodiment. [Figure 18] FIG. 1 illustrates a lane change decision artificial neural network module according to one embodiment. [Figure 19] FIG. 2 is a block diagram illustrating some components of a navigation device according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0024] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. When assigning reference numerals to components in each drawing, it should be noted that the same components are designated by the same numerals whenever possible, even if they are displayed in different drawings. Furthermore, in describing embodiments of the present invention, if a detailed description of related disclosed configurations or functions is deemed to hinder understanding of the embodiments of the present invention, the detailed description thereof will be omitted. Furthermore, although embodiments of the present invention will be described below, the technical concept of the present invention is not limited thereto, and may be modified and implemented in various ways by those skilled in the art.

[0025] Furthermore, the terms used in this specification are used to describe the embodiments and are not intended to limit and / or restrict the disclosed invention. A singular expression includes a plural expression unless the context clearly indicates otherwise.

[0026] In this specification, the terms "comprises," "includes," "has," and the like are intended to specify the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but do not preclude the possible presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0027] Furthermore, throughout the specification, when any part is "connected" to another part, this includes not only "directly connected" but also "indirectly connected" with an additional element interposed therebetween, and terms including ordinal numbers such as "first" and "second" used in this specification may be used to describe various components, but the above components are not limited to the above terms.

[0028] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the invention, and portions not related to the description will be omitted in order to clearly explain the present invention.

[0029] Fig. 1 is a block diagram showing some components of a roadway judgment device 100 using an artificial neural network according to one embodiment of the present invention, and Fig. 2 is a diagram showing input information input to and output information output from a roadway prediction artificial neural network module 120 according to one embodiment. For ease of explanation, the roadway judgment device 100 using an artificial neural network will be referred to as roadway judgment device 100 below.

[0030] Referring to FIG. 1, the roadway judgment device 100 includes an image information collection module 110, a roadway prediction artificial neural network module 120, an output information distribution calculation module 130, a reference information distribution calculation module 140, a reliability calibration module 150, a feature extraction module 160, and a lane change confirmation artificial neural network module 170.

[0031] The image information collection module 110 collects driving image information of the front and side of the vehicle captured through at least one camera mounted on the vehicle, and transmits the collected images to the road prediction artificial neural network module 120.

[0032] The roadway prediction artificial neural network module 120 is an artificial neural network module that receives as input information the vehicle driving video information 10 collected by the video information collection module 110, and outputs as output information roadway prediction information 20 inferred from which lane the current vehicle is traveling in, and reliability information 30 for the inferred roadway. The roadway prediction artificial neural network module 120 includes a learning session 121 that performs learning based on the input information and output information, and an inference session 122 that infers output information based on the input information.

[0033] The learning session 121 of the roadway prediction artificial neural network module 120 is a session that learns based on the input information 10 and the output information 30, 40, and the inference session 122 uses the learned roadway prediction artificial neural network module 120 to analyze the driving video information 10 input in real time, and can output roadway prediction information 20 that infers which road the vehicle is currently traveling on, along with reliability information 30 for this information.

[0034] For example, when an image is input to the roadway prediction artificial neural network module 120, the input image is analyzed to infer the roadway for the vehicle currently traveling. If it is determined that the traveling road is road 1, road 1 is output as roadway prediction information 20, and at the same time, reliability information 30 (e.g., 0.8) for determining that it is road 1 can be output as output information.

[0035] In addition, the roadway prediction information 20 output from the roadway prediction artificial neural network module 120 may be output based on information regarding the number of a lane relative to a specific lane. For example, roadway information regarding the number of a lane on which the vehicle is currently traveling may be output based on the first lane of the road, or conversely, roadway information regarding the number of a lane on which the vehicle is currently traveling may be output based on the last lane of the road.

[0036] When providing roadway information, if information is provided as to the number of a lane relative to a specific lane as in the present invention, the driver can understand the number of the lane relative to the reference lane closest to the vehicle, which has the effect of enabling the driver to more stably prepare for and change lanes along the driving route.

[0037] FIG. 3 is a diagram showing the structure of an artificial neural network module applied to the present invention, and FIG. 4 is a schematic diagram showing the calculation process in a node of the artificial neural network module.

[0038] Referring to FIG. 3, the multi-layer neural network model of the road prediction artificial neural network module 120 is composed of an input layer 60, a hidden layer 70, and an output layer 80.

[0039] The input layer 60 is composed of nodes corresponding to each input factor, and the number of nodes is the same as the number of input factors. The hidden layer 70 processes the linear combination of factor values ​​transmitted from the input layer 60 into a nonlinear function such as a sigmoid function and transmits the result to the output layer or another hidden layer. Due to space limitations, FIG. 3 illustrates the roadway prediction artificial neural network module 120 as having one hidden layer 70. However, this is not limited to this, and the roadway prediction artificial neural network module 120 may include multiple hidden layers 70. The output layer 80 has nodes corresponding to the output factors, and output nodes are generated as many times as the number of classes in the classification model. In the present invention, the output information is composed of roadway prediction information 20 indicating the lane number of the lane the vehicle is currently traveling on and its corresponding reliability information 30. The output layer 80 is illustrated as a single layer as shown in FIG. 3. However, the present invention is not limited to this, and multiple output layers 80 may be configured depending on the environment in which the roadway prediction artificial neural network module 120 is used.

[0040] As shown in Figure 4, when input information 10 is input to the roadway prediction artificial neural network module 120, actual calculations are performed at each node. The calculation process of the roadway prediction artificial neural network module 120 is mathematically designed to replicate the process that occurs in the neurons that make up the human neural network.

[0041] A node responds when it receives a stimulus above a certain magnitude, and the magnitude of the response, excluding bias, is roughly proportional to the product of the input value and the node's parameters (or weights). Generally, a node receives several inputs and has as many parameters as there are inputs. Therefore, by adjusting these parameters, different weights can be assigned to various inputs. Finally, all multiplied values ​​are added, and the sum is input to the activation function of the output layer 80. The result of the activation function corresponds to the node's output, and this output value is ultimately used in classification and regression analysis.

[0042] Each layer of a neural network model consists of at least one node, and the activation / deactivation of each node is determined by the input value. Input information becomes the input for the first layer (input layer), and the output of each layer then becomes the input for the next layer. All coefficients change slightly during the learning process, ultimately reflecting what inputs each node values. The "training" of the artificial neural network model mentioned above refers to the process of updating these coefficients.

[0043] Hereinafter, the structure of a CNN network that is typically applied to the roadway prediction artificial neural network module 120 of the present invention will be described with reference to Fig. 5. Fig. 5 is a diagram illustrating a process in which input information input to a CNN is converted into output information according to an embodiment of the present invention.

[0044] A convolutional neural network (CNN) is a type of feed-forward deep neural network (DNN) used to analyze visual images. It is a neural network consisting of one or several convolutional layers, a pooling layer, and a fully connected layer. In Korean, it is called a convolutional neural network. CNN has a structure suitable for learning two-dimensional data and can be trained via the backpropagation algorithm.

[0045] The CNN process involves repeating feature extraction, influence minimization, and classification to obtain global features that represent the entire image from local features.

[0046] 5, when an image collected by an image collection module 110 is input as input information 10 to a roadway prediction artificial neural network module 120, the roadway prediction artificial neural network module 120 applies a filter to the input image 11 to generate a first feature map 12. Applying a filter means performing a convolution operation on an image using a filter having specific coefficients.

[0047] The road prediction artificial neural network module 120 generates a first feature map 12 using a specific filter. Once the first feature map 12 is generated, it applies subsampling to the first feature map 12 to reduce the image size. A second feature map 13 having local features is generated through the subsampling process. The road prediction artificial neural network module 120 then applies a filter having a different feature to the second feature map 13, and repeatedly performs subsampling. Repeating this process produces a final feature map 14 having global features. The final feature map 14 thus obtained is connected as an input to the fully connected network 17, and output information can be generated via the output layer. In the present invention, the road prediction artificial neural network module 120 generates a final feature map for the input image 11 using the above-described method. The final feature map is then connected to the input of the fully connected network 17, and the final output information, road prediction information 20 and its corresponding reliability information 30, can be output via the activation function of the output layer 80.

[0048] As mentioned above, in the case of CNN, the number of variables is dramatically reduced by repeatedly applying a filter having the same coefficients to the entire image, which has the effect of obtaining invariance that is unrelated to topology changes. This effect can improve the accuracy of determining a roadway based on an input image, as in the present invention.

[0049] In addition, although the present specification has described the neural network applied to the roadway prediction artificial neural network module 120 of the present invention based on CNN, the neural network structure applied to the roadway prediction artificial neural network module 120 of the present invention is not limited to CNN, and various publicly known artificial neural network models useful for image detection, such as Google Mobile Net v2, VGGNet16, and ResNet50, may be applied.

[0050] FIG. 6 is a diagram illustrating the relationship between an artificial neural network module and each component according to an embodiment of the present invention, specifically illustrating the process by which the parameters of the artificial neural network module are updated through learning.

[0051] Referring to FIG. 6, the road prediction information 20 outputted by the road prediction artificial neural network module 120 and its corresponding reliability information 30 can be inputted from the output information distribution calculation module 130.

[0052] The output information distribution calculation module 130 is a module that specifically calculates the data distribution degree for the road prediction information 20 output from the road prediction artificial neural network module 120 and the reliability information 30 thereof.

[0053] For example, when the road prediction artificial neural network module 120 outputs road prediction information 20 indicating which road a vehicle is traveling on based on 100 input images and reliability information 30 therefor through inference, the output information distribution calculation module 130 receives the road prediction information 20 and reliability information 30 output by the road prediction artificial neural network module 120, analyzes the distribution of the information based on the received information, and generates a first data distribution degree (X). The first data distribution degree (X) means information regarding the distribution of values ​​of the output information output by inferring each image input to the road prediction artificial neural network module 120.

[0054] The reference information distribution calculation module 140 may collect reference information for the actual driving road prediction information of the vehicle, calculate the data distribution of the reference information, and generate a second data distribution (Y).

[0055] The reference information is called ground truth information, and serves as a basis for updating the parameters of the artificial neural network module. In the present invention, the reference information may include actual road information for the image input to the road prediction artificial neural network module 120.

[0056] The reliability calibration module 150 may perform calibration to change various parameter values ​​of the artificial neural network model so that the output value of the artificial neural network model reflects the actual reliability. For example, if the model output for input information A and output information B is 0.9, calibration can be performed to change various parameter values ​​of the artificial neural network model so that there is a 90% chance that it is B.

[0057] In calibrating the model of the road prediction artificial neural network module 120, the reliability calibration module 150 receives a first data distribution degree (X) from the output information distribution calculation module 130 and a second data distribution degree (Y) from the reference information distribution calculation module 140, and can perform calibration based on a method of updating the parameters of the road prediction artificial neural network module 120 so that the difference between the first data distribution degree (X) and the second data distribution degree (Y) is reduced based on the second data distribution degree (Y).

[0058] To explain one embodiment relating to lane prediction, for 100 input images, if the lane prediction information inferred by the lane prediction artificial neural network module 120 is such that the ratio of predicted lane 1 is 20%, the ratio of predicted lane 2 is 50%, and the ratio of predicted lane 3 is 30%, and if the reference information is such that the ratio of predicted lane 1 is 30%, the ratio of predicted lane 2 is 45%, and the ratio of predicted lane 3 is 25%, the reliability calibration module 150 can update the parameters of the lane prediction artificial neural network module 120 to match the actual reference information ratios.

[0059] This has the effect that the reliability information 30 output from the roadway prediction artificial neural network module 120 approaches the actual probability for the roadway prediction information 20 .

[0060] In addition, as one embodiment, the reliability calibration module 150 may perform calibration as a method of softening a softmax function corresponding to the output layer. Specifically, in the case of a multiple classification problem with K labels, calibration can be performed by converting a logit vector Z into the following equation (1) using a single scalar parameter T.

[0061] Formula (1)

number

[0062] q^ stands for calibrated probability, and the above formula means that calibration is performed by dividing the logit vector into T, a single scalar parameter, in a multi-classification problem with K labels.

[0063] In equation (3), when the single scalar parameter T is 1, it becomes the original formula for calculating the reliability using softmax. As T gets larger, the final q converges to 1 / K, and as T gets closer to 0, q gets closer to 1.

[0064] Furthermore, the optimal value of the single scalar parameter T is determined by learning. Specifically, learning can be performed on the single scalar parameter T using a calibration validation set, which is an additional validation set that has the same characteristics as the validation set of the lane prediction artificial neural network module 120.

[0065] Specifically, the reliability calibration module 150 may use a calibration validation set to train a single scalar parameter T so that the NLL (Negative Log Likelihood) value in the calibration validation set is minimized. When performing calibration in this manner, the single scalar parameter T can be calibrated without changing the maximum value of the softmax output. Therefore, this method has the advantage of being applicable to conventionally trained artificial neural network modules, since it only affects the calibration of the conventional model and does not affect its accuracy.

[0066] 7 and 8 are diagrams for explaining the effect of calibration performed by a reliability calibration module. FIG. 7(a) is program code for a general softmax function, and FIG. 7(b) is code that has been calibrated to the softmax function in advance. FIG. 8 is a graph showing the probability (P) values ​​calculated when applying T=0 (FIG. 8(a)), T=2 (FIG. 8(b)), and T=10 (FIG. 8(c)) to a single scalar parameter T according to FIG. 7(b).

[0067] As T is increased sequentially from 0, it is confirmed that the difference between each probability value decreases as shown in Figure 10. However, the order can be changed without affecting the accuracy. Therefore, when calibration is performed using the reliability calibration module of the present invention, it has the effect of improving the reliability of the road prediction artificial neural network module 120 without affecting the accuracy.

[0068] FIG. 9 is a diagram illustrating a further embodiment of the output layer of the present invention, showing how the information output by the artificial neural network of FIG. 10 is displayed on the display module of a navigation device.

[0069] Referring to FIG. 9, the input information input to the road prediction artificial neural network module 120 of the present invention is the same as that in FIG. 3, i.e., driving image information 10, but the output layer does not consist of a single layer but includes a first output layer 81 and a second output layer 82.

[0070] The first output layer 81 and the second output layer 82 act as activation functions, and typically, a softmax function or a cross entropy function may be used. For convenience of explanation, the following description will be based on the softmax function.

[0071] As shown in FIG. 9, when the lane prediction artificial neural network module 120 of the present invention uses a first softmax function and a second softmax function in the output layer 80, the first softmax function outputs first lane prediction information 21 regarding the lane number on which the currently traveling vehicle is traveling based on the first lane, and corresponding first lane reliability information 31, and the second softmax function outputs second lane prediction information 22 regarding the lane number on which the currently traveling vehicle is traveling based on the last lane, and corresponding second lane reliability information 32.

[0072] Therefore, as shown in FIG. 10, the output information of the first softmax function is the first lane prediction information 21, which is determined to be the second lane based on the first lane, and the corresponding first lane reliability information 31, which are output as 2 lanes and 0.999, respectively, and the output information of the second softmax function is the second lane prediction information 22, which is determined to be the second lane based on the last lane, and the corresponding reliability information 32, which are output as 2 lanes and 0.994, respectively.

[0073] Generally, there is no major problem if the front camera installed on a vehicle can capture all lanes of a road, but there are cases where the front camera does not capture all lanes depending on the camera's capabilities. Therefore, in such cases, using a single softmax function cannot extract information about the number of lanes in all lanes, making it difficult to provide accurate road information. However, classification using two softmax functions on a single network, as in the present invention, provides information about all lanes, thereby providing more accurate road information.

[0074] Furthermore, when lane information for a road can be extracted from a driving image as in the present invention, it is possible to easily create a map that includes lane information. That is, in the conventional technology, lane information cannot be automatically extracted from a road image, and a creator must manually label the lane information. However, the method and apparatus for determining a roadway for a driving vehicle using an artificial neural network according to an embodiment of the present invention can automatically extract lane information based on a driving image as described above, thereby making it possible to easily create a map that includes lane information. Therefore, it is possible to more accurately create maps for autonomous driving, which require accurate lane information, than in the conventional technology.

[0075] FIG. 11 is a diagram illustrating a method for calculating final roadway prediction information based on the roadway prediction information output by the roadway prediction artificial neural network module and its reliability information according to a further embodiment of the present invention.

[0076] Specifically, the blue line in Figure 11(a) is a diagram that displays the predicted roadway information predicted according to time in real time, and Figure 11(b) is a diagram that outputs reliability information for the predicted roadway information according to time.

[0077] Referring to Fig. 11, from time 0 to time t1, the reliability of the roadway prediction information is 1, and the roadway prediction information is output as road 2. Since the reliability is 1, there is almost no change in the blue line of the roadway shown in Fig. 11A.

[0078] However, as shown in FIG. 11B, there is a significant change in the reliability information between t1 and t2. This is due to a lane change. Therefore, the blue line shown in FIG. 11A also changes continuously from lane 2 to lane 1. In such a case, the present invention removes values ​​below a certain reliability level and selects only meaningful reliability values ​​to derive the green line shown in FIG. 11(a), and then derives the red line through a final correction process, thereby providing a corresponding result. If reliability information were not available, the process of correcting the result value for the reliability that changes in real time and calculating the final data would have to be performed manually. However, the present invention uses reliability information to automatically derive a result such as the red line based on a preset standard.

[0079] FIG. 12 is a diagram illustrating input and output information input to an artificial neural network module according to a further embodiment.

[0080] The roadway prediction artificial neural network module 120 shown in Figure 12 has the same basic structure as the roadway prediction artificial neural network module 120 described with reference to Figure 9, but while the input information is driving video information received from a single camera in Figure 9, in Figure 12, first driving video information 10 and second driving video information 50 acquired using two cameras can be input to the roadway prediction artificial neural network module 120 as input information. In this case, the first output layer 81 outputs first road prediction information 21 and first road reliability information 31 learned based on the first driving video information 10, and the second output layer 82 outputs second road prediction information 22 and second road reliability information 32 learned based on the second driving video information 50, where the second road refers to the last lane of the road.

[0081] Furthermore, the road prediction information 20 of the road prediction artificial neural network module 120 according to the embodiment of the present invention may be configured to include not only class information about roads but also exceptional class information.

[0082] Exception class information is class information classified for points on a road where lane divisions are unclear, and refers to points where it is difficult to accurately infer information about the lane using only the driving video information 10. Figure 13 is a diagram showing examples of points classified into class 0 according to an embodiment of the present invention, and as shown in Figure 13, these include (a) lane change points, (b) obstacle points, (c) intersection points, (d) lane merging points, (e) lane widening points, and (f) construction points.

[0083] Therefore, the roadway prediction artificial neural network module 120 according to an embodiment can learn and perform inference using the exception classes. Specifically, the learning session of the roadway prediction artificial neural network module 120 performs map learning based on information about the separately labeled exception classes, and when learning is complete, road information can be output as output information. The exception classes may be referred to as miscellaneous classes or zero (0) classes due to their characteristics.

[0084] The points classified as exceptional are points where it is difficult to accurately extract information about the lane only from the driving video information 10, and there are an extremely large number of such points due to the characteristics of the lane. However, since the amount of data available for learning is relatively small, it is not only difficult but also inefficient to learn for each point where the lane division is unclear.

[0085] However, if points with unclear lane divisions are classified into separately defined exception classes and learning is performed based on the remaining classes, as in the lane prediction artificial neural network module 120 according to an embodiment, learning is performed based only on data having relatively accurate information, which has the advantage of improving the reliability of the lane prediction artificial neural network module 120. Specific embodiments of this will be described with reference to FIGS.

[0086] 14 to 16 are diagrams showing a comparison of the output results of the road prediction artificial neural network module 120 that does not include exception class information and the output results of the neural network module 120 that includes exception class information.

[0087] Specifically, Figure 14(a) is a diagram showing the output information of the lane prediction artificial neural network module 120 that does not include exception class information at lane widening points, and Figure 14(b) is a diagram showing the output information of the lane prediction artificial neural network module 120 that includes exception class information at lane widening points.

[0088] Referring to FIG. 14(a), since the new lane expansion occurs at the leftmost point, if the lane prediction artificial neural network module 120 infers the lane without including the exception class, the first lane prediction information 21 and the corresponding first lane reliability information 31 cannot accurately determine the lane information, as shown in FIG. 14(a).

[0089] However, as shown in Figure 14(b), the lane prediction artificial neural network module 120, which includes exception class information, classifies lane widening points into exception classes and infers lanes, so that at lane widening points, the first lane prediction information 21 and the first lane reliability information 31 are output as a separate class value of 0, as shown in Figure 14(b). This has the effect of preventing confusion in the inferred lanes.

[0090] Figure 15(a) is a diagram showing the output information of the lane prediction artificial neural network module 120 that does not include exception class information at obstacle locations, and Figure 15(b) is a diagram showing the output information of the lane prediction artificial neural network module 120 that includes exception class information at obstacle locations. Figure 15(a) is a diagram showing the output information of the lane prediction artificial neural network module 120 that does not include exception class information at intersection locations, and Figure 15(b) is a diagram showing the output information of the lane prediction artificial neural network module 120 that includes exception class information at obstacle locations.

[0091] As with FIG. 14, in FIGS. 15 and 16, when inferring a lane using exception class information, the accuracy of lane inference decreases at points where lane divisions are unclear. However, there is an advantage in that, after classifying points where lane divisions are unclear into exception classes, lane information can be inferred more accurately by predicting the lane based on the remaining data.

[0092] In another embodiment of the present invention, the road prediction information 20 of the road prediction artificial neural network module 120 can be configured to include not only class information for the road but also information on a plurality of exception classes.

[0093] The plurality of exception class information may be class information obtained by classifying the above-described exception class information according to certain criteria, such as classifying lane change points into a first exception class, lane merging points into a second exception class, obstacle points into a third exception class, lane widening points into a fourth exception class, intersection points into a fifth exception class, and construction points into a sixth exception class, and then performing learning for each class, so that the lane prediction information 20 of the lane prediction artificial neural network module 120 further includes the above-described first through sixth exception class information. Therefore, the lane prediction artificial neural network module 120 according to the embodiment of the present invention can not only output lane information but also output separate information for points that are difficult to classify as lanes, thereby providing more diverse and accurate information when creating maps including lane information or performing road guidance.

[0094] FIG. 17 illustrates how a lane feature vector is extracted from a lane prediction artificial neural network module according to one embodiment, and FIG. 18 illustrates a lane change determination artificial neural network module according to one embodiment.

[0095] The roadway prediction artificial neural network module 120 in Figure 17 is the same as the roadway prediction artificial neural network module 120 described above, and the feature extraction module 160 is a module that outputs a roadway feature vector 16 for the driving video information 10 input to the roadway prediction artificial neural network module 120. Specifically, the feature extraction module 162 is a module that outputs vectors output from a fully connected layer 17 and a layer 15 before the output layer in a pre-trained image classification artificial neural network including a ConvNet as the roadway feature vector 16 for the input driving video information 10.

[0096] According to an embodiment of the present invention, the feature extraction module 160 can be configured to output an n-dimensional vector obtained by applying average value pooling (avg_pooling) to the feature extraction results at the last stage of the convolutional layer of a pre-trained image classification artificial neural network, as a roadway feature vector 61 for the input driving video information 10. As shown in FIG. 18, the lane change determination artificial neural network module 170 according to an embodiment of the present invention is configured as a recurrent neural network (RNN) or long-short term memory (LSTM) type recurrent artificial neural network that receives a lane feature vector as input data (xt) and outputs lane change determination information (yt).

[0097] Here, the input data consists of lane feature vectors 16 (xt, xt+1, xt+2...) after the lane change start time (t), which is after the lane change is confirmed by the lane prediction artificial neural network module 120. In an inference session of the lane change confirmation artificial neural network module 170 according to one embodiment of the present invention, information on lane feature vectors 16 sequentially input from xt is used as input data, and lane change confirmation information (yt, yt+1, yt+2...) is output through hidden layers (ht, ht+1, ht+2...). When the confidence value of the lane change confirmation information 18 is equal to or greater than a specific value, the lane prediction information, which is output information from the lane prediction artificial neural network module 120, is determined as confirmed lane information.

[0098] In connection with a learning session of the lane change determination artificial neural network module 170, the lane feature vector 16 can be configured as input data, the lane change determination information 18 can be configured as output data, and data on whether or not an actual lane change has occurred can be configured as comparison data (ground truth), and a loss function can be configured to update the parameters of the lane change determination artificial neural network module 170 in a direction that reduces the difference between the output data and the comparison data.

[0099] For example, if the lane prediction information at the time (t-1) before the lane change is the first lane, and the lane prediction information at the lane change start time (t) is changed to the second lane and the lane change is confirmed, the lane feature vector (xt, xt+1, xt+2...) after the lane change start time is input as input data to the lane change determination artificial neural network module 170.

[0100] Thereafter, the lane change confirmation artificial neural network module 170 sequentially outputs lane change confirmation information along with its confidence (yt='Lane change, 0.4', yt+1='Lane change, 0.8'...), and when lane change confirmation information with a certain confidence level or higher is output, the lane prediction information output from the lane prediction artificial neural network module 120 is confirmed as confirmed lane information ('second lane') and output.

[0101] According to the lane change determination artificial neural network module 170 of an embodiment of the present invention, when the lane prediction information, which is the output information of the lane prediction artificial neural network module 120, is repeatedly converted to different lanes and output during the process of changing lanes due to a lane change by a user, the actual time point of lane change can be determined. As a result, when operating a road-based navigation service or an autonomous driving service, the time point of lane change can be determined, thereby enabling a precise navigation service or a precise autonomous driving service.

[0102] FIG. 19 is a block diagram showing some components of a navigation device 200 according to one embodiment of the present invention.

[0103] 19, a navigation device 200 includes an image information collection module 210, a roadway prediction artificial neural network module 220, an output information distribution calculation module 230, a reference information distribution calculation module 240, a reliability calibration module 250, a feature extraction module 260, a roadway change determination artificial neural network module 270, and a display module 280. In FIG. 19, the components other than the display module 280 perform the same functions as the respective components described with reference to FIG. 1, and therefore detailed description thereof will be omitted.

[0104] The display module 280 can not only display the roadway prediction information 20 and its reliability information 30 output by the roadway prediction artificial neural network module 120, but also process the output information and display the implemented information together with a service that guides the vehicle's driving route.

[0105] Therefore, the display module 280 may include various display panels such as a liquid crystal display (LCD) panel, a light emitting diode (LED) panel, an organic light emitting diode (OLED) panel, etc. Meanwhile, if the display includes a graphical user interface (GUI) such as a touch pad, i.e., a software device, the display may also function as an input unit that receives user input.

[0106] In one embodiment, a roadway judgment device for a moving vehicle using an artificial neural network and a navigation device including the same perform calibration based on a method of updating parameters of the artificial neural network module based on the results inferred by the artificial neural network module and reference information, thereby achieving the effect of making the reliability information output from the artificial neural network module closer to the actual probability of roadway prediction information.

[0107] In addition, the roadway judgment device for a moving vehicle using an artificial neural network according to one embodiment can provide more reliable prediction information for roadways, which has the effect of enabling the vehicle to change lanes more stably during autonomous driving.

[0108] In addition, a navigation system including a roadway judgment device for a moving vehicle using an artificial neural network according to one embodiment can provide more accurate guidance to the driver in areas where accidents are likely to occur, such as when changing lanes or entering a turning area, based on more reliable roadway prediction information, thereby enabling the driver to drive stably.

[0109] However, components, units, modules, elements, etc. described herein as "modules" may be shared or separate, but may be implemented separately as interoperable logic devices. Depictions of different characteristics for modules, units, etc. are intended to highlight different functional embodiments and do not necessarily imply that they must be implemented by separate hardware or software components. Rather, functionality associated with one or more modules or units may be performed by separate hardware or software components or integrated within shared or separate hardware or software components.

[0110] A computer program (also known as a program, software, software application, script or code) may be written in any form of programming language, including compiled or interpreted languages, and a priori or procedural languages, and may take any form, including independent programs, modules, components, subroutines, or other units suitable for use in a computing environment.

[0111] Additionally, the logic flow and structural block diagrams described in this patent document describe corresponding functions supported by the disclosed structural means, corresponding acts supported by the steps, and / or specific methods, and may be used to construct corresponding software structures and algorithms and their equivalents.

[0112] The processes and logic flows described herein are executable by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output.

[0113] The written description sets forth the best mode of the invention and provides examples to explain the invention and to enable one skilled in the art to make and use the invention. The specification so written is not intended to limit the invention to the specific terms set forth.

[0114] Although the present invention has been described above with reference to preferred embodiments, it will be understood by those skilled in the art or those having ordinary knowledge in the art that various modifications and changes can be made to the present invention without departing from the spirit and technical scope of the present invention as set forth in the claims below. Therefore, the technical scope of the present invention should not be limited to the contents described in the detailed description of the specification, but should be determined by the claims.

Claims

1. A method for determining a roadway on which a vehicle is traveling, comprising: an operation of acquiring driving image information of the vehicle; inputting the vehicle driving image information into a first artificial neural network to generate roadway prediction information for the vehicle; an operation of detecting a lane change of the vehicle based on predicted lane information of the vehicle; generating lane change determination information indicating whether the vehicle has changed lanes using a second artificial neural network based on detecting the lane change of the vehicle; and Including, The second artificial neural network receives sequentially input roadway feature vectors extracted in time series from video information of the vehicle traveling after a lane change of the vehicle is detected.

2. 2. The method of claim 1, wherein the second artificial neural network comprises a recurrent neural network (RNN) or a long-short term memory (LSTM).

3. 2. The method of claim 1, wherein the operation of generating the vehicle's road prediction information includes an operation of inputting the vehicle's driving image information into the first artificial neural network to generate the vehicle's road prediction information and reliability information for the vehicle's road prediction information.

4. 2. The method of claim 1, wherein the operation of generating lane change confirmation information indicating whether the vehicle has changed lanes includes an operation of generating the lane change confirmation information and reliability information for the lane change confirmation information using the second artificial neural network.

5. 2. The method of claim 1, wherein the parameters of the second artificial neural network are updated so that a difference between output data of the second artificial neural network obtained by processing driving video data using the second artificial neural network and ground truth data corresponding to the driving video data decreases.

6. 1. An electronic device comprising: at least one processor; a memory for storing instructions; Including, The instructions, when executed by the at least one processor, cause the electronic device to perform a plurality of operations; The plurality of operations include: An operation of acquiring vehicle driving video information; inputting the vehicle driving image information into a first artificial neural network to generate roadway prediction information for the vehicle; an operation of detecting a lane change of the vehicle based on predicted lane information of the vehicle; generating lane change determination information indicating whether the vehicle has changed lanes using a second artificial neural network based on detecting the lane change of the vehicle; and Including, The second artificial neural network is an electronic device that sequentially receives, after a lane change of the vehicle is detected, lane feature vectors extracted in time series from video information of the vehicle traveling.

Images