Road surface marking identification method and road surface marking identification device

Abstract

This road surface marking identification method and road surface marking identification device identify road surface markings shown in a first image of the surroundings of a vehicle as captured by an image capture unit mounted on the vehicle. First machine learning of the relationship between color information of the first image and a first label is performed using a first convolutional neural network, and together therewith, second machine learning of the relationship between shape information of the first image and the first label is performed using a second convolutional neural network. During the first machine learning and second machine learning, a convolutional operation for each pixel constituting the convolutional layers of one of the first convolutional neural network and second convolutional neural network is selected on the basis of the output value of each pixel constituting the convolutional layers of the other convolutional neural network. The road surface markings are then identified using at least one of the first convolutional neural network and the second convolutional neural network.

Description

Road marking identification method and road marking identification device

[0001] The present invention relates to a road marking identification method and a road marking identification device.

[0002] An intelligent drive control method has been proposed that detects the vehicle's driving environment, obtains detection results for at least one category of target objects in the driving environment, and triggers a driving warning based on whether the detection results satisfy predetermined driving warning conditions and whether the vehicle's driving speed exceeds a vehicle speed threshold, thereby increasing the safety of vehicle driving and limiting the number of low-risk and / or low-urgency warnings based on the vehicle speed threshold (see Patent Document 1).

[0003] Special Publication No. 2021-504245

[0004] According to Patent Document 1, multiple types of lane marks are recognized using a convolutional neural network based on an image captured around a vehicle. However, because the convolutional neural network performs convolutional calculations, there is a significant loss of information in narrow areas present in the image. This results in a problem in that traffic signs with unclear contours or traffic signs with different colors in the distance are easily misrecognized.

[0005] The present invention has been made in view of the above-mentioned problems, and aims to provide a road marking identification method and a road marking identification device that, when using a convolutional neural network to recognize traffic signs reflected in an image, can suppress information loss in narrow areas present in the image and can suppress erroneous recognition of traffic signs with unclear contours or traffic signs with different colored lanes in the distance.

[0006] In order to solve the above-mentioned problems, a road marking identification method and road marking identification device according to one aspect of the present invention identify road markings reflected in a first image captured by an imaging unit mounted on a vehicle, and perform first machine learning using a first convolutional neural network to determine the relationship between color information of the first image and a first label, and perform second machine learning using a second convolutional neural network to determine the relationship between shape information of the first image and the first label. During the first and second machine learning operations, a convolution operation for each pixel constituting one of the first and second convolutional neural networks is selected based on an output value for each pixel constituting the other convolutional layer. Then, road markings are identified using at least one of the first and second convolutional neural networks.

[0007] According to the present invention, when recognizing traffic signs reflected in an image using a convolutional neural network, it is possible to suppress information loss in elongated areas present in the image and to suppress erroneous recognition of traffic signs such as traffic signs with unclear contours and traffic signs with different colored lanes in the distance.

[0008] FIG. 1 is a block diagram showing the configuration of a road marking identification device according to an embodiment of the present invention. FIG. 2 is a flowchart showing an example of processing by the road marking identification device according to an embodiment of the present invention. FIG. 3 is a block diagram showing the configuration of a convolutional neural network in the road marking identification device according to an embodiment of the present invention. FIG. 4A is a diagram showing an example of an image captured of the surroundings of a vehicle. FIG. 4B is a diagram showing an example of shape information generated based on the image. FIG. 4C is a diagram showing an example of the distribution of labels set for each pixel. FIG. 4D is a diagram showing a reference example of the distribution of labels set for each pixel.

[0009] Next, an embodiment of the present invention will be described in detail with reference to the drawings. In the description, the same components are designated by the same reference numerals and duplicated explanations will be omitted.

[0010] [Configuration of the road marking identification device] Fig. 1 is a block diagram showing the configuration of a road marking identification device according to this embodiment. As shown in Fig. 1, the road marking identification device according to this embodiment includes an acquisition unit 71, a database 73, a controller 100, and an output unit 400. The controller 100 is connected to the acquisition unit 71, the database 73, and the output unit 400 via wired or wireless communication paths.

[0011] The acquisition unit 71 acquires teacher data to be used for machine learning (or deep learning) in the controller 100. In particular, the acquisition unit 71 acquires a "teacher label image" as the teacher data. The "teacher label image" is an image in which a first image, which is an image of the area ahead of the vehicle captured by an imaging unit (such as a camera) mounted on the vehicle, is assigned a first label for identifying a road marking reflected in the first image for each pixel constituting the first image.

[0012] The road markings identified by the first label include various road markings. The road markings may include a variety of types, such as a "solid white line," a "broken white line," a "stop line," a "crosswalk," a "zebra crossing," an "arrow," a "bicycle navigation line," a "solid blue line," a "solid yellow line," and "other" that does not fall into any of the above categories. Furthermore, among the road markings, "arrows" may include "in intersection," "go straight, turn left," "turn left," "go straight," "turn right," and "go straight, turn right." The road markings are not limited to the examples given here.

[0013] The truth-labeled image may be generated by so-called "annotation," in which a person visually checks the first image and assigns a first label to each pixel constituting the first image. Furthermore, the truth-labeled image may be generated by performing viewpoint conversion on the truth-labeled image generated by "annotation" based on a virtual pose that the imaging unit could possibly capture. In other words, the truth-labeled image may be obtained by "padding" the truth-labeled image generated by "annotation."

[0014] The truth label image acquired by the acquisition unit 71 is input to the controller 100 and the database 73. Alternatively, the acquisition unit 71 may acquire a second image obtained by capturing an image of the surroundings of the vehicle using an imaging unit.

[0015] The database 73 stores the truth label images. In addition, the database 73 may store various other information generated by the controller 100. For example, the database 73 may store a learning model, which will be described later.

[0016] The output unit 400 outputs various information generated by the controller 100. For example, the output unit 400 may output to the outside an output from a learning model, which will be described later.

[0017] The controller 100 (an example of a control unit or processing unit) is a general-purpose computer equipped with a CPU (central processing unit), memory, and input / output units. A computer program (information processing program) for causing the controller 100 to function as part of the road sign identification device is installed in the controller 100. By executing the computer program, the controller 100 functions as multiple information processing circuits (110, 120, 130, 140, 150, 160) equipped in the road sign identification device.

[0018] Note that, here, an example is shown in which the multiple information processing circuits (110, 120, 130, 140, 150, 160) provided in the road sign identification device are realized by software. However, it is also possible to configure the information processing circuits (110, 120, 130, 140, 150, 160) by preparing dedicated hardware for executing each of the information processes described below. Also, the multiple information processing circuits (110, 120, 130, 140, 150, 160) may be configured by individual hardware.

[0019] The controller 100 includes a color information acquisition unit 110, a shape information acquisition unit 120, a first learning unit 130, a second learning unit 140, a convolution setting unit 150, and an evaluation unit 160 as multiple information processing circuits (110, 120, 130, 140, 150, 160).

[0020] The color information acquisition unit 110 acquires color information for each pixel constituting the first image based on the first image, from among the teacher label images that are teacher data.

[0021] For example, if the first image is a color image, the color information for each pixel includes information on the luminance of red ("R"), green ("G"), and blue ("B") at that pixel. Alternatively, if the first image is a black-and-white gray image, the color information for each pixel may be a grayscale indicating the luminance at that pixel. The color information for each pixel is determined based on the output from an imaging element such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) mounted in the imaging unit.

[0022] The shape information acquisition unit 120 acquires shape information for each pixel constituting the first image based on the first image, from among the teacher label images that are teacher data.

[0023] The shape information may include edge information of road markings reflected in the first image. The shape information acquisition unit 120 may generate edge information by applying a Sobel filter to the first image. Here, a Sobel filter is a spatial filter used in the field of image processing to detect the contours of features contained in an image. Alternatively, the shape information acquisition unit 120 may generate edge information using a smoothing filter, a differential filter, a Prewitt filter, or a filter that is a combination of these.

[0024] The shape information may include corner information of road markings reflected in the first image. The shape information acquisition unit 120 may generate the corner information by applying a Harris filter to the first image. Here, a Harris filter is a spatial filter used in the field of image processing to detect corners of images included in an image.

[0025] The first learning unit 130 performs first machine learning using a first convolutional neural network N1 consisting of multiple convolutional layers. The first learning unit 130 performs the first machine learning so that a first label is output for each pixel from the first convolutional neural network N1 to which color information has been input. As a result of the first machine learning by the first learning unit 130, parameters defining the first convolutional neural network N1 are set.

[0026] The second learning unit 140 performs second machine learning using a second convolutional neural network N2 consisting of multiple convolutional layers. The number of convolutional layers in the second convolutional neural network N2 is the same as the number of convolutional layers in the first convolutional neural network N1. The size of each convolutional layer in the second convolutional neural network N2 is the same as the size of each convolutional layer in the first convolutional neural network N1. The second learning unit 140 performs second machine learning so that a first label is output for each pixel from the second convolutional neural network N2 to which shape information has been input. As a result of the second machine learning by the second learning unit 140, parameters defining the second convolutional neural network N2 are set.

[0027] The configuration of the first convolutional neural network N1 and the configuration of the second convolutional neural network N2 are shown in Fig. 3. Fig. 3 is a block diagram showing the configuration of the convolutional neural network in the road sign recognition device according to this embodiment.

[0028] As shown in FIG. 3, the first convolutional neural network N1 consists of 2×N convolutional layers EL11 to EL1N and DL11 to DL1N, where N is an integer greater than or equal to 1. Each of the convolutional layers EL11 to EL1N and DL11 to DL1N consists of a plurality of units. Additionally, the first convolutional neural network N1 may include a pooling layer, a batch normalization layer, a rectified linear unit layer, an upsampling layer, a softmax output layer, etc.

[0029] The units in the convolutional layers EL11 to EL1N and DL11 to DL1N are connected to each other. More specifically, in the first convolutional neural network N1, multiple convolutional layers are arranged in the order of EL11, EL12, EL13, ..., EL1N, DL1N, ..., DL13, DL12, DL11, and the units in adjacent convolutional layers are connected to each other.

[0030] Note that by connecting units, the set of signals received by a convolutional layer of interest from adjacent convolutional layers can be considered as an "image." The set of signals is processed in the order of convolutional layers EL11, EL12, EL13, ..., EL1N, DL1N, ..., DL13, DL12, DL11, and the "images" processed in the order of the above convolutional layers can be considered to be transmitted sequentially. Therefore, the units constituting each convolutional layer can be considered as pixels of the "image" input / output to each convolutional layer (hereinafter, "unit" and "pixel" may be used interchangeably).

[0031] Each unit has an activation function (e.g., a sigmoid function, a normalized linear function, a softmax function, etc.). A weighted sum is calculated based on multiple inputs to the unit, and the value of the activation function, which uses the sum as a variable, becomes the output of the unit. For example, in the first machine learning, the weights used when calculating the sum in each unit of the first convolutional neural network N1 are adjusted as parameters defining the first convolutional neural network N1.

[0032] In the first convolutional neural network N1, convolutional layers EL11 to EL1N constitute an encoder and generate a low-dimensional latent representation of the input set of color information A1 (a set of color information for all pixels constituting the first image), while convolutional layers DL11 to DL1N constitute a decoder and generate a set of label information A3 from the low-dimensional latent representation.

[0033] Through the first machine learning, the first convolutional neural network N1 becomes a learning model that represents, for each first image, the relationship between the color information assigned to each pixel that constitutes the first image and the first label.

[0034] As shown in FIG. 3, the second convolutional neural network N2 consists of 2×N convolutional layers EL21 to EL2N and DL21 to DL2N, where N is an integer greater than or equal to 1. Each of the convolutional layers EL21 to EL2N and DL21 to DL2N consists of a plurality of units. Additionally, the second convolutional neural network N2 may include a pooling layer, a batch normalization layer, a rectified linear unit layer, an upsampling layer, a softmax output layer, etc.

[0035] More specifically, in the second convolutional neural network N2, multiple convolutional layers are arranged in the order of convolutional layers EL21, EL22, EL23, ..., EL2N, DL2N, ..., DL23, DL22, DL21, and units in adjacent convolutional layers are connected to each other.

[0036] Note that by connecting units, the set of signals received by a convolutional layer of interest from adjacent convolutional layers can be considered as an "image." The set of signals is processed in the order of convolutional layers EL21, EL22, EL23, ..., EL2N, DL2N, ..., DL23, DL22, DL21, and the "images" processed in the order of the above convolutional layers can be considered to be transmitted sequentially. Therefore, the units constituting each convolutional layer can be considered as pixels of the "image" input / output to each convolutional layer (hereinafter, "unit" and "pixel" may be used interchangeably).

[0037] Each unit has an activation function (e.g., a sigmoid function, a normalized linear function, a softmax function, etc.). A weighted sum is calculated based on multiple inputs to the unit, and the value of the activation function, which uses the sum as a variable, becomes the output of the unit. For example, in the second machine learning, the weights used when calculating the sum in each unit of the second convolutional neural network N2 are adjusted as parameters defining the second convolutional neural network N2.

[0038] In the second convolutional neural network N2, convolutional layers EL21 to EL2N constitute an encoder and generate a low-dimensional latent representation of the input set of shape information A2 (a set of shape information for all pixels constituting the first image).Furthermore, convolutional layers DL21 to DL2N constitute a decoder and generate a set of label information A3 from the low-dimensional latent representation.

[0039] Through the second machine learning, the second convolutional neural network N2 becomes a learning model that represents, for each first image, the relationship between the shape information assigned to each pixel that constitutes the first image and the first label.

[0040] When performing the first machine learning and the second machine learning to generate a learning model, color information or shape information is input to the input layer of the neural network that constitutes the learning model, and a set of probabilities indicated for each label is output from the output layer. Parameters related to the learning model are adjusted so that the probability for the first label is maximized in the set of probabilities indicated for each label. Additionally, when color information or shape information is input to the input layer and a label is output from the output layer, parameters related to the first convolutional neural network N1 and the second convolutional neural network N2 are adjusted so that the difference between the output label and the first label is reduced.

[0041] When performing the first machine learning and the second machine learning to generate a learning model, for example, gradient descent, stochastic gradient descent, or the like may be used to minimize errors related to the output of the neural network. Additionally, backpropagation may be used to calculate the gradient in gradient descent or stochastic gradient descent.

[0042] In machine learning using neural networks, generalization performance (the ability to discriminate against unknown data) and overfitting (a phenomenon in which the learning model adapts to the data used to create it but the generalization performance does not improve) can become problems.

[0043] Therefore, when generating a learning model by performing the first and second machine learning, a method such as regularization that restricts the degrees of freedom of weights during learning may be used to mitigate overfitting. Other methods may also be used, such as dropout, which probabilistically selects units in a neural network and disables other units. Furthermore, methods such as data regularization, data standardization, and data augmentation that eliminate bias in the data may also be used to improve generalization performance.

[0044] The convolution setting unit 150 selects a convolution operation for each pixel that constitutes the convolution layer of the first convolutional neural network based on an output value for each pixel that constitutes the convolution layer of the second convolutional neural network while the first machine learning and the second machine learning are performed. Similarly, the convolution setting unit 150 selects a convolution operation for each pixel that constitutes the convolution layer of the second convolutional neural network based on an output value for each pixel that constitutes the convolution layer of the first convolutional neural network while the first machine learning and the second machine learning are performed.

[0045] First, we will explain the "convolution operation." Suppose that the convolution layer CL1 of interest, among the multiple convolution layers that make up the first convolutional neural network N1 or the second convolutional neural network N2, receives an "image" from the adjacent convolution layer CL2, which consists of a set of signals "W1 × W2." The value of the signal included in the "image" is multiplied by x. ｉｊ (where i=0, 1, . . . , W1-1, j=0, 1, . . . , W2-1).

[0046] On the other hand, to define the convolution operation, consider a "filter" consisting of a set of "H1 x H2" signals. The size of the "filter" ("filter size") is "H1 x H2". If the value of the signal included in the "filter" is expressed as h ｐｑ (Note that p=0, 1, . . . , H1-1, q=0, 1, . . . , H2-1). "H1×H2" is an integer of 2 or more.

[0047] In this case, the convolution operation "u ｉｊ " is defined as the following equation 1 between the above "image" and "filter".

[0048]

[0049] In other words, each unit in the convolutional layer CL1 of interest is connected to "H1 x H2" units in the adjacent convolutional layer CL2. The value "h" defines the "filter" for each unit. ｐｑ " is a parameter that defines the convolutional neural network (first convolutional neural network N1 or second convolutional neural network N2) to which the convolutional layer CL1 belongs.

[0050] The convolution setting unit 150 selects a convolution operation for each unit constituting the convolution layer of the first convolutional neural network N1 based on the output value of each unit constituting the convolution layer of the second convolutional neural network N2. For example, the convolution setting unit 150 selects a convolution operation for each unit constituting the convolution layer EL11 of the first convolutional neural network N1 based on the output value of each unit constituting the convolution layer EL21 of the second convolutional neural network N2.

[0051] Similarly, the convolution operations in each unit constituting the convolutional layers EL12, EL13, ..., EL1N, DL1N, ..., DL13, DL12, and DL11 are selected based on the output values ​​of each unit constituting the convolutional layers EL22, EL23, ..., EL2N, DL2N, ..., DL23, DL22, and DL21, respectively.

[0052] For example, while the first machine learning and the second machine learning are being performed, the convolution setting unit 150 performs convolution with a filter size of 1 in a unit of the convolution layer of the first convolutional neural network N1 that corresponds to a unit having an output value equal to or greater than a predetermined threshold in the convolution layer of the second convolutional neural network N2.

[0053] More specifically, among the outputs of the units of the convolution layer of the second convolutional neural network N2, the convolution operation “u ｉｊ Consider the case where " is greater than or equal to a predetermined threshold. In this case, among the units of the convolution layer of the first convolutional neural network N1, the output "u ｉｊInstead of determining "x" by equation 1, ｉｊ " or a constant multiple thereof.

[0054] On the other hand, among the outputs of the units of the convolution layer of the second convolutional neural network N2, the convolution operation “u ｉｊ If " is less than a predetermined threshold, the output "u" of the corresponding unit among the units of the convolution layer of the first convolutional neural network N1 is ｉｊ " is determined by Equation 1.

[0055] When compared at the same convolutional layer level, the "image" flowing through the first convolutional neural network N1 and the "image" flowing through the second convolutional neural network N2 are the same size. Therefore, it can be said that the convolution setting unit 150 superimposes the "image" flowing through the second convolutional neural network N2 on the "image" flowing through the first convolutional neural network N1, thereby feeding back shape information into the analysis of color information.

[0056] The evaluation unit 160 inputs an image to the first convolutional neural network N1 and performs semantic segmentation on the input image. That is, the evaluation unit 160 inputs a second image of the surroundings of the vehicle captured by the imaging unit to the first convolutional neural network N1.

[0057] The evaluation unit 160 may obtain a second label for identifying a road marking reflected in the second image by calculating an output from the first convolutional neural network N1 to which the second image is input.

[0058] The evaluation unit 160 may calculate an output from the first convolutional neural network N1 to which the second image is input, thereby obtaining a second label for each pixel constituting the second image and identifying road markings reflected in the second image.

[0059] The imaging unit that captured the second image may be the same as or different from the imaging unit that captured the first image used to generate the learning model. Furthermore, the vehicle that has an area around it that appears in the second image may be the same as or different from the vehicle that has an area around it that appears in the first image.

[0060] Alternatively, the evaluation unit 160 may output the second label obtained for each pixel constituting the second image to the outside via the output unit 400 .

[0061] Furthermore, the evaluation unit 160 may determine whether additional learning is required to update the first convolutional neural network N1 and the second convolutional neural network N2. More specifically, the evaluation unit 160 may use a teacher label image to calculate a confusion matrix (TP / TN / FP / FN) between the results of estimation for each pixel by the learning model and the first labels (correct labels) assigned to each pixel in the teacher label image. Here, TP (true positive) represents the number of labels correctly estimated to be the correct label. TN (true negative) represents the number of labels correctly estimated to be incorrect. FP (false positive) represents the number of labels incorrectly estimated to be the correct label. FN (false negative) represents the number of labels incorrectly estimated to be incorrect.

[0062] Then, the evaluation unit 160 may calculate the accuracy, precision, recall, and evaluation index F-measure of the learning model based on the confusion matrix. Here, the accuracy of the learning model is calculated by "(TP+TN) / (TP+TN+FP+FN)." The precision is an index that measures the degree to which what is estimated to be the correct label is actually the correct label, and is calculated by "TP / (TP+FP)." The recall is an index that measures the degree to which what is actually the correct label is estimated to be the correct label, and is calculated by "TP / (TP+FN)."

[0063] The evaluation index F value is calculated as the harmonic mean of the precision and recall, ie, "2 x precision x recall / (precision + recall)."

[0064] The evaluation unit 160 may specify a label to be learned based on the calculated evaluation index F value. The processes in the color information acquisition unit 110, the shape information acquisition unit 120, the first learning unit 130, the second learning unit 140, and the convolution setting unit 150 may be repeated until a learning model that satisfies the target estimation accuracy is obtained.

[0065] For example, the evaluation unit 160 may calculate, for each road marking, a recognition rate when identifying the road marking using at least one of the trained first convolutional neural network and the trained second convolutional neural network. The evaluation unit 160 may then extract, as a target road marking, a road marking associated with a recognition rate less than a predetermined value. Furthermore, the evaluation unit 160 may extract, as a target image, a first image in which the number of pixels constituting the target road marking is equal to or greater than a predetermined number, and perform the first machine learning and the second machine learning based on the target image.

[0066] [Example of Processing by the Road Sign Identification Device] Fig. 2 is a flowchart showing an example of processing by the road sign identification device according to this embodiment. The processing by the road sign identification device shown in Fig. 2 may be repeatedly executed at a predetermined cycle.

[0067] In step S101, the acquisition unit 71 acquires an unselected teacher label image as teacher data.

[0068] In step S103, the color information acquisition unit 110 acquires color information for each pixel constituting the first image based on the first image included in the truth-labeled image, and the shape information acquisition unit 120 acquires shape information for each pixel constituting the first image based on the first image included in the truth-labeled image.

[0069] In step S105, an output value for each pixel of one of the first convolutional neural network N1 and the second convolutional neural network N2 is calculated. For example, the first learning unit 130 calculates the output value for each pixel of the first convolutional neural network N1. The second learning unit 140 calculates the output value for each pixel of the second convolutional neural network N2.

[0070] In step S107, the convolution setting unit 150 extracts pixels whose output values ​​are equal to or greater than a predetermined threshold value from among the pixels of one of the convolutional neural networks.

[0071] In step S109, the convolution setting unit 150 sets the convolution operation for each pixel of the other of the first convolutional neural network N1 and the second convolutional neural network N2.

[0072] Specifically, the convolution setting unit 150 sets a "filter" with a filter size of 1 for pixels of the first convolutional neural network N1 that correspond to pixels of the second convolutional neural network N2 whose output values ​​are equal to or greater than a predetermined threshold. On the other hand, the convolution setting unit 150 sets a "filter" with a filter size greater than 1 for pixels of the first convolutional neural network N1 that correspond to pixels of the second convolutional neural network N2 whose output values ​​are less than the predetermined threshold.

[0073] Alternatively, the convolution setting unit 150 sets a "filter" with a filter size of 1 for pixels of the second convolutional neural network N2 that correspond to pixels of the first convolutional neural network N1 whose output values ​​are equal to or greater than a predetermined threshold. On the other hand, the convolution setting unit 150 sets a "filter" with a filter size greater than 1 for pixels of the second convolutional neural network N2 that correspond to pixels of the first convolutional neural network N1 whose output values ​​are less than the predetermined threshold.

[0074] In step S111, the second learning unit 140 calculates the output value for each pixel of the other of the first convolutional neural network N1 and the second convolutional neural network N2. For example, the second learning unit 140 calculates the output value for each pixel of the second convolutional neural network N2. The first learning unit 130 calculates the output value for each pixel of the first convolutional neural network N1.

[0075] In step S113, the controller 100 determines whether or not there is an unselected truth label image.

[0076] If it is determined that there is an unselected truth-labeled image (YES in step S113), the process returns to step S101. On the other hand, if it is determined that there is no unselected truth-labeled image (NO in step S113), the process proceeds to step S115.

[0077] In step S115, the first learning unit 130 sets parameters for the first convolutional neural network N1 based on the output values ​​of the first convolutional neural network N1. The second learning unit 140 sets parameters for the second convolutional neural network N2 based on the output values ​​of the second convolutional neural network N2. Thereafter, the processing of the road marking identification device shown in FIG. 2 ends.

[0078] [Effects of the Embodiment] As described in detail above, the road marking identification method and road marking identification device according to this embodiment identify road markings reflected in a first image captured by an imaging unit mounted on a vehicle, around the vehicle. A first machine learning process is performed using a first convolutional neural network to determine the relationship between color information of the first image and a first label, and a second machine learning process is performed using a second convolutional neural network to determine the relationship between shape information of the first image and the first label. During the first and second machine learning processes, a convolution operation for each pixel constituting one of the first and second convolutional neural networks is selected based on the output value for each pixel constituting the other convolutional layer. Then, road markings are identified using at least one of the first and second convolutional neural networks.

[0079] This reduces information loss in narrow areas of an image when using a convolutional neural network to recognize traffic signs reflected in an image, and prevents misrecognition of traffic signs with unclear contours or traffic signs with different colored lanes in the distance.

[0080] For example, consider the case where semantic segmentation is performed on an image of the surroundings of a vehicle, as shown in Fig. 4A. In the road sign identification method and road sign identification device according to this embodiment, color information obtained for each pixel of the image shown in Fig. 4A is input to a first convolutional neural network.

[0081] The road marking identification method and road marking identification device according to this embodiment then generate shape information based on the image shown in Fig. 4A. Fig. 4B is a diagram showing an example of shape information generated based on the image shown in Fig. 4A. In Fig. 4B, the outline of the road marking reflected in the image of Fig. 4A is highlighted in white. The road marking identification method and road marking identification device according to this embodiment input the shape information obtained for each pixel of the image to a second convolutional neural network.

[0082] 4C shows the results of semantic segmentation performed on an image of the surroundings of a vehicle captured using the road marking identification method and road marking identification device according to this embodiment, which is a diagram showing an example of the distribution of labels set for each pixel.

[0083] Figure 4C shows how road markings TG1, TG2, and TG3 are accurately detected and labeled. The signals flowing through the second convolutional neural network are used to select the convolution operation performed by the first convolutional neural network, thereby feeding back shape information to the analysis of color information. As a result, road markings can be accurately detected, as shown in Figure 4C.

[0084] For comparison with Fig. 4C, Fig. 4D shows the results when the signal flowing through the second convolutional neural network is not used to select the convolution operation performed in the first convolutional neural network, which is a reference example of the distribution of labels set for each pixel.

[0085] Unlike Fig. 4C, in Fig. 4D, the road marking TG1, the road marking TG2 indicating a lane marking, and the road marking TG3 indicating an arrow cannot be detected with high accuracy. This is because the convolution operation performed in the first convolutional neural network causes information loss at the boundary portions of the road markings.

[0086] According to the road marking identification method and road marking identification device of this embodiment, the convolution operation in each unit constituting the convolution layer of the first convolutional neural network is selected, which results in suppressing information loss at the boundary portions of road markings and making it possible to suppress erroneous recognition of traffic signs such as traffic signs with unclear contours or traffic signs with different colored lanes in the distance.

[0087] Furthermore, the road marking identification method and road marking identification device according to this embodiment may perform convolution with a filter size of 1 on pixels in one convolution layer that correspond to pixels in the other convolution layer that have an output value equal to or greater than a predetermined threshold while performing the first machine learning and the second machine learning. This suppresses information loss at boundary portions of road markings. In particular, because the filter size is 1, blurring of contour positions during convolution is suppressed.

[0088] Furthermore, the road marking identification method and road marking identification device according to this embodiment may perform convolution with a filter size greater than 1 on pixels in one convolution layer that correspond to pixels in the other convolution layer that have an output value less than a predetermined threshold while performing the first machine learning and the second machine learning. This makes it possible to capture the appearance of continuous road markings in areas other than the boundaries of the road markings. As a result, road markings reflected in images can be detected with high accuracy.

[0089] In the road marking identification method and road marking identification device according to this embodiment, the shape information may include edge information of the road marking reflected in the first image, thereby suppressing information loss at the edge portions of the road marking.

[0090] Furthermore, the road marking identification method and road marking identification device according to this embodiment may generate edge information by applying a Sobel filter to the first image, which makes it possible to accurately capture the edge portions of road markings reflected in the first image and reflect the edge portions in machine learning.

[0091] In the road marking identification method and road marking identification device according to this embodiment, the shape information may include corner information of the road marking reflected in the first image, thereby suppressing information loss at the corners of the road marking.

[0092] Furthermore, the road marking identification method and road marking identification device according to this embodiment may generate corner information by applying a Harris filter to the first image, which makes it possible to accurately capture corners of road markings reflected in the first image and reflect the corners in machine learning.

[0093] Furthermore, the road sign identification method and road sign identification device according to the present embodiment may calculate, for each road sign, a recognition rate when identifying a road sign using at least one of a trained first convolutional neural network or a trained second convolutional neural network. Road signs associated with a recognition rate less than a predetermined value may be extracted as target road signs. A first image in which the number of pixels constituting the target road sign is equal to or greater than a predetermined number may be extracted as a target image. First machine learning and second machine learning may be performed based on the target image.

[0094] This allows for relearning to be performed on road markings with low recognition rates. In particular, images containing many pixels related to road markings with low recognition rates can be extracted and used in the first and second machine learning, improving the efficiency of relearning.

[0095] Each of the functions described in the above embodiments may be implemented by one or more processing circuits, including programmed processors, electrical circuits, and even devices such as application specific integrated circuits (ASICs), or circuit components arranged to perform the described functions.

[0096] Although the present invention has been described above based on the embodiments, it will be apparent to those skilled in the art that the present invention is not limited to these descriptions and that various modifications and improvements are possible. The descriptions and drawings that form part of this disclosure should not be understood as limiting the present invention. Various alternative embodiments, examples, and operating techniques will become apparent to those skilled in the art from this disclosure.

[0097] The present invention naturally includes various embodiments not described herein. Therefore, the technical scope of the present invention is defined only by the invention-specifying matters according to the scope of the claims that are appropriate from the above description.

[0098] 71 Acquisition unit 73 Database 100 Controller 110 Color information acquisition unit 120 Shape information acquisition unit 130 First learning unit 140 Second learning unit 150 Convolution setting unit 160 Evaluation unit 400 Output unit N1 First convolutional neural network N2 Second convolutional neural network

Claims

1. A road marking identification method for controlling a controller that identifies road markings reflected in a first image captured by an imaging unit mounted on a vehicle around the vehicle, the method comprising: The controller acquiring color information and shape information for each pixel constituting the first image based on the first image; performing first machine learning using a first convolutional neural network consisting of a plurality of convolutional layers so that, for each pixel, a first label that identifies the road marking is output from the first convolutional neural network to which the color information is input; and performing second machine learning using a second convolutional neural network including a plurality of convolutional layers, the number of convolutional layers and the size of each convolutional layer being the same as those of the first convolutional neural network, so that the first label is output from the second convolutional neural network to which the shape information is input, for each pixel; While performing the first machine learning and the second machine learning, selecting a convolution operation for each pixel constituting one of the convolution layers of the first convolutional neural network and the second convolutional neural network based on an output value for each pixel constituting the other of the convolutional layers of the first convolutional neural network and the second convolutional neural network; identifying the road markings using at least one of a trained first convolutional neural network whose parameters have been adjusted based on the first machine learning and a trained second convolutional neural network whose parameters have been adjusted based on the second machine learning; A road marking identification method comprising:

2. 2. A road marking identification method according to claim 1, comprising: The controller While performing the first machine learning and the second machine learning, performing convolution with a filter size of 1 on pixels of the other convolution layer corresponding to pixels having an output value equal to or greater than a predetermined threshold in the one convolution layer. A road marking identification method comprising:

3. A road marking identification method according to claim 1, comprising: The controller While performing the first machine learning and the second machine learning, performing convolution with a filter size greater than 1 on pixels of the other convolution layer corresponding to pixels having an output value less than a predetermined threshold in the one convolution layer. A road marking identification method comprising:

4. A road marking identification method according to claim 1, comprising: The shape information includes edge information of the road markings reflected in the first image. A road marking identification method comprising:

5. A road marking identification method according to claim 1, comprising: The shape information includes corner information of the road markings reflected in the first image. A road marking identification method comprising:

6. A road marking identification method according to any one of claims 1 to 5, The controller calculating a recognition rate for each road marking when identifying the road marking using at least one of the trained first convolutional neural network and the trained second convolutional neural network; extracting the road markings having a recognition rate less than a predetermined value as target road markings; extracting the first image in which the number of pixels constituting the target road marking is equal to or greater than a predetermined number as a target image; performing the first machine learning and the second machine learning based on the target image; A road marking identification method comprising:

7. A road marking identification device comprising an acquisition unit and a controller, The acquisition unit acquiring a first image of the surroundings of the vehicle by an imaging unit mounted on the vehicle; The controller acquiring color information and shape information for each pixel constituting the first image based on the first image; performing first machine learning using a first convolutional neural network consisting of a plurality of convolutional layers so that, for each pixel, a first label that identifies a road marking reflected in the first image is output from the first convolutional neural network to which the color information is input; and performing second machine learning using a second convolutional neural network including a plurality of convolutional layers, the number of convolutional layers and the size of each convolutional layer being the same as those of the first convolutional neural network, so that the first label is output from the second convolutional neural network to which the shape information is input, for each pixel; While performing the first machine learning and the second machine learning, selecting a convolution operation for each pixel constituting one of the convolution layers of the first convolutional neural network and the second convolutional neural network based on an output value for each pixel constituting the other of the convolutional layers of the first convolutional neural network and the second convolutional neural network; identifying the road markings using at least one of a trained first convolutional neural network whose parameters have been adjusted based on the first machine learning and a trained second convolutional neural network whose parameters have been adjusted based on the second machine learning; A road marking identification device characterized by the above.