Method for identifying a line structure in image data

Abstract

The invention relates to a method for identifying a line structure (14) in image data and determining the course thereof. In this case, the method comprises: a step of detecting (30) image data by means of an image sensor; a step of subdividing (32) the overall image data into a plurality of cells (18) and assigning each cell (18) with at least one line (22) of predefined and oriented fixed length; and a step of dividing (34) the at least one line (22) into a predetermined number of line segments (38). In addition, the method comprises: a step of calculating (42a), for at least one line (22) of each cell (18), a probability value of the presence of a line structure (14); a step of calculating (42b) an offset value (dn) from the start and end point (An, En) of the line segment (38) to the potential line structure (14); a step of "discarding (50) all lines (22) below a predefined probability threshold"; a step of inputting (54) the probability values and the offset values (dn) of the remaining lines (22) into the calculation function, and outputting the line (22) most similar to the line structure (14); and a step of determining (58) at least one partial course of the line structure (14) from the remaining lines (22) and the associated offset value (dn), all partial courses giving the overall course of the line structure (14).

Description

Technical Field

[0001] The present invention relates to a method for detecting line structures in image data and determining their course. Furthermore, the present invention additionally relates to a method for training a convolutional neural network to ascertain line structures and their course in image data. Background Art

[0002] With the increasing automation of vehicle functions, the recognition of lane markings and line structures in image data is becoming increasingly important. In this context, it is important that the vehicle can correctly and quickly recognize different lane markings.

[0003] DE 10 2004 057 188 A1 discloses a device for assisting vehicle driving. The scene in front of the vehicle is mapped into an image by a CCD camera. The number of image elements in each horizontal row required for driving the vehicle is stored, and based on the width of the vehicle and the ratio of the number of image elements of the road on which no vehicles are parked in the image to the number of image elements in each horizontal row, it is determined whether the vehicle can pass by parked vehicles.

[0004] EP 3 410 398 A1 discloses a system for identifying road information that can determine the position of lane markings on the other side of the lane after a lane change. The system includes means for identifying a front lane marking, means for identifying lateral lane markings, and means for estimating the front lane marking located on the other side of the lane after a lane change. Summary of the Invention

[0005] The object of the present invention is to specify a method by which line structures in image data can be detected and their course determined, and by which method line sections of finite length can also be detected in an improved manner.

[0006] This object is achieved by a method for identifying line structures in image data and determining their course, having the features of claim 1. Furthermore, the invention provides a method for training a convolutional neural network, having the features of claim 6. Preferred embodiments can be found in the dependent claims.

[0007] The present invention provides a method for identifying line structures in image data and determining their course. Line structures are understood to be object edges of three-dimensional objects or as marking lines, such as lane markings. Image data are understood to include 2D images and 3D images. The method comprises the steps of "detecting image data with the aid of an image sensor" and "subdividing the overall image data into a plurality of cells and assigning at least one predefined and oriented line of fixed length to each cell." The image data are detected by an image sensor, which is advantageously a camera, a lidar, or a radar sensor. According to the invention, a line is understood to mean not only a straight line. Accordingly, the term "line" also includes curved lines or curves.

[0008] In order to be able to detect line structures in all spatial directions, it is advantageous to assign a plurality of fixed-length lines of different orientations to each cell. However, in order to detect only line structures of a certain orientation, it is advantageous to use only one line with the direction to be detected. By pre-defining lines of fixed length, line segments of the line structure can be identified. Each line can thus be used to map a line segment of the line structure. Accordingly, for example, dashed lane markings can be identified, making it possible to distinguish between solid and dashed lane markings. The position of the lines within the dashed line can also be identified. This improves the recognition and differentiation of different line structures.

[0009] In a further method step, at least one line is divided into a predetermined number of line segments, a probability value for the presence of a line structure is calculated for at least one line in each cell, offsets from the start and end points of the line segments to the potential line structure are calculated, and all lines with a probability below a predetermined threshold are discarded. Line segments may be defined as sections of a line of fixed length. These line segments may all have the same or different lengths. Each line segment has a start and an end point that coincide with the start or end point of an adjacent line segment.

[0010] In this context, a value is understood to be a probability value: it not only indicates whether a line structure is generally present. Therefore, different lines can be weighted using this probability value, creating a criterion for deciding which line should be output to represent the line structure. By predefining a probability threshold, lines with low probability values ​​can be discarded even in the preparatory phase. This significantly reduces the computational effort in subsequent processes.

[0011] Additionally, the probability values ​​and offset values ​​for the remaining lines are input into a calculation function, which outputs the line most similar to the line structure. At least one partial course of these line structures is then determined from the remaining lines and their associated offset values, where all partial courses together represent the overall course of the line structure. The calculation function can be, for example, an algorithm that selects the lines most likely to correspond to the line structure based on the probability values ​​and offset values. These selected lines serve as the optimal starting points from which to arrive at the actual course of the line structure. By applying the offset values ​​to these lines, the actual course of the line structure is determined. Since each line only represents a partial course of the overall course of the line structure, this also allows for the display of interruptions in the line structure, enabling the precise location of, for example, dashed lane markings and partial lines to be identified.

[0012] In a preferred embodiment of the present invention, these steps are performed using a trained convolutional neural network. This trained convolutional neural network creates a generalized model based on training examples. After training, this network can be used to quickly and easily detect the actual course of line structures. This can be performed continuously, for example, during driving to identify lane markings.

[0013] In another preferred embodiment of the present invention, in addition to the probability value and the offset value, a length adjustment value is calculated, by which the length of a fixed-length line can be adjusted to match the line structure. The length adjustment value is the factor by which each line segment must be lengthened or shortened to reach the length of the line structure. This allows each line to be adjusted to match the length of the line structure. Thus, this method allows dashed line segments to be displayed with the correct length.

[0014] Preferably, the fixed-length line is given in the form of a parametric function. By giving the line in the form of a parametric function, it is easier to describe the line. Furthermore, multiple lines can be generated by varying a few parameters of the function, while still being represented by the same parametric function. This simplifies and speeds up the execution of the method.

[0015] In one advantageous development, at least one non-maximum suppression function is used for the calculation function. In this non-maximum suppression function, the highest probability values ​​are used as the starting point. Of the remaining lines, only those with the highest probability values ​​are retained, among the lines grouped according to the similarity function in the non-maximum suppression. This also makes it possible to identify multiple line structures in the image.

[0016] In addition, the present invention also provides a method for training a convolutional neural network to determine line structures and their directions. In a first step, training data are input, which at least includes sensor data with at least one line structure with a known direction and for which probability values ​​and offset values ​​are predetermined. The probability values ​​and offset values ​​determined by the convolutional neural network according to the method according to the present invention are compared with the predetermined probability values ​​and offset values. Deviations are evaluated using a cost function. Parameters characterizing the model behavior are varied with the goal that the evaluation obtained by the cost function is expected to improve when the training data is subsequently processed by the convolutional neural network, and these parameters are adopted if the determined accuracy value reaches a predetermined value.

[0017] Advantageously, a convolutional neural network is trained to estimate length adjustments for fixed-length wires relative to a wire structure. Additional training on the length adjustments allows these length adjustments to be determined more accurately using this method. Accordingly, the aforementioned advantages are achieved.

[0018] The object on which the present invention is based is additionally achieved by a control device which is configured to carry out the method according to the present invention.

[0019] The above method can in particular be computer-implemented and thus embodied in software. Therefore, the present invention also relates to a computer program having machine-readable instructions which, when executed on one or more computers, cause the computer(s) to carry out the described method.

[0020] Likewise, the invention relates to a machine-readable data carrier and / or a download product carrying the computer program. A download product is a digital product that can be transmitted via a data network, i.e., can be downloaded by a user of the data network, such as can be sold for immediate download in an online store.

[0021] Such a computer program can be run on one or more computers, which are arranged, for example, in the cloud. The advantages mentioned for the above method are achieved by such a computer running in the cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Embodiments of the present invention will be illustrated in the accompanying drawings and explained in more detail in the following description. The accompanying drawings show:

[0023] Figure 1 The roadway image detected by the camera device,

[0024] Figure 2 Schematic diagram of the method used to identify line structures and determine their direction,

[0025] Figure 3a representation of a predefined line subdivided into line segments and the line structure before length adjustment,

[0026] Figure 4 a representation of a predefined line subdivided into line segments and the line structure after length adjustment, and

[0027] Figure 5 A schematic diagram of a method for training a convolutional neural network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] Figure 1 An image of a roadway 10 captured by a camera of a motor vehicle is shown. The image displays line structures 14, such as solid lane markings 14a, dashed lane markings 14b, and sidewalk edges 14c. For the method according to the present invention, the image has been subdivided into a plurality of cells 18. One cell 18 is shown to have associated therewith a plurality of lines 22 of fixed length and in different orientations. Although this is shown for only one cell 18, all cells 18 have these lines 22.

[0029] Figure 2 1 shows a diagram of a method for identifying a line structure 14 and determining its course. In a first step 30 of the method, an image of the roadway 10 is detected. Figure 1 As shown, the image or image data is subdivided 32 into a plurality of cells 18. Figure 1 As shown in FIG. 1 , a line 22 is assigned to each cell 18 . Here, these lines 22 are oriented differently and have a fixed length. In a subsequent step 34 , these lines 22 are divided into a predetermined number of line segments 38 . Figure 3 A line 22 is shown divided into two line segments 38 of equal length.

[0030] In the next method step 42a, a probability value is calculated for each line 22 of each cell 18 that a line structure 14 is present in the corresponding cell 18. Figure 3 As shown, the offset value d is calculated 42b to the potential line structure 14. n As shown in this embodiment, the offset value d n It can be the vertical distance from line 22 to line structure 14. Here, the starting point A of line segment 38 is calculated respectively. n and the end point E n The offset value d n In addition to the offset value d n and the probability value, and simultaneously calculate the 42c length adjustment value. Figure 4As shown, the length of the line segment 38 is extended so that the line 22 corresponds to the length of the line structure 14. The new starting point A is adjusted accordingly. n and the new endpoint E n The offset value d n .

[0031] In order to reduce the computational workload, in the following step 50, all lines 22 whose probability values ​​are lower than the probability threshold are discarded. Subsequently, the probability values ​​and offset values ​​d of the remaining lines 22 are summed. n Input 54 is fed into the calculation function. The calculation function calculates the line 22 that is most similar to the line structure 14. If no line structure 14 is to be present in the image, then no line 22 is output accordingly. Similarly, if multiple line structures 14 are present, multiple lines 22 can also be output. Subsequently, the remaining lines 22 and the associated offset values ​​d are used to calculate the line 22 that is most similar to the line structure 14. n At least one partial course of the wire structure 14 is determined 58 , wherein all partial courses give the overall course of the wire structure 14 .

[0032] exist Figure 5 A diagram of a method for training a convolutional neural network according to an embodiment of the present invention is shown in FIG. The convolutional neural network is trained by this method so that the network can implement the Figure 2 In a first step 70, training data are input to the convolutional neural network. These training data include at least image data with at least one line structure 14 with a known course. In this case, a probability value and an offset value d are predefined for this line structure 14. n .

[0033] According to Figure 2 The method described above calculates the probability value and the offset value d in the next step 74. n Then, these values ​​are compared with the predetermined probability value, the predetermined offset value d n The values ​​are compared with the predetermined length adjustment values ​​78. For each of these values, the deviation from the predetermined value is calculated. In the next step 82, the deviation is evaluated by means of a cost function. Subsequently, the parameters characterizing the model behavior are changed with the goal that the evaluation obtained by means of the cost function is expected to improve during the subsequent processing of the training data by the convolutional neural network. This is carried out accordingly until the obtained probability values, the offset values ​​d n The accuracy of the determination of the sum of the length adjustment values ​​reaches a predetermined value. This value is advantageously the limit value of the learning curve, after which no further or significant improvement can be achieved after subsequent multiple runs.

Claims

1. A method for identifying line structures (14) in image data and determining their orientation, the method comprising the following steps: - detecting (30) image data by means of an image sensor, - subdividing (32) the overall image data into a plurality of cells (18) and assigning to each cell (18) at least one line (22) of a predefined and oriented fixed length, - dividing (34) the at least one line (22) into a predetermined number of line segments (38), - calculating (42a) for at least one line (22) of each cell (18) a probability value of "presence of a line structure (14)", - Calculate (42b) the distance from the start and end points (A) of the line segment (38) n , E n ) to the offset value (d) of the potential line structure (14) n ), - discarding (50) all lines (22) below a predefined probability threshold, - The probability values ​​and the offset values ​​(d n ) is input (54) into the calculation function, and the line (22) that is most similar to the line structure (14) is output, - from the remaining lines (22) and the associated offset values ​​(d n ) determines (58) at least a partial direction of the line structure (14), wherein The overall course of the wire structure (14) is determined by the course of all the parts.

2. The method according to claim 1, wherein: These steps are performed with the help of a trained convolutional neural network.

3. The method according to claim 1 or 2, characterized in that: In addition to the probability value and the offset value (d n ) and also calculates (42c) a length adjustment value by which the fixed-length wire (22) is adjusted in length in accordance with the wire structure (14).

4. The method according to any one of the preceding claims, characterized in that: The fixed length line (22) is given in the form of a parametric function.

5. The method according to any one of the preceding claims, characterized in that: At least one non-maxima suppression function is used for the calculation function.

6. A method for training a convolutional neural network to obtain line structures (14) and their orientations in image data, the method comprising the following steps: Input (70) of training data, said training data comprising at least image data with at least one line structure (14) of known course, for which a probability value and an offset value (d n ), - The probability value and offset value (d n ) and the predetermined probability value and offset value (d n ) for comparison (74), - using a cost function to evaluate (78) the deviation, - changing (82) parameters characterizing the model behavior, with the goal that the evaluation obtained by the cost function is expected to be improved during subsequent processing of the training data by the convolutional neural network, and delivering these parameters if the accuracy value obtained reaches a predetermined value.

7. The method according to claim 6, characterized in that: The convolutional neural network is trained to estimate the length adjustment value (d) of a fixed length line (22) relative to the line structure (14) n ).

8. A control unit of a motor vehicle for carrying out the method according to any one of the preceding claims.

9. A computer program comprising machine-readable instructions which, when executed on one or more computers, cause the computer or computers to carry out the method according to any one of claims 1 to 7.

10. A machine-readable data carrier and / or download product carrying a computer program according to claim 9.

11. A computer equipped with a computer program according to claim 9 and / or with a machine-readable data carrier and / or download product according to claim 10.

Images