Method for recognizing linear structures in image data

Abstract

The present invention relates to a method for recognizing linear structures (14) in image data and determining their trajectories. The method includes the steps of capturing (30) image data with an image sensor, partitioning (32) the entire image data into a number of cells (18) and assigning at least one predefined, oriented, fixed-length line (22) to each cell (18), and dividing (34) at least one line (22) into a predetermined number of line segments (38). In addition, the method includes the steps of calculating (42a) a probability value for the presence of a linear structure (14) for at least one line (22) in each cell (18), and determining the start and end points (A, B) of the line segment (38) for potential linear structures (14). n , E n ) of the remaining lines (22), discarding (50) all lines (22) that are below a predefined probability threshold, and calculating the probability and quantile values ​​(d n ) into a calculation function (54) and outputting the line (22) that most closely resembles the linear structure (14); and calculating the remaining line (22) and its associated quantile (d n ), determining (58) at least one partial trajectory of the linear structure (14), whereby all partial trajectories represent the entire trajectory of the linear structure (14).

Description

[Technical Field]

[0001] The present invention relates to a method for the recognition of linear structures in image data and the determination of their trajectories, but also to a method for training a convolutional neural network to determine linear structures and their trajectories in image data. [Background technology]

[0002] In the process of increasing automation of vehicle functions, the recognition of road markings and linear structures in image data is becoming increasingly important. In this regard, it is important that the various road markings can be recognized correctly and quickly by the vehicle.

[0003] German Patent Application Publication No. 102004057188 ​​discloses a device for assisting vehicle navigation. The scene ahead of the vehicle is imaged by a CCD camera. The number of pixels in each horizontal row required for the vehicle's navigation is stored, and a decision is made as to whether the vehicle can pass a parked vehicle based on the ratio of the number of pixels in each horizontal row based on the width of the vehicle to the number of pixels in the road where no vehicle is parked in the image.

[0004] EP 3410398 A1 discloses a system for recognizing road information, which is capable of determining the position of lane markings on the other side of a lane after a lane change. The system includes means for recognizing forward lane markings, means for recognizing lateral lane markings, and means for estimating forward lane markings on the other side of the lane after a lane change. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] German Patent Application Publication No. 102004057188 [Patent Document 2] European Patent Application Publication No. 3410398 Summary of the Invention [Problem to be solved by the invention]

[0006] The problem underlying the present invention is to provide a method that allows for the recognition of linear structures in image data and the determination of their trajectories, and in addition a method that allows for a better recognition of line segments of limited length. [Means for solving the problem]

[0007] This problem is solved by a method for recognizing linear structures in image data and determining their trajectories with the features of claim 1. Furthermore, the invention presents a method for training a convolutional neural network with the features of claim 6. Preferred embodiments can be read from the dependent claims.

[0008] The present invention provides a method for recognizing linear structures in image data and determining their trajectories. Linear structures are object edges or marking lines of three-dimensional objects, such as road markings. Image data can refer to both 2D and 3D images. The method includes capturing image data with an image sensor and dividing the entire image data into a number of cells and assigning at least one predefined, oriented, fixed-length line to each cell. The image data is captured by an image sensor, which is advantageously a camera sensor, a lidar sensor, or a radar sensor. According to the present invention, a line does not necessarily mean a straight line. Accordingly, the term line also includes curved lines or curves.

[0009] To recognize linear structures in all spatial directions, it is advantageous to assign a large number of lines of fixed length in various orientations to a cell. However, to detect only linear structures in a specific orientation, it is advantageous to use only lines having the direction to be detected. By setting lines of fixed length, it is possible to recognize line segments of a linear structure. This allows one line segment of a linear structure to be mapped via each line. Accordingly, it is possible to recognize, for example, dashed lane markings, thereby making it possible to distinguish between solid and dashed lane markings. The position of the line segments within the dashed lines can also be recognized. This improves the recognition and distinction of various linear structures.

[0010] In a further process step, at least one line is divided into a predetermined number of line segments, and for at least one line in each cell, a probability value for the presence of a linear structure is calculated. Displacement values ​​of the start and end points of the line segment relative to the potential linear structure are calculated, and all lines that are below a predefined probability threshold are discarded. A line segment is a partial section of a line with a fixed length. The line segments may all have the same or different lengths. Each line segment has a start point and an end point that overlap with the start point or end point of an adjacent line segment.

[0011] The probability value is a value that roughly indicates whether a linear structure exists. Therefore, various lines can be weighted through the probability value, so that there is a criterion that can determine which line should be used as a starting point to represent a linear structure. By setting a probability threshold, lines with low probability values ​​can be discarded in advance. This can significantly reduce the calculation cost of the further method.

[0012] In addition, the probability values ​​and displacement values ​​of the remaining lines are input into a calculation function, and the line that most closely resembles the linear structure is output. From the remaining lines and their associated displacement values, at least one partial trajectory of the linear structure is determined, where all partial trajectories represent the entire trajectory of the linear structure. This calculation function is, for example, an algorithm that selects the line that best corresponds to the linear structure based on the probability values ​​and displacement values. This selected line is the best starting point for determining the actual trajectory of the linear structure. By applying the displacement values ​​to this line, the true trajectory of the linear structure is obtained. Since each line represents only a partial trajectory of the entire trajectory of the linear structure, it can also represent interruptions in the linear structure, thus recognizing, for example, dashed lane markings and the exact position of the partial line.

[0013] In a preferred embodiment of the present invention, these steps are performed by a trained convolutional neural network, which creates a general-purpose model based on training examples. Thus, by training, such a network can be used to quickly and easily detect the actual trajectory of linear structures. This can be performed, in particular, to continuously recognize lane markings, for example, while driving.

[0014] In a further preferred embodiment of the present invention, in addition to the probability value and the displacement value, a length adaptation value is calculated, and the length of the fixed-length line is adapted to the linear structure via this length adaptation value. This length adaptation value is the proportion by which each line segment must be extended or shortened to reach the length of the linear structure. This allows each line to be adapted to the length of the linear structure. As a result, the dashed line segments can be represented with the correct length via this method.

[0015] Preferably, the fixed length lines are represented in the form of a parameter function. By representing the lines as a parameter function, the lines can be more easily described. In addition, by changing a few parameters of this function, it is possible to generate a large number of lines, which can be represented by the same parameter function. This simplifies and accelerates the implementation of the method.

[0016] In an advantageous variant, at least one non-maximum suppression function is used for the calculation function. In the case of non-maximum suppression, the highest probability value is used as the starting point. Of the remaining lines, only the line with the highest probability value is retained from the lines grouped according to the similarity function in non-maximum suppression. This means that it is also possible to recognize multiple linear structures in one image.

[0017] The present invention further provides a method for training a convolutional neural network to determine a linear structure and its trajectory. In a first step, training data is input, the training data including sensor data having at least one linear structure with a known trajectory, for which probability values ​​and quantiles are set. The probability values ​​and quantiles determined by the convolutional neural network according to the method of the present invention are compared with the set probability values ​​and quantiles. The error is determined using a cost function. During further processing of the training data by the convolutional neural network, parameters characterizing the model's behavior are changed with the aim of potentially improving the determination by the cost function, and these parameters are released when the determined accuracy value reaches a predetermined value.

[0018] Advantageously, the convolutional neural network is trained to estimate a length fit of a line of fixed length for a linear structure. This additional training of the length fit allows the method to determine the length fit more accurately. Accordingly, the advantages already described above are achieved.

[0019] Furthermore, the problem on which the invention is based is solved by a control device adapted to carry out the method according to the invention. The aforementioned methods may, inter alia, be implemented on a computer, for example, and thus embodied in software. The present invention therefore also relates to a computer program having machine-readable instructions which, when executed on one or more computers, cause the one or more computers to carry out the aforementioned methods.

[0020] The invention likewise relates to a machine-readable data storage medium and / or a download product comprising this computer program, which is a digital product that can be transmitted over a data network, i.e. downloaded by a user of the data network, and which digital product can be offered for immediate download, for example, in an online shop.

[0021] Such a computer program may be run on one or more computers located, for example, in a cloud, and the advantages mentioned for the method are achieved via computers running in such a cloud.

[0022] Exemplary embodiments of the invention are illustrated in the drawings and are explained in more detail in the following description. [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 1 shows an image of a roadway captured via a camera. [Figure 2] FIG. 1 illustrates a method for recognizing linear structures and determining their trajectories. [Figure 3]FIG. 1 illustrates predefined lines and linear structures partitioned into line segments before length fitting. [Figure 4] FIG. 10 shows predefined lines and linear structures partitioned into line segments after length fitting. [Figure 5] FIG. 1 illustrates a method for training a convolutional neural network according to one exemplary embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0024] 1 shows an image of a roadway 10 captured by a vehicle camera. The image shows linear structures 14, such as a solid road marking 14a, a dashed road marking 14b, and a sidewalk edge 14c. For the purposes of the method according to the present invention, the image was divided into a number of cells 18. Each cell 18 shows that it is assigned a number of differently oriented, fixed-length lines 22. Although this is shown for only one cell 18, all cells 18 have this line 22.

[0025] FIG. 2 shows a diagram of a method for recognizing linear structures 14 and determining their trajectories. In a first step 30 of the method, an image of the roadway 10 is captured. This image or image data is then divided 32 into a number of cells 18, as shown in FIG. 1. Each cell 18 is then assigned a line 22, as shown by one cell 18 in FIG. 1. These lines 22 are variously oriented and have a fixed length. In a subsequent step 34, the line 22 is divided into a predetermined number of line segments 38. In this regard, FIG. 3 shows the line 22 divided into two line segments 38 of equal length.

[0026] In the next process step 42a, for each line 22 of each cell 18, a probability value is calculated for the presence of a linear feature 14 within the corresponding cell 18. At the same time, a quantile value d for the potential linear feature 14 is calculated, as shown in FIG. nare calculated respectively. 42b. quantile d n may be the orthogonal spacing of the line 22 relative to the linear structure 14, as shown in the exemplary embodiment. n are the start and end points A of line segment 38. n , E n The quantile d is calculated from n 4, the length of the line segment 38 is extended so that the line 22 corresponds to the length of the linear structure 14. At this time, the new start and end points A n , E n quantile d n is adapted accordingly.

[0027] To reduce computational costs, in a subsequent step 50, all lines 22 whose probability values ​​are below a probability threshold are discarded. Then, the probability values ​​and quantiles d of the remaining lines 22 are calculated. n is input to a calculation function 54, which calculates the line 22 that is most similar to the linear structure 14. If the linear structure 14 is not present in the image, then no line 22 is output. Similarly, if multiple linear structures 14 are present, multiple lines 22 may be output. The remaining lines 22 and their associated displacement values ​​d n From this, at least one partial trajectory of the linear structure 14 is determined 58, whereby all partial trajectories represent the entire trajectory of the linear structure 14.

[0028] 5 shows a diagram of a method for training a convolutional neural network according to one exemplary embodiment of the present invention. A convolutional neural network is trained in this way so that it can perform the method according to FIG. 2. In a first step 70, training data is input to the convolutional neural network. The training data includes at least image data having at least one linear structure 14 with a known trajectory. A probability value and a quantile value d for this linear structure 14 are then calculated. nIt is set regarding.

[0029] According to the method based on FIG. 2, in the next step 74, the probability values, quantiles d n , and length fit values ​​are calculated. These values ​​are then combined with the set probability values, set quantile values ​​d n , and a set length fit value 78. For each of these values, the error relative to the set value is calculated. This error is determined via a cost function in the next step 82. Subsequently, the parameters characterizing the model's behavior are changed with the aim of potentially improving the determination of the cost function during further processing of the training data by this convolutional neural network. This correspondingly results in probability values, quantiles d n , and determination of the length fit value is carried out until the determined accuracy value reaches a predetermined value, which is advantageously the limit value of the learning curve, after which no further or substantial improvement is achieved by further cycles.

Claims

1. 1. A method for the recognition of linear structures (14) in image data and the determination of their trajectories, comprising: capturing (30) image data by an image sensor; - dividing (32) the entire image data into a number of cells (18) and allocating to each cell (18) at least one predefined, oriented, fixed-length line (22); - dividing (34) at least one said line (22) into a predetermined number of line segments (38); - calculating (42a) for at least one said line (22) of each cell (18) a probability value for the presence of a linear structure (14); - the start and end points (A) of said line segments (38) relative to the potential linear structure (14) ｎ , E ｎ ) quantile (d ｎ ) (42b); - discarding (50) all lines (22) that lie below a predefined probability threshold; the probability values ​​and the quantile values ​​(d ｎ ) into a calculation function and outputting the line (22) that most closely resembles said linear structure (14); - the line (22) that has remained and the displacement value (d ｎ determining (58) at least one partial trajectory of said linear structure (14) from said linear structure (14), wherein all partial trajectories represent the entire trajectory of said linear structure (14); A method comprising:

2. 10. The method of claim 1, wherein said steps are performed by a trained convolutional neural network.

3. The probability value and the quantile (d ｎ 3. The method according to claim 1, wherein a length adaptation value is calculated (42c) in addition to the linear structure (14), and via the length adaptation value, the length of the fixed length line (22) is adapted depending on the linear structure (14).

4. 4. A method according to any one of claims 1 to 3, characterized in that the fixed length line (22) is presented in the form of a parametric function.

5. 5. The method according to claim 1, wherein for the calculation function at least one non-maximal suppression function is used.

6. 1. A method for training a convolutional neural network to determine line structures (14) and their trajectories in image data, comprising: - containing image data having at least one linear structure (14) with a known trajectory, for which probability values ​​and displacement values ​​(d ｎ ) is set in the training data (70); the probability values ​​and the quantiles (d ｎ ) by the set probability value and quantile value (d ｎ ) and comparing (74) - determining (78) the error using a cost function; - modifying (82) parameters characterizing the model's behavior with a view to a possible improvement of the assessment by the cost function during further processing of training data by the convolutional neural network, and releasing said parameters when the established accuracy value reaches a predetermined value; A method comprising:

7. The convolutional neural network determines a length fit (d) of the fixed length line (22) to the linear structure (14). ｎ 7. The method of claim 6, wherein the method is trained to estimate

8. A control unit of a motor vehicle for carrying out the method according to any one of claims 1 to 7.

9. A computer program comprising machine-readable instructions which, when executed on one or more computers, cause said one or more computers to carry out the method of any one of claims 1 to 7.

10. A machine-readable data storage medium and / or download product comprising a computer program according to claim 9.

11. A computer comprising a computer program according to claim 9 and / or a machine-readable data storage medium and / or download product according to claim 10.

Images