Method for clustering key points belonging to the same lane marking, as well as control device and motor vehicle

Abstract

The invention relates to a method, a control device (9), and a motor vehicle (10) for clustering points belonging to the same lane marking (22). The method is carried out, in particular, by means of an artificial neural network (4) that captures key points from image data and is trained using predefined clusters (19). A predefined loss function is used to improve the separability of the clusters (19). After training, the neural network (4) is applied to image data (3) to generate cluster information. The cluster information is transformed into control data, which is then transmitted to a driver assistance system to enable at least partially automated longitudinal and / or lateral control of a motor vehicle (10).

Description

The invention relates to a method, a control device and a motor vehicle for clustering key points belonging to a same lane marking.Methods for identifying lane markings and / or rails are known. A common approach is to first identify key points in images or image data describing lane markings and / or rails and then order these key points by clustering or clustering or grouping. In principle, a plurality of lane markings can be depicted on a road. The lane markings may further comprise a plurality of lines and / or arrows. Challenges can arise here, in particular when assigning the key points to the respective lane markings, so that a releasability or separation between the different or respective lane markings and / or the different lines is made more difficult. The cluster information obtained from the clustering can then be transformed into control data. The control data can then be used to initiate technical functions, such as for a driver assistance system, in particular having a lane keeping assistance function and / or lane changing assistance function.The following publications disclose methods for lane recognition:Wang, Jinsheng, Yinchao Ma, Shaofei Huang, Tianrui Hui, Fei Wang, Chen Qian, and Tianzhu Zhang, "A Keypoint-Based Global Association Network for Lane Detection." In 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 1382-91. New Orleans, LA, USA: IEEE, 2022. https: / / doi.org / 10.1109 / cvpr52688.2022.00145.It is disadvantageous here, inter alia, that, in order to combine individual key points to form lane markings, the starting point of the lane of each key point is estimated, which, however, does not function in a robust manner.Li, Xiang, Jun Li, Xiaolin Hu, and Jian Yang. "Line-CNN: End-to-End Traffic Line Detection With Line Temporary Unit." IEEE Transactions on Intelligent Transportation Systems 21, no. 1 (2020): 248-58. https: / / doi.org / 10.1109 / TITS.2019,2890870.In this case, lane markings are modeled as offsets to previously defined anchors, the anchors being positioned at the image edges. However, this approach does not work when the training and inference resolutions differ.SCHILLING, Achim [et al.]: Quantizing the repairability of data classes in neural networks. Neural networks, Vol. 139, 2021, pp. 278-293. ISSN 0893-6080. https: / / doi.org / 10.1016 / j.nineet.2021.03.035Generalized Discrimination Value (GDV) is disclosed here as a measure of how well data classes can be separated in the layers of an artificial neural network.DE 10 2018 118 621 A1 discloses the processing of a point cloud generated by an environment detection device of a vehicle into input data for a neural network.DE 10 2020 211 678 A1 discloses a method for clustering detections.The invention is based on the object of improving the clustering of lane markings.The object is achieved by the subject matters of the independent claims. Advantageous refinements of the invention are described by the dependent patent claims, the following description and the figures.In traditional pattern recognition, the features or feature data suitable for clustering (i.e., for forming a cluster) must be selected from a predefined or predefined set of features. This requires a measure that describes the releasability of a set of features. Two known measures are based on the ratio between a class-overlapping interclass covariance matrix and an intraclass covariance matrix (i.e. the covariances within the respective classes): wherein Σ B describes the interclass covariance matrix (between the classes the covariance of the cluster centers of respective clusters) and Σ W describes the class-internal covariance matrix (the element-by-element mean value of the covariance matrices of the individual clusters).The intraclass covariance matrix and interclass covariance matrix can each be mapped as hyper-ellipsoids. For good releasability, the hyper-ellipsoid of the interclass covariance matrix between the classes must be large and that of the individual classes or clusters must be small. The trace is a measure of the size, because the following applies to a covariance matrix Σ: with the variances σ i on the diagonal of the covariance matrix and the singular values λ i of the covariance matrix. The singular values correspond to the variances of the main components of the underlying data distribution. tr(Σ) therefore corresponds to the sum of the variances of the main components and is invariant with respect to rotations of the data or data points.S 1 ignores the orientation between the inter- and intraclass covariance matrices, but S 2 takes them into account. Orientation refers to the location of data points or feature data or features in a high dimensional space.The invention relates to a method for clustering or grouping points or key points belonging to a same lane marking, wherein the method comprises the following steps, which are carried out in particular by means of a control device oriented for this purpose: a) acquisition of training image data and determination of a plurality of the key points (key points) in the training image data, wherein the key points are each part of the lane marking, b) regressive or prediction of at least one feature in the key points by means of an artificial neural network, whereby training prediction data are generated which represent predetermined clusters, characterized in that c) a first loss function loss S2 to be used is minimized for error quantification of the predetermined clusters, wherein the loss function describes a ratio between an interclass covariance matrix and an intraclass covariance matrix of the predetermined clusters, wherein S 2 is a measure of the quality of the releasability of the predetermined clusters, wherein Σ B is an interclass covariance matrix and Σ W is an intraclass covariance matrix, wherein the interclass covariance matrix is the covariance matrix of the centers of the predetermined clusters of the features and the intraclass covariance matrix is the average value of the covariance matrices of the features of the individual predetermined clusters, d) and / or for error quantification of the predetermined clusters loss S2 is approximated by the steps:decomposing the interclass covariance matrix Σ B into the form VS B V T,taking into account those singular values on the diagonal of S B, which exceed a predetermined threshold value,calculating S w by V Σ w V, where V consists of those lines of V that belong to the singular values considered, and thereby loss S2 is approximated by means of,e) repeating steps b) to d) until an optimization criterion of the neural network is fulfilled, and if the optimization criterion is fulfilled: f) capturing image data and ascertaining a plurality of key points in the image data which are part of a lane marking, g) regressive at least one feature in the key points by means of the artificial neural network, whereby prediction data are generated, h) clustering the prediction data, whereby cluster information is generated which represent clusters, wherein each cluster represents a lane marking and comprises key points which belong to the same cluster, whereby in particular a lane is represented overall, i) transforming the cluster information into control data, and routing the control data to a driver assistance system for realizing an at least partially automatically executed longitudinal and / or transverse guidance (e.g. level 1 to level 5) of a motor vehicle.The acquisition of the training image data can be realized, for example, by means of a camera or sensor device of the motor vehicle. The key points can be determined, for example, by means of CenterNet.In other words, the neural network generates one or more abstract features for each keypoint.Keypoint's may be represented by feature vectors describing their position and / or features.In other words, step d) can comprise the interclass covariance matrix being broken down into its individual elements and only the significant values (singular values) which exceed a specific or predefined threshold being taken into account. From this, a corrected intraclass covariance matrix can be derived to simplify the loss function. In step d), a spectral approximation of the loss function can thus be used, which can reduce the computing effort in particular.In other words, in step h), the at least one feature or abstract feature vector can be assigned to respective clusters by a clustering algorithm.The iterative minimization or optimization of the cost function or loss function can take place offline outside the motor vehicle on the basis of the training dataset or the training image data. During this, it may be provided that the neural network generates feature vectors for each determined keypoint. These can then be subjected to clustering. It can then be known which pixel belongs to which line, and this information can be further processed for the control of the vehicle or motor vehicle.Transforming the cluster information into control data may include converting the cluster information into the original image space as key points, for example. The key points may then be joined so that the original or ground truth lane markings are reconstructed. By "original image space" is meant the image space corresponding to the originally acquired image data. The result is therefore continuously detected lanes which can be reproduced, for example, as red lines via a display device. It is provided that, for example, a vehicle computer implements a motor vehicle technical control, for example a lane assistance function, by means of these recorded lanes. Additionally or alternatively, further sensor data, such as the speed of the motor vehicle and / or a lateral distance from another motor vehicle, which are required for realizing the motor vehicle technical control, can be detected.By means of the invention, (high-quality and / or good) separable cluster information can be generated. This makes it possible to achieve (reliable and / or improved lane recognition). In particular, the artificial neural network may be caused by the loss function(s) to generate separable features.The invention also includes developments which result in additional advantages.One development provides that the clustering is carried out by means of the OPTICS algorithm, Ordering Points To Identify the Clustering Structure, and / or DBSCAN, Density-Based Spatial Clustering of Applications with Noise, using the cosine distance and / or the L1 and L2 metrics as distance metrics.A further development provides that the mean value of the predefined clusters is shifted in the direction of a coordinate origin by means of a second loss function. I represents the number of features, C the number of predetermined clusters, N c the number of key points in the respective predetermined cluster and x c,i,nc the value of the i-th feature of the c-th cluster of the n c- th point of this cluster. In other words, the average values of the features in the individual clusters can be shifted in the direction of a common cluster center point. By aligning with a common cluster center point, numerical instabilities can be avoided. This can illustrate the separation between the clusters.A development provides that the optimization criterion comprises configuring the singular values as close as possible to the value 1, wherein i numbers the features and I is their number. In other words, the aim is to optimize the so-called singular values for each feature in such a way that they come as close as possible to the value 1.This specifies the size of the scatter of the cluster center points and thus prevents the scatter of the cluster center points from growing over all dimensions. Furthermore, this prevents the scattering within the individual clusters from decreasing ever further.A further development provides that the clusters are transformed into a two-dimensional space and filtered there. That is, the data points or features associated with the individual clusters may be arranged in a two-dimensional space, which in particular facilitates visualization and / or evaluation of the clusters. In a two-dimensional representation, patterns and / or outliers and / or intersections between the clusters can be recognized. This information is helpful in viewing whether the clusters are distinct from each other and / or whether certain data points have been incorrectly grouped.A development provides that the method further comprises:transforming the key points belonging to the clusters into a bird's eye view by means of a static transformation, in particular by means of perspective transformation,fitting an n-th degree polynomial to the key points belonging to the clusters in the bird's eye view by means of the random sample consensus algorithm, wherein in particular a large fault tolerance, e.g. 1 to 20 pixels or pixel coordinates, is permitted in order to remove outlier points;smoothing the adjusted polynomial curve by adjusting a B-spline curve, in particular to the remaining key points, which in particular have a fault tolerance above a predetermined threshold value.In particular: Evaluation of the B spline curve and inverse transformation of the resulting key points from the bird's eye view into the original image space. The inverse transformation of the key points into the original image space may allow resulting pixel coordinates to be interpreted into the context of the original image. The evaluation can make it possible to determine how exactly the B spline curve describes the remaining key points. This may be relevant for driving functions, such as for automatic lane guidance.For use cases or application situations which can arise in the method and which are not explicitly described here, provision can be made for an error message and / or a request for inputting a user feedback to be output and / or for a default setting and / or a predetermined initial state to be set according to the method.The invention also includes the control device or backend server device. The control device can have a data processing device or a processor device (processor circuit) which is configured to carry out an embodiment of the method according to the invention. For this purpose, the processor device can have at least one microprocessor and / or at least one microcontroller and / or at least one FPGA (field programmable gate array) and / or at least one DSP (digital signal processor). In particular, a CPU (central processing unit), a GPU (graphic processing unit) or an NPU (neural processing unit) can be used as the microprocessor in each case. Furthermore, the processor device can have program code which is configured to carry out the embodiment of the method according to the invention when executed by the processor device. The program code can be stored in a data memory of the processor device. The processor device can be based on at least one circuit board and / or on at least one SoC (system on chip), for example.The control device can be configured as a vehicle computer and / or backend server device or can comprise the vehicle computer and / or the backend server device.The motor vehicle may include this control device.The motor vehicle according to the invention is preferably designed as a motor vehicle, in particular as a passenger car or truck, or as a passenger bus or motorcycle.As a further solution, the invention also comprises a computer-readable storage medium comprising program code which, when executed by a computer or a computer cluster, causes the latter to execute an embodiment of the method according to the invention. The storage medium may be provided at least partially as a non-volatile data memory (e.g. as a flash memory and / or as an SSD-solid state drive) and / or at least partially as a volatile data memory (e.g. as a RAM-random access memory). The storage medium can be arranged in the computer or computer network. However, the storage medium can also be operated, for example, as a so-called store server and / or cloud server on the Internet. The computer or computer network can provide a processor circuit with, for example, at least one microprocessor. The program code can be provided as binary code and / or as assembler code and / or as source code of a programming language (e.g. C) and / or as a program script (e.g. Python). The computer-readable storage medium can alternatively be realized by a signal with computer-readable data, e.g. a time-variant voltage signal and / or a radio signal.The invention also includes the combinations of the features of the described embodiments. The invention therefore also comprises implementations which each have a combination of the features of a plurality of the described embodiments, provided that the embodiments have not been described as mutually exclusive.Exemplary embodiments of the invention are described below. The following shows: FIG. 1 shows a schematic illustration of an embodiment of the method according to the invention; FIG. 2 is a schematic diagram illustrating an interclass covariance matrix and an intaclass covariance matrix; and FIG. 3 shows a diagram for illustrating a transformation of key points into a bird's eye view.The exemplary embodiments explained below are preferred embodiments of the invention. In the exemplary embodiments, the described components of the embodiments each represent individual features of the invention that are to be considered independently of one another and that develop the invention also independently of one another. Therefore, the disclosure is intended to include combinations of the features of the embodiments other than those illustrated. Furthermore, the described embodiments can also be supplemented by further features of the invention that have already been described.In the figures, identical reference numerals designate functionally identical elements.FIG. 1 shows a motor vehicle 10 in which a user 1 is seated. The motor vehicle 10 comprises a sensor device 2, e.g. configured as a camera for capturing image data 3 and a display device 7. the image data 3 can be sent to a control device 9 and / or to a vehicle computer 6 and / or a backend server device 20, wherein the control device 9 and / or the vehicle computer 6 comprise a neural network 4 configured to ascertain lane markings 22, in particular key points representing lane markings 22. The control device 9 and the vehicle computer 6 can execute vehicle functionality, such as a lane keeping assist function and / or lane change assist function, by means of the neural network 4. The user 1 can initiate the lane keeping assistance function and / or the lane changing assistance function by means of the display device 7. The neural network 4 can have been trained for this purpose in the backend server device 20, wherein the backend server device 20 can likewise comprise the control device 9. The control device 9 and / or the vehicle computer 6 and / or the backend server device 20 can realize the following steps.According to FIG. 1, a method for clustering key points belonging to a same lane marking 22 can be provided, the method comprising the following steps: a) acquiring training image data and ascertaining a plurality of key points in the training image data, wherein the key points are each part of a lane marking 22, b) regressive at least one feature in the key points by means of an artificial neural network 4, whereby training prediction data are generated which represent predefined clusters 19, characterized in that c) for error quantification of the predefined clusters 19 a first loss function loss S2 to be used is minimized by means of or an approximation thereof, wherein the loss function describes a ratio between an interclass covariance matrix and an intraclass covariance matrix of the predefined clusters 19, where S 2 is a measure of the quality of the releasability of the specified clusters 19, where Σ B is an interclass covariance matrix and Σ W is an intraclass covariance matrix, where the interclass covariance matrix is the covariance matrix of the centers of the specified clusters 19 of the features and the intraclass covariance matrix is the mean value of the covariance matrices of the features of the individual specified clusters 19, d) and / or for error quantification of the specified clusters 19 loss S2 is approximated by the steps:decomposing the interclass covariance matrix Σ B into the form VS B V T,taking into account those singular values on the diagonal of S B, which exceed a predetermined threshold value,calculating S w by V T Σ w V, where V consists of those lines of V that belong to the singular values considered, and thereby the S2 is approximated by means of,e) repeating steps b) to d) until an optimization criterion of the neural network is fulfilled 4, and if the optimization criterion is fulfilled: f) capturing image data 3 and ascertaining a plurality of key points in the image data 3 which are part of a lane marking 22, g) regressive at least one feature in the key points by means of the artificial neural network 4, as a result of which prediction data are generated, h) clustering the prediction data, wherein cluster information is generated which represent clusters 19, wherein each cluster 19 represents a lane marking 22 and comprises key points which belong to the same cluster 19, as a whole, as a result of which in particular a lane is represented, i) transforming the cluster information into control data, and routing the control data to a driver assistance system for realizing an at least partially automatically executed longitudinal and / or lateral guidance of a motor vehicle 10.Steps S 1, S 3 and S 4 show, in a simplified manner, the steps of the method in an embodiment in the motor vehicle, comprising the acquisition of the image data 3 and the clustering of the key points, wherein an ordered cluster 13 is produced, represented by S 2. S 3 shows a representation of this cluster in the form of the associated lane marking. A step S 2 is not illustrated in FIG. 1, in which the points of the different classes lie through one another in the feature space (i.e. the classes are not separable). This is because no optimization takes place during execution in the motor vehicle, thereby eliminating S2. The optimization, i.e. steps a) to e), and thus step S 2 therefore only passes "offline" during development, while execution in the motor vehicle comprises steps f) to i). For simplification, steps a) and f), i.e. the acquisition of training image data or image data, is represented by step S 1.FIG. 2 is a schematic diagram illustrating an interclass covariance matrix and an intraclass covariance matrix represented as hyper-ellipsoids. For good releasability, the hyper-ellipsoid of the interclass covariance matrix between the classes must be large and the hyper-ellipsoid of the individual classes or clusters 19 must be small. The intraclass covariance matrix describes the mean scattering of the data within the individual clusters 19. The interclass covariance matrix describes the scattering of the data between the different clusters 19.FIG. 3 shows a diagram for illustrating a transformation of key points into a bird's eye view. It may occur that keypoint or keypoint that are unique to different lane markings 22 are clustered as outliers in the same marking. For this purpose, RANSAC (random sample consensus) can be used. In this case, the clustered points or key points can be transformed into a bird's eye view by a static transformation. In bird's eye view, a polynomial of 5thgrade may be adjusted by means of RANSAC and taking into account a large error tolerance, e.g. 1 to 20 pixels. The outliers can be used further for adapting a B spline. Samples from the B spline (base spline) can be transformed back to the original image space by means of a reversal or reconstruction of the transformation depicted in FIG. 3. This may be useful for knowing how modified curves or surfaces affect the original image or image space.Overall, the examples show how a method for recognizing lane markings 22 can be provided.

Claims

Method for clustering key points belonging to a same lane marking (22) (22), the method comprising the following steps: a) acquiring training image data and ascertaining a plurality of the key points in the training image data, wherein the key points are each part of the lane marking (22), b) regressive at least one feature (11) in the key points by means of an artificial neural network (4), whereby training prediction data representing predefined clusters (19) are generated, characterized in that c) a first loss function loss S2 to be used is minimized by means of 1 s 2 = tr ( ∑ B - 1 ∑ W ) for error quantification of the predefined clusters (19), wherein the loss function describes a ratio between an interclass covariance matrix and an intraclass covariance matrix of the predetermined clusters (19), wherein S 2 is a measure of the quality of the releasability of the predetermined clusters (19), wherein Σ B is an interclass covariance matrix and Σ W is an intraclass covariance matrix, wherein the interclass covariance matrix is the covariance matrix of the centers of the predetermined clusters (19) of the features (11) and the intraclass covariance matrix is the mean value of the covariance matrices of the features (11) of the individual predetermined clusters (19), d) and / or for error quantification of the predefined clusters (19) loss S2 by means of 1 s 2 = tr ( ∑ B - 1 ∑ W ) is approximated by the steps: - decomposing the interclass covariance matrix Σ B into the form VS B V T, - taking into account those singular values s i B on the diagonal of S B, which exceed a predefined threshold value, - calculating S W by V T Σ W V, wherein V consists of those rows of V which belong to the singular values taken into account, and thereby loss S2 by means of ∑ i I%0020̄ s%0020̄ i W s%0020̄ i B is approximated, e) repeating steps b) to d) until an optimization criterion of the neural network is fulfilled (4), and if the optimization criterion is fulfilled: f) acquiring image data (3) and determining a plurality of key points in the image data (3) which are part of a lane marking (22), g) regressive at least one feature (11) in the key points by means of the artificial neural network (4), thereby generating prediction data, h) clustering the prediction data, wherein cluster information is generated which represents clusters (19), wherein each cluster (19) represents a lane marking (22) and comprises key points belonging to the same cluster (19), i) Transforming the cluster information into control data, and routing the control data to a driver assistance system for realizing an at least partially automatically executed longitudinal and / or lateral guidance of a motor vehicle ( 10).The method of claim 1, wherein the clustering is performed using the OPTICS algorithm, Ordering Points To Identify the Clustering Structure, and / or DBSCAN, Density-Based Spatial Clustering of Applications with Noise, using the cosine distance and / or the L1 and L2 metrics as the distance metric.Method according to one of the preceding claims, further comprising: shifting the mean value of the predetermined clusters (19) in the direction of a coordinate origin, by means of a second loss function l o s s c e n t e r = 1 I ∑ i | 1 C ∑ c 1 N c ∑ n c x c, i, n c |, wherein I represents the number of features (11), C represents the number of predetermined clusters (19), N c represents the number of key points in the respective predetermined cluster (19), and x c,i,nc represents the value of the i -th feature of the c -th cluster (19) of the n c- th point of this cluster (19).Method according to one of the preceding claims, wherein the optimization criterion comprises configuring the singular values s i B as close as possible to the value 1 by means of l o s s B = 1 I ∑ i | 1 - s i B | wherein i numbers the features (11) and I is their number.Method according to one of the preceding claims, wherein the clusters (19) are transformed into a two-dimensional space and filtered there.Method according to one of the preceding claims, further comprising: - transforming the key points belonging to the clusters (19) into a bird's eye view by means of a static transformation, - adapting an n-th degree polynomial to the key points belonging to the clusters (19) in the bird's eye view by means of the random sample consensus algorithm, - smoothing the adapted polynomial curve by means of adapting a B-spline curve.A control device (9), wherein the control device (9) comprises a processor circuit comprising program instructions which, when executed by the processor circuit, cause the latter to perform a method according to one of the preceding method claims.Motor vehicle (10) comprising a control device (9) according to claim 7.

Images