Device and method for training a lane detector

Abstract

A computer-implemented method for training a lane detector (60) to detect a lane in a digital image (x), in particular based on pixel values ​​of the digital image (x), wherein the method comprises: Providing a first plural (T) of digital images with marked lanes that are labelled with lane markings; Obtaining a second plural (T') of digital images from the first plural of digital images by removing the lane markings from the digital images of the first plural of digital images, thereby generating corresponding digital images without markings; Providing data to characterize the lanes (y s ) in the digital images of the first plural of digital images (T); and Providing a data set (T') for training the lane detector (60), wherein the data set (T') comprises pairs of a digital image (x) and data that represent a basic truth of a lane (y). s ) characterize in the digital image (x), wherein the digital image (x) is taken from the second plural (T') of digital images, and the data characterize the lane identified in the corresponding digital image from the first plural (T) of digital images.

Description

[0001] The invention relates to a computer-implemented method for training a lane detector, a method for operating a vehicle that is at least partially automated, a data set, computer-readable storage media and computer programs. State of the art

[0002] DE 10 2010 062 129 B4 discloses a method for lane detection based on lane markings. Advantages of the invention

[0003] The method with the features of independent claim 1 has the advantage that the derived lane detector can operate reliably even when no lane markings are present.

[0004] Further improvements are described in the dependent claims. Further aspects of the invention are described in the parallel independent claims. Disclosure of the invention

[0005] In a first aspect, the invention relates to a computer-implemented method for training a lane detector to recognize a lane in a digital image, in particular based on pixel values ​​of the digital image. This method includes providing a first plurality of digital images with marked lanes, obtaining a second plurality of digital images by removing the lane markings, providing data for characterizing lanes in the first plurality of digital images, and providing a data set comprising pairs of a digital image from the second plurality and data characterizing a basic truth of a lane in the corresponding digital image from the first plurality.This method enables the creation of a high-quality training dataset for lane detectors that can accurately detect lanes on unmarked roads by using images with artificially removed lane markings, thereby improving the detector's ability to generalize from marked to unmarked road conditions without the need for expensive HD maps or manual labeling.

[0006] “No markings” can mean that the lane markings of the road ahead are removed; it does not necessarily imply that all lane markings anywhere in the image are removed, nor does it imply that all markings are removed. “Lane characterization data” can be a semantic segmentation of the image in which the pixels corresponding to the lane are labeled as such. Similarly, the “basic truth of a lane” can also be a semantic segmentation of the image in which the pixels corresponding to the lane are labeled as such.

[0007] In other words, the lane is identified from the image with the lane marking, the lane marking is removed from the image, and the identified lane is then used as the baseline to train the lane detector in a supervised manner to identify the lane without the lane marking.

[0008] In a second aspect, the invention addresses the step of providing data for characterizing lanes in the digital images of the first plurality of digital images by identifying the lanes in these digital images with a lane marking detector. This approach automates the labeling process of lane markings in the data set, significantly reducing the time and effort required for manual labeling, and ensures high accuracy in the basic truth data by using reliable lane marking detection algorithms.

[0009] The procedure further involves identifying lanes in the digital images of the first plural using a lane marking detector. This is achieved by identifying lane markings in the digital images and then identifying lanes based on these identified lane markings. This step refines the process of generating basic truth data for the training dataset by ensuring that lane identification depends directly on the accurate detection of lane markings, thereby improving the reliability and precision of the lane detection training process.

[0010] The lane marking removal step can be performed using a generative model, particularly a stable diffusion model, a Generative Adversarial Network (GAN), or a Variational Autoencoder (VAE). This involves removing the lane markings with a texture that closely matches the texture of an area near the lane marking in the image from which the lane marking is to be removed. Using a generative model, especially a stable diffusion model, to remove lane markings and replace them with a texture that matches the surrounding road surface allows for the generation of highly realistic, marking-free images. This improves the lane detector's ability to operate accurately under real-world conditions where lane markings may be faded or missing.

[0011] The process further includes training the lane detector with the provided dataset. Training the lane detector with a dataset containing images of removed lane markings and corresponding basic truth data significantly improves the detector's performance in recognizing lanes on unmarked roads, leading to more reliable and safer automated driving systems.

[0012] Training involves obtaining lane-characterizing data from a provided digital image within the dataset using the lane detector, and adjusting parameters that characterize the lane detector's behavior based on the obtained lane-characterizing data and the corresponding coupled data characterizing the lane's baseline truth from the provided dataset. This iterative training process, which includes adjusting the lane detector's parameters based on comparisons between its output and the baseline truth data, refines the detector's algorithms for improved lane detection accuracy and reliability, particularly under challenging, unmarked road conditions.

[0013] A method for operating a partially automated vehicle by first training the lane detector according to the described procedure, then providing images that characterize the vehicle's environment, inputting the provided image into the lane detector to obtain data that characterizes the lane in the provided image, and operating the vehicle based on the detected lane characterized by the acquired data. This method enables the practical application of the trained lane detector in partially automated vehicles, thereby enabling accurate, real-time lane detection, which is crucial for safe navigation and operation of the vehicle in environments with unmarked roads, thus improving the safety and reliability of automated driving systems. Description of the embodiments

[0014] Embodiments of the invention are discussed in more detail with reference to the following figures. The figures show the following: Fig. 1 a diagram illustrating an interaction between the lane detector and an actuator; Fig. 2 a vehicle that is at least partially automated and includes a lane detector; Fig. 3 a training system for training the lane detector; Fig. 4. In a flowchart, a procedure for generating a labeled video data set; Fig. 5. In a flowchart, a procedure for training a lane detector with this data set; Fig. 6. In a flowchart, a procedure for operating a vehicle with this lane detector; Fig. 7. An illustration of images with and without lane markings.

[0015] In Fig. Figure 1 shows an embodiment of actuator 10 in its environment 20. The actuator 10 interacts with the control system 40. The actuator 10 and its environment 20 are collectively referred to as the actuator system. At preferably uniform intervals, sensor 30, which preferably comprises an optical sensor, is configured to acquire images of the environment 20. An output signal S of sensor 30 (or, if sensor 30 comprises multiple sensors, an output signal S for each of the sensors), which encodes the images, is transmitted to the control system 40.

[0016] The control system 40 receives a stream of video signals S. It then calculates a series of actuator control commands A depending on the stream of video signals S, which are then transmitted to the actuator 10.

[0017] The control system 40 receives the video signal stream S from sensor 30 in an optional receiver 50. The receiver 50 converts the sensor signals S into input signals x. Alternatively, if no receiver 50 is available, any video signal S can be used directly as the input signal x. The input signal x can, for example, be specified as an extract from the video signal S. Alternatively, the video signal S can be processed to obtain the input signal x. The input signal x comprises image data corresponding to an image recorded by sensor 30. In other words, the input signal x is provided based on the video signal S.

[0018] The input signal x is then forwarded to lane detector 60, which can be provided, for example, by an artificial neural network.

[0019] The classifier 60 is parameterized by parameters ϕ, which are stored in parameter storage St1 and made available through it.

[0020] The lane detector 60 determines output signals y from the input signals x. The output signal y contains information that characterizes a lane in the input signal x. Output signals y are transmitted to planning unit 80, which converts the output signals y into control commands A. The actuator control commands A are then transmitted to the actuator 10 to control the actuator 10 accordingly. Alternatively, output signals y can be used directly as control commands A.

[0021] Actuator 10 receives actuator control commands A, is controlled accordingly, and performs an action that corresponds to the actuator control commands A. Actuator 10 may include control logic that converts the actuator control command A into another control command, which is then used to control actuator 10.

[0022] In further embodiments, the control system 40 can include the sensor 30. In still further embodiments, the control system 40 can alternatively or additionally include the actuator 10.

[0023] Furthermore, the control system 40 can comprise a processor 45 (or a plurality of processors) and at least one machine-readable storage medium 46 on which instructions are stored which, when executed, cause the control system 40 to execute a method according to an aspect of the invention.

[0024] Fig. Figure 2 shows an embodiment in which a control system 40 is used to control a vehicle 100 that is at least partially autonomous.

[0025] Sensor 30 comprises one or more video sensors. Some or all of these sensors are preferably, but not necessarily, integrated into the vehicle 100.

[0026] For example, the lane detector 60 can use the input signal x to detect a lane ahead of the at least partially autonomous vehicle 100. The output signal y can include information characterizing the location of the lane. The control command A can then be determined based on this information, for example, to steer along this lane.

[0027] The actuator 10, which is preferably integrated into the vehicle 100, can be a brake, a drive system, a motor, a powertrain, or a steering system of the vehicle 100. The actuator control commands A can be determined such that the actuator (or actuators) 10 is / are controlled in such a way that the vehicle 100 avoids collisions with the detected objects.

[0028] In Fig. Figure 3 shows an embodiment of training system 140 for training the lane detector 60. The training data unit 150 determines input signals x, i.e., images, which are forwarded to the lane detector 60. For example, the training data unit 150 can access a computer-implemented database St2 in which a set T' of training data is stored. The set T' comprises pairs of images x without markers and corresponding information about the desired lane segmentation y. sThe training data unit 150 selects samples from the set T', for example, randomly. The input signal x of a selected sample is forwarded to the lane detector 60. The desired output signal y s will be forwarded to assessment unit 180.

[0029] Data set extension unit 155 is used to compute a data set T' without markers, which contains modified images x, taken, for example, from the training set T, and their respective desired lane segmentation information y. s includes (derived from images of the training set T using a standard lane detection algorithm that identifies lane markings and then identifies lanes based on those lane markings), with which in Fig. 1 illustrated procedure.

[0030] The lane detector 60 is configured to calculate output signals y from input signals x. These output signals y are also forwarded to the evaluation unit 180.

[0031] Modification unit 160 determines updated parameters ϕ' depending on the input from evaluation unit 180. Updated parameters ϕ' are transferred to parameter storage St1 to replace the current parameters ϕ.

[0032] For example, it can be stipulated that the valuation unit 180 represents the value of a loss function. L depending on the output signals y and the desired output signals y s The modification unit 160 can then calculate updated parameters ϕ', for example by using stochastic gradient lowering to determine the loss function. L to optimize.

[0033] Furthermore, the training system 140 can comprise a processor 145 (or a plurality of processors) and at least one machine-readable storage medium 146 on which instructions are stored which, when executed, cause the control system 140 to execute a method according to an aspect of the invention.

[0034] Fig. Figure 4 shows a flowchart that discloses an embodiment of a method for automatically generating a large-scale labeled video data set T' for training the lane detector 60, which is operated on roads without lane markings.

[0035] First (1000) a large data set (T) is received, consisting of digital images with marked lanes from a user or a database.

[0036] Then (1100) an algorithm for recognizing lane markings on images of the received data set (T) is executed to identify lane markings (y) s ) to identify and document in each image.

[0037] Then (1200) a generative AI model, specifically stable diffusion models, is applied to remove the identified lane markings from the images. The models replace the removed lane markings with textures that closely resemble the surrounding lane surface, producing road images without markings.

[0038] Then (1300) each image is processed by the generative model to remove lane markings and transform the entire dataset. The transformed images are stored along with the original lane marking detections, which serve as the baseline truth data for the corresponding images without markings.

[0039] Then (1400) a dataset T' is provided which contains pairs of digital images x without markings and corresponding basic truth data y. s This includes characterizing the lanes captured in the original images x. This concludes this part of the procedure.

[0040] Fig. Figure 5 shows a flowchart that discloses an embodiment of a method for training the lane detector 60, which is suitable for identifying lanes in digital images, in particular those without explicit lane markings.

[0041] First (2000), the generated data set T', which is combined with the one in Fig. The illustrated procedure obtained is pairs of images x without markings and basic truth data y. s It includes lanes, provided.

[0042] Then (2100) a machine learning model 60 is provided for the lane detection task.

[0043] Then (2200) the images x without markings are fed into the machine learning model 60. The lane detection output y of the model is augmented with the basic truth data y. s For each image x, a comparison is made, and the model's parameters ϕ are adjusted based on the comparison to minimize a cost function that includes a term representing deviations between the detected lanes y and the basic truth y. s punished.

[0044] Then (2300), the performance of the trained model is optionally evaluated against a separate validation set. If necessary, the training process is repeated with adjusted parameters or a modified model architecture to improve accuracy and reliability.

[0045] Then (2400) the lane detector 60 is provided, which is trained to detect lanes in digital images without explicit lane markings and is therefore optimized for high accuracy and reliability. This completes this part of the procedure.

[0046] Fig. Figure 6 shows a flowchart of a procedure that discloses the use of the trained lane detector 60 in the at least partially automated vehicle 100 for real-time lane detection and vehicle guidance.

[0047] First (3000) the vehicle 100 is provided with the trained lane detector 60 integrated into the vehicle's on-board control system 40.

[0048] Then (3100), this control system 40 continuously receives images S characterizing the vehicle's surroundings from the camera 30 mounted on the vehicle 100. These images S are then processed by the trained lane detector 60 to identify lanes in real time.

[0049] Then (3200), the vehicle (60) uses the data characterizing the detected lanes y to inform the vehicle's steering and navigation systems. The vehicle's movement and speed are adjusted by the actuators (10) based on the detected lanes to maintain safe and accurate lane tracking. This completes this part of the process.

[0050] Fig.Figure 7 illustrates examples of lanes with lane markings (left column) and corresponding images without markings, showing the lane markings removed (right column). The lane markings to be removed are marked with an asterisk. Note that the other lane markings on the opposite side of the lane can be retained, as illustrated in this figure; that is, it is not necessary to remove all lane markings. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] DE 10 2010 062 129 B4

[0002]

Claims

[1] Computer-implemented method for training a lane detector (60) to detect a lane in a digital image (x), in particular based on pixel values ​​of the digital image (x), wherein the method comprises: Providing a first plural (T) of digital images with marked lanes that are labelled with lane markings; Obtaining a second plural (T') of digital images from the first plural of digital images by removing the lane markings from the digital images of the first plural of digital images, thereby generating corresponding digital images without markings; Providing data to characterize the lanes (y s ) in the digital images of the first plural of digital images (T); and Providing a data set (T') for training the lane detector (60), wherein the data set (T') comprises pairs of a digital image (x) and data that represent a basic truth of a lane (y). s ) characterize in the digital image (x), wherein the digital image (x) is taken from the second plural (T') of digital images, and the data characterize the lane identified in the corresponding digital image from the first plural (T) of digital images. [2] Method according to claim 1, wherein the step of providing data for characterizing lanes (y s ) in the digital images of the first plural (T) of digital images includes the identification of lanes in the digital images (x) of the first plural (T) of digital images (x) using a lane marking detector. [3] Method according to claim 2, wherein the step of identifying lanes in the digital images (x) of the first plurality (T) of digital images with the lane marking detector comprises identifying lane markings in the digital images and identifying lanes depending on the identified lane markings. [4] Method according to any one of claims 1 to 3, wherein the step of removing the lane markings is carried out by using a generative model, in particular a stable diffusion model, in particular by removing the lane markings with a texture that is approximately similar to the texture of an area near the lane marking in the image from which the lane marking is to be removed. [5] Method according to any one of claims 1 to 4, further comprising training the lane detector (60) with the data set. [6] Method according to claim 5, wherein the training comprises obtaining data for characterizing a lane (y) in a provided digital image (x) from the data set (T') using the lane detector (60), and adjusting parameters (ϕ) that characterize the behavior of the lane detector (60) depending on the obtained data for characterizing the lane (y) and the corresponding coupled data that define the basic truth of the lane (y) s ) characterize, from the set data set (T'). [7] Method for operating a vehicle (100) that is at least partially automated, comprising first training the lane detector (60) according to claim 5 and then providing images (x) that characterize the vehicle's (100's) surroundings; in particular, images (S) obtained by a camera (30) mounted on the vehicle (100), inputting the provided image (x) into the lane detector (60) to obtain data (y) that characterize the lane in the provided image (x), and operating the vehicle (100) depending on the detected lane characterized by the obtained data (y). [8] Data set (T') provided in the method according to claim 1. [9] Computer-readable storage medium (St2) in which the data set according to claim 8 is stored. [10] Computer program configured to perform the procedure according to one of the procedures 1 to 7. [11] Computer-readable storage medium in which the computer program according to claim 10 is stored. [12] Computer configured to perform the method according to any one of claims 1 to 6. [13] Lane detector (60) trained using the method according to claim 5.

Images