A high-fidelity lane line image generation method without re-labeling
By combining image preprocessing and multi-stage diffusion generation techniques with a pre-trained model, high-fidelity lane line images are directly generated, solving the problem of performance degradation of lane detection models under adverse weather and complex lighting conditions in existing technologies, and achieving efficient data generation without re-annotation or model fine-tuning.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ANHUI UNIV
- Filing Date
- 2025-08-14
- Publication Date
- 2026-08-04
AI Technical Summary
Existing lane detection models exhibit significant performance degradation under adverse weather and complex lighting conditions. They lack sample data, and existing data generation methods require re-labeling, model fine-tuning, or involve high computational overhead, making it difficult to meet real-time requirements.
By fusing lane line semantics and edge information through image preprocessing, and using a pre-trained diffusion model for multi-stage diffusion generation, high-fidelity lane line images are generated. This includes a VAE model, a back-diffusion process controlled by ControlNet and InstructPix2Pix, and combined with the EuLer sampler, data for various extreme scenarios are directly generated.
The generated images accurately preserve lane line semantics in extreme scenarios, reducing the cost of re-annotation and model fine-tuning. They support one-time generation of multiple extreme scenarios, significantly enhancing the robustness of the model in extreme environments.
Smart Images
Figure CN121010956B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of traffic image processing technology, specifically a method for generating high-fidelity lane line images without re-annotation. Background Technology
[0002] Lane detection is a critical task in autonomous driving and advanced driver assistance systems (ADAS), and its performance directly affects the vehicle's positioning accuracy and driving safety. Currently, mainstream lane detection models are mainly trained on public datasets such as CULane and TuSimple. However, the samples in these datasets are mostly concentrated in normal scenarios such as sunny days and daytime, lacking samples under adverse weather conditions such as rain, snow, and fog, as well as complex lighting conditions such as night and dusk.
[0003] This data deficiency leads to a significant performance degradation of existing detection models in extreme environments, posing serious security risks. Although data can be supplemented through manual collection, the low incidence of severe weather and the high cost of re-labeling make it difficult to achieve in the short term. Existing technologies, such as some methods that generate extreme scene data through image inpainting or domain transformation, require retraining for each weather type, and the additional image inpainting process increases computational overhead, making it difficult to meet real-time requirements. Other methods based on diffusion models can generate realistic data, but still require re-labeling and model fine-tuning, incurring additional costs.
[0004] Therefore, there is an urgent need for a method that can efficiently generate high-fidelity lane line data under severe weather and complex lighting conditions without requiring re-annotation and model fine-tuning. Summary of the Invention
[0005] The purpose of this invention is to address the shortcomings of existing lane detection datasets, such as the scarcity of samples under adverse weather (rain, snow, fog) and complex lighting (night, dusk) conditions, and the fact that existing data generation methods require re-labeling, model fine-tuning, or have high computational overhead. This invention provides a high-fidelity lane line image generation method that does not require re-labeling to solve the above problems.
[0006] To achieve the above objectives, the technical solution of the present invention is as follows:
[0007] A method for generating high-fidelity lane line images without re-annotation includes the following steps:
[0008] Image preprocessing: The original dataset is processed to generate a combined image that integrates lane line semantics and image edge information;
[0009] The first stage of diffusion generation is performed: the original image and the combined image are mapped to the latent space through the VAE model, and the first stage latent space map is generated through the ControlNet controlled by Canny information, positive prompt words, and negative prompt words, after passing through the first reverse diffusion module.
[0010] Second-stage diffusion generation: The first-stage latent space graph is input into the ControlNet controlled by InstructPix2Pix, and after passing through the second back diffusion module, the second-stage latent space graph is generated.
[0011] Pixel space transformation: The pixel space of the first-stage latent space map or the second-stage latent space map is transformed using the VAE model to obtain the final high-fidelity lane line generated image.
[0012] The image preprocessing includes the following steps:
[0013] The original CULane dataset is segmented into white and yellow lane line regions from the original image using a color thresholding algorithm to generate a color-thresholded image;
[0014] Generate a mask image based on the lane line annotations in the original CULane dataset;
[0015] Perform a bitwise AND operation on the color threshold image and the mask image to filter out non-lane line regions and obtain the real lane line image;
[0016] The Canny edge detection algorithm is applied to the original CULane dataset to obtain edge images;
[0017] Perform a bitwise OR operation between the edge image and the real lane line image to generate a combined image.
[0018] The first-stage diffusion generation includes the following steps:
[0019] The combined image is transformed into the latent space using a VAE model encoder, as shown in the following formula:
[0020] c f =ε1(c0),
[0021] z = ε2(x0),
[0022] Where x0 is the original image, c0 is the preprocessed combined image, ε1 is the encoder composed of convolutional layers, ε2 is the VAE model encoder, z is the representation of the original image in the latent space, and c f This represents the combined image in the latent space.
[0023] Further noise is added to z to obtain the original noisy image z. T The formula is as follows:
[0024]
[0025] in, Let be a constant, and γ be random noise following a standard normal distribution;
[0026] Positive and negative prompts are processed using a pre-trained CLIP model τ θ Encode the prompt word conditional code c. t The formula is:
[0027] c t =τ θ (Prompt word);
[0028] c f z T c t The input is fed into a pre-trained ControlNet, controlled by Canny information, and a first-stage backpropagation is performed to obtain the first-stage latent space map z0.
[0029] z0 = RD1(ControlNet Canny (c f ,z T ),c t ,z T ),
[0030] Among them, ControlNet Canny It is a pre-trained ControlNet controlled using Canny information, and RD1 is the first backpropagation module;
[0031] For each time step in this process, the following mathematical expression is used:
[0032] y c =F(x;Θ)+Z(F(x+Z(c;Θ)) z1 );Θ c );Θ z2 ),
[0033] Where c is the conditional vector, Z represents the zero convolutional layer, and Θ c These are the trained parameters copied from the original Stable Diffusion network, Θ. z1 and Θ z2 These are the parameters of two zero-convolutional layers, y c Θ is the output image feature map of ControlNet, F is the original Stable Diffusion network model, x is the input image feature map, and Θ is the fixed parameter of the original Stable Diffusion network.
[0034] The second-stage diffusion generation includes the following steps:
[0035] The first-stage latent space graph z0 is input into the ControlNet controlled by InstructPix2Pix and then passes through the second reverse diffusion module.
[0036] Performing the second-stage backdiffusion yields the two-stage latent space graph z′0, with the following formula:
[0037] z′0=RD2(ControlNet iP2P (x f ,z0),z0),
[0038] Among them, ControlNet iP2P The system is a pre-trained ControlNet controlled using InstructPix2Pix information, and RD2 is the second back-diffusion module; for each time step in this process, it is the same as the first stage.
[0039] The pixel space conversion includes the following steps:
[0040] Obtain pixel space conversion requirements.
[0041] When the target scene is snowy, rainy, or foggy, the first-stage latent space map z0 is transformed to pixel space through the VAE model to obtain the target scene image;
[0042] When the target scene is at night or dusk, the two-stage latent space map z′0 is transformed to the pixel space using a VAE model to obtain the target scene image. This process is represented by the following formula:
[0043] For snowy, rainy, and foggy days:
[0044] z′0=z0,
[0045] For snowy days, rainy days, foggy days, nights, and dusk:
[0046] x′0=D(z′0),
[0047] Here, x′0 is the final generated graph.
[0048] The first backdiffusion module RD1 or the second backdiffusion module RD2 both utilize the principle and mechanism of backdiffusion in the EuLer sampler, as detailed in the following formula:
[0049] x t-Δt =x t -Δt·ε θ (x t ,t),
[0050] Where, x t It is the image at the current time t, ε θ (x t x(t) represents the noise predicted by the model, Δt is the time step, and x is the noise. t-ΔtIt is the image at time t-Δt; by iterating continuously until t=0, a generated graph with stable characteristics is obtained, namely the one-stage latent space graph z0 or the two-stage latent space graph z′0.
[0051] A computer-readable storage medium storing a computer program that, when executed by a processor, enables a method for generating high-fidelity lane line images without re-annotation.
[0052] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, which, when executed by the processor, enables a method for generating high-fidelity lane line images without re-annotation.
[0053] Beneficial effects
[0054] The present invention provides a high-fidelity lane line image generation method that does not require re-annotation. Compared with the prior art, this method integrates lane line masks and edge information, and combines multi-stage diffusion control to generate images that can accurately preserve lane line semantics even in extreme scenarios, thus solving the problem of lane line information distortion in existing methods.
[0055] This invention utilizes pre-trained models and semantic preservation mechanisms to generate data that can directly use the original annotations without additional annotations or model fine-tuning, significantly reducing costs. It also supports the generation of multiple extreme scenarios at once, avoiding repeated training for single weather conditions, and the generation process is based on ComfyUI to achieve an efficient workflow design.
[0056] Experiments show that when training the state-of-the-art model CLRNet using the dataset generated by this invention, the overall mF1 score is improved by 20.87%, and the F1@50 scores for snowy and foggy scenes are improved by 38.8% and 26.84% respectively, significantly enhancing the model's robustness in extreme environments. Attached Figure Description
[0057] Figure 1 This is a sequence diagram of the method of the present invention;
[0058] Figure 2 This is a flowchart of a two-stage reverse diffusion process;
[0059] Figure 3 Flowchart of lane line image preprocessing;
[0060] Figure 4 This is a diagram illustrating the experimental results. Detailed Implementation
[0061] To provide a better understanding of the structural features and effects achieved by the present invention, a detailed description is provided below, accompanied by preferred embodiments and accompanying drawings:
[0062] like Figure 1 and Figure 2 As shown, this invention presents a high-fidelity lane line image generation method without re-annotation, proposing a high-fidelity generation framework based on a diffusion model. This invention extracts and preprocesses the CULane dataset, designs the workflow using ComfyUI, and controls the generation process through prompts and Canny information. Five new categories are generated under the original normal category: snow, rain, fog, night, and dusk. A new benchmark dataset for severe weather and lighting conditions is constructed without re-annotation or fine-tuning. Extensive comparative experiments and ablation studies on various baseline models demonstrate the effectiveness of the proposed framework.
[0063] Includes the following steps:
[0064] The first step, image preprocessing: The original dataset is processed to generate a combined image that integrates lane line semantics and image edge information. For example... Figure 3 As shown, the mask image is not used directly because the mask consists of continuous lines, while lane lines may be dashed, which will affect the generation result. Secondly, the Canny image is combined with the real lane image instead of using the Canny image alone because in some cases, Canny edge detection cannot accurately capture the edges of lane lines.
[0065] (1) The white and yellow lane line regions were segmented from the original image using the color thresholding algorithm to generate a color thresholded image.
[0066] (2) Generate a mask image based on the lane line annotations in the original CULane dataset.
[0067] (3) Perform a bitwise AND operation on the color threshold image and the mask image to filter out non-lane line areas and obtain the real lane line image.
[0068] (4) Apply the Canny edge detection algorithm to the original CULane dataset to obtain edge images.
[0069] (5) Perform a bitwise OR operation on the edge image and the real lane line image to generate a combined image.
[0070] The second step involves the first-stage diffusion generation: the original image and the combined image are mapped to the latent space using a VAE model, and then, through the ControlNet controlled by Canny information, positive prompts, and negative prompts, a first-stage latent space map is generated via the first back-diffusion module. Here, the entire process uses a pre-trained model without fine-tuning or retraining.
[0071] (1) Transform the combined image into the latent space using a VAE model encoder, as shown in the following formula:
[0072] c f =ε1(c0),
[0073] z = ε2(x0),
[0074] Where x0 is the original image, c0 is the preprocessed combined image, ε1 is the encoder composed of convolutional layers, ε2 is the VAE model encoder, z is the representation of the original image in the latent space, and c f This represents the combined image in the latent space.
[0075] (2) Further noise is added to z to obtain the original noisy image z. T The formula is as follows:
[0076] ,
[0077] in, Let be a constant, and γ be random noise that follows a standard normal distribution.
[0078] (3) Positive and negative prompts are processed using a pre-trained CLIP model τ θ Encode the prompt word conditional code c. t The formula is:
[0079] c t =τ θ (Prompt word)
[0080] (4) c f z T c t The input is fed into a pre-trained ControlNet, controlled by Canny information, and a first-stage backpropagation is performed to obtain the first-stage latent space map z0.
[0081] z0 = RD1(ControlNet Canny (c f ,z T ),c t ,z T ),
[0082] Among them, ControlNet Canny It is a pre-trained ControlNet controlled using Canny information, and RD1 is the first backpropagation module;
[0083] For each time step in this process, the following mathematical expression is used:
[0084] y c=F(x;Θ)+Z(F(x+Z(c;Θ)) z1 );Θ c );Θ z2 ),
[0085] Where c is the conditional vector, Z represents the zero convolutional layer, and Θ c These are the trained parameters copied from the original Stable Diffusion network, Θ. z1 and Θ z2 These are the parameters of two zero-convolutional layers, y c Θ is the output image feature map of ControlNet, F is the original Stable Diffusion network model, x is the input image feature map, and Θ is the fixed parameter of the original Stable Diffusion network.
[0086] The third step is to generate the second-stage diffusion graph: the first-stage latent space graph is input into the ControlNet controlled by InstructPix2Pix, and after passing through the second back diffusion module, the second-stage latent space graph is generated.
[0087] (1) Input the first-stage latent space graph z0 into ControlNet controlled by InstructPix2Pix, and then pass it through the second reverse diffusion module.
[0088] (2) Perform the second-stage back diffusion to obtain the two-stage latent space graph z′0, with the following formula:
[0089] z′0=RD2(ControlNet iP2P (x f ,z0),z0),
[0090] Among them, ControlNet iP2P The system is a pre-trained ControlNet controlled using InstructPix2Pix information, and RD2 is the second back-diffusion module; for each time step in this process, it is the same as the first stage.
[0091] Here, the principle and mechanism of back diffusion in the EuLer sampler are used in both the first back diffusion module RD1 and the second back diffusion module RD2. The specific formula is as follows:
[0092] x t-Δt =x t -Δt·ε θ (x t ,t),
[0093] Where, x t It is the image at the current time t, ε θ (x tx(t) represents the noise predicted by the model, Δt is the time step, and x is the noise. t-Δt It is the image at time t-Δt; by iterating continuously until t=0, a generated graph with stable characteristics is obtained, namely the one-stage latent space graph z0 or the two-stage latent space graph z′0.
[0094] The fourth step is to perform pixel space transformation: the first-stage latent space map or the second-stage latent space map is transformed into pixel space using the VAE model to obtain the final high-fidelity lane line generated image.
[0095] Obtain pixel space conversion requirements.
[0096] When the target scene is snowy, rainy, or foggy, only the first-stage latent space map z0 is transformed to the pixel space through the VAE model to obtain the target scene image;
[0097] When the target scene is at night or dusk, the two-stage latent space map z′0 is transformed to the pixel space using a VAE model to obtain the target scene image. This process is represented by the following formula:
[0098] For snowy, rainy, and foggy days:
[0099] z′0=z0,
[0100] For snowy days, rainy days, foggy days, nights, and dusk:
[0101] x′0=D(z′0),
[0102] Here, x′0 is the final generated graph.
[0103] The benchmark dataset generated based on this method contains 30,000 images across 6 categories (normal, snow, rain, fog, night, and dusk). The training, validation, and test sets are divided in a 7:1:2 ratio, as shown in the table below.
[0104]
[0105] Taking CLRNet (ResNet-18) as an example, the performance difference between "not using generated data" and "using generated data" is compared, and the results are shown in the table below:
[0106]
[0107]
[0108] As can be seen, after using this method to generate data, CLRNet's overall F1@50 improved from 68.74 to 88.49, and its overall mF1 improved from 42.16 to 63.03, validating the framework's effective improvement on model performance. In the "Snow" category, F1@50 improved from 46.08 to 84.88, and in the "Fog" category from 58.54 to 85.38, indicating that the severe weather data generated by this method can effectively compensate for the model's performance shortcomings in extreme scenarios. In the "Night" category, F1@50 improved from 70.09 to 91.59, and in the "Dusk" category from 79.95 to 91.99, indicating that the framework generates high-quality data for scenes with changing lighting conditions, enhancing the model's adaptability to low-light and backlight environments.
[0109] like Figure 4 As shown, this set of images compares road scenes under different weather and lighting conditions, demonstrating the diverse environments covered by the dataset and helping to validate the model's ability to generate lane lines in complex scenarios. They are as follows:
[0110] (a) normal,
[0111] This scene presents a road view under clear daylight conditions, with ample lighting and clear visibility. The details of roads, vehicles, and buildings are clearly defined, serving as a basic reference scene that can be used to compare image differences under adverse conditions and verify the model's performance in an undisturbed environment.
[0112] (b) at night,
[0113] The scene depicts a nighttime road, primarily illuminated by streetlights and shop lights, resulting in low overall brightness and a dark environment. Details in dark areas are easily lost, testing the model's ability to identify lane lines and distinguish road elements in low light conditions, thus meeting the requirements for generating nighttime scenes.
[0114] (c) Dusk,
[0115] At dusk, the sky is warm-toned, the light is soft with distinct transitions between light and shadow, creating a unique atmosphere. The angle and color of the light differ from those during the day, and the roads and buildings cast special shadows due to the light, which is used to verify the robustness of the model under complex lighting conditions.
[0116] (d) It snows.
[0117] The scene depicts a snowy landscape where the image is covered in white snow, obscuring roads, vehicles, and buildings, reducing visibility, and the white snow can easily be confused with lane lines. This test was used to evaluate the model's ability to generate lane lines in high-noise, low-contrast environments.
[0118] (e) It's raining.
[0119] In a rainy scene, the image shows traces of rainwater, and the light is scattered due to refraction from the raindrops, resulting in an overall dark image with blurred details. This simulates visual interference in rainy conditions to verify the model's performance under dynamic water interference and light scattering environments.
[0120] (f) There is fog.
[0121] In foggy scenes, the fog causes the image to be hazy, distant objects to be blurry, contrast to be low, and details to be severely lost. This test was used to evaluate the model's ability to generate lane lines and road elements in low-visibility, high-blur environments, a typical challenging scenario in severe weather.
[0122] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.
Claims
1. A method for generating high-fidelity lane line images without re-annotation, characterized in that, Includes the following steps: 11) Image preprocessing: Process the original dataset to generate a combined image that integrates lane line semantics and image edge information; 12) Perform the first stage of diffusion generation: Map the original image and the combined image to the latent space through the VAE model, and generate the first stage latent space map through the ControlNet controlled by Canny information, positive prompt words, and negative prompt words, after passing through the first reverse diffusion module. The first stage of diffusion generation is performed. Includes the following steps: 121) The combined image is transformed into the latent space through the VAE model encoder, as shown in the following formula: ， ， in, For the original image, The combined image obtained after preprocessing, An encoder consisting of convolutional layers. For VAE model encoder, This represents the original image in the latent space. This represents the combined image in the latent space. 122) To Further noise is added to obtain the original noisy image. The formula is as follows: ， in, It is a constant. For random noise that follows a standard normal distribution; 123) The positive and negative prompts are processed using a pre-trained CLIP model. Encode the prompt words to obtain conditional codes. The formula is: ； 124) will , , The input is fed into a pre-trained ControlNet, controlled by Canny information, and a first-stage backpropagation is performed to obtain the first-stage latent space map z0. ， in, For pre-trained ControlNet and control using Canny information, This is the first reverse diffusion module; For each time step in this process, the following mathematical expression is used: ， in, It is a condition vector. Indicates a zero convolutional layer. These are trained parameters copied from the original Stable Diffusion network. and These are the parameters of two zero-convolutional layers. It is the feature map of the image output by ControlNet. It is the original Stable Diffusion network model. It is the feature map of the input image. These are the fixed parameters of the original Stable Diffusion network; 13) Perform the second-stage diffusion generation: Input the first-stage latent space graph into ControlNet controlled by InstructPix2Pix, and generate the second-stage latent space graph through the second back diffusion module; The second-stage diffusion generation includes the following steps: 131) Input the first-stage latent space graph z0 into the ControlNet controlled by InstructPix2Pix, and then pass it through the second back diffusion module; 132) Perform the second-stage backdiffusion to obtain the two-stage latent space graph. The formula is: ， in, For pre-trained ControlNet and control using InstructPix2Pix information, This is the second reverse diffusion module; for each time step in this process, it is the same as the first stage. 14) Perform pixel space transformation: Perform pixel space transformation on the first-stage latent space map or the second-stage latent space map using the VAE model to obtain the final high-fidelity lane line generated image.
2. The method for generating high-fidelity lane line images without re-annotation according to claim 1, characterized in that, The image preprocessing includes the following steps: 21) Using a color thresholding algorithm, white and yellow lane line regions were segmented from the original image of the original CULane dataset to generate a color-thresholded image; 22) Generate a mask image based on the lane line annotations in the original CULane dataset; 23) Perform a bitwise AND operation on the color threshold image and the mask image to filter out non-lane line areas and obtain the real lane line image; 24) Apply the Canny edge detection algorithm to the original CULane dataset to obtain edge images; 25) Perform a bitwise OR operation on the edge image and the real lane line image to generate a combined image.
3. The method for generating high-fidelity lane line images without re-annotation according to claim 1, characterized in that, The pixel space conversion includes the following steps: Obtain pixel space conversion requirements. When the target scenario is a snowy, rainy, or foggy day, let the first-stage hidden space graph... The target scene image is obtained by converting it to pixel space using a VAE model. When the target scene is night or dusk, the two-stage latent space graph will be used. The target scene image is obtained by transforming it to pixel space using a VAE model. This process is represented by the following formula: For snowy, rainy, and foggy days: ， For snowy days, rainy days, foggy days, nights, and dusk: ， in, This is the final generated image.
4. The method for generating high-fidelity lane line images without re-annotation according to claim 1, characterized in that, The first reverse diffusion module Or the second reverse diffusion module The process utilizes the principle and mechanism of back diffusion in the EuLer sampler, as detailed in the following formulas: ， in, It is the current time. The image, It is noise in the model prediction. It is the time step. yes A timeline; through continuous iteration, until... When the time stops, a generative graph with stable characteristics is obtained, namely the one-stage latent space graph. Or two-stage hidden space diagram .
5. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, enables the generation of a high-fidelity lane line image without re-annotation as described in any one of claims 1-4.
6. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can implement the high-fidelity lane line image generation method without re-annotation as described in any one of claims 1-4.