Method for generating a road surface recognition dataset based on a diffusion model
By generating road surface recognition datasets using a diffusion model-based approach, the problems of difficult dataset collection and insufficient diversity were solved, achieving efficient and automated dataset generation and improving the detection accuracy and generalization ability of autonomous driving models.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHONGQING UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-04-17
- Publication Date
- 2026-07-14
Smart Images
Figure CN122391782A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of target detection technology and relates to a method for generating hyperspectral road surface recognition datasets based on diffusion models and incorporating dynamic semantic detection technology. Background Technology
[0002] Road surface recognition is a core perception task in autonomous driving, advanced driver assistance systems (ADAS), and intelligent transportation systems. It requires systems not only to detect common targets such as vehicles, pedestrians, and traffic signs in real time, but also to accurately understand complex road scenarios, including but not limited to diverse and dynamically changing elements such as lane lines, traffic signals, road surface damage, temporary construction zones, water accumulation, and debris. In recent years, the YOLO (You Only Look Once) series has been widely used in various fields due to its excellent balance between speed and accuracy, achieving true real-time monitoring.
[0003] However, deep learning methods like YOLO rely on large-scale, high-quality labeled data. For example, in the field of autonomous driving, lane recognition systems built on YOLOv5 require training on large-scale public datasets to ensure the model's generalization ability, robustness, and final accuracy. Manually collecting datasets covering all target categories, various scales, different lighting conditions, and different angles requires significant manpower and effort. Furthermore, this often results in abundant data for common categories but scarce data for rare categories, hindering further improvements in model performance.
[0004] Therefore, how to reduce the complexity of data collection is a problem that needs to be solved in the current field. Summary of the Invention
[0005] In view of this, the purpose of this invention is to provide a method for generating road surface recognition datasets based on a diffusion model, so as to solve the problems of difficulty in collecting target detection datasets, uncontrollable data content, and lack of diversity.
[0006] To achieve the above objectives, the present invention provides the following technical solution: A method for generating road surface recognition datasets based on a diffusion model, the method comprising: Obtain road aggregate sample blocks, take hyperspectral images of the sample blocks, and save them after preprocessing; A portion of the preprocessed images is selected to form the first training set. At the same time, the selected images are labeled according to the road targets to be identified. Multiple LoRa models are trained based on the first training set and the label files to obtain multiple dedicated LoRa models. A portion of the preprocessed images is selected, and bounding boxes are annotated for the road surface targets to be identified to form a second training set. Multiple YOLO models are trained using the second training set to obtain multiple YOLO models specifically for single target detection. A large number of road surface images were generated using the SD model, and the trained LoRa and YOLO models were used to adjust each generated road surface image, resulting in a large number of road surface images constituting a road surface recognition dataset.
[0007] Furthermore, the road aggregate sample blocks contained asphalt, limestone, and basalt.
[0008] Furthermore, images with clear visuals and consistent style are selected from the preprocessed images and their sizes are standardized to form the first training set.
[0009] Furthermore, three first training sets are obtained. Based on the road surface targets to be identified, including asphalt, limestone, and basalt, the images in each first training set are labeled. Specifically, in one training set, the image style is labeled, and a corresponding txt file is generated and saved; in another training set, the limestone in the images is labeled, and a corresponding txt file is generated and saved to different paths; in the last training set, the basalt in the images is labeled, and a corresponding txt file is generated and saved to different paths.
[0010] Furthermore, multiple LoRa models are trained based on the first training set and the label files; among them, a LoRa model for learning the style is trained using the label files corresponding to the style and the first training set; a LoRa model for learning the limestone features is trained using the label files corresponding to limestone and the first training set; and a LoRa model for learning the basalt features is trained using the label files corresponding to basalt and the first training set.
[0011] Furthermore, images with clear details are selected from the preprocessed images. Based on the need to identify limestone and basalt targets on the road surface, bounding boxes are labeled for the limestone and basalt in the images to form a second training set. The second training set is then used to train YOLO models specifically for limestone detection and YOLO models specifically for basalt detection.
[0012] Furthermore, the trained LoRa and YOLO models are used to adjust each generated road surface image, including: First, for the SD model to generate road surface images containing limestone and basalt, the Lora model that learns the art style is called to adjust the overall features of the generated images, and cue words are added to the CLIP encoder to optimize the content of the generated images and obtain the base image. Secondly, adjust the size of the base image and add cue words describing the characteristics of limestone and basalt, including positive and negative cue words; Then, two processing units are constructed. One processing unit calls the Lora model for learning limestone features and the YOLO model specifically for limestone detection. The other processing unit calls the Lora model for learning basalt features and the YOLO model specifically for basalt detection. Finally, the adjusted base image is processed sequentially by two processing units to redraw the limestone and basalt in the base image. The processing unit's process is as follows: first, the YOLO model is called to perform a full image scan, identify specific targets and output bounding boxes, convert the bounding boxes into binary masks, and crop the image of the masked area of the original image as a redrawing reference; then, prompt words are read, and the SD model calls the corresponding LoRa model to perform local redrawing. During the full image scan, the detection confidence threshold of the YOLO model is adjusted to 0.2~0.3 to achieve multiple detections and multiple renderings.
[0013] When the SD model calls the corresponding Lora model for local redrawing, the Lora model serves as a constraint. By injecting the low-rank matrix trained by the Lora model into the attention layer of the SD model U-Net and fusing it with the original weights, the SD model is guided to generate an image with target aggregate features during local redrawing.
[0014] The beneficial effects of this invention are as follows: (1) Traditional dataset generation methods require manual collection and photography of large amounts of data or reliance on existing databases. In contrast, the diffusion model-based dataset generation method proposed in this invention can generate large datasets with only a small amount of data, which can significantly shorten the dataset preparation cycle, reduce manpower input, and improve efficiency. In addition, this invention uses YOLO to detect and save target locations during the dataset generation process, which can reduce the workload of manual annotation.
[0015] (2) By utilizing the LoRA model and the diffusion model, this invention can train and generate a large number of diverse datasets, which not only improves the accuracy and precision of the target detection model during detection, but also enhances the generalization ability of the model, thereby improving the overall performance.
[0016] (3) This invention adopts the LoRa model. The core principle of the LoRa model is to assume that the weight change when the model adapts to a new task is "low rank". A large weight matrix is decomposed into the product of two small matrices, which reduces the number of trainable parameters. With the reduction of parameters, the amount of VRAM and computation required for training also decreases sharply.
[0017] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description
[0018] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 This is a schematic diagram of the process for generating a road surface recognition dataset based on a diffusion model. Figure 2 Images used to test the object detection model; Figure 3 The test results are for the model trained using data generated based on embodiments of the present invention; Figure 4 The test results are for models trained using data generated without the embodiments of the present invention. Detailed Implementation
[0019] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.
[0020] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.
[0021] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.
[0022] An embodiment of the present invention provides a method for generating a road surface recognition dataset based on a diffusion model, the method comprising: 1. Obtain the original dataset.
[0023] In this embodiment, a sample block containing asphalt, limestone, and basalt was first acquired. Then, a hyperspectral image was taken of the sample block to collect the most raw sample data. Subsequently, the acquired hyperspectral image was opened in FigSpec software, and the RGB channel values were adjusted until the image content was displayed most clearly. Finally, the image was output as a standard PNG format image.
[0024] 2. Build a multi-expert adapter (Lora) library.
[0025] Lora is a lightweight adaptation module that injects a low-rank matrix into the attention layer of a pre-trained diffusion model, enabling Lora to train the model with a small amount of data. Therefore, in this embodiment, multiple Lora models are trained with a small amount of data to build a Lora expert database. These models learn the characteristics of the art style and the road surface targets to be identified, such as asphalt, limestone, and basalt. The art style model is used to control the generation of the initial base map, while the Lora models for limestone and basalt are used to control the generation of targets locally.
[0026] From the output PNG images, select those that are clear and have a consistent style, and then crop and convert them to a uniform 256 pixel format. 256-pixel images were used as the dataset for training the LoRa model. Here, "art style" refers to the overall style of the images, specifically the style of the road surface within the images.
[0027] When redrawing images of different aggregate targets, a YOLO detection model is used for preliminary target localization to ensure accurate target location identification. Since redrawing target objects requires multiple detections and multiple renderings, the requirements for the number of sample datasets and the confidence level of the results are not high. Therefore, a small number of high-resolution aggregate images can be used to annotate the bounding boxes of the targets, constructing a small-sample dedicated detection dataset to train YOLO models specifically for detecting limestone and basalt. Similarly, clear images are selected from the output PNG images as the training set for training the YOLO models.
[0028] 3. Prepare the label file.
[0029] This method mainly involves training a LoRa model and generating a large amount of road surface data based on the trained LoRa model, covering materials such as basalt, limestone, and asphalt.
[0030] During training, three dedicated LoRa models are required: one to learn stylistic features of the images, and the other two to learn features of basalt and limestone, respectively. Therefore, when labeling, different labels are assigned to the three cases based on their learning focus, so that the three dedicated LoRa models can obtain different low-rank increments during the backpropagation update process of model training. Thus, three datasets for training the LoRa models need to be prepared, with each dataset used to train a separate LoRa model.
[0031] The tagging method is as follows: For style labels, a labeler is used to automatically label all images in the dataset, generate corresponding txt files, check the label content and manually delete redundant parts, and then save the label files.
[0032] For the labels of limestone and basalt, a labeler is used to automatically label all images in the dataset. Redundant labels are manually deleted, new txt files are generated, and saved to different paths for later use.
[0033] 4. Train the LoRa model.
[0034] Once the data images and label files are ready, begin training the LoRa model and adjust the training parameters, including the low-rank matrix parameters and the training parameters. During the training of the LoRa model, a larger rank parameter setting results in more information being retained in the fine-tuning coefficient matrix; however, a larger training parameter setting also increases memory usage.
[0035] Therefore, based on actual computer performance, this embodiment sets the rank to 8 and adjusts the corresponding Lora scaling factor to 1, while ensuring good training model performance.
[0036] Research has found that LoRa model training does not require adjustments to every layer of the model; fine-tuning only the attention layer is sufficient to achieve full parameter fine-tuning. Therefore, the parameters that need to be modified during LoRa model training are limited to the attention layer, such as query and value. The batch size parameter is adjusted to 4 based on device performance. The number of training epochs is set to 10.
[0037] After configuring the training parameters, run the LoRa training script to begin training. During this process, the LoRa model continuously calculates the loss and updates the parameters. After training, the final model is obtained. The LoRa model that learns the image style is used as the base model, and the LoRa model that learns the characteristics of limestone and basalt is used as the constraint condition to generate data for the SD model.
[0038] 5. Image generation is performed based on the trained LoRa and YOLO models.
[0039] 1) Generation of the base image This stage aims to learn the style of the original data and generate a non-disruptive background structure, which is achieved by deploying the StableDiffusion WebUI.
[0040] First, the pre-trained model is loaded, which is applied to the U-Net layer of the SD model. The overall features of the generated image are controlled by adjusting the output of each layer of the U-Net.
[0041] Secondly, to further adjust the quality of the generated base image, cue words are added, including positive and negative cue words. Positive cue words describe the macroscopic background of the road surface but lack clear and specific aggregate type features, leaving semantic gaps for subsequent local reconstruction; for example, "road surface" means it contains no aggregate type features, only the overall road surface structure. Negative cue words are filled with objects to suppress generation, such as "basalt" and "limestone," to keep the road surface background as clean as possible. These cue words are applied to the CLIP encoder to optimize the content of the generated image.
[0042] Finally, the parameters for generating images are set, including the size, number, seed value, and CFG sale (unsupervised classifier ratio, controlling the degree to which the model follows instructions). In this embodiment, the size of the generated images is adjusted to 512. 512, with a quantity of 500 cards, a seed value adjusted to -1 to indicate random generation, and CFG of 7 to indicate a balance between following the hints and freedom.
[0043] 2) Automated generation After the base image is generated, a dedicated target detection model is used to detect the two major targets, limestone and basalt. Then, the corresponding LoRa model is called to redraw the local area and retain the corresponding coordinate data, simplifying the annotation process and realizing automated annotation and detection.
[0044] Specifically, the adaptive detection and repair module is invoked in the WebUI interface, and the detection model and LoRa model called by this module are configured. Since two targets, limestone and basalt, need to be generated, two processing units need to be constructed and configured separately. The detection model calls a pre-trained YOLO model, and the confidence threshold during detection is lowered to 0.2 to 0.3 to achieve multiple detections. In addition, the corresponding LoRa model is loaded; the YOLO model specifically for limestone detection corresponds to a LoRa model that has learned limestone features, and the YOLO model specifically for basalt detection corresponds to a LoRa model that has learned basalt features. The size of the generated base image is adjusted to 256. 256, keeping other parameters unchanged, add positive and negative prompts describing the target features.
[0045] After the two units processing different types of stone are configured, automated generation begins. The SD model first calls the YOLO model to scan the entire image, identify specific targets, output bounding boxes, convert the bounding boxes into binary masks, and save their coordinate positions. The masked area of the original image is cropped as a redrawing reference. The prompt words are read, and the SD model calls the corresponding LoRa model for local redrawing. The LoRa model acts as a constraint; its trained low-rank matrix is injected into the attention layer of the SD model's U-Net, fused with the original weights, thus guiding the SD model to generate images with the characteristics of the target aggregate during local redrawing. The attention layer is a key module in U-Net responsible for establishing the association between image regions and text descriptions, determining "which word in the text should be focused on at which location in the image." The two processing units execute sequentially, ultimately generating a large amount of high-quality image datasets.
[0046] 6. Verify the effectiveness of the dataset.
[0047] The generated data is used as the dataset for training the YOLO model, and is divided into three parts: a training set, a test set, and a validation set. The training set is used for training the YOLO model, the validation set is used for model validation, and the test set is used to test the model's performance. Similarly, YOLO models trained on datasets not composed of the data generated in this embodiment are tested.
[0048] Compare the test results of YOLO models obtained using different training data, where the images used for testing are as follows: Figure 2 As shown, the test results for different models are as follows: Figure 3 and Figure 4 As shown. Figure 3 The test results are for the model trained using the data generated in this embodiment. Figure 4 The test results for the model trained using the data generated in this embodiment show that the model trained using the data generated in this embodiment performs better than the model trained using the data generated in this embodiment.
[0049] In summary, this invention provides a method for generating road surface recognition datasets based on a diffusion model. By utilizing the LoRa model and the diffusion model, a large number of diverse datasets can be generated, which not only improves the accuracy and precision of the target detection model during detection, but also enhances the model's generalization ability.
[0050] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A method for generating road surface recognition datasets based on a diffusion model, characterized in that, Obtain road aggregate sample blocks, take hyperspectral images of the sample blocks, and save them after preprocessing; A portion of the preprocessed images is selected to form the first training set, and the selected images are labeled according to the road targets to be identified. Multiple LoRa models are trained based on the first training set and label files to obtain multiple dedicated LoRa models; A portion of the preprocessed images is selected, and bounding boxes are annotated for the road surface targets to be identified to form a second training set; Multiple YOLO models were trained using the second training set to obtain multiple YOLO models specifically designed for single object detection; A large number of road surface images were generated using the SD model, and the trained LoRa and YOLO models were used to adjust each generated road surface image, resulting in a large number of road surface images constituting a road surface recognition dataset.
2. The method according to claim 1, characterized in that, The road aggregate sample blocks contain asphalt, limestone, and basalt.
3. The method according to claim 1, characterized in that, Images with clear visuals and consistent style were selected from the preprocessed images and their sizes were standardized to form the first training set.
4. The method according to claim 3, characterized in that, Obtain three sets of the first training set. Based on the road surface targets to be identified, including asphalt, limestone, and basalt, label the images in each set of the first training set. Specifically, in one training set, label the image style, generate corresponding txt files, and save them; in another training set, label the limestone in the images, generate corresponding txt files, and save them to different paths; in the last training set, label the basalt in the images, generate corresponding txt files, and save them to different paths.
5. The method according to claim 4, characterized in that, Multiple LoRa models were trained based on the first training set and the label files. Among them, a LoRa model for learning the style was trained using the label files corresponding to the style and the first training set; a LoRa model for learning the limestone features was trained using the label files corresponding to limestone and the first training set; and a LoRa model for learning the basalt features was trained using the label files corresponding to basalt and the first training set.
6. The method according to claim 1, characterized in that, Select clear images from the preprocessed images. Based on the need to identify limestone and basalt targets on the road surface, mark the bounding boxes of limestone and basalt in the images to form a second training set. Use the second training set to train YOLO models specifically for limestone detection and YOLO models specifically for basalt detection.
7. The method according to claim 1, characterized in that, The process of adjusting each generated road surface image by calling the trained LoRa and YOLO models includes: First, for the SD model to generate road surface images containing limestone and basalt, the Lora model that learns the art style is called to adjust the overall features of the generated images, and cue words are added to the CLIP encoder to optimize the content of the generated images and obtain the base image. Secondly, adjust the size of the base image and add cue words describing the characteristics of limestone and basalt, including positive and negative cue words; Then, two processing units are constructed. One processing unit calls the Lora model for learning limestone features and the YOLO model specifically for limestone detection. The other processing unit calls the Lora model for learning basalt features and the YOLO model specifically for basalt detection. Finally, the adjusted base image is processed sequentially by two processing units to redraw the limestone and basalt in the base image. The processing unit first calls the YOLO model to perform a full image scan, identifies specific targets and outputs bounding boxes, converts the bounding boxes into binary masks, and extracts the image of the mask area of the original image as a redrawing reference. Then, it reads the prompt words, and the SD model calls the corresponding Lora model to perform local redrawing.
8. The method according to claim 7, characterized in that, When performing a full-image scan, the detection confidence threshold of the YOLO model is adjusted to 0.2~0.3 to enable multiple detections and multiple renderings.
9. The method according to claim 7, characterized in that, When the SD model calls the corresponding Lora model for local redrawing, the Lora model serves as a constraint. By injecting the low-rank matrix obtained from training the Lora model into the attention layer of the SD model's U-Net and fusing it with the original weights, the SD model is guided to generate an image with the target aggregate features during local redrawing.