All-weather automatic driving data generation method and system based on diffusion model
By generating new image-label pairs using a diffusion model, the problem of insufficient coverage in all-weather autonomous driving datasets is solved, achieving low-cost and efficient data expansion and improving the model's generalization ability under various weather conditions.
Patent Information
- Application Number
- CN202310981806.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-07
- Publication Date
- 2025-12-19
- Estimated Expiration
- 2043-08-07
AI Technical Summary
Existing technologies struggle to create large-scale datasets that cover a wide range of weather conditions, resulting in insufficient generalization ability of deep learning models in all-weather autonomous driving.
A diffusion model is adopted to form obstacle masks by segmenting the existing dataset and generating new image-label pairs using text descriptions of different meteorological scenes, thereby expanding the scale of the dataset and the coverage of meteorological scenes.
The low-cost and rapid generation process significantly increases the scale of the dataset and the coverage of scenarios, improves the generalization ability of deep learning models in diverse weather environments, and supports all-weather autonomous driving.
Smart Images
Figure CN117079248B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the field of image processing, and particularly relates to a kind of all-weather automatic driving data generation method and system based on diffusion model. BACKGROUND
[0002] Deep learning has become a key technology to realize high-level automatic driving, and plays a central role in automatic driving core tasks such as obstacle detection, semantic segmentation, depth estimation, lane line detection and drivable area segmentation. Large-scale and high-quality data sets are one of the key driving forces for the success of deep learning technology. High-level automatic driving requires automatic driving vehicles to have the ability to work all-weather, that is, to adapt to complex and variable weather conditions (such as seasons, day and night, sunny, rain, snow, fog, dust, etc.). Having a large-scale data set covering all these conditions is the key to realizing all-weather automatic driving. However, the production of data sets is often expensive, especially for some rare weather conditions. Even in some environmental conditions, it may affect or damage the vehicle-mounted sensor, resulting in extremely high cost for collecting these scene data. In addition, the data labeling process requires a large amount of manpower, and large-scale data labeling is very expensive and time-consuming. Therefore, it is extremely expensive to produce a large-scale data set covering a wide range of weather conditions, which cannot be completed in a short period of time. How to solve the problem of limited coverage of the actually collected data set and make the deep learning model have stronger generalization ability is a technical problem to be solved at present. SUMMARY
[0003] The purpose of the present application is to solve the defect that it is difficult to produce a large-scale data set covering a wide range of weather conditions in the prior art, and to provide a kind of all-weather automatic driving data generation method, system, medium and equipment based on diffusion model. The present application can greatly improve the scale of data set and the coverage of various weather conditions (which can be called scene coverage), thereby enhancing the generalization ability of deep learning model in automatic driving, and further improving the all-weather working ability of automatic driving.
[0004] The specific technical solutions adopted by the present application are as follows:
[0005] In the first aspect, the present application provides a kind of all-weather automatic driving data generation method based on diffusion model, which includes:
[0006] S1, obtaining an existing data set corresponding to a target task in automatic driving, the existing data set is composed of a series of image-label pairs;
[0007] S2, performing target segmentation on the image in each image-label pair in the existing data set, and segmenting out the obstacles in the image that do not belong to the environment background to form an obstacle mask;
[0008] S3, taking the image-label pair in the existing data set, the obstacle mask corresponding to the image-label pair, and the text description of the target meteorological scene as the input of the trained diffusion model, generating new image-label pairs under different meteorological scenes based on different existing image-label pairs by constantly replacing different meteorological scene text descriptions, thereby expanding the size and meteorological scene coverage of the data set.
[0009] As a preferred embodiment of the first aspect, the target task is an obstacle detection task, a semantic segmentation task, a depth estimation task, a lane line detection task, or a drivable area segmentation task in the automatic driving process.
[0010] As a preferred embodiment of the first aspect, the target task is an obstacle detection task in the automatic driving process; the label in the image-label pair is an obstacle bounding box in the image.
[0011] The obstacle mask is input into the diffusion model to limit the obstacle region in the source image as an unchangeable region and the remaining regions as changeable regions during the model generation process, so that the label of the source image can still form a new image-label pair with the newly generated image.
[0012] As a preferred embodiment of the first aspect, the obstacle detection box is a planar two-dimensional detection box or a spatial three-dimensional detection box.
[0013] As a preferred embodiment of the first aspect, the target task is a semantic segmentation task in the automatic driving process; the label in the image-label pair is a pixel-level semantic classification map corresponding to the image.
[0014] The obstacle mask is input into the diffusion model to limit the obstacle region in the source image as an unchangeable region and the remaining regions as changeable regions during the model generation process, and the pixel-level semantic classification map of the source image is also input into the diffusion model as a label along with the source image, thereby preserving the obstacle information in the source image while ensuring that the generated new image has the same pixel-level classification as the source image, so that the label of the source image can still form a new image-label pair with the newly generated image.
[0015] As a preferred embodiment of the first aspect, when expanding the size and scene coverage of the existing data set, the meteorological scenes that need to be expanded in the existing data set or the meteorological scenes that do not exist in the existing data set are all taken as target scenes, and a text description of the scene is constructed for each target scene, and new image-label pairs are generated by inputting different existing image-label pairs and obstacle masks into the diffusion model.
[0016] As a preferred embodiment of the first aspect, the obstacles include vehicles and people in the image.
[0017] In a second aspect, the present application provides an all-weather autonomous driving data generation system based on a diffusion model, comprising:
[0018] A data set acquisition module is configured to acquire an existing data set corresponding to a target task in autonomous driving, wherein the existing data set is composed of a series of image-label pairs.
[0019] A preprocessing module is configured to perform target segmentation on an image in each image-label pair in the existing data set, segment out obstacles in the image that do not belong to the environment background, and form an obstacle mask.
[0020] A data generation module is configured to take the image-label pair in the existing data set, the obstacle mask corresponding to the image-label pair, and a textual description of a target meteorological scene as inputs of a trained diffusion model, generate new image-label pairs under different meteorological scenes based on different existing image-label pairs by constantly changing different meteorological scene textual descriptions, and thereby expand the scale and meteorological scene coverage of the data set.
[0021] In a second aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the diffusion model-based all-weather autonomous driving data generation method of any one of the above first aspects is implemented.
[0022] In a third aspect, the present application provides a computer electronic device comprising a memory and a processor.
[0023] The memory is configured to store a computer program.
[0024] The processor is configured to implement the diffusion model-based all-weather autonomous driving data generation method of any one of the above first aspects when executing the computer program.
[0025] Compared with the prior art, the present application has the following beneficial effects:
[0026] 1. The present application applies a diffusion model to all-weather autonomous driving data generation, and by controlling scene textual descriptions, it can generate and expand road data under various meteorological conditions, thereby improving the generalization of deep learning models under various meteorological environments (such as four seasons, day and night, rain, snow, fog, dust, etc.), and providing safety for autonomous driving.
[0027] 2、Traditional methods usually acquire data sets by collecting and manually annotating, which is extremely expensive. It is extremely costly to obtain large-scale and widely-scene-covered data, which requires long-term investment and accumulation. The present application can generate a large amount of new data with labels based on a small-scale existing data set through a generation method, greatly increase the size of the data set and expand the scene coverage of the data set. The generation process can be automated, the cost is very low and the generation speed is extremely fast. BRIEF DESCRIPTION OF DRAWINGS
[0028] Figure 1 A schematic diagram of the steps of the all-weather automatic driving data generation method based on the diffusion model of the present application is shown in the figure.
[0029] Figure 2 A data generation schematic diagram for the obstacle detection task in the embodiment of the present application is shown in the figure.
[0030] Figure 3 A data generation schematic diagram for the semantic segmentation task in the embodiment of the present application is shown in the figure.
[0031] Figure 4 A module schematic diagram of the all-weather automatic driving data generation system based on the diffusion model of the present application is shown in the figure. DETAILED DESCRIPTION
[0032] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. In the following description, a large number of specific details are set forth in order to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the spirit of the present application, therefore the present application is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present application can be combined accordingly without conflict.
[0033] As shown in the figure, in a preferred embodiment of the present application, an all-weather automatic driving data generation method based on a diffusion model is provided, which includes the following steps: Figure 1
[0034] S1, obtaining an existing data set corresponding to a target task in automatic driving, the existing data set being composed of a series of image-label pairs.
[0035] In the above S1 step, data preparation is mainly performed. However, it should be noted that the data preparation, i.e. preparing a certain number of image-label pairs (source dataset) with artificial labeling according to the task requirements (such as obstacle detection, semantic segmentation, depth estimation, lane line detection, drivable area segmentation, etc.), can be obtained by collecting cars and manually labeling, purchasing, or directly obtaining public datasets. The images are collected during the driving process of the autonomous vehicle.
[0036] S2, target segmentation is performed on the image in each image-label pair in the existing dataset, and the obstacles in the image that do not belong to the environment background are segmented to form an obstacle mask.
[0037] In the above S2 step, data preprocessing is mainly performed. However, it should be noted that in the present application, the dataset is used to train a deep learning model for performing a corresponding target task in the autonomous driving process, and the target task is an obstacle detection task, a semantic segmentation task, a depth estimation task, a lane line detection task, or a drivable area segmentation task in the autonomous driving process. The data preprocessing method is determined according to the task requirements, such as for obstacle detection or semantic segmentation tasks, the pixels corresponding to obstacles such as cars and people in the picture can be segmented to form an obstacle mask.
[0038] It should be noted that the specific form of the above obstacle can be determined according to the actual image, and the target that is not allowed to be changed in the image can be taken as the obstacle. Generally, the vehicles and people in the image can be taken as obstacles, and the regions where the vehicles and people are located are extracted to form an obstacle mask image.
[0039] S3, the image-label pair in the existing dataset, the obstacle mask corresponding to the image-label pair, and the textual description of the target weather scene are taken as the input of the trained diffusion model, and by constantly changing different weather scene textual descriptions, different existing image-label pairs are used to generate new image-label pairs in different weather scenes, thereby expanding the scale and weather scene coverage of the dataset.
[0040] Since different target tasks have different specific image-label pairs, the inputs of the diffusion model also differ.
[0041] In an embodiment of the present application, if the target task is an obstacle detection task in the autonomous driving process, the label in the image-label pair is an obstacle detection box (planar two-dimensional detection box or spatial three-dimensional detection box) in the image. Therefore, after the obstacle mask is input into the diffusion model, it is used to limit the obstacle region in the source image during the model generation process to be an unchangeable region, and the remaining regions are changeable regions, so that the label of the source image can still form a new image-label pair with the newly generated image.
[0042] In an embodiment of the present application, if the target task is a semantic segmentation task in the automatic driving process, the label in the image-label pair is a pixel-level semantic classification map corresponding to the image. Thus, the obstacle mask is input into the diffusion model to limit the obstacle area in the source image in the model generation process to be an unchangeable area, and the remaining area is a changeable area. In addition, the pixel-level semantic classification map of the source image is also input into the diffusion model as a label along with the source image, so as to retain the obstacle information in the source image while ensuring that the generated new image has the same pixel-level classification as the source image, and the label of the source image can still form a new image-label pair with the newly generated image.
[0043] In the above S2 step, data generation is mainly performed. In actual application, when augmenting the scale and scene coverage of the existing data set, the meteorological scenes that already exist in the existing data set but need to be expanded in sample size or the meteorological scenes that do not exist in the existing data set can be regarded as target scenes. A textual description describing the scene is constructed for each target scene, and different existing image-label pairs and obstacle masks are input into the diffusion model to generate new image-label pairs.
[0044] In order to expand the data samples as much as possible, all the data sets prepared in the first step can be traversed, and different scene textual descriptions can be replaced to cover various meteorological scenes (such as seasons, day and night, rain, snow, fog, dust, etc.). A large number of image-label pairs under various meteorological conditions can be generated, which greatly expands the scale and scene coverage of the data set and provides data support for model training of all-weather automatic driving.
[0045] It should be noted that the diffusion model used in the present application belongs to the prior art. The diffusion model needs to be trained in advance using a data set before actually generating data. However, the specific structure and training method of the diffusion model belong to the prior art, and can be referred to related existing implementation methods, and thus will not be described in detail.
[0046] In order to better demonstrate the specific implementation and technical effects of the all-weather automatic driving data generation method based on the diffusion model shown in S1-S3 above, two tasks of obstacle detection and semantic segmentation are used to demonstrate.
[0047] Example 1: Data generation of obstacle detection task
[0048] In this embodiment, the all-weather automatic driving obstacle detection data generation method based on the diffusion model is as follows:
[0049] Data preparation stage: a certain number of image-label pairs are obtained by collecting images through a vehicle, adding manual labeling, externally purchasing or downloading public data sets (such as KITTI, etc.). For the obstacle detection task, the label is a two-dimensional detection box in the image plane.
[0050] Data pre-processing stage: using a segmentation model to segment the obstacles (vehicles and pedestrians in the image) in each image in the image-label pair, and obtain the pre-processed result, i.e. the obstacle mask image.
[0051] Data generation stage: based on the source image and the mask as the input of the trained diffusion model, by controlling the description of the target weather scene input to the diffusion model, a new image is generated by the diffusion model.
[0052] Referring to Figure 2 As shown, the source image is a sunny road image taken in the daytime without snow, and the obstacle mask image can be extracted. The obstacle mask input to the model will limit the area that can be changed during the model generation process, and the white part of the mask corresponds to the area of the original image that remains unchanged after the diffusion model, which makes the generated image not only meet the requirements of the target scene described by the text, but also retain the basic information of the obstacles in the source image (such as size, relative position relationship, etc.), so the label corresponding to the source image (source label) is also applicable to the generated new image. In this embodiment, the input target weather scene text description can be designed according to actual expansion needs, for example, input "daytime, after snow, road covered with snow", the diffusion model will regenerate the part outside the obstacle mask in the source image to form a scene that meets the text description, but the vehicle area in the mask remains unchanged. Therefore, the generated new image and the source label form a new image-label pair, which can be used for the data set required for training the deep learning model for expanding the obstacle detection task.
[0053] By changing the text description while traversing the source image-label pair, a large number of new image-label pairs can be generated, especially by changing the text description to generate scene data that has not been covered in the source data set.
[0054] Embodiment 2 Data generation for semantic segmentation task
[0055] In this embodiment, the data generation of the semantic segmentation task is generally the same as the data generation method of the obstacle detection task in Embodiment 1, and also includes a data preparation stage, a data preprocessing stage and a data generation stage. The difference lies in that the label in the image-label pair obtained in the data preparation stage is a pixel-level semantic classification map of the image (i.e., the source picture pixel-level classification, different categories correspond to different colors). In addition, in the data generation stage, the source image and the label corresponding to the source image (i.e., the pixel-level semantic classification map) are also input as an input of the diffusion model to guide image generation (similar to the practice of controlnet), while the obstacle mask is also input to limit the obstacle region as an unchangeable region and the remaining region as a changeable region, so that the generated new image has the same pixel-level classification as the source image and the obstacle remains unchanged, thereby ensuring that the source label and the new image can be paired to form a new image-label pair. As shown in Figure 3 , in an example, the source image is a sunny road image taken in the daytime, but it is desired to generate a scene image in the rain based on it, so the input target weather scene text description is "moderate rain", and the diffusion model will regenerate the part outside the obstacle mask in the source image to form a background in line with the scene under moderate rain, but the vehicle region in the mask remains unchanged, and the pixel-level semantics of the whole image does not change, thereby generating a new image-label pair (the label still uses the label of the source image).
[0056] Therefore, in this embodiment, based on the source data set, a large number of new data-label pairs can be generated by changing the text description, greatly expanding the size and scene coverage of the data set required for deep learning model training in the semantic segmentation task.
[0057] Of course, in addition to the above two embodiments, the method of the present application can also be used for lane line detection, drivable area segmentation and other tasks, and the process of data generation can also follow the above two embodiments, which will not be described in detail here.
[0058] In addition, in another embodiment of the present application, as shown in Figure 4 , based on the same inventive concept as the above-mentioned diffusion model-based all-weather autonomous driving data generation method, a diffusion model-based all-weather autonomous driving data generation system is also provided, which comprises:
[0059] a data set acquisition module for acquiring an existing data set corresponding to a target task in autonomous driving, the existing data set being composed of a series of image-label pairs.
[0060] a preprocessing module for performing target segmentation on the image in each image-label pair in the existing data set to segment out the obstacles in the image that do not belong to the environmental background, forming an obstacle mask.
[0061] The data generation module is configured to input the image-label pairs in the existing data set, the obstacle masks corresponding to the image-label pairs, and the text description of the target meteorological scene as inputs of the trained diffusion model, replace different meteorological scene text descriptions, generate new image-label pairs under different meteorological scenes based on different existing image-label pairs, and thus expand the size and meteorological scene coverage of the data set.
[0062] It should be noted that the all-weather autonomous driving data generation method based on the diffusion model provided in the above embodiments can essentially exist in the form of a program processing flow. Therefore, based on the same inventive concept, another preferred embodiment of the present application also provides a computer electronic device corresponding to the all-weather autonomous driving data generation method based on the diffusion model provided in the above embodiments, which includes a memory and a processor.
[0063] The memory is configured to store a computer program.
[0064] The processor is configured to implement the all-weather autonomous driving data generation method based on the diffusion model when the computer program is executed.
[0065] Similarly, based on the same inventive concept, another preferred embodiment of the present application also provides a computer readable storage medium corresponding to the all-weather autonomous driving data generation method based on the diffusion model provided in the above embodiments, and the storage medium stores a computer program. When the computer program is executed by the processor, the all-weather autonomous driving data generation method based on the diffusion model is implemented.
[0066] It can be understood that the form of the above-mentioned memory and storage medium can include a random access memory (RAM) and a non-volatile memory (NVM), such as at least one disk memory. Meanwhile, the storage medium can also be a U disk, a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes. Of course, with the wide application of cloud servers, the above-mentioned software program can also be loaded on the cloud platform to provide corresponding services, so the computer readable storage medium is not limited to the form of local hardware.
[0067] It can be understood that the processor described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.
[0068] In addition, it should be noted that the skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device described above can refer to the corresponding process in the foregoing method embodiments, which will not be described here. In the embodiments provided in the present application, the division of steps or modules in the device and method is only a logical functional division, and actual implementation can have another division mode, for example, a plurality of modules or steps can be combined or integrated together, or a module or step can be split.
[0069] In addition, the logical instructions in the memory described above can be implemented in the form of a software functional unit and sold or used as an independent product when used, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in the embodiments of the present application.
[0070] The above-described embodiments are only a preferred scheme of the present application, and are not intended to limit the present application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present application.
Claims
1. A method for generating all-weather autonomous driving data based on a diffusion model, characterized in that, Comprise: S1, obtaining an existing data set corresponding to a target task in automatic driving, the existing data set consisting of a series of image-label pairs; S2, performing target segmentation on the image in each image-label pair in the existing data set, segmenting out the obstacles in the image that do not belong to the environment background to form an obstacle mask; S3, taking the image-label pair in the existing data set, the obstacle mask corresponding to the image-label pair, and the text description of the target meteorological scene as the input of the trained diffusion model, generating new image-label pairs under different meteorological scenes based on different existing image-label pairs by constantly changing different meteorological scene text descriptions, thereby expanding the size and meteorological scene coverage of the data set. 2.The all-weather automatic driving data generation method based on diffusion model according to claim 1, wherein, The target task is an obstacle detection task, a semantic segmentation task, a depth estimation task, a lane line detection task, or a drivable area segmentation task in the automatic driving process. 3.The all-weather automatic driving data generation method based on diffusion model of claim 1, wherein, The target task is an obstacle detection task in the automatic driving process; the label in the image-label pair is an obstacle detection box in the image; The obstacle mask is input into the diffusion model to limit the obstacle region in the source image as an unchangeable region and the remaining regions as changeable regions during the model generation process, so that the label of the source image can still form a new image-label pair with the newly generated image. 4.The all-weather automatic driving data generation method based on diffusion model of claim 3, wherein, The obstacle detection box is a planar two-dimensional detection box or a spatial three-dimensional detection box. 5.The all-weather automatic driving data generation method based on diffusion model according to claim 1, wherein, The target task is a semantic segmentation task in the automatic driving process; the label in the image-label pair is a pixel-level semantic classification map corresponding to the image; The obstacle mask is input into the diffusion model to limit the obstacle region in the source image as an unchangeable region and the remaining regions as changeable regions during the model generation process, and the pixel-level semantic classification map of the source image is also input into the diffusion model as a label along with the source image, thereby preserving the obstacle information in the source image while ensuring that the generated new image has the same pixel-level classification as the source image, so that the label of the source image can still form a new image-label pair with the newly generated image.
6. The all-weather autonomous driving data generation method based on a diffusion model according to claim 1, wherein, When expanding the size and scene coverage of the existing data set, meteorological scenes that need to be expanded in the existing data set or meteorological scenes that do not exist in the existing data set are all taken as target scenes, and a text description describing the scene is constructed for each target scene, and new image-label pairs are generated by inputting different existing image-label pairs and obstacle masks into the diffusion model.
7. The all-weather autonomous driving data generation method based on a diffusion model according to claim 1, wherein, The obstacles include vehicles and people in the image.
8. An all-weather autonomous driving data generation system based on a diffusion model, characterized by, Comprise: A data set acquisition module for acquiring an existing data set corresponding to a target task in automatic driving, the existing data set consisting of a series of image-label pairs; A preprocessing module for performing target segmentation on the image in each image-label pair in the existing data set, segmenting out the obstacles in the image that do not belong to the environment background to form an obstacle mask; The data generation module is configured to take the image-label pairs in the existing data set, the obstacle masks corresponding to the image-label pairs, and the text description of the target meteorological scene as inputs of the trained diffusion model, replace different meteorological scene text descriptions, generate new image-label pairs under different meteorological scenes based on different existing image-label pairs, and thus expand the scale and meteorological scene coverage of the data set.
9. A computer-readable storage medium, characterized in that, The storage medium has a computer program stored thereon, and when the computer program is executed by a processor, the diffusion model-based all-weather autonomous driving data generation method according to any one of claims 1-7 is implemented.
10. A computer electronic device, comprising: comprising a memory and a processor; The memory is configured to store a computer program. The processor is configured to implement the diffusion model-based all-weather autonomous driving data generation method according to any one of claims 1-7 when the computer program is executed.