Automatic driving algorithm training data synthesis method based on diffusion model

Abstract

The invention discloses an automatic driving algorithm training data generation system based on a diffusion model. In the invention, a driving environment-oriented controllable data sample generation scheme is provided, the scheme takes a diffusion model as a reference methodology, brand new data is generated and training samples of related perception tasks are expanded based on collected data samples and prior labels, so that the problems of uneven quality of existing training data samples, less difficult data and the like are solved; by generating the cue word, the sample acquisition cost and acquisition difficulty of the difficult case are reduced, the generalization effect of the model in the actual business is improved, and the algorithm is pushed to serve more businesses. Meanwhile, by combining the input of various forms of control conditions, the controllability of the data generation process is improved, and the generation result better conforms to the expectation. And finally, screening out generation results which do not meet requirements through a verification mechanism, and ensuring the generation quality.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and specifically is a system for generating training data for an autonomous driving algorithm based on a diffusion model. Background Art

[0002] For technologies related to autonomous driving, the quality of training data largely determines the effectiveness of the model. With the continuous advancement of autonomous driving technology, the demand for difficult data is also increasing. However, compared with the real road environment, the commonly used open source data sets usually have simpler road conditions and do not cover a comprehensive range of scenarios. As a result, the network models trained based on these data cannot be generalized to other complex environments, and it is difficult to achieve satisfactory results on various data in actual business. With the continuous development of artificial intelligence technology, generative artificial intelligence AIGC has had an important impact on production work in all walks of life. Among them, data generation is a booming field that aims to generate realistic or creative new pictures through deep learning. With the continuous iteration of data generation algorithms, using this technology to generate more data to solve the problem of single existing training data and few difficult examples has become a feasible solution.

[0003] The existing solution that is most similar to the present invention implements an image generation algorithm based on a diffusion model. The algorithm is a framework based on a U-shaped network as a whole, which converts noise from a standard Gaussian distribution to a real image through an iterative denoising process. The method predicts noise under the guidance of a classification label map to achieve a classification layout similar to the input label distribution after denoising.

[0004] The method consists of two parts: an encoder and a decoder. The encoder uses a diffusion coding residual block and an attention module to encode the noisy data. The diffusion coding residual block consists of a convolutional layer, an activation function SiLU, and a group normalization layer. The time step is introduced as a parameter so that the network can estimate the noise at different time steps t. The attention module is a self-attention layer with jump connections. In the decoder part, the method inputs a classification label map to guide the denoising process, and designs a diffusion decoding residual block. Unlike the diffusion coding residual block, the diffusion decoding residual block replaces the group normalization layer with a spatially adaptive normalization layer, and embeds the classification label map into the denoising network in a multi-layer spatially adaptive manner.

[0005] However, the controllability of the generation process of the existing technical solutions is insufficient. The existing image generation technology mainly controls the image generation process through a single guiding signal, such as a text prompt word, which leads to a large degree of freedom in the generation process, the generated results are not realistic enough, and it is difficult to achieve accurate control at the object level, resulting in the generation results cannot be directly used as autonomous driving training data. Lack of quality verification of the generated results. The existing image generation technology cannot automatically evaluate the generation results. Due to the randomness of the generation process, the expected generation results may not be met, and additional investment costs are required to screen the generation results. The versatility and generalization are poor. The existing image generation technology is usually trained using open source data sets. Since the open source data sets provide accurate and diverse label information, it provides great convenience for data generation. However, on more general data, due to the lack of labels, these data are difficult to use. Therefore, this type of data generation method has poor versatility and insufficient generalization of the generated data. It is difficult to use label information. Due to the randomness of the generation process, some label results of the existing image generation technology cannot be applied to the generated results, and additional costs are required to annotate the generated data. Summary of the invention

[0006] The purpose of the present invention is to provide an autonomous driving algorithm training data generation system based on a diffusion model in order to solve the above-mentioned problems.

[0007] The technical solution adopted by the present invention is as follows: a system for generating training data for an autonomous driving algorithm based on a diffusion model, the system comprising a data preprocessing module, a data generation module and a data verification module;

[0008] The data preprocessing module consists of four parts. The semantic segmentation inside the data preprocessing module is used to extract the semantic distribution of the input image. It is the most important link. The semantic guidance signal obtained through the semantic segmentation network will guide and supervise the overall distribution of the generated data. This system selects the InternImage-H model;

[0009] The data generation module uses a diffusion model to generate data, including a forward process and a reverse process. In the forward process, random noise is slowly and sequentially added to the sample, and then in the reverse process, parameters are learned and noise is fitted to recover the sample from the noise.

[0010] The data verification module uses the above-mentioned semantic segmentation and depth estimation algorithm to evaluate the generated results, calculates the semantic error and depth error according to the semantic pre-annotation and depth pre-annotation results of the original data, and judges whether the current generated results meet the requirements based on these two errors.

[0011] In a preferred embodiment, the InternImage-H model is a general visual large model based on multi-task learning. The core idea of ​​multi-task learning is to learn multiple related tasks by sharing model parameters. Compared with the method of training a model for each task separately in traditional single-task learning, multi-task learning uses a model to train the data of multiple tasks at the same time, sharing some or all of the model parameters. In this way, different tasks can influence each other through shared parameters, and this sharing and transfer can enable the model to better adapt to different tasks. In order to improve the generalization performance on different data and tasks, the foreground and background separation module mainly uses the results of semantic segmentation and separates the foreground and background according to the set semantic category. After the foreground object in the image is taken out, the pixel values ​​of this part of the position need to be filled to obtain a complete background image. This part of the function is implemented in the image repair module.

[0012] In a preferred embodiment, the data preprocessing module uses the LaMa method, which is based on fast Fourier convolution. It not only improves the quality of model restoration, but also reduces the number of model parameters. It can produce high-resolution restoration results using only low-resolution image training, and has good versatility. In existing data generation schemes, using only semantic graphs for data generation will lead to depth ambiguity problems in the same semantic area, that is, the relative depth relationship of objects cannot be known, resulting in confusion in the depth of the generated image. Therefore, this application adds a depth estimation module to solve this problem, and uses the MiDaS method for monocular depth estimation to constrain the relative depth relationship of foreground objects. This method uses a mixed data set for training and has good generalization.

[0013] In a preferred embodiment, the data generation module uses two diffusion models that do not share weights to generate data for the foreground and background in order to ensure the quality of their respective generation. In the background generation part, background prompt words and semantic maps are used to guide the generation process, wherein the semantic map is used to determine the semantic distribution of the entire image, and the background prompt words change the style of image generation when the semantic distribution is determined, such as guiding the generation of images of different weather or seasons. In the original data, there may be fewer images taken at a certain time, so this method can be used to supplement the scene data of the corresponding environment. It is found through experiments that there are usually fewer semantic ambiguities in the generation of the background, and the generation effect is better, so there is no need for depth information guidance. In the foreground generation part, foreground objects of different appearances are mainly generated through foreground prompt words, such as vehicles of various colors or models.

[0014] In a preferred embodiment, the data generation module adds a depth map as a guide signal to constrain the relative depth relationship of foreground objects in semantically identical areas. In the process of using prompt words, a text encoding network is required. This application uses the CLIP method, which is a model for matching text and images. The encoder part can be used to encode text into high-dimensional features for fusion with features in the diffusion model.

[0015] In a preferred embodiment, the specific implementation of the system includes the following steps:

[0016] S101 obtains data for generating an image, where the data includes an image and a labeling result consistent with the labeling task. For example, for a target detection task, the center coordinates of a target box and the length and width of the target box, or the coordinates of diagonal points of the target box are provided.

[0017] S102 inputs the image into a semantic segmentation network to obtain a semantic segmentation map.

[0018] S103 inputs the obtained semantic segmentation map into the foreground and background segmentation module, in which the foreground semantics are set to include pedestrians, cyclists, small cars, trucks, buses, motorcycles and bicycles, and the rest of the semantics are background. According to the set foreground / background semantics, the foreground image and the background image are obtained respectively.

[0019] S104 inputs the background image into the image inpainting network to obtain a inpainted background image.

[0020] S105 sends the patched background image back to the semantic segmentation network to obtain a semantic segmentation map of the patched background image. The semantic segmentation map of the background image is one of the inputs of the background generation part of the data generation module, and the pixel value is the category label value.

[0021] S106 sends the input image to the depth estimation network to obtain a depth prediction map. The depth prediction map has the same resolution as the input image, and its pixel value is the predicted value of the distance between the current pixel and the shooting camera;

[0022] S107 obtains the maximum value of the depth prediction map, and divides all the depth prediction maps by the maximum value to obtain a normalized depth prediction map. The pixel value range of the normalized depth prediction map is 0 to 1. The normalized depth prediction map is one of the inputs of the foreground generation part of the data generation module.

[0023] S201 constructs the foreground and background diffusion model networks of the data generation module respectively, and reads the model weight parameters.

[0024] S202 selects a prompt word, and sends the prompt word to a text encoding module to obtain text features.

[0025] S203 initializes a Gaussian noise and inputs it into the diffusion model. The noise is first input into the encoder composed of three layers of residual blocks and an attention module for encoding. In this process, the text features are input into the attention module for feature guidance. The features are then sent to the decoder composed of three layers of residual blocks. In this part, the semantic segmentation map is directly sent to the residual block for feature aggregation. For the foreground generation part, it is necessary to first splice the normalized depth prediction map and the semantic segmentation map in the feature dimension and then send them to the residual block. The decoder part finally outputs a new denoised image.

[0026] S204 combines the foreground generated image with the background generated image to obtain a complete generated image

[0027] S301 inputs the generated result into the semantic segmentation network to obtain a semantic segmentation map, and calculates the category weighted pixel accuracy with the semantic segmentation map of the input image.

[0028] S302 inputs the generated result into the depth estimation network to obtain a depth prediction map, and calculates the mean square error with the depth prediction map of the input image.

[0029] The root mean square error is calculated by looping through all pixel positions, calculating the depth difference between the input image depth prediction map and the generated image depth prediction map at that position, and accumulating the difference. The accumulated difference is squared and finally divided by the total number of pixels in the input image.

[0030] S303 determines whether the semantic category pixel accuracy and the depth mean square error meet the set thresholds respectively. If both meet the thresholds, the generated image and the annotation results corresponding to the input image are input into the training data set, otherwise the generated data is discarded.

[0031] In a preferred embodiment, in step S102, the semantic segmentation map has the same resolution as the input image, and each pixel position stores a category label value, indicating the category to which the pixel belongs, and pixels with the same category label value belong to the same category. The mapping relationship between the category label value and the true semantics is shown in Table 1, which contains 19 categories in total.

[0032] In a preferred embodiment, in step S103, the foreground image and the background image have the same resolution as the input image, wherein the foreground image is one of the inputs of the foreground generation part of the data generation module, and the pixel value is the category label value. The background image needs to be sampled based on the semantic segmentation map using the input image, and the pixel value is the input image pixel value, and the foreground object pixel part is filled with 0.

[0033] In a preferred embodiment, in step S104, the patched background image supplements the pixel portion filled with 0 in the background image according to the surrounding pixels.

[0034] In a preferred embodiment, in step S301, the category weighted pixel accuracy is calculated by looping all pixel positions, and comparing in turn whether the category predictions of the semantic segmentation map of the input image and the semantic segmentation map of the generated image at the position are consistent. If they are consistent, the number of correct pixels of the category is increased by 1. After all pixels are calculated, loop all occurring categories, and for each category, count the total number of pixels of the category in the semantic segmentation map of the input image, and the pixel accuracy of the category = the number of correct pixels of the category / the total number of pixels of the category. The pixel accuracy of all categories is weighted and summed, and the weight calculation method used is as follows: for each category, its pixel area ratio = the number of pixels of the category / the total area of ​​the input image. For the pixel area ratio of all categories, use the Softmax function to calculate its proportion weight, and the sum of the proportion weights of all categories is 1.

[0035] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0036] 1. In the present invention, in order to improve the controllability of the data generation process, this scheme uses various forms of supervisory signals such as semantics, depth of field, and text prompts as auxiliary inputs during the generation process, constrains the generation process from multiple dimensions, and maintains a consistent relationship between the semantic distribution and depth of field relationship of the generated image while maintaining a certain degree of randomness.

[0037] 2. In the present invention, in order to ensure that the generated data meets expectations and reduce data that may be harmful to training, this solution uses some methods with better effects to automatically evaluate the generated results and screen out low-quality data.

[0038] 3. In the present invention, in order to ensure that the generated data has sufficient generalization, this solution does not rely on various forms of labels of open source datasets, but chooses more general semantic segmentation and depth of field estimation for data pre-labeling, and then generates data based on these labels.

[0039] 4. In the present invention, in order to effectively utilize labels, this scheme generates data based on labels provided by relevant methods (such as location, category, etc.), and these labels are always stored during the generation process. Since the labels play a certain constraining role on the generation process to a certain extent, these labels are basically applicable to the generated results, reducing the annotation cost of the newly generated data. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flowchart of the present invention;

[0041] Figure 2 This is a structural diagram of the data preprocessing framework in the present invention;

[0042] Figure 3 Generate a framework structure diagram for the data in the present invention;

[0043] Figure 4 It is a structural diagram of the diffusion model in the present invention;

[0044] Figure 5 This is a structural diagram of the data verification framework in the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0046] Reference Figure 1-5 ,

[0047] Example:

[0048] An autonomous driving algorithm training data generation system based on a diffusion model includes a data preprocessing module, a data generation module and a data verification module.

[0049] First, the components of each module are introduced. The structure of the data preprocessing module is as follows: Figure 2 As shown, it mainly consists of four parts. Semantic segmentation is used to extract the semantic distribution of the input image. It is the most important link. The semantic guidance signal obtained by the semantic segmentation network will guide and supervise the overall distribution of the generated data. This application selects the InternImage-H model with relatively advanced performance. This is a general visual large model based on multi-task learning. The core idea of ​​multi-task learning is to learn multiple related tasks by sharing model parameters. Compared with the traditional single-task learning method of training a model for each task separately, multi-task learning uses a model to train data of multiple tasks at the same time, sharing part or all of the model parameters. In this way, different tasks can influence each other through shared parameters. This sharing and transfer can enable the model to better adapt to different tasks. In this way, the generalization performance on different data and tasks is improved. The foreground and background separation module mainly uses the results of semantic segmentation and separates the foreground and background according to the set semantic categories.

[0050] After taking out the foreground object from the image, the pixel values ​​at this position need to be filled to obtain a complete background image. This function is implemented in the image repair module. The module uses the LaMa method, which is based on fast Fourier convolution. It not only improves the quality of model repair, but also reduces the number of model parameters. It can produce high-resolution repair results using only low-resolution image training, and has good versatility. In existing data generation schemes, using only semantic graphs for data generation will lead to depth ambiguity problems in the same semantic area, that is, the relative depth relationship of objects cannot be known, resulting in confusion in the depth of the generated image. Therefore, this application adds a depth estimation module to solve this problem, and uses the MiDaS method for monocular depth estimation to constrain the relative depth relationship of foreground objects. This method uses mixed data sets for training and has good generalization.

[0051] The data generation module structure is as follows Figure 3 As shown. This module mainly uses the diffusion model for data generation. The diffusion model is a relatively advanced generation technology. It defines a diffusion step based on the Markov chain. It includes a forward process and a reverse process. In the forward process, random noise is slowly and sequentially added to the sample. Then, in the reverse process, parameters are learned and noise is fitted to recover the sample from the noise. The diffusion model structure used in this application is as follows: Figure 4 As shown. For the foreground and background, in order to ensure the quality of their respective generation, this application uses two diffusion models that do not share weights for data generation. In the background generation part, background prompt words and semantic maps are used to guide the generation process, where the semantic map is used to determine the semantic distribution of the entire image, and the background prompt words change the style of image generation when the semantic distribution is determined, such as guiding the generation of images in different weather or seasons. In the original data, there may be fewer images taken at a certain time, so this method can be used to supplement the scene data of the corresponding environment. Through experiments, it is found that there are usually fewer semantic ambiguities in the generation of the background, and the generation effect is better, so there is no need for depth information guidance.

[0052] In the foreground generation part, foreground objects of different appearances are mainly generated through foreground cue words, such as vehicles of various colors or models. In addition, compared with background generation, since foreground objects usually occupy fewer pixel values ​​and are prone to depth ambiguity, more precise generation is required. This application adds a depth map as a guiding signal to constrain the relative depth relationship of foreground objects in the same semantic area. In the process of using cue words, a text encoding network is required. This application uses the CLIP method, which is a model for matching text and images. Using its encoder part, text can be encoded into high-dimensional features for fusion with features in the diffusion model.

[0053] Data verification module such as Figure 5 This module mainly uses the above-mentioned semantic segmentation and depth estimation algorithms to evaluate the generated results, calculates the semantic error and depth error based on the semantic pre-annotation and depth pre-annotation results of the original data, and judges whether the current generated results meet the requirements based on these two errors.

[0054] Specific implementation steps:

[0055] The specific implementation of this application includes the following steps:

[0056] S101 obtains data for generating an image, where the data includes an image and a labeling result consistent with the labeling task. For example, for a target detection task, the center coordinates of a target box and the length and width of the target box, or the coordinates of diagonal points of the target box are provided.

[0057] S102 inputs the image into a semantic segmentation network to obtain a semantic segmentation map.

[0058] The semantic segmentation map has the same resolution as the input image, and each pixel position stores a category label value, indicating the category to which the pixel belongs. Pixels with the same category label value belong to the same category. The mapping relationship between the category label value and the true semantics is shown in Table 1, which contains a total of 19 categories.

[0059] Category label value Semantics Category label value Semantics 0 the way 9 Ramp 1 sidewalk 10 Sky 2 architecture 11 pedestrian 3 wall 12 Cyclist 4 Fences 13 Small car 5 Column 14 truck 6 Traffic lights 15 Bus 7 Traffic signs 16 train 8 vegetation 17 motorcycle 18 bike

[0060] Table 1 Semantic segmentation category label values ​​and semantic mapping relationships

[0061] S103 inputs the obtained semantic segmentation map into the foreground-background segmentation module, in which the foreground semantics are set to include pedestrians, cyclists, small cars, trucks, buses, motorcycles and bicycles, and the rest of the semantics are background. According to the set foreground / background semantics, a foreground image and a background image are obtained respectively.

[0062] The foreground image and background image have the same resolution as the input image, where the foreground image is one of the inputs of the foreground generation part of the data generation module, and the pixel value is the category label value. The background image needs to be sampled based on the semantic segmentation map using the input image, and the pixel value is the input image pixel value, and the foreground object pixel part is filled with 0.

[0063] S104 inputs the background image into the image inpainting network to obtain a inpainted background image.

[0064] The patched background image supplements the pixels filled with 0 in the background image according to the surrounding pixels.

[0065] S105 sends the patched background image back to the semantic segmentation network to obtain a semantic segmentation map of the patched background image.

[0066] The semantic segmentation map of the background image is one of the inputs of the background generation part of the data generation module, and the pixel value is the category label value.

[0067] S106 sends the input image to the depth estimation network to obtain a depth prediction map.

[0068] The depth prediction map has the same resolution as the input image, and its pixel value is the predicted distance of the current pixel from the shooting camera.

[0069] S107 obtains the maximum value of the depth prediction map, and divides all the depth prediction maps by the maximum value to obtain a normalized depth prediction map.

[0070] The pixel value range of the normalized depth prediction map is between 0 and 1. The normalized depth prediction map is one of the inputs of the foreground generation part of the data generation module.

[0071] S201 constructs the foreground and background diffusion model networks of the data generation module respectively, and reads the model weight parameters.

[0072] S202 selects a prompt word, and sends the prompt word to a text encoding module to obtain text features.

[0073] S203 initializes a Gaussian noise and inputs it into the diffusion model, such as Figure 4 As shown in the figure, the noise is first input into the encoder composed of three layers of residual blocks and attention modules for encoding. In this process, the text features are input into the attention module for feature guidance. The features are then sent to the decoder composed of three layers of residual blocks. In this part, the semantic segmentation map is directly sent to the residual block for feature aggregation. For the foreground generation part, the normalized depth prediction map and the semantic segmentation map need to be spliced ​​in the feature dimension before being sent to the residual block. The decoder part finally outputs a new denoised image.

[0074] S204 combines the foreground generated image with the background generated image to obtain a complete generated image

[0075] S301 inputs the generated result into the semantic segmentation network to obtain a semantic segmentation map, and calculates the category weighted pixel accuracy with the semantic segmentation map of the input image.

[0076] The calculation method of the category weighted pixel accuracy is: loop all pixel positions, and compare the category predictions of the semantic segmentation map of the input image and the semantic segmentation map of the generated image at the position in turn. If they are consistent, the number of correct pixels of the category is increased by 1. After all pixels are calculated, loop all the categories that appear. For each category, count the total number of pixels of the category in the semantic segmentation map of the input image. The pixel accuracy of the category = the number of correct pixels of the category / the total number of pixels of the category. The pixel accuracy of all categories is weighted and summed. The weight calculation method used is as follows: for each category, its pixel area ratio = the number of pixels of the category / the total area of ​​the input image. For the pixel area ratio of all categories, use the Softmax function to calculate its proportion weight, and the sum of the proportion weights of all categories is 1.

[0077] S302 inputs the generated result into the depth estimation network to obtain a depth prediction map, and calculates the mean square error with the depth prediction map of the input image.

[0078] The root mean square error is calculated by looping through all pixel positions, calculating the depth difference between the input image depth prediction map and the generated image depth prediction map at that position, and accumulating the difference. The accumulated difference is squared and finally divided by the total number of pixels in the input image.

[0079] S303 determines whether the semantic category pixel accuracy and the depth mean square error meet the set thresholds respectively. If both meet the thresholds, the generated image and the annotation results corresponding to the input image are input into the training data set, otherwise the generated data is discarded.

[0080] 1. In the present invention, in order to improve the controllability of the data generation process, this scheme uses various forms of supervisory signals such as semantics, depth of field, and text prompts as auxiliary inputs during the generation process, constrains the generation process from multiple dimensions, and maintains a consistent relationship between the semantic distribution and depth of field relationship of the generated image while maintaining a certain degree of randomness.

[0081] 2. In the present invention, in order to ensure that the generated data meets expectations and reduce data that may be harmful to training, this solution uses some methods with better effects to automatically evaluate the generated results and screen out low-quality data.

[0082] 3. In the present invention, in order to ensure that the generated data has sufficient generalization, this solution does not rely on various forms of labels of open source datasets, but chooses more general semantic segmentation and depth of field estimation for data pre-labeling, and then generates data based on these labels.

[0083] 4. In the present invention, in order to effectively utilize labels, this scheme generates data based on labels provided by relevant methods (such as location, category, etc.), and these labels are always stored during the generation process. Since the labels play a certain constraining role on the generation process to a certain extent, these labels are basically applicable to the generated results, reducing the annotation cost of the newly generated data.

[0084] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0085] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A system for generating training data for autonomous driving algorithms based on a diffusion model, characterized by: The system includes a data preprocessing module, a data generation module and a data verification module; The data preprocessing module consists of four parts; the semantic segmentation inside the data preprocessing module is used to extract the semantic distribution of the input image, which is the most important link. The semantic guidance signal obtained by the semantic segmentation network will guide and supervise the overall distribution of the generated data. This system selects the InternImage-H model; The data generation module uses a diffusion model to generate data; it includes a forward process and a reverse process, in which random noise is slowly and sequentially added to the sample in the forward process, and then in the reverse process, parameters are learned and noise is fitted to recover the sample from the noise; The data verification module uses the above-mentioned semantic segmentation and depth estimation algorithm to evaluate the generated results, calculates the semantic error and depth error according to the semantic pre-annotation and depth pre-annotation results of the original data, and judges whether the current generated results meet the requirements based on these two errors.

2. The system for generating training data for an autonomous driving algorithm based on a diffusion model according to claim 1, characterized in that: The InternImage-H model is a general visual large model based on multi-task learning. The core idea of ​​multi-task learning is to learn multiple related tasks by sharing model parameters. Compared with the traditional single-task learning method of training a model for each task separately, multi-task learning uses one model to train data of multiple tasks at the same time and shares part or all of the model parameters. In this way, different tasks can influence each other through shared parameters. This sharing and transfer can enable the model to better adapt to different tasks. In this way, the generalization performance on different data and tasks is improved. The foreground and background separation module mainly uses the results of semantic segmentation and separates the foreground and background according to the set semantic categories. After the foreground object in the image is taken out, the pixel values ​​of this part of the position need to be filled to obtain a complete background image. This function is implemented in the image inpainting module.

3. The system for generating training data for an autonomous driving algorithm based on a diffusion model according to claim 1, characterized in that: The data preprocessing module uses the LaMa method, which is based on fast Fourier convolution. It not only improves the restoration quality of the model, but also reduces the number of model parameters. It can produce high-resolution restoration results using only low-resolution image training, and has good versatility. In existing data generation schemes, using only semantic graphs for data generation will lead to depth ambiguity problems in the same semantic area, that is, the relative depth relationship of objects cannot be known, resulting in confusion in the depth of the generated image. Therefore, this application adds a depth estimation module to solve this problem, and uses the MiDaS method for monocular depth estimation to constrain the relative depth relationship of foreground objects. This method uses a mixed data set for training and has good generalization.

4. The system for generating training data for an autonomous driving algorithm based on a diffusion model according to claim 1, wherein: For the foreground and background, in order to ensure the generation quality of each, the data generation module uses two diffusion models that do not share weights for data generation. In the background generation part, background prompt words and semantic maps are used to guide the generation process, wherein the semantic map is used to determine the semantic distribution of the entire image, and the background prompt words change the style of image generation when the semantic distribution is determined, such as guiding the generation of images in different weather or seasons. In the original data, there may be fewer images taken at a certain time, so this method can be used to supplement the scene data of the corresponding environment. It is found through experiments that there are usually fewer semantic ambiguities in the generation of the background, and the generation effect is better, so there is no need for depth information guidance; in the foreground generation part, foreground objects of different appearances, vehicles of various colors or models are mainly generated through foreground prompt words.

5. The system for generating training data for an autonomous driving algorithm based on a diffusion model according to claim 1, characterized in that: The data generation module adds a depth map as a guiding signal to constrain the relative depth relationship of foreground objects in the same semantic area; in the process of using prompt words, a text encoding network is required. This application uses the CLIP method, which is a model for matching text and images. Using its encoder part, text can be encoded into high-dimensional features for fusion with the features in the diffusion model.

6. The system for generating training data for an autonomous driving algorithm based on a diffusion model according to claim 1, characterized in that: The specific implementation of the system includes the following steps: S101 acquires data for generating an image, the data including the image and a labeling result consistent with the labeling task, such as providing the center coordinates of a target frame and the length and width of the target frame, or the coordinates of diagonal points of the target frame for a target detection task; S102 inputs the image into a semantic segmentation network to obtain a semantic segmentation map; S103 inputs the obtained semantic segmentation map into the foreground-background segmentation module, in which the foreground semantics are set to include pedestrians, cyclists, small cars, trucks, buses, motorcycles and bicycles, and the rest of the semantics are background; according to the set foreground / background semantics, the foreground image and the background image are obtained respectively. S104 inputs the background image into an image inpainting network to obtain a inpainted background image; S105: the patched background image is re-sent to the semantic segmentation network to obtain a semantic segmentation map of the patched background image, wherein the semantic segmentation map of the background image is one of the inputs of the background generation part of the data generation module, and the pixel value is the category label value; S106 sends the input image to the depth estimation network to obtain a depth prediction map; the depth prediction map has the same resolution as the input image, and its pixel value is the predicted value of the distance between the current pixel and the shooting camera; S107 obtains the maximum value of the depth prediction map, and divides all the depth prediction maps by the maximum value to obtain a normalized depth prediction map; the pixel value range of the normalized depth prediction map is 0 to 1; the normalized depth prediction map is one of the inputs of the foreground generation part of the data generation module; S201 constructs the foreground and background diffusion model networks of the data generation module respectively, and reads the model weight parameters; S202: Select prompt words, send the prompt words to the text encoding module, and obtain text features; S203 initializes a Gaussian noise and inputs it into the diffusion model. The noise is first input into an encoder composed of three layers of residual blocks and an attention module for encoding. In this process, the text features are input into the attention module for feature guidance. The features are then fed into a decoder consisting of three layers of residual blocks, where the semantic segmentation map is directly fed into the residual block for feature aggregation; For the foreground generation part, the normalized depth prediction map and the semantic segmentation map need to be concatenated in the feature dimension before being sent to the residual block; The decoder part finally outputs a new denoised image; S204 combines the foreground generated image with the background generated image to obtain a complete generated image S301 inputs the generated result into the semantic segmentation network to obtain a semantic segmentation map, and calculates the category weighted pixel accuracy with the semantic segmentation map of the input image; S302 inputs the generated result into the depth estimation network to obtain a depth prediction map, and calculates the mean square error with the depth prediction map of the input image; The root mean square error is calculated by looping through all pixel positions, sequentially calculating the depth difference between the input image depth prediction map and the generated image depth prediction map at the position, and accumulating the difference; Take the square of the accumulated difference and finally divide it by the total number of pixels in the input image; S303 determines whether the semantic category pixel accuracy and the depth mean square error meet the set thresholds respectively. If both meet the thresholds, the generated image and the annotation results corresponding to the input image are input into the training data set, otherwise the generated data is discarded.

7. The system for generating training data for an autonomous driving algorithm based on a diffusion model according to claim 6, characterized in that: In step S102, the semantic segmentation map has the same resolution as the input image, and each pixel position stores a category label value, indicating the category to which the pixel belongs. Pixels with the same category label value belong to the same category. The mapping relationship between the category label value and the true semantics is shown in Table 1, which contains 19 categories in total.

8. The system for generating training data for an autonomous driving algorithm based on a diffusion model according to claim 6, characterized in that: In step S103, the foreground image and the background image have the same resolution as the input image, wherein the foreground image is one of the inputs of the foreground generation part of the data generation module, and the pixel value is the category label value; the background image needs to be sampled based on the semantic segmentation map using the input image, and the pixel value is the input image pixel value, and the foreground object pixel part is filled with 0.

9. The system for generating training data for an autonomous driving algorithm based on a diffusion model according to claim 6, characterized in that: In step S104, the patched background image supplements the pixel portion filled with 0 in the background image according to the surrounding pixels.

10. The system for generating training data for an autonomous driving algorithm based on a diffusion model according to claim 6, characterized in that: In step S301, the category weighted pixel accuracy is calculated by looping through all pixel positions, and comparing in turn whether the category predictions of the semantic segmentation map of the input image and the semantic segmentation map of the generated image at the position are consistent. If they are consistent, the number of correct pixels of the category is increased by 1; After all pixels are calculated, loop through all categories that appear. For each category, count the total number of pixels of this category in the semantic segmentation map of the input image. The pixel accuracy of this category = the number of correct pixels of this category / the total number of pixels of this category. Perform weighted summation on the pixel accuracy of all categories. The weight calculation method used is as follows: for each category, its pixel area ratio = the number of pixels of this category / the total area of ​​the input image. For the pixel area ratio of all categories, use the Softmax function to calculate its proportion weight, and the sum of the proportion weights of all categories is 1.

Images