A scene generation method for autonomous driving

Abstract

The present invention discloses a scene generation method for autonomous driving. First, the original image is obtained through a camera and decomposed according to the categories in the category list to form a benchmark data set; then a variational encoder is established, a total loss function is constructed and the variational encoder is trained; after the training is completed, when it is necessary to generate a scene image for autonomous driving, it is only necessary to input N groups of random numbers that conform to the Gaussian distribution into the decoder of the trained variational encoder to generate N groups of [decomposition images, masks] and reconstruct them into new images. The present invention makes up for the limitations of manually collected images and reduces the burden of manually collected images. To some extent, it is no longer necessary to manually collect images, and it also increases the diversity of the data sets required for training target detection models.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a method for generating an autonomous driving scene. Background Art

[0002] The autonomous driving system mainly includes three parts: environmental perception, decision-making planning and control. At present, the environmental perception part of the autonomous driving system is mainly implemented based on artificial intelligence methods. Among them, the artificial intelligence-based methods generally require training models, and the training models require corresponding and huge data sets. How to obtain effective and huge data sets is a problem worth solving.

[0003] In order to obtain effective and massive data sets, manual collection and data enhancement methods are generally used to obtain and increase the number of data sets. However, these methods are time-consuming and labor-intensive, and manually collected images cannot cover all situations, and it is impossible to effectively collect some rare samples. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a scene generation method for autonomous driving in view of the defects involved in the background technology.

[0005] The present invention adopts the following technical solutions to solve the above technical problems:

[0006] A method for generating a scene for autonomous driving comprises the following steps:

[0007] Step 1), obtain the original image through the camera, and for each original image, decompose it according to the preset category list: for each category in the category list, retain the image of the category in the original image and set the rest to blank as the decomposition diagram of the category, and generate a mask of the decomposition diagram to mark its position information on the original image, forming training data in the format of (original image, [decomposition diagram, mask], [decomposition diagram, mask], …, [decomposition diagram, mask]); all training data constitute the benchmark data set;

[0008] Step 2), establish a variational encoder, which includes an encoder and N decoders, where N is the number of categories in the category list;

[0009] The encoder is used to output N groups of parameters according to the input image, each group of parameters includes two parameters, parameter u and parameter v; the decoder is used to generate a corresponding decomposition graph and mask according to a group of random numbers that conform to the Gaussian distribution of the input;

[0010] Step 3), for each training data in the benchmark dataset:

[0011] Step 3.1), input the original image into the encoder in the variational encoder, the encoder outputs N groups of parameters, for each group of parameters, the parameter u is used as the mean and the parameter v is used as the variance to construct a group of random numbers that conform to the Gaussian distribution, forming N groups of random numbers that conform to the Gaussian distribution;

[0012] Step 3.2), input the N groups of random numbers that conform to the Gaussian distribution generated in step 3.1) into the N decoders one by one, and obtain N [decomposition graph, mask]; for each [decomposition graph, mask], its mask is processed using the Softmax function;

[0013] Step 3.3), reconstruct the N [decomposition images, masks] processed by the Softmax function into a new image;

[0014] Step 4), construct the total loss function and train the variational encoder, where the specific steps of constructing the loss function are as follows:

[0015] Step 4.1), construct N distribution loss functions, where the i-th distribution loss function is used to calculate the difference between the Gaussian distribution composed of the i-th group of parameters generated by the decoder and the standard Gaussian distribution, where i is a natural number greater than or equal to 1 and less than or equal to N;

[0016] Step 4.2), construct N reconstruction loss functions corresponding to the decoders one by one, wherein the i-th reconstruction loss function is used to calculate the difference between the [decomposition map, mask] generated by the i-th decoder and the i-th [decomposition map, mask] in the training data;

[0017] Step 4.3), construct a whole image loss function to calculate the difference between the reconstructed image and the original image;

[0018] Step 4.4), respectively add preset weight coefficients to the N distribution loss functions, N reconstruction loss functions, and the whole image loss function, and then add them up in sequence as the total loss function;

[0019] In step 5, when it is necessary to generate scene images for autonomous driving, N groups of random numbers that conform to the Gaussian distribution are input into the decoder of the trained variational encoder to generate N groups of [decomposition graph, mask]. For each [decomposition graph, mask], its mask is processed using the Softmax function; and the N [decomposition graph, mask] processed by the Softmax function are reconstructed into a new image.

[0020] As a further optimization scheme of the method for generating an autonomous driving scene of the present invention, the category list includes background, traffic roads, vehicles, pedestrians, and traffic signs.

[0021] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0022] The present invention can generate images of scenes designed by us and that are consistent with actual scenes to constitute a data set; on the one hand, the generated images are used to train the target detection model, which makes up for the limitations of manually collected images and reduces the burden of manually collected images. To a certain extent, manual collection of images is no longer necessary, and the diversity of the data set required for training the target detection model is also increased; on the other hand, the generated data set eliminates the need for manual labeling, because the content in the generated image is determined and labeled according to the mask when the image is generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of decomposing an image acquired by a front-view camera in the present invention;

[0024] Figure 2 It is a schematic diagram of the principle of the variational encoder in the present invention;

[0025] Figure 3 It is a schematic diagram of a process of generating a driving scene by a decoder in a variational encoder in the present invention. DETAILED DESCRIPTION

[0026] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings:

[0027] The present invention can be implemented in many different forms and should not be considered to be limited to the embodiments described herein. On the contrary, these embodiments are provided to make the present invention disclosed thoroughly and completely, and will fully express the scope of the present invention to those skilled in the art. In the accompanying drawings, components are enlarged for clarity.

[0028] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, assemblies, and / or parts, these components, assemblies, and / or parts are not limited by these terms. These terms are merely used to distinguish components, assemblies, and / or parts from each other. Therefore, the first component, assembly, and / or part discussed below can become a second component, assembly, or part without departing from the teachings of the present invention.

[0029] Based on an image dataset of a benchmark driving scene, the background, traffic roads, vehicles, pedestrians, traffic signs and other parts of an image are extracted, that is, multiple images are decomposed from one image, and the content of each decomposed image is a part of the original image. The decomposed images are superimposed to synthesize the original image or a part of the original image.

[0030] As for how to decompose an image that meets the above conditions from an image, on the one hand, you can use manual methods to decompose it initially. When the manually decomposed images reach a certain scale, you can use deep learning methods to learn and then use the learned model to decompose. On the other hand, you can also use autonomous driving simulation software (such as Carla) to generate driving scenes and extract the required images.

[0031] Through the above image preprocessing, a one-to-many image group can be generated, that is, one image is decomposed into multiple images, and these multiple images can be synthesized into the original image or a part of the original image. As for how many images an image needs to be decomposed into, it can be determined according to needs. Generally, it can be decomposed into multiple images according to factors such as the category of traffic participants in the driving scene and road and non-road.

[0032] We need to build a variational encoder (VAE) model with one encoder corresponding to multiple decoders. The number of decoders depends on how many images an image is decomposed into. Generally, the number of images that need to be decomposed into an image is the number of decoders.

[0033] After the input original image is processed, multiple sets of parameters are generated. These parameters are used to establish a Gaussian distribution (the number of distributions is consistent with the number of decoders), and then a random array that conforms to the established Gaussian distribution is constructed. The generated random array is input into a decoder, and each decoder can generate the decomposed image corresponding to the original image (each decoder only generates one image among the images decomposed by the original image, which is why the number of images decomposed into one image is the number of decoders).

[0034] A mask is added to the image generated or corresponding to each decoder. The mask can be used to know the position of each decomposed image or the image generated by the decoder in the original image (that is, the coordinate position of each pixel point). At the same time, it can ensure that the images generated by multiple decoders do not conflict with each other, so as to ensure the rationality and reality of the generated images.

[0035] After building a model with one encoder corresponding to multiple decoders based on the idea of ​​variational encoder (VAE), you can use the previously created image dataset to train the model.

[0036] When the model training is completed, the encoder part of the model is discarded and only the decoder part is retained, that is, we only need to use the decoder in the model to generate images. We only need to input a random array that conforms to the specified distribution into the decoder, and the decoder can generate the corresponding image according to the corresponding input, and then combine all the images generated by the decoder with the mask to synthesize a new image, thus completing the creation of a new scene.

[0037] The autonomous driving system is equipped with multiple cameras, which obtain environmental information from different directions and angles around the vehicle, and then integrate them to obtain all the environmental information around the vehicle.

[0038] Now, taking the vehicle's front-view or front camera as an example, the present invention explains how to obtain the initial reference data set, how to process the image acquired by the camera, and how to generate an image for imitating or replacing the image acquired by the front-view camera.

[0039] Step 1) Process the image acquired by the camera to obtain a benchmark data set.

[0040] The camera that decomposes the image obtained by the front-view camera is as follows: Figure 1 shown.

[0041] We decompose the traffic roads, vehicles and pedestrians in the original image captured by the camera into three images respectively, and obtain three decomposed images. At the same time, we generate masks of the pixel coordinates of the traffic roads, vehicles and pedestrians at the image locations to mark the position information of the targets in the decomposed images on the original image.

[0042] If simulation software (such as Carla) is used, the traffic information and target information such as traffic participants in the image obtained by the front-view camera can be directly generated by setting parameters, and then the image obtained by the front-view camera is synthesized. The method of generating images through simulation software can save the step of decomposing the original image.

[0043] This constructs the training data, completing the creation of the benchmark dataset.

[0044] Step 2), establish a variational encoder, which includes an encoder and N decoders, where N is the number of categories in the category list;

[0045] The encoder is used to output N groups of parameters according to the input image, each group of parameters includes two parameters, parameter u and parameter v; the decoder is used to generate a corresponding decomposition graph and mask according to a group of random numbers that conform to the Gaussian distribution of the input;

[0046] Step 3), for each training data in the benchmark dataset:

[0047] Step 3.1), input the original image into the encoder in the variational encoder, the encoder outputs N groups of parameters, for each group of parameters, the parameter u is used as the mean and the parameter v is used as the variance to construct a group of random numbers that conform to the Gaussian distribution, forming N groups of random numbers that conform to the Gaussian distribution;

[0048] Step 3.2), input the N groups of random numbers that conform to the Gaussian distribution generated in step 3.1) into the N decoders one by one, and obtain N [decomposition graph, mask]; for each [decomposition graph, mask], its mask is processed using the Softmax function;

[0049] Step 3.3), reconstruct the N [decomposition images, masks] processed by the Softmax function into a new image;

[0050] Step 4), construct the total loss function and train the variational encoder, where the specific steps of constructing the loss function are as follows:

[0051] Step 4.1), such as Figure 2 As shown, N distribution loss functions are constructed, wherein the i-th distribution loss function is used to calculate the difference between the Gaussian distribution composed of the i-th group of parameters generated by the decoder and the standard Gaussian distribution, and i is a natural number greater than or equal to 1 and less than or equal to N;

[0052] Step 4.2), construct N reconstruction loss functions corresponding to the decoders one by one, wherein the i-th reconstruction loss function is used to calculate the difference between the [decomposition map, mask] generated by the i-th decoder and the i-th [decomposition map, mask] in the training data;

[0053] Step 4.3), construct a whole image loss function to calculate the difference between the reconstructed image and the original image;

[0054] Step 4.4), respectively add preset weight coefficients to the N distribution loss functions, N reconstruction loss functions, and the whole image loss function, and then add them up in sequence as the total loss function;

[0055] Step 5), when it is necessary to generate scene images for autonomous driving, N groups of random numbers that conform to the Gaussian distribution are input into the decoder of the trained variational encoder to generate N groups of [decomposition graph, mask]. For each [decomposition graph, mask], its mask is processed by the Softmax function; and the N [decomposition graph, mask] processed by the Softmax function are reconstructed into a new image, such as Figure 3 shown.

[0056] Through the above steps, we only need to input random numbers that satisfy the Gaussian distribution into the decoder to generate a random scene, which can be used to train the perception module in the autonomous driving system.

[0057] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0058] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A scene generation method for autonomous driving, characterized in that: The following steps are involved: Step 1), obtain the original image through the camera, and for each original image, decompose it according to the preset category list: for each category in the category list, retain the image of the category in the original image and set the rest to blank as the decomposition diagram of the category, and generate a mask of the decomposition diagram to mark its position information on the original image, forming training data in the format of (original image, [decomposition diagram, mask], [decomposition diagram, mask], …, [decomposition diagram, mask]); all training data constitute the benchmark data set; Step 2), establish a variational encoder, which includes an encoder and N decoders, where N is the number of categories in the category list; The encoder is used to output N groups of parameters according to the input image, each group of parameters includes two parameters, parameter u and parameter v; the decoder is used to generate a corresponding decomposition graph and mask according to a group of random numbers that conform to the Gaussian distribution of the input; Step 3), for each training data in the benchmark dataset: Step 3.1), input the original image into the encoder in the variational encoder, the encoder outputs N groups of parameters, for each group of parameters, the parameter u is used as the mean and the parameter v is used as the variance to construct a group of random numbers that conform to the Gaussian distribution, forming N groups of random numbers that conform to the Gaussian distribution; Step 3.2), input the N groups of random numbers that conform to the Gaussian distribution generated in step 3.1) into the N decoders one by one, and obtain N [decomposition graph, mask]; for each [decomposition graph, mask], its mask is processed using the Softmax function; Step 3.3), reconstruct the N [decomposition images, masks] processed by the Softmax function into a new image; Step 4), construct the total loss function and train the variational encoder, where the specific steps of constructing the loss function are as follows: Step 4.1), construct N distribution loss functions, where the i-th distribution loss function is used to calculate the difference between the Gaussian distribution composed of the i-th group of parameters generated by the decoder and the standard Gaussian distribution, where i is a natural number greater than or equal to 1 and less than or equal to N; Step 4.2), construct N reconstruction loss functions corresponding to the decoders one by one, wherein the i-th reconstruction loss function is used to calculate the difference between the [decomposition map, mask] generated by the i-th decoder and the i-th [decomposition map, mask] in the training data; Step 4.3), construct a whole image loss function to calculate the difference between the reconstructed image and the original image; Step 4.4), respectively add preset weight coefficients to the N distribution loss functions, N reconstruction loss functions, and the whole image loss function, and then add them up in sequence as the total loss function; In step 5, when it is necessary to generate scene images for autonomous driving, N groups of random numbers that conform to the Gaussian distribution are input into the decoder of the trained variational encoder to generate N groups of [decomposition graph, mask]. For each [decomposition graph, mask], its mask is processed using the Softmax function; and the N [decomposition graph, mask] processed by the Softmax function are reconstructed into a new image.

2. The scene generation method for autonomous driving according to claim 1, characterized in that: The category list includes background, traffic roads, vehicles, pedestrians, and traffic signs.

Images