Automatic driving simulation scene generation method and device and electronic equipment

By adjusting the sampling results of the generative model by using the gradient and control intensity obtained by the perceptual model, the target autonomous driving simulation scene image is generated, which solves the problem of increasing the number of samples generated by the generative model in the prior art and the simulation scene does not meet the requirements, and realizes the problem of the increase in the number of samples generated by the generative model generated by the existing technology. More efficient simulation scenario generation and better downstream task prediction accuracy.

CN119026313APending Publication Date: 2024-11-26TSINGHUA UNIVERSITY +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410909542.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-08
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The prior art cannot generate the autonomous driving simulation scenarios that are truly required for downstream tasks under the premise of sampling as few samples as possible, resulting in an increase in the number of generated samples of the generative model, and the generated simulation scene images do not meet the needs of downstream tasks.

Method used

By using the pre-trained generative model, the gradient and control intensity of the autonomous driving simulation scene image are obtained using the perceptual model, the sampling results of the generated model are adjusted, and the target autonomous driving simulation scene image is generated, which is used to train the optimized perceptual model and reduce the generated sampling. and improve the quality of the simulation scenario.

Benefits of technology

The number of generation samples of the generative model is reduced, so that the generated autonomous driving simulation scene images are more in line with the needs of downstream tasks, and the prediction accuracy of the perceptual model on downstream tasks is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119026313A_ABST
    Figure CN119026313A_ABST
Patent Text Reader

Abstract

The invention provides an automatic driving simulation scene generation method and device and electronic equipment, and the method comprises the steps: generating an automatic driving simulation scene image according to predicted normal distribution based on a pre-trained generative model; on the basis of a to-be-trained and optimized perception model, gradient and control intensity corresponding to the automatic driving simulation scene image are acquired; based on the generative model, generating a target automatic driving simulation scene image according to the gradient and the control intensity corresponding to the automatic driving simulation scene image; the target automatic driving simulation scene image is used for training the optimization perception model. According to the method, the directivity information for adjusting the automatic driving simulation scene image is provided by using the perception model, and the sampling result of the generative model is improved, so that the sampling generation frequency required by the generative model is reduced, the target automatic driving simulation scene image generated by the generative model can better meet the requirements of downstream tasks, and the efficiency of the automatic driving simulation scene image is improved. And the prediction accuracy of the perception model on a downstream task is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an autonomous driving simulation scene generation method, device and electronic equipment. Background Art

[0002] Existing technologies have achieved simulation of autonomous driving scenarios through generative models, but have not yet proposed a technical solution on how to generate autonomous driving simulation scenarios that are truly required for downstream tasks while ensuring that as few samples as possible are sampled. Summary of the invention

[0003] The present invention provides an autonomous driving simulation scene generation method, device and electronic device, which are used to overcome the defect that the prior art cannot generate the autonomous driving simulation scene that is truly required by downstream tasks under the premise of ensuring that the number of samples is sampled as little as possible, improve the assistance of the autonomous driving simulation scene images generated by the generative model to the downstream tasks, reduce the number of generation sampling times of the generative model, and make the generated autonomous driving simulation scene images more in line with the needs of downstream tasks and be better utilized.

[0004] On the one hand, the present invention provides a method for generating an autonomous driving simulation scene, comprising: based on a pre-trained generative model, generating an autonomous driving simulation scene image according to a predicted normal distribution; wherein the generative model is obtained by training and optimizing a training sample data set composed of image samples and their corresponding noise image samples; based on a perception model to be trained and optimized, obtaining a gradient and a control strength corresponding to the autonomous driving simulation scene image; wherein the gradient is used to provide directional information for adjusting the autonomous driving simulation scene image, and the control strength is used to amplify or reduce the influence of the gradient; based on the generative model, generating a target autonomous driving simulation scene image according to the gradient and the control strength corresponding to the autonomous driving simulation scene image; wherein the target autonomous driving simulation scene image is used to train and optimize the perception model.

[0005] Furthermore, based on the generative model, a target autonomous driving simulation scene image is generated according to the gradient and control strength corresponding to the autonomous driving simulation scene image, including: determining a target normal distribution according to the gradient and control strength corresponding to the autonomous driving simulation scene image; based on the generative model, generating a target autonomous driving simulation scene image according to the target normal distribution.

[0006] Furthermore, determining the target normal distribution according to the gradient and control strength corresponding to the autonomous driving simulation scene image includes: predicting the mean and variance of the image generated in the next time step based on the generative model; calculating the target mean according to the gradient and control strength corresponding to the autonomous driving simulation scene image, and the mean; and determining the target normal distribution according to the target mean and the variance.

[0007] Further, calculating the target mean according to the gradient and control strength corresponding to the autonomous driving simulation scene image, and the mean, includes: calculating the target mean by setting a formula, and the setting formula is as follows:

[0008] in, represents the target mean, represents the mean of the image generated at the next time step predicted by the generative model, Indicates the control strength, Represents the gradient.

[0009] Furthermore, the method of obtaining the gradient and control strength corresponding to the autonomous driving simulation scene image based on the perception model to be trained and optimized includes: determining a calculation graph corresponding to the perception model; and calculating the gradient corresponding to the autonomous driving simulation scene image according to the calculation graph during back propagation.

[0010] Furthermore, the value range of the control strength includes [0,10].

[0011] Furthermore, the generating of the target autonomous driving simulation scene image then includes: using the target autonomous driving simulation scene image to train and optimize the perception model; and using the trained perception model to predict the actual autonomous driving simulation scene image of the downstream task.

[0012] In a second aspect, the present invention also provides an autonomous driving simulation scene generation device, comprising: an autonomous driving simulation scene image generation module, for generating an autonomous driving simulation scene image based on a pre-trained generative model and a predicted normal distribution; wherein the generative model is obtained by training and optimizing a training sample data set consisting of image samples and their corresponding noise image samples; a gradient and control strength acquisition module, for acquiring the gradient and control strength corresponding to the autonomous driving simulation scene image based on a perception model to be trained and optimized; wherein the gradient is used to provide directional information for adjusting the autonomous driving simulation scene image, and the control strength is used to amplify or reduce the influence of the gradient; a target autonomous driving simulation scene image generation module, for generating a target autonomous driving simulation scene image based on the generative model and according to the gradient and control strength corresponding to the autonomous driving simulation scene image; wherein the target autonomous driving simulation scene image is used to train and optimize the perception model.

[0013] In a third aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an autonomous driving simulation scene generation method as described in any one of the above is implemented.

[0014] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for generating an autonomous driving simulation scene as described in any one of the above.

[0015] The automatic driving simulation scene generation method provided by the present invention generates an automatic driving simulation scene image according to a predicted normal distribution based on a pre-trained generative model; wherein the generative model is obtained by training and optimizing a training sample data set composed of image samples and their corresponding noise image samples; and based on the perception model to be trained and optimized, the gradient and control strength corresponding to the automatic driving simulation scene image are obtained; wherein the gradient is used to provide directional information for adjusting the automatic driving simulation scene image, and the control strength is used to amplify or reduce the influence of the gradient; and then based on the generative model, according to the gradient and control strength corresponding to the automatic driving simulation scene image, a target automatic driving simulation scene image is generated; wherein the target automatic driving simulation scene image is used to train and optimize the perception model. The method improves the sampling results of the generative model by using the perception model to provide directional information for adjusting the automatic driving simulation scene image. This process not only reduces the number of generation samplings required by the generative model, but also enables the target automatic driving simulation scene image generated by the generative model to better meet the requirements of downstream tasks, thereby improving the prediction accuracy of the perception model on downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flowchart of the method for generating an autonomous driving simulation scene provided by an embodiment of the present invention.

[0018] Figure 2 It is a schematic diagram of the overall process of the method for generating an autonomous driving simulation scene provided by an embodiment of the present invention.

[0019] Figure 3 It is a schematic diagram of multiple rounds of interactive feedback between a generative model and a perceptual model provided in an embodiment of the present invention.

[0020] Figure 4 It is a structural schematic diagram of an automatic driving simulation scene generation device provided in an embodiment of the present invention.

[0021] Figure 5 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] It should be noted that although the existing technology has achieved the simulation of autonomous driving scenarios through generative models, it has not yet proposed a technical solution on how to generate the autonomous driving simulation scenarios that are truly required for downstream tasks while ensuring that as few samples as possible are sampled.

[0024] In the existing technology, the generative model uses the pre-learned generative model to predict the mean and variance of the next generated image in each sampling process: At this moment, the model is based on the current noise image Predicting the next step The mean of the time and variance Then, from the mean and variance Sample the image generated in the next step from the normal distribution with parameters This sampling process is carried out iteratively times, we can get the standard Gaussian noise from Sampling to obtain autonomous driving road images However, this process does not take into account the needs of perceptual model learning and lacks sampling efficiency.

[0025] In view of this, the present invention proposes a method for generating an autonomous driving simulation scene. Specifically, Figure 1 A schematic diagram of a flow chart of a method for generating an autonomous driving simulation scene provided in an embodiment of the present invention is shown.

[0026] like Figure 1 As shown, the method includes steps S110-S130, and these steps will be described in detail below with reference to the accompanying drawings.

[0027] S110, based on a pre-trained generative model, generating an autonomous driving simulation scene image according to a predicted normal distribution; wherein the generative model is trained and optimized based on a training sample data set consisting of image samples and their corresponding noise image samples.

[0028] It can be understood that this step pre-trains a generative model by sampling noise from a standard normal distribution as the input of the generative model, thereby obtaining an output autonomous driving simulation scene image.

[0029] In a specific embodiment, the generative model is a diffusion model.

[0030] The diffusion model generates data by simulating a process that gradually goes from an ordered state to a disordered state (forward process) and from a disordered state to an ordered state (reverse process). In the simulation of the autonomous driving scenario, the diffusion model first gradually introduces noise into the image in the forward process, eventually reaching a state close to the standard normal distribution. Then, in the reverse process, the diffusion model gradually predicts and removes this noise, and finally restores a clear image.

[0031] It is worth mentioning that the generative model in this step is pre-trained. Specifically, during training, an image sample is randomly selected, uniformly sampled to obtain time round t, and the sampling noise is sampled from the standard normal distribution. The noise image sample after the tth round of noise is obtained by the formula. Then, the noise image sample and the time step t are input into the generative model together to predict the noise value, and the difference between the sampling noise and the predicted noise value is used as the loss to update the optimization model parameters. After multiple iterations, the trained generative model can be obtained.

[0032] S120, based on the perception model to be trained and optimized, obtain the gradient and control strength corresponding to the autonomous driving simulation scene image; wherein the gradient is used to provide directional information for adjusting the autonomous driving simulation scene image, and the control strength is used to amplify or reduce the influence of the gradient.

[0033] It can be understood that, based on the fact that step S110 generates the autonomous driving simulation scene image according to the predicted normal distribution based on the pre-trained generative model, step S120 will further utilize the sampling result perception model of the improved generative model to obtain the gradient and control strength corresponding to the autonomous driving simulation scene image.

[0034] Specifically, the perception model in this step is a model that has not been trained and optimized, or a model that has been initially trained and optimized but needs further training and optimization. The calculation in the perception model can be regarded as a combination of a series of operations (such as addition, multiplication, activation function, etc.), which can be represented as nodes in the calculation graph, and the connections between nodes represent data dependencies. Each node has one or more inputs (outputs from other nodes) and an output.

[0035] In the forward propagation, the autonomous driving simulation scene image flows from the input layer of the perception model to the output layer, passes through each node in the computational graph, and generates corresponding outputs, which will serve as inputs to subsequent nodes until the prediction results of the perception model are finally obtained.

[0036] Then, a loss function is adopted to evaluate the gap between the prediction results of the perception model and the true results and minimize it during the training process.

[0037] Next, we start from the loss function node and propagate backwards along the computational graph. In the backward propagation process, we can only calculate the gradient of each node, that is, the partial derivative of the loss function with respect to the input of this stage. These gradients indicate how to adjust the input of the node to minimize the loss function.

[0038] With the gradient of each parameter, we can use gradient descent to update the parameters of the perception model. Through repeated forward propagation, loss calculation, backpropagation, and parameter update, the perception model will gradually learn to extract useful features from the input data and produce accurate predictions.

[0039] Based on the above process, the gradient corresponding to the autonomous driving simulation scene image can be solved.

[0040] In addition, the control intensity in this step is used to amplify or reduce the influence of the gradient, and can be manually adjusted according to actual conditions, and is not specifically limited here.

[0041] It should be emphasized that the process of training and optimizing the perception model described above is a preliminary training and optimization process. In the process of further training and optimizing the perception model, the training samples used are the target autonomous driving simulation scene images generated in step S130. Since the target autonomous driving simulation scene images are more in line with the requirements of downstream tasks, using them to train and optimize the perception model is more effective and the performance of the perception model is better.

[0042] It should also be noted that the model architecture of the perception model can be adjusted according to the requirements of downstream tasks and is not specifically limited here.

[0043] Among them, downstream tasks include but are not limited to target detection tasks, semantic segmentation tasks, etc., which are not specifically limited here.

[0044] S130, based on the generative model, generating a target autonomous driving simulation scene image according to the gradient and control strength corresponding to the autonomous driving simulation scene image; wherein the target autonomous driving simulation scene image is used to train and optimize the perception model.

[0045] It can be understood that, based on the fact that step S120 obtains the gradient and control strength corresponding to the autonomous driving simulation scene image based on the perception model to be trained and optimized, step S130 further generates the target autonomous driving simulation scene image based on the generative model and the gradient and control strength corresponding to the autonomous driving simulation scene image.

[0046] Specifically, first, the generative model is used to predict the mean and variance of the next generated image; then, according to the gradient, control strength and the aforementioned predicted mean corresponding to the autonomous driving simulation scene image, the target mean of the normal distribution sampled by the generative model is calculated; finally, based on the generative model, the target autonomous driving simulation scene image is generated according to the normal distribution determined by the target mean and variance.

[0047] In other words, the gradient and control strength obtained by the perception model directly affect the mean of the normal distribution sampled by the generative model, thereby making the generated target autonomous driving simulation scene image more in line with the actual needs of downstream tasks.

[0048] In other words, there is a mutual feedback relationship between the perception model and the generative model. First, the generative model feeds back the autonomous driving simulation scene images it generates to the perception model. Then, the perception model obtains the gradient and control strength corresponding to the autonomous driving simulation scene images, and feeds them back to the generative model, telling the generative model "in which scenes the simulation effects still need to be improved". Then, the generative model generates target autonomous driving simulation scene images in the direction given by the perception model. In this way, the target autonomous driving simulation scene images that are more in line with downstream tasks are used to train and optimize the perception model, so that the perception model can have better prediction accuracy on downstream tasks.

[0049] It should also be noted that, in the present embodiment, steps S110-S130 are a continuously iterative process, which can be performed synchronously on the perception model and the generative model, or asynchronously on different graphics cards (perception model, generative model), and the data can be organized into queues.

[0050] In this embodiment, an autonomous driving simulation scene image is generated according to a predicted normal distribution based on a pre-trained generative model; wherein the generative model is obtained by training and optimizing a training sample data set composed of image samples and their corresponding noise image samples; and based on the perception model to be trained and optimized, a gradient and control strength corresponding to the autonomous driving simulation scene image are obtained; wherein the gradient is used to provide directional information for adjusting the autonomous driving simulation scene image, and the control strength is used to amplify or reduce the influence of the gradient; and then based on the generative model, a target autonomous driving simulation scene image is generated according to the gradient and control strength corresponding to the autonomous driving simulation scene image; wherein the target autonomous driving simulation scene image is used to train and optimize the perception model. The method improves the sampling results of the generative model by using the perception model to provide directional information for adjusting the autonomous driving simulation scene image. This process not only reduces the number of generation samplings required by the generative model, but also enables the target autonomous driving simulation scene image generated by the generative model to better meet the requirements of downstream tasks, thereby improving the prediction accuracy of the perception model on downstream tasks.

[0051] On the basis of the above-mentioned embodiments, the following will further describe in detail the process of the generative model generating the target autonomous driving simulation scene image according to the gradient and control strength fed back by the perception model.

[0052] According to the gradient and control strength corresponding to the autonomous driving simulation scene image, a target autonomous driving simulation scene image is generated, including: determining a target normal distribution according to the gradient and control strength corresponding to the autonomous driving simulation scene image; and generating a target autonomous driving simulation scene image according to the target normal distribution based on a generative model.

[0053] It can be understood that after using the perception model to obtain the gradient and control strength corresponding to the autonomous driving simulation scene image, the target normal distribution is first determined based on the gradient and control strength corresponding to the autonomous driving simulation scene image.

[0054] Specifically, after generating the autonomous driving simulation scene image, the generative model will predict the mean and variance of the image generated in the next time step. On this basis, the target mean is calculated according to the gradient and control strength corresponding to the autonomous driving simulation scene image, as well as the mean.

[0055] Specifically, the target mean is calculated by setting a formula, and the setting formula is as follows (1).

[0056] (1)

[0057] in, represents the target mean, represents the mean of the image generated at the next time step predicted by the generative model, Indicates the control strength, Represents the gradient.

[0058] Based on the calculated target mean and the variance predicted by the generative model, a normal distribution, namely the target normal distribution, can be determined.

[0059] Then, the generative model samples the standard Gaussian noise in the target normal distribution to generate the target autonomous driving simulation scene image. Up to this point, it is a closed-loop feedback process.

[0060] Next, in the second closed-loop feedback process, the target autonomous driving simulation scene image is fed back to the perception model, and the perception model further obtains the gradient and control strength corresponding to the target autonomous driving simulation scene image. Then, according to the gradient and control strength corresponding to the target autonomous driving simulation scene image, another target normal distribution is determined. Finally, the generative model is used to sample the aforementioned target normal distribution to generate another target autonomous driving simulation scene image.

[0061] By repeating the aforementioned closed-loop feedback process, the target autonomous driving simulation scene image generated at each time step is more in line with the specific requirements of the downstream tasks. This method not only improves the quality of the generated images, but also reduces the necessary number of sampling times, thereby improving the overall sample efficiency.

[0062] In this embodiment, the target normal distribution is determined according to the gradient and control strength corresponding to the autonomous driving simulation scene image, and the target autonomous driving simulation scene image is generated according to the target normal distribution based on the generative model. This method improves the sampling results of the generative model by using the perception model to provide directional information for adjusting the autonomous driving simulation scene image. This process not only reduces the number of generation samplings required by the generative model, but also enables the target autonomous driving simulation scene image generated by the generative model to better meet the needs of downstream tasks, thereby improving the prediction accuracy of the perception model on downstream tasks.

[0063] On the basis of the above-mentioned embodiments, the following will further describe in detail the process of using the perception model to obtain the gradient and control strength corresponding to the autonomous driving simulation image, and the process of training and optimizing the perception model.

[0064] Based on the perception model to be trained and optimized, the gradient and control strength corresponding to the autonomous driving simulation scene image are obtained, including: determining a calculation graph corresponding to the perception model; in the back propagation process, calculating the gradient corresponding to the autonomous driving simulation scene image according to the calculation graph.

[0065] It is understandable that the perception model in this embodiment is a model that has not been trained and optimized, or a model that has been initially trained and optimized but needs further training and optimization. The calculation in the perception model can be regarded as a combination of a series of operations (such as addition, multiplication, activation function, etc.), which can be represented as nodes in the calculation graph, and the connections between the nodes represent data dependencies. Each node has one or more inputs (outputs from other nodes) and an output.

[0066] In the forward propagation, the autonomous driving simulation scene image flows from the input layer of the perception model to the output layer, passes through each node in the computational graph, and generates corresponding outputs, which will serve as inputs to subsequent nodes until the prediction results of the perception model are finally obtained.

[0067] Then, a loss function is adopted to evaluate the gap between the prediction results of the perception model and the true results and minimize it during the training process.

[0068] Next, we start from the loss function node and propagate backwards along the computational graph. In the backward propagation process, we can only calculate the gradient of each node, that is, the partial derivative of the loss function with respect to the input of this stage. These gradients indicate how to adjust the input of the node to minimize the loss function.

[0069] With the gradient of each parameter, we can use gradient descent to update the parameters of the perception model. Through repeated forward propagation, loss calculation, backpropagation, and parameter update, the perception model will gradually learn to extract useful features from the input data and produce accurate predictions.

[0070] Based on the above process, the gradient corresponding to the autonomous driving simulation scene image can be solved.

[0071] In addition, the control intensity in this step is used to amplify or reduce the influence of the gradient, and can be manually adjusted according to actual conditions, and is not specifically limited here.

[0072] For example, in a specific embodiment, the value range of the control strength is [0, 10].

[0073] It should be emphasized that the process of training and optimizing the perception model described above is a preliminary training and optimization process, and the samples used are autonomous driving simulation scene images generated by a generative model. In the embodiment provided by the present invention, the training samples used for training and optimizing the perception model are the target autonomous driving simulation scene images generated in step S130. Since the target autonomous driving simulation scene images are more in line with the requirements of downstream tasks, using them to train and optimize the perception model is more effective, the performance of the perception model is better, and the prediction accuracy is higher.

[0074] It should be noted that the process of training and optimizing the perception model using the target autonomous driving simulation image in this embodiment can refer to the process of training and optimizing the perception model described above except that the training samples are different.

[0075] After the perception model is trained, the trained perception model can be used to predict some downstream tasks. The downstream tasks here include but are not limited to target detection tasks, semantic segmentation tasks, etc.

[0076] In this embodiment, by determining the computational graph corresponding to the perception model, and calculating the gradient corresponding to the autonomous driving simulation scene image according to the computational graph during the back propagation process, manually setting the control strength, and then based on the generative model, generating the target autonomous driving simulation scene image according to the gradient and control strength corresponding to the autonomous driving simulation scene image; wherein the target autonomous driving simulation scene image is used to train and optimize the perception model. This method improves the sampling results of the generative model by using the perception model to provide directional information for adjusting the autonomous driving simulation scene image. This process not only reduces the number of generation samplings required by the generative model, but also enables the target autonomous driving simulation scene image generated by the generative model to better meet the needs of downstream tasks, thereby improving the prediction accuracy of the perception model on downstream tasks.

[0077] Additionally, in some embodiments, Figure 2 The figure shows the overall process diagram of the method for generating an autonomous driving simulation scene provided by an embodiment of the present invention.

[0078] Specifically, Figure 2As shown in FIG, first, the generative model generates an autonomous driving simulation scene image according to the predicted normal distribution, and feeds the autonomous driving simulation scene image back to the perception model. The perception model obtains the gradient and control strength corresponding to the autonomous driving simulation scene image.

[0079] Then, a target normal distribution is determined based on the gradient and control strength corresponding to the autonomous driving simulation scene image, as well as the mean and variance of the image generated at the next time step predicted by the generative model.

[0080] Next, the target normal distribution is sampled using the generative model to generate the target autonomous driving simulation scene image.

[0081] Then, the target autonomous driving simulation scene images are used to train and optimize the perception model to obtain a trained perception model.

[0082] Finally, the trained perception model is used to make high-accuracy predictions on actual autonomous driving simulation scene images for downstream tasks.

[0083] It should be noted that only a closed-loop feedback process for generating the target autonomous driving simulation scene image is described here, but it is actually a cyclical process that is not expanded here.

[0084] In some other embodiments, Figure 3 A schematic diagram of multiple rounds of interactive feedback between a generative model and a perceptual model provided in an embodiment of the present invention is shown.

[0085] according to Figure 3 It can be seen that the interactive feedback between the generative model and the perceptual model is a multi-round, cyclical process.

[0086] In each closed-loop feedback process, the perception model will feed back a corresponding gradient and control strength to the generative model, and then obtain a target normal distribution. The mean of the target normal distribution is , the variance is , represents the current time step, is the mean of the images generated at the next time step predicted by the generative model, To control the intensity, is the gradient.

[0087] Corresponding to the autonomous driving simulation scene generation method described in the above embodiments, the present invention also proposes an autonomous driving simulation scene generation device. Specifically, Figure 4 A schematic diagram of the structure of an automatic driving simulation scene generation device provided in an embodiment of the present invention is shown.

[0088] like Figure 4As shown, the device includes: an autonomous driving simulation scene image generation module 410, which is used to generate an autonomous driving simulation scene image according to a predicted normal distribution based on a pre-trained generative model; wherein the generative model is obtained by training and optimizing a training sample data set composed of image samples and their corresponding noise image samples; a gradient and control strength acquisition module 420, which is used to acquire the gradient and control strength corresponding to the autonomous driving simulation scene image based on the perception model to be trained and optimized; wherein the gradient is used to provide directional information for adjusting the autonomous driving simulation scene image, and the control strength is used to amplify or reduce the influence of the gradient; a target autonomous driving simulation scene image generation module 430, which is used to generate a target autonomous driving simulation scene image based on the generative model and according to the gradient and control strength corresponding to the autonomous driving simulation scene image; wherein the target autonomous driving simulation scene image is used to train and optimize the perception model.

[0089] In this embodiment, the automatic driving simulation scene image generation module 410 generates the automatic driving simulation scene image according to the predicted normal distribution based on the pre-trained generative model; wherein the generative model is obtained by training and optimizing the training sample data set composed of the image sample and its corresponding noise image sample; the gradient and control strength acquisition module 420 acquires the gradient and control strength corresponding to the automatic driving simulation scene image based on the perception model to be trained and optimized; wherein the gradient is used to provide directional information for adjusting the automatic driving simulation scene image, and the control strength is used to amplify or reduce the influence of the gradient; and then the target automatic driving simulation scene image generation module 430 generates the target automatic driving simulation scene image based on the generative model according to the gradient and control strength corresponding to the automatic driving simulation scene image; wherein the target automatic driving simulation scene image is used to train and optimize the perception model. The device improves the sampling result of the generative model by using the perception model to provide directional information for adjusting the automatic driving simulation scene image. This process not only reduces the number of generation samplings required by the generative model, but also enables the target automatic driving simulation scene image generated by the generative model to better meet the requirements of downstream tasks, thereby improving the prediction accuracy of the perception model on downstream tasks.

[0090] It should be noted that the autonomous driving simulation scene generation device provided in this embodiment can be referenced to each other with the autonomous driving simulation scene generation method described in the above embodiments, and will not be repeated here.

[0091] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the method for generating an autonomous driving simulation scene, the method comprising: generating an autonomous driving simulation scene image based on a pre-trained generative model according to a predicted normal distribution; wherein the generative model is obtained by training and optimizing a training sample data set composed of image samples and their corresponding noise image samples; obtaining a gradient and a control strength corresponding to the autonomous driving simulation scene image based on a perception model to be trained and optimized; wherein the gradient is used to provide directional information for adjusting the autonomous driving simulation scene image, and the control strength is used to amplify or reduce the influence of the gradient; based on the generative model, generating a target autonomous driving simulation scene image according to the gradient and the control strength corresponding to the autonomous driving simulation scene image; wherein the target autonomous driving simulation scene image is used to train and optimize the perception model.

[0092] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0093] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the autonomous driving simulation scene generation method provided by the above-mentioned methods, the method comprising: generating an autonomous driving simulation scene image based on a predicted normal distribution based on a pre-trained generative model; wherein the generative model is obtained by training and optimizing a training sample data set consisting of image samples and their corresponding noise image samples; obtaining a gradient and a control strength corresponding to the autonomous driving simulation scene image based on a perception model to be trained and optimized; wherein the gradient is used to provide directional information for adjusting the autonomous driving simulation scene image, and the control strength is used to amplify or reduce the influence of the gradient; based on the generative model, generating a target autonomous driving simulation scene image according to the gradient and the control strength corresponding to the autonomous driving simulation scene image; wherein the target autonomous driving simulation scene image is used to train and optimize the perception model.

[0094] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0095] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating an autonomous driving simulation scene, It is characterized in that include: Based on a pre-trained generative model, an autonomous driving simulation scene image is generated according to a predicted normal distribution; wherein the generative model is trained and optimized based on a training sample data set consisting of image samples and their corresponding noise image samples; Based on the perception model to be trained and optimized, the gradient and control strength corresponding to the autonomous driving simulation scene image are obtained; wherein the gradient is used to provide directional information for adjusting the autonomous driving simulation scene image, and the control strength is used to amplify or reduce the influence of the gradient; Based on the generative model, a target autonomous driving simulation scene image is generated according to the gradient and control strength corresponding to the autonomous driving simulation scene image; wherein the target autonomous driving simulation scene image is used to train and optimize the perception model.

2. The method for generating an autonomous driving simulation scene according to claim 1, It is characterized in that The generating a target autonomous driving simulation scene image based on the generative model and according to the gradient and control strength corresponding to the autonomous driving simulation scene image includes: Determining a target normal distribution according to a gradient and a control strength corresponding to the autonomous driving simulation scene image; Based on the generative model, a target autonomous driving simulation scene image is generated according to the target normal distribution.

3. The method for generating an autonomous driving simulation scene according to claim 2, It is characterized in that The determining of the target normal distribution according to the gradient and control strength corresponding to the autonomous driving simulation scene image includes: Based on the generative model, predict the mean and variance of the image generated at the next time step; Calculating a target mean value according to the gradient and control strength corresponding to the autonomous driving simulation scene image and the mean value; The target normal distribution is determined according to the target mean and the variance.

4. The method for generating an autonomous driving simulation scene according to claim 3, It is characterized in that The step of calculating the target mean value according to the gradient and control strength corresponding to the autonomous driving simulation scene image and the mean value comprises: The target mean is calculated by setting a formula, which is as follows: in, represents the target mean, represents the mean of the image generated at the next time step predicted by the generative model, Indicates the control strength, Represents the gradient.

5. The method for generating an autonomous driving simulation scene according to claim 1, It is characterized in that The step of obtaining the gradient and control strength corresponding to the autonomous driving simulation scene image based on the perception model to be trained and optimized includes: Determining a computational graph corresponding to the perception model; During the back propagation process, the gradient corresponding to the autonomous driving simulation scene image is calculated according to the calculation graph.

6. The method for generating an autonomous driving simulation scene according to claim 1, It is characterized in that The value range of the control strength includes [0,10].

7. The method for generating an autonomous driving simulation scene according to any one of claims 1 to 6, It is characterized in that The generating of the target autonomous driving simulation scene image then includes: Using the target autonomous driving simulation scene image, training and optimizing the perception model; The trained perception model is used to predict actual autonomous driving simulation scene images for downstream tasks.

8. An automatic driving simulation scene generation device, It is characterized in that include: An autonomous driving simulation scene image generation module, used to generate an autonomous driving simulation scene image based on a pre-trained generative model and a predicted normal distribution; wherein the generative model is obtained by training and optimizing a training sample data set consisting of image samples and their corresponding noise image samples; A gradient and control strength acquisition module, used to acquire the gradient and control strength corresponding to the autonomous driving simulation scene image based on the perception model to be trained and optimized; wherein the gradient is used to provide directional information for adjusting the autonomous driving simulation scene image, and the control strength is used to amplify or reduce the influence of the gradient; The target autonomous driving simulation scene image generation module is used to generate a target autonomous driving simulation scene image based on the generative model and according to the gradient and control strength corresponding to the autonomous driving simulation scene image; wherein the target autonomous driving simulation scene image is used to train and optimize the perception model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, it implements the method for generating an autonomous driving simulation scene image according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method for generating an autonomous driving simulation scene image according to any one of claims 1 to 7 is implemented.