Automatic driving simulation method, storage medium and computer equipment

Abstract

The invention discloses an automatic driving simulation method, a storage medium and computer equipment. The method comprises the following steps: inputting a camera pose into a trained neural radiation field network; the neural radiation field network performs rendering based on the camera pose to obtain a simulation image corresponding to the camera pose; and the simulation image is transmitted to an automatic driving algorithm, and the automatic driving algorithm performs simulation testing based on the simulation image. In this way, the authenticity of the scene in the automatic driving simulation task can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an autonomous driving simulation method, storage medium and computer equipment. Background Art

[0002] At present, the scenes and sensor simulation data in the autonomous driving simulation tasks are mainly generated through game engine-based methods. However, there are differences between the sensor simulation data generated by this method and the real data, which will affect the detection accuracy of perception, and this is particularly prominent in the autonomous driving parking task. Because the perception requirements of the parking task are high in accuracy, and when there are obstacles very close to the vehicle, the data simulated by the sensors based on the game engine simulation is not realistic enough, which will affect the detection accuracy of the perception system at close range, and it is impossible to conduct a closed-loop simulation that truly introduces perception, and it cannot meet the autonomous driving simulation test requirements for parking. At the same time, the sensors based on game engine simulation also rely on hardware equipment, have high construction costs, and the generated images support few sensor types. Summary of the invention

[0003] This application mainly provides an autonomous driving simulation method, storage medium and computer equipment to solve the problem of low scene authenticity in autonomous driving simulation tasks.

[0004] To solve the above technical problems, a technical solution adopted in the present application is: to provide an autonomous driving simulation method, comprising: inputting the camera pose into a trained neural radiation field network; the neural radiation field network is rendered based on the camera pose to obtain a simulated image corresponding to the camera pose; the simulated image is transmitted to the autonomous driving algorithm, and the autonomous driving algorithm performs a simulation test based on the simulated image.

[0005] Rendering images through the trained neural radiation field network is conducive to improving the realism of the simulated image and making the simulated image closer to the effect of real camera shooting. The neural radiation field network can synthesize images of any new perspective based on known perspective images, increase the environmental information that can be obtained by the autonomous driving simulation vehicle, and have a flexible simulation perspective. The neural radiation field network also supports different autonomous driving simulation sensors and has higher versatility. The rendered images are processed by the autonomous driving algorithm, the simulated driving environment is constructed, the autonomous driving process is executed, and the autonomous driving simulation tests based on simulated images such as autonomous driving algorithm training, software in loop testing (Software in Loop, SIL), world simulation (WorldSim) or log simulation (LogSim) are completed.

[0006] In some embodiments, before inputting the camera pose into the trained neural radiation field network, it also includes: inputting the neural radiation field network with multiple groups of camera poses and real camera images as training sets; calculating the distortion parameters of the sampling rays in the neural radiation field network through camera distortion parameters to determine the correspondence between the position points on the real camera image and the positions in the scene under the corresponding camera pose; performing image rendering based on the correspondence between the position points on the real camera image and the positions in the scene to obtain a simulated image under the corresponding camera pose; and updating the parameters of the neural radiation field network based on the difference between the simulated image and the real camera image.

[0007] By considering the camera distortion parameters, the position of the sampling ray on the real camera image can be calculated more accurately, thereby establishing an accurate correspondence between the position in the scene and the position on the image, improving the realism of the image rendering and the accuracy of the image information. Combining the camera distortion parameters enables the neural radiation field network to directly use the distorted images taken by the on-board camera for training and rendering, reducing training costs and improving rendering efficiency.

[0008] In some embodiments, updating the parameters of the neural radiation field network based on the difference between the simulated image and the real camera image includes: calculating a loss function based on the difference between the simulated image and the real camera image; updating the parameters of the neural radiation field network by gradient backpropagation using the gradient associated with the loss function; and ending the training of the neural radiation field network in response to the convergence of the loss function.

[0009] Updating network parameters through gradient backpropagation can accurately guide the optimization direction of network parameters, converge to the optimal solution more quickly, improve the rendering quality of the neural radiation field network model, and make the generated images more accurate and realistic.

[0010] In some embodiments, the loss function includes at least one of: color loss, structural similarity loss, regularization loss, depth loss, perceptual loss, or mask loss.

[0011] Color loss is used to measure the color difference between the rendered image and the real image, structural similarity loss is used to evaluate the similarity between images, regularization loss is used to constrain the complexity of network parameters, depth loss is used to measure the difference between the estimated depth and the real depth in the rendered image, mask loss is used to train the impact of occluded areas on the training process, and perceptual loss is used to capture the semantic and structural similarities between images. Through the above loss functions alone or in combination, the neural radiation field network can generate more visually realistic rendered images.

[0012] In some embodiments, after the neural radiation field network is input with multiple groups of camera poses and real camera images as training sets, it also includes: adding a mask on the real camera image to segment the body area and non-body area in the real camera image; the image rendering based on the correspondence between the position points on the real camera image and the positions in the scene includes: respectively rendering the images of the body area and the non-body area based on the correspondence between the position points in the body area and the non-body area and the positions in the scene; the simulated image under the corresponding camera pose includes: a simulated image of the body area and a simulated image of the non-body area under the corresponding camera pose.

[0013] By training the vehicle body and non-vehicle parts in the original image separately, the vehicle body area moving with the vehicle can be prevented from affecting the rendering of the stationary non-vehicle area in the scene, thereby improving the effect of neural radiation field network training.

[0014] In some embodiments, the neural radiation field network performs rendering based on the camera pose to obtain a simulated image corresponding to the camera pose, including: the neural radiation field network renders the body area and non-body area of ​​the real camera image onto the simulated image based on the camera pose.

[0015] By rendering the body and non-body parts of the original image separately and then overlaying them after rendering, the influence of the body part on the rendering of the non-body part is avoided, and the rendering effect of the neural radiation field network is improved.

[0016] In some embodiments, the distortion parameters of the sampling rays in the neural radiation field network are calculated by using the camera distortion parameters to determine the correspondence between the position points on the camera's real image and the positions in the scene under the corresponding camera pose, including: calculating the reference positions of the sampling points on the sampling rays by using the camera distortion parameters; adjusting the direction and position of the sampling rays based on the reference positions of the sampling points to determine the correspondence between the position points on the camera's real image and the positions in the scene under the corresponding camera pose.

[0017] The reference position of the sampling point is calculated through the distortion parameters, and the sampling rays are adjusted so that the neural radiation field network can process distorted images with the same distortion parameters, avoiding the need to remove the distortion of the image separately before input, thereby improving the efficiency of image training and rendering.

[0018] In some embodiments, the camera pose is derived from the real camera image.

[0019] By directly extracting from the real camera image through image processing technology, additional collection of camera pose data is avoided, the workload of obtaining training data is reduced, and the training efficiency of the neural radiation field network is improved.

[0020] In order to solve the above-mentioned technical problems, another technical solution adopted in the present application is: providing a storage medium on which program data is stored, and when the program data is executed by a processor, the steps of the above-mentioned autonomous driving simulation method are implemented.

[0021] The beneficial effects of the storage medium refer to the beneficial effects of the above-mentioned autonomous driving simulation method and will not be repeated here.

[0022] The present application also provides a computer device, comprising a processor and a memory connected to each other, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the autonomous driving simulation method as described above are implemented.

[0023] The beneficial effects of the computer device refer to the beneficial effects of the above-mentioned autonomous driving simulation method, which will not be repeated here.

[0024] The beneficial effects of the present application are as follows: Different from the prior art, the present application discloses an autonomous driving simulation method, storage medium and computer equipment. Inputting the camera posture into the trained neural radiation field network, and generating a real-time simulation image through the updated neural radiation field network, is conducive to improving the realism of the simulation image, which is closer to the effect of actual camera shooting, increases the environmental information that can be obtained by the autonomous driving simulation vehicle, has a flexible simulation perspective, and reduces the cost of rendering the simulated autonomous driving image. The autonomous driving algorithm performs simulation testing based on the simulated image to complete the simulated autonomous driving, which reduces the cost of rendering the simulated autonomous driving image and improves the rendering efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0026] Figure 1 It is a flowchart of an embodiment of an autonomous driving simulation method provided by the present application;

[0027] Figure 2 Yes Figure 1 A flow chart of another embodiment of an autonomous driving simulation method is shown;

[0028] Figure 3 Yes Figure 1 The flowchart of step 20 of the automatic driving simulation method shown in FIG.

[0029] Figure 4It is a comparison between the real camera image (a) and the simulated image (b);

[0030] Figure 5 Yes Figure 1 The flowchart of step 40 of the automatic driving simulation method shown in FIG.

[0031] Figure 6 Yes Figure 1 A schematic flow chart of another embodiment of the autonomous driving simulation method shown;

[0032] Figure 7 It is a structural schematic diagram of an embodiment of a storage medium provided by the present application;

[0033] Figure 8 It is a structural diagram of an embodiment of a computer device provided by the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] The terms "first", "second", "third" in the embodiments of the present application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second", "third" can expressly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0036] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0037] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of an autonomous driving simulation method provided by the present application, and the autonomous driving simulation method includes:

[0038] Step 50: Input the camera pose into the trained neural radiance field network.

[0039] The camera pose of the vehicle currently undergoing simulation test is input into the trained neural radiation field network for image rendering to obtain a highly realistic simulation image.

[0040] Step 60: The neural radiation field network performs rendering based on the camera pose to obtain a simulated image corresponding to the camera pose.

[0041] The neural radiance field network determines the observation angle of the vehicle in the virtual scene based on the vehicle position and orientation defined by the camera pose, and synthesizes the pixel color combination at this angle by combining the distortion of the sampling rays to obtain a distorted simulation image corresponding to the camera pose.

[0042] Rendering images through the trained neural radiation field network helps improve the realism of the rendered images and make them closer to the effects of actual camera shooting. The neural radiation field network can synthesize images of any new perspective based on known perspective images, increase the environmental information that can be obtained by the autonomous driving simulation vehicle, and have a flexible simulation perspective. The neural radiation field network also supports different autonomous driving simulation sensors and has higher versatility.

[0043] Step 70: Transmit the simulation image to the autonomous driving algorithm, and the autonomous driving algorithm performs a simulation test based on the simulation image.

[0044] A neural radiation field network is used to generate highly realistic simulated images for simulation testing based on preset dynamically changing scene parameters to simulate the driving conditions of autonomous vehicles in the real world.

[0045] By processing the rendered images with the autonomous driving algorithm, a simulated driving environment is constructed, and the autonomous driving process is executed. Training or testing based on the simulation environment, such as autonomous driving algorithm training, software-in-the-loop testing, world simulation or log simulation, is completed, thereby reducing the cost of autonomous driving simulation testing based on simulated images.

[0046] See also Figure 2 , Figure 2 Yes Figure 1 The flowchart of another embodiment of the automatic driving simulation method shown in FIG. 5 is a flowchart of another embodiment of the automatic driving simulation method. Before step 50, the automatic driving simulation method further includes:

[0047] Step 10: Input the neural radiance field network with multiple sets of camera poses and real camera images as training sets.

[0048] Multiple groups of camera poses and real camera images are collected to construct a training set for training the neural radiation field network. The training set is input into the neural radiation field network to train the neural radiation field network.

[0049] Neural Radiance Field (NeRF) is a fully connected neural network that can reconstruct complex 3D scenes from a partial 2D image set. NeRF receives the 3D spatial position and 2D viewing direction obtained from the camera pose, outputs 4D color and volume density, and then uses rendering to draw the output into a 2D simulated image. NeRF is used to create realistic visual effects, simulations, and scenes. Compared with traditional game engine simulation technology, NeRF can render simulated images with higher realism, build more complex virtual scenes, and require simpler data and lower construction costs.

[0050] The camera pose is used to describe the spatial position and orientation of the camera when taking an image. The camera pose is directly input into the neural radiation field network as input data for image rendering. During the training process of the neural radiation field network, the spatial position and orientation of the current image are determined by the camera pose to match the corresponding real image of the camera; during the rendering process of the neural radiation field network, the camera pose is used to render a simulated image in the current direction to construct a virtual driving scene.

[0051] Specifically, the camera pose is derived from the real camera image. Methods for obtaining the camera pose from the real camera image include: Structure from Motion (SFM) technology, which infers the three-dimensional structure of the scene and the motion of the camera by analyzing the images captured by the camera at different perspectives; Multi-View Stereo (MVS), which analyzes the target scene images taken by multiple cameras from different perspectives, finds the corresponding feature points in the image as the matching points of the spatial position between the images, and uses the position information of the matching points in the image to restore the three-dimensional structure of the scene; Depth camera, which can directly obtain the three-dimensional information of the object under test without complex image processing and calculation; Stereo Vision technology, which estimates the depth and three-dimensional structure of objects in the scene by comparing and analyzing two or more pictures taken from different angles.

[0052] The real camera image is used as a reference image for the training of the neural radiation field network. It corresponds to the camera pose and represents the actual image captured by the camera at the current camera pose. The real camera image comes from multiple cameras installed on the self-driving car. The shooting range of the camera covers the area around the vehicle body. These cameras shoot and record the driving scene, that is, the environment around the vehicle during driving, forming a large amount of driving scene camera image data as real camera data. The training of the neural radiation field network only requires image data obtained during driving, without the need to collect additional training set data, which reduces the workload of constructing the early training set and reduces the training cost.

[0053] By using multiple sets of camera poses and real camera images as training sets to input the neural radiation field network, the three-dimensional space establishment capability of the neural radiation field network is trained to improve the quality and realism of the generated simulation scenes.

[0054] Step 20: Calculate the distortion parameters of the sampled rays in the neural radiation field network through the camera distortion parameters to determine the correspondence between the position points on the camera real image and the positions in the scene under the corresponding camera posture.

[0055] Obtain the distortion parameters of the camera that takes the real image, and use the distortion parameters to add distortion to the sampling rays in the neural radiation field network, so that the position points in the distorted real image of the camera correspond to the positions in the actual scene or the virtual scene corresponding to the image, and determine the position correspondence between the two-dimensional image and the three-dimensional space.

[0056] Camera distortion parameters describe the image distortion caused by the curvature of the optical system and other factors during the shooting process. Camera distortion parameters can be obtained by using camera calibration technology, by shooting multiple calibration plate images of known geometric shapes, and using the feature points in these images to solve the camera intrinsic and extrinsic parameters to obtain camera distortion parameters.

[0057] Sampling rays are a series of rays emitted along a specific direction from the camera position that captures the real image, representing the correspondence between the position points in the image and the positions in the actual scene or the virtual scene. When the image is distorted, the light path is distorted, and the original sampling rays cannot accurately describe the correspondence between the position points in the image and the positions in the actual scene or the virtual scene. Therefore, distortion is added to the sampling rays to ensure that the position points in the distorted image can accurately correspond to the positions in the actual or virtual scene.

[0058] The position point on the real camera image can be the relative position of the pixel in the image, or a certain area or point in the image coordinate system. The position in the scene represents the spatial position in the actual scene or virtual scene.

[0059] Integrating the camera's distortion parameters into the calculation of the neural radiation field network can more accurately simulate the real camera image captured by the camera, which is conducive to improving the quality and authenticity of the neural radiation field network rendered images. The distorted sampling rays allow the neural radiation field network to directly use distorted images for training and obtain accurate distorted simulated images when rendering. At the same time, it also avoids the loss of image information caused by cropping distorted images, improving the training efficiency of the neural radiation field network.

[0060] Specifically, see Figure 3 , Figure 3 Yes Figure 1 The flowchart of step 20 of the automatic driving simulation method shown in FIG. 1 further includes:

[0061] Step 21: Calculate the reference position of the sampling point on the sampling ray using the camera distortion parameters.

[0062] After determining the direction and starting point of the sampling ray, a series of sampling points are generated on the ray according to the sampling step size and the number of sampling times. Due to the existence of camera distortion, the actual positions of these sampling points on the image will change, and the distorted positions of the sampling points need to be calculated using the camera distortion parameters as the reference positions of the sampling points.

[0063] The reference position of the sampling point is used to adjust the direction of the sampling ray so that the position point in the real image of the camera can correspond to the position in the actual scene through the sampling ray.

[0064] Step 22: Adjust the direction and position of the sampling ray based on the reference position of the sampling point, and determine the corresponding relationship between the position point on the real image of the camera and the position in the scene under the corresponding camera posture.

[0065] The camera distortion model is used to convert the undistorted points into the reference positions of the actual sampling points after distortion. Based on the offset of the sampling points relative to the reference positions, the small angle that needs to be adjusted for the direction of the sampling ray is estimated, and the direction and position of the sampling ray are adjusted to determine the correspondence between the position points on the real image of the camera and the positions in the scene under the corresponding camera posture.

[0066] Step 30: Perform image rendering based on the correspondence between the position points on the real camera image and the positions in the scene to obtain a simulated image under the corresponding camera posture.

[0067] The position, posture and lens parameters of the simulated camera are matched with the posture captured in the real image through the neural radiation field network to ensure the consistency of the perspective between the simulated image and the real image. The image information is mapped on the constructed three-dimensional scene using the correspondence between the known position points on the real camera image and the corresponding positions in the scene, and the simulated image is rendered to increase the realism of the simulated image.

[0068] The simulated image is a virtual scene image obtained by the neural radiation field network through the light propagation, lighting effects and image optimization in the virtual scene. Specifically, the neural radiation field network simulates the propagation and interaction of light in the scene, calculates the color value of each pixel, and thus produces realistic lighting effects and shadows; considers the influence of all light sources in the scene on objects, as well as the reflection and refraction between objects, so that the simulated image obtains a more realistic lighting effect; and performs post-processing on the rendering results, such as color correction, anti-aliasing, depth of field effect, etc., to improve the quality of the simulated image.

[0069] Image rendering is performed through the correspondence between the position points on the camera's real image and the positions in the scene, so that the neural radiation field network can directly use distorted images for training and rendering, and improve the rendering efficiency and the quality of the simulated image.

[0070] Step 40: Update the parameters of the neural radiance field network based on the difference between the simulated image and the real camera image.

[0071] Compare the rendered simulated image with the real image actually captured by the camera, see Figure 4 , Figure 4 The figure shows the comparison between the real camera image (a) and the simulated image (b). The difference between the simulated image (b) and the real camera image (a) is obtained, and the parameters of the neural radiation field network are updated, so that the simulated image (b) rendered by the neural radiation field network is closer to the real camera image (a) actually taken, thereby improving the authenticity of the simulated image (b).

[0072] Specifically, see Figure 5 , Figure 5 Yes Figure 1 The flowchart of step 40 of the automatic driving simulation method shown in FIG. 1 is an embodiment of the present invention, wherein step 40 further includes:

[0073] Step 41: Calculate the loss function based on the difference between the simulated image and the real camera image.

[0074] Compare the simulated image with the real camera image, obtain the difference between the simulated image and the real camera image, and quantify the loss function.

[0075] The loss function is used to quantify the difference between the rendered image and the real image. The change in the loss function can reflect the degree of training of the current neural radiation field network. When the loss function converges or is less than the preset threshold, the training of the neural radiation field network is completed, and the rendered simulated image has a high degree of similarity with the real camera image and meets the training standards.

[0076] Specifically, the loss function includes: at least one of: color loss, structural similarity loss, regularization loss, depth loss, perceptual loss or mask loss.

[0077] Among the loss functions, color loss is the most basic loss function, which is used to measure the difference in color between the simulated image and the real camera image. The commonly used color loss is the mean square error, which is to calculate the average of the square of the difference between the color value of each pixel in the simulated image and the color value of the corresponding pixel in the real camera image. In addition to color loss, structural similarity loss can also be considered to evaluate the similarity between images. Structural similarity loss not only considers the difference between pixel values, but also considers the structural information of the image, such as contrast, brightness, etc., so that the quality of the simulated image can be more comprehensively evaluated. In order to prevent overfitting, a regularization term can be added to the loss function. Regularization loss can be a penalty term for network weights to constrain the complexity of network parameters. If the real camera image also contains depth information, depth loss can be used as part of the training network. Depth loss is used to measure the difference between the estimated depth in the rendered image and the real depth. When there are occluded or unobserved areas in the input image, mask loss can be used to ignore the impact of these areas on the training process. Mask loss is usually used in combination with color loss, and loss is calculated only for the valid areas indicated by the mask. Perceptual loss is a loss function based on high-level feature representation, which uses a pre-trained neural radiance field network to extract high-level features of images and calculate the differences between these features. Perceptual loss can capture the semantic and structural similarities between images, thereby generating more visually realistic rendered images.

[0078] Step 42: Using the gradient associated with the loss function, update the parameters of the neural radiation field network through gradient backpropagation.

[0079] During the training process, the neural radiance field network needs to optimize its parameters so that it can more accurately predict the scene view under given camera parameters. Using the gradient associated with the loss function, the parameters of the neural radiance field network are updated through the gradient backpropagation algorithm, so that the loss function gradually tends to the minimum through the update of the neural radiance field network parameters, that is, the similarity between the simulated image and the real camera image is the highest.

[0080] The gradient associated with the loss function refers to a vector or matrix consisting of the partial derivatives of the loss function with respect to the parameters of the neural radiation field network in machine learning or deep learning. This gradient vector or matrix indicates how to adjust the parameters of the model to minimize the loss function, that is, to improve the rendering performance of the model.

[0081] Gradient backpropagation is an algorithm for calculating the gradient of the loss function with respect to the network parameters, starting from the output layer of the network, propagating the gradient layer by layer to the input layer, and calculating the gradient of each parameter according to the chain rule. These gradients are then used to update the parameters of the network to minimize the loss function.

[0082] The gradient information of the loss function is used to guide the optimization direction of the parameters, thereby continuously improving the performance of the network and the realism of the simulation image.

[0083] Step 43: In response to the convergence of the loss function, the training of the neural radiation field network is terminated.

[0084] Under the guidance of the gradient associated with the loss function, after updating the parameters of the neural radiation field network through the gradient backpropagation algorithm multiple times, the loss function converges, the update of the neural radiation field network parameters is ended, and the trained neural radiation field network is saved for downstream calls for simulation testing.

[0085] The trained neural radiance field network can render highly realistic simulation images based on the input camera pose for simulation testing.

[0086] Optionally, see Figure 6 , Figure 6 Yes Figure 1 The flowchart of another embodiment of the autonomous driving simulation method shown in the figure further includes: after multiple sets of camera poses and real camera images are used as training sets to input the neural radiation field network, the method further includes:

[0087] Step 11: Add a mask to the real camera image to segment the body area and non-body area in the real camera image.

[0088] Since the camera installed on the car will capture part of the car body, and the car body is a dynamic object that moves with the car, it will affect the rendering of other static parts in the scene. Therefore, image processing technology is used to detect the car body area and non-car body area in the image, and then a mask is added to the real camera image, in which the car body area is marked as 1 and the non-car body area is marked as 0, so as to separate the car body area and non-car body area in the real camera image, which helps to improve the clarity of the rendered image of the two parts, thereby improving the target detection accuracy of the autonomous driving algorithm using the real camera image.

[0089] After step 11, this embodiment further performs a process as step 20.

[0090] Specifically, image rendering is performed based on the correspondence between the position points on the real camera image and the positions in the scene to obtain a simulated image under the corresponding camera posture, including:

[0091] Step 31: Based on the correspondence between the position points of the vehicle body area and the non-vehicle area and the positions in the scene, the images of the vehicle body area and the non-vehicle area are rendered respectively to obtain a simulated image of the vehicle body area and a simulated image of the non-vehicle area under the corresponding camera posture.

[0092] The position, posture and lens parameters of the simulated camera are matched with the posture captured in the real image through the neural radiation field network to ensure the consistency of the perspective between the simulated image and the real image. The image information is mapped on the constructed three-dimensional scene using the correspondence between the known position points of the body area and non-body area on the real camera image and the corresponding positions of the body area and non-body area in the scene, and the simulated image is rendered separately to increase the realism of the simulated image of the body area and non-body area.

[0093] After step 31 , this embodiment further executes processes such as steps 40 and 50 .

[0094] Specifically, the neural radiation field network is rendered based on the camera pose to obtain a simulated image corresponding to the camera pose, including:

[0095] Step 61: The neural radiance field network renders the vehicle body area and non-vehicle body area of ​​the real camera image onto the simulated image based on the camera pose.

[0096] The neural radiation field network determines the observation angle of the vehicle in the virtual scene based on the vehicle position and orientation defined by the camera pose, and synthesizes the pixel color combination of the body area and non-body area at this perspective in combination with the distortion of the sampling ray, and obtains the body area and non-body area in the distorted simulation image corresponding to the camera pose. Through volume rendering, the color and opacity of each sampling point on the sampling ray are accumulated along the direction of the light. The accumulated color value is the color of the pixel point in the body area and non-body area in the final simulation image, and a complete simulation image is obtained.

[0097] After step 61 , this embodiment further executes a process as step 70 .

[0098] See also Figure 7 , Figure 7 301 is a schematic diagram of a storage medium according to an embodiment of the present application. The storage medium 300 stores program data 301. When the program data 301 is executed by a processor, the following is achieved: Figure 1 The steps of the described autonomous driving simulation method.

[0099] The program data 301 is stored in a storage medium 300, and includes a number of instructions for enabling a network device (such as a router, a personal computer, a server, or other network device) or a processor to execute all or part of the steps of the method described in each embodiment of the present application.

[0100] Optionally, the storage medium 300 may be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, or other medium that can store program data.

[0101] See also Figure 8 , Figure 8 It is a structural schematic diagram of an embodiment of a computer device provided by the present application.

[0102] The computer device 400 cooperates with each sensor module and control module in the simulated vehicle to realize visual perception, sensor data processing and image display process. The computer device 400 includes a processor 420 and a memory 410 connected to each other. The memory 410 stores a computer program. When the processor 420 executes the computer program, the steps of the above-mentioned automatic driving simulation method are realized.

[0103] Different from the prior art, the present application provides an autonomous driving simulation method, storage medium and computer device. By inputting the camera posture into the neural radiation field network for image rendering, a simulation image with high realism is obtained for the training of the autonomous driving algorithm, thereby reducing the training cost of the autonomous driving algorithm and improving the realism of the simulation image. The neural radiation field network is trained only through the driving scene image data of the actual driving vehicle, thereby reducing the cost of obtaining network training data.

[0104] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium embodiment and the computer device embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0105] The present application can be used in many general or special vehicle-mounted computing system environments or configurations, such as personal computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputers, and distributed computing environments including any of the above systems or devices.

[0106] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not performed.

[0107] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0108] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0109] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An automatic driving simulation method, characterized in that: include: Input the camera pose into the trained neural radiance field network; The neural radiation field network performs rendering based on the camera pose to obtain a simulated image corresponding to the camera pose; The simulation image is transmitted to an autonomous driving algorithm, and the autonomous driving algorithm performs a simulation test based on the simulation image.

2. The automatic driving simulation method according to claim 1, characterized in that: Before inputting the camera pose into the trained neural radiation field network, the method further includes: Using multiple sets of camera poses and real camera images as training sets to input the neural radiation field network; Calculating the distortion parameters of the sampled rays in the neural radiation field network through the camera distortion parameters to determine the corresponding relationship between the position points on the real image of the camera and the positions in the scene under the corresponding camera posture; Performing image rendering based on the correspondence between the position points on the real camera image and the positions in the scene to obtain a simulated image at the corresponding camera position; Parameters of the neural radiance field network are updated based on the difference between the simulated image and the camera real image.

3. The automatic driving simulation method according to claim 2, characterized in that: The updating of the parameters of the neural radiation field network based on the difference between the simulated image and the real camera image comprises: Calculate a loss function based on the difference between the simulated image and the real camera image; Using the gradient associated with the loss function, updating the parameters of the neural radiation field network by gradient backpropagation; In response to the convergence of the loss function, the training of the neural radiation field network is terminated.

4. The automatic driving simulation method according to claim 3, characterized in that: The loss function includes: at least one of color loss, structural similarity loss, regularization loss, depth loss, perceptual loss or mask loss.

5. The automatic driving simulation method according to claim 2, characterized in that: After the neural radiation field network is input with multiple sets of camera poses and real camera images as training sets, the method further includes: Adding a mask to the real camera image to segment the vehicle body area and the non-vehicle body area in the real camera image; The image rendering is performed based on the correspondence between the position points on the real image of the camera and the positions in the scene to obtain a simulated image under the corresponding camera posture, including: Based on the correspondence between the position points of the body area and the non-body area and the positions in the scene, the images of the body area and the non-body area are rendered respectively to obtain the simulated image of the body area and the simulated image of the non-body area under the corresponding camera posture.

6. The automatic driving simulation method according to claim 5, characterized in that: The neural radiation field network performs rendering based on the camera pose to obtain a simulated image corresponding to the camera pose, including: The neural radiance field network renders the vehicle body area and the non-vehicle body area of ​​the real camera image onto the simulated image respectively based on the camera pose.

7. The automatic driving simulation method according to claim 2, characterized in that: The method of calculating the distortion parameters of the sampled rays in the neural radiation field network by using the camera distortion parameters to determine the corresponding relationship between the position point on the real image of the camera and the position in the scene under the corresponding camera posture includes: Calculating the reference position of the sampling point on the sampling ray by using the camera distortion parameter; The direction and position of the sampling ray are adjusted based on the reference position of the sampling point, and the corresponding relationship between the position point on the real image of the camera and the position in the scene under the corresponding camera posture is determined.

8. The automatic driving simulation method according to claim 2, characterized in that: The camera pose is derived from the real camera image.

9. A storage medium having program data stored thereon, characterized in that: When the program data is executed by the processor, the steps of the automatic driving simulation method as described in any one of claims 1 to 8 are implemented.

10. A computer device, characterized in that: Includes interconnected processors and storage The memory stores a computer program, and when the processor executes the computer program, Implement the steps of the autonomous driving simulation method as described in any one of claims 1-8.

Images