Virtual scene construction method based on point cloud data, medium and equipment

Abstract

The invention discloses a virtual scene construction method based on point cloud data, a medium and equipment. The method comprises the following steps: constructing an initial virtual scene based on original point cloud data; constructing a depth map corresponding to the point cloud data in the initial virtual scene, and determining a shielding relationship among the objects in the initial virtual scene based on the depth map; performing physics-based rendering on an object in the initial virtual scene, and processing the rendered initial virtual scene by adopting global illumination; and integrating the occlusion relationship into the rendered initial virtual scene to generate a new virtual scene. Through the above mode, the method can improve the trueness of building the virtual scene through the radar point cloud data.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving, and in particular to a method, medium and device for constructing a virtual scene based on point cloud data. Background Art

[0002] With the rapid development of autonomous driving technology, the demand for highly accurate and diverse environmental perception data is growing. Traditional data collection mainly relies on actual road testing. However, this method is not only costly, but also limited by geographical location, weather conditions, and difficult-to-reproduce traffic scenarios. It is difficult to meet the requirements of autonomous driving algorithms for large data volumes and full scene coverage. Therefore, efficiently generating high-quality simulation data has become a key way to solve this problem.

[0003] The existing methods of creating virtual environments and objects through 3D rendering are mainly aimed at known models or designed scenes. 3D rendering is often based on predefined models and lighting conditions, and it is difficult to adaptively solve the occlusion logic between new objects and original scene elements, resulting in limited realism and accuracy of the generated content and unrealistic virtual scenes. Summary of the invention

[0004] The present application mainly provides a method, medium and device for constructing a virtual scene based on point cloud data to solve the problem of low realism of the virtual scene.

[0005] To solve the above technical problems, a technical solution adopted in the present application is: to provide a method for constructing a virtual scene based on point cloud data, including: constructing an initial virtual scene based on the original point cloud data; constructing a depth map corresponding to the point cloud data in the initial virtual scene, and determining the occlusion relationship between objects in the initial virtual scene based on the depth map; performing physically-based rendering on the objects in the initial virtual scene, and using global illumination to process the rendered initial virtual scene; integrating the occlusion relationship into the rendered initial virtual scene to generate a new virtual scene.

[0006] By directly constructing a depth map from the original point cloud data, the occlusion relationship between objects in the point cloud data is enhanced, ensuring that the foreground objects in the picture can correctly occlude the background objects, thereby enhancing the realism of the picture; by calculating the lighting effect on the surface of each object, the light and shadow relationship in the virtual scene is made more natural and realistic.

[0007] In some embodiments, before constructing the depth map corresponding to the point cloud data in the initial virtual scene, it also includes: performing physical collision simulation on the objects in the initial virtual scene through a physical engine, and fusing the additional point cloud data generated by the collision with the original point cloud data; updating the initial virtual scene based on the fused point cloud data.

[0008] Through physical collision simulation, the dynamic behavior of objects in virtual space can be accurately simulated to improve the realism of virtual scenes. The richness and diversity of the acquired point cloud data can be increased through collision point cloud data, and the fused point cloud data can be used to generate higher-quality virtual scenes, maintaining the compactness of the point cloud data by updating the original point cloud data.

[0009] In some embodiments, constructing a depth map corresponding to the point cloud data in the initial virtual scene includes: constructing a virtual camera model based on radar sensor parameters and virtual camera configuration parameters related to the original point cloud data; and generating the depth map based on the virtual camera model.

[0010] By constructing a virtual camera model and directly generating a depth map using radar point cloud data, the accurate mapping of depth information is ensured, which is conducive to better handling the occlusion relationship of objects in space.

[0011] In some embodiments, after generating the depth map based on the virtual camera model, the method further includes: filling the hole areas in the depth map by image processing technology; and processing the depth map by an edge-preserving smoothing algorithm.

[0012] The integrity of the depth map is maintained by filling the hole areas of the depth map. The noise and staircase effect in the depth map are reduced by the edge-preserving smoothing algorithm, the edge clarity is maintained, and the overall quality of the depth map is improved.

[0013] In some embodiments, generating the depth map based on the virtual camera model includes: projecting the point cloud data in the original coordinate system to a two-dimensional image plane through the virtual camera model; adding distortion to the point cloud data in the two-dimensional image plane based on the virtual camera configuration parameters to obtain projection points corresponding to the point cloud data in the two-dimensional image plane; fusing the projection points with the point cloud data on the two-dimensional image plane to generate a depth map corresponding to the point cloud data.

[0014] The camera model is used to add distortion to the point cloud projection on the two-dimensional image plane, taking into account the deformation during perspective projection, ensuring the accurate mapping of the depth information on the image, and intuitively showing the distance from each point in the scene to the camera, making the depth information easier to understand and analyze.

[0015] In some embodiments, the initial virtual scene rendered using global illumination processing includes: simulating the lighting effect of light sources on objects in the initial virtual scene through a global illumination algorithm; and dynamically updating the lighting effects of objects in the initial virtual scene in response to dynamic changes in the initial virtual scene.

[0016] Physically based rendering technology is used to simulate direct lighting, indirect lighting, and ambient occlusion in complex lighting environments to enhance the realism of virtual scenes.

[0017] In some embodiments, simulating the lighting effect of the light source on the objects in the initial virtual scene through the global illumination algorithm includes: calculating the lighting information of the objects in the initial virtual scene and storing the lighting information as a light map; applying the light map on the objects to simulate the lighting effect of the light source on the objects in the initial virtual scene.

[0018] By calling pre-stored lightmaps, the runtime calculation burden is reduced. Dynamic lighting handles the lighting effects caused by dynamic objects and environmental changes to ensure the consistency and real-time performance of lighting.

[0019] In some embodiments, constructing an initial virtual scene based on the original point cloud data includes: preprocessing the original point cloud data and aligning the preprocessed point cloud data to a target coordinate frame to form an initial virtual scene; evaluating the distribution density of the point cloud data in the initial virtual scene to divide the point cloud data into sparse areas and dense areas; and filling the sparse areas of the point cloud data in the initial virtual scene by interpolation.

[0020] By preprocessing the acquired radar point cloud data, noise points and outliers in the point cloud data are removed to ensure the purity and consistency of the data; the density and uniformity of the point cloud data are increased to make the point cloud more continuous, laying the foundation for generating high-resolution depth maps.

[0021] In order to solve the above technical problems, another technical solution adopted by the present application is: providing a storage medium on which program data is stored, and when the program data is executed by a processor, the steps of the virtual scene construction method as described above are implemented.

[0022] The beneficial effects of the storage medium refer to the beneficial effects of the virtual scene construction method described above, which will not be repeated here.

[0023] The present application also provides a computer device, including a processor and a memory connected to each other, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the virtual scene construction method as described above are implemented.

[0024] The beneficial effects of the computer device refer to the beneficial effects of the above-mentioned virtual scene construction method, which will not be repeated here.

[0025] The beneficial effects of the present application are as follows: Different from the prior art, the present application discloses a method, medium and device for constructing a virtual scene based on point cloud data. The initial virtual scene is constructed through the original point cloud data, and a depth map corresponding to the point cloud data in the initial virtual scene is directly constructed to determine the occlusion relationship between the objects in the original point cloud model, thereby enhancing the processing accuracy of the occlusion relationship between objects in the point cloud data; after the objects in the initial virtual scene are rendered based on physics, the global illumination dynamic processing is used to render the initial virtual scene, so that the light and shadow relationship in the virtual scene is dynamically variable, thereby enhancing the realism of the virtual scene. Integrating the occlusion relationship into the rendered initial virtual scene solves the problem of unnatural occlusion and unrealistic effects between objects in the virtual scene, significantly improves the realism of the virtual scene simulated by point cloud data, and provides more abundant and diverse training resources for the autonomous driving algorithm, which is conducive to improving the recognition and response capabilities of the autonomous driving algorithm in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0027] Figure 1 It is a flowchart of an embodiment of a method for constructing a virtual scene based on point cloud data provided by the present application;

[0028] Figure 2 Yes Figure 1 The flowchart of step 10 of the virtual scene construction method shown in FIG.

[0029] Figure 3 Yes Figure 1 The flowchart of step 40 of the virtual scene construction method shown in FIG.

[0030] Figure 4 Yes Figure 3 The flowchart of step 42 of the virtual scene construction method shown in FIG.

[0031] Figure 5 Yes Figure 1 The flowchart of step 50 of the virtual scene construction method shown in FIG.

[0032] Figure 6 Yes Figure 5 The flowchart of step 51 of the virtual scene construction method shown in FIG.

[0033] Figure 7 It is a structural schematic diagram of an embodiment of a storage medium provided by the present application;

[0034] Figure 8 It is a structural diagram of an embodiment of a computer device provided by the present application. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] The terms "first", "second", "third" in the embodiments of the present application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second", "third" can expressly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0037] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0038] See also Figure 1 , Figure 1 : is a flow chart of an embodiment of a method for constructing a virtual scene based on point cloud data provided by the present application, and the method for constructing a virtual scene includes:

[0039] Step 10: Build the initial virtual scene based on the original point cloud data.

[0040] Import the radar point cloud data into the 3D modeling system, build the initial virtual scene in the modeling system based on the 3D model constructed based on the point cloud data.

[0041] Point cloud is a collection of three-dimensional spatial data collected by the LiDAR system. These data collections are composed of a large number of discrete points, each of which represents a spatial position. LiDAR point cloud data is calculated by the LiDAR by emitting a pulsed laser beam and measuring the time it takes for the laser beam to return. The point cloud data of the target object at different angles is obtained through a rotating or scanning mechanism.

[0042] The initial virtual scene is a three-dimensional environment constructed by multiple point clouds that can be displayed, analyzed and interacted in a computer. It contains a blank object frame constructed by point cloud coordinates. Constructing the initial virtual scene can determine the position and shape of each object in the virtual space, and on this basis, add materials, lighting effects, etc. to the virtual objects to make the virtual scene more realistic.

[0043] Specifically, point cloud data can be acquired through vehicle-mounted lidar, drone-mounted lidar, ground laser scanners and other equipment.

[0044] The initial virtual scene is constructed through the original point cloud data, laying the foundation for the construction of high-realism and high-quality point cloud scenes.

[0045] Further, see Figure 2 , Figure 2 Yes Figure 1 The flowchart of step 10 of the embodiment of the virtual scene construction method shown in FIG. 1 further includes: constructing the initial virtual scene based on the original point cloud data;

[0046] Step 11: Preprocess the original point cloud data and align the preprocessed point cloud data to the target coordinate frame to form an initial virtual scene.

[0047] The original point cloud acquired by the lidar system is preprocessed by denoising and filtering, and the preprocessed point cloud data is aligned to the same coordinate system. After rotation or translation, it is aligned with the preset target coordinate frame to form an initial virtual scene.

[0048] Specifically, by applying statistical or geometric denoising algorithms, such as neighborhood-based average filtering and random sampling consensus (RANSAC) algorithms, noise points and outliers in the point cloud can be removed to ensure the purity and coherence of the point cloud data. Neighborhood-based average filtering calculates the average value of each pixel in the image and its neighborhood, and replaces the original pixel value with this average value, thereby achieving the purpose of smoothing the image and reducing noise. The random sampling consensus algorithm estimates the model by repeatedly selecting a subset of the data and using the model to test all data points to identify which points fit the model.

[0049] Alignment is to convert point cloud data from different sources and coordinate systems into a unified target coordinate frame by defining a common origin and coordinate axis direction, and ensure that the point cloud coordinates of each corner are aligned with the corner coordinates of the target coordinate frame to eliminate errors and deviations caused by inconsistent coordinate systems, which helps to improve the accuracy and efficiency of subsequent processing and analysis.

[0050] The rotation operation of the point cloud is implemented using a rotation matrix. The rotation matrix describes the rotation transformation around the X-axis, Y-axis, and Z-axis. For each point in the point cloud, its coordinates are multiplied by the rotation matrix to obtain the rotated coordinates. The translation operation of the point cloud is implemented using a translation vector. The translation vector is used to represent the translation direction and distance. Each component of the translation vector corresponds to the translation amount on the X-axis, Y-axis, and Z-axis. For each point in the point cloud, its coordinates are added to the translation vector to obtain the translated coordinates.

[0051] The accuracy and reliability of point cloud data can be significantly improved by removing noise points. At the same time, by simplifying point cloud data, redundant information can be reduced and the speed and efficiency of subsequent processing can be improved. Through operations such as preprocessing and coordinate alignment, it can be ensured that the point cloud data in the virtual scene has higher accuracy and richer detail information. It helps to enhance the realism and immersion of the virtual scene.

[0052] Step 12: Evaluate the distribution density of the point cloud data in the initial virtual scene to divide the sparse areas and dense areas in the point cloud data.

[0053] Point cloud data contains three-dimensional coordinates (X, Y, Z). Points are usually projected onto a two-dimensional plane containing XY, and the plane density is used to evaluate the distribution density of point cloud data. The greater the distribution density of point cloud data, the more points there are on the two-dimensional plane within the same area, and vice versa. Based on the overall distribution of point cloud data, one or more density thresholds are set to divide the sparse and dense areas in the point cloud data, providing a reference area for the balance of point cloud data.

[0054] Step 13: Fill the sparse areas of point cloud data in the initial virtual scene through interpolation.

[0055] In sparse areas, methods such as K-Nearest Neighbors (KNN) interpolation, Inverse Distance Weight (IDW) interpolation or Radial Basis Function (RBF) interpolation are used to estimate and insert new points based on the information of adjacent points to fill the gaps in the point cloud data in the initial virtual scene, increase the density and uniformity of the point cloud, make the point cloud more continuous, and lay the foundation for generating high-resolution depth maps.

[0056] Interpolation is to construct a curve or surface passing through the known data points through the information of the known data points, so as to make predictions or estimates between the known data points. Specifically, K nearest neighbor interpolation is to find the K known sample points closest to it in the feature space, and then predict the value of the unknown point based on the values ​​of these K known sample points. Inverse distance weighted interpolation is to assign weights according to the distance between the known point and the unknown point. The closer the distance, the greater the weight, and vice versa. This method assumes that each measured point has a local influence, and this influence decreases with the increase of distance. Radial basis function interpolation is to define a radial basis function around each data point, which depends only on the distance between the data point and the point to be interpolated. By adjusting the weight coefficients of these basis functions, the interpolation function can best approximate the known data points.

[0057] Step 20: Perform physical collision simulation on the objects in the initial virtual scene through the physical engine, and fuse the additional point cloud data generated by the collision with the original point cloud data.

[0058] Create or import the required object models in the virtual scene, including static and dynamic objects. Then set the corresponding physical properties for each object in the physical engine, such as mass, density, friction coefficient, elastic coefficient, etc., and use high-performance physical computing engines such as Physx to perform physical collision simulation on the objects in the initial virtual scene, simulate the precise collision detection between objects in the virtual scene, such as the physical effects after the collision, such as speed, direction, rotation, etc., and the collision point, collision direction, collision force and other responses of the objects when the collision occurs, and record the collision point coordinates of each collision to form additional point cloud coordinate data in the point cloud space. The additional point cloud data is merged with the original point cloud data to obtain new point cloud data, increasing the richness and diversity of the point cloud data.

[0059] Physics engines are used to simulate the movement and interaction of objects in the virtual world and calculate the behavior of these objects in various physical environments. Physics engines give objects real physical properties such as mass, momentum, friction, and elasticity, making the behavior of these objects in the virtual world more realistic and in line with the laws of physics.

[0060] Specifically, Physx is a high-performance physical computing engine that is used to make the movement of objects in virtual scenes more consistent with the physical laws of the real world, thereby enhancing the realism of the virtual scenes.

[0061] In the process of point cloud fusion, since the original point cloud data and the additional point cloud data may be in different coordinate systems or positions, they need to be aligned to the same coordinate system first. Specifically, the alignment process needs to use an appropriate algorithm to estimate the transformation matrix between the two point clouds, and transform the additional point cloud into the coordinate system of the original point cloud data, and merge the transformed additional point cloud data with the original point cloud data into a single point cloud data set as the inherent point cloud data in the virtual scene, so that the point cloud data set generated this time can be distinguished from the additional point cloud data collected during the next collision. The fusion of the additional point cloud data and the original point cloud data is completed by transforming the additional point cloud data and merging it into the point cloud data set of the original point cloud data.

[0062] Optionally, after the additional point cloud data is fused with the original point cloud data, the fused new point cloud data is deduplicated and density optimized. Deduplication is to find and delete duplicate points in the fused point cloud data. Density optimization is to adjust the density of the point cloud to improve its uniformity and reduce noise. Specifically, algorithms such as voxel grid downsampling can be used to reduce the number of points in the point cloud while maintaining its shape and features. Other filtering algorithms such as moving least squares, Gaussian filtering, etc. can also be applied to smooth the point cloud and reduce noise. In this embodiment, there is no specific limitation on the methods of deduplication and density optimization.

[0063] By fusing the original point cloud with the additional point cloud data generated by the collision, high-quality new point cloud data is generated through deduplication and density optimization algorithms, maintaining the compactness and efficiency of the point cloud data.

[0064] Step 30: Update the initial virtual scene based on the fused point cloud data.

[0065] Based on the fused point cloud data, the 3D model, physical engine settings and rendering effects in the initial virtual scene are updated respectively, making the point cloud-based virtual scene more realistic, dynamic and interactive.

[0066] Step 40: construct a depth map corresponding to the point cloud data in the initial virtual scene, and determine the occlusion relationship between the objects in the initial virtual scene based on the depth map.

[0067] Select a fixed viewing angle as the viewing angle for observing the initial virtual scene. Based on the viewing angle, calculate the projection of the point cloud data in the initial virtual scene on the plane corresponding to the viewing angle. Project the three-dimensional point cloud data onto the two-dimensional plane. Determine the pixel position of each point in the plane based on the depth value of each point in the three-dimensional space. Map the projected depth value to the pixel at each corresponding position to generate a depth map. Based on the depth map, determine the occlusion relationship by comparing the depth values ​​of different pixels. If the depth value of a pixel is less than the depth value of its adjacent pixel, it means that the object represented by the pixel is visually located in front of other objects, that is, it occludes other objects.

[0068] The depth map is a grayscale image or a numerical image, in which the grayscale value or numerical size of a pixel represents the distance from the location to the observation point. The rendering order of each object in the virtual scene is determined according to the size of the depth value. Objects with smaller depth values ​​are rendered first, and objects with larger depth values ​​are rendered later. In this way, the correct occlusion relationship will be formed in the final displayed image.

[0069] For example, the initial virtual scene contains multiple objects: a table, a chair, and a character model. Based on the depth value of each pixel in the depth image, we can determine the relative position of each point in the scene in three-dimensional space. Points with smaller depth values ​​indicate that the point is closer to the camera, so they should be drawn first during rendering in order to correctly occlude points with larger depth values. If the depth value of a part of the table is smaller than the depth value of the character model, the table should block the view of the character model under that part during rendering.

[0070] By directly constructing a depth map from the original point cloud data, the occlusion relationship between objects in the point cloud data is enhanced, ensuring that the foreground objects in the picture can correctly occlude the background objects, enhancing the realism of the picture. The depth map can be used to perform depth testing on the rendered virtual scene, so as to correctly remove the occluded parts of the image and make the overall virtual scene more realistic.

[0071] Further, see Figure 3 , Figure 3 Yes Figure 1 The flowchart of step 40 of the virtual scene construction method shown in the figure is an embodiment of the invention, wherein constructing a depth map corresponding to the point cloud data in the initial virtual scene includes:

[0072] Step 41: Construct a virtual camera model based on radar sensor parameters and virtual camera configuration parameters related to the original point cloud data.

[0073] The parameters of the radar sensor that forms the original point cloud data and the configuration parameters of the virtual camera currently used as the observation point are obtained, the point cloud data generated by the radar sensor is fused with the imaging data of the virtual camera model, the virtual camera model is calibrated according to the configuration parameters of the virtual camera model, the camera intrinsic parameters and camera extrinsic parameters of the virtual camera model are determined, and the virtual camera model is constructed.

[0074] In this embodiment, the radar sensor is mainly a laser radar sensor, which measures three-dimensional spatial information such as the distance, shape, speed, and surface features of a target object based on the emission and reception of lasers to form point cloud data.

[0075] The virtual camera model is a parameter model that simulates the camera's viewing angle in a virtual scene. Through the virtual camera model, the image of the scene under the viewing angle can be observed from different angles in the virtual scene.

[0076] The virtual camera configuration parameters include the camera intrinsic parameters and camera extrinsic parameters of the virtual camera model. The camera intrinsic parameter calibration mainly involves determining the camera's focal length, principal point coordinates, distortion coefficient and other parameters. These parameters are inherent properties of the camera imaging process and determine the mapping relationship from points in the three-dimensional world to the two-dimensional image plane. The camera extrinsic parameter calibration is used to determine the transformation relationship between the radar coordinate system and the camera coordinate system, including the rotation matrix and the translation vector.

[0077] By constructing a virtual camera model and directly generating a depth map using radar point cloud data, the accurate mapping of depth information is ensured, which is conducive to better handling the occlusion relationship of objects in space.

[0078] Step 42: Generate a depth map based on the virtual camera model.

[0079] Using a virtual camera model, the point cloud data in the three-dimensional space is directly projected onto the two-dimensional image plane to generate a depth map, in which the position of each point is determined by its depth value in the three-dimensional space.

[0080] Further, see Figure 4 , Figure 4 Yes Figure 3 The flowchart of step 42 of the embodiment of the virtual scene construction method shown in the figure, generating a depth map based on the virtual camera model includes:

[0081] Step 421: Project the point cloud data in the original coordinate system onto a two-dimensional image plane through a virtual camera model.

[0082] Each point in the point cloud data is transformed from the original coordinate system of the point cloud in space to the camera coordinate system by multiplying the rotation matrix of the camera and adding the translation vector. The point cloud coordinates in the camera coordinate system after the transformation are defined with the camera as the center. Using the camera intrinsic parameters of the virtual camera model, the three-dimensional point (x, y, z) in the camera coordinate system is projected onto the two-dimensional image plane (x, y) coordinates, providing an image basis for the subsequent generation of the depth map.

[0083] Specifically, the original coordinate system is a coordinate system constructed with the relevant radar as the center when the point cloud data is generated.

[0084] Step 422: Add distortion to the point cloud data in the two-dimensional image plane based on the virtual camera configuration parameters to obtain projection points corresponding to the point cloud data in the two-dimensional image plane.

[0085] Based on the internal parameters of the virtual camera model, a distortion model is applied to the point cloud data in the two-dimensional image plane to increase the distortion, so as to obtain the distorted projection points corresponding to the point cloud data in the two-dimensional image plane.

[0086] The projection point is the position of the point cloud data combined with the camera's intrinsic distortion parameters on the two-dimensional image plane. According to the camera's distortion model, the distortion offset of the projection point is calculated, and then the distortion offset is applied to the projection point to obtain the distorted projection point. Through the projection point, the point cloud can be projected onto the two-dimensional image plane in a distorted form to form a distorted image corresponding to the observation point.

[0087] The camera intrinsic parameters of the virtual camera model are used to make the projection position of the point cloud data on the two-dimensional plane more accurate, so as to ensure the accurate mapping of the depth information in the depth map.

[0088] Step 423: Fuse the projection points with the point cloud data on the two-dimensional image plane to generate a depth map corresponding to the point cloud data.

[0089] Point cloud data is a collection of discrete points in three-dimensional space, each point contains X, Y, and Z coordinates. The two-dimensional image plane is a plane composed of pixels, each pixel has x and y coordinates, which are used to represent the position of the point cloud data mapped to the two-dimensional image. Through methods such as coordinate matching or data fusion algorithms, the distorted projection points are fused with the point cloud data on the two-dimensional image plane, and a depth value is assigned to each pixel at the correct position, that is, the distance between the object surface corresponding to the pixel and the camera. The depth value is mapped to grayscale or color value to generate a depth map corresponding to the point cloud data.

[0090] By fusing the projection points with the point cloud data on the two-dimensional image plane into a point cloud projection on the two-dimensional image plane to increase distortion, the deformation during perspective projection is taken into account, ensuring the accurate mapping of the depth information on the image.

[0091] Optionally, after generating the depth map based on the virtual camera model, the method further includes:

[0092] Step 43: Fill the hole areas in the depth map using image processing techniques.

[0093] Due to occlusion between objects and other reasons, there are some holes in the generated depth map. Image processing techniques such as morphological operations or image inpainting algorithms are applied to fill the holes in the depth map to maintain the integrity of the depth map.

[0094] Holes are areas in an image or dataset where some part of the information is missing or not recorded correctly. These areas can be caused by sensor failure, occlusion, view limitation, insufficient depth information, etc. In depth and / or 3D point clouds, holes are formed because all details may not be fully captured from a single view or sensor data.

[0095] Specifically, the hole areas in the depth map can be filled by image processing technology, which can be achieved by using morphological operations through two basic operations of dilation and erosion in set theory, and can be further combined into more complex operations, such as opening operations, closing operations, morphological gradients, top hat operations, black hat operations, and hit-miss operations. Image inpainting algorithms can also be used to repair damaged areas or hole areas in the image through algorithms such as interpolation, image restoration, and deep learning, so as to restore the integrity and visual effects of the image.

[0096] Step 44: Reduce noise and staircase effects in the depth map by edge-preserving smoothing algorithm.

[0097] Through edge-preserving smoothing algorithms such as bilateral filtering or guided filtering, the noise and staircase effect in the depth map are reduced while maintaining the clarity of the edges, thereby improving the overall quality of the depth map.

[0098] Edge-preserving smoothing algorithms take into account the edge information of the image when smoothing it to avoid blurring or destroying the edges during the smoothing process. Such algorithms usually determine the degree and direction of smoothing based on local features of the image, such as gradients, color differences, or structural similarities.

[0099] Specifically, bilateral filtering is a nonlinear spatial filtering method that takes into account the difference in Euclidean distance and value between the output pixel and other pixels in the neighborhood, that is, it takes into account the difference between the spatial domain and the value domain. This enables bilateral filtering to effectively retain edge information while smoothing the image, avoiding the edge blurring problem that may be caused by traditional filtering methods during the smoothing process. Guided filtering is an image filtering technology that filters the depth map using a guide image, removes noise and retains depth edges through guided filtering. Guided filtering has a faster processing speed and better edge preservation effect. Optionally, the noise and step effect in the depth map can also be reduced by algorithms such as non-local mean denoising or total variation denoising, which is not specifically limited in this application.

[0100] The edge-preserving smoothing algorithm can reduce the noise and staircase effect in the depth map and maintain the clarity of the edge of the depth map, which can significantly improve the quality of the depth map and the effect of subsequent processing.

[0101] Step 50: Perform physically based rendering on the objects in the initial virtual scene, and use global illumination to process the rendered initial virtual scene.

[0102] Physical material properties such as the base color, metalness, transmittance, and roughness of objects in the initial virtual scene are defined through physical rendering to form the appearance of objects in the initial virtual scene. At the same time, physical rendering uses complex lighting models to simulate the interaction between light and the surface of objects to ensure that the visual performance of the surface of objects remains consistent and realistic under different lighting conditions. Global illumination is used to simulate the propagation path of light in the scene, calculate the interaction between light and the surface of objects, and consider the interaction between all light sources and objects in the scene, including direct lighting, indirect lighting, and occlusion relationships.

[0103] Physically Based Rendering (PBR) is a rendering technology based on physical principles. It generates more realistic and consistent images by simulating the behavior of light in the real world. When applying physically based rendering, first set the physical material properties of the object material, such as base color, metalness, transmittance, and roughness. The physical material properties directly affect the lighting and reflection effects of the object surface; secondly, use physically based lighting models to calculate the reflected and refracted light. These models take into account the interaction between light and the surface, such as reflection, scattering, and refraction; finally, apply the material properties and lighting models to the objects in the scene, and use the rendering engine for rendering. The rendering process will take into account the propagation and interaction of light in the scene to generate a more realistic virtual scene.

[0104] After determining the light sources and objects in the scene, global illumination simulates the propagation path of light emitted by the camera in the scene and calculates the interaction between light and the surface of the object. The physically based rendering results are integrated with the calculation results of global illumination to generate the initial virtual scene.

[0105] For example, an indoor virtual scene contains a wooden table, a metal chair, and a fabric sofa, with the windows and indoor lights of the virtual scene as light sources. Set the material of the wooden table to a diffuse material with wood texture, and add a certain glossiness to increase surface details. Use a metal material for the metal chair, set a high reflectivity and a suitable metal color, and add specular reflection and Fresnel reflection effects. Use a diffuse material with a cloth texture for the fabric sofa, and add a subsurface scattering effect to simulate the absorption and scattering of light by the cloth. Use a high dynamic range image as an ambient light map to simulate natural light entering from the window. Set up multiple point lights or spotlights in the scene to simulate indoor lighting, such as chandeliers and table lamps. Adjust the position, intensity, and color of the light source to ensure that the lighting effect is natural and in line with the atmosphere of the scene. Adjust the global illumination parameters according to the actual scene, such as indirect lighting intensity, diffuse reflection, number of reflections, etc. Render the scene after ensuring that the global illumination effect matches the light source and material in the scene. The virtual scene will present a more realistic lighting effect and material texture. The texture of the wooden table is clearly visible, the reflection of the metal chair is realistic, and the soft texture of the fabric sofa is perfectly presented. At the same time, global illumination makes the light distribution in the scene more natural, and the shadows and reflections between objects are more delicate and realistic.

[0106] Further, Figure 5 Yes Figure 1 The flowchart of step 50 of the embodiment of the virtual scene construction method shown in FIG. 1 shows that the initial virtual scene rendered by global illumination processing includes:

[0107] Step 51: Simulate the lighting effect of the light source on the objects in the initial virtual scene through the global illumination algorithm.

[0108] After defining the physical material properties of the objects in the initial virtual scene, set the position, color, intensity and other parameters of the light source in the initial virtual scene, calculate the direct lighting effect on the objects in the scene according to the light source type such as point light source, parallel light, etc., and select appropriate global illumination algorithms such as ray tracing, radiosity algorithm, etc. to simulate indirect lighting effects, synthesize the effects of direct and indirect lighting, and obtain the lighting effect of the objects in the initial virtual scene.

[0109] Specifically, in the process of calculating global illumination, the radiosity algorithm and Monte Carlo path tracing are used to simulate direct illumination, indirect illumination and ambient occlusion in complex light source environments. Through ray tracing algorithms such as Monte Carlo path tracing, light is projected from the camera to each pixel, and the intersection of the light and the objects in the scene and the illumination conditions at the intersection are tracked; through the radiosity algorithm or the instant radiosity algorithm, the radiosity field in the scene is constructed and the radiosity field equation is solved to calculate the illumination.

[0110] The radiosity algorithm is a global illumination algorithm that can generate globally consistent lighting effects by simulating multiple reflections of light in a scene, making the lighting of virtual scenes more realistic and natural.

[0111] Monte Carlo path tracing is a ray tracing technology based on the Monte Carlo method. It simulates global illumination by randomly sampling the propagation paths of light in the initial virtual scene and calculating the contribution of these paths to the final image pixels. Monte Carlo path tracing simulates the random propagation process of light in the initial virtual scene by randomly selecting the emission direction and reflection point of light. Monte Carlo path tracing can gradually approach the real lighting effect, obtain more accurate and stable results, and form a globally consistent lighting effect.

[0112] Further, see Figure 6 , Figure 6 Yes Figure 5 The flowchart of step 51 of the embodiment of the virtual scene construction method shown in the figure, simulating the lighting effect of the light source on the objects in the initial virtual scene by the global illumination algorithm includes:

[0113] Step 511: Calculate the lighting information of the objects in the initial virtual scene, and store the lighting information as a light map.

[0114] The static lighting information of each object in the initial virtual scene is calculated through image processing technology such as offline baking technology, and the lighting information is stored as a light map. When used, the pre-stored light map is directly called to form the visual effect in the initial virtual scene, reducing the amount of calculation in the scene rendering process.

[0115] Offline baking technology calculates the lighting, material, and shadow of the scene in advance to generate baking results such as light maps or shadow maps. These baking results contain the lighting information of static objects in the scene under specific lighting conditions, such as color, brightness, and shadows. During rendering, these baking results will be directly applied to the static objects in the scene, quickly presenting a lighting effect close to real-time rendering.

[0116] By pre-calculating the lighting information in the scene and directly using these pre-calculated results to render the scene at runtime, the computational burden of real-time rendering is reduced, and rendering efficiency and picture quality are improved.

[0117] Step 512: Apply the light map to the object to simulate the lighting effect of the light source on the object in the initial scene.

[0118] After setting up the light source in the initial virtual scene, apply the pre-stored light map to the corresponding object as part of the material, and ensure that the light map is correctly applied to all related objects to simulate the lighting effect of the light source on the objects in the initial scene.

[0119] By storing lighting information in maps and applying these maps to corresponding objects during rendering, the lighting effect of light sources on objects can be efficiently simulated, which helps to reduce the computational burden during real-time rendering and improve rendering efficiency and picture quality.

[0120] For example, in an indoor scene containing multiple static objects, simulate sunlight coming in from the window, as well as the lighting effects of indoor lights on objects in the scene. First, create and import all static objects into the scene in the 3D modeling software, and mark all objects that need to apply light maps as static. Set up light sources in the scene, including parallel light to simulate sunlight and point light sources or spotlights to simulate indoor lights, and adjust the position, intensity, color, and shadow settings of the light sources. Configure the parameters of the light map, such as resolution, quality, etc. to generate the light map. After baking, the light map is automatically applied to the static objects in the scene. When rendering, the engine will use the light map to simulate the indirect lighting effects of the light source on the object, such as shadows, reflections, and global illumination.

[0121] Step 52: In response to the dynamic changes of the initial virtual scene, dynamically update the lighting effects of the objects in the initial virtual scene.

[0122] According to the changes in the perspective and light source of the virtual camera model, the lighting effects caused by dynamic objects and environmental changes are processed through real-time global illumination such as ray tracing and radiosity transmission, and different light maps are applied for rendering to update the lighting effects in the initial virtual scene in real time.

[0123] Optionally, when dealing with the lighting effects caused by dynamic objects and environmental changes, voxel cone tracing technology or spherical harmonics can be used. Voxel cone tracing is a technology used in computer graphics to achieve real-time global illumination. This technology achieves efficient and realistic lighting effects by dividing the scene into voxel grids and using cones in each voxel to approximate the propagation path of light. Spherical harmonics are the decomposition of functions defined on a sphere, which can simplify the lighting model and avoid accurate estimation of the light source direction by linearly representing the pixel values ​​on the surface. This method reduces the impact of lighting on textures and improves the efficiency of graphics processing. Using spherical harmonic basis functions to represent lighting can compress lighting information into a limited number of coefficients, which is convenient for storage and transmission. Spherical harmonics are used to simulate global lighting effects, such as indirect lighting and soft shadows. By considering the interaction between the light source and other objects in the scene, more realistic lighting effects can be generated.

[0124] By dynamically updating the lighting effects of objects in the initial virtual scene, a more realistic and immersive rendering result can be generated, making the scene's lighting changes more flexible and improving the authenticity and visual effects of the initial virtual scene.

[0125] Step 60: Integrate the occlusion relationship into the rendered initial virtual scene to generate a new virtual scene.

[0126] According to the occlusion relationship between objects, the rendering order is adjusted to ensure that during the rendering process, the occluded objects are visually located behind the objects that occlude them, and a new virtual scene is generated through graphics drawing calls in the graphics rendering pipeline.

[0127] The graphics rendering pipeline is the process of integrating, calculating, and finally displaying rendering data such as models, materials, and textures on the screen.

[0128] See also Figure 7 , Figure 7 It is a structural diagram of an embodiment of the storage medium provided by the present application.

[0129] The storage medium 300 stores program data 301. When the program data 301 is executed by the processor, the following is achieved: Figure 1 The steps of the described virtual scene construction method.

[0130] The program data 301 is stored in a storage medium 300, and includes a number of instructions for enabling a network device (such as a router, a personal computer, a server, or other network device) or a processor to execute all or part of the steps of the virtual scene construction method described in each embodiment of the present application.

[0131] Optionally, the storage medium 300 may be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, or other medium that can store program data.

[0132] See also Figure 8 , Figure 8 It is a structural schematic diagram of an embodiment of a computer device provided by the present application.

[0133] The computer device 400 cooperates with each sensor module and control module in the simulation training vehicle to realize environmental perception, data processing and display of virtual scenes. The computer device 400 includes a processor 420 and a memory 410 that are interconnected. The memory 410 stores a computer program. When the processor 420 executes the computer program, the steps of the virtual scene construction method as described above are implemented.

[0134] Different from the prior art, the present application provides a method, medium and device for constructing a virtual scene based on point cloud data. By deeply integrating physically based rendering, global illumination technology and physical collision simulation into the virtual scene, a virtual scene with high realism and high-quality visual effects is constructed. The original radar point cloud data is also directly used to generate a depth map, which greatly optimizes the accuracy of the occlusion relationship between objects in the virtual scene, solves the problem of unnatural occlusion and unrealistic effects between objects in the virtual scene, and significantly improves the authenticity of point cloud data simulation of virtual scenes, providing more abundant and diverse training resources for autonomous driving algorithms, which is conducive to improving the recognition and response capabilities of autonomous driving algorithms in complex environments.

[0135] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium embodiment and the computer device embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0136] The present application can be used in many general or special vehicle-mounted computing system environments or configurations, such as personal computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputers, and distributed computing environments including any of the above systems or devices.

[0137] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not performed.

[0138] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0139] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0140] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for constructing a virtual scene based on point cloud data, characterized in that: include: Constructing an initial virtual scene based on the original point cloud data; Constructing a depth map corresponding to the point cloud data in the initial virtual scene, and determining the occlusion relationship between objects in the initial virtual scene based on the depth map; Performing physically based rendering on objects in the initial virtual scene, and using global illumination to process the rendered initial virtual scene; The occlusion relationship is integrated into the rendered initial virtual scene to generate a new virtual scene.

2. The method for constructing a virtual scene based on point cloud data according to claim 1, characterized in that: Before constructing the depth map corresponding to the point cloud data in the initial virtual scene, the method further includes: Performing physical collision simulation on the objects in the initial virtual scene through a physical engine, and fusing the additional point cloud data generated by the collision with the original point cloud data; The initial virtual scene is updated based on the fused point cloud data.

3. The method for constructing a virtual scene based on point cloud data according to claim 1 or 2, characterized in that: The constructing a depth map corresponding to the point cloud data in the initial virtual scene includes: Building a virtual camera model based on radar sensor parameters and virtual camera configuration parameters related to the original point cloud data; The depth map is generated based on the virtual camera model.

4. The method for constructing a virtual scene based on point cloud data according to claim 3, characterized in that: After generating the depth map based on the virtual camera model, the method further includes: Filling the hole areas in the depth map by image processing technology; The depth map is processed by an edge-preserving smoothing algorithm.

5. The method for constructing a virtual scene based on point cloud data according to claim 3, characterized in that: The generating the depth map based on the virtual camera model comprises: Projecting the point cloud data in the original coordinate system onto a two-dimensional image plane through the virtual camera model; Adding distortion to the point cloud data in the two-dimensional image plane based on the virtual camera configuration parameters to obtain projection points corresponding to the point cloud data in the two-dimensional image plane; The projection points are fused with the point cloud data on the two-dimensional image plane to generate a depth map corresponding to the point cloud data.

6. The method for constructing a virtual scene based on point cloud data according to claim 1, characterized in that: The initial virtual scene rendered by global illumination processing includes: Simulating the lighting effect of light sources on objects in the initial virtual scene by using a global illumination algorithm; In response to the dynamic changes of the initial virtual scene, the lighting effects of the objects in the initial virtual scene are dynamically updated.

7. The method for constructing a virtual scene based on point cloud data according to claim 6, characterized in that: The simulating the lighting effect of the light source on the objects in the initial virtual scene by using a global illumination algorithm includes: Calculating lighting information of objects in the initial virtual scene, and storing the lighting information as a light map; The light map is applied to the object to simulate the lighting effect of the light source on the object in the initial virtual scene.

8. The method for constructing a virtual scene based on point cloud data according to claim 1, characterized in that: The constructing an initial virtual scene based on the original point cloud data comprises: Preprocessing the original point cloud data, and aligning the preprocessed point cloud data to a target coordinate frame to form an initial virtual scene; evaluating the distribution density of the point cloud data in the initial virtual scene to divide the point cloud data into sparse areas and dense areas; The sparse areas of the point cloud data in the initial virtual scene are filled by interpolation.

9. A storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the steps of the virtual scene construction method according to any one of claims 1 to 8 are implemented.

10. A computer device, characterized in that: Includes interconnected processors and storage The memory stores a computer program, and when the processor executes the computer program, Implement the steps of the virtual scene construction method as described in any one of claims 1-8.

Images