3D scene reconstruction method, device, electronic device and computer readable medium

Abstract

The present application provides a 3D scene reconstruction method, device, electronic device and computer-readable medium, which relates to the field of computer vision technology, including: acquiring image data of a vehicle traveling in a target scene, performing sparse reconstruction using the image data to obtain a sparse point cloud of the target scene, and densely reconstructing the sparse point cloud through a pre-trained 3D Gaussian distribution representation model to obtain a first 3D scene reconstruction result, wherein the first 3D scene reconstruction result includes multiple first Gaussian distribution point clouds, the 3D Gaussian distribution representation model is used for dense reconstruction based on a 3D Gaussian splash algorithm, and the training samples of the 3D Gaussian distribution representation model are multiple real ground images and the sparse point clouds corresponding to each of the multiple real ground images. While improving the reconstruction efficiency, the weak texture area contained in the sparse point cloud obtained by sparse reconstruction is densely reconstructed to improve the accuracy of 3D scene reconstruction.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a 3D scene reconstruction method, device, electronic device and computer-readable medium. Background Art

[0002] With the rapid development of deep learning technology, visual perception algorithms require a large amount of 3D reconstruction data for annotation. Currently, most mainstream 3D reconstruction methods use traditional 3D reconstruction methods or NERF (Neural Radiance Fields) methods for 3D reconstruction.

[0003] However, it is difficult to reconstruct areas with weak textures based on traditional 3D reconstruction methods, while the reconstruction method based on NERF takes a lot of reconstruction time, making it difficult to improve the accuracy and efficiency of 3D scene reconstruction. Summary of the invention

[0004] In view of this, embodiments of the present application provide a 3D scene reconstruction method, device, electronic device and computer-readable medium to overcome the above problems or at least partially solve the above problems.

[0005] A first aspect of an embodiment of the present application provides a 3D scene reconstruction method, the method comprising:

[0006] Acquire image data of a vehicle traveling in a target scene;

[0007] Performing sparse reconstruction using the image data to obtain a sparse point cloud of the target scene;

[0008] The sparse point cloud is densely reconstructed through a pre-trained 3D Gaussian distribution representation model to obtain a first 3D scene reconstruction result, wherein the first 3D scene reconstruction result includes multiple first Gaussian distribution point clouds, and the 3D Gaussian distribution representation model is used for dense reconstruction based on a 3D Gaussian splash algorithm, and the training samples of the 3D Gaussian distribution representation model are multiple real ground images and the sparse point clouds corresponding to each of the multiple real ground images.

[0009] Optionally, the 3D Gaussian distribution representation model is trained by the following steps:

[0010] Collecting sample data collected on a vehicle, wherein the sample data includes sparse point clouds corresponding to a plurality of real images;

[0011] The sparse point cloud corresponding to the real image is densely reconstructed by using the 3D Gaussian distribution representation model to be trained to obtain a second 3D scene reconstruction result corresponding to the real image, wherein the second 3D scene reconstruction result includes a plurality of second Gaussian distribution point clouds, and the parameters of the second Gaussian distribution point clouds include: the position of the center point, the ellipsoid shape, the color, the spherical harmonics, and the opacity;

[0012] Rendering the second Gaussian distribution point cloud on the 2D image in a differentiable rasterized manner according to the parameters to obtain a rasterized image;

[0013] Calculating a loss function according to the rasterized image and the real image to obtain a loss value;

[0014] Updating the parameters of the 3D Gaussian distribution representation model to be trained according to the loss value;

[0015] Repeat the above steps until the loss value is less than the preset loss threshold, indicating that the 3D Gaussian distribution represents that the model training is completed.

[0016] Optionally, after obtaining a second 3D scene reconstruction result corresponding to the real image, the method further includes:

[0017] Copying a second Gaussian distribution point cloud of a first region in the second 3D scene reconstruction result to obtain a third Gaussian distribution point cloud of the first region, where the first region is a region where the dense reconstruction sufficiency is less than a first threshold;

[0018] Splitting a second Gaussian distribution point cloud of a second region in the second 3D scene reconstruction result to obtain a fourth Gaussian distribution point cloud of the second region, where the second region is a region where the dense reconstruction sufficiency is greater than a second threshold;

[0019] Deleting a second Gaussian distribution point cloud of a third area in the second 3D scene reconstruction result to obtain a fifth Gaussian distribution point cloud of the third area, where the third area is an area where the opacity is greater than a third threshold;

[0020] A first difference is obtained based on the second Gaussian distribution point cloud of the first area and the second Gaussian distribution point cloud of the first area, a second difference is obtained based on the second Gaussian distribution point cloud of the second area and the fourth Gaussian distribution point cloud of the second area, and a third difference is obtained based on the second Gaussian distribution point cloud of the third area and the fifth Gaussian distribution point cloud of the third area;

[0021] In the step of repeatedly executing the above steps until the loss value is less than the preset loss threshold, indicating that the 3D Gaussian distribution representation model training is completed, the method further includes:

[0022] The parameters of the 3D Gaussian distribution representation model to be trained are updated according to the first difference, the second difference and the third difference to accelerate the convergence of the 3D Gaussian distribution representation model to be trained.

[0023] Optionally, after acquiring the image data of the vehicle traveling in the target scene, the method further includes:

[0024] Acquire a plurality of sub-videos based on the image data;

[0025] For the sub-video, extract a plurality of key frames from the sub-video according to a preset time threshold and a preset feature point matching pair threshold, wherein the number of feature point matching pairs between two adjacent key frames satisfies the preset feature point matching pair threshold;

[0026] The step of performing sparse reconstruction using the image data to obtain a sparse point cloud of the target scene includes:

[0027] Sparse reconstruction is performed on the key frame to obtain a sparse point cloud of the target scene.

[0028] Optionally, performing sparse reconstruction on the key frame to obtain a sparse point cloud of the target scene includes:

[0029] Reconstruct the viewpoint of the sub-videos in the same time period that have the most feature point matches in the key frames;

[0030] Reconstruct the key frames contained in each sub-video by incremental reconstruction and / or global reconstruction to obtain a sparse point cloud corresponding to each sub-video;

[0031] The sparse point cloud corresponding to each sub-video is bundled and adjusted with the camera pose of the key frame contained in the sub-video to obtain the reconstruction result of each sub-video;

[0032] The reconstruction results of each sub-video are fused to obtain a sparse point cloud of the target scene.

[0033] Optionally, acquiring a plurality of sub-videos based on the image data includes:

[0034] Dividing the image data into a plurality of sub-videos according to a preset duration and a preset overlap rate threshold;

[0035] The overlap rate between each frame included in each of the sub-videos is greater than the preset overlap rate threshold.

[0036] Optionally, the method further comprises at least one of the following:

[0037] After obtaining the first 3D scene reconstruction result, completing a semantic segmentation task through a semantic segmentation model based on the first 3D scene reconstruction result;

[0038] The first 3D scene reconstruction result is projected by volume rendering, and the 2D image annotation task is completed according to the projection result.

[0039] A second aspect of an embodiment of the present application provides a 3D scene reconstruction method and device, the device comprising:

[0040] An image data acquisition module is configured to acquire image data of a vehicle traveling in a target scene;

[0041] A sparse reconstruction module is configured to perform sparse reconstruction using the image data to obtain a sparse point cloud of the target scene;

[0042] The dense reconstruction module is configured to perform dense reconstruction on the sparse point cloud through a pre-trained 3D Gaussian distribution representation model to obtain a first 3D scene reconstruction result, wherein the first 3D scene reconstruction result includes multiple first Gaussian distribution point clouds, the 3D Gaussian distribution representation model is used to perform dense reconstruction based on a 3D Gaussian splashing algorithm, and the training samples of the 3D Gaussian distribution representation model are multiple real ground images and the sparse point clouds corresponding to each of the multiple real ground images.

[0043] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described in the first aspect.

[0044] In a fourth aspect of the embodiments of the present application, a computer-readable medium is provided, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the method described in the first aspect is implemented.

[0045] Beneficial effects of this application:

[0046] The embodiments of the present application provide a 3D scene reconstruction method, device, electronic device and computer-readable medium, which obtain image data of a vehicle traveling in a target scene, use the image data to perform sparse reconstruction, and obtain a sparse point cloud of the target scene. The sparse point cloud is densely reconstructed using a pre-trained 3D Gaussian distribution representation model to obtain a first 3D scene reconstruction result, wherein the first 3D scene reconstruction result includes multiple first Gaussian distribution point clouds, the 3D Gaussian distribution representation model is used to densely reconstruct the sparse point cloud based on a 3D Gaussian splash algorithm, and the training samples of the 3D Gaussian distribution representation model are multiple real ground images and the sparse point clouds corresponding to each of the multiple real ground images.

[0047] The sparse point cloud obtained by sparse reconstruction of image data collected from a vehicle traveling in a target scene can be densely reconstructed through a pre-trained 3D Gaussian distribution representation model to obtain a first 3D scene reconstruction result based on the target scene, thereby achieving dense reconstruction of weak texture areas contained in the sparse point cloud obtained by sparse reconstruction while improving the reconstruction efficiency, thereby improving the accuracy and efficiency of 3D scene reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application.

[0049] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0050] Figure 1 is a flow chart of a 3D scene reconstruction method provided by an embodiment of the present application;

[0051] Figure 2 This is a flow chart of completing a labeling task by using a 3D Gaussian distribution representation model in a 3D scene reconstruction method provided by an embodiment of the present application;

[0052] Figure 3 It is a schematic diagram of the framework of a 3D scene reconstruction device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0053] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] With the introduction of the 3D Gaussian splash algorithm, this application provides a new idea for 3D scene reconstruction, which can reconstruct scenes with higher accuracy than the NERF algorithm in a shorter time frame. Therefore, using the 3D Gaussian splash algorithm for 3D scene reconstruction and subsequent data annotation can greatly improve the annotation efficiency, thereby improving the iteration efficiency of the visual perception algorithm.

[0056] Figure 1 is a flow chart of a 3D scene reconstruction method provided by an embodiment of the present application. Figure 2 FIG. 1 is a flowchart of completing a labeling task by using a 3D Gaussian distribution representation model in a 3D scene reconstruction method provided by an embodiment of the present application, such as Figure 1 and Figure 2 As shown, this embodiment provides a 3D scene reconstruction method, the method comprising:

[0057] In step S101, image data of a vehicle traveling in a target scene is acquired;

[0058] In step S102, sparse reconstruction is performed using the image data to obtain a sparse point cloud of the target scene;

[0059] In step S103, the sparse point cloud is densely reconstructed by using a pre-trained 3D Gaussian distribution representation model to obtain a first 3D scene reconstruction result, wherein the first 3D scene reconstruction result includes multiple first Gaussian distribution point clouds, and the 3D Gaussian distribution representation model is used for dense reconstruction based on a 3D Gaussian splashing algorithm, and the training samples of the 3D Gaussian distribution representation model are multiple real ground images and the sparse point clouds corresponding to each of the multiple real ground images.

[0060] First, in step S101, image data of a vehicle traveling in a target scene is obtained. Figure 2 As shown, a vehicle for data collection can be given, and multiple cameras can be deployed in different directions of the vehicle. Each camera has been calibrated with internal and external parameters. At the same time, an RTK sensor is deployed on the vehicle and time synchronized. After the cameras and sensors are deployed, the vehicle is driven on a road section of a target scene, and image data related to the target scene is collected and acquired through the camera, wherein the image data can be driving data such as videos or pictures.

[0061] Optionally, after the image data is acquired, the image data is uploaded to a server so that a technician can process the image data.

[0062] Optionally, the target scene may be a road section with relatively more or more complex target objects, such as a road section containing lane lines, traffic lights, ground arrows, or a large number of lanes.

[0063] Further, in step S102, the image data is used for sparse reconstruction to obtain a sparse point cloud of the target scene. In an embodiment of the present application, after the image data is acquired, the collected image data is sparsely reconstructed to obtain a sparse point cloud of the image data of the vehicle in the target scene, wherein the sparse reconstruction method may be based on algorithms such as MVS or SFM.

[0064] Finally, in step S103, the sparse point cloud is densely reconstructed through a pre-trained 3D Gaussian distribution representation model to obtain a first 3D scene reconstruction result, wherein the first 3D scene reconstruction result includes multiple first Gaussian distribution point clouds, and the 3D Gaussian distribution representation model is used for dense reconstruction based on a 3D Gaussian splashing algorithm, and the training samples of the 3D Gaussian distribution representation model are multiple real ground images and the sparse point clouds corresponding to each of the multiple real ground images.

[0065] In an embodiment of the present application, after obtaining the sparse point cloud of the target scene after sparse reconstruction, since the sparse point cloud obtained by sparse reconstruction cannot reflect a good representation effect for the weak texture area, it is necessary to input the sparse point cloud of the target scene into a pre-trained 3D Gaussian distribution representation model, so that the 3D Gaussian distribution representation model densely reconstructs the sparse point cloud in the target scene to obtain a first 3D scene reconstruction result based on the target scene, wherein the first 3D scene reconstruction result includes multiple first Gaussian distribution point clouds, and the parameters of the first Gaussian distribution point clouds include: the position of the center point, the shape of the ellipsoid, the color, the spherical harmonics, the opacity, etc. The first Gaussian distribution point cloud can be based on different directions of the vehicle and different driving time periods. In the process of dense reconstruction, the 3D Gaussian distribution representation model is densely reconstructed based on the 3D Gaussian splashing algorithm, and the 3D Gaussian distribution representation model is trained using multiple real ground images and the sparse point clouds corresponding to the multiple real ground images as training samples.

[0066] Through the above embodiment, by acquiring image data of a vehicle traveling in a target scene, sparse reconstruction is performed using the image data to obtain a sparse point cloud of the target scene, and the sparse point cloud is densely reconstructed using a pre-trained 3D Gaussian distribution representation model to obtain a first 3D scene reconstruction result, wherein the first 3D scene reconstruction result includes multiple first Gaussian distribution point clouds, the 3D Gaussian distribution representation model is used to densely reconstruct the sparse point cloud based on a 3D Gaussian splash algorithm, and the training samples of the 3D Gaussian distribution representation model are multiple real ground images and the sparse point clouds corresponding to each of the multiple real ground images.

[0067] The sparse point cloud obtained by sparse reconstruction of image data collected from a vehicle traveling in a target scene can be densely reconstructed through a pre-trained 3D Gaussian distribution representation model to obtain a first 3D scene reconstruction result based on the target scene, thereby achieving dense reconstruction of weak texture areas contained in the sparse point cloud obtained by sparse reconstruction while improving the reconstruction efficiency, thereby improving the accuracy and efficiency of 3D scene reconstruction.

[0068] Optionally, the 3D Gaussian distribution representation model is trained by the following steps:

[0069] Collecting sample data collected on a vehicle, wherein the sample data includes sparse point clouds corresponding to a plurality of real images;

[0070] The sparse point cloud corresponding to the real image is densely reconstructed by using the 3D Gaussian distribution representation model to be trained to obtain a second 3D scene reconstruction result corresponding to the real image, wherein the second 3D scene reconstruction result includes a plurality of second Gaussian distribution point clouds, and the parameters of the second Gaussian distribution point clouds include: the position of the center point, the ellipsoid shape, the color, the spherical harmonics, and the opacity;

[0071] Rendering the second Gaussian distribution point cloud on the 2D image in a differentiable rasterized manner according to the parameters to obtain a rasterized image;

[0072] Calculating a loss function according to the rasterized image and the real image to obtain a loss value;

[0073] Updating the parameters of the 3D Gaussian distribution representation model to be trained according to the loss value;

[0074] Repeat the above steps until the loss value is less than the preset loss threshold, indicating that the 3D Gaussian distribution represents that the model training is completed.

[0075] Specifically, in the embodiment of the present application, the 3D Gaussian distribution representation model is trained by the following steps:

[0076] First, the real images collected on the vehicle are processed and sparsely reconstructed to obtain sparse point clouds corresponding to multiple real images. The sparse point clouds corresponding to the multiple real images of each vehicle are used as a set of sample data to initialize the parameters of the 3D Gaussian distribution representation model to be trained. The sparse point clouds corresponding to the multiple real images are input into the 3D Gaussian distribution representation model to be trained to obtain a second 3D scene reconstruction result corresponding to the real image. The second 3D scene reconstruction result includes multiple second Gaussian distribution point clouds, that is, each sparse point cloud corresponds to a second Gaussian distribution point cloud, and the parameters of each second Gaussian distribution point cloud include: the position of the center point, the shape of the ellipsoid, the color, the spherical harmonics, and the opacity. The second Gaussian distribution point clouds corresponding to the sample data are initialized according to their parameters. Rendering on a 2D image in a differential rasterization manner to obtain multiple rasterized images, and calculating the loss function of the difference between each rasterized image and the corresponding real image to obtain the loss value between each rasterized image and the corresponding real image, and updating the parameters of the 3D Gaussian distribution representation model to be trained by using the loss value through back propagation, and inputting the sample data collected by another car according to the above steps into the 3D Gaussian distribution representation model to be trained after the parameters are updated, repeating the above training and parameter updating steps, and after each loss value is calculated, comparing the loss value with a preset loss threshold, until the loss value is less than the preset loss threshold, indicating that the 3D Gaussian distribution representation model training is completed, and a pre-trained 3D Gaussian distribution representation model is obtained.

[0077] Optionally, after obtaining a second 3D scene reconstruction result corresponding to the real image, the method further includes:

[0078] Copying a second Gaussian distribution point cloud of a first region in the second 3D scene reconstruction result to obtain a third Gaussian distribution point cloud of the first region, where the first region is a region where the dense reconstruction sufficiency is less than a first threshold;

[0079] Splitting a second Gaussian distribution point cloud of a second region in the second 3D scene reconstruction result to obtain a fourth Gaussian distribution point cloud of the second region, where the second region is a region where the dense reconstruction sufficiency is greater than a second threshold;

[0080] Deleting a second Gaussian distribution point cloud of a third area in the second 3D scene reconstruction result to obtain a fifth Gaussian distribution point cloud of the third area, where the third area is an area where the opacity is greater than a third threshold;

[0081] A first difference is obtained based on the second Gaussian distribution point cloud of the first area and the second Gaussian distribution point cloud of the first area, a second difference is obtained based on the second Gaussian distribution point cloud of the second area and the fourth Gaussian distribution point cloud of the second area, and a third difference is obtained based on the second Gaussian distribution point cloud of the third area and the fifth Gaussian distribution point cloud of the third area;

[0082] In the step of repeatedly executing the above steps until the loss value is less than the preset loss threshold, indicating that the 3D Gaussian distribution representation model training is completed, the method further includes:

[0083] The parameters of the 3D Gaussian distribution representation model to be trained are updated according to the first difference, the second difference and the third difference to accelerate the convergence of the 3D Gaussian distribution representation model to be trained.

[0084] Specifically, in the training phase of the 3D Gaussian distribution representation model, the following steps may also be included:

[0085] During the training phase, an adaptive control density module can also be applied to perform corresponding pruning on the second Gaussian distribution point cloud. Specifically, after obtaining the second 3D scene reconstruction result corresponding to the real image, the second 3D scene reconstruction result can be divided into multiple regions according to the reconstruction effect. The reconstruction effect can be reflected by dense reconstruction adequacy and opacity. For example, according to dense reconstruction adequacy and opacity, the second 3D scene reconstruction result can be divided into a first region, a second region, and a third region. The first region is a region where the dense reconstruction adequacy is less than a first threshold. The dense reconstruction adequacy is less than the first threshold, indicating that the reconstruction result is not sufficient and the reconstruction effect is not ideal. The second region is a region where the dense reconstruction adequacy is greater than the second threshold. The dense reconstruction adequacy is greater than the second threshold, indicating that the reconstruction is excessive. For example, an overlapping area appears between two adjacent point clouds. The third region is a region where the opacity is greater than the third threshold. The opacity is greater than the third threshold, indicating that the region in the real image corresponding to the region is blank and has no information, and no dense reconstruction is required.

[0086] For the processing of the first area, each second Gaussian distribution point cloud included in the first area in the second 3D scene reconstruction result can be copied to obtain a third Gaussian distribution point cloud of the first area. The third Gaussian distribution point cloud can make the dense reconstruction of the first area more sufficient, thereby improving the reconstruction effect of the 3D Gaussian distribution representation model.

[0087] For the processing of the second area, each second Gaussian distribution point cloud included in the second area in the second 3D scene reconstruction result can be split to obtain a fourth Gaussian distribution point cloud of the second area. The fourth Gaussian distribution point cloud can make the dense reconstruction effect of the second area better, and the overlapping point clouds are split to improve the reconstruction effect of the 3D Gaussian distribution representation model;

[0088] For the processing of the third area, each second Gaussian distribution point cloud included in the third area in the second 3D scene reconstruction result can be deleted to obtain a fifth Gaussian distribution point cloud of the third area, wherein the fifth Gaussian distribution point cloud is empty. The fifth Gaussian distribution point cloud can be used to simplify the area in the third area that does not require dense reconstruction, thereby improving the reconstruction effect of the 3D Gaussian distribution representation model.

[0089] Furthermore, after each area is processed, the loss of the Gaussian distribution point clouds before and after the processing is calculated, including: obtaining a first difference based on the second Gaussian distribution point cloud of the first area and the second Gaussian distribution point cloud of the first area, obtaining a second difference based on the second Gaussian distribution point cloud of the second area and the fourth Gaussian distribution point cloud of the second area, and obtaining a third difference based on the second Gaussian distribution point cloud of the third area and the fifth Gaussian distribution point cloud of the third area, wherein the first difference, the second difference, and the third difference may be the difference in parameters between each second Gaussian distribution point cloud and the Gaussian distribution point cloud obtained after its processing, and the parameters of the 3D Gaussian distribution representation model to be trained are updated by back propagation based on the first difference, the second difference, and the third difference, so as to accelerate the convergence of the 3D Gaussian distribution representation model to be trained.

[0090] Optionally, the first threshold, the second threshold, and the third threshold may be set according to actual conditions and are not specifically limited here.

[0091] Optionally, after acquiring the image data of the vehicle traveling in the target scene, the method further includes:

[0092] Acquire a plurality of sub-videos based on the image data;

[0093] For the sub-video, extract a plurality of key frames from the sub-video according to a preset time threshold and a preset feature point matching pair threshold, wherein the number of feature point matching pairs between two adjacent key frames satisfies the preset feature point matching pair threshold;

[0094] The step of performing sparse reconstruction using the image data to obtain a sparse point cloud of the target scene includes:

[0095] Sparse reconstruction is performed on the key frame to obtain a sparse point cloud of the target scene.

[0096] Specifically, in the embodiments of the present application, Figure 2 As shown, after acquiring image data of a vehicle traveling in a target scene, processing of the image data may include: acquiring multiple sub-videos from the image data, where the sub-videos may be in the form of video clips, and for each sub-video, in order to improve the efficiency of 3D scene reconstruction, 3D scene reconstruction is not performed on every frame in the sub-video, but multiple key frames are extracted therefrom, and 3D scene reconstruction is performed on the key frames.

[0097] For the extraction of key frames, first a preset number of video frames are selected from each sub-video according to a preset time threshold, and then for each of the selected video frames, whether it is a key frame is determined according to whether the number of feature point matching pairs between two adjacent video frames meets the preset feature point matching pair threshold, thereby obtaining multiple key frames for each sub-video.

[0098] As an example, the length of sub-video A is 10 seconds, including a total of 120 video frames. According to a preset time threshold (such as 3 seconds), 40 video frames are selected to determine whether each of the 40 selected video frames meets the requirements of key frames. The extraction needs to be based on the principle of whether the number of feature point matching pairs between two adjacent video frames in the selected 40 video frames meets the preset feature point matching pair threshold.

[0099] For sub-video A, 40 video frames are selected, and one video frame is randomly selected as the first key frame of sub-video A, such as the 5th video frame as the first key frame, and the feature points of the 4th video frame and the feature points of the 6th video frame need to be paired with the feature points of the 5th video frame respectively, and the feature point matching pairs between the 5th video frame and the 4th video frame and the 6th video frame are calculated. The feature point matching pair is two feature points in the two video frames that meet the similarity threshold as a feature point matching pair (for example, when the feature point X in the 4th video frame and the feature point Y in the 5th video frame meet the similarity threshold, they are used as a feature point matching pair, and when the feature point M in the 5th video frame and the feature point N in the 6th video frame meet the similarity threshold, they are used as a feature point matching pair). Calculating the number of feature point matching pairs between two adjacent video frames can determine whether the two video frames are similar. For similar For two or more video frames, only one of the video frames needs to be selected as the key frame. Therefore, for the similarity comparison of the video frames, a preset feature point matching pair threshold can be set in advance. The preset feature point matching pair threshold can be set according to actual needs. For example, the preset feature point matching pair threshold can be set to 1500. When the number of feature point matching pairs between the 5th video frame and the 6th video frame is greater than 1500, it is considered that the 5th video frame and the 6th video frame have a high similarity. At this time, since the 5th video frame is a key frame, the 6th video frame will not be used as a key frame. The above steps can be continued to compare the similarity of the 5th video frame with the 7th video frame to determine whether the 7th video frame is a key frame. The judgment of other video frames is similar and will not be repeated here until all the selected 40 key frames are judged and multiple key frames of sub-video A are obtained.

[0100] Optionally, when a video frame in a sub-video includes one or more feature points that can improve the 3D scene reconstruction effect, the video frame including the one or more feature points may be directly used as a key frame without determining a feature point matching pair.

[0101] Optionally, acquiring a plurality of sub-videos based on the image data includes:

[0102] Dividing the image data into a plurality of sub-videos according to a preset duration and a preset overlap rate threshold;

[0103] The overlap rate between each frame included in each of the sub-videos is greater than the preset overlap rate threshold.

[0104] Specifically, in an embodiment of the present application, multiple sub-videos are obtained from image data by dividing the sub-videos according to a preset duration and a preset overlap threshold. In other words, the division of the sub-videos needs to meet the preset duration and the preset overlap threshold, wherein the image data may be a video collection, and the video collection is divided according to the preset duration to obtain multiple video fragments, i.e., the preset duration is met as a prerequisite for the sub-video. For each video fragment, it is necessary to determine the overlap rate between each video frame included in the video fragment. Specifically, it is necessary to determine whether the overlap rate between two adjacent video frames is greater than the preset overlap rate threshold. When the overlap rate between two adjacent video frames is greater than the preset overlap rate threshold, it means that at this time, in order to meet the overlap rate of the sub-videos, it is necessary to divide the sub-videos based on two video frames whose overlap rate is greater than the preset overlap rate threshold as the boundary to ensure the overlap rate of the video frames contained in each sub-video.

[0105] Optionally, performing sparse reconstruction on the key frame to obtain a sparse point cloud of the target scene includes:

[0106] Reconstruct the viewpoint of the sub-videos in the same time period that have the most feature point matches in the key frames;

[0107] Reconstruct the key frames contained in each sub-video by incremental reconstruction and / or global reconstruction to obtain a sparse point cloud corresponding to each sub-video;

[0108] The sparse point cloud corresponding to each sub-video is bundled and adjusted with the camera pose of the key frame contained in the sub-video to obtain the reconstruction result of each sub-video;

[0109] The reconstruction results of each sub-video are fused to obtain a sparse point cloud of the target scene.

[0110] Specifically, in the embodiments of the present application, Figure 2 As shown, after the key frame selection, the key frame needs to be sparsely reconstructed to obtain the sparse point cloud of the target scene by first matching the feature points of the key frames corresponding to the multiple sub-videos obtained by shooting with different viewing angles of the camera in the same time period, and reconstructing the viewpoints with the most feature point matches among the key frames corresponding to the multiple sub-videos, and then reconstructing the key frames contained in each sub-video through incremental reconstruction and / or global reconstruction to obtain the sparse point cloud corresponding to each sub-video, and bundling and adjusting the sparse point cloud corresponding to each sub-video with the camera posture of the key frame contained in the sub-video to obtain the reconstruction result of each sub-video, and merging the reconstruction results of each sub-video to obtain the sparse point cloud of the target scene.

[0111] Specifically, assuming that there are 3D point clouds under a common perspective between the two sets of sparse point clouds, there is a Euclidean transformation between the two sets of sparse point clouds to transform the two sets of point clouds into the same coordinate system. After the Euclidean transformation, a bundle adjustment is performed to optimize the camera pose and sparse point cloud of the entire target scene.

[0112] Optionally, the method further comprises at least one of the following:

[0113] After obtaining the first 3D scene reconstruction result, completing a semantic segmentation task through a semantic segmentation model based on the first 3D scene reconstruction result;

[0114] The first 3D scene reconstruction result is projected by volume rendering, and the 2D image annotation task is completed according to the projection result.

[0115] Specifically, in the embodiments of the present application, Figure 2 As shown, after the first 3D scene reconstruction result is obtained through the pre-trained 3D Gaussian distribution representation model, it can be used for actual downstream tasks. For example, a semantic segmentation model based on 3D GS is used to perform semantic segmentation on the high-precision first 3D scene reconstruction result to complete the semantic segmentation task. The road surface, lane lines and other static objects can also be 3D labeled based on the first 3D scene reconstruction result. After the 3D labeling is completed, the first 3D scene reconstruction result is projected back to the 2D image taken by the surround view camera by volume rendering, so as to obtain a 2D labeled image to complete the 2D image labeling task.

[0116] In summary, the pure visual 3D scene reconstruction method based on the 3D Gaussian distribution representation model is more accurate than the 3D reconstruction method based on multi-sensor fusion, and the reconstruction accuracy will not decrease due to sensor jitter during the driving process of the vehicle. Therefore, the pure visual 3D scene reconstruction results based on the 3D Gaussian distribution representation model are used for subsequent labeling tasks to make the labeling results more accurate. In addition, compared with the labeling method based on NERF, the pure visual labeling method based on the 3D Gaussian distribution representation model can significantly reduce the reconstruction time while ensuring the accuracy of the 3D reconstruction results. Therefore, the use of dense reconstruction based on the 3D Gaussian distribution representation model can improve the reconstruction efficiency, thereby improving the iteration efficiency of perception algorithms such as BEV and Occpancy Network, and improving the quality of autonomous driving of automobile companies.

[0117] Figure 3 is a schematic diagram of a framework of a 3D scene reconstruction device provided in an embodiment of the present application, such as Figure 3 As shown, this embodiment provides a 3D scene reconstruction device, the device comprising:

[0118] An image data acquisition module 11 is configured to acquire image data of a vehicle traveling in a target scene;

[0119] A sparse reconstruction module 12 is configured to perform sparse reconstruction using the image data to obtain a sparse point cloud of the target scene;

[0120] The dense reconstruction module 13 is configured to perform dense reconstruction on the sparse point cloud through a pre-trained 3D Gaussian distribution representation model to obtain a first 3D scene reconstruction result, wherein the first 3D scene reconstruction result includes multiple first Gaussian distribution point clouds, and the 3D Gaussian distribution representation model is used to perform dense reconstruction based on a 3D Gaussian splash algorithm, and the training samples of the 3D Gaussian distribution representation model are multiple real ground images and the sparse point clouds corresponding to each of the multiple real ground images.

[0121] Optionally, the device further comprises:

[0122] A collection module is configured to collect sample data collected on a vehicle, wherein the sample data includes sparse point clouds corresponding to a plurality of real images;

[0123] A dense reconstruction training module is configured to perform dense reconstruction on the sparse point cloud corresponding to the real image through a 3D Gaussian distribution representation model to be trained, so as to obtain a second 3D scene reconstruction result corresponding to the real image, wherein the second 3D scene reconstruction result includes a plurality of second Gaussian distribution point clouds, and the parameters of the second Gaussian distribution point clouds include: the position of the center point, the ellipsoid shape, the color, the spherical harmonics, and the opacity;

[0124] A rendering module is configured to render the second Gaussian distribution point cloud on the 2D image in a differentiable rasterized manner according to the parameters to obtain a rasterized image;

[0125] A calculation module, configured to calculate a loss function according to the rasterized image and the real image to obtain a loss value;

[0126] A first updating module is configured to update the parameters of the 3D Gaussian distribution representation model to be trained according to the loss value;

[0127] The repeated execution module is configured to repeatedly execute the above steps until the loss value is less than a preset loss threshold, indicating that the 3D Gaussian distribution represents that the model training is completed.

[0128] Optionally, the device further comprises:

[0129] a copy module configured to, after obtaining a second 3D scene reconstruction result corresponding to the real image, copy a second Gaussian distribution point cloud of a first area in the second 3D scene reconstruction result to obtain a third Gaussian distribution point cloud of the first area, wherein the first area is an area whose dense reconstruction sufficiency is less than a first threshold;

[0130] a splitting module configured to split a second Gaussian distribution point cloud of a second area in the second 3D scene reconstruction result to obtain a fourth Gaussian distribution point cloud of the second area, where the second area is an area where the dense reconstruction sufficiency is greater than a second threshold;

[0131] a deleting module configured to delete a second Gaussian distribution point cloud of a third area in the second 3D scene reconstruction result to obtain a fifth Gaussian distribution point cloud of the third area, wherein the third area is an area where the opacity is greater than a third threshold;

[0132] a difference calculation module, configured to obtain a first difference according to the second Gaussian distribution point cloud of the first area and the second Gaussian distribution point cloud of the first area, obtain a second difference according to the second Gaussian distribution point cloud of the second area and the fourth Gaussian distribution point cloud of the second area, and obtain a third difference according to the second Gaussian distribution point cloud of the third area and the fifth Gaussian distribution point cloud of the third area;

[0133] The device also includes:

[0134] The second updating module is configured to update the parameters of the 3D Gaussian distribution representation model to be trained according to the first difference, the second difference and the third difference before the training of the 3D Gaussian distribution representation model is completed, so as to accelerate the convergence of the 3D Gaussian distribution representation model to be trained.

[0135] Optionally, the device further comprises:

[0136] a sub-video acquisition module, configured to acquire a plurality of sub-videos based on the image data after acquiring the image data of the vehicle traveling in the target scene;

[0137] A key frame extraction module is configured to extract a plurality of key frames from the sub-video according to a preset time threshold and a preset feature point matching pair threshold, wherein the number of feature point matching pairs between two adjacent key frames satisfies the preset feature point matching pair threshold;

[0138] The step of performing sparse reconstruction using the image data to obtain a sparse point cloud of the target scene includes:

[0139] Sparse reconstruction is performed on the key frame to obtain a sparse point cloud of the target scene.

[0140] Optionally, the sparse reconstruction module 12 includes:

[0141] A perspective reconstruction module is configured to reconstruct the perspective with the most matching pairs of feature points in the key frames contained in the sub-videos within the same time period;

[0142] The incremental reconstruction and / or global reconstruction module is configured to reconstruct the key frames contained in each sub-video by incremental reconstruction and / or global reconstruction to obtain a sparse point cloud corresponding to each sub-video;

[0143] A sub-video reconstruction result acquisition module is configured to bundle and adjust the sparse point cloud corresponding to each sub-video with the camera pose of the key frame contained in the sub-video to obtain the reconstruction result of each sub-video;

[0144] The sub-video reconstruction result fusion module is configured to fuse the reconstruction results of each sub-video to obtain a sparse point cloud of the target scene.

[0145] Optionally, the sub-video acquisition module includes:

[0146] A sub-video division module is configured to divide the image data into a plurality of sub-videos according to a preset duration and a preset overlap rate threshold;

[0147] The overlap rate between each frame included in each of the sub-videos is greater than the preset overlap rate threshold.

[0148] Optionally, the device further comprises at least one of the following:

[0149] A semantic segmentation module, configured to complete a semantic segmentation task through a semantic segmentation model based on the first 3D scene reconstruction result after obtaining the first 3D scene reconstruction result;

[0150] The 2D image annotation module is configured to project the first 3D scene reconstruction result by volume rendering, and complete the 2D image annotation task according to the projection result.

[0151] Based on the same inventive concept, another embodiment of the present application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the 3D scene reconstruction method as described in any of the above embodiments.

[0152] Based on the same inventive concept, another embodiment of the present application further provides a computer-readable medium on which a computer program is stored, wherein when the computer program is executed by a processor, the 3D scene reconstruction method as described in any of the above embodiments is implemented.

[0153] As for the device, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0154] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0155] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0156] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0157] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0158] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0159] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0160] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0161] The above is a detailed introduction to a 3D scene reconstruction method, device, electronic device and computer-readable medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A 3D scene reconstruction method, characterized in that: The method comprises: Acquire image data of a vehicle traveling in a target scene; Performing sparse reconstruction using the image data to obtain a sparse point cloud of the target scene; The sparse point cloud is densely reconstructed by using a pre-trained 3D Gaussian distribution representation model to obtain a first 3D scene reconstruction result, wherein the first 3D scene reconstruction result includes a plurality of first Gaussian distribution point clouds, the 3D Gaussian distribution representation model is used for dense reconstruction based on a 3D Gaussian splashing algorithm, and the training samples of the 3D Gaussian distribution representation model are a plurality of real ground images and the sparse point clouds corresponding to each of the plurality of real ground images; The 3D Gaussian distribution representation model is trained through the following steps: Collecting sample data collected on a vehicle, wherein the sample data includes sparse point clouds corresponding to a plurality of real images; The sparse point cloud corresponding to the real image is densely reconstructed by using the 3D Gaussian distribution representation model to be trained to obtain a second 3D scene reconstruction result corresponding to the real image, wherein the second 3D scene reconstruction result includes a plurality of second Gaussian distribution point clouds, and the parameters of the second Gaussian distribution point clouds include: the position of the center point, the ellipsoid shape, the color, the spherical harmonics, and the opacity; Rendering the second Gaussian distribution point cloud on the 2D image in a differentiable rasterized manner according to the parameters to obtain a rasterized image; Calculating a loss function according to the rasterized image and the real image to obtain a loss value; Updating the parameters of the 3D Gaussian distribution representation model to be trained according to the loss value; Repeat the above steps until the loss value is less than the preset loss threshold, indicating that the 3D Gaussian distribution represents that the model training is completed.

2. The 3D scene reconstruction method according to claim 1, characterized in that: After obtaining a second 3D scene reconstruction result corresponding to the real image, the method further includes: Copying a second Gaussian distribution point cloud of a first region in the second 3D scene reconstruction result to obtain a third Gaussian distribution point cloud of the first region, where the first region is a region where the dense reconstruction sufficiency is less than a first threshold; Splitting a second Gaussian distribution point cloud of a second region in the second 3D scene reconstruction result to obtain a fourth Gaussian distribution point cloud of the second region, where the second region is a region where the dense reconstruction sufficiency is greater than a second threshold; Deleting a second Gaussian distribution point cloud of a third area in the second 3D scene reconstruction result to obtain a fifth Gaussian distribution point cloud of the third area, where the third area is an area where the opacity is greater than a third threshold; A first difference is obtained based on the second Gaussian distribution point cloud of the first area and the second Gaussian distribution point cloud of the first area, a second difference is obtained based on the second Gaussian distribution point cloud of the second area and the fourth Gaussian distribution point cloud of the second area, and a third difference is obtained based on the second Gaussian distribution point cloud of the third area and the fifth Gaussian distribution point cloud of the third area; In the step of repeatedly executing the above steps until the loss value is less than the preset loss threshold, indicating that the 3D Gaussian distribution representation model training is completed, the method further includes: The parameters of the 3D Gaussian distribution representation model to be trained are updated according to the first difference, the second difference and the third difference to accelerate the convergence of the 3D Gaussian distribution representation model to be trained.

3. The 3D scene reconstruction method according to claim 1, characterized in that: After acquiring the image data of the vehicle traveling in the target scene, the method further includes: Acquire a plurality of sub-videos based on the image data; For the sub-video, extract a plurality of key frames from the sub-video according to a preset time threshold and a preset feature point matching pair threshold, wherein the number of feature point matching pairs between two adjacent key frames satisfies the preset feature point matching pair threshold; The step of performing sparse reconstruction using the image data to obtain a sparse point cloud of the target scene includes: Sparse reconstruction is performed on the key frame to obtain a sparse point cloud of the target scene.

4. The 3D scene reconstruction method according to claim 3, characterized in that: The step of performing sparse reconstruction on the key frame to obtain a sparse point cloud of the target scene includes: Reconstruct the viewpoint of the sub-videos in the same time period that have the most feature point matches in the key frames; Reconstruct the key frames contained in each sub-video by incremental reconstruction and / or global reconstruction to obtain a sparse point cloud corresponding to each sub-video; The sparse point cloud corresponding to each sub-video is bundled and adjusted with the camera pose of the key frame contained in the sub-video to obtain the reconstruction result of each sub-video; The reconstruction results of each sub-video are fused to obtain a sparse point cloud of the target scene.

5. The 3D scene reconstruction method according to claim 3, characterized in that: The acquiring a plurality of sub-videos based on the image data comprises: Dividing the image data into a plurality of sub-videos according to a preset duration and a preset overlap rate threshold; The overlap rate between each frame included in each of the sub-videos is greater than the preset overlap rate threshold.

6. The 3D scene reconstruction method according to any one of claims 1 to 5, characterized in that: The method further comprises at least one of the following: After obtaining the first 3D scene reconstruction result, completing a semantic segmentation task through a semantic segmentation model based on the first 3D scene reconstruction result; The first 3D scene reconstruction result is projected by volume rendering, and the 2D image annotation task is completed according to the projection result.

7. A 3D scene reconstruction device, characterized in that: The device comprises: An image data acquisition module is configured to acquire image data of a vehicle traveling in a target scene; A sparse reconstruction module is configured to perform sparse reconstruction using the image data to obtain a sparse point cloud of the target scene; A dense reconstruction module is configured to perform dense reconstruction on the sparse point cloud through a pre-trained 3D Gaussian distribution representation model to obtain a first 3D scene reconstruction result, wherein the first 3D scene reconstruction result includes a plurality of first Gaussian distribution point clouds, the 3D Gaussian distribution representation model is used for dense reconstruction based on a 3D Gaussian splash algorithm, and the training samples of the 3D Gaussian distribution representation model are a plurality of real ground images and the sparse point clouds corresponding to each of the plurality of real ground images; A collection module is configured to collect sample data collected on a vehicle, wherein the sample data includes sparse point clouds corresponding to a plurality of real images; A dense reconstruction training module is configured to perform dense reconstruction on the sparse point cloud corresponding to the real image through a 3D Gaussian distribution representation model to be trained, so as to obtain a second 3D scene reconstruction result corresponding to the real image, wherein the second 3D scene reconstruction result includes a plurality of second Gaussian distribution point clouds, and the parameters of the second Gaussian distribution point clouds include: the position of the center point, the ellipsoid shape, the color, the spherical harmonics, and the opacity; A rendering module is configured to render the second Gaussian distribution point cloud on the 2D image in a differentiable rasterized manner according to the parameters to obtain a rasterized image; A calculation module, configured to calculate a loss function according to the rasterized image and the real image to obtain a loss value; A first updating module is configured to update the parameters of the 3D Gaussian distribution representation model to be trained according to the loss value; The repeated execution module is configured to repeatedly execute the above steps until the loss value is less than a preset loss threshold, indicating that the 3D Gaussian distribution represents that the model training is completed.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by the processor, the 3D scene reconstruction method according to any one of claims 1 to 6 is implemented.

9. A computer-readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the 3D scene reconstruction method according to any one of claims 1 to 6 is implemented.

Images