A scene reconstruction method, device, storage medium and electronic device

Abstract

The application provides a scene reconstruction method and device, a storage medium and an electronic device, and applies to the technical field of three-dimensional reconstruction. Through a multi-source data fusion strategy, laser point cloud data, visible light image data and aerial image data are comprehensively used for joint pose solution and sparse point cloud generation, so that the accuracy of pose solution and the integrity of sparse point cloud in a complex scene are ensured. On this basis, binary mask data is generated by introducing semantic segmentation, and the three-dimensional reconstruction process is constrained by the binary mask data. The pixels corresponding to the segmented object region are excluded from the reconstruction optimization, so that the problem that the interference object is included in the reconstruction model to cause texture aliasing and structure distortion is avoided. The strategy is particularly suitable for scenes with dynamic objects, and high-fidelity reconstruction can be completed without physical isolation. Meanwhile, the target scene is divided into multiple sub-scenes for reconstruction based on the sparse point cloud, so that high-precision scene reconstruction in a large-scale complex scene is realized.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional reconstruction technology, and in particular to a scene reconstruction method, apparatus, storage medium, and electronic device. Background Technology

[0002] 3D reconstruction technology is a core technological support for fields such as digital twins, virtual reality, and digital preservation of cultural heritage. 3D Gaussian Splatting (3DGS) has become one of the mainstream technologies in this field due to its fast rendering speed and high detail rendering capabilities.

[0003] However, 3DGS technology requires the scene to be fully controlled, free from any objects or factors that could interfere with the acquisition. Otherwise, these interfering factors will be incorporated into the reconstruction and optimization process, leading to texture aliasing and structural distortion in the model. Outdoor real-world scenes typically contain continuously moving elements such as crowds and vehicles. Reconstructing these scenes using traditional methods requires temporarily closing off the site, involving significant coordination work. Furthermore, some scenes objectively lack the conditions for closure, making large-scale deployment of this technology difficult.

[0004] Therefore, how to achieve high-precision scene reconstruction in large-scale complex scenarios has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of the above problems, the present invention provides a scene reconstruction method, apparatus, storage medium, and electronic device to overcome or at least partially solve the above problems, the technical solution of which is as follows:

[0006] A scene reconstruction method, comprising:

[0007] Obtain laser point cloud data, visible light image data, and aerial image data of the target scene;

[0008] Based on the visible light image data and the aerial image data, joint pose calculation is performed to obtain the pose calculation result;

[0009] Using the pose calculation results and the laser point cloud data, sparse point cloud data corresponding to the target scene is generated;

[0010] Semantic segmentation is performed using the visible light image data and the aerial image data to generate binary mask data;

[0011] The target scene is reconstructed in three dimensions using the sparse point cloud data and the binary mask data to obtain a three-dimensional scene model corresponding to the target scene.

[0012] Optionally, the step of using the sparse point cloud data and the binary mask data to perform 3D reconstruction of the target scene to obtain a 3D scene model corresponding to the target scene includes:

[0013] The target scene is divided into multiple sub-scenes using the sparse point cloud data;

[0014] Obtain the local sparse point cloud data and local binary mask data corresponding to each sub-scene;

[0015] 3D reconstruction is performed based on the local sparse point cloud data and local binary mask data corresponding to each sub-scene to obtain the sub-scene model corresponding to each sub-scene.

[0016] The sub-scene models corresponding to each sub-scene are stitched together to obtain the three-dimensional scene model.

[0017] Optionally, the step of performing 3D reconstruction based on the local sparse point cloud data and local binary mask data corresponding to each sub-scene to obtain the sub-scene model corresponding to each sub-scene includes:

[0018] By utilizing the local sparse point cloud data corresponding to the current sub-scene, a three-dimensional Gaussian distribution set is obtained;

[0019] The effective pixel region is determined using the local binary mask data corresponding to the current sub-scene;

[0020] Calculate the loss value based on the effective pixel region;

[0021] The Gaussian parameters in the three-dimensional Gaussian distribution set are iteratively optimized using the loss value;

[0022] The iteration stops when the loss value meets the preset convergence condition, and the sub-scene model corresponding to the current sub-scene is obtained.

[0023] Optionally, dividing the target scene into multiple sub-scenes using the sparse point cloud data includes:

[0024] Obtain the spatial bounding box of the sparse point cloud data, divide the spatial bounding box according to the preset resource limit, and obtain the initial partitioning result;

[0025] Establish an association index relationship between each three-dimensional point in the sparse point cloud data and each image in the visible light image data and the aerial image data;

[0026] The number of images corresponding to each initial partition result is counted based on the associated index relationship;

[0027] The initial partition with a corresponding number of images lower than the preset image number threshold is merged with the adjacent partition until the number of images in each partition after merging reaches the preset image number threshold.

[0028] Optionally, the step of stitching together the sub-scene models corresponding to each sub-scene to obtain the 3D scene model includes:

[0029] Extract the common sparse point cloud between adjacent sub-scene models;

[0030] The coordinates of each sub-scene model are aligned using the aforementioned common sparse point cloud.

[0031] The coordinate-aligned sub-scene models are then merged and stitched together to obtain the 3D scene model.

[0032] Optionally, the step of performing joint pose calculation based on the visible light image data and the aerial image data to obtain the pose calculation result includes:

[0033] Obtain the initial camera pose data of the visible light image data;

[0034] By utilizing the common feature point data between the visible light image data and the aerial image data, and taking the initial camera pose data as a reference, the poses of all cameras are jointly optimized to obtain the pose calculation results.

[0035] Optionally, in the binary mask data, the dynamic object region is marked as an invalid region, and the main scene region is marked as a valid region.

[0036] A scene reconstruction device includes: a data acquisition unit, a pose calculation unit, a sparse point cloud generation unit, a mask generation unit, and a 3D reconstruction unit;

[0037] The data acquisition unit is used to acquire laser point cloud data, visible light image data and aerial image data of the target scene;

[0038] The pose calculation unit is used to perform joint pose calculation based on the visible light image data and the aerial image data to obtain the pose calculation result.

[0039] The sparse point cloud generation unit is used to generate sparse point cloud data corresponding to the target scene using the pose calculation results and the laser point cloud data.

[0040] The mask generation unit is used to perform semantic segmentation using the visible light image data and the aerial image data to generate binary mask data.

[0041] The 3D reconstruction unit is used to perform 3D reconstruction of the target scene using the sparse point cloud data and the binary mask data, so as to obtain a 3D scene model corresponding to the target scene.

[0042] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the scene reconstruction method described above.

[0043] An electronic device includes at least one processor, at least one memory connected to the processor, and a bus; wherein the processor and the memory communicate with each other via the bus; the processor is used to call program instructions in the memory to execute the scene reconstruction method.

[0044] By employing the above technical solutions, this invention provides a scene reconstruction method, apparatus, storage medium, and electronic device. Through a multi-source data fusion strategy, it integrates laser point cloud data, visible light image data, and aerial image data for joint pose calculation and sparse point cloud generation, ensuring the accuracy of pose calculation and the integrity of the sparse point cloud in complex scenes. Furthermore, it introduces semantic segmentation to generate binary mask data and uses this to constrain the 3D reconstruction process, excluding pixels corresponding to segmented object regions from reconstruction optimization. This avoids the problem of texture aliasing and structural distortion caused by interference objects being included in the reconstruction model. This strategy is particularly suitable for scenes with dynamic objects, achieving high-fidelity reconstruction without physical isolation. Simultaneously, it employs a strategy of dividing the target scene into multiple sub-scenes based on sparse point clouds for reconstruction, achieving high-precision scene reconstruction in large-scale complex scenes.

[0045] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0046] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0047] Figure 1 A flowchart illustrating one embodiment of the scene reconstruction method provided by this invention is shown.

[0048] Figure 2 The diagram shows a specific implementation of step S140 in the scene reconstruction method provided by the present invention.

[0049] Figure 3 A schematic diagram of the scene reconstruction device provided in an embodiment of the present invention is shown;

[0050] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention is shown. Detailed Implementation

[0051] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0052] like Figure 1 The diagram shows a flowchart of one embodiment of the scene reconstruction method provided by this invention. The method may include:

[0053] S100: Obtain laser point cloud data, visible light image data, and aerial image data of the target scene.

[0054] The target scene refers to the physical space area to be reconstructed in 3D, which includes the static scene elements and any dynamic objects that may exist. The static scene elements can include objects that do not change position over time, such as buildings, roads, landscape facilities, and vegetation, while dynamic objects can include moving objects such as people and vehicles.

[0055] Among them, laser point cloud data refers to the three-dimensional spatial point set data obtained by actively scanning the target scene using LiDAR. Each point contains three-dimensional spatial coordinate information and is used to characterize the geometric structure of the scene surface.

[0056] Visible light image data refers to two-dimensional color image data acquired by capturing a target scene using a visible light camera, containing the scene's texture and color information. Optionally, this visible light image data can be acquired by a visible light camera mounted on an acquisition device.

[0057] Aerial image data refers to two-dimensional image data acquired by cameras mounted on an aerial platform capturing images of a target scene. The acquisition angle is higher than that of ground acquisition equipment, and it is used to obtain top-down or surround-view images of the target scene.

[0058] Specifically, embodiments of the present invention can control a data acquisition device equipped with a lidar and a visible light camera to move along a ground path of the target scene. The lidar actively scans the scene surface to obtain lidar point cloud data, and the visible light camera simultaneously captures images to obtain visible light image data. During the acquisition process, the moving speed is kept uniform so that the overlap rate of adjacent visible light images is not less than a first preset overlap threshold. The camera on the aerial platform is controlled to take aerial photos of the target scene according to a preset flight path to obtain aerial image data. The flight path planning ensures that the overlap rate of adjacent aerial images is not less than a second preset overlap threshold. The obtained lidar point cloud data, visible light image data, and aerial image data are timestamped and aligned.

[0059] As examples, the data acquisition equipment can be a self-assembling visual radar scanner (with a 10Hz lidar sampling frequency, 4K camera resolution, and built-in RTK (Real-Time Kinematic) and SLAM (Simultaneous Localization and Mapping) modules) or an industrial-grade aerial drone (equipped with a high-definition aerial camera and RTK module). The scanner is controlled to move slowly along the bridge surface, street, and building facades at a speed of 0.5m / s to 1.0m / s, simultaneously acquiring lidar point cloud data and 4K visible light images, with an overlap rate of no less than 70% between adjacent images. The drone's flight altitude is set to 20-40m, its flight speed to 3m / s, employing a hybrid orbital and grid flight path. The camera tilt angles are set to 45° depression, 0° horizontal, and 30° elevation for aerial photography, with an overlap rate of no less than 75% between adjacent aerial images. Simultaneously, close-up photography is performed on key areas such as the bridge structure.

[0060] Optionally, the close-up photography acquisition provided in this embodiment of the invention may include: setting multiple acquisition heights on both sides of the key area in the target scene, taking pictures at camera tilt angles of looking down, looking at eye level, and looking up at positions above, at eye level, and below the area, respectively; taking multiple pictures at camera angles of vertical downward, tilted at 45°, and horizontal directly above the area; and taking multiple pictures at camera angles of extreme upward and horizontal directly below the area.

[0061] S110. Perform joint pose calculation based on visible light image data and aerial image data to obtain pose calculation results.

[0062] The pose calculation result refers to the set of parameters obtained by solving the camera position and attitude parameters corresponding to each frame of image acquired by the acquisition device. It includes the three-dimensional position coordinates and rotation attitude angle of the camera in space, which is used to determine the shooting orientation of each frame of image in three-dimensional space.

[0063] Specifically, in this embodiment of the invention, the initial camera pose of each frame of visible light image can be calculated based on the real-time dynamic positioning data (RTK) and real-time positioning (SLAM) and mapping data built into the acquisition device. Common feature points between the visible light image and the aerial image are extracted. Based on the initial camera pose, the pose of all cameras is globally jointly optimized by the bundle adjustment method to correct the positioning error between different acquisition devices and obtain the optimized camera pose as the pose calculation result.

[0064] As examples, embodiments of the present invention can call the scanner's SLAM algorithm combined with RTK absolute positioning data to calculate the camera intrinsic and extrinsic parameters of each frame of visible light image as the initial camera pose; extract SIFT (Scale-Invariant Feature Transform) feature points from all scanner images and UAV images and perform spatial matching, filter common feature pairs between the two types of images, use the RANSAC (Random Sample Consensus) algorithm to remove mismatched points, and then perform global bundle adjustment to iteratively optimize the pose of all cameras, unifying the pose coordinate system of UAV images and scanner images.

[0065] S120. Using the pose calculation results and laser point cloud data, generate sparse point cloud data corresponding to the target scene.

[0066] Sparse point cloud data refers to a set of three-dimensional spatial points generated by triangulation of matching feature points in visible light image data and aerial image data, and then spatially fused with laser point cloud data to obtain a global sparse three-dimensional point set. It is used to characterize the main geometric structure of the target scene and provide an initial basis for three-dimensional reconstruction.

[0067] Specifically, in this embodiment of the invention, the matching feature points in the visible light image and the aerial image can be triangulated according to the optimized camera pose to obtain a feature point cloud. The feature point cloud is then spatially fused with the laser point cloud data to generate sparse point cloud data covering the target scene. Furthermore, an association index relationship is established between each three-dimensional point in the sparse point cloud data and each acquired image.

[0068] As some examples, embodiments of the present invention can perform three-dimensional triangulation on the matched feature points based on the optimized poses of all cameras to generate an initial feature point cloud. The triangulation result is then spatially fused with the original point cloud of the lidar to generate a global three-dimensional sparse point cloud covering the scenic spot. At the same time, an association index table between the sparse point cloud and each acquired image is established to record all acquired image numbers corresponding to each three-dimensional point.

[0069] S130. Perform semantic segmentation using visible light image data and aerial image data to generate binary mask data.

[0070] Binary mask data refers to the binary label data generated by semantic segmentation of each image in visible light image data and aerial image data. It is used to distinguish between dynamic object regions and static scene regions in the image. The dynamic object region is labeled with the first value, and the static scene region is labeled with the second value.

[0071] Specifically, in this embodiment of the invention, visible light image data and aerial image data can be input into a semantic segmentation model to identify dynamic object regions and static scene regions in each image frame by frame. Based on the identification results, dynamic object regions are labeled with a first value, and static scene regions are labeled with a second value. The labeling results corresponding to each image are aggregated into binary mask data, so that each frame of the image corresponds to a separate set of binary mask data. The first value corresponds to invalid regions, and the second value corresponds to valid regions.

[0072] As examples, embodiments of the present invention can batch load all acquired images, deploy a lightweight portrait semantic segmentation model, and identify dynamic object regions such as tourists and pedestrians, as well as static scene regions such as buildings, roads, and greenery, frame by frame. Pixels in dynamic object regions are marked as 0 (invalid regions), and pixels in static scene regions are marked as 1 (valid regions). A binary mask file with the same name is generated for each image and stored in conjunction with the original image. Randomly selected images are used to manually verify the segmentation effect, and the segmentation model parameters are fine-tuned for dense crowds and occluded crowds.

[0073] S140. Use sparse point cloud data and binary mask data to perform 3D reconstruction of the target scene and obtain a 3D scene model corresponding to the target scene.

[0074] Among them, the three-dimensional scene model refers to the three-dimensional scene representation obtained after modeling the target scene through the three-dimensional Gaussian splash reconstruction method. It consists of a set of multiple three-dimensional Gaussian distribution functions. Each Gaussian distribution includes spatial position, opacity, geometric shape parameters and color features, which are used to render and generate a two-dimensional view of the target scene.

[0075] Specifically, in this embodiment of the invention, sparse point cloud data and binary mask data can be used to divide the target scene into multiple sub-scenes according to spatial distribution, perform 3D reconstruction on each sub-scene, and merge the reconstruction results of each sub-scene to obtain a 3D scene model corresponding to the target scene.

[0076] Furthermore, in this embodiment of the invention, the target scene can be divided into multiple sub-scenes based on the spatial distribution of sparse point cloud data, so that the data volume of each sub-scene matches a preset storage limit or computing power limit. A three-dimensional Gaussian distribution set is initialized based on the local sparse point cloud data corresponding to each sub-scene, and effective pixel regions are extracted from the corresponding images based on the binary mask data corresponding to each sub-scene to calculate the loss value. The Gaussian parameters are iteratively optimized until convergence, obtaining a three-dimensional Gaussian splash model corresponding to each sub-scene. The three-dimensional Gaussian splash models corresponding to each sub-scene are then aligned and stitched together based on a common sparse point cloud to obtain a three-dimensional scene model.

[0077] As examples, embodiments of the present invention can read global sparse point clouds, calculate the spatial bounding box of the point cloud, and perform initial uniform partitioning based on the storage limit and computing power requirements of a single partition on the graphics workstation (e.g., a maximum of 2000 images per partition, and a maximum of 10 million Gaussian points). When the number of associated images in a partition is less than a threshold of 100, adjacent partitions are expanded and merged sequentially until the threshold is reached. Mask-constrained 3DGS (3D Gaussian Splatting) reconstruction is performed independently on each sub-partition: the local sparse point cloud of the partition is loaded to initialize the 3D Gaussian set, and the associated images and binary masks of the partition are loaded. During training, only the effective pixels of the mask area are used to calculate L1 loss and SSIM (Structural Similarity Index) loss, iterating for 30,000 rounds. Iteration stops when the loss value no longer decreases for 1000 consecutive rounds. After reconstruction of each partition, the common sparse point cloud of the overlapping areas of the partitions is used for coordinate alignment and fusion to eliminate stitching gaps and obtain a complete 3D scene model.

[0078] This invention provides a scene reconstruction method that utilizes a multi-source data fusion strategy to jointly calculate pose and generate sparse point clouds from laser point cloud data, visible light image data, and aerial image data. This ensures the accuracy of pose calculation and the integrity of the sparse point cloud in complex scenes. Furthermore, semantic segmentation is introduced to generate binary mask data, which is then used to constrain the 3D reconstruction process. Pixels corresponding to segmented object regions are excluded from reconstruction optimization, thus avoiding the inclusion of interfering objects in the reconstruction model and preventing texture aliasing and structural distortion. This strategy is particularly suitable for scenes with dynamic objects, enabling high-fidelity reconstruction without physical isolation. Simultaneously, a strategy of dividing the target scene into multiple sub-scenes based on sparse point clouds is employed to achieve high-precision scene reconstruction in large-scale complex scenes.

[0079] Optional, based on Figure 1 The method shown is as follows: Figure 2 The diagram shows a specific implementation of step S140 in the scene reconstruction method provided by this invention. Step S140 may specifically include:

[0080] S200: The target scene is divided into multiple sub-scenes using sparse point cloud data.

[0081] In this context, a sub-scene refers to several independent sub-space regions obtained by dividing the target scene according to its spatial distribution. The combination of all sub-scenes covers the complete target scene, and the sub-scenes are spatially overlapping or adjacent to each other through common boundary areas.

[0082] Specifically, in this embodiment of the invention, the target scene can be initially divided into multiple initial partitions according to a preset resource limit, and the initial partitions with insufficient data can be merged to obtain multiple sub-scenes.

[0083] S210. Obtain the local sparse point cloud data and local binary mask data corresponding to each sub-scene.

[0084] Local sparse point cloud data refers to a subset of 3D point clouds within a single sub-scene space that falls within the global sparse point cloud data. Each sub-scene corresponds to a set of local sparse point cloud data, and the sub-scene to which each 3D point belongs can be determined by the association index relationship between each 3D point in the sparse point cloud data and the acquired image.

[0085] Local binary mask data refers to the set of binary mask data corresponding to the images associated with each sub-scene. Each frame of image corresponds to one set of binary mask data, and images and binary mask data are stored in a one-to-one correspondence. The binary mask data of a sub-scene and all its associated images constitute the local binary mask data of that sub-scene.

[0086] Specifically, embodiments of the present invention can determine all images associated with 3D points falling within the spatial range of each sub-scene based on the association index relationship between each 3D point and each image in the sparse point cloud data. 3D points falling within the spatial range of each sub-scene are extracted from the sparse point cloud data and used as the local sparse point cloud data corresponding to each sub-scene. Binary masks corresponding to the images associated with each sub-scene are extracted from the binary mask data and used as the local binary mask data corresponding to each sub-scene.

[0087] S220. Perform 3D reconstruction based on the local sparse point cloud data and local binary mask data corresponding to each sub-scene to obtain the sub-scene model corresponding to each sub-scene.

[0088] The sub-scene model refers to the 3D scene representation corresponding to a single sub-scene obtained after performing 3D Gaussian splash reconstruction. It consists of a set of multiple 3D Gaussian distribution functions, each of which includes spatial location, opacity, geometric parameters, and color features.

[0089] Specifically, in this embodiment of the invention, the local binary mask data corresponding to the current sub-scene can be used to constrain the iterative optimization process of the three-dimensional Gaussian distribution parameters generated based on local sparse point cloud data. When the iterative optimization meets the preset convergence condition, the optimization stops and the sub-scene model corresponding to the current sub-scene is obtained.

[0090] S230. Combine the sub-scene models corresponding to each sub-scene to obtain a 3D scene model.

[0091] Specifically, in this embodiment of the invention, the spatial positions of each sub-scene model can be aligned, and the aligned sub-scene models can be stitched together to obtain a three-dimensional scene model.

[0092] The present invention employs a strategy of dividing the target scene into multiple sub-scenes, reconstructing them separately, and then stitching them together. This overcomes the technical problem that large scenes cannot be completely reconstructed due to limitations in storage space and computing power of terminal devices, thereby reducing the performance dependence of 3D reconstruction on terminal hardware and enabling the method to adapt to scenes of different scales.

[0093] Optionally, in the above Figure 2 Based on one or more corresponding embodiments, in another optional embodiment provided by the present invention, step S220 may specifically include:

[0094] A three-dimensional Gaussian distribution set is obtained by using the local sparse point cloud data corresponding to the current sub-scene.

[0095] In this context, the 3D Gaussian distribution set refers to a collection of multiple 3D Gaussian distribution functions, each of which characterizes the spatial distribution characteristics of a local region in the target scene. In 3D Gaussian splatter reconstruction, the 3D Gaussian distribution set is the basic unit constituting the 3D scene model, and each Gaussian distribution includes spatial position parameters, opacity parameters, geometric shape parameters, and color feature parameters.

[0096] Specifically, in this embodiment of the invention, local sparse point cloud data corresponding to the current sub-scene can be loaded. Each 3D point in the local sparse point cloud data is initialized as a 3D Gaussian distribution function. Initial attribute parameters are assigned to each 3D Gaussian distribution function, including 3D spatial coordinates, opacity, a covariance matrix used to define the rotation and direction of the Gaussian distribution, and spherical harmonic coefficients used to characterize viewpoint-related color features. The set of all 3D Gaussian distribution functions is taken as the 3D Gaussian distribution set.

[0097] As examples, embodiments of the present invention can load local sparse point cloud data (e.g., containing approximately 500,000 three-dimensional points) corresponding to the current sub-partition, initialize each three-dimensional point as a three-dimensional Gaussian distribution function, assign the spatial coordinates of the three-dimensional point to the center position of the three-dimensional Gaussian distribution function, initialize the opacity to 0.5, initialize the covariance matrix to the identity matrix, and initialize the spherical harmonic coefficients to random values, thereby obtaining a set of three-dimensional Gaussian distributions containing approximately 500,000 Gaussian distributions.

[0098] Optionally, the three-dimensional Gaussian distribution function provided in this embodiment of the invention is defined by the following expression:

[0099] ;

[0100] in, Represents the coordinates of any point in three-dimensional space; This indicates the center location of the three-dimensional Gaussian distribution function; The covariance matrix represents the 3D Gaussian distribution function, used to define its rotation direction and spatial shape. Each 3D Gaussian distribution function is also equipped with learnable attribute parameters, including opacity. and the spherical harmonic coefficients corresponding to the color features related to the viewing angle. .

[0101] The effective pixel region is determined by using the local binary mask data corresponding to the current sub-scene.

[0102] In this context, the effective pixel region refers to the set of pixels belonging to the static scene region selected from the original image based on the binary mask data. In the binary mask data, pixels in the static scene region are labeled with the second value, and pixels in the dynamic object region are labeled with the first value; the effective pixel region is the set of all pixels whose mask value is equal to the second value.

[0103] Specifically, in this embodiment of the invention, local binary mask data corresponding to the current sub-scene can be loaded. This local binary mask data contains binary masks that correspond one-to-one with each image associated with the current sub-scene. Based on the marker values ​​of static scene regions and dynamic object regions in the binary mask, pixels marked as static scene regions are selected from the corresponding original images, and the selected set of pixels is determined as the valid pixel region.

[0104] As examples, embodiments of the present invention can load 2000 images associated with the current sub-partition and their corresponding binary mask files. Taking one frame as an example, its resolution is 3840×2160 pixels, and in the corresponding binary mask, the pixels in the crowd area are marked as 0, and the pixels in the main scene area are marked as 1. Based on this binary mask, all pixels marked as 1 in the image (e.g., approximately 6 million pixels) are filtered out as the valid pixel area of ​​the image.

[0105] The loss value is calculated based on the effective pixel area.

[0106] The loss value refers to a quantitative metric calculated based on the difference between the rendered image and the real image in the effective pixel region during the iterative optimization process of 3D Gaussian splash reconstruction. The loss value measures the accuracy of the current 3D Gaussian distribution set in representing the target scene; a smaller loss value indicates higher reconstruction quality.

[0107] Specifically, embodiments of the present invention can perform differentiable rendering based on the current three-dimensional Gaussian distribution set to generate a rendered image corresponding to the current sub-scene. Rendered pixel values ​​corresponding to the effective pixel region positions are extracted from the rendered image. A loss value is calculated based on the difference between the rendered pixel values ​​and the corresponding pixel values ​​in the original image, whereby the loss value is used to measure the accuracy of the current three-dimensional Gaussian distribution set in representing the target scene.

[0108] As examples, embodiments of the present invention can perform differentiable rendering on the current viewpoint based on the current set of three-dimensional Gaussian distributions to generate a rendered image with the same resolution as the original image. Pixel values ​​(approximately 6 million pixels) corresponding to the effective pixel region locations are extracted from the rendered image. The L1 norm difference and SSIM structure difference between these rendered pixel values ​​and the corresponding pixel values ​​in the original image are calculated. The weighted sum of the L1 loss and the SSIM loss is used as the loss value for the current iteration.

[0109] Optionally, the differentiable rendering provided in this embodiment of the invention specifically includes: sorting the three-dimensional Gaussian distribution functions corresponding to the current sub-scene according to the camera depth order; splashing the sorted three-dimensional Gaussian distribution functions sequentially into the screen space; and accumulating the color contribution of each three-dimensional Gaussian distribution function in the screen space pixel by pixel through alpha blending to obtain the color value of each pixel in the rendered image. Wherein, any pixel position... The color value is calculated as follows:

[0110] ;

[0111] in, Indicates the image index; Represents pixel coordinates; Indicates the pixel position The number of contributing three-dimensional Gaussian distribution functions; Indicates the first Opacity of a three-dimensional Gaussian distribution function; Indicates the first The color values ​​of a three-dimensional Gaussian distribution function in the view direction; Indicates the first A three-dimensional Gaussian distribution function at pixel position The two-dimensional projection value at that location. The two-dimensional projection value is calculated in the following way:

[0112] ;

[0113] in, This represents a projection operation used to transform a three-dimensional Gaussian distribution function. Based on the camera intrinsic parameter matrix and extrinsic parameter matrix Projected onto a two-dimensional image plane, a two-dimensional Gaussian distribution is generated. .

[0114] Optionally, the loss value provided in the embodiments of the present invention can be calculated in the following way:

[0115] ;

[0116] in, This represents the valid pixel region in the binary mask data; This indicates that the L1 loss between the rendered image and the original image is calculated within the effective pixel area; This indicates that the structural similarity loss between the rendered image and the original image is calculated within the effective pixel area; This represents the coefficient used to balance the weights between the L1 loss and the SSIM loss, with a value of 0.2. Through the above loss function, the calculation of the loss value is limited to the valid pixel area; pixels corresponding to invalid areas in the binary mask data are not included in the loss value calculation.

[0117] The Gaussian parameters in a three-dimensional Gaussian distribution set are optimized through iterative optimization using loss values.

[0118] Here, Gaussian parameters refer to the set of learnable properties of the three-dimensional Gaussian distribution function, including the three-dimensional spatial coordinates of the Gaussian distribution center point, the opacity value, the covariance matrix used to define the rotation and orientation of the Gaussian distribution, and the spherical harmonic coefficients used to characterize viewpoint-related color features. During iterative optimization, the Gaussian parameters are continuously updated to gradually reduce the loss value.

[0119] Specifically, in this embodiment of the invention, the loss value can be used as the optimization objective. The gradient descent method is used to backpropagate the gradient of the loss value with respect to each Gaussian parameter in the three-dimensional Gaussian distribution set. The Gaussian parameters are updated based on the gradient, gradually reducing the loss value. During the iteration process, Gaussian distribution encryption, cloning, and pruning operations are performed on the three-dimensional Gaussian distribution set to adjust the number and density of Gaussian distributions.

[0120] As examples, embodiments of the present invention can use the calculated loss value as the optimization objective, employ the Adam (Adaptive Moment Estimation) optimizer to backpropagate the gradient of the loss value with respect to each Gaussian parameter, and update each Gaussian parameter in the three-dimensional Gaussian distribution set with a learning rate of 0.001. Every 100 iterations, the three-dimensional Gaussian distribution set undergoes a Gaussian distribution encryption and cloning operation (increasing the number of Gaussian distributions in regions with high loss values), while simultaneously performing a pruning operation (removing Gaussian distributions with an opacity lower than 0.1). After 2000 iterations, the number of Gaussian distributions increases from the initial 500,000 to approximately 2 million.

[0121] Optionally, during the iteration process, embodiments of the present invention may further refine the three-dimensional Gaussian distribution functions whose contribution to the loss value is lower than a preset threshold, based on the degree of contribution of each three-dimensional Gaussian distribution function to the loss value. The refinement operation includes: within the neighborhood space of the three-dimensional Gaussian distribution function whose contribution is lower than the preset threshold, using the center position of the original three-dimensional Gaussian distribution function as a reference, generating at least two new three-dimensional Gaussian distribution functions along at least one eigenvector direction of the covariance matrix, and distributing the parameters of the original three-dimensional Gaussian distribution function to each new three-dimensional Gaussian distribution function according to a preset ratio; the contribution is determined based on the weight coefficient of the three-dimensional Gaussian distribution function at the projection position in the effective pixel region.

[0122] The iteration stops when the loss value meets the preset convergence condition, and the sub-scene model corresponding to the current sub-scene is obtained.

[0123] The preset convergence condition refers to a pre-defined criterion used to determine whether the iterative optimization process should be terminated. Optionally, the preset convergence condition provided in this embodiment of the invention may include one or more of the following conditions: the number of iterations reaches a preset upper limit, the loss value no longer decreases after multiple consecutive rounds, and the decrease in the loss value is less than a preset decrease threshold. When the preset convergence condition is met, the iterative optimization stops and the current sub-scene model is output.

[0124] Specifically, in this embodiment of the invention, after each iteration, it can be determined whether the loss value meets a preset convergence condition. When the loss value meets the preset convergence condition, the iteration stops, and the current set of three-dimensional Gaussian distributions is determined as the sub-scene model corresponding to the current sub-scene. The preset convergence condition includes the number of iterations reaching a preset upper limit or the loss value no longer decreasing for multiple consecutive rounds.

[0125] As examples, embodiments of the present invention can set a maximum number of iterations to 30,000 rounds, with the loss value no longer decreasing after 1,000 consecutive rounds as an early termination condition. When iterating to approximately 25,000 rounds, if the loss value does not show a significant decrease for 1,200 consecutive rounds, it is determined that the preset convergence condition is met, the iterative optimization is stopped, and the current set of three-dimensional Gaussian distributions (approximately 2.5 million Gaussian distributions) is output as the sub-scene model of the current sub-scene.

[0126] This invention uses binary mask data to determine the effective pixel region and calculates the loss value based on the effective pixel region to constrain the iterative optimization process of Gaussian parameters. This excludes the pixel region corresponding to the dynamic object from the loss calculation and does not participate in the optimization of the Gaussian parameters. This solves the problem of texture aliasing and structural distortion caused by the inclusion of dynamic objects in the reconstruction model, and thus realizes high-fidelity 3D reconstruction without the need for physical isolation of dynamic objects.

[0127] Optionally, in the above Figure 2 Based on one or more corresponding embodiments, in another optional embodiment provided by the present invention, step S200 may specifically include:

[0128] Obtain the spatial bounding box of the sparse point cloud data, divide the spatial bounding box according to the preset resource limit, and obtain the initial partitioning result.

[0129] The spatial bounding box refers to the smallest axis-aligned cuboid spatial region containing all three-dimensional points in sparse point cloud data. It is defined by the minimum and maximum values ​​of the three-dimensional point cloud in the three coordinate axes and is used to determine the overall spatial range of the target scene.

[0130] The preset resource limit refers to the maximum amount of resources that a single sub-scene can occupy in advance. Resources include storage space limit and / or computing resource limit, which are used to control the data size of a single sub-scene from not exceeding the carrying capacity of the terminal device.

[0131] The initial partitioning result refers to the multiple initial spatial blocks obtained after uniformly dividing the spatial bounding box according to the preset resource limit. The size of each initial spatial block is determined by the size ratio of the spatial bounding box in the three coordinate axes and the preset resource limit.

[0132] Specifically, embodiments of the present invention can obtain the maximum and minimum values ​​of each 3D point in sparse point cloud data along the three coordinate axes. Based on the maximum and minimum values ​​on each coordinate axis, a minimum axis-aligned cuboid spatial region containing all 3D points is defined as a spatial bounding box. Based on the size ratio of the spatial bounding box along the three coordinate axes and combined with a preset resource limit, the number of divisions along each coordinate axis is determined. The spatial bounding box is then uniformly divided according to the number of divisions to obtain multiple initial partitioning results.

[0133] As examples, assume the global sparse point cloud ranges from -50m to +50m along the X-axis, from -30m to +30m along the Y-axis, and from 0m to +20m along the Z-axis, with a bounding box ratio of 5:3:1. The preset resource limit is 2000 images and 10 million Gaussian points per partition. Based on this limit, the X-axis is divided into 5 parts, the Y-axis into 3 parts, and the Z-axis into 1 part, resulting in 15 initial partitions, each 20m × 20m × 20m.

[0134] Establish the association index relationship between each 3D point in sparse point cloud data and each image in visible light image data and aerial image data.

[0135] The association index relationship refers to the correspondence between each 3D point in the sparse point cloud data and each image in the visible light image data and aerial image data. It is used to determine in which acquired images each sparse 3D point is observed, and which sparse 3D points are contained in each acquired image.

[0136] Specifically, in this embodiment of the invention, based on the camera pose parameters corresponding to each image obtained during the joint pose calculation process, each 3D point in the sparse point cloud data is projected onto the imaging plane of each image, and it is determined whether the projected position of each 3D point falls within the imaging range of each image. For 3D points whose projected positions fall within the imaging range, a corresponding record between the 3D point and the image is established, ultimately forming an association index relationship between each 3D point in the sparse point cloud data and each image.

[0137] As an example, if a three-dimensional point with spatial coordinates (10m, 5m, 3m) in the target scene is taken as an example, the three-dimensional point is projected onto the imaging plane of all 5000 captured images. It is determined whether the projection position of the three-dimensional point falls within the imaging range of each image. Finally, it is recorded that the three-dimensional point is observed by 120 of the images, forming a corresponding record between the three-dimensional point and these 120 images. Based on this, an association index table between all three-dimensional points and all images is established.

[0138] The number of images corresponding to each initial partition result is counted based on the association index relationship.

[0139] The number of images refers to the total number of acquired images associated with a single initial partition, including all images from visible light image data and aerial image data that fall within the spatial range of that initial partition.

[0140] Specifically, in this embodiment of the invention, based on the spatial range of each initial partitioning result, all 3D points in the sparse point cloud data falling within that spatial range are traversed. According to the association index relationship, all images associated with these 3D points are obtained, and duplicates are counted on all obtained images to obtain the number of images corresponding to each initial partitioning result.

[0141] As examples, embodiments of the present invention can, for an initial partition (such as the 3rd block in the X direction, the 2nd block in the Y direction, and the 1st block in the Z direction, with a spatial range of 0m to 20m, 0m to 20m, and 0m to 20m), traverse all three-dimensional points (approximately 80,000 three-dimensional points in total) falling within the spatial range, obtain all images associated with these 80,000 three-dimensional points according to the association index table, and after deduplication, count the number of images corresponding to the initial partition as 150.

[0142] The initial partition with a corresponding number of images lower than the preset image number threshold is merged with the adjacent partition until the number of images in each partition reaches the preset image number threshold after merging.

[0143] The preset image number threshold refers to the minimum number of images required to be associated with a single sub-scene, used to determine whether the initial partition needs to be merged with adjacent partitions. When the number of images corresponding to the initial partition is lower than this threshold, the initial partition is merged with its adjacent partitions to ensure that each sub-scene is associated with a sufficient number of images for 3D reconstruction.

[0144] Specifically, in this embodiment of the invention, initial partitions with a corresponding image count lower than a preset image count threshold are selected and marked as partitions to be merged. Initial partitions adjacent to the partitions to be merged are determined, and the partitions to be merged are merged with their adjacent partitions. After merging, the image count corresponding to the merged partitions is recounted. If the image count after merging is still lower than the preset image count threshold, the process continues to expand outwards to merge adjacent partitions until the image count corresponding to each merged partition reaches the preset image count threshold or there are no more adjacent partitions available for merging.

[0145] As examples, if the preset image count threshold is 100, and the initial partition A has 45 images (below the 100-image threshold), while its adjacent initial partition B has 80 images (also below the 100-image threshold), merging partitions A and B and recounting the images, the merged partition has 110 images, reaching the 100-image threshold, and the merging stops. After merging, 4 of the original 15 initial partitions were below the threshold; after 3 merging operations, 12 sub-scenes are ultimately formed, each with more than 100 images.

[0146] The embodiments of the present invention employ a strategy of initial spatial division based on a preset resource limit and merging partitions with insufficient image quantity. This ensures that after the target scene is decomposed into multiple sub-scenes, each sub-scene can still be associated with a sufficient number of images for 3D reconstruction. This avoids the problem of overfitting in the 3D reconstruction process caused by insufficient associated images of factor scenes and guarantees the reconstruction quality of each sub-scene.

[0147] Optionally, in the above Figure 2 Based on one or more corresponding embodiments, in another optional embodiment provided by the present invention, step S230 may specifically include:

[0148] Extract the common sparse point cloud between adjacent sub-scene models.

[0149] In this context, the common sparse point cloud refers to the set of 3D points located within the overlapping spatial region of adjacent sub-scene models in the sparse point cloud data. Each 3D point in this set simultaneously exists in the local sparse point cloud data corresponding to each of the two adjacent sub-scenes. During the sub-scene model stitching process, the common sparse point cloud between adjacent sub-scene models is extracted and used as the registration reference to calculate the spatial transformation relationship between adjacent sub-scene models, thereby achieving coordinate alignment and fusion stitching of each sub-scene model.

[0150] Specifically, in this embodiment of the invention, adjacent sub-scene pairs can be determined based on the spatial positional relationship of each sub-scene, and the set of three-dimensional point coordinates in the local sparse point cloud data corresponding to each adjacent sub-scene can be obtained. Three-dimensional points in two sets whose spatial distance is less than a preset distance threshold are determined to be the same three-dimensional point, and the same three-dimensional point is used as the common sparse point cloud between adjacent sub-scene models.

[0151] As examples, suppose there are 12 sub-scenes. Sub-scene A and sub-scene B are considered as an adjacent sub-scene pair. The local sparse point cloud of sub-scene A contains approximately 80,000 3D points, and the local sparse point cloud of sub-scene B contains approximately 100,000 3D points. Using a spatial distance of less than 0.1m as a criterion, the common 3D points that match each other in the two local sparse point clouds are extracted, ultimately yielding approximately 6,000 common 3D points, which constitute the common sparse point cloud between the adjacent sub-scene pairs.

[0152] Coordinate alignment of each sub-scene model is performed using a common sparse point cloud.

[0153] Specifically, in this embodiment of the invention, the rotation matrix and translation vector between two local coordinate systems can be calculated based on the three-dimensional coordinates of the common sparse point cloud in the respective local coordinate systems of adjacent sub-scene models. The three-dimensional Gaussian distribution set of one sub-scene model is transformed to the coordinate system of the other sub-scene model based on the rotation matrix and translation vector, so that the geometric positions of the adjacent sub-scene models coincide in the overlapping region.

[0154] As examples, taking sub-scenes A and B as examples, based on the 3D coordinates of the 6000 common 3D points extracted in the local coordinate systems of sub-scenes A and B, the singular value decomposition method is used to calculate the rotation matrix and translation vector between the two local coordinate systems. All 3D Gaussian distributions of sub-scene B are transformed to the coordinate system of sub-scene A according to the rotation matrix and translation vector, so that the geometric positions of the two sub-scene models coincide in the overlapping area. The average registration error after coordinate alignment is 0.05m.

[0155] The coordinate-aligned sub-scene models are then merged and stitched together to obtain a 3D scene model.

[0156] Specifically, embodiments of the present invention can merge the 3D Gaussian distributions of adjacent sub-scene models within the overlapping region after coordinate alignment. The merging process includes deduplication of Gaussian distributions with repeated positions in the overlapping region, weighted averaging and fusion of Gaussian distributions with close positions, and smoothing of the Gaussian distributions at the seams. The 3D Gaussian distribution sets of all sub-scene models are merged into a complete 3D Gaussian distribution set, which serves as the 3D scene model.

[0157] As examples, assuming that all 12 sub-scene models are sequentially aligned in coordinates, this embodiment of the invention can perform a fusion process on the 3D Gaussian distributions within the overlapping areas of adjacent sub-scene models: Gaussian distributions with a spatial distance of less than 0.01m are identified as duplicates and deduplicated; Gaussian distributions with a spatial distance between 0.01m and 0.05m are weighted and averaged based on position and attributes; and Gaussian distributions around the seams are finely smoothed. After fusion, the 3D Gaussian distribution sets of all 12 sub-scenes are merged into a complete 3D Gaussian distribution set, ultimately obtaining a 3D scene model covering all scenes.

[0158] The present invention employs a strategy of extracting common sparse point clouds between adjacent sub-scene models for coordinate alignment and fusion stitching. This allows each sub-scene model to determine its spatial transformation relationship through the common sparse point cloud during the stitching process, avoiding the problem of misalignment or gaps caused by inconsistent coordinate systems of each sub-scene during direct stitching. This ensures the geometric continuity and overall consistency of the stitched 3D scene model.

[0159] Optionally, in the above Figure 1 Based on one or more corresponding embodiments, in another optional embodiment provided by the present invention, step S110 may specifically include:

[0160] Initial camera pose data for obtaining visible light image data.

[0161] The initial camera pose data refers to the initial values ​​of the camera position and attitude parameters corresponding to each frame of visible light image obtained by solving the real-time dynamic positioning data and real-time positioning and mapping data built into the acquisition device. It includes the three-dimensional position coordinates and rotation attitude angle of the camera in space, and serves as the initial reference for subsequent joint pose optimization.

[0162] Specifically, in this embodiment of the invention, while the acquisition device moves along the target scene and simultaneously acquires laser point cloud data and visible light image data, it can simultaneously acquire absolute spatial position data output by the real-time dynamic positioning module built into the acquisition device, and relative pose estimation data output by the real-time positioning and mapping module. The absolute spatial position data and relative pose estimation data corresponding to each frame of visible light image are fused and calculated to obtain the initial values ​​of the camera position and attitude parameters corresponding to each frame of visible light image, which are used as the initial camera pose data.

[0163] As examples, in embodiments of the present invention, as the visual radar scanner moves along the bridge surface at a speed of 0.5 m / s, the RTK positioning module outputs the absolute spatial coordinates of the scanner in real time at a frequency of 10 Hz, and the SLAM module simultaneously outputs the relative pose estimation data corresponding to each frame of the scanner image. The RTK position data at the acquisition time of each visible light image frame is fused with the SLAM relative pose data to obtain the initial camera pose of each visible light image frame. For example, the initial camera pose corresponding to the 120th frame image is the position coordinates (longitude 102.1856°E, latitude 29.9142°N, altitude 1012m), with a pitch angle of -5°, a yaw angle of 25°, and a roll angle of 0°.

[0164] By utilizing the common feature point data between visible light image data and aerial image data, and taking the initial camera pose data as a reference, the pose of all cameras is jointly optimized to obtain the pose calculation result.

[0165] The common feature point data refers to the set of two-dimensional projection point pairs corresponding to the same three-dimensional spatial points that appear in both types of images, extracted from visible light image data and aerial image data. Each pair of feature points in the common feature point data corresponds to the imaging position of the same three-dimensional spatial location in the target scene in different images, and is used to establish the spatial geometric constraint relationship between the two types of images, serving as the constraint basis in the joint pose optimization process.

[0166] As examples, embodiments of the present invention can extract SIFT feature points from all 5000 frames of visible light images and aerial images (approximately 2000 feature points per frame). Common feature point pairs between the two types of images are obtained through feature matching. After removing mismatched points using the RANSAC algorithm, approximately 15000 pairs of valid common feature points are retained. Starting from the initial camera pose, and with the optimization objective of minimizing the sum of reprojection errors of common feature points, global bundle adjustment (BA) is used to iteratively optimize the pose of all cameras. After approximately 100 iterations, the reprojection error converges from the initial 4.5 pixels to 0.8 pixels, yielding the optimized pose calculation results for all images.

[0167] This invention employs a strategy of jointly optimizing the poses of all cameras using common feature point data between visible light image data and aerial image data, based on initial camera pose data. This strategy enables the camera poses corresponding to the visible light image and the aerial image to achieve consistent convergence in a unified spatial coordinate system, avoiding the problem of inaccurate pose calculation caused by cumulative positioning errors of a single device. This provides an accurate pose data foundation for subsequent sparse point cloud generation and 3D reconstruction.

[0168] Optionally, in the above Figure 1 Based on one or more corresponding embodiments, in another optional embodiment provided by the present invention, the dynamic object region in the binary mask data is marked as an invalid region and the main scene region is marked as a valid region.

[0169] Specifically, in this embodiment of the invention, marking dynamic object regions in binary mask data as invalid regions means assigning a specific mask value to the pixel positions of dynamic objects (such as pedestrians, vehicles, etc.) in the image to indicate that the region will not participate in the subsequent 3D reconstruction optimization process. Correspondingly, marking the main scene region as a valid region means assigning another specific mask value to the pixel positions of static scenes (such as buildings, roads, and vegetation) in the image to indicate that the region will participate in the subsequent 3D reconstruction optimization process.

[0170] Understandably, the binary mask data corresponds one-to-one with each image, possessing the same spatial resolution. Pixels corresponding to invalid regions are masked during the loss calculation stage of 3D reconstruction; their pixel values ​​do not participate in the loss function calculation, nor do they generate gradient information for optimizing the 3D Gaussian distribution parameters. Pixels corresponding to valid regions participate normally in the loss function calculation, and their generated gradient information is used to update the values ​​of each Gaussian parameter in the 3D Gaussian distribution set. Through the above marking and masking mechanism, the 3D reconstruction process only optimizes and iterates based on the pixel information of the main scene region; the pixel information of dynamic object regions has no impact on the entire reconstruction optimization process. In terms of acquisition method, the binary mask data is obtained by inputting visible light image data and aerial image data into the semantic segmentation model for frame-by-frame inference. The semantic segmentation model outputs pixel-by-pixel classification results with the same resolution as the input image, marking pixels classified as dynamic objects as invalid regions and pixels classified as the main scene region as valid regions.

[0171] Although the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous.

[0172] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0173] Corresponding to the above method embodiments, this invention also provides a scene reconstruction device, the structure of which is as follows: Figure 3 As shown, it may include: a data acquisition unit 10, a pose calculation unit 20, a sparse point cloud generation unit 30, a mask generation unit 40, and a 3D reconstruction unit 50.

[0174] The data acquisition unit 10 is used to acquire laser point cloud data, visible light image data and aerial image data of the target scene.

[0175] The pose calculation unit 20 is used to perform joint pose calculation based on visible light image data and aerial image data to obtain pose calculation results.

[0176] The sparse point cloud generation unit 30 is used to generate sparse point cloud data corresponding to the target scene using pose calculation results and laser point cloud data.

[0177] The mask generation unit 40 is used to perform semantic segmentation using visible light image data and aerial image data to generate binary mask data.

[0178] The 3D reconstruction unit 50 is used to perform 3D reconstruction of the target scene using sparse point cloud data and binary mask data, and obtain a 3D scene model corresponding to the target scene.

[0179] Optionally, the 3D reconstruction unit 50 may include: a sub-scene division sub-unit, a local data acquisition sub-unit, a sub-scene model acquisition sub-unit, and a sub-scene model stitching sub-unit.

[0180] Sub-scene division sub-units are used to divide the target scene into multiple sub-scenes using sparse point cloud data.

[0181] The local data acquisition sub-unit is used to obtain local sparse point cloud data and local binary mask data corresponding to each sub-scene.

[0182] The sub-scene model obtains sub-units, which are used to perform 3D reconstruction based on the local sparse point cloud data and local binary mask data corresponding to each sub-scene, and obtain the sub-scene model corresponding to each sub-scene.

[0183] The sub-scene model splicing sub-unit is used to splice together the sub-scene models corresponding to each sub-scene to obtain a 3D scene model.

[0184] Optionally, the sub-scene model obtains sub-units, which can be used to obtain a three-dimensional Gaussian distribution set by utilizing the local sparse point cloud data corresponding to the current sub-scene; determine the effective pixel region by utilizing the local binary mask data corresponding to the current sub-scene; calculate the loss value based on the effective pixel region; iteratively optimize the Gaussian parameters in the three-dimensional Gaussian distribution set through the loss value; stop iterating when the loss value meets the preset convergence condition, and obtain the sub-scene model corresponding to the current sub-scene.

[0185] Optionally, the sub-scene is divided into sub-units, which can be used to obtain the spatial bounding box of sparse point cloud data. The spatial bounding box is divided according to the preset resource limit to obtain the initial partitioning result; the association index relationship between each 3D point in the sparse point cloud data and each image in the visible light image data and aerial image data is established; the number of images corresponding to each initial partitioning result is counted according to the association index relationship; the initial partitions with the number of corresponding images lower than the preset image number threshold are merged with the adjacent partitions until the number of images corresponding to each partition after merging reaches the preset image number threshold.

[0186] Optionally, the sub-scene model splicing sub-unit can be used to extract the common sparse point cloud between adjacent sub-scene models; use the common sparse point cloud to align the coordinates of each sub-scene model; and then fuse and splice the coordinate-aligned sub-scene models to obtain a 3D scene model.

[0187] Optionally, the pose calculation unit 20 can be used to obtain the initial camera pose data of the visible light image data; using the common feature point data between the visible light image data and the aerial image data, the poses of all cameras are jointly optimized based on the initial camera pose data to obtain the pose calculation results.

[0188] Optionally, dynamic object regions in the binary mask data are marked as invalid regions, while the main scene region is marked as a valid region.

[0189] This invention provides a scene reconstruction device that, through a multi-source data fusion strategy, integrates laser point cloud data, visible light image data, and aerial image data for joint pose calculation and sparse point cloud generation. This ensures the accuracy of pose calculation and the integrity of the sparse point cloud in complex scenes. Furthermore, semantic segmentation is introduced to generate binary mask data, which is then used to constrain the 3D reconstruction process. Pixels corresponding to segmented object regions are excluded from reconstruction optimization, thus avoiding the inclusion of interfering objects in the reconstruction model and preventing texture aliasing and structural distortion. This strategy is particularly suitable for scenes with dynamic objects, enabling high-fidelity reconstruction without physical isolation. Simultaneously, a strategy of dividing the target scene into multiple sub-scenes based on sparse point clouds for reconstruction is employed to achieve high-precision scene reconstruction in large-scale complex scenes.

[0190] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0191] The scene reconstruction device includes a processor and a memory. The data acquisition unit 10, pose calculation unit 20, sparse point cloud generation unit 30, mask generation unit 40 and 3D reconstruction unit 50 are all stored in the memory as program units. The processor executes the program units stored in the memory to realize the corresponding functions.

[0192] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and by adjusting kernel parameters, a multi-source data fusion strategy is employed. This strategy integrates laser point cloud data, visible light image data, and aerial image data for joint pose calculation and sparse point cloud generation, ensuring the accuracy of pose calculation and the integrity of the sparse point cloud in complex scenes. Furthermore, semantic segmentation is introduced to generate binary mask data, which constrains the 3D reconstruction process. Pixels corresponding to segmented object regions are excluded from reconstruction optimization, thus avoiding the inclusion of interfering objects in the reconstruction model, which can lead to texture aliasing and structural distortion. This strategy is particularly suitable for scenes with dynamic objects, achieving high-fidelity reconstruction without physical isolation. Simultaneously, a strategy based on sparse point clouds is used to divide the target scene into multiple sub-scenes for reconstruction, achieving high-precision scene reconstruction in large-scale complex scenes.

[0193] This invention provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the scene reconstruction method.

[0194] This invention provides a processor for running a program, wherein the program executes the scene reconstruction method during runtime.

[0195] like Figure 4 As shown, this embodiment of the invention provides an electronic device 1000, which includes at least one processor 1001, at least one memory 1002 connected to the processor 1001, and a bus 1003. The processor 1001 and the memory 1002 communicate with each other via the bus 1003. The processor 1001 is used to call program instructions in the memory 1002 to execute the aforementioned scene reconstruction method. The electronic device in this document can be a server, PC, PAD, mobile phone, etc.

[0196] The present invention also provides a computer program product that, when executed on an electronic device, is suitable for executing a program that initializes a scene reconstruction method step.

[0197] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0198] In a typical configuration, an electronic device includes one or more processors (CPUs), memory, and a bus. The electronic device may also include input / output interfaces, network interfaces, etc.

[0199] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM, and memory includes at least one memory chip. Memory is an example of computer-readable media.

[0200] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0201] In the description of this invention, it should be understood that if the terms "upper", "lower", "front", "rear", "left" and "right" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the position or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0202] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0203] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0204] The above are merely embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the present invention.

Claims

1. A scene reconstruction method, characterized in that, include: Obtain laser point cloud data, visible light image data, and aerial image data of the target scene; Based on the visible light image data and the aerial image data, joint pose calculation is performed to obtain the pose calculation result; Using the pose calculation results and the laser point cloud data, sparse point cloud data corresponding to the target scene is generated; Semantic segmentation is performed using the visible light image data and the aerial image data to generate binary mask data; The target scene is reconstructed in three dimensions using the sparse point cloud data and the binary mask data to obtain a three-dimensional scene model corresponding to the target scene.

2. The method according to claim 1, characterized in that, The step of using the sparse point cloud data and the binary mask data to perform 3D reconstruction of the target scene to obtain a 3D scene model corresponding to the target scene includes: The target scene is divided into multiple sub-scenes using the sparse point cloud data; Obtain the local sparse point cloud data and local binary mask data corresponding to each sub-scene; 3D reconstruction is performed based on the local sparse point cloud data and local binary mask data corresponding to each sub-scene to obtain the sub-scene model corresponding to each sub-scene. The sub-scene models corresponding to each sub-scene are stitched together to obtain the three-dimensional scene model.

3. The method according to claim 2, characterized in that, The step involves performing 3D reconstruction based on the local sparse point cloud data and local binary mask data corresponding to each sub-scene to obtain the sub-scene model corresponding to each sub-scene, including: By utilizing the local sparse point cloud data corresponding to the current sub-scene, a three-dimensional Gaussian distribution set is obtained; The effective pixel region is determined using the local binary mask data corresponding to the current sub-scene; Calculate the loss value based on the effective pixel region; The Gaussian parameters in the three-dimensional Gaussian distribution set are iteratively optimized using the loss value; The iteration stops when the loss value meets the preset convergence condition, and the sub-scene model corresponding to the current sub-scene is obtained.

4. The method according to claim 2, characterized in that, The process of dividing the target scene into multiple sub-scenes using the sparse point cloud data includes: Obtain the spatial bounding box of the sparse point cloud data, divide the spatial bounding box according to the preset resource limit, and obtain the initial partitioning result; Establish an association index relationship between each three-dimensional point in the sparse point cloud data and each image in the visible light image data and the aerial image data; The number of images corresponding to each initial partition result is counted based on the associated index relationship; The initial partition with a corresponding number of images lower than the preset image number threshold is merged with the adjacent partition until the number of images in each partition after merging reaches the preset image number threshold.

5. The method according to claim 2, characterized in that, The step of stitching together the sub-scene models corresponding to each sub-scene to obtain the 3D scene model includes: Extract the common sparse point cloud between adjacent sub-scene models; The coordinates of each sub-scene model are aligned using the aforementioned common sparse point cloud. The coordinate-aligned sub-scene models are then merged and stitched together to obtain the 3D scene model.

6. The method according to claim 1, characterized in that, The joint pose calculation based on the visible light image data and the aerial image data to obtain the pose calculation result includes: Obtain the initial camera pose data of the visible light image data; By utilizing the common feature point data between the visible light image data and the aerial image data, and taking the initial camera pose data as a reference, the poses of all cameras are jointly optimized to obtain the pose calculation results.

7. The method according to any one of claims 1 to 6, characterized in that, In the binary mask data, the dynamic object region is marked as an invalid region, and the main scene region is marked as a valid region.

8. A scene reconstruction device, characterized in that, include: The system comprises a data acquisition unit, a pose calculation unit, a sparse point cloud generation unit, a mask generation unit, and a 3D reconstruction unit. The data acquisition unit is used to acquire laser point cloud data, visible light image data and aerial image data of the target scene; The pose calculation unit is used to perform joint pose calculation based on the visible light image data and the aerial image data to obtain the pose calculation result. The sparse point cloud generation unit is used to generate sparse point cloud data corresponding to the target scene using the pose calculation results and the laser point cloud data. The mask generation unit is used to perform semantic segmentation using the visible light image data and the aerial image data to generate binary mask data. The 3D reconstruction unit is used to perform 3D reconstruction of the target scene using the sparse point cloud data and the binary mask data, so as to obtain a 3D scene model corresponding to the target scene.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the scene reconstruction method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, The electronic device includes at least one processor, at least one memory connected to the processor, and a bus; wherein the processor and the memory communicate with each other through the bus; the processor is used to call program instructions in the memory to execute the scene reconstruction method as described in any one of claims 1 to 7.

Images