Scene reconstruction method, system, apparatus, medium, and electronic device

Abstract

The present disclosure provides a scene reconstruction method, system, device, medium and electronic equipment. The method comprises: determining the internal parameters and pose information of each camera in the image collector according to the images of the target scene collected by the image collector; determining the point cloud data corresponding to each image based on the internal parameters and pose information of the camera corresponding to each image; determining the supplementary data corresponding to the point cloud data based on the pose information of the camera corresponding to each image; and performing scene reconstruction on the target scene based on the point cloud data and the supplementary data by using an improved three-dimensional Gaussian splatting algorithm. By inputting the point cloud data and the supplementary data into the improved three-dimensional Gaussian splatting algorithm, the three-dimensional Gaussian splatting algorithm can perform scene reconstruction on the target scene according to the point cloud data and the supplementary data, which can improve the rendering effect of the zero-parallax area in the target scene, eliminate the color inconsistency caused by occlusion between cameras, and improve the accuracy of the target scene reconstruction.

Description

Technical Field

[0001] This disclosure relates, and more particularly, to a scene reconstruction method, system, apparatus, medium, and electronic device. Background Technology

[0002] The realization of VR (Virtual Reality) technology relies on the intensive capture of high-quality data and real-time, high-fidelity rendering algorithms in the target viewing space. However, there is a lack of data available for intensively collecting data from large, open real-world scenes, and rendering algorithms often produce flaws when reconstructing scenes from the data, resulting in poor scene realism during VR viewing. Summary of the Invention

[0003] In view of this, the present disclosure provides a scene reconstruction method, system, apparatus, medium and electronic device.

[0004] According to a first aspect of the present disclosure, a scene reconstruction method is provided, the method comprising:

[0005] Based on the images of the target scene acquired by the image acquisition device, the pose information of each camera in the image acquisition device is determined, wherein the image acquisition device includes multiple cameras;

[0006] Based on the internal parameters and pose information of the camera corresponding to each image, the point cloud data corresponding to the image is determined;

[0007] Based on the internal parameters and pose information of the camera corresponding to each image, supplementary data corresponding to the point cloud data is determined, wherein the supplementary data is used to supplement the point cloud data;

[0008] Based on the point cloud data and the supplementary data, the target scene is reconstructed using an improved 3D Gaussian splashing algorithm.

[0009] In one embodiment, determining the internal parameters and pose information of each camera in the image acquisition device based on the image of the target scene acquired by the image acquisition device includes:

[0010] Extract two-dimensional feature points from each image of the target scene and the intrinsic parameters of each camera;

[0011] Match the two-dimensional feature points of each image with the three-dimensional feature points of the target scene;

[0012] Based on the two-dimensional and three-dimensional feature points matched in each image, and the internal parameters of the camera corresponding to each image, the pose information of the camera corresponding to each image is calculated.

[0013] In one embodiment, determining the supplementary data corresponding to the point cloud data based on the internal parameters and pose information of the camera corresponding to each image includes:

[0014] Based on the internal parameters and pose information of the camera corresponding to each image, the central camera in the image acquisition device is determined;

[0015] Based on the image captured by the central camera, at least one target camera in the image acquisition unit is determined, and the image captured by the target camera has a field of view overlap with the image captured by the central camera.

[0016] Based on the pose information of the central camera and the pose information of the target cameras, the farthest camera that is farthest from the central camera is determined among the target cameras.

[0017] The supplementary data is determined based on the pose information of the central camera and the furthest camera.

[0018] In one embodiment, determining the supplementary data based on the pose information of the central camera and the furthest camera includes:

[0019] Based on the pose information of the central camera and the farthest camera, determine the target distance between the central camera and the zero parallax region in the target scene;

[0020] Construct a target sphere with the target distance as its radius;

[0021] A certain amount of point cloud is set on the surface of the target sphere and assigned at least one random color to obtain supplementary data for the three-dimensional Gaussian splashing algorithm.

[0022] In one embodiment, the step of reconstructing the target scene based on the point cloud data and the supplementary data using a 3D Gaussian splashing algorithm includes:

[0023] Based on the point cloud data and the supplementary data, the target scene is rendered using a 3D Gaussian splashing algorithm to obtain an initial rendering result;

[0024] Based on the pixel information in the initial rendering result and the real pixel information of the target scene, the parameters of the three-dimensional Gaussian splashing algorithm are adjusted by a preset loss function to obtain the adjusted three-dimensional Gaussian splashing algorithm.

[0025] The target scene is reconstructed using the adjusted 3D Gaussian splashing algorithm.

[0026] In one embodiment, the step of rendering the target scene using a 3D Gaussian splashing algorithm based on the point cloud data and the supplementary data to obtain an initial rendering result includes:

[0027] Based on the point cloud data and the supplementary data, the target scene is rendered using a three-dimensional Gaussian splashing algorithm to obtain a first rendering result;

[0028] Based on the color transformation parameters, the pixels in the first rendering result are color-converted to obtain the initial rendering result.

[0029] In one embodiment, the preset loss function includes:

[0030]

[0031]

[0032]

[0033] in, , , Here, u and v are the weight parameters, representing the coordinates of pixels in the target rendering result. Represents the i-th image. Represents the gradient operator. For weight parameters, GT represents the color of a pixel in the target rendering result, while GT represents the pixel's true color. For the 3D Gaussian splash algorithm, the regularization term for the elliptical shape is... This is the regular expression for pixel color.

[0034] According to a second aspect of the present disclosure, a scene reconstruction system is provided, the system comprising:

[0035] Image acquisition device, the image acquisition device being used to acquire images of the target scene;

[0036] processor;

[0037] Memory used to store processor-executable instructions;

[0038] The processor is used to implement the steps of the method described in the first aspect above.

[0039] In one embodiment, the image acquisition device includes:

[0040] A camera assembly, comprising a camera base and a plurality of cameras, wherein the camera base is hemispherical and the plurality of cameras are evenly arranged on a protruding side of the camera base;

[0041] A bracket, which is fixedly connected to the camera assembly;

[0042] A movable base, wherein the bracket is fixedly connected to the movable base.

[0043] According to a third aspect of the present disclosure, a scene reconstruction apparatus is provided, the apparatus comprising:

[0044] The first determining unit is used to determine the internal parameters and pose information of each camera in the image acquisition device based on the image of the target scene acquired by the image acquisition device, wherein the image acquisition device includes multiple cameras;

[0045] The second determining unit is used to determine the point cloud data corresponding to the image based on the internal parameters and pose information of the camera corresponding to each image;

[0046] The third determining unit is used to determine supplementary data corresponding to the point cloud data based on the pose information of the camera corresponding to each image, wherein the supplementary data is used to supplement the point cloud data;

[0047] The reconstruction unit is used to reconstruct the target scene based on the point cloud data and the supplementary data using an improved 3D Gaussian splashing algorithm.

[0048] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect above.

[0049] According to a fifth aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described in the first aspect above.

[0050] The technical solutions provided in this disclosure may have the following beneficial effects:

[0051] By inputting point cloud data and supplementary data into the improved 3D Gaussian splashing algorithm, the algorithm can reconstruct the target scene based on the point cloud data and supplementary data, thereby improving the rendering effect of zero parallax regions in the target scene and improving the accuracy of target scene reconstruction. According to the designed loss function, the algorithm's sensitivity to complex lighting conditions can be reduced. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating a scene reconstruction method according to an exemplary embodiment of the present disclosure;

[0053] Figure 2 This is a flowchart illustrating an exemplary embodiment of the present disclosure for determining a camera pose;

[0054] Figure 3This is a flowchart illustrating an exemplary embodiment of the present disclosure of a method for generating supplementary data;

[0055] Figure 4 This is a flowchart illustrating an exemplary embodiment of the present disclosure for determining supplementary data;

[0056] Figure 5 This is a flowchart illustrating an exemplary embodiment of the present disclosure of a target scene reconstruction;

[0057] Figure 6 This is a flowchart illustrating a method for rendering a target scene according to an exemplary embodiment of this disclosure;

[0058] Figure 7 This is a schematic diagram of the structure of an image acquisition device shown in an exemplary embodiment of the present disclosure;

[0059] Figure 8 This is a schematic diagram illustrating the installation of a camera assembly according to an exemplary embodiment of this disclosure;

[0060] Figure 9 This is a block diagram of a scene reconstruction apparatus illustrated in an exemplary embodiment of the present disclosure;

[0061] Figure 10 This is a hardware structure diagram of an electronic device illustrated in an exemplary embodiment of the present disclosure. Detailed Implementation

[0062] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0063] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0064] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0065] Six-DOF VR experiences rely on intensive capture of high-quality data and efficient, high-fidelity rendering algorithms. Among related technologies, algorithms for scene reconstruction and new perspective synthesis mainly fall into three paradigms: image-based rendering, implicit neural radiation field rendering, and explicit radiation field rendering. 3DGS (3D Gaussian Splatting) uses sparse point clouds generated by SFM (Structure from Motion) as initialization, but it is limited by its inability to accurately estimate the depth of zero-parallax regions such as the sky or distant objects. Therefore, during subsequent iterative optimization, it is prone to generating floating objects in the foreground, causing errors such as "sky collapse." Secondly, the unavoidable presence of external factors (such as floating clouds) and internal factors (such as camera autofocus and white balance) profoundly affects the captured natural scene images, resulting in many flaws in the scene reconstruction algorithm. Furthermore, it easily causes discontinuous color transitions when switching between adjacent viewpoints, leading to significant differences between the scene observed by the user in VR and the real scene.

[0066] Based on this, the present disclosure provides a scene reconstruction method that can be applied to electronic devices such as computers and can effectively improve the scene reconstruction effect.

[0067] Please see Figure 1 The flowchart shown is as follows. Figure 1 A flowchart illustrating a scene reconstruction method according to an illustrative embodiment of this disclosure includes the following steps:

[0068] S101, based on the image of the target scene acquired by the image acquisition device, determine the internal parameters and pose information of each camera in the image acquisition device.

[0069] The target scene can be an indoor or outdoor scene. The camera pose refers to the camera's position and orientation in space, which can be described and represented by a transformation matrix. The image acquisition unit includes multiple cameras, and each camera in the camera acquisition unit will acquire images for the same scene. For example, in the same location in the same room, multiple cameras in the camera acquisition unit can simultaneously acquire multiple images.

[0070] After the image acquisition device captures images, the camera pose of the corresponding camera can be determined based on each image. For example, for any given camera, the camera's intrinsic parameters can be obtained first, and then the camera's intrinsic parameters can be calculated using a calibration algorithm to obtain the camera's intrinsic parameter matrix. Then, feature detection and matching are performed on the image to obtain the matching feature points in the image. Finally, the camera pose can be determined based on the coordinates of the feature points in the image and the camera's intrinsic parameter matrix.

[0071] In one embodiment, the process of determining the camera pose can be referred to Figure 2 The flowchart shown includes:

[0072] S201, extract the two-dimensional feature points of each image and the internal parameters of each camera from the image of the target scene.

[0073] S202, matching the two-dimensional feature points of each image with the three-dimensional feature points of the target scene.

[0074] S203, based on the two-dimensional and three-dimensional feature points matched in each image, and the internal parameters of the camera corresponding to each image, calculate the pose information of the camera corresponding to each image.

[0075] First, the intrinsic parameters of each camera are obtained, and the intrinsic parameter matrix of each camera is calculated. Simultaneously, two-dimensional feature points are extracted from each image, specifically using algorithms such as SIFT (Scale-invariant feature transform) and ORB (Oriented Fast and Rotated Brief). Then, corresponding three-dimensional feature points are matched in the target scene. These three-dimensional feature points can be feature points with known coordinates in the target scene, or they can be obtained through multi-view reconstruction. Furthermore, the two-dimensional and three-dimensional feature points are matched to determine the corresponding matching two-dimensional and three-dimensional feature points. Then, based on the coordinates of the matched two-dimensional and three-dimensional feature points, and the intrinsic parameter matrix of the corresponding camera, the pose information of each camera can be calculated. For example, the coordinates of the matched two-dimensional and three-dimensional feature points, along with the intrinsic parameter matrix of the corresponding camera, can be input into the PnP (Perspective-n-Point) algorithm. Solving the algorithm yields the pose matrix of each camera.

[0076] S102, based on the internal parameters of the camera corresponding to each image and the camera pose information, determine the point cloud data corresponding to the image.

[0077] There are several methods for converting 2D images into 3D point cloud data. For example, using two 2D images from different perspectives, the depth information of the target scene can be recovered to generate point cloud data. Alternatively, a convolutional neural network can be used to predict the depth information of each pixel in the 2D image, converting each pixel into 3D coordinates to generate point cloud data. Another method is SFM (Structure from Motion), which converts 2D images into 3D point cloud data.

[0078] In one embodiment, three-dimensional world coordinates can be converted into two-dimensional image coordinates based on the camera's internal parameters. Then, feature extraction and feature point matching are performed on the image. Based on the matched feature points, triangulation is performed using the camera's internal parameters and pose information. Finally, point cloud data is generated based on the results of the triangulation.

[0079] S103, based on the pose information of the camera corresponding to each image, determine the supplementary data corresponding to the point cloud data.

[0080] The supplementary data is used to supplement the point cloud data, addressing potential data gaps that may occur when converting a 2D image into 3D point cloud data. For example, in natural scenes, areas such as the sky and distant objects lack parallax information (i.e., no significant positional change from different viewpoints), making it difficult to accurately estimate depth during point cloud data conversion. This can lead to problems like "sky collapse" (the sky area is incorrectly compressed into the foreground) or floating foreground objects during reconstruction. Therefore, by supplementing the point cloud data, supplementary data can be constructed to cover the entire target scene with zero parallax regions, preventing scene reconstruction failure due to data gaps.

[0081] In one embodiment, the method for generating the supplementary data can be found in [reference needed]. Figure 3 The flowchart shown includes the following steps:

[0082] S301, Based on the pose information of the camera corresponding to each image, determine the center camera in the image acquisition device.

[0083] S302, Based on the image captured by the central camera, determine at least one target camera in the image acquisition unit.

[0084] S303, based on the pose information of the central camera and the pose information of the target camera, determine the farthest camera among the target cameras that is farthest from the central camera.

[0085] S304, determine the supplementary data based on the pose information of the central camera and the farthest camera.

[0086] The central camera is determined based on the perspective center method. It is the camera that covers the largest area of ​​the target scene among the multiple cameras; that is, the central camera is determined based on the field of view of each camera. The core idea of ​​the perspective center method is to observe the projection of an object or scene and use known geometric relationships to infer the perspective center of the camera. The perspective center refers to the optical center of the camera, that is, the point through which all light rays entering the camera lens pass. When determining the central camera based on the perspective center method, the extrinsic parameters of each camera can be determined based on the pose information of each camera. Then, based on the rotation matrix and camera translation vector in the extrinsic parameters, the central camera is determined.

[0087] The images captured by the target camera and the images captured by the central camera have overlapping fields of view. After determining the central camera, the target camera with an overlapping field of view with the image captured by the central camera can be identified from among the multiple cameras of the image acquisition device.

[0088] Then, based on the 3D coordinates from the pose information of the central camera and the 3D coordinates from the pose information of each target camera, the distance between the central camera and each target camera is calculated. Subsequently, the farthest target camera, located furthest from the central camera, is identified. Finally, supplementary point cloud data for the target scene is determined using the pose information of the central camera and the farthest camera.

[0089] In one embodiment, the process of determining supplementary data based on the center camera and the furthest camera can be found in [link to relevant documentation]. Figure 4 The flowchart shown includes:

[0090] S401, based on the pose information of the central camera and the farthest camera, determine the target distance between the central camera and the zero parallax region in the target scene.

[0091] S402, construct a target sphere with the target distance as the radius.

[0092] S403, a certain amount of point cloud is set on the surface of the target sphere and assigned at least one random color to obtain supplementary data for the three-dimensional Gaussian splashing algorithm.

[0093] A zero-parallax region refers to a region where parallax is zero. Parallax refers to the relative positional difference between two viewpoints (e.g., the left and right eyes or two cameras). In other words, a zero-parallax region is the region in a visual system where depth perception difference is minimal or nonexistent under specific conditions. Zero parallax means that this region lies at the point where the optical axes of the two cameras coincide. Therefore, a zero-parallax region typically occurs in the overlapping area of ​​the center camera and the furthest camera. Correspondingly, the depth of the zero-parallax region corresponds to the distance between the intersection points of the optical axes of the two cameras, i.e., the baseline length between the two cameras, which is the physical distance between them. Therefore, the extrinsic parameters of the center camera can be determined based on its pose information, and the extrinsic parameters of the furthest camera can be determined based on its pose information. Then, the baseline length is calculated based on the translation vectors in the extrinsic parameters of the two cameras.

[0094] Then, the target distance between the central camera and the zero parallax region is calculated based on the geometric model. The geometric model can be referenced in formula (1), as follows:

[0095] (1)

[0096] Where D represents the target distance, f represents the focal length of the center camera, B represents the baseline length, and d represents the parallax. The parallax d in the zero parallax region is close to 0 and can be determined by calibrating the overlapping boundary of the images taken by the farthest camera and the center camera.

[0097] After determining the target distance between the central camera and the zero parallax region, a target sphere can be constructed with the target distance as the radius, and a point cloud of random colors can be uniformly distributed on the surface of the target sphere to obtain supplementary data for the point cloud data in the final 3D Gaussian splash algorithm.

[0098] Supplementary data is a global structure for the entire target scene. By filling in the data gaps in the zero parallax region, providing initial constraints, and enhancing reconstruction stability, it can ensure that areas that are difficult to capture in natural scenes can also be accurately modeled, thereby improving the accuracy of target scene reconstruction.

[0099] S104, Based on the point cloud data and the supplementary data, the target scene is reconstructed using an improved three-dimensional Gaussian splashing algorithm.

[0100] After determining the supplementary data for the point cloud data, both the point cloud data and the supplementary data can be input into the improved 3D Gaussian splashing algorithm to reconstruct the target scene. When reconstructing the target scene using the 3D Gaussian splashing algorithm, the Gaussian ellipsoid in the algorithm needs to be defined to obtain a 3D Gaussian ellipsoid. Then, the 3D Gaussian ellipsoid is mapped onto a 2D plane and rasterized for imaging, ultimately generating a rendered image to complete the scene reconstruction of the target scene.

[0101] In some embodiments, the method for reconstructing the target scene can be found in [reference needed]. Figure 5 The flowchart shown includes:

[0102] S501, Based on the point cloud data and the supplementary data, the target scene is rendered using a three-dimensional Gaussian splashing algorithm to obtain an initial rendering result.

[0103] First, after inputting the point cloud data and supplementary data into the 3D Gaussian splashing algorithm, the position of the Gaussian ellipsoid is defined based on the 3D spatial coordinates in the point cloud data. The shape and orientation of the Gaussian ellipsoid are also initialized, as is its opacity during rendering. Then, the point clouds of the target sphere's surface from the point cloud data and supplementary data are merged as the initial distribution basis for the Gaussian ellipsoid. The point cloud covering the target sphere's surface ensures zero parallax coverage, preventing crashes due to insufficient depth during the initial optimization phase.

[0104] After defining the Gaussian ellipsoid, the target scene can be initially rendered based on the Gaussian ellipsoid to obtain the initial rendering result. In one embodiment, during the initial rendering of the target scene, an affine color transformation model can be introduced to constrain the color consistency of the target scene in the initial rendering result. For details, please refer to [link to relevant documentation]. Figure 6 The flowchart shown includes the following steps:

[0105] S601, based on the point cloud data and the supplementary data, the target scene is rendered using a three-dimensional Gaussian splashing algorithm to obtain a first rendering result.

[0106] S602, according to the color transformation parameters, the pixels in the first rendering result are color-converted to obtain the initial rendering result.

[0107] After rendering the target scene based on point cloud data and supplementary data using the 3D Gaussian splash algorithm to obtain the first rendering result, a view-independent color transformation parameter is introduced. This allows us to constrain the first rendering result and obtain the initial rendering result. Here, i represents the camera sequence. It can be The transformation matrix, It can be The offset vector. The process of color conversion for pixels in the first rendering result can be found in formula (2), as follows:

[0108] (2)

[0109] in, This represents the color in the initial rendering result. This represents the color in the first rendered result.

[0110] It should be noted that the color transformation parameters can be set arbitrarily during initialization, and the color transformation parameters will also be optimized and adjusted when optimizing the 3D Gaussian splashing algorithm.

[0111] By introducing color transformation parameters, the problem of inconsistent colors in the target scene in the rendering results caused by changes in lighting or camera differences can be corrected. By correcting color differences through affine transformation, the realism of the target scene rendering is improved.

[0112] S502, based on the pixel information in the initial rendering result and the real pixel information of the target scene, the parameters of the three-dimensional Gaussian splashing algorithm are adjusted through a preset loss function to obtain the adjusted three-dimensional Gaussian splashing algorithm.

[0113] After obtaining the initial rendering result of the target scene, the parameters of the 3D Gaussian splashing algorithm can be adjusted and optimized based on the pixel information in the initial rendering result and the real pixel information in the target scene. In one embodiment, the parameters of the 3D Gaussian splashing algorithm can be optimized by a preset loss function to obtain the adjusted 3D Gaussian splashing algorithm. The preset loss function can be found in formula (3), as follows:

[0114]

[0115]

[0116]

[0117] Where Loss represents the loss value. , , Here, u and v are the weight parameters, representing the coordinates of pixels in the target rendering result. Represents the i-th image. Represents the gradient operator. For weight parameters, GT represents the color of a pixel in the target rendering result, while GT represents the pixel's true color. This represents the photometric loss term in the improved 3D Gaussian splashing algorithm. This represents the photometric loss term in the original 3D Gaussian splashing algorithm. for The weight parameters, For the 3D Gaussian splash algorithm, the regularization term for the elliptical shape is... This is the regular expression for pixel color.

[0118] Iterative training of the 3D Gaussian splashing algorithm using a pre-defined loss function makes the algorithm more sensitive to image edge information during iteration. The L1 loss term, based on image gradients, enhances the algorithm's ability to perceive image gradients and learn more details. By introducing... Regularization terms can prevent excessively long or flat ellipsoids from affecting the rendering quality of the target scene.

[0119] S503, the target scene is reconstructed using the adjusted three-dimensional Gaussian splashing algorithm.

[0120] After adjusting the 3D Gaussian splash algorithm, the target scene can be reconstructed using the algorithm, and the reconstruction results can be integrated into the platform engine. By setting up a virtual camera, users can achieve a six-degrees-of-freedom VR experience in a head-mounted display.

[0121] The scene reconstruction method provided in this disclosure can handle the inevitable zero parallax region reconstruction errors and color inconsistencies between adjacent viewpoints when reconstructing natural scenes, achieve high-fidelity rendering, further enhance the immersive VR experience, and improve the color consistency of the scene viewed during VR interaction.

[0122] Because six-degrees-of-freedom VR experiences rely not only on efficient and high-fidelity rendering algorithms, but also on the intensive capture of high-quality data. In related technologies, data for large-scale, intensively collected scenes is scarce; typically, images of the target scene can only be acquired through fixed-point shooting. While fixed-point shooting offers relatively high acquisition density, the explorable VR space is limited to a small area. Alternatively, some image acquisition technologies focus only on indoor scenes, and their acquisition density still has room for improvement, making it impossible to support long-term capture of dynamic scenes (video capture). Due to the limitations of dataset acquisition, the realism of scene reconstruction is relatively poor. Therefore, this disclosure also provides an embodiment of a scene reconstruction system. The scene reconstruction system includes:

[0123] Image acquisition device, the image acquisition device being used to acquire images of the target scene;

[0124] processor;

[0125] Memory used to store processor-executable instructions;

[0126] The processor is used to implement the steps described in the scene reconstruction method.

[0127] In one embodiment, the image acquisition device includes:

[0128] A camera assembly, comprising a camera base and a plurality of cameras, wherein the camera base is hemispherical and the plurality of cameras are evenly arranged on a protruding side of the camera base;

[0129] A bracket, which is fixedly connected to the camera assembly;

[0130] A movable base, wherein the bracket is fixedly connected to the movable base.

[0131] In some embodiments, the structure of the image acquisition device can be seen from [reference needed]. Figure 7 , Figure 7 This is a schematic diagram of an image acquisition device provided in this disclosure. The camera assembly 71 may include multiple high-definition action cameras 711 and a hemispherical camera base 712. The action cameras 711 are evenly mounted on one side of the protruding portion of the camera base 712 via a mortise and tenon structure, forming a... Figure 8 The installation diagram is shown below. For example, camera assembly 71 may include 46 action cameras, with an average distance of 18cm between any two cameras. These 46 action cameras can be evenly mounted on a transparent camera base. During installation, holes can be drilled in the camera base 712 according to the camera mounting positions. Then, a matching tenon-and-mortise structure bracket is designed for each action camera. Finally, adhesive is used to attach the brackets to the holes in the camera base 712, allowing the cameras to capture images of the target scene from the camera base 712. The installed camera assembly 71 has a field of view exceeding 220°, enabling multi-directional image acquisition of the target scene.

[0132] Furthermore, to enable the camera assembly 71 to dynamically acquire images of the target scene, it can be mounted on a movable base 63 via a bracket 62. By controlling the movable base 63 to move according to guide markers, the camera assembly 71 can be controlled to acquire images from multiple angles within the target scene. For example, based on the characteristics of the target scene, the movable base 63 can be moved using tile lines or floor lines as guide markers, and the camera assembly 71 can be controlled to acquire an image every 30cm. An object-oriented approach can be used for image acquisition. In specific scenes, to further enhance the user's immersive experience, 360° shooting was performed, including both front and rear perspectives.

[0133] The image datasets acquired by the image acquisition device have an average acquisition density of 134 viewpoints per cubic meter in each target scene, achieving a resolution of 5K in each scene, including details such as lighting changes and glass reflections. It can also capture target scene spaces ranging from 6 to 39 cubic meters in volume, providing considerable exploration space for six-degrees-of-freedom VR experiences.

[0134] In some embodiments, when acquiring images of the target scene using an image acquisition device, the multiple motion cameras in the camera assembly 71 can be initialized first, the status of each motion camera can be queried, and the camera shooting parameters and shooting modes can be set uniformly. During the shooting process, a synchronous acquisition command is issued to multiple cameras to enable the multiple cameras in the camera assembly 71 to acquire images synchronously. Furthermore, the working status of each camera is monitored during the shooting process to prevent image data loss due to camera offline. Upon completion of shooting, a synchronous stop command is issued to multiple cameras to control the camera assembly 71 to stop acquiring images. Finally, the images and / or video data acquired by the camera assembly 71 are aggregated to obtain the original images for reconstructing the target scene.

[0135] The scene reconstruction system can quickly, efficiently, and comprehensively capture information about the target scene, enriching the density and accuracy of image data in the target scene, which is beneficial to improving the accuracy of target scene reconstruction and rendering precision. The image acquisition device provided in this disclosure embodiment can quickly construct high-resolution, high-density image data of large indoor and outdoor scenes, and can acquire video sequences of dynamic scenes over long periods of time.

[0136] Corresponding to the embodiments of the aforementioned scene reconstruction method, this disclosure also provides embodiments of a scene reconstruction apparatus.

[0137] Please refer to Figure 9 , Figure 9 This is a block diagram of a scene reconstruction apparatus according to an exemplary embodiment of the present disclosure, the apparatus comprising:

[0138] The first determining unit 901 is used to determine the internal parameters and pose information of each camera in the image acquisition device based on the image of the target scene acquired by the image acquisition device, wherein the image acquisition device includes multiple cameras;

[0139] The second determining unit 902 is used to determine the point cloud data corresponding to the image based on the internal parameters and pose information of the camera corresponding to each image;

[0140] The third determining unit 903 is used to determine supplementary data corresponding to the point cloud data based on the pose information of the camera corresponding to each image, wherein the supplementary data is used to supplement the point cloud data.

[0141] The reconstruction unit 904 is used to reconstruct the target scene based on the point cloud data and the supplementary data using an improved three-dimensional Gaussian splashing algorithm.

[0142] In some embodiments, the first determining unit 901 is configured to:

[0143] Extract two-dimensional feature points from each image of the target scene and the intrinsic parameters of each camera;

[0144] Match the two-dimensional feature points of each image with the three-dimensional feature points of the target scene;

[0145] Based on the two-dimensional and three-dimensional feature points matched in each image, and the internal parameters of the camera corresponding to each image, the pose information of the camera corresponding to each image is calculated.

[0146] In some embodiments, the third determining unit 903 is configured to:

[0147] Based on the pose information of the camera corresponding to each image, the central camera in the image acquisition device is determined;

[0148] Based on the image captured by the central camera, at least one target camera in the image acquisition unit is determined, and the image captured by the target camera has a field of view overlap with the image captured by the central camera.

[0149] Based on the pose information of the central camera and the pose information of the target cameras, the farthest camera that is farthest from the central camera is determined among the target cameras.

[0150] The supplementary data is determined based on the pose information of the central camera and the furthest camera.

[0151] In some embodiments, the third determining unit 903 is configured to:

[0152] Based on the pose information of the central camera and the farthest camera, determine the target distance between the central camera and the zero parallax region in the target scene;

[0153] Construct a target sphere with the target distance as its radius;

[0154] A certain amount of point cloud is set on the surface of the target sphere and assigned at least one random color to obtain supplementary data for the three-dimensional Gaussian splashing algorithm.

[0155] In some embodiments, the reconstruction unit 904 is configured to:

[0156] Based on the point cloud data and the supplementary data, the target scene is rendered using a 3D Gaussian splashing algorithm to obtain an initial rendering result;

[0157] Based on the pixel information in the initial rendering result and the real pixel information of the target scene, the parameters of the three-dimensional Gaussian splashing algorithm are adjusted by a preset loss function to obtain the adjusted three-dimensional Gaussian splashing algorithm.

[0158] The target scene is reconstructed using the adjusted 3D Gaussian splashing algorithm.

[0159] In some embodiments, the reconstruction unit 904 is configured to:

[0160] Based on the point cloud data and the supplementary data, the target scene is rendered using a three-dimensional Gaussian splashing algorithm to obtain a first rendering result;

[0161] Based on the color transformation parameters, the pixels in the first rendering result are color-converted to obtain the initial rendering result.

[0162] In some embodiments, the preset loss function includes:

[0163]

[0164]

[0165]

[0166] in, , , Here, u and v are the weight parameters, representing the coordinates of pixels in the target rendering result. Represents the i-th image. Represents the gradient operator. For weight parameters, GT represents the color of a pixel in the target rendering result, while GT represents the pixel's true color. For the 3D Gaussian splash algorithm, the regularization term for the elliptical shape is... This is the regular expression for pixel color.

[0167] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0168] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0169] Embodiments of the scene reconstruction device disclosed herein can be applied to electronic devices. The device embodiments can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of the electronic device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 10 The diagram shown is a hardware structure diagram of the electronic device containing the scene reconstruction device of this disclosure, except... Figure 10 In addition to the processor, memory, network interface, and non-volatile memory shown, the electronic device in which the device is located in the embodiment may also include other hardware depending on the actual function of the electronic device, which will not be described in detail here.

[0170] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0171] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A scene reconstruction method, characterized by, The method comprises: determining the internal parameters and pose information of each camera in the image collector according to images of a target scene collected by the image collector, wherein the image collector comprises a plurality of cameras; determining point cloud data corresponding to each image based on the internal parameters and pose information of the corresponding camera of each image; determining supplementary data corresponding to the point cloud data based on the pose information of the corresponding camera of each image, wherein the supplementary data is used to supplement the point cloud data; reconstructing the target scene based on the point cloud data and the supplementary data by using an improved three-dimensional Gaussian splatting algorithm; determining the supplementary data corresponding to the point cloud data based on the pose information of the corresponding camera of each image, comprising: determining a center camera in the image collector based on the pose information of the corresponding camera of each image; determining at least one target camera in the image collector according to an image captured by the center camera, wherein the image captured by the target camera has a field of view overlap region with the image captured by the center camera; determining a farthest camera in the target camera farthest from the center camera based on the pose information of the center camera and the pose information of the target camera; determining the supplementary data according to the pose information of the center camera and the farthest camera; determining the supplementary data according to the pose information of the center camera and the farthest camera, comprising: determining a target distance between the center camera and a zero-parallax region in the target scene according to the pose information of the center camera and the farthest camera; constructing a target sphere with the target distance as the radius; setting at least one random color on the surface of the target sphere to obtain the supplementary data of the three-dimensional Gaussian splatting algorithm.

2. The method of claim 1, wherein, determining the internal parameters and pose information of each camera in the image collector according to images of a target scene collected by the image collector, comprising: extracting two-dimensional feature points of each image in the images of the target scene and internal parameters of each camera; matching the two-dimensional feature points of each image with three-dimensional feature points of the target scene; calculating the pose information of the corresponding camera of each image based on the matched two-dimensional feature points and three-dimensional feature points in each image and the internal parameters of the corresponding camera of each image.

3. The method of claim 1, wherein, reconstructing the target scene based on the point cloud data and the supplementary data by using an improved three-dimensional Gaussian splatting algorithm, comprising: rendering the target scene based on the point cloud data and the supplementary data by using a three-dimensional Gaussian splatting algorithm to obtain an initial rendering result; adjusting parameters of the three-dimensional Gaussian splatting algorithm by using a preset loss function according to pixel information in the initial rendering result and real pixel information of the target scene to obtain an adjusted three-dimensional Gaussian splatting algorithm; reconstructing the target scene by using the adjusted three-dimensional Gaussian splatting algorithm.

4. The method of claim 3, wherein, rendering the target scene based on the point cloud data and the supplementary data by using a three-dimensional Gaussian splatting algorithm to obtain an initial rendering result, comprising: According to the point cloud data and the supplementary data, the target scene is rendered by a three-dimensional Gaussian splatting algorithm to obtain a first rendering result; According to the color transformation parameter, the pixels in the first rendering result are color-converted to obtain the initial rendering result.

5. The method of claim 3, wherein, The preset loss function comprises: wherein Loss represents a loss value, , , is a weight parameter, and u, v represent coordinates of a pixel in a target rendering result, represents the ith image, represents a gradient operator, is a weight parameter, represents a color of a pixel in a target rendering result, and GT represents a true color of the pixel, represents a photometric loss term, is a regular term of an elliptical shape in a three-dimensional Gaussian splatting algorithm, represents a structural loss term, is a regular term of a pixel color.

6. A scene reconstruction system, characterized by The system comprises: An image collector configured to collect an image of a target scene; A processor; A memory configured to store processor-executable instructions; The processor is configured to implement the steps of any one of the methods of claims 1-5.

7. The system of claim 6, wherein, The image collector comprises: A camera assembly comprising a camera base and a plurality of cameras, the camera base being hemispherical, and the plurality of cameras being uniformly arranged on one side of the camera base; A support fixedly connected with the camera assembly; A movable base fixedly connected with the support.

8. A scene reconstruction apparatus, characterized by comprising: The device comprises: A first determination unit configured to determine internal parameters and pose information of each camera in an image collector according to an image of a target scene collected by the image collector, wherein the image collector comprises a plurality of cameras; A second determination unit configured to determine point cloud data corresponding to each image based on the internal parameters and the pose information of the corresponding camera; A third determination unit configured to determine supplementary data corresponding to the point cloud data based on the pose information of the corresponding camera of each image, wherein the supplementary data is used to supplement the point cloud data; A reconstruction unit configured to perform scene reconstruction on the target scene based on the point cloud data and the supplementary data by an improved three-dimensional Gaussian splatting algorithm; The third determination unit is configured to: Determine a center camera in the image collector based on the pose information of the corresponding camera of each image; Determine at least one target camera in the image collector based on an image captured by the center camera, wherein the image captured by the target camera has a field of view overlap area with the image captured by the center camera; Determine a farthest camera in the target camera farthest from the center camera based on the pose information of the center camera and the pose information of the target camera; Determine the supplementary data based on the pose information of the center camera and the pose information of the farthest camera; The third determination unit is further configured to: Determine a target distance of the center camera to a zero-parallax area in the target scene based on the pose information of the center camera and the pose information of the farthest camera; Construct a target sphere with the target distance as a radius; Set a certain amount of point clouds on the surface of the target sphere and assign at least one random color to obtain supplementary data of the three-dimensional Gaussian splatting algorithm.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program, when executed by a processor, implements the steps of any one of the methods of claims 1-5.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor, when executing the program, implements the steps of any one of the methods of claims 1-5.

Images