Object-level data acquisition and reconstruction method based on three-dimensional gaussian splashing

Abstract

This invention provides a method for object-level data acquisition and reconstruction based on 3D Gaussian splashing, relating to the field of computer 3D modeling. In this invention, firstly, a multi-view image sequence of the target object is synchronously acquired from multiple preset heights, and this multi-view image sequence is limited to multiple images of a single background acquired at intervals during one rotation of the target object. Secondly, one image is randomly selected for bounding box annotation, and then the annotation results are propagated across viewpoints to obtain bounding box annotations for all images, thereby segmenting and obtaining the corresponding mask image. Then, based on all images and their mask images, the camera pose is estimated to construct an initial 3D point cloud. Finally, starting from the initial 3D point cloud, the Gaussian sphere parameters are iteratively optimized to obtain the 3D reconstruction result of the target object. This method generates high-fidelity, clean object-level 3D data through comprehensive acquisition and background optimization processing, significantly reducing reliance on manual annotation, while omitting the voxel clipping step to reduce computational complexity.

Description

Technical Field

[0001] This invention relates to the field of computer 3D modeling, specifically to a method for object-level data acquisition and reconstruction based on 3D Gaussian splashing. Background Technology

[0002] As an important research direction in computer vision and graphics, 3D Gaussian splashing technology has the advantages of fast rendering, low modeling cost, and ultra-high visual fidelity compared to traditional explicit 3D representation.

[0003] In related technologies, purely visual (image or video-based) 3D reconstruction methods typically require capturing images of the target object from multiple angles to obtain sufficient viewpoint coverage. For example, patent application CN119784959A discloses a fast object reconstruction method based on 3D Gaussian sputtering segmentation, comprising: acquiring images of the target object's appearance, obtaining image quality and 3D reconstruction information, and selecting and optimizing the image set based on image quality; converting sparse point clouds into voxels, cropping and optimizing the image set to obtain 2D regions for each viewpoint, using camera pose and 2D regions to crop voxels to obtain target voxels; rendering target voxels as depth maps, generating sampling points based on the depth maps, segmenting and optimizing the image set using the sampling points to obtain target object masks, and cropping to obtain segmentation results for each viewpoint; projecting the segmentation results back to the initial viewpoint, generating depth maps, comparing them with the target viewpoint masks to obtain consistency evaluation values, iterating the segmentation results based on the evaluation values ​​to obtain a set of target object masks; and reconstructing a 3D Gaussian sputtering model based on the cropped voxels and Gaussian points.

[0004] However, the above schemes rely on voxel clipping and geometric constraints for post-processing, which incurs high computational costs and cannot eliminate background noise. Summary of the Invention

[0005] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides a method for object-level data acquisition and reconstruction based on three-dimensional Gaussian splashing, which solves the technical problems of high computational cost and inability to completely eliminate background noise.

[0006] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: A method for object-level data acquisition and reconstruction based on 3D Gaussian splashing includes: A multi-view image sequence of a target object is acquired synchronously from multiple preset heights; wherein, the multi-view image sequence includes multiple images of a single background acquired at intervals during one rotation of the target object; Based on multi-view image sequences at various altitudes, an initial 3D point cloud is constructed, including: Select at least one image randomly from all images and receive bounding boxes labeled with the target objects in the selected image; The annotation results are propagated across viewpoints to obtain bounding box annotations for all images, and the corresponding mask image for each image is obtained through segmentation. The camera pose is estimated based on all images and their mask maps to construct an initial 3D point cloud; The Gaussian sphere parameters are iteratively optimized starting from the initial 3D point cloud to obtain the 3D reconstruction results of the target object.

[0007] Preferably, a multi-view image sequence is acquired using an imaging device, which includes a support, a magic hand, a camera, a turntable, and a control device. One end of each magic hand is connected to the corresponding height of the bracket, and the other end is connected to the corresponding camera; Each camera is aligned with the axis of rotation of the turntable; The turntable provides a platform for carrying the target object; The control device is used to control the rotation of the turntable and, during the rotation, to control each camera to synchronously acquire an image of a single background of the target object at the corresponding height.

[0008] Preferably, the step of simultaneously acquiring multi-view image sequences of the target object from multiple preset heights includes: Place the target object in the center of the turntable; Repeat the following steps to acquire images: The turntable is activated and rotated to a preset angle via a control signal; and each camera is controlled to synchronously acquire images of the target object at the corresponding height via a camera synchronization signal. If the cumulative rotation angle of the target object is greater than or equal to 360 degrees, output a multi-view image sequence at different heights.

[0009] Preferably, in the process of iteratively optimizing the Gaussian sphere parameters, a mask loss function and a cumulative opacity loss are designed as optimization objectives; wherein, the mask loss function includes a 1-norm loss based on the mask image and a structural similarity loss.

[0010] Preferably, the 1-norm loss is expressed as: Where J represents the height, M represents the number of viewpoints divided around the target object, and L1 represents the mean absolute error value. For the selected j-th height and m-th viewpoint θ m Images captured below, for The mask image, Rendered for a Gaussian sphere and with Aligned images, It is the matrix dot product.

[0011] Preferably, the structural similarity loss is expressed as: Where J represents the height, and M represents the number of viewpoints divided around the target object. SSIM is a differentiable structural similarity index, where the subscript D indicates differentiability and SSIM represents structural similarity. For the selected j-th height and m-th viewpoint θ m Images captured below, for The mask image, The result of rendering a Gaussian sphere with The corresponding image, It is the matrix dot product.

[0012] Preferably, the cumulative opacity loss is expressed as: Where p is the pixel position index in the rendered image, mask(p)=0 represents all pixel positions in the background region of the rendered image, and α∈[0,1] is the opacity of the Gaussian sphere.

[0013] An object-level data acquisition and reconstruction system based on 3D Gaussian splashing includes: The acquisition module is used to synchronously acquire multi-view image sequences of a target object from multiple preset heights; wherein, the multi-view image sequence includes multiple images of a single background acquired at intervals during one rotation of the target object; The initialization module is used to construct an initial 3D point cloud based on multi-view image sequences at various heights, including: The annotation unit is used to randomly select at least one image from all images and receive bounding annotations for target objects in the selected image; The propagation and segmentation unit is used to propagate the annotation results across viewpoints, obtain bounding box annotations for all images, and segment to obtain the mask image corresponding to each image. The building unit is used to estimate the camera pose based on all images and their mask maps to construct an initial 3D point cloud; The reconstruction module is used to iteratively optimize the Gaussian sphere parameters starting from the initial 3D point cloud to obtain the 3D reconstruction result of the target object.

[0014] A storage medium storing a computer program that causes a computer to execute the object-level data acquisition and reconstruction system method as described above.

[0015] An electronic device, comprising: One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing object-level data acquisition and reconstruction systems as described above.

[0016] (III) Beneficial Effects This invention provides a method for object-level data acquisition and reconstruction based on three-dimensional Gaussian splashing. Compared with existing technologies, it has the following advantages: In this invention, firstly, a multi-view image sequence of the target object is synchronously acquired from multiple preset heights, and this multi-view image sequence is limited to multiple images of a single background acquired at intervals during one rotation of the target object. Secondly, an initial 3D point cloud is constructed, including: at least one image is randomly selected for bounding box annotation, and then the annotation results are propagated across viewpoints to obtain bounding box annotations for all images, so as to segment and obtain the corresponding mask image. Then, the camera pose is estimated based on all images and their mask images to construct the initial 3D point cloud. Finally, the Gaussian sphere parameters are iteratively optimized starting from the initial 3D point cloud to obtain the 3D reconstruction result of the target object. This method generates high-fidelity, clean object-level 3D data through comprehensive acquisition and background optimization processing, significantly reducing the dependence on manual annotation, while omitting the voxel clipping step to reduce computational complexity. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating an object-level data acquisition and reconstruction method based on three-dimensional Gaussian splashing, provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a shooting device provided in an embodiment of the present invention; Figure 3 A flowchart for obtaining a mask image of all images is provided in an embodiment of the present invention; Figure 4 (a) to (b) in the figure are comparison images of the reconstruction results using a moving camera and a changing background combination versus using a fixed camera and a single background combination; Figure 5 (a) to (c) in the figure are comparison images of the reconstruction effect using a scaffold and a solid color background with a turntable, using a solid color background with a turntable but without a scaffold, and using a scaffold and a solid color background but without a turntable. Figure 6 (a) to (b) in the figure are comparison charts of reconstruction effects with and without cumulative opacity loss.

[0019] Explanation of reference numerals in the attached figures: 1-Turntable, 2-Camera, 3-Magic Hand, 4-Claw Clamp, 5-Stand, 6-Background Wall, 7-Control Device. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] This application provides a method for object-level data acquisition and reconstruction based on three-dimensional Gaussian splashing, which solves the technical problems of high computational cost and inability to completely eliminate background noise. The overall idea is as follows: 1. Achieve efficient acquisition of object-level data. By designing a specialized acquisition process and equipment, the question of "how many angles should be captured" is quantified into the rotation angle of a turntable, preventing repeated or missed angles during the shooting process, ensuring all-round visual coverage of the target object, and reducing reconstruction defects caused by missed shooting angles and significant changes in accuracy.

[0022] 2. Automatic preprocessing removes irrelevant background data. Combined with intelligent video tracking algorithms, data preprocessing effectively distinguishes target objects from background information during the acquisition or reconstruction phase, reducing the impact of irrelevant backgrounds on model quality. Unlike traditional manual camera movement shooting methods, this approach easily eliminates backgrounds from all angles due to the fixed, single background.

[0023] 3. A loss function based on mask image and opacity constraint is adopted, where the opacity loss constraint is used to remove Gaussian sphere noise in non-target areas.

[0024] 4. Improve the integrity and accuracy of 3D models. Through comprehensive data acquisition and background optimization, generate high-fidelity, clean object-level 3D data to provide high-quality basic data for subsequent analysis, display, or application.

[0025] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0026] Example 1: like Figure 1As shown, this embodiment of the invention provides a method for object-level data acquisition and reconstruction based on three-dimensional Gaussian splashing, including: S1. Synchronously acquire multi-view image sequences of the target object from multiple preset heights; wherein, the multi-view image sequence includes multiple images of a single background acquired at intervals during the target object's rotation. S2. Based on multi-view image sequences at various heights, construct an initial 3D point cloud, including: S21. Randomly select at least one image from all images and receive bounding box annotations for the target objects in the selected image; S22. Perform cross-view propagation on the annotation results to obtain bounding box annotations for all images, and then segment to obtain the mask image corresponding to each image. S23. Estimate the camera pose based on all images and their mask images to construct an initial 3D point cloud; S3. Iteratively optimize the Gaussian sphere parameters starting from the initial 3D point cloud to obtain the 3D reconstruction results of the target object.

[0027] The embodiments of the present invention generate high-fidelity, clean object-level 3D data through comprehensive acquisition and background optimization processing, significantly reducing the reliance on manual annotation, while omitting the voxel clipping step to reduce computational complexity.

[0028] The following will detail each step of the above solution: In step S1, a multi-view image sequence of the target object is acquired synchronously from multiple preset heights.

[0029] First, it should be noted that, in contrast to the traditional method of "camera moving around the object while the object remains stationary," the embodiments of the present invention adopt a "camera fixed while the object rotates" acquisition method to satisfy the requirement that the multi-view image sequence includes multiple images of a single background acquired at intervals during the process of the target object rotating once.

[0030] For example, in this embodiment of the invention, a multi-view image sequence is acquired through a preset shooting device, which includes a bracket, a magic hand, a camera, a turntable, and a control device. One end of each magic hand is connected to the corresponding height of the support, and the other end is connected to the corresponding camera; each camera is aligned with the rotation axis of the turntable; the turntable provides a platform to support the target object; the control device is used to control the rotation of the turntable and, during the rotation, control each camera to synchronously acquire an image of a single background of the target object at the corresponding height.

[0031] It should be noted that, according to the structural complexity of the target object, the embodiments of the present invention can freely set a number of cameras at different heights and image acquisition frequencies, and no absolute limitation is made here.

[0032] For example, the target object can be photographed from three different heights—top, middle, and bottom—to ensure that images of the target object at different angles can cover its various perspectives, thus including structural details such as the top, front, and bottom of the target object in the images; or, the control device can control the turntable to rotate 10 or 20 degrees, simultaneously controlling each camera to capture images from the corresponding perspective.

[0033] Continuing with the example of the shooting device described above, this step involves simultaneously acquiring a multi-view image sequence of the target object from multiple preset heights, specifically including: S11. Place the target object at the center of the turntable; S12. Repeat the following operations to acquire images: The turntable is activated and rotated to a preset angle via a control signal; and each camera is controlled to synchronously acquire images of the target object at the corresponding height via a camera synchronization signal. S13. If the cumulative rotation angle of the target object is greater than or equal to 360 degrees, output a multi-view image sequence at different heights.

[0034] Understandably, the embodiments of the present invention achieve efficient acquisition of object-level data. By designing specialized acquisition hardware and processes, the question of "how many angles should be captured" is quantified into the rotation angle of a turntable, preventing repeated or missed angles during the shooting process, ensuring all-round visual coverage of the target object, and reducing reconstruction defects caused by missed shooting angles and significant changes in accuracy.

[0035] Specifically, taking the demonstration of a magic hand as an example, such as... Figure 2 As shown, this embodiment of the invention provides a shooting device that can be used in the above-described acquisition process, specifically including a bracket 1, a magic hand 2, a gripper 3, a camera 4, a turntable 5, a background wall 6, and a control device 7; wherein: Bracket 1, fixed to a desktop or the ground, is used to provide a fixed mounting base.

[0036] Magic Hand 2 uses a hinged linkage to connect to Gripper 3. Because the hinged linkage is omnidirectional, the angle of Magic Hand 2 can be adjusted arbitrarily according to actual conditions.

[0037] Two grippers 3 are connected to both ends of the magic hand 2 for fixed connection with the bracket 1 and the camera 4, respectively.

[0038] Camera 4 is fixedly connected to magic hand 2 via gripper 3 located away from bracket 1. By adjusting the positions of magic hand 2 and gripper 3, images of objects can be acquired from a specific perspective. Optionally, camera 4 can be an industrial camera, a consumer-grade camera, or a depth camera (such as RealSense), which is not limited here.

[0039] Turntable 5, whose upper surface is used to support the target object, is located between support 1 and background wall 6 to ensure that the target object can be covered by the camera 4 from all directions during continuous rotation around the central axis.

[0040] Background wall 6, as the shooting background, can be a solid color, simple pattern or other single background wall (such as a white or gray wall) to help with the automatic segmentation processing of the image in post-processing.

[0041] The control device 7 is connected to the camera 4 and the turntable 5 via wired or wireless means. It is mainly used to provide two functions: first, to control the rotation of the turntable, that is, to control the rotation of the turntable 5 to a fixed angle; second, to act as a camera synchronization generator, to send a synchronization signal to control the camera 4 to synchronously acquire images.

[0042] In step S2, an initial 3D point cloud is constructed based on the multi-view image sequence at various heights, including: S21. Randomly select at least one image from all images and receive bounding boxes labeled with respect to the target objects in the selected image.

[0043] First, it should be noted that under normal circumstances, only one image can be randomly selected for bounding box annotation. However, in a few extreme cases (such as when the top and bottom structures of the target object are very different), in order to avoid distortion in subsequent reconstruction, 1 to 2 additional images can be annotated. But this is not the usual procedure.

[0044] For example, such as Figure 3 As shown, taking only a randomly selected image as an example, the bounding box of the target object in a single viewpoint is given by manually annotating it.

[0045] S22. Perform cross-view propagation on the annotation results to obtain bounding box annotations for all images, and then segment to obtain the mask image corresponding to each image.

[0046] Continuing with the example above, such as Figure 3 As shown, the annotation results are propagated across viewpoints through the detection box propagation model. After the detection boxes are automatically propagated, the bounding box annotations of all images are obtained.

[0047] It should be noted that the detection box propagation model in the embodiments of the present invention can be a large-scale visual model such as DINO (Self-Supervised Visual Transformer model), a video tracking algorithm such as DeepSORT (Deep Correlation Multi-Object Tracking Algorithm) model, or a three-dimensional image matching algorithm such as LoFTR (Local Feature Transformer Matcher) model, etc., and no absolute limitation is made here.

[0048] Continuing with the example above, such as Figure 3 As shown, after obtaining the bounding box annotations of all images, the (binary) mask of the selected image is obtained through the segmentation model.

[0049] It should be noted that the segmentation model in the embodiments of the present invention can be ViT (Visual Transformer model), SAM2 (second generation general image segmentation model), etc., and no absolute limitation is made here.

[0050] S23. Estimate the camera pose based on all images and their mask maps to construct an initial 3D point cloud.

[0051] For example, in embodiments of the present invention, all images and their mask images are used as input to a 3D reconstruction system to estimate camera pose, thereby completing the construction of an initial 3D point cloud.

[0052] It should be noted that the three-dimensional reconstruction system in this embodiment of the invention can be COLMAP (multi-view) Figure 3 Examples of systems include 3D reconstruction systems and VGGT (Visual Geometry Fundamentals Transformer), but no absolute restrictions are imposed here.

[0053] Understandably, through the above-mentioned automatic generation and propagation of object masks, this embodiment of the invention achieves the retention of only the relevant data of the target object, avoiding the background being reconstructed into the 3D model. The entire process is automated, avoiding manual image cutout, and greatly reducing labor and time costs.

[0054] In step S3, the Gaussian sphere parameters are iteratively optimized starting from the initial 3D point cloud to obtain the 3D reconstruction result of the target object.

[0055] In an optional implementation, to effectively remove 3D Gaussian noise from the background region during reconstruction, this embodiment of the invention designs a mask loss function and a cumulative opacity loss as optimization objectives during iterative optimization of the Gaussian sphere parameters. Specifically, the mask loss function includes a 1-norm loss based on the mask image and a structural similarity loss.

[0056] Understandably, the 1-norm loss is mainly used to constrain the accurate color reconstruction of the target region; the structural similarity loss is mainly used to maintain the authenticity of the local structure of the target object; and the cumulative opacity loss is mainly used to automatically suppress invalid Gaussian balls in the background region, replacing manual thresholding.

[0057] For example, the mask loss function and the cumulative opacity loss are expressed as follows: Where L is the total loss, These are the 1-norm loss and structural similarity loss based on the mask image, respectively. α λ1 represents the cumulative opacity loss; λ2 and λ3 represent the weighting coefficients of the structural similarity loss based on the mask image and the cumulative opacity loss, respectively.

[0058] Specifically: 1) The formula for calculating the 1-norm loss is as follows: Where J is the height, M is the number of viewpoints divided around the target object, and L1 is the mean absolute error value, which is used to reflect the pixel signal difference between the input image signal and the reconstructed image signal. For the selected j-th height and m-th viewpoint θ m Images captured below, for The mask image, Rendered for a Gaussian sphere and with Aligned images; For matrix dot product, the dot product of the mask image and the color image is used here before calculating the relevant metrics. This calculation shows the model that only the loss of the target object region is of concern, while the loss of pixels in non-target regions is completely ignored.

[0059] 2) The formula for calculating structural similarity loss is as follows: in, The SSIM (structural similarity) index is a differentiable index used to reflect the structural similarity of images, specifically measuring the consistency of images at a specific scale; the subscript D indicates differentiability, and SSIM indicates structural similarity.

[0060] 3) The formula for calculating the cumulative opacity loss is as follows: Among them, L α The value represents the overall opacity of all points along a certain rendering direction; p is the pixel position index in the rendered image, and mask(p)=0 represents all pixel positions in the background area (non-target object area) of the rendered image; α∈[0,1] is the opacity of the Gaussian sphere, the closer the value is to 1, the more opaque it is, and vice versa.

[0061] It is understood that embodiments of the present invention use L1 and L2 based on mask images. D-SSIM and cumulative transparency loss L α This invention employs a hard partitioning method to separately divide the loss calculation regions for the target object region and the background region (non-target object region), achieving region-specific optimization and improving modeling accuracy. Furthermore, unlike traditional methods that require manually setting transparency thresholds to remove low-contribution Gaussian spheres, this embodiment automatically filters effective Gaussian spheres through backpropagation of the cumulative transparency loss gradient, significantly reducing manual intervention.

[0062] To help better understand the advantages of the method provided in the embodiments of the present invention, the following comparative experiments are provided: Figure 4 (a) to (b) in the figure show a comparison of the reconstruction results using a moving camera with varying backgrounds versus using a fixed camera with a single background. Figure 4 (a) shows a moving camera shot. The uncertain background introduces various noises to mask propagation and segmentation, which can lead to increased segmentation difficulty, loss of small targets (such as bamboo in the panda's hand), gaps, and low-quality segmentation edges. Figure 4 The aforementioned problems can be avoided by using a fixed camera and a single background in (b) of the presentation.

[0063] Figure 5 Images (a) through (c) show a comparison of the reconstruction results using a scaffold and a solid background with a turntable, using a solid background without a scaffold with a turntable, and using a scaffold and a solid background without a turntable (replaced by manually rotating the target object). It can be observed that, as... Figure 5 As shown in (b) to (c), the combination of the latter two results in obvious distortion and deformation in the reconstruction of small target objects, proving the necessity and effectiveness of the acquisition device design.

[0064] Figure 6 In the figures (a) to (b), the reconstruction results are compared with and without cumulative opacity loss. It can be observed that... Figure 6 The background region in (b) shows obvious black reconstruction noise, proving that the opacity loss effectively suppresses the reconstruction noise in the background region (non-target region).

[0065] Example 2: This invention provides an object-level data acquisition and reconstruction system based on three-dimensional Gaussian splashing, comprising: The acquisition module is used to synchronously acquire multi-view image sequences of a target object from multiple preset heights; wherein, the multi-view image sequence includes multiple images of a single background acquired at intervals during one rotation of the target object; The initialization module is used to construct an initial 3D point cloud based on multi-view image sequences at various heights, including: The annotation unit is used to randomly select at least one image from all images and receive bounding annotations for target objects in the selected image; The propagation and segmentation unit is used to propagate the annotation results across viewpoints, obtain bounding box annotations for all images, and segment to obtain the mask image corresponding to each image. The building unit is used to estimate the camera pose based on all images and their mask maps to construct an initial 3D point cloud; The reconstruction module is used to iteratively optimize the Gaussian sphere parameters starting from the initial 3D point cloud to obtain the 3D reconstruction result of the target object.

[0066] Example 3: This invention provides a storage medium storing a computer program that causes a computer to execute the object-level data acquisition and reconstruction system method as described in Embodiment 1.

[0067] Example 4: This invention provides an electronic device, comprising: One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing object-level data acquisition and reconstruction system methods as described in Embodiment 1.

[0068] It is understood that the object-level data acquisition and reconstruction system, storage medium and electronic device based on three-dimensional Gaussian splashing provided in the embodiments of the present invention correspond to the object-level data acquisition and reconstruction method based on three-dimensional Gaussian splashing provided in the embodiments of the present invention. The explanation, examples and beneficial effects of the relevant contents can be referred to the corresponding parts of the method, and will not be repeated here.

[0069] In summary, compared with existing technologies, it has the following beneficial effects: 1. This embodiment generates high-fidelity, clean object-level 3D data through comprehensive data acquisition and background optimization processing, significantly reducing the reliance on manual annotation, while omitting the voxel clipping step to reduce computational complexity.

[0070] 2. The embodiments of the present invention achieve efficient acquisition of object-level data. By designing specialized acquisition hardware and processes, the question of "how many angles should be captured" is quantified into the rotation angle of the turntable, preventing repeated or missed angles during the shooting process, ensuring all-round visual coverage of the target object, and reducing reconstruction defects caused by missed shooting angles and significant changes in accuracy.

[0071] 3. This invention employs L1 and D-SSIM based on mask images, along with cumulative transparency loss, to rigidly divide the loss calculation regions for the target object region and the background region (non-target object region), achieving region-specific optimization and improving modeling accuracy. Furthermore, unlike traditional methods that require manually setting transparency thresholds to remove low-contribution Gaussian spheres, this invention automatically filters effective Gaussian spheres through gradient backpropagation of the cumulative transparency loss, significantly reducing manual intervention.

[0072] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0073] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for object-level data acquisition and reconstruction based on three-dimensional Gaussian splashing, characterized in that, include: A multi-view image sequence of a target object is acquired synchronously from multiple preset heights; wherein, the multi-view image sequence includes multiple images of a single background acquired at intervals during one rotation of the target object; Based on multi-view image sequences at various altitudes, an initial 3D point cloud is constructed, including: Select at least one image randomly from all images and receive bounding boxes labeled with the target objects in the selected image; The annotation results are propagated across viewpoints to obtain bounding box annotations for all images, and the corresponding mask image for each image is obtained through segmentation. The camera pose is estimated based on all images and their mask maps to construct an initial 3D point cloud; The Gaussian sphere parameters are iteratively optimized starting from the initial 3D point cloud to obtain the 3D reconstruction results of the target object.

2. The object-level data acquisition and reconstruction method as described in claim 1, characterized in that, A multi-view image sequence is acquired through an imaging device, which includes a bracket, a magic hand, a camera, a turntable, and a control device. One end of each magic hand is connected to the corresponding height of the bracket, and the other end is connected to the corresponding camera; Each camera is aligned with the axis of rotation of the turntable; The turntable provides a platform for carrying the target object; The control device is used to control the rotation of the turntable and, during the rotation, to control each camera to synchronously acquire an image of a single background of the target object at the corresponding height.

3. The object-level data acquisition and reconstruction method as described in claim 2, characterized in that, The method of synchronously acquiring multi-view image sequences of a target object from multiple preset heights includes: Place the target object in the center of the turntable; Repeat the following steps to acquire images: The turntable is activated and rotated to a preset angle via a control signal; and each camera is controlled to synchronously acquire images of the target object at the corresponding height via a camera synchronization signal. If the cumulative rotation angle of the target object is greater than or equal to 360 degrees, output a multi-view image sequence at different heights.

4. The object-level data acquisition and reconstruction method as described in claim 1, characterized in that, In the process of iteratively optimizing the Gaussian sphere parameters, a mask loss function and a cumulative opacity loss are designed as optimization objectives; wherein, the mask loss function includes a 1-norm loss based on the mask image and a structural similarity loss.

5. The object-level data acquisition and reconstruction method as described in claim 4, characterized in that, The 1-norm loss is expressed as: Where J represents the height, M represents the number of viewpoints divided around the target object, and L1 represents the mean absolute error value. For the selected j-th height and m-th viewpoint θ m Images captured below, for The mask image, Rendered for a Gaussian sphere and with Aligned images, It is the matrix dot product.

6. The object-level data acquisition and reconstruction method as described in claim 4, characterized in that, The structural similarity loss is expressed as: Where J represents the height, and M represents the number of viewpoints divided around the target object. SSIM is a differentiable structural similarity index, where the subscript D indicates differentiability and SSIM represents structural similarity. For the selected j-th height and m-th viewpoint θ m Images captured below, for The mask image, The result of rendering a Gaussian sphere with The corresponding image, It is the matrix dot product.

7. The object-level data acquisition and reconstruction method as described in claim 4, characterized in that, The cumulative opacity loss is expressed as: Where p is the pixel position index in the rendered image, mask(p)=0 represents all pixel positions in the background region of the rendered image, and α∈[0,1] is the opacity of the Gaussian sphere.

8. A system for object-level data acquisition and reconstruction based on three-dimensional Gaussian splashing, characterized in that, include: The acquisition module is used to synchronously acquire multi-view image sequences of a target object from multiple preset heights; wherein, the multi-view image sequence includes multiple images of a single background acquired at intervals during one rotation of the target object; The initialization module is used to construct an initial 3D point cloud based on multi-view image sequences at various heights, including: The annotation unit is used to randomly select at least one image from all images and receive bounding annotations for target objects in the selected image; The propagation and segmentation unit is used to propagate the annotation results across viewpoints, obtain bounding box annotations for all images, and segment to obtain the mask image corresponding to each image. The building unit is used to estimate the camera pose based on all images and their mask maps to construct an initial 3D point cloud; The reconstruction module is used to iteratively optimize the Gaussian sphere parameters starting from the initial 3D point cloud to obtain the 3D reconstruction result of the target object.

9. A storage medium, characterized in that, It stores a computer program that causes a computer to perform the object-level data acquisition and reconstruction system method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: One or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the object-level data acquisition and reconstruction system method as described in any one of claims 1 to 7.

Images