A panoramic 3D Gaussian splash reconstruction method and apparatus for unmanned systems

Abstract

This application provides a panoramic 3D Gaussian splash reconstruction method and apparatus for unmanned systems. The method includes: acquiring a panoramic image sequence of the target scene at the current moment; processing the panoramic image sequence to obtain sparse point cloud data; processing the sparse point cloud data to obtain a 3D Gaussian scene containing multiple 3D Gaussian ellipsoids; in the compaction stage of the 3D Gaussian ellipsoids, using multiple 3D Gaussian ellipsoids to obtain a rendered image with an expanded boundary for each surround-view camera; determining the luminance error based on the luminance of each pixel of the expanded boundary and the luminance of the corresponding real image from a nearby viewpoint; updating the two-dimensional aggregated gradient of the pixels of the expanded boundary based on the luminance error; thereby determining the two-dimensional aggregated gradient of each 3D Gaussian ellipsoid; and determining whether to perform a compaction operation based on the two-dimensional aggregated gradient of each 3D Gaussian ellipsoid. This application can achieve high-fidelity seamless panoramic 3D scene reconstruction and rendering.

Description

Technical Field

[0001] This application relates to the field of three-dimensional Gaussian splash technology, and in particular to a panoramic three-dimensional Gaussian splash reconstruction method and apparatus for unmanned systems. Background Technology

[0002] Existing 3D Gaussian splashing techniques for panoramic images have the following technical problems: First, there is a lack of a self-calibration mechanism for the inherent pose noise of panoramic cameras. In real-world scenarios, panoramic cameras suffer from optical distortion and stitching errors, resulting in noise in the camera poses extracted by traditional structure-of-motion (SOR) algorithms. Panoramic images are often processed by equivalently projecting them into cube maps with a perspective view; however, existing 3DGS frameworks lack overall constraints on the six facets of the cube. This causes the cameras on each facet to drift during backpropagation, leading to topological tearing in the 3D scene.

[0003] Second, cube projection can cause field-of-view cutoffs and breaks at adjacent physical junctions. Existing methods mostly rely on pre-trained depth estimation models for processing, but at the edges of the field of view, depth estimation is usually accompanied by a lot of noise, which causes serious deviations in the reprojection of cross-plane pixels, resulting in distortion artifacts in the reconstructed 3D scene at the view boundaries.

[0004] Third, the cloning and splitting mechanism of 3DGS causes the Gaussian ellipsoid to be incorrectly stretched at the seams of the panoramic cube, which seriously reduces the scene reconstruction effect. Summary of the Invention

[0005] In view of this, this application provides a panoramic three-dimensional Gaussian splash reconstruction method and apparatus for unmanned systems to solve the above-mentioned technical problems.

[0006] Firstly, this application provides a panoramic 3D Gaussian splash reconstruction method for unmanned systems, including: Using six surround-view cameras of an unmanned system, a panoramic image sequence of the target scene at the current moment is acquired, the panoramic image sequence including real images from six perspectives; The panoramic image sequence is processed to obtain sparse point cloud data; the sparse point cloud data is processed to obtain a three-dimensional Gaussian scene containing multiple three-dimensional Gaussian ellipsoids. In the compaction stage of the 3D Gaussian ellipsoid, a rendered image with an expanded boundary is obtained for each panoramic camera using multiple 3D Gaussian ellipsoids; the luminance error is determined based on the luminance of each pixel of the expanded boundary and the luminance of the corresponding real image from the neighboring viewpoint; the 2D aggregated gradient of the pixels of the expanded boundary is updated based on the luminance error; the 2D aggregated gradient of each 3D Gaussian ellipsoid is determined using the updated 2D aggregated gradient of the pixels of the expanded boundary of each rendered image; and the 2D aggregated gradient of each 3D Gaussian ellipsoid is used to determine whether to perform a compaction operation.

[0007] In one possible implementation, during the compaction stage of the 3D Gaussian ellipsoids, a rendered image with an expanded boundary for each surround-view camera is obtained using multiple 3D Gaussian ellipsoids; a luminance error is determined based on the luminance of each pixel of the expanded boundary and the luminance of the corresponding real image from a nearby viewpoint; the 2D aggregated gradient of the pixels of the expanded boundary is updated based on the luminance error; the 2D aggregated gradient of each 3D Gaussian ellipsoid is determined using the updated 2D aggregated gradient of the pixels of the expanded boundary of each rendered image; and a compaction operation is determined based on the 2D aggregated gradient of each 3D Gaussian ellipsoid, including: Step S1: Calibrate the position and attitude of each surround view camera to obtain the calibrated position and attitude of each surround view camera; Step S2: Using the calibrated position and pose of each surround-view camera and multiple 3D Gaussian ellipsoids, generate a rendered image with an expanded boundary for each surround-view camera, and calculate the first 2D aggregate gradient of each rendered image with an expanded boundary. Step S3: Determine the photometric error based on the expanded boundary of each rendered image and the real image of the adjacent viewpoint; determine the edge compensation gradient based on the photometric error; Step S4: Determine the second two-dimensional aggregate gradient based on the first two-dimensional aggregate gradient and edge compensation gradient of the expansion boundary of each rendered image; Step S5: Determine the aggregated two-dimensional gradient of each three-dimensional Gaussian ellipsoid based on the second two-dimensional aggregated gradient of the expanded boundaries of all rendered images; Step S6: Sequentially determine whether the aggregated two-dimensional gradient of each three-dimensional Gaussian ellipsoid is greater than a preset threshold. If yes, perform a cloning or splitting operation on the three-dimensional Gaussian ellipsoid to obtain an updated three-dimensional Gaussian ellipsoid and proceed to step S2; otherwise, end.

[0008] In one possible implementation, the position and attitude of each surround-view camera are calibrated to obtain the calibrated position and attitude of each surround-view camera; including: The camera coordinate system of the forward-looking camera is defined as a three-dimensional global coordinate system; Obtain the position of the coordinate system of the camera group consisting of six panoramic cameras in the three-dimensional global coordinate system. and posture ; Obtain the fixed orthogonal rotation matrix of the camera coordinate system of each panoramic camera relative to the coordinate system of the camera group. ,in, The serial number of the panoramic camera; Determine the calibrated position of the camera coordinate system of each surround-view camera relative to the 3D global coordinate system. The calibrated attitude is .

[0009] In one possible implementation, the photometric error is determined based on the expanded boundary of each rendered image and the real image of the neighboring viewpoint; including: Get the pixels of the expansion boundary of a rendered image. Photometric value ,in, The sequence number of the surround-view camera corresponding to the rendered image; Intrinsic parameter matrix based on surround view camera , will pixels Back projection into a three-dimensional ray vector : Construct the transformation matrix between surround-view camera i and neighboring surround-view camera j. Including rotation matrix Translation vector ; Based on the transformation matrix Three-dimensional ray vector The corresponding pixels are obtained by converting the image onto the real image from the nearby surround-view camera j. Get pixels Photometric value ; Calculate pixels Photometric error : in, It represents the sum of squares.

[0010] In one possible implementation, the edge compensation gradient is determined based on photometric error; including: The partial derivative of the photometric error with respect to the pixel coordinates is used as the edge compensation gradient; Multiply the edge compensation gradient by the trigger interval coefficient to obtain the scaled edge compensation gradient.

[0011] In one possible implementation, a second two-dimensional aggregate gradient is determined based on a first two-dimensional aggregate gradient of the expanded boundary and the edge compensation gradient of each rendered image, including: The sum of the first two-dimensional aggregated gradient and the edge compensation gradient of each pixel at the expansion boundary of the rendered image is determined as the second two-dimensional aggregated gradient of the pixel.

[0012] Secondly, embodiments of this application provide a panoramic three-dimensional Gaussian splash reconstruction device for unmanned systems, comprising: The acquisition unit is used to acquire a panoramic image sequence of the target scene at the current moment using six surround-view cameras of the unmanned system. The panoramic image sequence includes real images from six perspectives. The processing unit is used to process the panoramic image sequence to obtain sparse point cloud data; and to process the sparse point cloud data to obtain a three-dimensional Gaussian scene containing multiple three-dimensional Gaussian ellipsoids. The compaction unit is used in the compaction stage of the three-dimensional Gaussian ellipsoid to obtain a rendered image with an expanded boundary for each panoramic camera using multiple three-dimensional Gaussian ellipsoids; to determine the luminance error based on the luminance of each pixel of the expanded boundary and the luminance of the corresponding real image of the neighboring viewpoint; to update the two-dimensional aggregated gradient of the pixels of the expanded boundary based on the luminance error; to determine the two-dimensional aggregated gradient of each three-dimensional Gaussian ellipsoid using the updated two-dimensional aggregated gradient of the pixels of the expanded boundary of each rendered image; and to determine whether to perform a compaction operation based on the two-dimensional aggregated gradient of each three-dimensional Gaussian ellipsoid.

[0013] Thirdly, embodiments of this application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method of embodiments of this application.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the methods of embodiments of this application.

[0015] Fifthly, embodiments of this application provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the method of embodiments of this application.

[0016] This application enables the reconstruction and rendering of high-fidelity seamless panoramic 3D scenes. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 A flowchart of a panoramic 3D Gaussian splash reconstruction method for unmanned systems provided in this application embodiment; Figure 2 Functional structure diagram of the panoramic three-dimensional Gaussian splash reconstruction device for unmanned systems provided in the embodiments of this application; Figure 3 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0020] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0021] First, a brief introduction to the design concept of the embodiments of this application will be given.

[0022] Unmanned systems (such as autonomous vehicles, drones, and intelligent inspection robots) place extremely high demands on high-precision, all-around 3D perception capabilities of their surroundings when performing autonomous navigation, path planning, and target reconnaissance tasks in complex environments. 3D scene reconstruction and novel view synthesis technologies, as core support for environmental perception in unmanned systems, are important research directions in the field of computer vision. In recent years, 3D Gaussian Splatting (3DGS), with its explicit 3D representation and realistic reconstruction effects, has been able to provide high-fidelity 3D scenes for unmanned systems, greatly promoting the development of this field. Meanwhile, panoramic cameras, due to their ability to provide images with a large field of view, are widely deployed on various unmanned system platforms for large-scale scene data acquisition. Utilizing panoramic image sequences acquired by unmanned systems to construct 3D scenes is an inevitable trend for realizing large-scale scene reconstruction on unmanned platforms.

[0023] To address the aforementioned problems, this application provides a 3D Gaussian reconstruction method based on joint pose optimization and manifold continuity constraints for panoramic images. This method can effectively adjust the pose of the panoramic camera and, without requiring an external depth estimation model, utilizes geometric analysis and cross-plane gradient aggregation techniques to solve the boundary discontinuities and densification blind spots caused by panoramic projection, achieving high-fidelity 3D Gaussian reconstruction based on panoramic input. This method can be widely applied to environmental perception in unmanned systems (such as autonomous vehicles, drones, and intelligent inspection robots) and in scenarios requiring high-fidelity 3D reconstruction using panoramic data, such as virtual reality.

[0024] After introducing the application scenarios and design concepts of the embodiments of this application, the technical solutions provided by the embodiments of this application will be described below.

[0025] like Figure 1 As shown in the figure, this embodiment provides a panoramic 3D Gaussian splash reconstruction method for unmanned systems, including: Step 101: Using the six surround-view cameras of the unmanned system, acquire a panoramic image sequence of the target scene at the current moment, the panoramic image sequence including real images from six perspectives; Specifically, due to severe polar distortion in spherical panoramic images, each frame of panoramic image is first converted into a standard six-sided cubemap through reprojection, generating six realistic images from conventional perspectives: front, right, back, left, top, and bottom.

[0026] Step 102: Process the panoramic image sequence to obtain sparse point cloud data; process the sparse point cloud data to obtain a three-dimensional Gaussian scene containing multiple three-dimensional Gaussian ellipsoids; For example, the Structure for Motion Restoration (SfM) algorithm is used to extract and match features from the transformed panoramic image sequence to obtain sparse 3D point cloud data and initial camera intrinsics. Extrinsic parameters. Using sparse point cloud data as spatial priors, the property parameters of the three-dimensional Gaussian radiation field are initialized, including the center position matrix. Covariance matrix Opacity And spherical harmonic coefficients (SH).

[0027] Step 103: In the compaction stage of the 3D Gaussian ellipsoid, a rendered image with an expanded boundary is obtained for each panoramic camera using multiple 3D Gaussian ellipsoids; the luminance error is determined based on the luminance of each pixel of the expanded boundary and the luminance of the corresponding real image from the neighboring viewpoint; the 2D aggregated gradient of the pixels of the expanded boundary is updated based on the luminance error; the 2D aggregated gradient of each 3D Gaussian ellipsoid is determined using the updated 2D aggregated gradient of the pixels of the expanded boundary of each rendered image; and the 2D aggregated gradient of each 3D Gaussian ellipsoid is used to determine whether to perform a compaction operation.

[0028] This embodiment can trigger the cloning or splitting of a three-dimensional Gaussian ellipsoid, thereby adaptively generating a new three-dimensional Gaussian ellipsoid in the seam region to fill the geometric fault.

[0029] This embodiment utilizes photometric loss to optimize the color attributes and positional parameters of a 3D Gaussian ellipsoid, thereby achieving high-fidelity seamless panoramic 3D scene reconstruction and rendering. This effectively avoids artifacts caused by forced cross-surface stretching after the boundary Gaussian body loses its splitting ability, and eliminates the need for cross-surface constraint calculations in the later stages of training, significantly reducing the overall training time of the model.

[0030] In some embodiments, during the densification stage of the three-dimensional Gaussian ellipsoids, a rendered image with an expanded boundary for each surround-view camera is obtained using multiple three-dimensional Gaussian ellipsoids; a luminance error is determined based on the luminance of each pixel of the expanded boundary and the luminance of the corresponding real image from a nearby viewpoint; the two-dimensional aggregated gradient of the pixels of the expanded boundary is updated based on the luminance error; the two-dimensional aggregated gradient of each three-dimensional Gaussian ellipsoid is determined using the updated two-dimensional aggregated gradient of the pixels of the expanded boundary of each rendered image; and a determination is made based on the two-dimensional aggregated gradient of each three-dimensional Gaussian ellipsoid to determine whether to perform a densification operation, including: Step S1: Calibrate the position and attitude of each surround view camera to obtain the calibrated position and attitude of each surround view camera; Step S2: Using the calibrated position and pose of each surround-view camera and multiple 3D Gaussian ellipsoids, generate a rendered image with an expanded boundary for each surround-view camera, and calculate the first 2D aggregate gradient of each rendered image with an expanded boundary. Step S3: Determine the photometric error based on the expanded boundary of each rendered image and the real image of the adjacent viewpoint; determine the edge compensation gradient based on the photometric error; Step S4: Determine the second two-dimensional aggregate gradient based on the first two-dimensional aggregate gradient and edge compensation gradient of the expansion boundary of each rendered image; Step S5: Determine the aggregated two-dimensional gradient of each three-dimensional Gaussian ellipsoid based on the second two-dimensional aggregated gradient of the expanded boundaries of all rendered images; Step S6: Sequentially determine whether the aggregated two-dimensional gradient of each three-dimensional Gaussian ellipsoid is greater than a preset threshold. If yes, perform a cloning or splitting operation on the three-dimensional Gaussian ellipsoid to obtain an updated three-dimensional Gaussian ellipsoid and proceed to step S2; otherwise, end.

[0031] For example, if the original image resolution is Then expand downwards, to the right, and outwards respectively. A 1-pixel wide field of view is obtained. A 2D image including the seam boundaries is then rendered within this expanded field of view.

[0032] In some embodiments, the position and orientation of each surround-view camera are calibrated to obtain the calibrated position and orientation of each surround-view camera; including: The camera coordinate system of the forward-looking camera is defined as a three-dimensional global coordinate system; Obtain the position of the coordinate system of the camera group consisting of six panoramic cameras in the three-dimensional global coordinate system. and posture ; Obtain the fixed orthogonal rotation matrix of the camera coordinate system of each panoramic camera relative to the coordinate system of the camera group. ,in, The serial number of the panoramic camera; Determine the calibrated position of the camera coordinate system of each surround-view camera relative to the 3D global coordinate system. The calibrated attitude is .

[0033] Specifically, all real images share the same optical center, and the relative translation vectors between real images are strictly equal to... .

[0034] To address the issue of initial pose noise generated by panoramic cameras, this embodiment implements panoramic camera pose calibration, avoiding scene tearing caused by independent optimization of each real image. It ensures that the internal topology of the projected hexahedron remains rigidly correlated during external pose optimization. Manifold continuity regularization and cross-face gradient aggregation effectively solve the problem of scene consistency at the edges of the panoramic image cube projection view, significantly improving scene reconstruction accuracy.

[0035] In some embodiments, the photometric error is determined based on the expanded boundary of each rendered image and the real image of the adjacent viewpoint; including: Get the pixels of the expansion boundary of a rendered image. Photometric value ,in, The sequence number of the surround-view camera corresponding to the rendered image; Intrinsic parameter matrix based on surround view camera , will pixels Back projection into a three-dimensional ray vector : Construct the transformation matrix between surround-view camera i and neighboring surround-view camera j. Including rotation matrix Translation vector ; Based on the transformation matrix Three-dimensional ray vector The corresponding pixels are obtained by converting the image onto the real image from the nearby surround-view camera j. Get pixels Photometric value ; Calculate pixels Photometric error : in, It represents the sum of squares.

[0036] To eliminate view discontinuities caused by cube projection, this embodiment proposes a depth-independent cross-plane ray mapping mechanism. Since introducing manifold constraints increases computational overhead, a trigger interval parameter is set. (e.g., triggered once every 3 iterations). In addition, to prevent geometric logic errors, invalid back faces with a depth direction (Z-axis) of the 3D ray vector less than or equal to 0 are removed.

[0037] In some embodiments, determining the edge compensation gradient based on photometric error includes: The partial derivative of the photometric error with respect to the pixel coordinates is used as the edge compensation gradient; Multiply the edge compensation gradient by the trigger interval coefficient to obtain the scaled edge compensation gradient.

[0038] In some embodiments, a second two-dimensional aggregate gradient is determined based on a first two-dimensional aggregate gradient of the expansion boundary and an edge compensation gradient of each rendered image, including: The sum of the first two-dimensional aggregated gradient and the edge compensation gradient of each pixel at the expansion boundary of the rendered image is determined as the second two-dimensional aggregated gradient of the pixel.

[0039] Based on the same inventive concept, this application provides a panoramic three-dimensional Gaussian splash reconstruction device for unmanned systems, see reference. Figure 2 As shown, the panoramic three-dimensional Gaussian splash reconstruction device 200 for unmanned systems provided in this application embodiment includes at least: The acquisition unit 201 is used to acquire a panoramic image sequence of the target scene at the current moment using six surround-view cameras of the unmanned system. The panoramic image sequence includes real images from six perspectives. Processing unit 202 is used to process the panoramic image sequence to obtain sparse point cloud data; and to process the sparse point cloud data to obtain a three-dimensional Gaussian scene containing multiple three-dimensional Gaussian ellipsoids. The densification unit 203 is used in the densification stage of the three-dimensional Gaussian ellipsoid to obtain a rendered image with an expanded boundary for each panoramic camera using multiple three-dimensional Gaussian ellipsoids; to determine the luminance error based on the luminance of each pixel of the expanded boundary and the luminance of the corresponding real image of the neighboring viewpoint; to update the two-dimensional aggregated gradient of the pixels of the expanded boundary based on the luminance error; to determine the two-dimensional aggregated gradient of each three-dimensional Gaussian ellipsoid using the updated two-dimensional aggregated gradient of the pixels of the expanded boundary of each rendered image; and to determine whether to perform a densification operation based on the two-dimensional aggregated gradient of each three-dimensional Gaussian ellipsoid.

[0040] It should be noted that the principle of the panoramic three-dimensional Gaussian splash reconstruction device 200 for unmanned systems provided in this application embodiment to solve the technical problem is similar to the method provided in this application embodiment. Therefore, the implementation of the panoramic three-dimensional Gaussian splash reconstruction device 200 for unmanned systems provided in this application embodiment can refer to the implementation of the method provided in this application embodiment, and the repeated parts will not be described again.

[0041] Based on the same inventive concept, embodiments of this application also provide an electronic device, such as... Figure 3 As shown, it includes a memory and a processor. The memory stores an executable program, and the processor executes the executable program to implement the steps of the panoramic three-dimensional Gaussian splash reconstruction method for unmanned systems provided in the above embodiments.

[0042] The aforementioned processor can be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0043] Since the electronic device described in this application embodiment is an electronic device equipped with a memory for implementing the panoramic three-dimensional Gaussian splash reconstruction method for unmanned systems disclosed in this application embodiment, those skilled in the art can understand the structure and variations of the electronic device described in this application embodiment based on the panoramic three-dimensional Gaussian splash reconstruction method for unmanned systems disclosed in this application embodiment, and therefore will not be described again here.

[0044] This application also provides a computer-readable storage medium storing a computer program thereon. When the computer program is run by a processor, it implements the steps of the panoramic three-dimensional Gaussian splash reconstruction method for unmanned systems provided in the above embodiments.

[0045] The storage medium in this embodiment may be included in an electronic device; or it may exist independently and not be assembled into an electronic device. The storage medium carries one or more computer programs, which, when executed, implement the steps of the panoramic three-dimensional Gaussian splash reconstruction method for unmanned systems provided in the above embodiment.

[0046] It should be understood that the various solutions in this embodiment have the same technical effects as those in the above method embodiments, and will not be repeated here.

[0047] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. Optionally, specific examples in this embodiment can refer to the examples described in any embodiment of this application, which will not be repeated here. Obviously, those skilled in the art should understand that the various modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular hardware and software combination.

[0048] This application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the panoramic three-dimensional Gaussian splash reconstruction method for unmanned systems provided in the above embodiments.

[0049] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions targeted in the blocks may occur in a different order than those targeted in the drawings. For example, two consecutively represented blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0050] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. Multitasking and parallel processing may be advantageous in certain environments. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this application. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

Claims

1. A panoramic 3D Gaussian splash reconstruction method applied to unmanned systems, characterized in that, include: Using six surround-view cameras of an unmanned system, a panoramic image sequence of the target scene at the current moment is acquired, the panoramic image sequence including real images from six perspectives; The panoramic image sequence is processed to obtain sparse point cloud data; Sparse point cloud data is processed to obtain a three-dimensional Gaussian scene containing multiple three-dimensional Gaussian ellipsoids; In the compaction stage of the three-dimensional Gaussian ellipsoid, multiple three-dimensional Gaussian ellipsoids are used to obtain the rendered image with the expanded boundary of each panoramic camera; The luminance error is determined based on the luminance of each pixel at the expansion boundary and the luminance of the corresponding real image at a nearby viewpoint; the two-dimensional aggregate gradient of the pixels at the expansion boundary is updated based on the luminance error; The updated 2D aggregate gradient of each pixel of the expanded boundary of each rendered image is used to determine the 2D aggregate gradient of each 3D Gaussian ellipsoid, and the densification operation is determined based on the 2D aggregate gradient of each 3D Gaussian ellipsoid.

2. The method according to claim 1, characterized in that, In the compaction stage of the three-dimensional Gaussian ellipsoid, a rendered image with an expanded boundary is obtained for each panoramic camera using multiple three-dimensional Gaussian ellipsoids; the luminance error is determined based on the luminance of each pixel of the expanded boundary and the luminance of the corresponding real image of the neighboring viewpoint; and the two-dimensional aggregate gradient of the pixels of the expanded boundary is updated based on the luminance error. Using the updated 2D aggregated gradient of the pixels of the expanded boundary of each rendered image, the 2D aggregated gradient of each 3D Gaussian ellipsoid is determined. Based on the 2D aggregated gradient of each 3D Gaussian ellipsoid, it is determined whether to perform a compaction operation, including: Step S1: Calibrate the position and attitude of each surround view camera to obtain the calibrated position and attitude of each surround view camera; Step S2: Using the calibrated position and pose of each surround-view camera and multiple 3D Gaussian ellipsoids, generate a rendered image with an expanded boundary for each surround-view camera, and calculate the first 2D aggregate gradient of each rendered image with an expanded boundary. Step S3: Determine the photometric error based on the expanded boundary of each rendered image and the real image of the adjacent viewpoint; determine the edge compensation gradient based on the photometric error; Step S4: Determine the second two-dimensional aggregate gradient based on the first two-dimensional aggregate gradient and edge compensation gradient of the expansion boundary of each rendered image; Step S5: Determine the aggregated two-dimensional gradient of each three-dimensional Gaussian ellipsoid based on the second two-dimensional aggregated gradient of the expanded boundaries of all rendered images; Step S6: Sequentially determine whether the aggregated two-dimensional gradient of each three-dimensional Gaussian ellipsoid is greater than a preset threshold. If yes, perform a cloning or splitting operation on the three-dimensional Gaussian ellipsoid to obtain an updated three-dimensional Gaussian ellipsoid and proceed to step S2; otherwise, end.

3. The method according to claim 2, characterized in that, The position and attitude of each surround-view camera are calibrated to obtain the calibrated position and attitude of each surround-view camera; including: The camera coordinate system of the forward-looking camera is defined as a three-dimensional global coordinate system; Obtain the position of the coordinate system of the camera group consisting of six panoramic cameras in the three-dimensional global coordinate system. and posture ; Obtain the fixed orthogonal rotation matrix of the camera coordinate system of each panoramic camera relative to the coordinate system of the camera group. ,in, The serial number of the panoramic camera; Determine the calibrated position of the camera coordinate system of each surround-view camera relative to the 3D global coordinate system. The calibrated attitude is .

4. The method according to claim 2, characterized in that, Based on the expansion boundary of each rendered image and the real image from a neighboring viewpoint, the photometric error is determined; including: Get the pixels of the expansion boundary of a rendered image. Photometric value ,in, The sequence number of the surround-view camera corresponding to the rendered image; Intrinsic parameter matrix based on surround view camera , will pixels Back projection into a three-dimensional ray vector : Construct the transformation matrix between surround-view camera i and neighboring surround-view camera j. Including rotation matrix Translation vector ; Based on the transformation matrix Three-dimensional ray vector The corresponding pixels are obtained by converting the image onto the real image from the nearby surround-view camera j. Get pixels Photometric value ; Calculate pixels Photometric error : in, It represents the sum of squares.

5. The method according to claim 4, characterized in that, Determining the edge compensation gradient based on photometric error; including: The partial derivative of the photometric error with respect to the pixel coordinates is used as the edge compensation gradient; Multiply the edge compensation gradient by the trigger interval coefficient to obtain the scaled edge compensation gradient.

6. The method according to claim 2, characterized in that, Based on the first two-dimensional aggregated gradient of the expansion boundary and the edge compensation gradient of each rendered image, the second two-dimensional aggregated gradient is determined, including: The sum of the first two-dimensional aggregated gradient and the edge compensation gradient of each pixel at the expansion boundary of the rendered image is determined as the second two-dimensional aggregated gradient of the pixel.

7. A panoramic three-dimensional Gaussian splash reconstruction device for unmanned systems, characterized in that, include: The acquisition unit is used to acquire a panoramic image sequence of the target scene at the current moment using six surround-view cameras of the unmanned system. The panoramic image sequence includes real images from six perspectives. The processing unit is used to process the panoramic image sequence to obtain sparse point cloud data; Sparse point cloud data is processed to obtain a three-dimensional Gaussian scene containing multiple three-dimensional Gaussian ellipsoids; The compaction unit is used to obtain a rendered image with an expanded boundary for each surround-view camera during the compaction stage of the three-dimensional Gaussian ellipsoid. The luminance error is determined based on the luminance of each pixel at the expansion boundary and the luminance of the corresponding real image at a nearby viewpoint; the two-dimensional aggregate gradient of the pixels at the expansion boundary is updated based on the luminance error; The updated 2D aggregate gradient of each pixel of the expanded boundary of each rendered image is used to determine the 2D aggregate gradient of each 3D Gaussian ellipsoid, and the densification operation is determined based on the 2D aggregate gradient of each 3D Gaussian ellipsoid.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the method as described in any one of claims 1-6.

Images