Three-dimensional reconstruction method, data processing method, rendering method and apparatus

Abstract

A three-dimensional reconstruction method, a data processing method, a rendering method, and an apparatus, applied to the field of artificial intelligence, for use in assigning a plurality of images in a scene into a plurality of regions within the scene and performing scene reconstruction on the basis of at least one image of each region to realize patch-based scene reconstruction. The three-dimensional reconstruction method comprises: acquiring a plurality of images for a scene, the scene comprising a plurality of regions; on the basis of poses of photographing devices corresponding to the plurality of images, assigning the plurality of images into the plurality of regions to obtain at least one image of each region; acquiring at least one three-dimensional Gaussian body of each region on the basis of the at least one image of each region, the at least one three-dimensional Gaussian body being used for representing the region in the scene; and combining the at least one three-dimensional Gaussian body of each region to obtain a combined three-dimensional Gaussian body, the combined three-dimensional Gaussian body representing the scene.

Description

A three-dimensional reconstruction method, data processing method, rendering method and device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on February 2, 2024, with application number 202410157546.1 and application name “A three-dimensional reconstruction method, data processing method, rendering method and device”, and claims priority to the Chinese patent application filed with the State Intellectual Property Office on September 13, 2024, with application number 202411291776.3 and application name “A three-dimensional reconstruction method, data processing method, rendering method and device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence, and in particular to a three-dimensional reconstruction method, a data processing method, a rendering method, and a device. Background Art

[0003] 3D reconstruction refers to the establishment of a mathematical model of a three-dimensional object suitable for computer representation and processing. It is the basis for processing, operating and analyzing its properties in a computer environment, and is also a key technology for establishing virtual reality in a computer to express the objective world.

[0004] Currently, existing solutions mainly use Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) methods for scene reconstruction. Among them, NeRF can reconstruct three-dimensional scenes based on images or videos, and uses deep learning technology to extract the geometric shape and texture information of objects from images from multiple perspectives. Therefore, it can present highly realistic three-dimensional models at any angle and distance. However, this method requires a lot of computing resources and time for training, and cannot take into account both rendering quality and speed at the same time. As for the 3DGS method, although it can complete high-quality scene reconstruction based on images or videos and realize real-time rendering, this method cannot be applied to the reconstruction of large scenes.

[0005] Since high-quality large-scene reconstruction and real-time rendering can provide users with an immersive online tour experience, achieving high-quality large-scene reconstruction and real-time rendering has become a problem that needs to be solved urgently. Summary of the Invention

[0006] Embodiments of the present application provide a three-dimensional reconstruction method, a data processing method, a rendering method, and a device for dividing multiple images in a scene into multiple areas in the scene, and reconstructing the scene based on at least one image in each area to achieve block scene reconstruction.

[0007] In view of this, in a first aspect, the present application provides a three-dimensional reconstruction method, comprising: first, obtaining multiple images of a scene, the scene including multiple regions; then, dividing the multiple images into the multiple regions according to the posture of the shooting device corresponding to the multiple images, so that each region in the scene includes at least one image; based on the at least one image of each region in the scene, at least one three-dimensional Gaussian body can be obtained for each region, and the at least one three-dimensional Gaussian body is used to represent the region in the scene; combining the at least one three-dimensional Gaussian body in each region to obtain a combined three-dimensional Gaussian body, and the combined three-dimensional Gaussian body can be used to represent the scene.

[0008] In an embodiment of the present application, by dividing the scene into regions, the scene can be reconstructed for each region separately using a three-dimensional Gaussian splashing method based on the image in each region, thereby realizing block scene reconstruction, and the number of three-dimensional Gaussian bodies in each region is smaller than the number of three-dimensional Gaussian bodies in the scene, thereby improving the accuracy of scene reconstruction and improving the quality of scene reconstruction.

[0009] In a possible embodiment, the aforementioned dividing multiple images into multiple areas according to the postures of the shooting devices corresponding to the multiple images to obtain at least one image for each area may include: determining the projection information of each area in the shooting device corresponding to each image according to the posture of the shooting device corresponding to each image, the projection information including the projection area of ​​each area in the shooting device; dividing each image into corresponding areas among the multiple areas according to the projection information of each area in the shooting device corresponding to each image to obtain at least one image for each area.

[0010] In an embodiment of the present application, images can be classified according to the posture of the shooting device, fully considering the visibility of the shooting device. The images can be divided according to the projection area of ​​each area in each shooting device, thereby improving the accuracy of image division into corresponding areas and improving the quality of scene reconstruction.

[0011] In a possible embodiment, the method further includes: obtaining multiple point cloud data corresponding to each image; dividing the multiple point cloud data according to the posture corresponding to each image to obtain a regional point cloud for each area, the regional point cloud including at least one point cloud data, and the regional point cloud of each area is used to obtain a three-dimensional Gaussian body for each area.

[0012] In one possible implementation, dividing the plurality of point cloud data according to the pose corresponding to each image to obtain the regional point cloud for each region may include: determining a point cloud ratio of the camera corresponding to each image based on the pose corresponding to each image, where the point cloud ratio is the ratio of the number of point clouds in each region observed by the camera to the total number of point cloud data observed by the camera. Dividing the plurality of point cloud data into corresponding regions of the plurality of regions based on the point cloud ratio of the camera corresponding to each image.

[0013] In a possible embodiment, the aforementioned division of multiple point cloud data into corresponding areas among multiple areas based on the point cloud ratio of the shooting device corresponding to each image may include: determining the shooting device in each area based on the point cloud ratio of the shooting device corresponding to each image; and dividing the multiple point cloud data into corresponding areas among multiple areas based on the shooting device in each area, to obtain at least one point cloud data for each area.

[0014] In a possible implementation, the method may further include: optimizing parameters of the three-dimensional Gaussian body to obtain an optimized three-dimensional Gaussian body, wherein the parameters are used to adjust appearance parameters of the three-dimensional Gaussian body after rendering.

[0015] In a possible embodiment, the aforementioned optimization of the parameters of the three-dimensional Gaussian body to obtain an optimized three-dimensional Gaussian body may include: rendering the three-dimensional Gaussian body to obtain a first rendered image; analyzing the first rendered image and appearance latent variables through a neural network model to obtain a second rendered image, where the appearance latent variables are used to identify the first rendered image; and optimizing the parameters of the three-dimensional Gaussian body based on the difference between the first rendered image and the sample image and the difference between the second rendered image and the sample image to obtain the optimized three-dimensional Gaussian body.

[0016] In an embodiment of the present application, the difference between the second rendered image and the sample image can be taken into consideration to optimize the parameters of the three-dimensional Gaussian body, wherein the appearance parameters of the second rendered image are consistent. By increasing the consideration of the comparison between the second rendered image and the sample image, the brightness of the image rendered based on the optimized three-dimensional Gaussian body can be made consistent, thereby reducing artifacts caused by different image exposure parameters, thereby improving the rendering quality.

[0017] In a possible implementation, the aforementioned analysis of the first rendered image and the appearance latent variables through a neural network model to obtain the second rendered image may include: downsampling the first rendered image to obtain a temporary rendered image; and analyzing the temporary rendered image and the appearance latent variables through a neural network model to obtain the second rendered image.

[0018] In a possible embodiment, the aforementioned optimization of the parameters of the three-dimensional Gaussian body based on the difference between the first rendered image and the sample image and the difference between the second rendered image and the sample image may include: calculating a first loss function between the first rendered image and the sample image; calculating a second loss function between the second rendered image and the sample image; and optimizing the parameters of the three-dimensional Gaussian body based on the first loss function and the second loss function to obtain an optimized three-dimensional Gaussian body.

[0019] In a possible implementation, the aforementioned optimization of the parameters of the three-dimensional Gaussian body to obtain an optimized three-dimensional Gaussian body may include: analyzing through a fully connected network according to the posture to obtain appearance-related parameters, and the appearance-related parameters are used to adjust the appearance parameters of the image after the three-dimensional Gaussian body is rendered; and optimizing the parameters of the three-dimensional Gaussian body according to the appearance-related parameters.

[0020] In a second aspect, the present application provides a data processing method, including: obtaining appearance latent variables, the appearance latent variables are used to identify the rendered image obtained by rendering a three-dimensional Gaussian body in the area; according to the appearance latent variables, optimizing the parameters of the three-dimensional Gaussian body to obtain an optimized three-dimensional Gaussian body, and the optimized three-dimensional Gaussian body is used to represent the area.

[0021] In a possible embodiment, the aforementioned optimization of the parameters of the three-dimensional Gaussian body according to the appearance latent variables to obtain the optimized three-dimensional Gaussian body may include: rendering the three-dimensional Gaussian body in the area to obtain a first rendered image; analyzing the first rendered image and the appearance latent variables to obtain a second rendered image; and optimizing the parameters of the three-dimensional Gaussian body according to the difference between the first rendered image and the sample image and the difference between the second rendered image and the sample image to obtain the optimized three-dimensional Gaussian body.

[0022] In a possible implementation, the aforementioned optimization of the parameters of the three-dimensional Gaussian body according to the appearance latent variables to obtain the optimized three-dimensional Gaussian body may include: rendering the three-dimensional Gaussian body in the area to obtain a first rendered image; analyzing the first rendered image and the appearance latent variables to obtain a second rendered image; and optimizing the parameters of the three-dimensional Gaussian body according to the difference between the second rendered image and the sample image to obtain the optimized three-dimensional Gaussian body.

[0023] In one possible implementation, the aforementioned analysis of the first rendered image and the appearance latent variables to obtain the second rendered image may include: downsampling the first rendered image to obtain a temporary rendered image; and analyzing the temporary rendered image and feature parameters through a neural network model to obtain the second rendered image.

[0024] In a possible embodiment, the aforementioned optimization of the parameters of the three-dimensional Gaussian body according to the difference between the first rendered image and the sample image and the difference between the second rendered image and the sample image to obtain the optimized three-dimensional Gaussian body may include: calculating a first loss function between the first rendered image and the sample image; calculating a second loss function between the second rendered image and the sample image; and optimizing the parameters of the three-dimensional Gaussian body according to the first loss function and the second loss function to obtain the optimized three-dimensional Gaussian body.

[0025] In a possible embodiment, the aforementioned optimizing the parameters of the three-dimensional Gaussian body according to the difference between the second rendered image and the sample image to obtain the optimized three-dimensional Gaussian body may include: calculating a second loss function between the second rendered image and the sample image; optimizing the parameters of the three-dimensional Gaussian body according to the second loss function to obtain the optimized three-dimensional Gaussian body.

[0026] In a possible implementation, the aforementioned optimization of the parameters of the three-dimensional Gaussian body according to the appearance latent variables to obtain the optimized three-dimensional Gaussian body may include: determining appearance-related parameters according to the appearance latent variables, the appearance-related parameters being used to adjust the appearance parameters of the image after the three-dimensional Gaussian body is rendered; and optimizing the parameters of the three-dimensional Gaussian body according to the appearance-related parameters.

[0027] In a third aspect, the present application provides a data processing method, including: obtaining a three-dimensional Gaussian body of an area based on an image in the area, where the area is any one of multiple areas in a scene; rendering the three-dimensional Gaussian body to obtain a first rendered image; analyzing the first rendered image and feature parameters through a neural network model to obtain a second rendered image, and the feature parameters are used to adjust the appearance parameters of the first rendered image; optimizing the parameters of the three-dimensional Gaussian body based on the difference between the first rendered image and the sample image and the difference between the second rendered image and the sample image to obtain an optimized three-dimensional Gaussian body, and the optimized three-dimensional Gaussian body is used to represent the area.

[0028] In a possible embodiment, the aforementioned analysis of the first rendered image and feature parameters through a neural network model to obtain the second rendered image may include: downsampling the first rendered image to obtain a temporary rendered image; and analyzing the temporary rendered image and feature parameters through a neural network model to obtain the second rendered image.

[0029] In one possible implementation, optimizing the parameters of the three-dimensional Gaussian body based on the aforementioned differences with the sample image and the differences between the second rendered image and the sample image to obtain an optimized three-dimensional Gaussian body may include: calculating a first loss function between the first rendered image and the sample image; calculating a second loss function between the second rendered image and the sample image; and optimizing the parameters of the three-dimensional Gaussian body based on the first loss function and the second loss function to obtain an optimized three-dimensional Gaussian body.

[0030] In a fourth aspect, the present application provides a rendering method, including: obtaining the posture of a shooting device in a scene, where the scene is divided into multiple areas; determining a three-dimensional Gaussian body in the area corresponding to the shooting device based on the posture of the shooting device; rendering the three-dimensional Gaussian body of each area to obtain a rendered image of each area; and combining the rendered images of each area to obtain a target rendered image.

[0031] In the embodiment of the present application, each block area can be processed separately based on the scene block strategy, thereby reducing the amount of calculation and improving the rendering speed to achieve real-time rendering.

[0032] In a possible implementation, the aforementioned determination of the three-dimensional Gaussian body in the area corresponding to the shooting device based on the posture of the shooting device may include: determining the area corresponding to the shooting device based on the posture of the shooting device; deleting the invisible three-dimensional Gaussian body in the area based on the marking information of the three-dimensional Gaussian body in the area, and obtaining the three-dimensional Gaussian body in the area corresponding to the shooting device.

[0033] In an embodiment of the present application, the three-dimensional Gaussian bodies that are invisible in the area can be deleted, and only the three-dimensional Gaussian bodies that are visible in the area are rendered, which effectively reduces the amount of calculation when rendering the three-dimensional Gaussian bodies and improves the rendering speed. In addition, the three-dimensional Gaussian bodies that are invisible in the area are deleted, which effectively reduces the storage space. It can be understood that in an embodiment of the present application, deleting the three-dimensional Gaussian bodies that are invisible in the area means that when rendering the three-dimensional Gaussian bodies in the area, the three-dimensional Gaussian bodies that are invisible in the area are excluded, and the three-dimensional Gaussian bodies that are invisible in the area are not rendered, and it does not mean that there are no three-dimensional Gaussian bodies that are invisible to the area. The three-dimensional Gaussian bodies that are invisible to the area may be visible to other areas, and can be rendered to obtain rendered images of other areas.

[0034] In a possible implementation, the aforementioned rendering of the three-dimensional Gaussian body of each area to obtain a rendered image of each area may include: calculating the depth of the three-dimensional Gaussian body of each area from the camera device; determining the target three-dimensional Gaussian body of each area based on the depth and the projected area of ​​the three-dimensional Gaussian body of each area; and rendering the target three-dimensional Gaussian body of each area to obtain a rendered image of each scene.

[0035] In a possible embodiment, the aforementioned determination of the target three-dimensional Gaussian body in each region based on the depth and the projected area of ​​the three-dimensional Gaussian body in each region may include: determining the three-dimensional Gaussian body to be deleted in each region from the three-dimensional Gaussian body in each region based on the depth, and the depth is positively correlated with the number of three-dimensional Gaussian bodies to be deleted; determining a first preset threshold value for the projected area of ​​the three-dimensional Gaussian body in each region based on the depth; and deleting the three-dimensional Gaussian bodies whose projected area is less than the preset threshold from the three-dimensional Gaussian bodies to be deleted based on the three-dimensional Gaussian bodies to be deleted and the first preset threshold, to obtain the target three-dimensional Gaussian body.

[0036] In this embodiment of the present application, layered rendering can be performed based on both the depth and projected area of ​​the 3D Gaussian volume. The rendering of distant 3D Gaussian volumes can be reduced, and 3D Gaussian volumes with projected areas smaller than a preset value can be deleted to reduce the number of 3D Gaussian volumes to be rendered, thereby improving rendering speed.

[0037] In a possible implementation, the aforementioned rendering of the three-dimensional Gaussian body of each region to obtain a rendered image of each region may include: randomly deleting a preset number of three-dimensional Gaussian bodies from the three-dimensional Gaussian body of each region to obtain a target three-dimensional Gaussian body of each region; and rendering the target three-dimensional Gaussian body of each region to obtain a rendered image of each region.

[0038] In a possible implementation, the aforementioned rendering of the three-dimensional Gaussian body of each region to obtain a rendered image of each region may include: calculating a first volume of the three-dimensional Gaussian body of each region; based on the first volume, deleting the three-dimensional Gaussian body whose first volume exceeds a second preset threshold to obtain a target three-dimensional Gaussian body; and rendering based on the target three-dimensional Gaussian body in each region to obtain a rendered image of each region.

[0039] In one possible implementation, the aforementioned rendering of the three-dimensional Gaussian body of each region to obtain a rendered image of each region may include: randomly determining a three-dimensional Gaussian body to be deleted from the three-dimensional Gaussian body of each region; calculating the second volume of the three-dimensional Gaussian body of each region; deleting the three-dimensional Gaussian body whose second volume exceeds a third preset threshold in the three-dimensional Gaussian body to be deleted to obtain a target three-dimensional Gaussian body; and rendering according to the target three-dimensional Gaussian body in each region to obtain a rendered image of each region.

[0040] In a fifth aspect, the present application provides a rendering method, including: determining a target three-dimensional Gaussian body from the three-dimensional Gaussian body based on the marking information of the three-dimensional Gaussian body in the area, the target three-dimensional Gaussian body being a three-dimensional Gaussian body recognized by the shooting equipment in the area; rendering the target three-dimensional Gaussian body to obtain a rendered image of the area.

[0041] In a possible implementation, the aforementioned determining the target three-dimensional Gaussian body from the three-dimensional Gaussian body based on the marking information of the three-dimensional Gaussian body in the region may include: based on the marking information of the three-dimensional Gaussian body in the region, using the three-dimensional Gaussian body visible in the region as the target three-dimensional Gaussian body.

[0042] In a sixth aspect, the present application provides a rendering method, including: obtaining a three-dimensional Gaussian body of the region based on an image in the region; determining a target three-dimensional Gaussian body of the region from the three-dimensional Gaussian body of the region based on one or more of the depth of the three-dimensional Gaussian body, the projected area or volume of the three-dimensional Gaussian body, and the number of target three-dimensional Gaussian bodies is less than the number of three-dimensional Gaussian bodies; rendering the target three-dimensional Gaussian body to obtain a rendered image of the region.

[0043] In a possible embodiment, the aforementioned determining the target three-dimensional Gaussian body of the region from the three-dimensional Gaussian body of the region based on one or more of the depth, projected area or volume of the three-dimensional Gaussian body may include: determining the target three-dimensional Gaussian body of the region from the three-dimensional Gaussian body of the region based on the depth and projected area of ​​the three-dimensional Gaussian body; determining the target three-dimensional Gaussian body of the region from the three-dimensional Gaussian body of the region based on the volume of the three-dimensional Gaussian body.

[0044] In a possible embodiment, the aforementioned determination of the target three-dimensional Gaussian body of the region from the three-dimensional Gaussian body of the region based on the depth and projection area of ​​the three-dimensional Gaussian body may include: determining the three-dimensional Gaussian body to be rendered in the region from the three-dimensional Gaussian body of the region based on the depth of the three-dimensional Gaussian body; determining a first preset value of the projection area of ​​the three-dimensional Gaussian body in the region based on the depth of the three-dimensional Gaussian body; and based on the three-dimensional Gaussian body to be rendered in the region and the first preset threshold, taking the three-dimensional Gaussian body in the three-dimensional Gaussian body to be rendered whose projection area is greater than or equal to the first preset value as the target three-dimensional Gaussian body.

[0045] In a possible embodiment, the aforementioned determination of the target three-dimensional Gaussian body of the region from the three-dimensional Gaussian body of the region based on the volume of the three-dimensional Gaussian body may include: calculating the first volume of the three-dimensional Gaussian body in the region; and taking the three-dimensional Gaussian body whose first volume is less than or equal to a second preset threshold as the target three-dimensional Gaussian body.

[0046] In a possible embodiment, the aforementioned determination of the target three-dimensional Gaussian body of the region from the three-dimensional Gaussian body of the region based on the volume of the three-dimensional Gaussian body may include: randomly determining the three-dimensional Gaussian body to be rendered from the three-dimensional Gaussian body of the region; calculating the second volume of the three-dimensional Gaussian body in the region; and taking the three-dimensional Gaussian body whose second volume in the three-dimensional Gaussian body to be rendered is less than or equal to a third preset threshold as the target three-dimensional Gaussian body.

[0047] In a possible implementation, before rendering the target three-dimensional Gaussian body to obtain the rendered image of the region, the method may further include: randomly determining a preset number of three-dimensional Gaussian bodies from the three-dimensional Gaussian bodies in the region as the target three-dimensional Gaussian bodies.

[0048] In a possible implementation, the region is any one of the multiple regions in the aforementioned first aspect.

[0049] In a seventh aspect, the present application provides a three-dimensional reconstruction device, comprising:

[0050] An acquisition module, configured to acquire a plurality of images of a scene, where the scene includes a plurality of regions;

[0051] a division module, configured to divide the plurality of images into a plurality of regions according to the positions of the plurality of images corresponding to the shooting device, and obtain at least one image in each region;

[0052] The aforementioned acquisition module is further used to acquire at least one three-dimensional Gaussian volume for each region based on at least one image of each region, wherein the at least one three-dimensional Gaussian volume is used to represent the region in the scene;

[0053] The combining module is used to combine at least one three-dimensional Gaussian volume in each area to obtain a combined three-dimensional Gaussian volume, where the combined three-dimensional Gaussian volume represents the scene.

[0054] In one possible embodiment, the division module is specifically used to determine the projection information of each region in the shooting device corresponding to each image based on the posture of the shooting device corresponding to each image, and the projection information includes the projection area of ​​each region in the shooting device; according to the projection information of each region in the shooting device corresponding to each image, each image is divided into corresponding regions among the multiple regions to obtain at least one image of each region.

[0055] In a possible implementation, the device may further include:

[0056] The aforementioned acquisition module is further used to obtain multiple point cloud data corresponding to each image;

[0057] The aforementioned division module is also used to divide multiple point cloud data according to the posture corresponding to each image to obtain a regional point cloud for each area. The regional point cloud includes at least one point cloud data. The regional point cloud of each area is used to obtain a three-dimensional Gaussian body for each area.

[0058] In one possible embodiment, the partitioning module is specifically configured to determine, based on the pose corresponding to each image, a point cloud ratio of the camera corresponding to each image, where the point cloud ratio is the ratio of the number of point clouds in each region observed by the camera to the total number of point cloud data observed by the camera. Based on the point cloud ratio of the camera corresponding to each image, the plurality of point cloud data are partitioned into corresponding regions of the plurality of regions.

[0059] In one possible embodiment, the division module is specifically used to: determine the shooting device in each area based on the point cloud ratio of the shooting device corresponding to each image; and divide the multiple point cloud data into corresponding areas in the multiple areas according to the shooting device in each area to obtain at least one point cloud data for each area.

[0060] In a possible implementation, the device may further include:

[0061] The optimization module is used to optimize the parameters of the three-dimensional Gaussian body to obtain an optimized three-dimensional Gaussian body. The parameters are used to adjust the appearance parameters of the three-dimensional Gaussian body after rendering.

[0062] In one possible implementation, the optimization module is specifically used to render a three-dimensional Gaussian body to obtain a first rendered image; analyze the first rendered image and appearance latent variables through a neural network model to obtain a second rendered image, where the appearance latent variables are used to identify the first rendered image; and optimize the parameters of the three-dimensional Gaussian body based on the difference between the first rendered image and the sample image and the difference between the second rendered image and the sample image to obtain an optimized three-dimensional Gaussian body.

[0063] In a possible implementation, the optimization module is specifically configured to downsample the first rendered image to obtain a temporary rendered image; and analyze the temporary rendered image and appearance latent variables using a neural network model to obtain a second rendered image.

[0064] In one possible implementation, the optimization module is specifically used to calculate a first loss function between a first rendered image and a sample image; calculate a second loss function between a second rendered image and a sample image; and optimize the parameters of a three-dimensional Gaussian body according to the first loss function and the second loss function to obtain an optimized three-dimensional Gaussian body.

[0065] In one possible implementation, the optimization module is specifically used to analyze the posture through a fully connected network to obtain appearance-related parameters, and the appearance-related parameters are used to adjust the appearance parameters of the image after the three-dimensional Gaussian body is rendered; based on the appearance-related parameters, the parameters of the three-dimensional Gaussian body are optimized.

[0066] In an eighth aspect, the present application provides a data processing device, comprising:

[0067] An acquisition module, used to acquire appearance latent variables, where the appearance latent variables are used to identify a rendered image obtained by rendering a three-dimensional Gaussian volume in a region;

[0068] The optimization module is used to optimize the parameters of the three-dimensional Gaussian body according to the appearance latent variables to obtain an optimized three-dimensional Gaussian body. The optimized three-dimensional Gaussian body is used to represent the region.

[0069] In one possible implementation, the aforementioned optimization module is specifically used to: render the three-dimensional Gaussian body in the area to obtain a first rendered image; analyze the first rendered image and the appearance latent variables to obtain a second rendered image; and optimize the parameters of the three-dimensional Gaussian body based on the difference between the first rendered image and the sample image and the difference between the second rendered image and the sample image to obtain an optimized three-dimensional Gaussian body.

[0070] In one possible implementation, the aforementioned optimization module is specifically used to: render the three-dimensional Gaussian body in the area to obtain a first rendered image; analyze the first rendered image and the appearance latent variables to obtain a second rendered image; and optimize the parameters of the three-dimensional Gaussian body based on the difference between the second rendered image and the sample image to obtain an optimized three-dimensional Gaussian body.

[0071] In a possible implementation, the aforementioned optimization module is specifically configured to: downsample the first rendered image to obtain a temporary rendered image; and analyze the temporary rendered image and feature parameters using a neural network model to obtain a second rendered image.

[0072] In one possible implementation, the aforementioned optimization module is specifically used to: calculate a first loss function between the first rendered image and the sample image; calculate a second loss function between the second rendered image and the sample image; and optimize the parameters of the three-dimensional Gaussian body based on the first loss function and the second loss function to obtain an optimized three-dimensional Gaussian body.

[0073] In a possible implementation, the aforementioned optimization module is specifically configured to: calculate a second loss function between the second rendered image and the sample image; and optimize parameters of the three-dimensional Gaussian body according to the second loss function to obtain an optimized three-dimensional Gaussian body.

[0074] In a possible implementation, the aforementioned optimization module is specifically used to: determine appearance-related parameters based on appearance latent variables, where the appearance-related parameters are used to adjust the appearance parameters of the image after three-dimensional Gaussian rendering; and optimize the parameters of the three-dimensional Gaussian based on the appearance-related parameters.

[0075] In a ninth aspect, the present application provides a data processing device, comprising:

[0076] An acquisition module, configured to acquire a three-dimensional Gaussian volume of a region according to an image in the region, where the region is any one of multiple regions in the scene;

[0077] A rendering module, configured to render the three-dimensional Gaussian volume to obtain a first rendered image;

[0078] an analysis module, configured to analyze the first rendered image and feature parameters using a neural network model to obtain a second rendered image, wherein the feature parameters are used to adjust appearance parameters of the first rendered image;

[0079] The optimization module is used to optimize the parameters of the three-dimensional Gaussian body according to the difference between the first rendered image and the sample image and the difference between the second rendered image and the sample image to obtain an optimized three-dimensional Gaussian body, and the optimized three-dimensional Gaussian body is used to represent the area.

[0080] In a tenth aspect, the present application provides a rendering device, comprising:

[0081] The acquisition module is used to obtain the position and posture of the camera in the scene. The scene is divided into multiple areas;

[0082] A determination module, configured to determine a three-dimensional Gaussian volume in an area corresponding to the shooting device according to the position of the shooting device;

[0083] A rendering module is used to render the three-dimensional Gaussian volume of each region to obtain a rendered image of each region;

[0084] The combination module is used to combine the rendered images of each area to obtain a target rendered image.

[0085] In one possible implementation, the determination module is specifically used to determine the area corresponding to the shooting device based on the posture of the shooting device; based on the marking information of the three-dimensional Gaussian body in the area, delete the invisible three-dimensional Gaussian body in the area to obtain the three-dimensional Gaussian body in the area corresponding to the shooting device.

[0086] In one possible implementation, the rendering module is specifically used to calculate the depth of the three-dimensional Gaussian body of each area from the camera device; determine the target three-dimensional Gaussian body of each area based on the depth and the projected area of ​​the three-dimensional Gaussian body of each area; and render the target three-dimensional Gaussian body of each area to obtain a rendered image of each area.

[0087] In one possible embodiment, the rendering module is specifically used to determine, from the three-dimensional Gaussian bodies in each region according to the depth, a three-dimensional Gaussian body to be deleted in each region; determine, according to the depth, a first preset threshold value of the projected area of ​​the three-dimensional Gaussian body in each region; and, based on the three-dimensional Gaussian bodies to be deleted in each region and the first preset threshold value, delete the three-dimensional Gaussian bodies to be deleted whose projected area is smaller than the preset threshold value to obtain a target three-dimensional Gaussian body.

[0088] In one possible implementation, the rendering module is specifically configured to randomly delete a preset number of three-dimensional Gaussian bodies from the three-dimensional Gaussian body of each region to obtain a target three-dimensional Gaussian body of each region; and render the target three-dimensional Gaussian body of each region to obtain a rendered image of each region.

[0089] In one possible implementation, the rendering module is specifically configured to calculate a first volume of a three-dimensional Gaussian body in each region; based on the first volume, delete three-dimensional Gaussian bodies whose first volume exceeds a second preset threshold to obtain a target three-dimensional Gaussian body; and render based on the target three-dimensional Gaussian body in each region to obtain a rendered image of each region.

[0090] In one possible embodiment, the rendering module is specifically used to randomly determine a three-dimensional Gaussian body to be deleted from the three-dimensional Gaussian body of each region; calculate the second volume of the three-dimensional Gaussian body of each region; delete the three-dimensional Gaussian body whose second volume exceeds a third preset threshold in the three-dimensional Gaussian body to be deleted to obtain a target three-dimensional Gaussian body; and render according to the target three-dimensional Gaussian body of each region to obtain a rendered image of each region.

[0091] In an eleventh aspect, the present application provides a rendering device, comprising:

[0092] a determination module, configured to determine a target three-dimensional Gaussian body from the three-dimensional Gaussian bodies according to the label information of the three-dimensional Gaussian bodies in the region, where the target three-dimensional Gaussian body is the three-dimensional Gaussian body recognized by the shooting device in the region;

[0093] The rendering module is used to render the target three-dimensional Gaussian volume to obtain a rendered image of the area.

[0094] In a possible implementation, the aforementioned determination module is specifically configured to: select a visible three-dimensional Gaussian body in the region as a target three-dimensional Gaussian body according to label information of the three-dimensional Gaussian body in the region.

[0095] In a twelfth aspect, the present application provides a rendering device, comprising:

[0096] An acquisition module, configured to acquire a three-dimensional Gaussian volume of a region according to an image in the region;

[0097] a determination module, configured to determine target three-dimensional Gaussian bodies of the region from the three-dimensional Gaussian bodies of the region based on one or more of the depth of the three-dimensional Gaussian body and the projected area or volume of the three-dimensional Gaussian body, wherein the number of the target three-dimensional Gaussian bodies is less than the number of the three-dimensional Gaussian bodies;

[0098] The rendering module is used to render the target three-dimensional Gaussian volume to obtain a rendered image of the area.

[0099] In a possible implementation, the aforementioned determination module is specifically used to: determine the target three-dimensional Gaussian body of the region from the three-dimensional Gaussian body of the region based on the depth and projection area of ​​the three-dimensional Gaussian body; determine the target three-dimensional Gaussian body of the region from the three-dimensional Gaussian body of the region based on the volume of the three-dimensional Gaussian body.

[0100] In a possible implementation, the aforementioned determination module is specifically used to: determine the three-dimensional Gaussian body to be rendered in the region from the three-dimensional Gaussian body in the region according to the depth of the three-dimensional Gaussian body; determine a first preset value of the projected area of ​​the three-dimensional Gaussian body in the region according to the depth of the three-dimensional Gaussian body; and, based on the three-dimensional Gaussian body to be rendered in the region and the first preset threshold, select the three-dimensional Gaussian body in the three-dimensional Gaussian body to be rendered whose projected area is greater than or equal to the first preset value as the target three-dimensional Gaussian body.

[0101] In a possible implementation, the aforementioned determination module is specifically configured to: calculate a first volume of a three-dimensional Gaussian body in the region; and use a three-dimensional Gaussian body whose first volume is less than or equal to a second preset threshold as a target three-dimensional Gaussian body.

[0102] In a possible implementation, the aforementioned determination module is specifically used to: randomly determine a three-dimensional Gaussian body to be rendered from the three-dimensional Gaussian bodies in the region; calculate the second volume of the three-dimensional Gaussian body in the region; and use the three-dimensional Gaussian body in the three-dimensional Gaussian body to be rendered whose second volume is less than or equal to a third preset threshold as the target three-dimensional Gaussian body.

[0103] In a possible implementation, before rendering the target three-dimensional Gaussian body to obtain the rendered image of the region, the determination module is further configured to randomly determine a preset number of three-dimensional Gaussian bodies from the three-dimensional Gaussian bodies in the region as target three-dimensional Gaussian bodies.

[0104] In a possible embodiment, the region is any one of the multiple regions in the aforementioned seventh aspect.

[0105] In the thirteenth aspect, the present application provides a three-dimensional reconstruction device, which includes: a processor, a memory, an input and output device, and a bus; the memory stores computer instructions; when the processor executes the computer instructions in the memory, the memory stores computer instructions; when the processor executes the computer instructions in the memory, it is used to implement the first aspect or any one of the implementation methods of the first aspect.

[0106] In the fourteenth aspect, the present application provides a data processing device, which includes: a processor, a memory, an input and output device, and a bus; the memory stores computer instructions; when the processor executes the computer instructions in the memory, the memory stores computer instructions; when the processor executes the computer instructions in the memory, it is used to implement the second aspect, the third aspect, any one of the implementation methods of the second aspect, or any one of the implementation methods of the third aspect.

[0107] In the fifteenth aspect, the present application provides a rendering device, which includes: a processor, a memory, an input and output device, and a bus; the memory stores computer instructions; when the processor executes the computer instructions in the memory, the memory stores computer instructions; when the processor executes the computer instructions in the memory, it is used to implement the fourth aspect, the fifth aspect, the sixth aspect, any one of the implementation methods of any aspect of the fourth aspect, any one of the implementation methods of any aspect of the fifth aspect, or any one of the implementation methods of any aspect of the sixth aspect.

[0108] In the sixteenth aspect, an embodiment of the present application provides a chip system, which includes a processor and an input / output port, wherein the processor is used to implement the processing functions involved in the method described in the first, second, third, fourth, fifth or sixth aspect above, and the input / output port is used to implement the transceiver functions involved in the method described in the first, second, third, fourth, fifth or sixth aspect above.

[0109] In one possible design, the chip system also includes a memory, which is used to store program instructions and data for implementing the functions involved in the methods described in the first, second, third, fourth, fifth or sixth aspects above.

[0110] The chip system may be composed of chips, or may include chips and other discrete devices.

[0111] In a seventeenth aspect, embodiments of the present application provide a computer-readable storage medium. The computer-readable storage medium stores computer instructions that, when executed on a computer, cause the computer to execute the method described in any possible implementation of the first, second, third, fourth, fifth, or sixth aspects.

[0112] In an eighteenth aspect, an embodiment of the present application provides a computer program product. The computer program product includes a computer program or instructions, which, when executed on a computer, causes the computer to perform the method as described in any possible implementation of the first, second, third, fourth, fifth, or sixth aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0113] FIG1 is a schematic diagram of a system architecture provided by this application;

[0114] FIG2 is a schematic diagram of a flow chart of a 3D reconstruction method provided by the present application;

[0115] FIG3 is a schematic diagram of selecting a shooting device based on the projection area of ​​the bounding box of the scene area on the shooting device;

[0116] FIG4 illustrates the camera and image selection process in the scene division of this application;

[0117] FIG5 is a schematic diagram of a flow chart of another 3D reconstruction method provided by the present application;

[0118] FIG6 is a schematic diagram of a flow chart of another 3D reconstruction method provided by the present application;

[0119] FIG7 is a flow chart of a data processing method provided by the present application;

[0120] FIG8 is a schematic diagram of a flow chart of a rendering method provided by this application;

[0121] FIG9 is a schematic diagram of a flow chart of another rendering method provided by the present application;

[0122] FIG10 is a schematic diagram of screening visible three-dimensional Gaussian bodies when performing block rendering of different scenes;

[0123] FIG11 is a schematic diagram of a flow chart of another rendering method provided by the present application;

[0124] FIG12 is a schematic diagram of a flow chart of another rendering method provided by the present application;

[0125] FIG13 is a schematic diagram of screening based on the depth, index number, and projection area of ​​a three-dimensional Gaussian volume;

[0126] FIG14 is a schematic diagram of a flow chart of another rendering method provided by the present application;

[0127] FIG15 is a schematic structural diagram of a three-dimensional reconstruction device provided by the present application;

[0128] FIG16 is a schematic structural diagram of a data processing device 1500 provided by the present application;

[0129] FIG17 is a schematic structural diagram of a data processing device 1600 provided by the present application;

[0130] FIG18 is a schematic structural diagram of a rendering device 1700 provided by this application;

[0131] FIG19 is a schematic structural diagram of a rendering device 1800 provided by this application;

[0132] FIG20 is a schematic structural diagram of a rendering device 1900 provided by this application;

[0133] FIG21 is a schematic structural diagram of another 3D reconstruction device provided by the present application;

[0134] FIG22 is a schematic structural diagram of another data processing device provided by the present application;

[0135] FIG23 is a schematic structural diagram of another rendering device provided by the present application;

[0136] FIG24 is a schematic diagram of the structure of a chip provided in this application. DETAILED DESCRIPTION

[0137] The following will describe the technical solutions in the embodiments of this application in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of this application.

[0138] The method provided in this application can be applied to artificial intelligence (AI) scenarios. AI is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.

[0139] First, we describe the overall workflow of AI systems. The following section elaborates on the aforementioned AI framework from two perspectives: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, this could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. Throughout this process, data undergoes a condensed journey from "data-information-knowledge-wisdom." The "IT value chain," spanning the underlying infrastructure of human intelligence, information (provided and processed by technology), and the system's industrial ecosystem, reflects the value that AI brings to the information technology industry.

[0140] (1) Infrastructure

[0141] Infrastructure provides computing power for AI systems, enabling communication with the outside world and supporting this through a foundational platform. External communication occurs through sensors; computing power is provided by intelligent chips (CPUs, NPUs, GPUs, ASICs, FPGAs, and other hardware accelerators). The foundational platform includes a distributed computing framework and network-related platform guarantees and support, including cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to acquire data, which is then fed into the intelligent chips within the distributed computing system provided by the foundational platform for computation.

[0142] (2) Data

[0143] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0144] (3) Data processing

[0145] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0146] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0147] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0148] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0149] (4) General ability

[0150] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0151] (5) Smart products and industry applications

[0152] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.

[0153] The embodiments of the present application involve related applications of neural networks. In order to better understand the solutions of the embodiments of the present application, the relevant terms and concepts of neural networks that may be involved in the embodiments of the present application are first introduced below.

[0154] (1) Neural Network

[0155] A neural network can be composed of neural units. A neural unit can refer to an operation unit with xs and intercept 1 as input. The output of the operation unit can be shown as formula (1-1):

[0156] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal of the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0157] (2) Convolutional Neural Network

[0158] A convolutional neural network (CNN) is a deep neural network with a convolutional architecture. It consists of a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons that performs convolution processing on the input signal. In a convolutional layer of a CNN, a neuron can only connect to a subset of neurons in adjacent layers. A convolutional layer typically contains several feature planes, each of which is composed of a rectangular arrangement of neurons. Neurons in the same feature plane share weights, which are referred to as convolution kernels. Shared weights can be understood as position-independent feature extraction. Convolution kernels can be formalized as matrices of random size, and during CNN training, the kernels can be learned to acquire reasonable weights. Furthermore, shared weights have the direct benefit of reducing the number of connections between layers of the CNN, thereby reducing the risk of overfitting.

[0159] (3) Loss function

[0160] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually a process of optimization before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict lower, and continue to adjust until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function (loss function) or objective function (objective function). They are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible. The loss function can usually include loss functions such as mean square error, cross entropy, logarithm, exponential, etc. For example, the mean square error can be used as a loss function, defined as The specific loss function can be selected according to the actual application scenario.

[0161] (4) Neural Radiance Fields (NeRF)

[0162] The scene expression is recorded in the deep neural network through implicit expression, and a static three-dimensional scene is implicitly learned using the deep neural network to indirectly complete tasks such as three-dimensional reconstruction of the scene and novel view image generation (Novel View Synthesis).

[0163] (5) 3D Gaussian Splatting (3DGS)

[0164] 3D Gaussian splatting is a rasterization technique used to render realistic scenes. It uses 3D Gaussian distributions to describe the scene and projects these distributions onto the image plane as 2D Gaussian distributions to render the view.

[0165] (6) Three-dimensional Gaussian

[0166] A three-dimensional Gaussian body, also known as a Gaussian sphere, is used to describe a Gaussian distribution in three-dimensional space. The shape of a Gaussian body is similar to a sphere, and its shape and size are determined by its standard deviation. A larger standard deviation results in a flatter Gaussian body, and vice versa.

[0167] (7) Rendering

[0168] In computer graphics, rendering generally refers to outputting visual information such as lighting, textures, and colors of a three-dimensional model onto a two-dimensional screen. During the rendering process, the graphics processing unit (GPU) is the primary hardware device that performs rendering calculations. The main work of the GPU graphics rendering pipeline can be divided into two parts: converting 3D coordinates into 2D coordinates and converting 2D coordinates into actual colored pixels. The specific implementation can be divided into six stages, including vertex shader, shape assembly, geometry shader, rasterization, fragment shader, and test and blending. Among them, rasterization is the process of converting data into visible pixels.

[0169] The following introduces some possible system architectures provided by the embodiments of this application.

[0170] For example, as shown in FIG1 , a terminal device 101 is connected to a server cluster 102 via a communication network. In an embodiment of the present application, the terminal device 101 can capture an image and send the captured image to the server cluster 102. The server cluster 102 executes the 3D reconstruction method, data processing method, and rendering method provided in the present application, and outputs the obtained image through the 3D reconstruction method to obtain a 3D Gaussian volume. The 3D Gaussian volume can also be optimized through the data processing method, and the optimized 3D Gaussian volume is rendered through the rendering method to obtain a rendering result, which is sent to the terminal device 101. Finally, the terminal device 101 displays the rendering result.

[0171] Exemplarily, in an embodiment of the present application, image acquisition can also be performed only by the terminal device 101, and the image can be converted into a three-dimensional Gaussian body through the three-dimensional reconstruction method provided by the present application, and the parameters of the three-dimensional Gaussian body can be optimized to obtain an optimized three-dimensional Gaussian body, and the optimized three-dimensional Gaussian body can be rendered through the rendering method provided by the present application, and the rendering result can be displayed.

[0172] Among them, the terminal device 101 can be a mobile phone, augmented reality (AR) / virtual reality (VR) glasses, a tablet computer, a laptop computer or a TV, etc., and is not limited here.

[0173] Existing solutions for large-scale scene reconstruction primarily use NeRF. However, this approach requires significant computational resources and time for training, and cannot balance rendering quality and speed. Furthermore, while the 3DGS approach can construct high-quality scenes and achieve real-time rendering, it is computationally intensive due to the large number of 3D Gaussian volumes involved in rendering, making it unsuitable for large-scale scene reconstruction.

[0174] To address these current challenges, the present invention proposes a 3D reconstruction method, data processing method, rendering method, and device. These methods can divide a large scene into multiple small scenes (also referred to as regions), and then use 3DGS to perform scene reconstruction and real-time rendering for these small scenes, ultimately achieving high-quality reconstruction of the large scene. A large scene can typically be an indoor scene area exceeding 1,000 square meters, or a scene that requires regional division.

[0175] It should be understood that the present application can be applied not only to the reconstruction and rendering of large scenes, but also to the reconstruction and rendering of small scenes. For small scenes, the small scene can be divided into multiple areas first, and the 3DGS method can be used to reconstruct the scene for each area. There is no specific limitation.

[0176] The following describes the method flow provided by this application in conjunction with the aforementioned system architecture.

[0177] It is understandable that the 3D reconstruction method and rendering method provided by the present application may include a training phase and an inference phase. The training phase may be referred to as a 3D reconstruction method, and the inference phase may be referred to as a process of a rendering method.

[0178] 1. Training phase.

[0179] In an embodiment of the present application, the scene can be divided into multiple areas, and then the 3DGS method is used to reconstruct the scene based on each area. The process of scene reconstruction based on scene block is introduced below.

[0180] Referring to FIG2 , a flow chart of a three-dimensional reconstruction method provided by the present application is described below.

[0181] 201. Acquire multiple images of a scene;

[0182] Typically, 3D scene reconstruction involves an environment in three-dimensional space, such as an indoor room, an outdoor building, or a natural environment. The scene can be divided into multiple regions, and subsequent reconstruction can be performed based on the images in each region. This allows for block-by-block reconstruction, avoiding memory constraints or poor reconstruction accuracy caused by an excessive number of 3D Gaussian volumes, thereby improving scene reconstruction quality.

[0183] Among them, multiple images in the scene can be captured by a shooting device, and image extraction can also be performed on the video captured by the shooting device to obtain multiple images in the scene. The specific details are not limited here.

[0184] 202. Divide the multiple images into multiple regions to obtain at least one image in each region;

[0185] Before dividing the multiple images, the posture corresponding to each image, that is, the posture of the shooting device in the scene, can also be obtained, so that the images can be divided into corresponding areas according to the posture of the images.

[0186] Optionally, the multiple images may be divided according to the pose of each image in the scene, so that at least one image is obtained in each area of ​​the scene, and each area in the scene may be represented by at least one image.

[0187] Optionally, the projection area of ​​each region on the camera corresponding to each image can be determined based on the pose corresponding to each image. Due to the visibility of the camera, the visibility angle of the camera can be fully utilized. Based on the projection area of ​​each region on each camera, cameras with projection areas exceeding a preset value can be screened out. Subsequently, the images corresponding to these cameras are divided into corresponding regions, thereby completing the division of multiple images in the scene.

[0188] Among them, the bounding box of each area can be used as the projection area of ​​each shooting device. The bounding box is a cube with the area of ​​the area as the base and infinite height, as shown in Figure 3, so as to fully consider the surface of the area and the airspace area. By shooting the airspace area of ​​the area, the three-dimensional Gaussian body floating in the air can be deleted to suppress the generation of floating objects in the air, thereby improving the quality of scene reconstruction.

[0189] Exemplarily, the scene can be randomly divided into multiple areas, such as four areas A, B, C and D. The scene can also be evenly divided into multiple areas in a checkerboard manner, which is not limited here, as shown in Figure 4 (a); after obtaining multiple areas, usually, only the image in the area is acquired based on the shooting device in the area. For example, the image in area B can be acquired based on the shooting device in area B, as shown in Figure 4 (b); however, the shooting device has visibility, and the shooting device in area A, C or D may also capture area B. Therefore, the shooting device with a projection area greater than a preset value can be screened out based on the projection area of ​​area B in each shooting device in the scene, as shown in Figure 4 (c), and the image corresponding to the screened shooting device is divided into area B, as shown in Figure 4 (d).

[0190] In the embodiment of the present application, the visibility of the shooting equipment can be fully utilized to divide multiple images in the scene into corresponding areas, and the surface and airspace areas of the objects are fully considered to suppress the generation of floating objects in the air, thereby improving the quality of scene reconstruction.

[0191] 203. Obtain at least one three-dimensional Gaussian volume for each region according to at least one image of each region.

[0192] Typically, a three-dimensional Gaussian volume can be used to represent a scene. The three-dimensional Gaussian volume is the smallest element in a 3DGS scene representation. Therefore, a three-dimensional Gaussian volume of each region can be obtained based on the image in each region to represent the constructed scene.

[0193] Specifically, the image in the acquired scene can be analyzed by using a structure from motion (SFM) algorithm to obtain point cloud data corresponding to the image, and the corresponding three-dimensional Gaussian volume can be obtained by creating a three-dimensional Gaussian function.

[0194] 204. Combine at least one three-dimensional Gaussian volume in each region to obtain a combined three-dimensional Gaussian volume.

[0195] After obtaining the three-dimensional Gaussian volume in each region, the three-dimensional Gaussian volume in each region can be combined to obtain multiple combined three-dimensional Gaussian volumes, and the multiple combined three-dimensional Gaussian volumes can be used to represent the scene to achieve scene reconstruction.

[0196] 5 , the present application provides a flow chart of another 3D reconstruction method, as described below.

[0197] 501. Acquire multiple point cloud data in the scene;

[0198] The point cloud data may be calculated by an SFM algorithm based on multiple images in the scene, or may be collected by a depth camera or an inertial sensor, and the specifics are not limited here.

[0199] 502. Divide the plurality of point cloud data according to the position of the shooting device in the scene to obtain regional point cloud data of each region;

[0200] Before classifying the multiple point cloud data in the scene, the pose of the camera in the scene can also be obtained. This pose can be calculated using an SFM algorithm based on the image in the scene, or can be acquired using a depth camera or an inertial sensor, the specific details of which are not limited here. In addition, the pose of the camera in the scene corresponds to the pose of the image corresponding to the camera.

[0201] After obtaining the position and posture of the shooting device, the plurality of point cloud data in the scene may be divided according to the position and posture of the shooting device, so that each area in the scene includes at least one point cloud data.

[0202] Specifically, the point cloud ratio of the number of regional point clouds of each area observed by the shooting device to the number of all point cloud data that the shooting device can observe can be determined based on the posture of the shooting device. When the point cloud ratio exceeds a preset value, the shooting device corresponding to each area is determined; subsequently, the point cloud data corresponding to each area can be obtained based on the shooting device of each area.

[0203] 503. Acquire multiple three-dimensional Gaussian volumes of each region based on the point cloud data of each region.

[0204] After obtaining the point cloud data for each region, a Gaussian distribution can be created based on the point cloud data to obtain multiple 3D Gaussian volumes for each region. These multiple 3D Gaussian volumes are used to represent each region. After obtaining the 3D Gaussian volumes for each region, the 3D Gaussian volumes for each region can be combined to obtain a combined 3D Gaussian volume, which can represent the scene.

[0205] The following introduces the optimization process of scene reconstruction for an area in the scene.

[0206] Referring to FIG6 , a flow chart of a three-dimensional reconstruction method provided by the present application is described below.

[0207] 601. Acquire an image in a region;

[0208] Among them, the area can be any one of multiple areas in the scene, the image in the area can be captured by a shooting device, and image extraction can be performed on the video captured by the shooting device to obtain multiple images in the scene. The specific details are not limited here.

[0209] 602. Acquire multiple three-dimensional Gaussian volumes in the region according to the image in the region;

[0210] After obtaining the image in the area, the point cloud data of the image can be obtained through the SFM algorithm. Based on the point cloud data, a Gaussian distribution is created to obtain multiple three-dimensional Gaussian volumes in the area. The three-dimensional Gaussian volumes are used to preliminarily represent the area. The three-dimensional Gaussian volumes can be subsequently optimized to obtain a higher quality area.

[0211] 603. Optimize the parameters of the three-dimensional Gaussian body to obtain an optimized three-dimensional Gaussian body.

[0212] Usually, due to the different exposure changes of different images, artifacts will be generated after scene reconstruction. To solve the current problem, the exposure changes of different pictures can be separated, thereby reducing the artifact problem caused by exposure transformation and improving rendering quality.

[0213] Specifically, by comparing and analyzing the differences between the first rendered image and the sample image, and the differences between the second rendered image and the sample image, the parameters of the 3D Gaussian volume can be optimized to obtain an optimized 3D Gaussian volume. The optimized 3D Gaussian volume is then rendered to obtain the region. The rendered images of each region are then combined to obtain a scene, so that the scene's apparent brightness is consistent. The second rendered image can be obtained by analyzing the first rendered image, and the first rendered image can be obtained by rendering the obtained 3D Gaussian volume.

[0214] Optionally, after obtaining the first rendered image, the appearance latent variable can be combined and analyzed through a convolutional neural network to obtain a second rendered image, and the appearance latent variable is used to identify the first rendered image; then, the parameters of the three-dimensional Gaussian body can be optimized by comparing and analyzing the differences between the first rendered image and the sample image, as well as the differences between the second rendered image and the sample image, to obtain an optimized three-dimensional Gaussian body.

[0215] Optionally, the first rendered image and the appearance latent variables can be analyzed by a convolutional neural network to obtain a second rendered image; and according to the difference between the second rendered image and the sample image, the parameters of the three-dimensional Gaussian body are optimized to obtain an optimized three-dimensional Gaussian body.

[0216] The first rendered image may be downsampled to obtain a temporary rendered image, and the temporary rendered image and the appearance latent variables may be analyzed by a convolutional neural network to obtain a second rendered image.

[0217] Optionally, in addition to analyzing the first rendered image and the appearance latent variables through a neural network model to obtain the second rendered image, the parameters used to express the color in the three-dimensional Gaussian body can be analyzed according to the appearance latent variables to obtain the second rendered image, or other methods can be used in combination with the appearance latent variables for analysis to obtain the second rendered image, which is not limited here.

[0218] Optionally, a first loss function between the first rendered image and the sample image and a second loss function between the second rendered image and the sample image may be calculated, wherein both the first loss function and the second loss function may satisfy the formula L = (1-λ)L1 + λL D-SSIM , where L1 loss usually refers to the mean absolute error loss, L D-SSIM is the structural similarity loss between the first rendered image or the second rendered image and the sample image, and λ is a weight. Subsequently, the parameters of the 3D Gaussian volume can be optimized according to the first and second loss functions until a preset condition is satisfied, thereby obtaining an optimized 3D Gaussian volume. This optimized 3D Gaussian volume is used to represent the region.

[0219] Optionally, a second loss function between the second rendered image and the sample image may be used, where the second loss function satisfies the aforementioned formula L=(1-λ)L1+λL D-SSIM Subsequently, the parameters of the three-dimensional Gaussian body are optimized according to the second loss function until the preset conditions are met, thereby obtaining an optimized three-dimensional Gaussian body.

[0220] The parameters of the three-dimensional Gaussian body include the center (position) μ, opacity α, three-dimensional covariance matrix ∑, and color c. By adjusting one or more of μ, α, ∑, or c, the brightness of the image rendered according to the optimized three-dimensional Gaussian body is made consistent. The color c can be represented by spherical harmonics or RGB values ​​to present a perspective-dependent appearance. All parameters of the three-dimensional Gaussian body are learnable and can be optimized through backpropagation. Therefore, the parameters of the three-dimensional Gaussian body can be updated according to the first loss function and the second loss function.

[0221] Optionally, a fully connected network can be constructed to analyze the posture of the shooting device to obtain appearance-related parameters. The appearance-related parameters can be used to adjust the exposure parameters of the image after rendering based on the three-dimensional Gaussian body. The appearance-related parameters may include exposure change parameters, color parameters or other parameters related to the appearance, which are not limited here; based on the appearance-related parameters, the parameters of the three-dimensional Gaussian body are optimized.

[0222] Optionally, appearance-related parameters can be obtained based on the appearance latent variables; then, the parameters of the three-dimensional Gaussian body can be optimized based on the appearance-related parameters. Since the appearance latent variables can be used to identify an image rendered based on the three-dimensional Gaussian body, and since the three-dimensional Gaussian body corresponds to an image captured by a camera, the appearance latent variables can also be used to identify the image corresponding to the camera. The appearance-related parameters can be used to adjust the exposure parameters of an image rendered based on the three-dimensional Gaussian body. Therefore, the appearance-related parameters can also be obtained based on the appearance latent variables. The image corresponding to the camera is determined based on the appearance latent variables, and the appearance-related parameters can then be determined based on the image.

[0223] Specifically, according to the numerical value of the obtained appearance-related parameters, one or more of μ, α, ∑ or c in the parameters of the three-dimensional Gaussian body can be directly adjusted to make the appearance of the optimized three-dimensional Gaussian body consistent after rendering. If the appearance-related parameter at this time is an exposure change parameter, the brightness of the optimized three-dimensional Gaussian body after rendering can be made consistent.

[0224] The appearance-related parameters may be global parameters or pixel-by-pixel parameters. When the appearance-related parameters are global parameters, the appearance of the image may be adjusted frame by frame according to the appearance-related parameters. When the appearance-related parameters are pixel-by-pixel parameters, the appearance of the image may be adjusted pixel by pixel according to the appearance-related parameters.

[0225] In an embodiment of the present application, the parameters of the three-dimensional Gaussian body can be optimized by taking into account the loss function between the second rendered image and the sample image. The second rendered image can be an image obtained by adjusting the rendered image in combination with the characteristic parameters, and the appearance parameters of the second rendered image are consistent. Therefore, by comparing and analyzing the loss between the second rendered image and the sample image, the parameters of the three-dimensional Gaussian body can be optimized to ensure a consistent appearance after rendering the optimized three-dimensional Gaussian body, thereby reducing artifacts and improving rendering quality.

[0226] Referring to FIG7 , a flow chart of a data processing method provided by the present application is shown as follows.

[0227] 701. Obtain appearance latent variables;

[0228] The appearance latent variable can be used to identify a rendered image, which is obtained by rendering a 3D Gaussian volume in the region. Furthermore, the appearance latent variable corresponds to the camera in the region and can include the camera ID. Since the 3D Gaussian volume corresponding to the first rendered image is obtained based on the image corresponding to the camera, the appearance latent variable can also be used to identify the image corresponding to the camera.

[0229] 702. Optimize the parameters of the three-dimensional Gaussian body according to the appearance latent variables to obtain an optimized three-dimensional Gaussian body.

[0230] In the embodiment of the present application, since the appearance latent variable corresponds to the shooting device and the image corresponding to the shooting device can be identified, the appearance-related parameters can be determined based on the appearance latent variable, and then the parameters of the three-dimensional Gaussian body can be optimized based on the appearance-related parameters.

[0231] Optionally, appearance-related parameters can be determined based on the acquired appearance latent variables, and the appearance-related parameters are used to adjust the appearance parameters of the image after three-dimensional Gaussian rendering; then, the parameters of the three-dimensional Gaussian can be optimized based on the appearance-related parameters so that the brightness of the image rendered according to the optimized three-dimensional Gaussian is consistent.

[0232] Optionally, the three-dimensional Gaussian body in the area can be rendered to obtain a first rendered image; then, the first rendered image and the appearance latent variables can be analyzed by a neural network model to obtain a second rendered image. It should be understood that, optionally, the parameters used to express the color in the three-dimensional Gaussian body can also be analyzed according to the appearance latent variables to obtain a second rendered image, or other methods can be used in combination with the appearance latent variables to analyze and obtain the second rendered image, which is not limited here; after obtaining the first rendered image and the second rendered image, the parameters of the three-dimensional Gaussian body can be optimized by comparing and analyzing the differences between the first rendered image and the sample image and the differences between the second rendered image and the sample image to obtain an optimized three-dimensional Gaussian body. The specific process of rendering the parameters of the three-dimensional Gaussian body according to the rendered image and the appearance latent variables is similar to the content described in step 603 in Figure 6 above, and will not be repeated here.

[0233] Optionally, based on the rendered first image, a neural network model can be used to analyze the first rendered image and appearance latent variables to obtain a second rendered image. The second rendered image can also be obtained by analyzing color parameters in the three-dimensional Gaussian volume based on the appearance latent variables. Alternatively, other methods combined with appearance latent variables can be used to analyze the second rendered image, without limitation. Subsequently, based on the difference between the second rendered image and the sample image, the parameters of the three-dimensional Gaussian volume are optimized to obtain an optimized three-dimensional Gaussian volume.

[0234] 2. Reasoning stage.

[0235] Referring to FIG8 , a flowchart of a rendering method provided in an embodiment of the present application is shown as follows.

[0236] 801. Determine an area of ​​the shooting device according to the position of the shooting device;

[0237] The position and posture of the shooting device may be obtained from other devices, or may be calculated using an SFM algorithm based on images captured by the shooting device, and the specifics are not limited here.

[0238] After obtaining the position and posture of the camera, the region to which the camera belongs can be determined based on the position and posture information of the camera. It should be understood that the region can be obtained by dividing a large scene or a small scene, and the specific details are not limited here. After determining the scene region to which the camera belongs, subsequent block rendering can be performed according to different scene regions.

[0239] 802. Determine a three-dimensional Gaussian volume in an area of ​​the shooting device according to the area of ​​the shooting device.

[0240] An image of the area can be acquired, point cloud data of the area can be obtained using an SFM algorithm, and Gaussian mapping can be performed on the point cloud data to obtain a three-dimensional Gaussian volume in the area. The image of the area can be obtained by capturing an image using a camera, or by extracting an image based on a video captured by the camera, and the specifics are not limited here.

[0241] 803. Rendering is performed based on the three-dimensional Gaussian volume of each region to obtain a rendered image of each region;

[0242] After obtaining the three-dimensional Gaussian volume in each region, each region can be processed in parallel in blocks to obtain a rendered image of each region.

[0243] Specifically, the three-dimensional Gaussian volume in the area can be divided into small triangular fragments by the renderer so that it can better adapt to different tile (area) sizes or screen resolutions of the terminal device. A tile is an area on the screen of the terminal device, which can be a pixel or a combination of multiple pixels; then, each triangular fragment is mapped to the corresponding tile on the screen; after mapping to the tile, the position, depth value, etc. of each triangular fragment on the screen can be calculated by fast rasterization (tile-based rasterizer), and each small triangular fragment can be converted into a pixel on the screen; during the rasterization process, a depth test is performed to determine whether to draw the pixel by comparing the depth value of the triangular fragment with the depth value of the corresponding pixel, so as to ensure that the occlusion relationship is correctly handled and the accuracy of the rendering result is improved; finally, the color value of each pixel is calculated according to the properties of the triangular fragment (such as color, texture, etc.), so as to achieve rendering according to the three-dimensional Gaussian volume to obtain a rendered image of each area.

[0244] 804. Combine the rendered images of each region to obtain a target rendered image.

[0245] After obtaining a rendered image for each region, each rendered image can be combined to obtain a target rendered image, which can be used to represent the target scene. It should be understood that in the embodiments of the present application, after rendering the three-dimensional Gaussian volume in the region according to the position of the camera to obtain a rendered image, the rendered images of each region do not need to be combined. Instead, only the three-dimensional Gaussian volume in the currently visible region can be rendered according to the position of the camera to obtain the current rendered image.

[0246] In an embodiment of the present application, a region partitioning strategy can be used to render multiple regions obtained by dividing the scene, reducing the number of three-dimensional Gaussian bodies rendered each time, thereby increasing the rendering speed to achieve real-time rendering.

[0247] Referring to FIG9 , a flowchart of another rendering method provided in an embodiment of the present application is shown as follows.

[0248] 901. Determine an area of ​​the shooting device according to the posture of the shooting device;

[0249] In the embodiment of the present application, step 901 is similar to step 801 described in FIG8 , and the details are not repeated here.

[0250] 902. Determine, based on the area of ​​the shooting device, a three-dimensional Gaussian volume visible in the area;

[0251] After the scene is divided into multiple areas, the visible three-dimensional Gaussian body in the area can be determined based on the marking information of the three-dimensional Gaussian body in the area. As shown in Figure 10, for the indoor scene area, since there may be walls, for the shooting device shown in Figure 10, the three-dimensional Gaussian body in the area numbered 5 is not visible to the shooting device due to the obstruction of the wall.

[0252] Among them, the marking information includes whether the three-dimensional Gaussian body is visible in the current area. Therefore, the three-dimensional Gaussian body that is invisible in the area can be deleted according to the marking information of the three-dimensional Gaussian body, thereby obtaining the three-dimensional Gaussian body that is visible in the area. It can be understood that in the embodiment of the present application, deleting the three-dimensional Gaussian body that is invisible in the area means that when rendering the three-dimensional Gaussian body in the area, the three-dimensional Gaussian body that is invisible in the area is excluded, and the three-dimensional Gaussian body that is invisible in the area is not rendered, and it does not mean that there is no three-dimensional Gaussian body that is invisible to the area. The three-dimensional Gaussian body that is invisible to the area may be visible to other areas, and can be rendered to obtain rendered images of other areas.

[0253] 903. Rendering is performed based on the three-dimensional Gaussian volume visible in each region to obtain a rendered image of each region;

[0254] After obtaining the visible three-dimensional Gaussian volume in each region, the rendering method described in step 803 in FIG. 8 may be used to project the visible three-dimensional Gaussian volume into a two-dimensional image space for rendering to obtain a rendered image of each region.

[0255] 904. Combine the rendered images of each region to obtain a target rendered image.

[0256] In the embodiment of the present application, step 904 is similar to step 804 described in FIG8 , and the details are not repeated here. It should be understood that in the embodiment of the present application, rendering can be performed based on the three-dimensional Gaussian volume visible in the region to obtain a rendered image under the current camera perspective, and the rendered images of each region do not need to be combined.

[0257] In an embodiment of the present application, only the three-dimensional Gaussian bodies visible in the area can be rendered, which reduces the number of three-dimensional Gaussian bodies to be rendered, thereby improving the rendering speed, and reducing storage space by deleting the three-dimensional Gaussian bodies that are not visible in the area.

[0258] Referring to FIG. 11 , a flowchart of another rendering method provided in an embodiment of the present application is shown as follows.

[0259] 1101. Determine a target three-dimensional Gaussian body from the three-dimensional Gaussian bodies according to label information of the three-dimensional Gaussian bodies in the region;

[0260] Typically, when rendering the area corresponding to the shooting device according to the posture of the shooting device, all three-dimensional Gaussian bodies will be rendered. However, not all three-dimensional Gaussian bodies are visible to the area. In order to improve rendering efficiency, the target three-dimensional Gaussian body can be determined from the three-dimensional Gaussian body based on the marking information of the three-dimensional Gaussian body. The target three-dimensional Gaussian body is the three-dimensional Gaussian body recognized by the shooting device in the area, that is, the three-dimensional Gaussian body visible to the area, so that only the three-dimensional Gaussian body visible to the area can be rendered subsequently, reducing the number of three-dimensional Gaussian bodies rendered, thereby improving rendering efficiency.

[0261] Specifically, a visible three-dimensional Gaussian body in the region may be used as a target three-dimensional Gaussian body according to the marking information of the three-dimensional Gaussian body, wherein the marking information includes information on whether the three-dimensional Gaussian body is visible in the current region.

[0262] 1102. Render the target three-dimensional Gaussian volume to obtain a rendered image of the region.

[0263] After obtaining the target three-dimensional Gaussian body of the area, only the target three-dimensional Gaussian body can be rendered. When the shooting device switches its posture, the three-dimensional Gaussian body corresponding to the area can be loaded according to the area corresponding to the current posture of the shooting device to achieve dynamic rendering based on regional visibility.

[0264] In an embodiment of the present application, the target three-dimensional Gaussian body is rendered to obtain a rendered image by using the rendering method described in step 803 in Figure 8 above, projecting the target three-dimensional Gaussian body into a two-dimensional image space for rendering to obtain a rendered image of the area.

[0265] Referring to FIG. 12 , a flowchart of another rendering method provided in an embodiment of the present application is shown as follows.

[0266] 1201. Determine an area of ​​the shooting device according to the posture of the shooting device;

[0267] 1202. Determine a three-dimensional Gaussian volume in an area of ​​the shooting device according to the area of ​​the shooting device.

[0268] In the embodiment of the present application, step 1201 and step 1202 are similar to step 801 and step 802 described in FIG8 , and the details are not repeated here.

[0269] 1203. Determine a target three-dimensional Gaussian body according to the depth and projection area corresponding to the three-dimensional Gaussian body in each region;

[0270] After obtaining the three-dimensional Gaussian bodies in the region, the three-dimensional Gaussian bodies can be screened according to their depths and projected areas to obtain target three-dimensional Gaussian bodies.

[0271] Optionally, the depth of the 3D Gaussian volume from the camera device can be calculated, and based on this depth, a 3D Gaussian volume to be deleted from the 3D Gaussian volumes or a 3D Gaussian volume to be rendered from the 3D Gaussian volumes of the region can be determined. A first preset threshold for the projected area of ​​the 3D Gaussian volume can also be determined based on this depth. Based on this first preset threshold, 3D Gaussian volumes with projected areas less than the first preset threshold are deleted from the 3D Gaussian volumes to be deleted, thereby obtaining a target 3D Gaussian volume. Alternatively, 3D Gaussian volumes with projected areas greater than or equal to the first preset threshold are selected from the 3D Gaussian volumes to be rendered as the target 3D Gaussian volume.

[0272] For example, as shown in FIG13 , based on the depth of the three-dimensional Gaussian body, the depth can be divided into multiple segments L0, L1, L2, ..., Ln. In the L1 segment, based on the index number of the three-dimensional Gaussian body within the depth range, the three-dimensional Gaussian body with an index number multiple of 2 can be used as the three-dimensional Gaussian body to be deleted, and the three-dimensional Gaussian body with a projected area less than a first preset threshold can be screened out from the three-dimensional Gaussian body to be deleted; in the L2 segment, the three-dimensional Gaussian body with an index number multiple of 2 or 3 can be used as the three-dimensional Gaussian body to be deleted, and the three-dimensional Gaussian body with a projected area less than the first preset threshold can be screened out from the three-dimensional Gaussian body to be deleted. It should be understood that in the embodiment of the present application, the three-dimensional Gaussian body that meets the preset conditions in different depth ranges can also be deleted based solely on the depth of the three-dimensional Gaussian body. For example, the three-dimensional Gaussian body with an index number multiple of 2 in the L1 segment can be deleted, and the three-dimensional Gaussian body with an index number multiple of 2 or 3 in the L2 segment can be deleted to obtain the target three-dimensional Gaussian body. In addition, the three-dimensional Gaussian volume whose projected area is smaller than the preset threshold may be deleted based on the preset threshold, thereby obtaining the target three-dimensional Gaussian volume.

[0273] Optionally, a preset number of 3D Gaussian volumes may be randomly deleted to obtain a target 3D Gaussian volume, or a preset number of 3D Gaussian volumes may be randomly selected as the target 3D Gaussian volume. For example, half of the 3D Gaussian volumes in the L1 segment may be deleted or half of the 3D Gaussian volumes in the L1 segment may be selected as the target 3D Gaussian volume.

[0274] Optionally, the volumes of all three-dimensional Gaussian bodies in the scene can be pre-calculated, and the three-dimensional Gaussian bodies whose volumes exceed a second preset threshold can be deleted to obtain a target three-dimensional Gaussian body, or the three-dimensional Gaussian body whose volume is less than or equal to the second preset threshold can be used as the target three-dimensional Gaussian body.

[0275] Alternatively, a 3D Gaussian volume to be deleted may be randomly determined, and then, among the 3D Gaussian volumes to be deleted, those with a volume exceeding a third preset threshold may be deleted to obtain a target 3D Gaussian volume. Alternatively, a 3D Gaussian volume to be rendered may be randomly determined, and then, among the 3D Gaussian volumes to be rendered, those with a volume less than or equal to the third preset threshold may be used as the target 3D Gaussian volume.

[0276] In the embodiment of the present application, the three-dimensional Gaussian body that meets the preset conditions is deleted in order to filter out the target three-dimensional Gaussian body from the three-dimensional Gaussian bodies in the region, so that subsequent rendering can only render the target three-dimensional Gaussian body, thereby improving rendering efficiency.

[0277] 1204. Render the target three-dimensional Gaussian volume in each region to obtain a rendered image of each region.

[0278] After obtaining the target three-dimensional Gaussian volume, the rendering method described in step 803 in FIG. 8 may be used to project the target three-dimensional Gaussian volume in each region into a two-dimensional image space for rendering, thereby obtaining a rendered image of each region.

[0279] 1205. Combine the rendered images of each region to obtain a target rendered image.

[0280] In the embodiment of the present application, step 1205 is similar to step 904 described in FIG. 9 , and the details are not repeated here.

[0281] Referring to FIG. 14 , a flowchart of another rendering method provided in an embodiment of the present application is shown as follows.

[0282] 1401. Determine an area of ​​the shooting device according to the position of the shooting device;

[0283] 1402. Determine, based on the area of ​​the shooting device, a three-dimensional Gaussian volume visible in the area;

[0284] In the embodiment of the present application, step 1401 and step 1402 are similar to step 901 and step 902 described in FIG. 9 , and the details are not repeated here.

[0285] 1403. Determine a target three-dimensional Gaussian body based on the depth and projection area of ​​the three-dimensional Gaussian body visible in each region.

[0286] After obtaining the visible three-dimensional Gaussian body in each area, the target three-dimensional Gaussian body can be determined according to the method of determining the target three-dimensional Gaussian body in step 1203 described in Figure 12 above, based on the depth and projection area corresponding to the visible three-dimensional Gaussian body. The specific implementation method is similar and will not be repeated here.

[0287] 1404. Render the target three-dimensional Gaussian volume in each region to obtain a rendered image of each region.

[0288] 1405. Combine the rendered images of each region to obtain a target rendered image.

[0289] In the embodiment of the present application, steps 1404 and 1405 are similar to steps 1204 and 1205 described in FIG12 , and are not described in detail here. Similarly, the rendered images of each region may not be combined, and only the region under the current viewing angle may be rendered according to the position of the camera.

[0290] The above describes the method flow provided by the present application. Based on the above method flow, the following describes the device provided by the present application.

[0291] Referring to FIG15 , a schematic diagram of the structure of a 3D reconstruction device provided by the present application includes:

[0292] An acquisition module 1501 is configured to acquire multiple images of a scene, where the scene includes multiple areas.

[0293] A division module 1502 is configured to divide the multiple images into multiple regions according to the poses of the multiple images corresponding to the shooting device, and obtain at least one image in each region;

[0294] The aforementioned acquisition module 1501 is further configured to acquire at least one three-dimensional Gaussian volume for each region based on at least one image of each region, where the at least one three-dimensional Gaussian volume is used to represent the region in the scene;

[0295] The combining module 1503 is configured to combine at least one three-dimensional Gaussian volume in each region to obtain a combined three-dimensional Gaussian volume, where the combined three-dimensional Gaussian volume represents the scene.

[0296] In one possible embodiment, the division module 1502 is specifically used to determine the projection information of each region in the shooting device corresponding to each image based on the posture of the shooting device corresponding to each image, and the projection information includes the projection area of ​​each region in the shooting device; according to the projection information of each region in the shooting device corresponding to each image, each image is divided into corresponding regions among the multiple regions to obtain at least one image of each region.

[0297] In a possible implementation, the device may further include:

[0298] The aforementioned acquisition module 1501 is further used to acquire a plurality of point cloud data corresponding to each image;

[0299] The aforementioned division module 1502 is also used to divide multiple point cloud data according to the posture corresponding to each image to obtain a regional point cloud for each area. The regional point cloud includes at least one point cloud data. The regional point cloud of each area is used to obtain a three-dimensional Gaussian body for each area.

[0300] In one possible implementation, the segmentation module 1502 is specifically configured to determine, based on the pose corresponding to each image, a point cloud ratio of the camera corresponding to each image, where the point cloud ratio is the ratio of the number of point clouds in each region observed by the camera to the total number of point cloud data observed by the camera. Based on the point cloud ratio of the camera corresponding to each image, the plurality of point cloud data are segmented into corresponding regions of the plurality of regions.

[0301] In one possible implementation, the division module 1502 is specifically used to: determine the shooting device in each area based on the point cloud ratio of the shooting device corresponding to each image; and divide the multiple point cloud data into corresponding areas in the multiple areas based on the shooting device in each area to obtain at least one point cloud data for each area.

[0302] In a possible implementation, the device may further include:

[0303] The optimization module 1504 is used to optimize the parameters of the three-dimensional Gaussian body to obtain an optimized three-dimensional Gaussian body. The parameters are used to adjust the appearance parameters of the three-dimensional Gaussian body after rendering.

[0304] In one possible implementation, the optimization module 1504 is specifically configured to render a three-dimensional Gaussian body to obtain a first rendered image; analyze the first rendered image and appearance latent variables using a neural network model to obtain a second rendered image, where the appearance latent variables are used to identify the first rendered image; and optimize the parameters of the three-dimensional Gaussian body based on the difference between the first rendered image and the sample image and the difference between the second rendered image and the sample image to obtain an optimized three-dimensional Gaussian body.

[0305] In a possible implementation, the optimization module 1504 is specifically configured to downsample the first rendered image to obtain a temporary rendered image; and analyze the temporary rendered image and appearance latent variables using a neural network model to obtain a second rendered image.

[0306] In one possible implementation, the optimization module 1504 is specifically configured to calculate a first loss function between the first rendered image and the sample image; calculate a second loss function between the second rendered image and the sample image; and optimize the parameters of the three-dimensional Gaussian body based on the first loss function and the second loss function to obtain an optimized three-dimensional Gaussian body.

[0307] In one possible implementation, the optimization module 1504 is specifically configured to perform analysis based on the posture through a fully connected network to obtain appearance-related parameters, where the appearance-related parameters are used to adjust the appearance parameters of the image after the three-dimensional Gaussian body is rendered; and optimize the parameters of the three-dimensional Gaussian body based on the appearance-related parameters.

[0308] Referring to FIG. 16 , a schematic diagram of the structure of a data processing device 1600 provided by the present application includes:

[0309] An acquisition module 1601 is used to acquire appearance latent variables, where the appearance latent variables are used to identify a rendered image obtained by rendering a three-dimensional Gaussian volume in a region;

[0310] The optimization module 1602 is used to optimize the parameters of the three-dimensional Gaussian body according to the appearance latent variables to obtain an optimized three-dimensional Gaussian body. The optimized three-dimensional Gaussian body is used to represent the region.

[0311] In one possible implementation, the aforementioned optimization module 1602 is specifically used to: render the three-dimensional Gaussian body in the area to obtain a first rendered image; analyze the first rendered image and the appearance latent variables to obtain a second rendered image; and optimize the parameters of the three-dimensional Gaussian body based on the difference between the first rendered image and the sample image and the difference between the second rendered image and the sample image to obtain an optimized three-dimensional Gaussian body.

[0312] In one possible implementation, the aforementioned optimization module 1602 is specifically used to: render the three-dimensional Gaussian body in the area to obtain a first rendered image; analyze the first rendered image and the appearance latent variables to obtain a second rendered image; and optimize the parameters of the three-dimensional Gaussian body based on the difference between the second rendered image and the sample image to obtain an optimized three-dimensional Gaussian body.

[0313] In a possible implementation, the aforementioned optimization module 1602 is specifically configured to: downsample the first rendered image to obtain a temporary rendered image; and analyze the temporary rendered image and feature parameters using a neural network model to obtain a second rendered image.

[0314] In one possible implementation, the aforementioned optimization module 1602 is specifically used to: calculate a first loss function between the first rendered image and the sample image; calculate a second loss function between the second rendered image and the sample image; and optimize the parameters of the three-dimensional Gaussian body according to the first loss function and the second loss function to obtain an optimized three-dimensional Gaussian body.

[0315] In a possible implementation, the aforementioned optimization module 1602 is specifically configured to: calculate a second loss function between the second rendered image and the sample image; and optimize parameters of the three-dimensional Gaussian body according to the second loss function to obtain an optimized three-dimensional Gaussian body.

[0316] In a possible implementation, the aforementioned optimization module 1602 is specifically used to: determine appearance-related parameters based on appearance latent variables, where the appearance-related parameters are used to adjust the appearance parameters of the image after three-dimensional Gaussian rendering; and optimize the parameters of the three-dimensional Gaussian based on the appearance-related parameters.

[0317] Referring to FIG. 17 , a schematic diagram of the structure of a data processing device 1700 provided by the present application includes:

[0318] An acquisition module 1701 is configured to acquire a three-dimensional Gaussian volume of a region according to an image in the region, where the region is any one of multiple regions in the scene;

[0319] A rendering module 1702 is configured to render the three-dimensional Gaussian volume to obtain a first rendered image;

[0320] An analysis module 1703 is configured to analyze the first rendered image and feature parameters using a neural network model to obtain a second rendered image, wherein the feature parameters are used to adjust appearance parameters of the first rendered image;

[0321] The optimization module 1704 is configured to optimize the parameters of the 3D Gaussian volume according to the difference between the first rendered image and the sample image and the difference between the second rendered image and the sample image, to obtain an optimized 3D Gaussian volume, which is used to represent the region.

[0322] Referring to FIG. 18 , a schematic diagram of the structure of a rendering device 1800 provided by the present application includes:

[0323] An acquisition module 1801 is used to acquire the position and posture of the camera in a scene, where the scene is divided into multiple areas;

[0324] A determination module 1802 is configured to determine a three-dimensional Gaussian volume in an area corresponding to the camera according to the position of the camera;

[0325] A rendering module 1803 is used to render the three-dimensional Gaussian volume of each region to obtain a rendered image of each region;

[0326] The combining module 1804 is configured to combine the rendered images of each region to obtain a target rendered image.

[0327] In one possible implementation, the determination module 1802 is specifically used to determine the area corresponding to the shooting device based on the posture of the shooting device; based on the marking information of the three-dimensional Gaussian body in the area, delete the invisible three-dimensional Gaussian body in the area to obtain the three-dimensional Gaussian body in the area corresponding to the shooting device.

[0328] In one possible implementation, the rendering module 1803 is specifically used to calculate the depth of the three-dimensional Gaussian body of each area from the camera device; determine the target three-dimensional Gaussian body of each area based on the depth and the projected area of ​​the three-dimensional Gaussian body of each area; and render the target three-dimensional Gaussian body of each area to obtain a rendered image of each area.

[0329] In one possible implementation, the rendering module 1803 is specifically configured to determine, from the three-dimensional Gaussian bodies in each region according to the depth, a three-dimensional Gaussian body to be deleted in each region; determine, according to the depth, a first preset threshold value of a projected area of ​​the three-dimensional Gaussian body in each region; and, according to the three-dimensional Gaussian bodies to be deleted in each region and the first preset threshold value, delete the three-dimensional Gaussian bodies whose projected areas are smaller than the preset threshold value from the three-dimensional Gaussian bodies to be deleted, to obtain a target three-dimensional Gaussian body.

[0330] In one possible implementation, the rendering module 1803 is specifically configured to randomly delete a preset number of three-dimensional Gaussian bodies from the three-dimensional Gaussian body of each region to obtain a target three-dimensional Gaussian body of each region; and render the target three-dimensional Gaussian body of each region to obtain a rendered image of each region.

[0331] In one possible implementation, the rendering module 1803 is specifically configured to calculate a first volume of a three-dimensional Gaussian body for each region; based on the first volume, delete the three-dimensional Gaussian bodies whose first volume exceeds a second preset threshold to obtain a target three-dimensional Gaussian body; and render the target three-dimensional Gaussian body in each region to obtain a rendered image of each region.

[0332] In one possible implementation, the rendering module 1803 is specifically configured to randomly determine a three-dimensional Gaussian body to be deleted from the three-dimensional Gaussian body of each region; calculate the second volume of the three-dimensional Gaussian body of each region; delete the three-dimensional Gaussian body whose second volume exceeds a third preset threshold from the three-dimensional Gaussian body to be deleted to obtain a target three-dimensional Gaussian body; and render according to the target three-dimensional Gaussian body of each region to obtain a rendered image of each region.

[0333] Please refer to FIG. 19 , which is a schematic structural diagram of a rendering device 1900 provided in this application, as described below.

[0334] A determination module 1901 is configured to determine a target 3D Gaussian body from the 3D Gaussian bodies according to the label information of the 3D Gaussian bodies in the region, where the target 3D Gaussian body is the 3D Gaussian body recognized by the camera in the region;

[0335] The rendering module 1902 is used to render the target three-dimensional Gaussian volume to obtain a rendered image of the region.

[0336] In a possible implementation, the aforementioned determination module 1801 is specifically configured to: select a visible three-dimensional Gaussian body in the region as a target three-dimensional Gaussian body according to label information of the three-dimensional Gaussian body in the region.

[0337] Please refer to FIG. 20 , which is a schematic structural diagram of a rendering device 2000 provided in this application, as described below.

[0338] An acquisition module 2001 is configured to acquire a three-dimensional Gaussian volume of a region according to an image in the region;

[0339] a determination module 2002, configured to determine target three-dimensional Gaussian bodies of the region from the three-dimensional Gaussian bodies of the region based on one or more of the depth of the three-dimensional Gaussian body and the projected area or volume of the three-dimensional Gaussian body, wherein the number of the target three-dimensional Gaussian bodies is less than the number of the three-dimensional Gaussian bodies;

[0340] The rendering module 2003 is used to render the target three-dimensional Gaussian volume to obtain a rendered image of the region.

[0341] In a possible implementation, the aforementioned determination module 2002 is specifically used to: determine the target three-dimensional Gaussian body of the region from the three-dimensional Gaussian body of the region based on the depth and projection area of ​​the three-dimensional Gaussian body; and determine the target three-dimensional Gaussian body of the region from the three-dimensional Gaussian body of the region based on the volume of the three-dimensional Gaussian body.

[0342] In a possible implementation, the aforementioned determination module 2002 is specifically used to: determine the three-dimensional Gaussian body to be rendered in the region from the three-dimensional Gaussian body in the region according to the depth of the three-dimensional Gaussian body; determine a first preset value of the projected area of ​​the three-dimensional Gaussian body in the region according to the depth of the three-dimensional Gaussian body; and, based on the three-dimensional Gaussian body to be rendered in the region and the first preset threshold, select the three-dimensional Gaussian body in the three-dimensional Gaussian body to be rendered whose projected area is greater than or equal to the first preset value as the target three-dimensional Gaussian body.

[0343] In a possible implementation, the aforementioned determination module 2002 is specifically configured to: calculate a first volume of a three-dimensional Gaussian body in the region; and use a three-dimensional Gaussian body whose first volume is less than or equal to a second preset threshold as a target three-dimensional Gaussian body.

[0344] In a possible implementation, the aforementioned determination module 2002 is specifically used to: randomly determine a three-dimensional Gaussian body to be rendered from the three-dimensional Gaussian bodies in the region; calculate the second volume of the three-dimensional Gaussian body in the region; and use the three-dimensional Gaussian body in the three-dimensional Gaussian body to be rendered whose second volume is less than or equal to a third preset threshold as the target three-dimensional Gaussian body.

[0345] In a possible implementation, before rendering the target three-dimensional Gaussian body to obtain the rendered image of the region, the determination module 2002 is further configured to randomly determine a preset number of three-dimensional Gaussian bodies from the three-dimensional Gaussian bodies in the region as target three-dimensional Gaussian bodies.

[0346] In a possible implementation, the region is any one of the multiple regions in the aforementioned three-dimensional reconstruction device.

[0347] Please refer to FIG. 21 , which is a schematic structural diagram of another three-dimensional reconstruction device provided in the present application, as described below.

[0348] The 3D reconstruction apparatus may include a processor 2101 and a memory 2102. The processor 2101 and the memory 2102 are interconnected via a circuit. The memory 2102 stores program instructions and data.

[0349] The memory 2102 stores program instructions and data corresponding to the steps in FIG. 2 , FIG. 5 and FIG. 6 .

[0350] The processor 2101 is configured to execute the method steps performed by the three-dimensional reconstruction apparatus shown in any of the embodiments in FIG. 2 , FIG. 5 and FIG. 6 .

[0351] Optionally, the three-dimensional reconstruction apparatus may further include a transceiver 2103 for receiving or sending data.

[0352] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a program, which, when executed on a computer, enables the computer to execute the steps of the method described in the embodiments shown in Figures 2, 5 and 6 above.

[0353] Optionally, the three-dimensional reconstruction device shown in the aforementioned FIG. 21 is a chip.

[0354] An embodiment of the present application also provides a three-dimensional reconstruction device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to execute the method steps performed by the three-dimensional reconstruction device shown in any of the embodiments in Figures 2, 5 and 6 above.

[0355] Please refer to FIG. 22 , which is a schematic structural diagram of another data processing device provided in the present application, as described below.

[0356] The data processing device may include a processor 2201 and a memory 2202. The processor 2201 and the memory 2202 are interconnected via a circuit. The memory 2202 stores program instructions and data.

[0357] The memory 2202 stores program instructions and data corresponding to the steps in FIG. 7 .

[0358] The processor 2201 is configured to execute the method steps performed by the data processing device shown in the embodiment of FIG. 7 .

[0359] Optionally, the data processing device may further include a transceiver 2203 for receiving or sending data.

[0360] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a program, which, when executed on a computer, enables the computer to execute the steps of the method described in the embodiment shown in FIG. 7 .

[0361] Optionally, the data processing device shown in the aforementioned FIG. 22 is a chip.

[0362] An embodiment of the present application also provides a data processing device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to execute the method steps performed by the data processing device shown in the embodiment of Figure 7 above.

[0363] Please refer to FIG. 23 , which is a schematic structural diagram of another rendering device provided in this application, as described below.

[0364] The rendering device may include a processor 2301 and a memory 2302. The processor 2301 and the memory 2302 are interconnected via a circuit. The memory 2302 stores program instructions and data.

[0365] The memory 2302 stores program instructions and data corresponding to the steps in the aforementioned Figures 8, 9, 11, 12 and 14.

[0366] The processor 2301 is used to execute the method steps performed by the rendering device shown in Figures 8, 9, 11, 12 and 14 above.

[0367] Optionally, the rendering device may further include a transceiver 2303 for receiving or sending data.

[0368] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a program, which, when executed on a computer, enables the computer to execute the steps of the method described in the embodiments shown in the aforementioned Figures 8, 9, 11, 12 and 14.

[0369] Optionally, the rendering device shown in the aforementioned FIG. 18 is a chip.

[0370] An embodiment of the present application also provides a rendering device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to execute the method steps performed by the training device shown in the embodiments of Figures 8, 9, 11, 12 and 14 above.

[0371] The present application also provides a digital processing chip. The digital processing chip integrates circuits and one or more interfaces for implementing the functions of the above-mentioned processor 2101, processor 2201, and processor 2301, or processor 2101, processor 2201, and processor 2301. When the digital processing chip integrates a memory, the digital processing chip can perform the method steps of any one or more of the aforementioned embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory via a communication interface. The digital processing chip implements the actions performed by the three-dimensional reconstruction device or rendering device in the above-mentioned embodiments according to the program code stored in the external memory.

[0372] An embodiment of the present application also provides a computer program product, which, when running on a computer, enables the computer to execute the method steps described in the embodiments shown in Figures 2, 5, 6, 7, 8, 9, 11, 12 or 14.

[0373] The three-dimensional reconstruction device or rendering device provided in the embodiments of the present application can be a chip, and the chip includes: a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, a pin or a circuit. The processing unit can execute the computer-executable instructions stored in the storage unit to enable the chip in the server to execute the method described in the embodiments shown in Figures 2, 5, 6, 7, 8, 9, 11, 12 or 14. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0374] Specifically, the aforementioned processing unit or processor may be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0375] For example, see Figure 24, which is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip can be represented as a neural network processor NPU 2400. NPU 2400 is mounted on the host CPU (host CPU) as a coprocessor and assigned tasks by the host CPU. The core of the NPU is arithmetic circuit 2403, which is controlled by controller 2404 to extract matrix data from memory and perform multiplication operations.

[0376] In some implementations, the arithmetic circuit 2403 includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 2403 is a two-dimensional systolic array. The arithmetic circuit 2403 may also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 2403 is a general-purpose matrix processor.

[0377] For example, assume there are input matrix A, weight matrix B, and output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from weight memory 2402 and caches it on each PE in the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from input memory 2401 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 2408.

[0378] Unified memory 2406 is used to store input and output data. Weight data is directly transferred to weight memory 2402 via direct memory access controller (DMAC) 2405. Input data is also transferred to unified memory 2406 via DMAC.

[0379] The bus interface unit (BIU) 2410 is used for interaction between the AXI bus and the DMAC and instruction fetch buffer (IFB) 2409.

[0380] The bus interface unit 2410 (BIU) is used for the instruction fetch memory 2409 to obtain instructions from the external memory, and is also used for the storage unit access controller 2405 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0381] DMAC is mainly used to move input data in the external memory DDR to the unified memory 2406 or to move weight data to the weight memory 2402 or to move input data to the input memory 2401.

[0382] The vector calculation unit 2407 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0383] In some implementations, the vector calculation unit 2407 can store the processed output vector to the unified memory 2406. For example, the vector calculation unit 2407 can apply a linear function and / or a nonlinear function to the output of the operation circuit 2403, such as linear interpolation of the feature plane extracted by the convolution layer, or accumulate a vector of values ​​to generate an activation value. In some implementations, the vector calculation unit 2407 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 2403, for example, for use in a subsequent layer in a neural network.

[0384] An instruction fetch buffer 2409 connected to the controller 2404 is used to store instructions used by the controller 2404;

[0385] Unified memory 2406, input memory 2401, weight memory 2402, and instruction fetch memory 2409 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0386] Among them, the processor mentioned in any of the above places can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the program of the method of Figure 2, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, Figure 11, Figure 12 or Figure 14 above.

[0387] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0388] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and conciseness of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0389] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0390] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0391] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0392] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0393] The terms "first," "second," "third," "fourth," and the like (if any) in the specification and claims of this application and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0394] Finally, it should be noted that the above is only a specific implementation method of the present application, but the protection scope of the present application is not limited to this. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the protection scope of the present application.

Claims

1. A three-dimensional reconstruction method, characterized in that: include: Acquire a plurality of images of a scene, the scene comprising a plurality of regions; Dividing the plurality of images into the plurality of regions according to the positions of the photographing device corresponding to the plurality of images, and obtaining at least one image in each region; Acquire at least one three-dimensional Gaussian volume for each region according to at least one image of each region, wherein the at least one three-dimensional Gaussian volume is used to represent the region in the scene; At least one three-dimensional Gaussian volume in each region is combined to obtain a combined three-dimensional Gaussian volume, where the combined three-dimensional Gaussian volume represents the scene.

2. The method according to claim 1, characterized in that The step of dividing the plurality of images into the plurality of regions according to the positions of the photographing device corresponding to the plurality of images to obtain at least one image in each region includes: Determining, according to the posture of the shooting device corresponding to each image, projection information of each region in the shooting device corresponding to each image, wherein the projection information includes a projection area of each region in the shooting device; According to the projection information of each area in the shooting device corresponding to each image, each image is divided into corresponding areas among the multiple areas to obtain at least one image of each area.

3. The method according to claim 2, characterized in that The method further comprises: Acquire a plurality of point cloud data of each image; According to the posture of each image, the multiple point cloud data are divided to obtain a regional point cloud of each area, the regional point cloud includes at least one point cloud data, and the regional point cloud of each area is used to obtain a three-dimensional Gaussian body of each area.

4. The method according to claim 3, characterized in that The step of dividing the plurality of point cloud data according to the pose of each image to obtain a regional point cloud for each region includes: Determine, according to the pose of each image, a point cloud ratio of the shooting device corresponding to each image, where the point cloud ratio is a ratio of the number of point clouds in each area observed by the shooting device to the total number of point cloud data observed by the shooting device; The plurality of point cloud data are divided into corresponding areas among the plurality of areas according to a point cloud ratio of a shooting device corresponding to each image.

5. The method according to claim 4, characterized in that The dividing the plurality of point cloud data into corresponding areas among the plurality of areas according to the point cloud ratio of the shooting device corresponding to each image includes: determining the shooting device in each area according to the point cloud ratio of the shooting device corresponding to each image; According to the shooting equipment in each area, the plurality of point cloud data are divided into corresponding areas among the plurality of areas to obtain at least one point cloud data in each area.

6. The method according to any one of claims 1 or 5, characterized in that The method further comprises: Optimizing the parameters of the three-dimensional Gaussian body to obtain an optimized three-dimensional Gaussian body, wherein the parameters are used to adjust the appearance parameters of the three-dimensional Gaussian body after rendering.

7. The method according to claim 6, characterized in that Optimizing the parameters of the three-dimensional Gaussian body to obtain an optimized three-dimensional Gaussian body includes: Rendering the three-dimensional Gaussian volume to obtain a first rendered image; Analyzing the first rendered image and appearance latent variables using a neural network model to obtain a second rendered image, wherein the appearance latent variables are used to identify the first rendered image; Parameters of the three-dimensional Gaussian volume are optimized according to a difference between the first rendered image and the sample image and a difference between the second rendered image and the sample image to obtain the optimized three-dimensional Gaussian volume.

8. The method according to claim 7, characterized in that The step of analyzing the first rendered image and the appearance latent variables using a neural network model to obtain a second rendered image includes: downsampling the first rendered image to obtain a temporary rendered image; The temporary rendered image and the appearance latent variables are analyzed by the neural network model to obtain the second rendered image.

9. The method according to any one of claims 7 or 8, characterized in that Optimizing the parameters of the three-dimensional Gaussian volume according to the difference between the first rendered image and the sample image and the difference between the second rendered image and the sample image to obtain the optimized three-dimensional Gaussian volume includes: calculating a first loss function between the first rendered image and the sample image; calculating a second loss function between the second rendered image and the sample image; Optimizing the parameters of the three-dimensional Gaussian body according to the first loss function and the second loss function to obtain the optimized three-dimensional Gaussian body.

10. The method according to claim 6, characterized in that Optimizing the parameters of the three-dimensional Gaussian body to obtain an optimized three-dimensional Gaussian body includes: Analyzing the posture through a fully connected network to obtain appearance-related parameters, wherein the appearance-related parameters are used to adjust the appearance parameters of the image after the three-dimensional Gaussian volume is rendered; Optimizing parameters of the three-dimensional Gaussian volume according to the appearance-related parameters.

11. A data processing method, characterized in that: include: Obtaining appearance latent variables, where the appearance latent variables are used to identify a rendered image obtained by rendering a three-dimensional Gaussian volume in the region; According to the appearance latent variables, the parameters of the three-dimensional Gaussian body are optimized to obtain an optimized three-dimensional Gaussian body, and the optimized three-dimensional Gaussian body is used to represent the region.

12. The method according to claim 11, characterized in that Optimizing the parameters of the three-dimensional Gaussian body according to the appearance latent variables to obtain an optimized three-dimensional Gaussian body includes: Rendering the three-dimensional Gaussian volume in the region to obtain a first rendered image; Analyzing the first rendered image and the appearance latent variable to obtain a second rendered image; Parameters of the three-dimensional Gaussian volume are optimized according to a difference between the first rendered image and the sample image and a difference between the second rendered image and the sample image to obtain the optimized three-dimensional Gaussian volume.

13. The method according to claim 11, characterized in that Optimizing the parameters of the three-dimensional Gaussian body according to the appearance latent variables to obtain an optimized three-dimensional Gaussian body includes: Rendering the three-dimensional Gaussian volume in the region to obtain a first rendered image; Analyzing the first rendered image and the appearance latent variable to obtain a second rendered image; According to the difference between the second rendered image and the sample image, the parameters of the three-dimensional Gaussian volume are optimized to obtain the optimized three-dimensional Gaussian volume.

14. The method according to claim 11, characterized in that Optimizing the parameters of the three-dimensional Gaussian body according to the appearance latent variables to obtain an optimized three-dimensional Gaussian body includes: Determining appearance-related parameters according to the appearance latent variables, wherein the appearance-related parameters are used to adjust the appearance parameters of the image after the three-dimensional Gaussian volume rendering; Optimizing parameters of the three-dimensional Gaussian volume according to the appearance-related parameters.

15. A data processing method, characterized in that: include: Acquire a three-dimensional Gaussian volume of the region according to an image in the region, where the region is any one of multiple regions in the scene; Rendering the three-dimensional Gaussian volume to obtain a first rendered image; Analyzing the first rendered image and feature parameters using a neural network model to obtain a second rendered image, wherein the feature parameters are used to adjust appearance parameters of the first rendered image; Optimizing parameters of the three-dimensional Gaussian volume according to differences between the first rendered image and the sample image and differences between the second rendered image and the sample image to obtain an optimized three-dimensional Gaussian volume, wherein the optimized three-dimensional Gaussian volume is used to represent the region.

16. A rendering method, characterized in that: include: Obtaining a position of a camera in a scene, where the scene is divided into multiple areas; Determining a three-dimensional Gaussian volume in an area corresponding to the shooting device according to the position and posture of the shooting device; Rendering the three-dimensional Gaussian volume of each region to obtain a rendered image of each region; The rendered images of each region are combined to obtain a target rendered image.

17. The method according to claim 16, characterized in that The determining, according to the posture of the shooting device, a three-dimensional Gaussian volume in an area corresponding to the shooting device includes: Determining an area corresponding to the shooting device according to the posture of the shooting device; According to the marking information of the three-dimensional Gaussian bodies in the area, the invisible three-dimensional Gaussian bodies in the area are deleted to obtain the three-dimensional Gaussian bodies in the area corresponding to the shooting device.

18. The method according to any one of claims 16 or 17, characterized in that Rendering the three-dimensional Gaussian volume of each region to obtain a rendered image of each region includes: Calculate the depth of the three-dimensional Gaussian volume of each area from the camera device; determining a target three-dimensional Gaussian body for each region according to the depth and the projection area of the three-dimensional Gaussian body for each region; The target three-dimensional Gaussian volume of each region is rendered to obtain a rendered image of each region.

19. The method according to claim 18, characterized in that Determining a target three-dimensional Gaussian body for each region according to the depth and the projection area of the three-dimensional Gaussian body for each region includes: determining, according to the depth, a three-dimensional Gaussian volume to be deleted in each region from the three-dimensional Gaussian volumes in each region; Determining a first preset threshold value of a projection area of a three-dimensional Gaussian volume of each region according to the depth; According to the three-dimensional Gaussian volumes to be deleted in each region and the first preset threshold, the three-dimensional Gaussian volumes whose projected areas are smaller than the first preset threshold are deleted from the three-dimensional Gaussian volumes to be deleted to obtain the target three-dimensional Gaussian volume.

20. The method according to any one of claims 16 or 17, characterized in that Rendering the three-dimensional Gaussian volume of each region to obtain a rendered image of each region includes: Randomly deleting a preset number of three-dimensional Gaussian bodies from the three-dimensional Gaussian bodies in each region to obtain a target three-dimensional Gaussian body in each region; The target three-dimensional Gaussian volume of each region is rendered to obtain a rendered image of each region.

21. The method according to any one of claims 16 or 17, characterized in that Rendering the three-dimensional Gaussian volume of each region to obtain a rendered image of each region includes: Calculating a first volume of a three-dimensional Gaussian body for each region; According to the first volume, deleting the three-dimensional Gaussian volume whose volume exceeds a second preset threshold value in the first volume to obtain the target three-dimensional Gaussian volume; Rendering is performed according to the target three-dimensional Gaussian volume in each region to obtain a rendered image of each region.

22. The method according to any one of claims 16 or 17, characterized in that Rendering the three-dimensional Gaussian volume of each region to obtain a rendered image of each region includes: Randomly determine a three-dimensional Gaussian volume to be deleted from the three-dimensional Gaussian volumes in each region; Calculating a second volume of the three-dimensional Gaussian body of each region; Deleting the three-dimensional Gaussian volume whose second volume exceeds a third preset threshold from the three-dimensional Gaussian volume to be deleted, to obtain the target three-dimensional Gaussian volume; Rendering is performed according to the target three-dimensional Gaussian volume of each region to obtain a rendered image of each region.

23. A rendering method, characterized in that: include: determining a target three-dimensional Gaussian body from the three-dimensional Gaussian bodies according to label information of the three-dimensional Gaussian bodies in the region, where the target three-dimensional Gaussian body is a three-dimensional Gaussian body recognized by a photographing device in the region; The target three-dimensional Gaussian volume is rendered to obtain a rendered image of the area.

24. A rendering method, characterized in that: include: Acquire a three-dimensional Gaussian volume of the region according to an image in the region; determining target three-dimensional Gaussian bodies in the region from the three-dimensional Gaussian bodies in the region according to one or more of the depth of the three-dimensional Gaussian body and the projected area or volume of the three-dimensional Gaussian body, wherein the number of the target three-dimensional Gaussian bodies is less than the number of the three-dimensional Gaussian bodies; The target three-dimensional Gaussian volume is rendered to obtain a rendered image of the area.

25. The method according to claim 24, characterized in that The determining, from the three-dimensional Gaussian bodies of the region, a target three-dimensional Gaussian body of the region according to one or more of the depth of the three-dimensional Gaussian body and the projected area or volume of the three-dimensional Gaussian body, comprises: determining a target three-dimensional Gaussian volume in the region from the three-dimensional Gaussian volumes in the region according to the depth of the three-dimensional Gaussian volume and the projected area; A target three-dimensional Gaussian volume of the region is determined from the three-dimensional Gaussian volumes of the region according to the volume of the three-dimensional Gaussian volume.

26. The method according to claim 25, characterized in that The determining, from the three-dimensional Gaussian volumes of the region according to the depth and the projected area of the three-dimensional Gaussian volume, a target three-dimensional Gaussian volume of the region comprises: determining, according to the depth of the three-dimensional Gaussian volume, a three-dimensional Gaussian volume to be rendered in the region from the three-dimensional Gaussian volumes in the region; determining a first preset value of a projection area of the three-dimensional Gaussian body in the region according to the depth of the three-dimensional Gaussian body; According to the three-dimensional Gaussian volumes to be rendered in the region and the first preset threshold, a three-dimensional Gaussian volume with a projected area greater than or equal to the first preset value in the three-dimensional Gaussian volume to be rendered is used as the target three-dimensional Gaussian volume.

27. The method according to claim 25, characterized in that The step of determining a target three-dimensional Gaussian volume of the region from the three-dimensional Gaussian volumes of the region according to the volume of the three-dimensional Gaussian volume comprises: Calculate a first volume of a three-dimensional Gaussian body in the region; A three-dimensional Gaussian body whose first volume is smaller than or equal to a second preset threshold is used as the target three-dimensional Gaussian body.

28. The method according to claim 25, characterized in that The step of determining a target three-dimensional Gaussian volume of the region from the three-dimensional Gaussian volumes of the region according to the volume of the three-dimensional Gaussian volume comprises: randomly determining the three-dimensional Gaussian volume to be rendered from the three-dimensional Gaussian volumes in the region; calculating a second volume of a three-dimensional Gaussian body in the region; A three-dimensional Gaussian body whose second volume is smaller than or equal to a third preset threshold in the three-dimensional Gaussian body to be rendered is used as the target three-dimensional Gaussian body.

29. The method according to any one of claims 24 to 28, characterized in that The region is any one of the plurality of regions according to any one of claims 1 to 10.

30. A three-dimensional reconstruction device, characterized in that: include: An acquisition module, configured to acquire a plurality of images of a scene, wherein the scene includes a plurality of areas; a division module, configured to divide the plurality of images into the plurality of regions according to the positions of the plurality of images corresponding to the shooting device, and obtain at least one image in each region; The acquisition module is further configured to acquire at least one three-dimensional Gaussian volume of each region based on the at least one image of each region, wherein the at least one three-dimensional Gaussian volume is used to represent the region in the scene; The combining module is configured to combine at least one three-dimensional Gaussian volume in each region to obtain a combined three-dimensional Gaussian volume, wherein the combined three-dimensional Gaussian volume represents the scene.

31. A data processing device, characterized in that: include: an acquisition module, configured to acquire appearance latent variables, wherein the appearance latent variables are used to identify a rendered image obtained by rendering a three-dimensional Gaussian volume in a region; An optimization module is used to optimize the parameters of the three-dimensional Gaussian body according to the appearance latent variables to obtain an optimized three-dimensional Gaussian body, and the optimized three-dimensional Gaussian body is used to represent the region.

32. A data processing device, characterized in that: include: An acquisition module, configured to acquire a three-dimensional Gaussian volume of a region according to an image in the region, wherein the region is any one of a plurality of regions in the scene; a rendering module, configured to render the three-dimensional Gaussian volume to obtain a first rendered image; an analysis module, configured to analyze the first rendered image and feature parameters using a neural network model to obtain a second rendered image, wherein the feature parameters are used to adjust appearance parameters of the first rendered image; An optimization module is used to optimize the parameters of the three-dimensional Gaussian volume according to the difference between the first rendered image and the sample image and the difference between the second rendered image and the sample image, so as to obtain an optimized three-dimensional Gaussian volume, wherein the optimized three-dimensional Gaussian volume is used to represent the region.

33. A rendering device, characterized in that include: An acquisition module, configured to acquire a position of a camera in a scene, wherein the scene is divided into a plurality of areas; a determination module, configured to determine a three-dimensional Gaussian volume in an area corresponding to the photographing device according to a position of the photographing device; A rendering module, configured to render the three-dimensional Gaussian volume of each region to obtain a rendered image of each region; The combination module is used to combine the rendering images of each area to obtain a target rendering image.

34. A rendering device, characterized in that include: a determination module, configured to determine a target three-dimensional Gaussian body from the three-dimensional Gaussian bodies in the region based on label information of the three-dimensional Gaussian bodies, wherein the target three-dimensional Gaussian body is a three-dimensional Gaussian body recognized by a photographing device in the region; A rendering module is used to render the target three-dimensional Gaussian volume to obtain a rendered image of the area.

35. A rendering device, characterized in that include: An acquisition module, configured to acquire a three-dimensional Gaussian volume of the region according to an image in the region; a determination module, configured to determine target three-dimensional Gaussian bodies in the region from the three-dimensional Gaussian bodies in the region based on one or more of the depth of the three-dimensional Gaussian body and the projected area or volume of the three-dimensional Gaussian body, wherein the number of the target three-dimensional Gaussian bodies is less than the number of the three-dimensional Gaussian bodies; A rendering module is used to render the target three-dimensional Gaussian volume to obtain a rendered image of the area.

36. A three-dimensional reconstruction device, characterized in that: include: a processor and a memory, the processor being coupled to the memory; The memory is used to store programs; The processor is configured to execute the program in the memory so as to perform the method according to any one of claims 1 to 10.

37. A data processing device, characterized in that: include: a processor and a memory, the processor being coupled to the memory; The memory is used to store programs; The processor is configured to execute the program in the memory so as to perform the method according to any one of claims 11 to 14 or 15.

38. A rendering device, characterized in that include: a processor and a memory, the processor being coupled to the memory; The memory is used to store programs; The processor is configured to execute the program in the memory so as to perform the method according to any one of claims 16 to 22, 23, or 24 to 29.

39. A computer-readable storage medium comprising instructions, which, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 10, 11 to 14, 15, 16 to 22, 23, or 24 to 29.

40. A computer program product comprising instructions which, when run on a computer, cause the computer to perform the method of any one of claims 1 to 10, 11 to 14, 15, 16 to 22, 23, or 24 to 29.

Images