Scene division method, parameter training method, rendering method, and apparatus

Abstract

Embodiments of the present application provide a scene division method, a parameter training method, a rendering method, and an apparatus, used for dividing a scene on the basis of the layout and blocking status of the scene, to obtain a plurality of areas, so that photographing devices in a same area have stronger correlation. The method comprises: acquiring layout information of a scene, the layout information being used for indicating layout of objects in the scene (step 201); and dividing the scene on the basis of the layout information of the scene, to obtain a plurality of areas, the areas being used for performing three-dimensional reconstruction and / or scene rendering (step 202).

Description

A scene segmentation method, parameter training method, rendering method, and device

[0001] This application claims priority to Chinese Patent Application No. 202410940567.0, filed on July 12, 2024, entitled "A Scene Segmentation Method, Parameter Training Method, Rendering Method and Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of scene reconstruction, and in particular to a scene segmentation method, parameter training method, rendering method, and apparatus. Background Technology

[0003] With the rapid development of industries such as Virtual Reality (VR), Augmented Reality (AR), and gaming, the demand for realistic large-scale scene reconstruction is constantly increasing. Currently, scene reconstruction can typically employ methods such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS). A common approach for large-scale scene reconstruction is to divide the large scene into multiple smaller scenes, reconstruct each individual scene, and finally merge the reconstructed smaller scenes to obtain the larger scene.

[0004] Currently, methods such as VastGaussian and CityGaussian can be used for large-scale scene reconstruction. VastGaussian works by uniformly dividing the camera coordinates of the colmap into grids, then independently training each grid region using a customized camera and point cloud selection strategy in parallel, ultimately stitching them together to form a complete scene. CityGaussian first constructs a coarse-grained 3D Gaussian splash model based on the full dataset, then normalizes and compresses the model coordinates to optimize the distribution of the 3D Gaussian volume (Gaussian sphere), and then uniformly divides the space into grids based on the compressed space, training each region in parallel, and finally stitching them together to form a complete scene.

[0005] However, existing large-scene reconstruction methods mainly employ scene segmentation strategies by directly and uniformly dividing the scene. This scene segmentation strategy may result in a single target region containing multiple mutually occluding areas. For example, a region may be divided into areas 1, 2, and 3, and occlusions may exist between these areas. Cameras in area 1 cannot capture scenes in areas 2 and 3. Therefore, cameras in area 1 cannot contribute to the scene reconstruction of other areas, and this will reduce the average contribution of cameras in the target region to the reconstruction of the target region. Summary of the Invention

[0006] This application provides a scene segmentation method, parameter training method, rendering method, and apparatus for dividing a scene into multiple regions based on its layout and occlusion, so that shooting devices in the same region have stronger correlation.

[0007] In a first aspect, this application provides a scene segmentation method, comprising: first, obtaining the layout information of the scene, which is used to instruct objects in the scene to be laid out; then, dividing the scene according to the layout information to obtain multiple regions, and subsequently performing 3D reconstruction and / or scene rendering based on the multiple regions obtained from the segmentation, so as to realize the scene segmentation reconstruction and rendering acceleration.

[0008] In this embodiment, the layout of the scene is taken into consideration. The scene can be divided into regions according to the distribution of objects in the scene, so that there are no parts that block each other in the regions after division. When the parts that block each other in the region are reduced, the visibility between shooting devices can be improved, thereby increasing the average contribution of the shooting devices in the region to the reconstruction of the region.

[0009] In one possible implementation, the aforementioned process of dividing the scene into multiple regions based on the scene layout information may include: clustering the shooting devices in the scene based on the scene layout information to obtain multiple clustering results; and dividing the scene into multiple regions based on the multiple clustering results.

[0010] In this embodiment, the scene can be divided according to the category of the shooting device, so that the correlation between the shooting devices in the area is stronger, thereby improving the average contribution of the shooting device to the scene reconstruction.

[0011] In one possible implementation, the aforementioned clustering of shooting devices in the scene based on the scene layout information to obtain multiple clustering results may include: determining occlusion information in the scene based on the scene layout information, wherein the occlusion information is used to indicate the location information of occluders in the scene; constructing an occlusion-aware attribute scene graph based on the shooting devices in the scene and the occlusion information, wherein the occlusion-aware attribute scene graph includes multiple nodes and multiple edges, wherein nodes represent shooting devices, edges represent the visibility relationship between two shooting devices, and the visibility relationship represents the similarity between the images corresponding to the two shooting devices; and clustering multiple nodes using an attribute graph clustering algorithm based on the nodes and edges to obtain multiple clustering results.

[0012] In this embodiment, an occlusion-aware attribute scene graph can be constructed to represent the occlusion relationships in the scene, and the scene can be divided into regions based on the visual relationships between shooting devices. Since the shooting devices in each region have visual relationships, the average contribution of the shooting devices to region reconstruction can be improved.

[0013] In one possible implementation, the attributes of the aforementioned nodes represent the location coordinates of the shooting device, and the weights of the aforementioned edges represent the number of matching feature points in the corresponding images of the two shooting devices. The matching feature points are image feature points whose similarity between the images is higher than a first preset value.

[0014] In this embodiment, the weight of the edge can be used to represent whether the shooting devices have a common visual relationship, that is, whether the shooting devices have similar perspectives, so that shooting devices with more similar shooting perspectives can be divided into the same area to improve the correlation of shooting devices in the same area.

[0015] In one possible implementation, the aforementioned clustering of multiple nodes using an attribute graph clustering algorithm based on nodes and edges to obtain multiple clustering results may include: clustering multiple nodes using a graph convolution algorithm based on node attributes and edge weights to obtain multiple clustering results.

[0016] In this embodiment, the shooting devices can be clustered according to their location information and corresponding image features. This allows shooting devices that are close to each other and visually similar to be grouped into one category, making the shooting devices in the regions obtained by subsequent region division based on the clustering results more correlated.

[0017] In one possible implementation, after clustering multiple nodes using an attribute graph clustering algorithm based on nodes and edges to obtain multiple clustering results, the method may further include: optimizing the multiple clustering results to obtain multiple optimized clustering results.

[0018] In this embodiment of the application, after clustering the shooting devices to obtain multiple clustering results, the multiple clustering results can be optimized so that the number of regions obtained by subsequent region division based on the clustering results is more accurate.

[0019] In one possible implementation, the aforementioned optimization of multiple clustering results to obtain multiple optimized clustering results may include: when the overlapping area between the region corresponding to the first clustering result and the region corresponding to the second clustering result exceeds a second preset value, merging the first clustering result and the second clustering result among the multiple clustering results to obtain multiple optimized clustering results; when the number of shooting devices in the first clustering result is less than a third preset value, deleting the first clustering result among the multiple clustering results to obtain multiple optimized clustering results; when the number of shooting devices in the first clustering result is greater than a fourth preset value, obtaining multiple optimized clustering results based on the clustering results of clustering the shooting devices in the first clustering result.

[0020] In one possible implementation, when the number of shooting devices in the first clustering result is higher than the fourth preset value, obtaining multiple optimized clustering results based on the clustering results of the shooting devices in the first clustering result may include: when the number of shooting devices in the first clustering result is higher than the fourth preset value, using an attribute graph clustering algorithm to cluster the shooting devices in the first clustering result to obtain multiple third clustering results; updating the first clustering result to multiple third clustering results to obtain multiple optimized clustering results.

[0021] In this embodiment of the application, when there are too many shooting devices in the clustering result, the shooting devices in the clustering result can be further divided, so that the division of the clustering result can be more accurate.

[0022] In one possible implementation, the aforementioned process of dividing the scene into multiple regions based on multiple clustering results may include: dividing the occlusion perception attribute scene map into multiple regions based on multiple optimized clustering results.

[0023] In one possible implementation, the aforementioned process of dividing the occlusion-aware attribute scene map into multiple regions based on multiple optimized clustering results may include: obtaining the boundary lines of multiple regions using a support vector machine model based on the attributes of multiple optimized clustering results and multiple nodes, with the boundary lines used to divide the multiple optimized clustering results; and obtaining multiple regions based on the boundary lines of multiple optimized clustering results and multiple regions.

[0024] In this embodiment, the boundary lines of multiple regions can be calculated based on the coordinates of the shooting device and the multiple clustering results of the shooting device, thereby clarifying the range and boundary of each region and realizing the division of the scene into multiple regions.

[0025] Secondly, this application provides a parameter training method, which may include: setting training frequencies for different types of shooting devices in the region, with each type of shooting device having a different training frequency; training the parameters of the three-dimensional Gaussian volume of the region according to the training frequencies of the different types of shooting devices, to obtain the trained three-dimensional Gaussian volume of the region, and the trained three-dimensional Gaussian volume is used to represent the region.

[0026] In this embodiment of the application, when training the parameters of the three-dimensional Gaussian volume of the region, different training frequencies can be set according to different types of training cameras to improve the overall training efficiency.

[0027] In one possible implementation, before training the parameters of the three-dimensional Gaussian body of the region according to the training frequency of different types of shooting devices to obtain the trained three-dimensional Gaussian body of the region, the method may further include: obtaining at least one three-dimensional Gaussian body of the region based on at least one image of the region.

[0028] In one possible implementation, the aforementioned types of imaging devices include boundary imaging devices and core imaging devices. The boundary imaging devices are used to supervise the boundaries of the region, and the core imaging devices are imaging devices within the region. The first training frequency of the boundary imaging devices is lower than the second training frequency of the core imaging devices. The aforementioned training of the parameters of the three-dimensional Gaussian body of the region according to the training frequencies of different types of imaging devices to obtain the trained three-dimensional Gaussian body of the region may include: training the parameters of the three-dimensional Gaussian body of each region according to the first training frequency of the boundary imaging devices and the second training frequency of the core imaging devices to obtain the trained three-dimensional Gaussian body.

[0029] In this embodiment, the imaging device can be divided into a boundary imaging device and a core imaging device. Since the boundary imaging device is only used to constrain the size of the three-dimensional Gaussian body at the boundary of the region, it does not need to be trained frequently. Based on this, different training frequencies can be set for imaging devices with different uses to improve the overall training efficiency.

[0030] In one possible implementation, the region is any one of the multiple regions mentioned in the first aspect above.

[0031] Thirdly, this application provides a rendering method, which includes: dividing the region into multiple sub-regions by moving the boundary line of the region into the region by a preset distance; rendering a three-dimensional Gaussian volume in the target sub-region according to the target sub-region corresponding to the rendering viewpoint, thereby obtaining a rendered image of the target sub-region, wherein the target sub-region is any one of the multiple sub-regions.

[0032] In this embodiment, the region can be further divided into multiple sub-regions, and the sub-regions corresponding to the rendering viewpoint are rendered in blocks according to the position of the rendering viewpoint. By dividing the region into multiple sub-regions, the influence of the 3D Gaussian volume of the neighboring region is reduced when each sub-region is rendered, and the number of 3D Gaussian volumes in a single rendering is reduced by rendering the region in blocks, thereby improving the real-time rendering efficiency.

[0033] In one possible implementation, before rendering the three-dimensional Gaussian volume in the target sub-region according to the target sub-region corresponding to the rendering viewpoint to obtain the rendered image of the target sub-region, the method may further include: deleting the invisible three-dimensional Gaussian volume in the target sub-region according to the marking information of the three-dimensional Gaussian volume in the target sub-region to obtain the three-dimensional Gaussian volume in the target sub-region.

[0034] In this embodiment, invisible 3D Gaussian volumes within a sub-region can be deleted, and only visible 3D Gaussian volumes within the sub-region can be rendered, effectively reducing the computational load when rendering 3D Gaussian volumes and improving rendering speed.

[0035] In one possible implementation, before deleting the invisible three-dimensional Gaussian bodies in the target sub-region based on the labeling information of the three-dimensional Gaussian bodies in the target sub-region to obtain the three-dimensional Gaussian bodies in the target sub-region, the method may further include: labeling the three-dimensional Gaussian bodies based on the rendering contribution value of the three-dimensional Gaussian bodies to the pixels in the rendered image to obtain the labeling information of the three-dimensional Gaussian bodies.

[0036] In this embodiment of the application, the three-dimensional Gaussian volume can be marked so that the invisible three-dimensional Gaussian volume can be deleted later based on the marking information of the three-dimensional Gaussian volume.

[0037] In one possible implementation, the aforementioned method of marking the three-dimensional Gaussian body based on its rendering contribution value to pixels in the rendered image to obtain the marking information of the three-dimensional Gaussian body may include: marking the three-dimensional Gaussian body as a visible three-dimensional Gaussian body when the selected rendering contribution value of the three-dimensional Gaussian body is higher than or equal to a preset value; and marking the three-dimensional Gaussian body as an invisible three-dimensional Gaussian body when the selected rendering contribution value of the three-dimensional Gaussian body is lower than the preset value.

[0038] In one possible implementation, the region is any one of the multiple regions mentioned in the first aspect above.

[0039] Fourthly, this application provides a scene segmentation device, comprising: an acquisition module for acquiring layout information of a scene, the layout information being used to instruct objects in the scene to be laid out; and a segmentation module for segmenting the scene according to the layout information to obtain multiple regions, the regions being used for 3D reconstruction and / or scene rendering.

[0040] In one possible implementation, the aforementioned segmentation module is specifically used to: cluster the shooting devices in the scene according to the scene layout information to obtain multiple clustering results; and divide the scene according to the multiple clustering results to obtain multiple regions.

[0041] In one possible implementation, the aforementioned partitioning module is specifically used for: determining occlusion information in the scene based on the scene layout information, wherein the occlusion information is used to indicate the location information of occluders in the scene; constructing an occlusion-aware attribute scene graph based on the shooting devices and occlusion information in the scene, wherein the occlusion-aware attribute scene graph includes multiple nodes and multiple edges, wherein nodes represent shooting devices, edges represent the visual relationship between two shooting devices, and the visual relationship represents the similarity between the images corresponding to the two shooting devices; and clustering multiple nodes using an attribute graph clustering algorithm based on the nodes and edges to obtain multiple clustering results.

[0042] In one possible implementation, the attributes of the aforementioned nodes represent the location coordinates of the shooting device, and the weights of the aforementioned edges represent the number of matching feature points of the corresponding images of the two shooting devices. The matching feature points are image feature points whose similarity between the images is higher than a first preset value.

[0043] In one possible implementation, the aforementioned partitioning module is specifically used to: cluster multiple nodes using a graph convolution algorithm based on the node attributes and edge weights to obtain multiple clustering results.

[0044] In one possible implementation, after the aforementioned clustering of multiple nodes using an attribute graph clustering algorithm based on nodes and edges to obtain multiple clustering results, the device may further include: an optimization module for optimizing the multiple clustering results to obtain multiple optimized clustering results.

[0045] In one possible implementation, the aforementioned optimization module is specifically used for: when the overlapping area between the region corresponding to the first clustering result and the region corresponding to the second clustering result exceeds a second preset value, merging the first clustering result and the second clustering result among multiple clustering results to obtain multiple optimized clustering results; when the number of shooting devices in the first clustering result is lower than a third preset value, deleting the first clustering result among multiple clustering results to obtain multiple optimized clustering results; when the number of shooting devices in the first clustering result is higher than a fourth preset value, obtaining multiple optimized clustering results based on the clustering results of clustering the shooting devices in the first clustering result.

[0046] In one possible implementation, the aforementioned optimization module is specifically used to: when the number of shooting devices in the first clustering result is higher than a fourth preset value, use an attribute graph clustering algorithm to cluster the shooting devices in the first clustering result to obtain multiple third clustering results; update the first clustering result to multiple third clustering results to obtain multiple optimized clustering results.

[0047] In one possible implementation, the aforementioned partitioning module is specifically used to: partition the occlusion perception attribute scene map according to multiple optimized clustering results to obtain multiple regions.

[0048] In one possible implementation, the aforementioned partitioning module is specifically used to: obtain the boundary lines of multiple regions through a support vector machine model based on multiple optimized clustering results and the attributes of multiple nodes, and the boundary lines are used to partition the multiple optimized clustering results; and obtain multiple regions based on the multiple optimized clustering results and the boundary lines of multiple regions.

[0049] Fifthly, this application provides a parameter training device, comprising: a setting module for setting training frequencies for different types of shooting devices in a region, wherein the training frequency for each type of shooting device is different; and a training module for training the parameters of a three-dimensional Gaussian body of the region according to the training frequencies of different types of shooting devices, thereby obtaining a trained three-dimensional Gaussian body of the region, wherein the trained three-dimensional Gaussian body is used to represent the region.

[0050] In one possible implementation, before training the parameters of the three-dimensional Gaussian body of the region according to the training frequency of different types of shooting devices to obtain the trained three-dimensional Gaussian body of the region, the device may further include: an acquisition module for acquiring at least one three-dimensional Gaussian body of the region based on at least one image of the region.

[0051] In one possible implementation, the aforementioned imaging devices include boundary imaging devices and core imaging devices. The boundary imaging devices are used to supervise the boundaries of the region, and the core imaging devices are imaging devices within the region. The first training frequency of the boundary imaging devices is lower than the second training frequency of the core imaging devices. The aforementioned training module is specifically used to: train the parameters of the three-dimensional Gaussian body of each region according to the first training frequency of the boundary imaging devices and the second training frequency of the core imaging devices, so as to obtain the trained three-dimensional Gaussian body.

[0052] In one possible implementation, the region is any one of the multiple regions mentioned in the fourth aspect above.

[0053] Sixthly, this application provides a rendering apparatus, comprising: a partitioning module, configured to partition a region by moving the boundary line of the region within the region by a preset distance to obtain multiple sub-regions; and a rendering module, configured to render a three-dimensional Gaussian volume in a target sub-region according to a target sub-region corresponding to a rendering viewpoint, thereby obtaining a rendered image of the target sub-region, wherein the target sub-region is any one of the multiple sub-regions.

[0054] In one possible implementation, before rendering the three-dimensional Gaussian volume in the target sub-region according to the target sub-region corresponding to the rendering viewpoint to obtain the rendered image of the target sub-region, the device may further include: a deletion module, used to delete invisible three-dimensional Gaussian volumes in the target sub-region according to the marking information of the three-dimensional Gaussian volumes in the target sub-region, so as to obtain the three-dimensional Gaussian volumes in the target sub-region.

[0055] In one possible implementation, before deleting the invisible three-dimensional Gaussian bodies in the target sub-region based on the marking information of the three-dimensional Gaussian bodies in the target sub-region to obtain the three-dimensional Gaussian bodies in the target sub-region, the device may further include: a marking module, used to mark the three-dimensional Gaussian bodies based on the rendering contribution value of the three-dimensional Gaussian bodies to the pixels in the rendered image, to obtain the marking information of the three-dimensional Gaussian bodies.

[0056] In one possible implementation, the aforementioned marking module is specifically used to: mark the three-dimensional Gaussian body as a visible three-dimensional Gaussian body when the selected rendering contribution value of the three-dimensional Gaussian body is higher than or equal to a preset value; and mark the three-dimensional Gaussian body as an invisible three-dimensional Gaussian body when the selected rendering contribution value of the three-dimensional Gaussian body is lower than the preset value.

[0057] In one possible implementation, the region is any one of the multiple regions mentioned in the fourth aspect above.

[0058] In a seventh aspect, a scene partitioning apparatus is provided, comprising: a processor, a memory, an input / output device, and a bus; the memory stores computer instructions; when the processor executes the computer instructions in the memory, the memory stores computer instructions; when the processor executes the computer instructions in the memory, it is used to implement as in the first aspect and any one of the implementations of the first aspect.

[0059] Eighthly, a parameter training apparatus is provided, comprising: a processor, a memory, an input / output device, and a bus; the memory stores computer instructions; when the processor executes the computer instructions in the memory, the memory stores computer instructions; when the processor executes the computer instructions in the memory, it is used to implement as in the second aspect and any one of the implementations of the second aspect.

[0060] A ninth aspect provides a rendering apparatus, the rendering apparatus comprising: a processor, a memory, an input / output device, and a bus; the memory storing computer instructions; the processor executing the computer instructions in the memory, wherein the memory storing computer instructions; and the processor executing the computer instructions in the memory being configured to implement as described in the third aspect and any one of the implementations of the third aspect.

[0061] In a tenth aspect, embodiments of this application provide a computer-readable storage medium. The computer-readable storage medium stores computer instructions; when the computer instructions are executed on a computer, the computer causes the computer to perform the method as described in any one of the possible implementations of the first, second, or third aspects.

[0062] Eleventhly, embodiments of this application provide a computer program product. The computer program product includes a computer program or instructions that, when executed on a computer, cause the computer to perform the method described in any of the possible implementations of the first, second, or third aspects.

[0063] The technical effects of the fourth to eleventh aspects or any of the possible implementations can be found in the technical effects of the relevant possible implementations of the first to third aspects, and will not be repeated here. Attached Figure Description

[0064] Figure 1 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0065] Figure 2 is a flowchart illustrating a scene segmentation method provided in an embodiment of this application;

[0066] Figure 3 is an exemplary occlusion perception attribute scene diagram;

[0067] Figure 4 is a flowchart illustrating the process of scene division based on the coordinates of the shooting device and the visual relationship;

[0068] Figure 5 is a flowchart illustrating a parameter training method provided in an embodiment of this application;

[0069] Figure 6 is a schematic diagram of a three-dimensional Gaussian body that is too large or too long at the boundary of the region.

[0070] Figure 7 is a schematic diagram of expanding the region by adding a selected boundary camera;

[0071] Figure 8 is a flowchart illustrating a rendering method provided in an embodiment of this application;

[0072] Figure 9 is an example diagram showing the division of the region according to the second boundary line;

[0073] Figure 10 shows an example of rendering based on region visibility.

[0074] Figure 11 is a schematic diagram of a scene segmentation device provided in an embodiment of this application;

[0075] Figure 12 is a schematic diagram of a parameter training device provided in an embodiment of this application;

[0076] Figure 13 is a schematic diagram of a rendering device provided in an embodiment of this application;

[0077] Figure 14 is a schematic diagram of another scene segmentation device provided in an embodiment of this application;

[0078] Figure 15 is a schematic diagram of another parameter training device provided in an embodiment of this application;

[0079] Figure 16 is a schematic diagram of another rendering device provided in an embodiment of this application. Detailed Implementation

[0080] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0081] The method provided in this application can be applied to artificial intelligence (AI) scenarios. AI is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to have the functions of perception, reasoning, and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0082] First, the overall workflow of an artificial intelligence system is described. The following sections elaborate on the aforementioned AI framework from two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed by technology) to the industrial ecosystem of the system.

[0083] (1) Infrastructure

[0084] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0085] (2) Data

[0086] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0087] (3) Data processing

[0088] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0089] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0090] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0091] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0092] (4) General ability

[0093] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0094] (5) Smart Products and Industry Applications

[0095] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0096] To better understand the solutions of the embodiments of this application, the relevant terms and concepts that may be involved in the embodiments of this application will be introduced first.

[0097] (1) 3D Gaussian Splatting (3DGS)

[0098] 3D Gaussian splashing is a rasterization technique used to render realistic scenes. It uses a 3D Gaussian distribution to describe the scene and projects these distributions onto the image plane as a 2D Gaussian distribution to render the view.

[0099] (2) Three-dimensional Gaussian body

[0100] A three-dimensional Gaussian volume, also known as a Gaussian sphere, is used to describe a Gaussian distribution in three-dimensional space. The shape of a Gaussian volume resembles a sphere. The standard deviation of the Gaussian volume determines its shape and size; the larger the standard deviation, the flatter the shape, and vice versa.

[0101] (3) Rendering

[0102] In computer graphics, rendering typically refers to outputting visual information such as lighting, texture, and color of a 3D model to a 2D screen. During rendering, the Graphics Processing Unit (GPU) is the primary hardware device performing rendering calculations. The main work of the GPU graphics rendering pipeline can be divided into two parts: converting 3D coordinates to 2D coordinates and converting 2D coordinates into actual colored pixels. The specific implementation can be divided into six stages, including vertex shaders, shape assemblies, geometry shaders, rasterization, fragment shaders, and testing and blending. Rasterization is the process of converting data into visible pixels.

[0103] (4) Property Graphs

[0104] A property graph is a representation of graph data that adds additional attribute information. In a property graph, nodes (also called vertices or nodes) and relationships (also called edges) both have labels and properties. A property graph consists of nodes and relationships, where nodes represent entities in the graph and can be any object with a unique identifier. Relationships represent connections or interactions between nodes.

[0105] (5) Scene

[0106] A scene represents a specific region in the geographical world.

[0107] To make the technical solution of this application clearer and easier to understand, the system architecture of this application will be described below with reference to the accompanying drawings.

[0108] Referring to Figure 1, a system architecture 100 provided in this application is illustrated. This system architecture 100 includes a terminal device 110 and a server 120. The terminal device 110 and the server 120 can communicate with each other via a network, which can be a wired network or a wireless network. The terminal device 110 can be a mobile phone, augmented reality (AR) / virtual reality (VR) glasses, a tablet computer, a laptop computer, or a television, etc., and is not specifically limited here. The server 120 can be a standalone server or a server cluster consisting of multiple servers, such as a cloud server, a central server, an edge server, or a local server in a local data center, etc., and is not specifically limited here.

[0109] Specifically, terminal device 110 can acquire images of the scene and send them to server 120. Server 120 then executes the scene segmentation method, parameter training method, and rendering method proposed in this application embodiment. Before performing 3D reconstruction of the scene, the layout and occlusion relationships of the scene can be considered. Using the scene segmentation method proposed in this application, the scene is divided into multiple regions. Subsequently, the 3D Gaussian volume of the region can be determined from the multiple images sent by terminal device 110. Server 120 then renders the 3D Gaussian volume of the region according to the region where the rendering viewpoint is located using the rendering method proposed in this application embodiment, and sends the rendering result to terminal device 110. Finally, terminal device 110 displays the rendering result.

[0110] In this embodiment, the server 120 can also execute the parameter training method proposed in this embodiment to train the parameters of the three-dimensional Gaussian volume in the region, so that the region can be represented by the trained three-dimensional Gaussian volume. It is worth noting that the system architecture shown in Figure 1 is only an example and is not intended to limit its specific implementation to this example.

[0111] For large-scene reconstruction, existing solutions employ VastGaussian and CityGaussian. Based on common large-scene reconstruction methods, the large scene is first divided into multiple smaller scenes, each reconstructed independently, and then the reconstructed smaller scenes are merged to obtain the large scene. Existing solutions primarily use uniform grid-based partitioning for large scenes, neglecting scene layout and occlusion relationships. This results in smaller scenes potentially containing multiple mutually occluding regions (region 1, region 2, and region 3). For example, a camera in region 1 cannot capture regions 2 and 3. Therefore, during the reconstruction of this smaller scene, the camera in region 1 cannot contribute to the reconstruction of regions 2 and 3, thus reducing the average contribution of the cameras to the scene reconstruction.

[0112] To address the current problems, this application proposes a scene segmentation method that considers scene layout and occlusion relationships. Based on the position coordinates of the shooting devices within the scene and their visibility, the scene is divided into multiple regions to increase the correlation of shooting devices within the same region and improve the average contribution of each shooting device to scene reconstruction. This application also proposes a parameter training and rendering method. The parameters of a 3D Gaussian volume in a region are trained according to different training frequencies corresponding to different types of shooting devices in the region. The trained 3D Gaussian volume represents the region. Furthermore, the region can be further subdivided into multiple sub-regions, and then the 3D Gaussian volumes of these sub-regions are rendered in blocks to improve rendering efficiency.

[0113] The method flow provided in this application will be described below in conjunction with the aforementioned system architecture.

[0114] Referring to Figure 2, a flowchart of a scene segmentation method provided in this application is shown below.

[0115] Step 201: Obtain the scene layout information;

[0116] Reconstructing a large scene typically involves dividing it into multiple smaller scenes (or regions), then reconstructing each region piece by piece, and finally combining these reconstructed regions to obtain the larger scene. When dividing a large scene into multiple regions, to avoid overlapping areas, the scene's layout information can be obtained first. This layout information instructs the objects in the scene to be laid out correctly. Based on this layout information, the distribution of each object in the scene can be determined, allowing subsequent region division based on this layout information to avoid overlapping areas.

[0117] The layout information of the scene can be obtained from images or videos captured by the shooting device, or from text descriptions of the scene; no specific limitation is made here.

[0118] Step 202: Divide the scene into multiple areas based on the scene layout information.

[0119] The scene layout information includes occlusion information in the scene. This occlusion information is used to indicate the location information of occluding objects in the scene. Occluding objects are objects in the scene that divide the scene into multiple areas, such as walls.

[0120] In this embodiment, after obtaining the scene layout information, the shooting devices in the scene can be clustered according to the scene layout information to obtain multiple clustering results. Subsequently, the scene is divided according to the multiple clustering results to obtain multiple regions. After obtaining multiple regions, three-dimensional Gaussian volumes in multiple regions can be obtained for scene reconstruction and / or scene rendering, etc.

[0121] Specifically, occlusion information in the scene can be determined based on the scene layout information. Then, an occlusion-aware attribute scene graph can be constructed based on the shooting devices and occlusion information in the scene. This occlusion-aware attribute scene graph can represent the occlusion relationships in the scene. For example, as shown in Figure 3, which is an exemplary occlusion-aware attribute scene graph, the graph includes multiple nodes and multiple edges. Nodes represent shooting devices in the scene, and their attributes include the location information of the shooting devices, such as their coordinates. Edges represent the visual relationship between two shooting devices, indicating the similarity between the images corresponding to the two devices. Specifically, the visual relationship can indicate the existence of matching feature points between the images corresponding to the shooting devices, where the matching feature points are image feature points with a similarity higher than a first preset value.

[0122] For example, when the images corresponding to two shooting devices in a scene contain feature points with a similarity higher than a first preset value, it can be indicated that the two shooting devices share a common visual relationship, and the two shooting devices can be represented by connecting them with edges in the occlusion-aware attribute scene graph. Furthermore, shooting devices in the occlusion-aware attribute scene graph can be directly connected by edges, with the edge weight representing the visual relationship between the two connected shooting devices. When the edge weight is 0, it can be indicated that the two shooting devices do not share a common visual relationship, meaning that the shooting devices do not capture the area where the other shooting device is located. The larger the edge weight value, the higher the similarity between the images corresponding to the two shooting devices, and the more similar the viewpoints of the two shooting devices.

[0123] In this embodiment, the layout of the scene and the occlusion of the shooting devices by occluders in the scene are considered. An occlusion-aware attribute scene graph is constructed based on the shooting devices in the scene. The visibility relationship between two shooting devices can be represented by whether there are edges connecting nodes in the occlusion-aware attribute scene graph, or by the weight of the edges. Subsequently, the scene can be divided into multiple regions based on this occlusion-aware attribute scene graph, with shooting devices in each region having visibility relationships, thereby improving the average contribution of the shooting devices to scene reconstruction.

[0124] After constructing the occlusion-aware attribute scene graph, multiple nodes can be clustered based on the nodes and edges in the graph to cluster the shooting devices in the scene. This allows for subsequent region division of the scene based on the shooting devices with different clustering results. The attribute graph clustering algorithm can be a graph convolution algorithm, K-means algorithm, or graph segmentation algorithm, etc., and is not limited here.

[0125] Optionally, based on the attributes of nodes and the weights of edges in the occlusion-aware attribute scene graph, a graph convolution algorithm can be used to cluster multiple nodes, resulting in multiple clustering results. The edge weights can be used to represent the number of matching feature points among the feature points in the images acquired by the two respective imaging devices.

[0126] After obtaining multiple clustering results, these results can be optimized, and the occlusion perception attribute scene map can be divided based on the optimized clustering results to obtain multiple regions.

[0127] Specifically, it can be determined whether the overlapping area of ​​the region corresponding to the shooting device in the first clustering result and the region corresponding to the shooting device in the second clustering result exceeds a second preset value. When the overlapping area of ​​the regions corresponding to the two clustering results exceeds the first preset value, the first clustering result and the second clustering result in multiple clustering results can be merged into one clustering result to obtain multiple optimized clustering results.

[0128] It can also determine whether the number of cameras in the first cluster result exceeds a third preset value. When the number of cameras in the first cluster result is lower than the third preset value, the first cluster result can be deleted from multiple cluster results, and the remaining cluster results are used as optimized cluster results. After dividing the regions based on the optimized cluster results, the cameras in the deleted cluster results are assigned to the corresponding region according to the range of the regions. When the number of cameras in the first cluster result is higher than or equal to a fourth preset value, the cameras in the first cluster result can be further clustered to obtain multiple optimized cluster results.

[0129] Optionally, when the number of shooting devices in the first clustering result is higher than or equal to the fourth preset value, the attribute graph clustering algorithm can be used to cluster the shooting devices in the first clustering result to obtain multiple third clustering results; then, the first clustering result is updated to multiple third clustering results to obtain multiple optimized clustering results.

[0130] After obtaining multiple optimized clustering results, a support vector machine model can be used to obtain the boundary line of each region based on the optimized clustering results and the attributes of multiple nodes. This boundary line is a linear equation that can be used to partition the multiple optimized clustering results. Subsequently, the occlusion-aware attribute scene map can be divided into multiple regions based on the multiple optimized clustering results and the boundary lines of the multiple regions.

[0131] For example, as shown in Figure 4, which is a flowchart illustrating scene segmentation based on the coordinates of the shooting devices and their visual relationships, the shooting devices in the occlusion perception attribute scene map can be clustered according to their coordinates and the visual relationships between them. The resulting clustering results are then optimized, and the scene is segmented and reconstructed based on the optimized clustering results.

[0132] Referring to Figure 5, which is a flowchart of a parameter training method proposed in an embodiment of this application, as described below.

[0133] Step 501: Set training frequencies for different categories of shooting devices in the area. The training frequency is different for each category of shooting devices.

[0134] The region can be any one of the multiple regions obtained by dividing the scene, or it can represent any scene; there is no specific limitation here.

[0135] In this embodiment of the application, when the region is any one of the multiple regions obtained by dividing the scene, the scene can be divided into multiple regions first. This can be based on a uniform grid division, or it can be based on the division method in the aforementioned scene division method, taking into account the scene layout and occlusion, to divide the scene into multiple regions. The specific method is not limited here.

[0136] Typically, excessively large or long 3D Gaussian volumes may exist at the boundaries of regions, as shown in Figure 6. For each region, the 3D Gaussian volume outside the region is clipped. To avoid the problem of excessively large or long 3D Gaussian volumes at the boundaries of each region spanning two regions, when reconstructing each region, in addition to selecting images corresponding to other regions based on the visibility of the shooting device, a boundary shooting device can also be added based on the region's boundary. That is, a shooting device close to the boundary of an adjacent region can be selected as the boundary shooting device for that region to avoid excessively large or long 3D Gaussian volumes at the boundaries. As shown in Figure 7, Figure 7 is a schematic diagram of expanding the region and adding a selected boundary camera. As shown in Figure 7, by expanding the boundary of the region, a selected camera can be added from near the region boundary as a boundary camera. This boundary camera is used to supervise the boundary and constrain the size of the 3D Gaussian volume at the boundary.

[0137] Based on this, in order to improve the training efficiency of the 3D Gaussian body, different training frequencies can be set for different types of shooting devices in the region. Since the boundary shooting device is only used to constrain the size of the 3D Gaussian body at the boundary, the boundary shooting device does not need to be trained frequently, and a lower training frequency can be set for the boundary shooting device.

[0138] Specifically, the types of shooting equipment can include boundary shooting equipment and core shooting equipment, with core shooting equipment being the shooting equipment within the area. The first training frequency for boundary shooting equipment is lower than the second training frequency for core shooting equipment.

[0139] Step 502: Train the parameters of the three-dimensional Gaussian volume of the region according to the training frequency of different types of shooting equipment to obtain the three-dimensional Gaussian volume of the region after training.

[0140] In this embodiment, the three-dimensional Gaussian volume is the smallest element in the 3DGS scene representation and can be used to represent the scene. Before training the parameters of the three-dimensional Gaussian volume in the region, the three-dimensional Gaussian volume in the region can be obtained first.

[0141] Specifically, at least one image in the region can be acquired. Based on the at least one image in the region, the image in the acquired region is analyzed using the structure from motion (SFM) algorithm to obtain the point cloud data corresponding to the image. Then, the corresponding three-dimensional Gaussian volume is obtained through the created three-dimensional Gaussian function.

[0142] After obtaining the three-dimensional Gaussian volume of the region, the parameters of the three-dimensional Gaussian volume of the region can be trained according to the training frequency of different types of shooting equipment to obtain the trained three-dimensional Gaussian volume, and the region can be represented by the trained three-dimensional Gaussian volume.

[0143] Based on the first training frequency of the boundary imaging device and the second training frequency of the core imaging device, the parameters of the three-dimensional Gaussian body of each region are trained to obtain the trained three-dimensional Gaussian body of each region.

[0144] In this embodiment of the application, in order to avoid the three-dimensional Gaussian volume at the region boundary being too long or too large, a boundary shooting device can be added at the region boundary, and different training frequencies can be set for the added boundary shooting device and the core shooting device. The training frequency can be adaptively set according to the different functions of the shooting device to improve the training efficiency.

[0145] Referring to Figure 8, which is a flowchart of a rendering method proposed in an embodiment of this application, as described below.

[0146] Step 801: Divide the region into multiple sub-regions by moving the boundary line of the region within the region by a preset distance;

[0147] When rendering a scene, it is usually divided into multiple regions for block rendering to improve rendering speed. In this embodiment, a region can be any one of the multiple regions obtained from dividing the scene, or it can represent any scene; the specific meaning is not limited here. Furthermore, when rendering each region in blocks, when the rendering viewpoint is close to the boundary of a region, a 3D Gaussian volume of the adjacent region appears in the rendering viewpoint, causing computational redundancy in the rendering. Therefore, to reduce the impact of the 3D Gaussian volume at the region boundary on the region rendering, the region can be further divided, and each region can also be rendered in blocks.

[0148] Specifically, one or more first boundary lines of a region can be moved inward by a preset distance to obtain one or more second boundary lines. These second boundary lines are used to divide the region into multiple sub-regions, so that when rendering the region, the region can be rendered in blocks according to the sub-region where the rendering view is located.

[0149] After obtaining one or more second boundary lines, the region is divided into multiple sub-regions to allow for further block rendering, as shown in Figure 9. Figure 9 is an example of region division based on the second boundary lines. As shown in Figure 9, region 1 is divided into four sub-regions, including three boundary regions and one internal region. The region can be block-rendered according to the sub-region where the rendering viewpoint is located. When the rendering viewpoint is in region 1, only a portion of region 1 can be rendered. This reduces the impact of the 3D Gaussian volume of adjacent regions at the region boundaries on the region rendering. Furthermore, further subdividing the region for block rendering allows rendering only the sub-region where the rendering viewpoint is located, improving the efficiency of real-time rendering.

[0150] Step 802: Render the three-dimensional Gaussian volume in the target sub-region according to the target sub-region corresponding to the rendering viewpoint to obtain the rendered image of the target sub-region.

[0151] After further dividing the region, multiple sub-regions within the region can be rendered in blocks based on the target sub-region where the rendering viewpoint is located. However, during the preprocessing and depth sorting of the 3D Gaussian rendering process, all 3D Gaussian volumes in front of the camera are processed, including many occluded and invisible Gaussian volumes, thus reducing rendering efficiency and speed. Therefore, before rendering the 3D Gaussian volume of the target sub-region, the invisible 3D Gaussian volumes in the target sub-region can be deleted first. Figure 10 shows an example of rendering based on region visibility. As can be seen from Figure 10, because there are obstructions in front of the camera, all invisible 3D Gaussian volumes in front of the camera can be deleted, and only the visible 3D Gaussian volumes are rendered.

[0152] Optionally, invisible 3D Gaussian volumes in the target sub-region can be deleted based on the marking information of the 3D Gaussian volumes in the target sub-region.

[0153] Specifically, a 3D Gaussian volume can be marked based on its rendering contribution to pixels in the rendered image. When the rendering contribution value of the 3D Gaussian volume is higher than or equal to a preset value, the 3D Gaussian volume can be marked as a visible 3D Gaussian volume; when the rendering contribution value of the 3D Gaussian volume is lower than the preset value, the 3D Gaussian volume can be marked as an invisible 3D Gaussian volume.

[0154] In this embodiment, the region obtained based on scene segmentation can be further subdivided into multiple sub-regions. These sub-regions are then rendered in blocks according to the rendering viewpoint's location. After segmentation, the influence of the 3D Gaussian volume of neighboring regions on each sub-region is reduced. Furthermore, block rendering based on the region where the rendering viewpoint is located reduces the number of 3D Gaussian volumes rendered in a single pass, thereby improving real-time rendering efficiency. Additionally, before rendering the 3D Gaussian volumes in a region, invisible 3D Gaussian volumes can be deleted, significantly increasing rendering speed.

[0155] The method flow provided in this application has been described above. The apparatus provided in this application will now be described based on the aforementioned method flow.

[0156] Referring to Figure 11, a schematic diagram of a scene segmentation device provided in this application includes:

[0157] The acquisition module 1101 is used to acquire the layout information of the scene, which is used to instruct objects in the scene to be laid out.

[0158] The segmentation module 1102 is used to divide the scene into multiple regions based on the scene's layout information. These regions are used for 3D reconstruction and / or scene rendering.

[0159] In one possible implementation, the aforementioned segmentation module 1102 is specifically used for: clustering the shooting devices in the scene according to the scene layout information to obtain multiple clustering results; and dividing the scene according to the multiple clustering results to obtain multiple regions.

[0160] In one possible implementation, the aforementioned partitioning module 1102 is specifically used for: determining occlusion information in the scene based on the scene layout information, wherein the occlusion information is used to indicate the position information of occluders in the scene; constructing an occlusion-aware attribute scene graph based on the shooting devices and occlusion information in the scene, wherein the occlusion-aware attribute scene graph includes multiple nodes and multiple edges, wherein nodes represent shooting devices, edges represent the visual relationship between two shooting devices, and the visual relationship represents the similarity between the images corresponding to the two shooting devices; and clustering multiple nodes using an attribute graph clustering algorithm based on the nodes and edges to obtain multiple clustering results.

[0161] In one possible implementation, the attributes of the aforementioned nodes represent the location coordinates of the shooting device, and the weights of the aforementioned edges represent the number of matching feature points of the corresponding images of the two shooting devices. The matching feature points are image feature points whose similarity between the images is higher than a first preset value.

[0162] In one possible implementation, the aforementioned partitioning module 1102 is specifically used to: cluster multiple nodes using a graph convolution algorithm based on the attributes of the nodes and the weights of the edges, to obtain multiple clustering results.

[0163] In one possible implementation, after the aforementioned clustering of multiple nodes using an attribute graph clustering algorithm based on nodes and edges to obtain multiple clustering results, the device may further include: an optimization module 1103, used to optimize the multiple clustering results to obtain multiple optimized clustering results.

[0164] In one possible implementation, the aforementioned optimization module 1103 is specifically used for: when the overlapping area between the region corresponding to the first clustering result and the region corresponding to the second clustering result exceeds a second preset value, merging the first clustering result and the second clustering result among multiple clustering results to obtain multiple optimized clustering results; when the number of shooting devices in the first clustering result is lower than a third preset value, deleting the first clustering result among multiple clustering results to obtain multiple optimized clustering results; when the number of shooting devices in the first clustering result is higher than a fourth preset value, obtaining multiple optimized clustering results based on the clustering results of clustering the shooting devices in the first clustering result.

[0165] In one possible implementation, the aforementioned optimization module 1103 is specifically used to: when the number of shooting devices in the first clustering result is higher than the fourth preset value, use the attribute graph clustering algorithm to cluster the shooting devices in the first clustering result to obtain multiple third clustering results; update the first clustering result to multiple third clustering results to obtain multiple optimized clustering results.

[0166] In one possible implementation, the aforementioned partitioning module 1102 is specifically used to: partition the occlusion perception attribute scene map according to multiple optimized clustering results to obtain multiple regions.

[0167] In one possible implementation, the aforementioned partitioning module 1102 is specifically used to: obtain the boundary lines of multiple regions through a support vector machine model based on multiple optimized clustering results and the attributes of multiple nodes, and the boundary lines are used to partition the multiple optimized clustering results; and obtain multiple regions based on the multiple optimized clustering results and the boundary lines of multiple regions.

[0168] Referring to Figure 12, which is a schematic diagram of the structure of a parameter training device provided in this application, including:

[0169] The setting module 1201 is used to set the training frequency for different categories of shooting devices in the area, and the training frequency is different for each category of shooting devices.

[0170] The training module 1202 is used to train the parameters of the three-dimensional Gaussian volume of the region according to the training frequency of different types of shooting devices, so as to obtain the trained three-dimensional Gaussian volume of the region. The trained three-dimensional Gaussian volume is used to represent the region.

[0171] In one possible implementation, before training the parameters of the three-dimensional Gaussian body of the region according to the training frequency of different types of shooting devices to obtain the trained three-dimensional Gaussian body of the region, the device may further include: an acquisition module 1203, used to acquire at least one three-dimensional Gaussian body of the region based on at least one image of the region.

[0172] In one possible implementation, the aforementioned imaging devices include boundary imaging devices and core imaging devices. The boundary imaging devices are used to supervise the boundaries of the region, and the core imaging devices are imaging devices within the region. The first training frequency of the boundary imaging devices is lower than the second training frequency of the core imaging devices. The aforementioned training module 1202 is specifically used to: train the parameters of the three-dimensional Gaussian body of each region according to the first training frequency of the boundary imaging devices and the second training frequency of the core imaging devices, so as to obtain the trained three-dimensional Gaussian body.

[0173] In one possible implementation, the region is one of the multiple regions in the aforementioned scene segmentation device.

[0174] Referring to Figure 13, which is a schematic diagram of the structure of a rendering device provided in this application, including:

[0175] The partitioning module 1301 is used to partition the region by moving the boundary line of the region within the region by a preset distance, thereby obtaining multiple sub-regions;

[0176] The rendering module 1302 is used to render the three-dimensional Gaussian volume in the target sub-region according to the target sub-region corresponding to the rendering viewpoint, so as to obtain the rendered image of the target sub-region. The target sub-region can be any one of multiple sub-regions.

[0177] In one possible implementation, before rendering the three-dimensional Gaussian volume in the target sub-region according to the target sub-region corresponding to the rendering viewpoint to obtain the rendered image of the target sub-region, the device may further include: a deletion module 1303, used to delete invisible three-dimensional Gaussian volumes in the target sub-region according to the marking information of the three-dimensional Gaussian volumes in the target sub-region, so as to obtain the three-dimensional Gaussian volumes in the target sub-region.

[0178] In one possible implementation, before deleting the invisible three-dimensional Gaussian bodies in the target sub-region based on the marking information of the three-dimensional Gaussian bodies in the target sub-region to obtain the three-dimensional Gaussian bodies in the target sub-region, the device may further include: a marking module 1304, used to mark the three-dimensional Gaussian bodies according to the rendering contribution value of the three-dimensional Gaussian bodies to the pixels in the rendered image, and obtain the marking information of the three-dimensional Gaussian bodies.

[0179] In one possible implementation, the aforementioned marking module 1304 is specifically used to: mark the three-dimensional Gaussian body as a visible three-dimensional Gaussian body when the selected rendering contribution value of the three-dimensional Gaussian body is higher than or equal to a preset value; and mark the three-dimensional Gaussian body as an invisible three-dimensional Gaussian body when the selected rendering contribution value of the three-dimensional Gaussian body is lower than the preset value.

[0180] In one possible implementation, the region is one of the multiple regions in the aforementioned scene segmentation device.

[0181] Referring to Figure 14, a schematic diagram of another scene segmentation device provided in this application is shown below.

[0182] The scene partitioning device may include a processor 1401 and a memory 1402. The processor 1401 and the memory 1402 are interconnected via a circuit. The memory 1402 stores program instructions and data.

[0183] The memory 1402 stores the program instructions and data corresponding to the steps in Figure 2 above.

[0184] The processor 1401 is used to execute the method steps performed by the scene segmentation device shown in any of the embodiments of FIG2 above.

[0185] Optionally, the scene segmentation device may also include a transceiver 1403 for receiving or sending data.

[0186] Referring to Figure 15, a schematic diagram of another parameter training device provided in this application is shown below.

[0187] The parameter training device may include a processor 1501 and a memory 1502. The processor 1501 and the memory 1502 are interconnected via a circuit. The memory 1502 stores program instructions and data.

[0188] The memory 1502 stores the program instructions and data corresponding to the steps in Figure 5 above.

[0189] The processor 1501 is used to execute the method steps performed by the parameter training device shown in any of the embodiments of FIG5 above.

[0190] Optionally, the parameter training device may also include a transceiver 1503 for receiving or sending data.

[0191] Referring to Figure 16, a schematic diagram of another rendering apparatus provided in this application is shown below.

[0192] The rendering apparatus may include a processor 1601 and a memory 1602. The processor 1601 and the memory 1602 are interconnected via a circuit. The memory 1602 stores program instructions and data.

[0193] The memory 1602 stores the program instructions and data corresponding to the steps in Figure 8 above.

[0194] The processor 1601 is used to execute the method steps performed by the rendering apparatus shown in any of the embodiments in FIG8 above.

[0195] Optionally, the rendering apparatus may also include a transceiver 1603 for receiving or sending data.

[0196] This application also provides a computer-readable storage medium storing a program that, when run on a computer, causes the computer to perform the steps of the method described in the embodiments shown in FIG2, FIG5 or FIG8 above.

[0197] This application also provides a computer program product that, when run on a computer, causes the computer to perform the method steps described in the embodiments shown in FIG2, FIG5 or FIG8 above.

[0198] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0199] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, the specific working process of the systems, devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0200] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0201] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0202] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0203] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0204] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0205] Finally, it should be noted that the above are only specific embodiments of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A scene segmentation method, characterized in that, include: Obtain the layout information of the scene, which is used to instruct the objects in the scene to be laid out; Based on the layout information of the scene, the scene is divided into multiple regions, which are used for 3D reconstruction and / or scene rendering.

2. The method according to claim 1, characterized in that, The scene is divided into multiple regions based on the layout information of the scene, including: Based on the layout information of the scene, the shooting devices in the scene are clustered to obtain multiple clustering results; Based on the multiple clustering results, the scene is divided into multiple regions.

3. The method according to claim 2, characterized in that, The step involves clustering the shooting devices in the scene based on the scene layout information, resulting in multiple clustering results, including: Based on the layout information of the scene, occlusion information in the scene is determined, and the occlusion information is used to indicate the position information of the occluding objects in the scene; Based on the shooting devices in the scene and the occlusion information, an occlusion perception attribute scene graph is constructed. The occlusion perception attribute scene graph includes multiple nodes and multiple edges. The nodes are used to represent the shooting devices, the edges are used to represent the visual relationship between two shooting devices, and the visual relationship is used to represent the similarity between the images corresponding to the two shooting devices. Based on the nodes and the edges, the attribute graph clustering algorithm is used to cluster the multiple nodes to obtain the multiple clustering results.

4. The method according to claim 3, characterized in that, The attributes of the node represent the location coordinates of the shooting device, and the weight of the edge represents the number of matching feature points in the corresponding images of the two shooting devices. The matching feature points are image feature points whose similarity between the images is higher than a first preset value.

5. The method according to claim 4, characterized in that, The method involves clustering the multiple nodes using an attribute graph clustering algorithm based on the nodes and the edges, resulting in multiple clustering results, including: Based on the attributes of the nodes and the weights of the edges, a graph convolution algorithm is used to cluster the multiple nodes to obtain the multiple clustering results.

6. The method according to any one of claims 3 to 5, characterized in that, After clustering the multiple nodes using an attribute graph clustering algorithm based on the nodes and the edges to obtain multiple clustering results, the method further includes: The multiple clustering results are optimized to obtain multiple optimized clustering results.

7. The method according to claim 6, characterized in that, The optimization of the multiple clustering results yields multiple optimized clustering results, including: When the area of ​​overlap between the region corresponding to the first clustering result and the region corresponding to the second clustering result exceeds a second preset value, the first clustering result and the second clustering result among the plurality of clustering results are merged to obtain the plurality of optimized clustering results; When the number of shooting devices in the first clustering result is lower than a third preset value, the first clustering result in the plurality of clustering results is deleted to obtain the plurality of optimized clustering results; When the number of shooting devices in the first clustering result is higher than the fourth preset value, multiple optimized clustering results are obtained based on the clustering results of the shooting devices in the first clustering result.

8. The method according to claim 7, characterized in that, When the number of shooting devices in the first clustering result is higher than the fourth preset value, based on the clustering result of clustering the shooting devices in the first clustering result, multiple optimized clustering results are obtained, including: When the number of shooting devices in the first clustering result is higher than the fourth preset value, the attribute graph clustering algorithm is used to cluster the shooting devices in the first clustering result to obtain multiple third clustering results; The first clustering result is updated with the multiple third clustering results to obtain the multiple optimized clustering results.

9. The method according to any one of claims 6 to 8, characterized in that, The scene is divided into multiple regions based on the multiple clustering results, including: The occlusion-aware attribute scene map is divided based on the multiple optimized clustering results to obtain the multiple regions.

10. The method according to claim 9, characterized in that, The occlusion-aware attribute scene map is divided according to the multiple optimized clustering results to obtain the multiple regions, including: Based on the multiple optimized clustering results and the attributes of the multiple nodes, the boundary lines of the multiple regions are obtained through a support vector machine model. The boundary lines are used to divide the multiple optimized clustering results. The multiple regions are obtained based on the multiple optimized clustering results and the boundary lines of the multiple regions.

11. A parameter training method, characterized in that, include: Set training frequencies for different categories of shooting devices in the region, with each category having a different training frequency; Based on the training frequency of different types of shooting equipment, the parameters of the three-dimensional Gaussian volume of the region are trained to obtain the trained three-dimensional Gaussian volume of the region, which is used to represent the region.

12. The method according to claim 11, characterized in that, Before training the parameters of the three-dimensional Gaussian volume of the region according to the training frequency of different types of shooting devices to obtain the trained three-dimensional Gaussian volume of the region, the method further includes: Based on at least one image of the region, obtain at least one three-dimensional Gaussian volume of the region.

13. The method according to claim 11 or 12, characterized in that, The types of imaging devices include boundary imaging devices and core imaging devices. The boundary imaging devices are used to supervise the boundaries of the region, and the core imaging devices are the imaging devices within the region. The first training frequency of the boundary imaging devices is lower than the second training frequency of the core imaging devices. The step of training the parameters of the three-dimensional Gaussian volume of the region according to the training frequencies of different types of imaging devices to obtain the trained three-dimensional Gaussian volume of the region includes: The parameters of the three-dimensional Gaussian volume of the region are trained based on the first training frequency of the boundary imaging device and the second training frequency of the core imaging device to obtain the trained three-dimensional Gaussian volume.

14. The method according to any one of claims 11 to 13, characterized in that, The region is any one of the regions in any one of the preceding claims 1 to 10.

15. A rendering method, characterized in that, include: By moving the boundary line of the region within the region by a preset distance, the region is divided into multiple sub-regions; Based on the target sub-region corresponding to the rendering viewpoint, the three-dimensional Gaussian volume in the target sub-region is rendered to obtain the rendered image of the target sub-region, wherein the target sub-region is any one of the plurality of sub-regions.

16. The method according to claim 15, characterized in that, Before rendering the 3D Gaussian volume in the target sub-region according to the target sub-region corresponding to the rendering viewpoint to obtain the rendered image of the target sub-region, the method further includes: Based on the marking information of the three-dimensional Gaussian volume in the target sub-region, the invisible three-dimensional Gaussian volume in the target sub-region is deleted to obtain the three-dimensional Gaussian volume in the target sub-region.

17. The method according to claim 16, characterized in that, Before deleting invisible 3D Gaussian volumes in the target sub-region based on the marking information of the 3D Gaussian volumes in the target sub-region to obtain the 3D Gaussian volumes in the target sub-region, the method further includes: The three-dimensional Gaussian volume is labeled according to its rendering contribution value to the pixels in the rendered image, thereby obtaining the labeling information of the three-dimensional Gaussian volume.

18. The method according to claim 17, characterized in that, The step of marking the three-dimensional Gaussian volume based on its rendering contribution value to pixels in the rendered image, to obtain the marking information of the three-dimensional Gaussian volume, includes: When the selected rendering contribution value of the three-dimensional Gaussian volume is higher than or equal to the preset value, the three-dimensional Gaussian volume is marked as a visible three-dimensional Gaussian volume; When the selected rendering contribution value of the three-dimensional Gaussian volume is lower than the preset value, the three-dimensional Gaussian volume is marked as an invisible three-dimensional Gaussian volume.

19. The method according to any one of claims 15 to 18, characterized in that, The region is any one of the regions in any one of the preceding claims 1 to 10.

20. A scene segmentation device, characterized in that, include: An acquisition module is used to acquire layout information of a scene, the layout information being used to instruct objects in the scene to be laid out. The segmentation module is used to divide the scene into multiple regions based on the layout information of the scene, and the regions are used for 3D reconstruction and / or scene rendering.

21. A parameter training device, characterized in that, include: The settings module is used to set the training frequency for different categories of shooting devices in the area, with each category of shooting devices having a different training frequency; The training module is used to train the parameters of the three-dimensional Gaussian volume of the region according to the training frequency of different types of shooting devices, so as to obtain the trained three-dimensional Gaussian volume of the region, which is used to represent the region.

22. A rendering apparatus, characterized in that, include: The partitioning module is used to divide the region into multiple sub-regions by moving the boundary line of the region within the region by a preset distance. The rendering module is used to render a three-dimensional Gaussian volume in the target sub-region according to the target sub-region corresponding to the rendering viewpoint, so as to obtain a rendered image of the target sub-region, wherein the target sub-region is any one of the plurality of sub-regions.

23. A scene segmentation device, characterized in that, include: A processor and a memory, wherein the processor is coupled to the memory; The memory is used to store programs; The processor is configured to execute a program in the memory, such that the method as described in any one of claims 1 to 10 is performed.

24. A parameter training device, characterized in that, include: A processor and a memory, wherein the processor is coupled to the memory; The memory is used to store programs; The processor is configured to execute a program in the memory, such that the method described in any one of claims 11 to 14 is performed.

25. A rendering apparatus, characterized in that, include: A processor and a memory, wherein the processor is coupled to the memory; The memory is used to store programs; The processor is configured to execute a program in the memory, such that the method described in any one of claims 15 to 19 is performed.

26. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the method as claimed in any one of claims 1 to 10, 11 to 14, or 15 to 19.

27. A computer program product comprising instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1 to 10, 11 to 14, or 15 to 19.

Images