Scene understanding method and device, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses a scene understanding method and device, electronic equipment and a computer readable storage medium, relates to the technical field of three-dimensional scene understanding, and aims to solve the technical problem that three-dimensional scene understanding based on two-dimensional features is not accurate enough. The method comprises the following steps: acquiring multiple frames of depth images of a scene; constructing a plurality of three-dimensional Gaussian points of the scene according to the multiple frames of depth images; the three-dimensional Gaussian points comprise high-dimensional language features and geometric features; clustering the plurality of three-dimensional Gaussian points according to the similarity of the high-dimensional language features and the similarity of the geometric features of the plurality of three-dimensional Gaussian points to obtain a plurality of three-dimensional super primitives; and fusing the plurality of three-dimensional super primitives to obtain a three-dimensional instance segmentation representation of the scene. According to the scheme, three-dimensional scene understanding can be realized directly based on the three-dimensional Gaussian points.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of three-dimensional scene understanding, and specifically to a scene understanding method, device, electronic device, and computer-readable storage medium. Background Art

[0002] 3D scene understanding is a key task in the field of computer vision and is widely used in scenarios such as intelligent robots, augmented reality (AR) and autonomous driving. 3D scene understanding is the process by which computers process and analyze scene data to understand the geometric structure and semantic information in the scene.

[0003] The 3D scene understanding methods in related technologies mainly focus on the 2D semantic segmentation task, which extracts and learns features after rendering the 3D scene to a 2D image plane. However, the domain difference between the 2D feature space and the 3D feature space is significant, which leads to the limitation of language feature learning. Therefore, the 3D scene understanding methods in related technologies need to be improved. Summary of the invention

[0004] The embodiments of the present application provide a scene understanding method, device, electronic device and computer-readable storage medium, which can directly implement three-dimensional scene understanding based on three-dimensional Gaussian points.

[0005] In a first aspect, an embodiment of the present application provides a scene understanding method, including: Acquire multiple frames of depth images of the scene; Constructing a plurality of three-dimensional Gaussian points of the scene according to the multiple frames of the depth image; the three-dimensional Gaussian points include: high-dimensional language features and geometric features; Clustering the plurality of three-dimensional Gaussian points according to the similarity of the high-dimensional linguistic features and the similarity of the geometric features of the plurality of three-dimensional Gaussian points to obtain a plurality of three-dimensional super-primitives; A plurality of the three-dimensional superprimitives are fused to obtain a three-dimensional instance segmentation representation of the scene.

[0006] In one embodiment, constructing a plurality of three-dimensional Gaussian points of the scene according to the plurality of frames of the depth image comprises: Constructing a plurality of initial three-dimensional Gaussian points of the scene according to the pixel points in the multiple frames of the depth image; the initial three-dimensional Gaussian points include: the geometric features and the semantic coding; the semantic coding of the initial three-dimensional Gaussian points is learned from the three-plane features of the latent space pyramid; Inputting the semantic coding of the initial three-dimensional Gaussian points into a three-dimensional decoder to obtain the high-dimensional language features; The high-dimensional language features and the initial three-dimensional Gaussian points are combined to obtain the three-dimensional Gaussian points of the scene.

[0007] In one embodiment, the step of training the 3D decoder at least comprises: Obtain semantic coding samples of initial three-dimensional Gaussian point samples; Determining, according to the semantic coding samples of the initial three-dimensional Gaussian point samples, high-dimensional language feature samples corresponding to the initial three-dimensional Gaussian point samples and the confidence of the high-dimensional language feature samples; Inputting the initial three-dimensional Gaussian point samples into an initial three-dimensional decoder to obtain predicted high-dimensional language features of the initial three-dimensional Gaussian point samples; Constructing a loss function according to the confidence of the high-dimensional language feature sample and the cosine similarity between the predicted high-dimensional language feature corresponding to the initial three-dimensional Gaussian point sample and the high-dimensional language feature sample; Based on the loss function, the initial 3D decoder is trained to obtain the trained 3D decoder.

[0008] In one embodiment, determining the high-dimensional language feature samples corresponding to the initial three-dimensional Gaussian point samples according to the semantic coding samples of the initial three-dimensional Gaussian point samples includes: Acquire multiple frames of depth image samples of scene samples corresponding to the initial three-dimensional Gaussian point samples; Extracting language feature maps of multiple frames of depth image samples; Performing multi-angle projection on the initial three-dimensional Gaussian point samples to obtain two-dimensional pixel position information corresponding to the initial three-dimensional Gaussian point samples in multiple frames of the depth image samples; Acquire the visual relationship between the initial three-dimensional Gaussian point samples in multiple frames of depth image samples; Determining, according to the visible relationship, a target depth image sample corresponding to the initial three-dimensional Gaussian point sample; the initial three-dimensional Gaussian point sample is visible in the target depth image sample; Determine the two-dimensional language features corresponding to the initial three-dimensional Gaussian point samples in the target depth image samples according to the language feature graphs of the multiple frames of the depth image samples and the two-dimensional pixel position information corresponding to the initial three-dimensional Gaussian point samples in the target depth image samples; Average pooling is performed on the two-dimensional language features corresponding to the initial three-dimensional Gaussian point samples in the target depth image samples to obtain high-dimensional language feature samples corresponding to the initial three-dimensional Gaussian point samples.

[0009] In one embodiment, determining the confidence of the high-dimensional language feature sample corresponding to the initial three-dimensional Gaussian point sample according to the semantic coding sample of the initial three-dimensional Gaussian point sample includes: Obtaining the variance of the two-dimensional language features corresponding to the initial three-dimensional Gaussian point samples in the target depth image samples; The confidence of the high-dimensional language feature sample corresponding to the initial three-dimensional Gaussian point sample is determined according to the visual relationship and the variance of the two-dimensional language feature corresponding to the initial three-dimensional Gaussian point sample.

[0010] In one embodiment, fusing the plurality of the three-dimensional superprimitives to obtain a three-dimensional instance segmentation representation of the scene includes: Determine affine coefficients between the plurality of the three-dimensional superprimitives according to projections of the three-dimensional Gaussian points of the plurality of the three-dimensional superprimitives and instance segmentation masks of the plurality of frames of the depth images; According to the affine coefficients between the plurality of the three-dimensional superprimitives, a plurality of the three-dimensional superprimitives whose affine coefficients are less than an affine coefficient threshold are merged to obtain a plurality of preliminary three-dimensional instance segmentation representations; Iteratively fuse the multiple preliminary 3D instance segmentation representations to obtain a 3D instance segmentation representation of the scene.

[0011] In one embodiment, determining the affine coefficients between the plurality of the 3D superprimitives according to the 3D Gaussian projections of the plurality of the 3D superprimitives and the instance segmentation masks of the plurality of frames of the depth images comprises: Projecting the three-dimensional superprimitive to obtain a plurality of two-dimensional images corresponding to a plurality of frames of the depth image of the three-dimensional superprimitive; Determine, according to the instance segmentation masks of the two-dimensional image and the multiple frames of the depth image, a distribution of the instance segmentation masks in the multiple two-dimensional images; Determine affine coefficients between the plurality of three-dimensional superprimitives in the depth image of the same frame according to distribution of instance segmentation masks of the two-dimensional images of the depth image of the same frame corresponding to the plurality of three-dimensional superprimitives; Obtaining a visible ratio of the two-dimensional image in the corresponding depth image; Affine coefficients between the plurality of three-dimensional superprimitives are determined according to affine coefficients between the plurality of three-dimensional superprimitives in the depth images of the same frames and visible ratios of the two-dimensional images in the depth images of the same frames.

[0012] In a second aspect, an embodiment of the present application provides a scene understanding device, including: An acquisition module, used to acquire multiple frames of depth images of a scene; A construction module, used to construct a plurality of three-dimensional Gaussian points of the scene according to the multiple frames of the depth image; the three-dimensional Gaussian points include: high-dimensional language features and geometric features; A clustering module, used for clustering the plurality of three-dimensional Gaussian points according to the similarity of the high-dimensional linguistic features and the similarity of the geometric features of the plurality of three-dimensional Gaussian points to obtain a plurality of three-dimensional super-primitives; A fusion module is used to fuse the multiple three-dimensional super primitives to obtain a three-dimensional instance segmentation representation of the scene.

[0013] In one embodiment, the building blocks include: A construction unit, configured to construct a plurality of initial three-dimensional Gaussian points of the scene according to pixel points in the multiple frames of the depth image; the initial three-dimensional Gaussian points include: the geometric features and semantic coding; the semantic coding of the initial three-dimensional Gaussian points is learned from the three-plane features of the latent space pyramid; An input unit, used for inputting the semantic coding of the initial three-dimensional Gaussian point into a three-dimensional decoder to obtain the high-dimensional language feature; A combining unit is used to combine the high-dimensional language features and the initial three-dimensional Gaussian points to obtain the three-dimensional Gaussian points of the scene.

[0014] In one embodiment, the step of training the 3D decoder at least comprises: Obtain semantic coding samples of initial three-dimensional Gaussian point samples; Determining, according to the semantic coding samples of the initial three-dimensional Gaussian point samples, high-dimensional language feature samples corresponding to the initial three-dimensional Gaussian point samples and the confidence of the high-dimensional language feature samples; Inputting the initial three-dimensional Gaussian point samples into an initial three-dimensional decoder to obtain predicted high-dimensional language features of the initial three-dimensional Gaussian point samples; Constructing a loss function according to the confidence of the high-dimensional language feature sample and the cosine similarity between the predicted high-dimensional language feature corresponding to the initial three-dimensional Gaussian point sample and the high-dimensional language feature sample; Based on the loss function, the initial 3D decoder is trained to obtain the trained 3D decoder.

[0015] In one embodiment, determining the high-dimensional language feature samples corresponding to the initial three-dimensional Gaussian point samples according to the semantic coding samples of the initial three-dimensional Gaussian point samples includes: Acquire multiple frames of depth image samples of scene samples corresponding to the initial three-dimensional Gaussian point samples; Extracting language feature maps of multiple frames of depth image samples; Performing multi-angle projection on the initial three-dimensional Gaussian point samples to obtain two-dimensional pixel position information corresponding to the initial three-dimensional Gaussian point samples in multiple frames of the depth image samples; Acquire the visual relationship between the initial three-dimensional Gaussian point samples in multiple frames of depth image samples; Determining, according to the visible relationship, a target depth image sample corresponding to the initial three-dimensional Gaussian point sample; the initial three-dimensional Gaussian point sample is visible in the target depth image sample; Determine the two-dimensional language features corresponding to the initial three-dimensional Gaussian point samples in the target depth image samples according to the language feature graphs of the multiple frames of the depth image samples and the two-dimensional pixel position information corresponding to the initial three-dimensional Gaussian point samples in the target depth image samples; Average pooling is performed on the two-dimensional language features corresponding to the initial three-dimensional Gaussian point samples in the target depth image samples to obtain high-dimensional language feature samples corresponding to the initial three-dimensional Gaussian point samples.

[0016] In one embodiment, determining the confidence of the high-dimensional language feature sample corresponding to the initial three-dimensional Gaussian point sample according to the semantic coding sample of the initial three-dimensional Gaussian point sample includes: Obtaining the variance of the two-dimensional language features corresponding to the initial three-dimensional Gaussian point samples in the target depth image samples; The confidence of the high-dimensional language feature sample corresponding to the initial three-dimensional Gaussian point sample is determined according to the visual relationship and the variance of the two-dimensional language feature corresponding to the initial three-dimensional Gaussian point sample.

[0017] In one embodiment, the fusion module includes: A determining unit, configured to determine affine coefficients between the plurality of the three-dimensional superprimitives according to projections of the three-dimensional Gaussian points of the plurality of the three-dimensional superprimitives and instance segmentation masks of the plurality of frames of the depth images; a fusion unit, configured to fuse, according to the affine coefficients between the plurality of the three-dimensional superprimitives, the plurality of three-dimensional superprimitives whose affine coefficients are less than an affine coefficient threshold, to obtain a plurality of preliminary three-dimensional instance segmentation representations; The iterative unit is used to iteratively fuse the multiple preliminary 3D instance segmentation representations to obtain the 3D instance segmentation representation of the scene.

[0018] In one embodiment, the determining unit includes: A projection subunit, configured to project the three-dimensional superprimitive to obtain a plurality of two-dimensional images of the three-dimensional superprimitive corresponding to a plurality of frames of the depth image; a distribution determination subunit, configured to determine the distribution of instance segmentation masks in a plurality of the two-dimensional images according to the instance segmentation masks of the two-dimensional image and the plurality of frames of the depth image; a coefficient determination subunit, configured to determine affine coefficients between a plurality of the three-dimensional superprimitives in the depth image of the same frame according to distribution of instance segmentation masks of the two-dimensional images of the depth image of the same frame corresponding to each of the plurality of the three-dimensional superprimitives; A ratio acquisition subunit, used to acquire a visible ratio of the two-dimensional image in the corresponding depth image; The affine coefficient determination subunit is used to determine the affine coefficients between the multiple three-dimensional superprimitives according to the affine coefficients between the multiple three-dimensional superprimitives in the depth images of the multiple same frames and the visible ratio of the two-dimensional image in the depth images of the multiple same frames.

[0019] In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps in the above-mentioned scene understanding method when executed by the processor.

[0020] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above-mentioned scene understanding method are implemented.

[0021] In a fifth aspect, the embodiments of the present application further provide a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations described in the embodiments of the present application.

[0022] In summary, in the embodiment of the present application, multiple three-dimensional Gaussian points of a scene can be directly constructed based on the multi-frame depth images of the scene, and the three-dimensional Gaussian points include high-dimensional language features and geometric features, and then the multiple three-dimensional Gaussian points can be clustered according to the similarity of the high-dimensional language features and the similarity of the geometric features of the three-dimensional Gaussian points to obtain multiple three-dimensional super primitives, and the multiple three-dimensional super primitives are fused to obtain a three-dimensional instance segmentation representation of the scene. In this way, three-dimensional scene understanding is achieved directly based on three-dimensional Gaussian points, solving the problem of domain differences between two-dimensional feature space and three-dimensional feature space. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a schematic diagram of the steps of a scene understanding method provided by an embodiment of the present application; Figure 2 is a structural diagram of a scene understanding device provided by an embodiment of the present application; Figure 3 It is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0026] 3D scene understanding can support downstream tasks such as target detection, path planning, and virtual scene interaction, providing intelligent agents with comprehensive environmental perception capabilities. Especially in open vocabulary scene understanding tasks, the system needs to be able to understand diverse semantic information, which places higher demands on modeling capabilities and semantic expression.

[0027] In recent years, neural rendering techniques, such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), have shown remarkable performance in novel perspective synthesis tasks. 3DGS has attracted particular attention for its fast training and rendering capabilities, as well as explicit scene representation based on point clouds. Related technologies usually combine visual language models (VLMs) to achieve open vocabulary scene understanding using two-dimensional pixel-level semantic features. However, most methods in related technologies still focus on two-dimensional semantic segmentation tasks, performing feature extraction and learning after rendering to a two-dimensional image plane. Although these methods have achieved certain success in two-dimensional semantic segmentation tasks, in three-dimensional scene understanding tasks, there are still problems such as inaccurate three-dimensional language feature learning and missing instance-level information modeling.

[0028] In related technologies, the feature representation of discrete Gaussian primitives may destroy the intrinsic smoothness of semantically similar objects. At the same time, due to the feature accumulation method based on alpha-blending, the domain difference between the two-dimensional feature space and the three-dimensional feature space is significant, resulting in limitations in language feature learning. In addition, the feature compression and distillation process in the two-dimensional space will further weaken the distinguishing ability of the features. This will cause the problem of inaccurate three-dimensional language feature learning.

[0029] In related technologies, object detection is usually performed through similarity heat maps of language features and text queries. However, this method cannot distinguish multiple instances of objects with the same semantics, resulting in inconsistent instance-level segmentation results and the lack of instance-level information modeling, which is crucial for pan-semantic 3D scene understanding.

[0030] The reasons for the inaccurate learning of 3D language features and the lack of instance-level information modeling are that most related technologies remain at the feature distillation of 2D space, ignoring the interactive relationship between semantic and geometric features in 3D space. At the same time, the lack of instance information is mainly due to the lack of an effective 3D graph structure modeling mechanism, which makes it impossible to use semantic consistency and geometric relationships to perform instance-level division of scenes.

[0031] Based on the above problems, an embodiment of the present application proposes a scene understanding method, which is based on three-dimensional Gaussian projection and aims to fundamentally improve the ability of three-dimensional scene semantics and instance-level understanding.

[0032] The three-dimensional panoramic segmentation technology of the related art has the following problems: a large number of three-dimensional points or voxels need to be explicitly stored in the reconstruction of multi-view scenes, resulting in huge storage overhead and low operating efficiency; the open vocabulary adaptability is poor, and the methods of the related technology rely on a fixed set of labels, and the adaptability to unseen categories or open vocabulary is insufficient, and it is difficult to generalize to complex actual scenes; the joint segmentation performance of semantics and geometry is insufficient, and there is a lack of effective semantic, geometric and multi-view feature fusion mechanism, resulting in inaccurate segmentation results.

[0033] The goal of one embodiment of the present application is to achieve efficient and accurate 3D panoptic segmentation in an open vocabulary environment, and to solve the problems of storage overhead, open vocabulary adaptation, and semantic-geometry joint optimization by improving scene representation and segmentation algorithms.

[0034] In order to achieve the above objectives, an embodiment of the present application provides the following technical solutions: Use 3D Gaussian points to express the scene and reconstruct the scene's geometry and appearance information. Instead of directly binding independent high-frequency feature encodings to each 3D Gaussian, use low-dimensional multi-resolution triplanes to model the smooth semantic characteristics of the 3D scene, and use a 3D feature decoder to elevate the low-dimensional semantic encoding to a high-dimensional language feature space, so that the scene understanding and segmentation of the opening words can be performed. In this way, the number of parameters required for scene expression and semantic modeling can be effectively reduced.

[0035] In order to achieve 3D instance segmentation, the geometric normal vectors and high-dimensional language features obtained by 3D Gaussian point reconstruction are used as the basis, and millions of 3D Gaussians in the scene are used to construct the vertices of graph clustering, thereby reducing the number of parameters of the clustering algorithm and speeding up the clustering speed. The segmentation model is used to obtain segmentation masks under different perspectives to construct the affine relationship between graph vertices. Based on the relationship between the constructed graph vertices and affine edges, progressive clustering is performed to obtain a consistent 3D instance segmentation result.

[0036] Therefore, the technical solution proposed in one embodiment of the application can effectively solve the problems of storage overhead, open vocabulary adaptability and segmentation accuracy in the current field of 3D panoramic segmentation, and has broad practical application value.

[0037] In one embodiment, Figure 1 As shown, a scene understanding method is provided. Although the logical order is shown in the step diagram, in some cases, the steps shown or described may be performed in an order different from that shown in the figure. Specifically, the scene understanding method can be applied to an electronic device, which may be a terminal or a server. Among them, the terminal may include but is not limited to one or more of a smart phone, a tablet computer, a portable computer, a desktop computer, and a car computer. The server may be a physical server or a cloud server that provides various cloud services. It is worth noting that the present application does not limit the number of terminals or servers. According to the implementation needs, there may be any number of terminals or servers. For example, the server may be a single server or a server cluster consisting of multiple servers, and so on.

[0038] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

[0039] according to Figure 1 The scene understanding method shown in the figure includes at least steps S110 to S140, which are described in detail as follows: In step S110 , multiple frames of depth images of the scene are acquired.

[0040] The scene can be any three-dimensional scene. A depth image is an image with depth information. Depth information refers to the distance information between the photographed object and the photographing device. A depth image can be an image taken by a color depth (Red Green Blue Depth, RGB-D) camera. A multi-frame depth image can be a multi-frame depth image taken continuously by a color depth camera.

[0041] In step S120, a plurality of three-dimensional Gaussian points of the scene are constructed based on the multiple frames of the depth image; the three-dimensional Gaussian points include: high-dimensional language features and geometric features.

[0042] Three-dimensional Gaussian points are points that follow Gaussian distribution in three-dimensional space. Three-dimensional Gaussian points include high-dimensional language features and geometric features. High-dimensional language features are obtained by decoding low-dimensional semantic codes, and have higher dimensions than low-dimensional semantic codes. Among them, the semantic code is learned from the three-plane features of the latent space pyramid. Geometric features include position features, rotation features and / or scale features, etc. In one embodiment of the present application, three-dimensional Gaussian points are used to characterize the scene.

[0043] Compared with low-dimensional semantic coding, high-dimensional language features have stronger feature expression capabilities. Higher dimensions mean a larger feature space and can encode more information. Therefore, high-dimensional language features can capture richer and more detailed semantic information, such as more complex textures, shapes, or contextual relationships. High-dimensional language features have finer-grained discrimination capabilities and can make more detailed distinctions between similar but different semantics; for example, based on high-dimensional language features, the specific breed of a cat can be distinguished, while based on low-dimensional language features, it may only be possible to distinguish whether an animal is a cat. High-dimensional language features may contain more abstract information. High-dimensional language features can encode higher-level abstract information, such as the functions of objects, semantic relationships of scenes, etc., which is very important for complex tasks (such as image generation and semantic segmentation).

[0044] The three-dimensional Gaussian points are constructed based on the multi-frame depth images of the captured scene. Based on the multi-frame depth images, the image color and depth sequence can be obtained. Optionally, a depth sequence can be a sequence composed of the depth information of an object in the multi-frame depth images. When there are multiple objects, there can be multiple depth sequences. Optionally, a depth sequence can be a sequence composed of the depth information of the pixel points representing the same physical point in the multi-frame depth images.

[0045] According to the image color and depth sequence, the six-degree-of-freedom pose of the color depth camera can be calculated by using the motion recovery structure algorithm by analyzing the depth images taken by the color depth camera at different positions. The six-degree-of-freedom pose of the color depth camera refers to the position and posture of the color depth camera in three-dimensional space, specifically including three translational degrees of freedom and three rotational degrees of freedom. The three translational degrees of freedom refer to the degrees of freedom of movement along the three rectangular coordinate axes of x, y, and z, and the three rotational degrees of freedom refer to the degrees of freedom of rotation around the three coordinate axes of x, y, and z.

[0046] Optionally, in order to save storage overhead and improve operating efficiency, each depth image can be sampled to obtain multiple sampled pixels. Based on the six-degree-of-freedom posture of the camera and the depth information of the sampled pixels, project them back into three-dimensional space to obtain a sampled point cloud corresponding to the sampled pixels. For example, 20% of the pixels can be uniformly sampled from each depth image to obtain multiple sampled pixels, and then a sampled sparse point cloud can be obtained based on the multiple sampled pixels. Determine the information of the point cloud corresponding to the pixel based on the pixel.

[0047] The three-dimensional Gaussian points are initialized according to the sampled point cloud to obtain multiple original three-dimensional Gaussian points, each of which corresponds to a sampled point cloud. The color features, opacity features, position features, rotation features and / or scale features of each original three-dimensional Gaussian point can be determined. The color features and position features of the original three-dimensional Gaussian point can be determined based on the color features and position features of the corresponding point cloud. The opacity of the original three-dimensional Gaussian point can be initialized to 0.5, and the rotation features can be initialized to the unit matrix. The scale features of the original three-dimensional Gaussian point can be determined based on the distance between the original three-dimensional Gaussian point and the surrounding adjacent three-dimensional Gaussian points.

[0048] Assign a semantic code to the original three-dimensional Gaussian point to obtain the initial three-dimensional Gaussian point. The dimension of the semantic code of the initial three-dimensional Gaussian point can be 6, and the semantic code is learned from the three-plane features of the latent space pyramid. The three-plane of the latent space pyramid contains fine-grained coding features and coarse-grained coding features; the fine-grained coding feature is a shape of [3, Hf , W ,6] optimization variables; the coarse-grained encoding feature is a shape of [3, Hc , W ,6] optimization variables. 3 represents the three decomposition planes of xy, xz, and yz, and 6 represents the feature dimension of semantic encoding. Hf , W , Hc , W They represent the fine-grained and coarse-grained plane resolutions, respectively. Optimization variables refer to the parameters or variables that need to be adjusted by the optimization algorithm during model training.

[0049] Using the initial 3D Gaussian points to compactly and efficiently represent the 3D scene can reduce storage requirements, while using the pyramid three-plane structure to encode the potential feature field can enhance the semantic expression ability of the 3D scene.

[0050] By inputting the semantic coding of the initial three-dimensional Gaussian point into the trained three-dimensional decoder, the high-dimensional language features corresponding to the initial three-dimensional Gaussian point output by the three-dimensional decoder can be obtained. By combining the high-dimensional language features corresponding to the initial three-dimensional Gaussian point with the initial three-dimensional Gaussian point, the three-dimensional Gaussian point of the scene can be obtained. The original three-dimensional Gaussian point does not have semantic coding or high-dimensional language features; the initial three-dimensional Gaussian point has semantic coding; the three-dimensional Gaussian point has high-dimensional language features.

[0051] In step S130, the three-dimensional Gaussian points are clustered according to the similarity of the high-dimensional linguistic features and the similarity of the geometric features of the three-dimensional Gaussian points to obtain a plurality of three-dimensional super-primitives.

[0052] The similarity of the high-dimensional language features between the multiple three-dimensional Gaussian points can be determined by calculating the cosine similarity or Euclidean distance of the high-dimensional language features between the multiple three-dimensional Gaussian points. The similarity of the geometric features between the multiple three-dimensional Gaussian points can be calculated by using a Gaussian kernel function or other distance measurement methods. Based on the similarity of the high-dimensional language features and the similarity of the geometric features of the multiple three-dimensional Gaussian points, the multiple three-dimensional Gaussian points are clustered to obtain multiple three-dimensional super primitives.

[0053] Optionally, a language-guided Graph Cuts algorithm may be used to cluster the multiple 3D Gaussian points to obtain multiple 3D super-primitives. Optionally, a Spectral Clustering, Hierarchical Clustering or K-means clustering algorithm may be used to cluster the multiple 3D Gaussian points to obtain multiple 3D super-primitives.

[0054] In one embodiment, two three-dimensional Gaussian points with the smallest sum of similarity of high-dimensional language features and similarity of geometric features can be clustered into one category to obtain a cluster, and the mean of the high-dimensional language features of the three-dimensional Gaussian points in the cluster is determined as the high-dimensional language features of the cluster center, and the mean of the geometric features of the three-dimensional Gaussian points in the cluster is determined as the geometric features of the cluster center. A high-dimensional language feature similarity threshold and a geometric feature similarity threshold are obtained. It is sequentially determined whether each of the remaining three-dimensional Gaussian points belongs to an existing cluster. When the similarity between the high-dimensional language features of the three-dimensional Gaussian points and the high-dimensional language features of the cluster center is greater than the high-dimensional language feature similarity threshold, and the similarity between the geometric features of the three-dimensional Gaussian points and the geometric features of the cluster center is greater than the geometric feature similarity threshold, the three-dimensional Gaussian points are clustered into the cluster corresponding to the cluster center, and the high-dimensional language features and geometric features of the three-dimensional Gaussian points are used to update the high-dimensional language features and geometric features of the cluster center. When a three-dimensional Gaussian point cannot be clustered into the cluster corresponding to the cluster center, another cluster can be generated based on the three-dimensional Gaussian point, and it is continued to be determined whether each of the remaining three-dimensional Gaussian points belongs to the existing cluster, until all three-dimensional Gaussian points are clustered and each cluster is determined as a three-dimensional super-element.

[0055] The 3D Gaussian points in the 3D superprimitive have geometric and semantic consistency, and the 3D Gaussian points in the 3D superprimitive have similar geometric features and high-dimensional language features. The 3D superprimitive can be used as a vertex unit for subsequent graph clustering.

[0056] In step S140, a plurality of the three-dimensional superprimitives are fused to obtain a three-dimensional instance segmentation representation of the scene.

[0057] The Segment Anything Model (SAM) is used to obtain the instance segmentation mask of the depth image, which can distinguish different targets in the depth image and determine the type of each target. The affine coefficients between multiple 3D superprimitives are determined based on the 3D Gaussian projections of multiple 3D superprimitives and the instance segmentation masks of multiple frames of depth images. The affine coefficients can determine the specific form of the transformation.

[0058] The three-dimensional superprimitives are taken as vertices in the graph structure, and the affine coefficients between the three-dimensional superprimitives are taken as edges in the graph structure. According to the affine coefficients between multiple three-dimensional superprimitives, multiple three-dimensional superprimitives whose affine coefficients are less than the affine coefficient threshold are fused to obtain multiple preliminary three-dimensional instance segmentation representations.

[0059] The preliminary 3D instance segmentation representation is used as the vertex in the new graph structure, and the affine coefficients between the preliminary 3D instance segmentation representations are used as the edges in the new graph structure to fuse multiple preliminary 3D instance segmentation representations. Repeat the fusion multiple times to achieve gradual iterative fusion from local to global, and finally obtain the 3D instance segmentation representation of the scene, complete the 3D instance segmentation, and realize 3D scene understanding.

[0060] By adopting the technical solution of the embodiment of the present application, multiple three-dimensional Gaussian points of the scene can be directly constructed based on the multi-frame depth images of the scene, and the three-dimensional Gaussian points include high-dimensional language features and geometric features. Then, multiple three-dimensional Gaussian points can be clustered according to the similarity of the high-dimensional language features and the similarity of the geometric features of the three-dimensional Gaussian points to obtain multiple three-dimensional super primitives, and the multiple three-dimensional super primitives are fused to obtain a three-dimensional instance segmentation representation of the scene. In this way, three-dimensional scene understanding is realized directly based on three-dimensional Gaussian points, and the problem of domain differences between two-dimensional feature space and three-dimensional feature space is solved.

[0061] On the basis of the above technical solution, as an embodiment, constructing multiple three-dimensional Gaussian points of the scene according to multiple frames of the depth image may include: constructing multiple initial three-dimensional Gaussian points of the scene according to pixel points in the multiple frames of the depth image; the initial three-dimensional Gaussian points include: the geometric features and semantic coding; the semantic coding of the initial three-dimensional Gaussian points is learned from the three-plane features of the latent space pyramid; the semantic coding of the initial three-dimensional Gaussian points is input into a three-dimensional decoder to obtain the high-dimensional language features; and the high-dimensional language features and the initial three-dimensional Gaussian points are combined to obtain the three-dimensional Gaussian points of the scene.

[0062] Each depth image is uniformly sampled to obtain a plurality of sampled pixels. According to the six-degree-of-freedom posture of the camera and the depth information of the sampled pixels, the depth image can be projected back into the three-dimensional space to obtain a sampled point cloud corresponding to the sampled pixels. The information of the point cloud corresponding to the pixel is determined according to the pixel. The three-dimensional Gaussian points are initialized according to the sampled point cloud to obtain a plurality of original three-dimensional Gaussian points, each of which corresponds to a sampled point cloud. The color features, opacity features, position features, rotation features and / or scale features of each original three-dimensional Gaussian point can be determined. The geometric features of the original three-dimensional Gaussian points are determined according to the position features, rotation features and / or scale features of the original three-dimensional Gaussian points.

[0063] Assign a semantic code to the original three-dimensional Gaussian point to obtain the initial three-dimensional Gaussian point. Semantic coding is a representation of semantics and contains semantic information. The dimension of the semantic code of the initial three-dimensional Gaussian point can be 6, and the semantic code is learned from the three-plane features of the latent space pyramid. The three-plane of the latent space pyramid contains fine-grained coding features and coarse-grained coding features; the fine-grained coding feature is a shape of [3, Hf , W ,6] optimization variables; the coarse-grained encoding feature is a shape of [3, Hc , W ,6] optimization variables. 3 represents the three decomposition planes of xy, xz, and yz, and 6 represents the feature dimension of semantic encoding. Hf , W , Hc , W Represent the fine-grained and coarse-grained plane resolutions, respectively.

[0064] The three planes of the latent space pyramid include three decomposition planes: xy, xz, and yz. Data sampling is performed on these three planes. For each position in the three-dimensional scene, information is obtained from different perspectives of the three planes to fully capture the semantic features of each initial three-dimensional Gaussian point in the three-dimensional space. A pyramid-like feature structure is constructed on each decomposition plane. By downsampling the original data to different degrees, feature layers of different resolutions are obtained to form a pyramid shape. The low-level coarse-grained encoding features have a large receptive field and can capture overall and macroscopic semantic information; the high-level fine-grained encoding features have a small receptive field and can retain more detailed information. The features extracted from the three decomposition planes contain spatial information in different directions. These features are fused, and the features from the three decomposition planes are combined to form a feature representation containing the three decomposition planes. The fusion method can be element-by-element addition or splicing. In the pyramid structure, features at different levels have different semantic granularity. In order to make full use of this information, cross-level feature fusion can be performed to fuse the low-level coarse-grained features with the high-level fine-grained features. A neural network model can be used to learn the mapping relationship between the three-plane features of the latent space pyramid and the semantic encoding of the initial three-dimensional Gaussian points. The neural network model can take the fused three-plane features of the pyramid as input, and further abstract and refine the features through multiple layers of convolution, pooling, attention and other operations, and gradually learn the semantic encoding. For example, the convolution layer in the convolutional neural network can automatically extract local patterns in the features, the pooling layer can reduce the dimension of the features and extract the main features, and the attention mechanism can dynamically focus on the importance of different features, thereby learning semantic information. By training the neural network model, the semantic encoding can be made more distinguishable and representative.

[0065] By inputting the semantic coding of the initial three-dimensional Gaussian point into the pre-trained three-dimensional decoder, the high-dimensional language features corresponding to the initial three-dimensional Gaussian point output by the three-dimensional decoder can be obtained. By combining the high-dimensional language features corresponding to the initial three-dimensional Gaussian point with the initial three-dimensional Gaussian point, the three-dimensional Gaussian point of the scene can be obtained. The original three-dimensional Gaussian point does not have semantic coding or high-dimensional language features; the initial three-dimensional Gaussian point has semantic coding; the three-dimensional Gaussian point has high-dimensional language features.

[0066] High-dimensional language features have higher dimensions than low-dimensional semantic encodings. Compared with low-dimensional semantic encodings, high-dimensional language features have stronger feature expression capabilities. Higher dimensions mean a larger feature space and can encode more information. Therefore, based on high-dimensional language features, richer and more detailed semantic information can be captured, such as more complex textures, shapes, or contextual relationships. High-dimensional language features have a more fine-grained distinction capability and can make more detailed distinctions between similar but different semantics. High-dimensional language features may contain more abstract information. High-dimensional language features can encode higher-level abstract information, such as the functions of objects, semantic relationships of scenes, etc., which is very important for complex tasks (such as image generation and semantic segmentation).

[0067] By adopting the technical solution of the embodiment of the present application, the three-dimensional scene can be compactly and efficiently represented using the initial three-dimensional Gaussian points, thereby reducing storage requirements. At the same time, the potential feature field is encoded using the pyramid three-plane structure to enhance the semantic expression ability of the three-dimensional scene. By acquiring high-dimensional language features through a three-dimensional decoder, richer and more detailed semantic information can be obtained, which is conducive to the subsequent three-dimensional scene understanding based on the three-dimensional Gaussian points containing high-dimensional language features.

[0068] In another embodiment, a deep learning network that specifically processes point cloud data can be used to obtain the semantic coding of the initial three-dimensional Gaussian points; the point cloud corresponding to the initial three-dimensional Gaussian points is input into the deep learning network, and the deep learning network automatically extracts local and global features of the point cloud through multi-layer convolution, pooling and other operations, learns the semantic information of the point cloud, and determines the semantic coding of the initial three-dimensional Gaussian points corresponding to the point cloud based on the semantic information of the point cloud.

[0069] In another embodiment, the high-dimensional language features may be obtained by referring to the method for obtaining high-dimensional language feature samples of initial three-dimensional Gaussian point samples described below.

[0070] On the basis of the above technical scheme, as an embodiment, the training step of the three-dimensional decoder at least includes: obtaining semantic coding samples of initial three-dimensional Gaussian point samples; determining high-dimensional language feature samples corresponding to the initial three-dimensional Gaussian point samples and the confidence of the high-dimensional language feature samples according to the semantic coding samples of the initial three-dimensional Gaussian point samples; inputting the initial three-dimensional Gaussian point samples into the initial three-dimensional decoder to obtain the predicted high-dimensional language features of the initial three-dimensional Gaussian point samples; constructing a loss function according to the confidence of the high-dimensional language feature samples, and the cosine similarity between the predicted high-dimensional language features corresponding to the initial three-dimensional Gaussian point samples and the high-dimensional language feature samples; based on the loss function, training the initial three-dimensional decoder to obtain the trained three-dimensional decoder.

[0071] The method for obtaining the initial three-dimensional Gaussian point samples can refer to the method for obtaining the initial three-dimensional Gaussian points described above, and the method for obtaining the semantic coding samples of the initial three-dimensional Gaussian point samples can refer to the method for obtaining the semantic coding of the initial three-dimensional Gaussian points described above.

[0072] According to the semantic coding samples of the initial three-dimensional Gaussian point samples, the high-dimensional language feature samples corresponding to the initial three-dimensional Gaussian point samples can be determined. According to the semantic coding samples of the initial three-dimensional Gaussian point samples, the confidence of the high-dimensional language feature samples corresponding to the initial three-dimensional Gaussian point samples can be determined. The specific method of determining the high-dimensional language feature samples corresponding to the initial three-dimensional Gaussian point samples and the confidence of the high-dimensional language feature samples will be described in detail later. The confidence of the high-dimensional language feature sample can characterize the accuracy of the high-dimensional language feature sample.

[0073] The initial three-dimensional decoder is a three-dimensional decoder to be trained. The initial three-dimensional decoder can predict high-dimensional language features, but the initial three-dimensional decoder has not been trained yet, so the predicted high-dimensional language features are not accurate enough. The initial three-dimensional Gaussian point sample is input into the initial three-dimensional decoder, and the predicted high-dimensional language features corresponding to the initial three-dimensional Gaussian point sample output by the initial three-dimensional Gaussian point sample can be obtained.

[0074] A loss function is constructed based on the high-dimensional language feature samples, and the initial three-dimensional decoder is supervisedly trained using the loss function so that the predicted high-dimensional language features output by the initial three-dimensional decoder can be close to the high-dimensional language feature samples. Optionally, a loss function can be constructed based on the confidence of the high-dimensional language feature samples and the cosine similarity between the predicted high-dimensional language features corresponding to the initial three-dimensional Gaussian point samples and the high-dimensional language feature samples. With the goal of minimizing the loss function, the initial three-dimensional decoder is trained to obtain a trained three-dimensional decoder.

[0075] Alternatively, the loss function can be determined by the following formula: ; in, Representation loss function, Characterizes the number of initial three-dimensional Gaussian point samples, Characterization The confidence of the high-dimensional language feature samples corresponding to the initial three-dimensional Gaussian point samples, Representation prediction of high-dimensional language feature samples, Characterize the high-dimensional language feature samples corresponding to the initial three-dimensional Gaussian point samples, Characterizes the cosine distance.

[0076] By adopting the technical solution of the embodiment of the present application, a three-dimensional decoder can be pre-trained to obtain high-dimensional language features of three-dimensional Gaussian points. The three-dimensional decoder can be used to quickly obtain high-dimensional language features of three-dimensional Gaussian points, thereby improving the efficiency of obtaining high-dimensional language features.

[0077] On the basis of the above technical solution, as an embodiment, determining the high-dimensional language feature samples corresponding to the initial three-dimensional Gauss point samples according to the semantic coding samples of the initial three-dimensional Gauss point samples may include: obtaining multiple frames of depth image samples of the scene samples corresponding to the initial three-dimensional Gauss point samples; extracting language feature maps of multiple frames of the depth image samples; performing multi-angle projection on the initial three-dimensional Gauss point samples to obtain the two-dimensional pixel position information corresponding to the initial three-dimensional Gauss point samples in the multiple frames of the depth image samples; obtaining the visual relationship of the initial three-dimensional Gauss point samples in the multiple frames of the depth image samples; determining the target depth image samples corresponding to the initial three-dimensional Gauss point samples according to the visual relationship; the initial three-dimensional Gauss point samples are visible in the target depth image samples; determining the two-dimensional language features corresponding to the initial three-dimensional Gauss point samples in the target depth image samples according to the language feature maps of the multiple frames of the depth image samples and the two-dimensional pixel position information corresponding to the initial three-dimensional Gauss point samples in the target depth image samples; performing average pooling on the two-dimensional language features corresponding to the initial three-dimensional Gauss point samples in the target depth image samples to obtain the high-dimensional language feature samples corresponding to the initial three-dimensional Gauss point samples.

[0078] The scene sample can be any three-dimensional scene. The scene sample is photographed using a color depth camera to obtain multiple frames of depth image samples. From the multiple frames of depth image samples, the initial three-dimensional Gaussian point samples can be obtained. The semantic segmentation model can be used to extract features from the depth image samples to obtain a language feature map of the shape of H×W×D corresponding to each frame of the depth image sample, where D represents the dimension of the extracted language feature and H×W represents the size. Optionally, the semantic segmentation model can be a language-driven semantic segmentation (LSeg) model.

[0079] Based on the multi-frame depth image samples, the initial three-dimensional Gaussian point samples are obtained. According to the angle of the depth image samples, the initial three-dimensional Gaussian point samples are projected at multiple angles to obtain the two-dimensional pixel position information corresponding to each initial three-dimensional Gaussian point sample in the multi-frame depth image samples. According to the two-dimensional pixel position information corresponding to the initial three-dimensional Gaussian point samples in the multi-frame depth image samples, the distance information of the two-dimensional pixels corresponding to the initial three-dimensional Gaussian point samples in the multi-frame depth image samples can be determined. According to the distance information of the two-dimensional pixels corresponding to the initial three-dimensional Gaussian point samples in the depth image samples, and the depth distance of the initial three-dimensional Gaussian point samples at the viewing angle corresponding to the depth image samples, the visual relationship of the initial three-dimensional Gaussian point samples at the viewing angle corresponding to the depth image samples can be determined. If the distance between the two-dimensional pixels corresponding to the initial three-dimensional Gaussian point sample in the depth image sample is smaller than the depth distance of the initial three-dimensional Gaussian point sample at the viewing angle corresponding to the depth image sample, and the difference between the depth distance of the initial three-dimensional Gaussian point sample at the viewing angle corresponding to the depth image sample and the distance between the two-dimensional pixels corresponding to the initial three-dimensional Gaussian point sample in the depth image sample is smaller than the visible threshold, it means that the initial three-dimensional Gaussian point sample is blocked, and the visual relationship at the viewing angle corresponding to the depth image sample is invisible.

[0080] According to the visual relationship between each initial three-dimensional Gaussian point sample in the multi-frame depth image samples, the target depth image corresponding to each initial three-dimensional Gaussian point sample is determined. The visual relationship between the initial three-dimensional Gaussian point sample in the corresponding target depth image is visible.

[0081] According to the two-dimensional pixel position information corresponding to the initial three-dimensional Gaussian point sample in the target depth image sample and the language feature map of the target depth image sample corresponding to the initial three-dimensional Gaussian point sample, the two-dimensional language feature corresponding to the initial three-dimensional Gaussian point sample in the corresponding target depth image sample is determined. The two-dimensional language features corresponding to the initial three-dimensional Gaussian point sample in multiple target depth image samples are average-pooled to obtain a high-dimensional language feature sample corresponding to the initial three-dimensional Gaussian point sample.

[0082] By adopting the technical solution of the embodiment of the present application, the high-dimensional language feature samples of the initial three-dimensional Gauss point samples can be determined based on the visual relationship of the initial three-dimensional Gauss point samples under multiple perspectives and the two-dimensional language features of the initial three-dimensional Gauss point samples under multiple perspectives, so that the calculated high-dimensional language feature samples of the initial three-dimensional Gauss point samples can be used as the learning target of the three-dimensional decoder to quickly acquire the high-dimensional language features.

[0083] On the basis of the above technical solution, as an embodiment, determining the confidence of the high-dimensional language feature sample corresponding to the initial three-dimensional Gauss point sample according to the semantic coding sample of the initial three-dimensional Gauss point sample can include: obtaining the variance of the two-dimensional language feature corresponding to the initial three-dimensional Gauss point sample in the target depth image sample; determining the confidence of the high-dimensional language feature sample corresponding to the initial three-dimensional Gauss point sample according to the visual relationship and the variance of the two-dimensional language feature corresponding to the initial three-dimensional Gauss point sample.

[0084] Obtain the variance of the two-dimensional language features corresponding to the initial three-dimensional Gaussian point samples in the visible target depth image samples. Because an initial three-dimensional Gaussian point sample corresponds to a physical point, the two-dimensional language features of the initial three-dimensional Gaussian point sample in multiple target depth image samples should be the same or close. Therefore, if the variance of the two-dimensional language features corresponding to the initial three-dimensional Gaussian point samples in the visible target depth image samples is large, the confidence of the high-dimensional language feature samples obtained based on the two-dimensional language features of the initial three-dimensional Gaussian point samples is not high.

[0085] The confidence of the high-dimensional language feature sample corresponding to the initial three-dimensional Gaussian point sample can be calculated based on the visual relationship of the initial three-dimensional Gaussian point sample and the variance of the two-dimensional language feature corresponding to the initial three-dimensional Gaussian point sample. Optionally, the confidence can be calculated according to the following formula: ; in, Represents confidence, Represents the number of visible times, It represents the variance of the two-dimensional language features corresponding to the initial three-dimensional Gaussian point samples; the number of visible times represents the number of target depth image samples corresponding to the initial three-dimensional Gaussian point samples.

[0086] By adopting the technical solution of the embodiment of the present application, the confidence of the high-dimensional language feature samples corresponding to the initial three-dimensional Gaussian point samples can be calculated based on the variance and visual relationship of the two-dimensional language features corresponding to the initial three-dimensional Gaussian point samples, so that the three-dimensional decoder can learn more high-confidence and more accurate high-dimensional language feature samples during learning, thereby improving the performance of the three-dimensional decoder.

[0087] On the basis of the above technical solution, as an embodiment, the fusing of the multiple three-dimensional superprimitives to obtain the three-dimensional instance segmentation representation of the scene may include: determining the affine coefficients between the multiple three-dimensional superprimitives based on the projections of the three-dimensional Gaussian points of the multiple three-dimensional superprimitives and the instance segmentation masks of the multiple frames of the depth images; based on the affine coefficients between the multiple three-dimensional superprimitives, fusing the multiple three-dimensional superprimitives whose affine coefficients are less than the affine coefficient threshold to obtain multiple preliminary three-dimensional instance segmentation representations; iteratively fusing the multiple preliminary three-dimensional instance segmentation representations to obtain the three-dimensional instance segmentation representation of the scene.

[0088] Multiple frames of depth images can be input into the segmentation model separately to obtain instance segmentation masks of each frame of depth image. The instance segmentation mask can distinguish different targets in the depth image and determine the type of each target. Different types of targets have different labels for instance segmentation masks.

[0089] The three-dimensional Gaussian points in multiple three-dimensional superprimitives are projected into a multi-frame depth image. According to the projections of the three-dimensional Gaussian points in the three-dimensional superprimitives in the multi-frame depth image and the instance segmentation masks of the multi-frame depth image, the distribution of labels of the instance segmentation masks of the projections of the three-dimensional superprimitives in the multi-frame depth image can be determined.

[0090] According to the distribution of multiple three-dimensional superprimitives, the affine coefficients between multiple three-dimensional superprimitives can be obtained by calculating the JS divergence (Jensen-Shannon divergence) of different distributions. The affine coefficients can determine the specific form of the transformation.

[0091] Taking 3D superprimitives as vertices in the graph structure and affine coefficients between 3D superprimitives as edges in the graph structure, multiple 3D superprimitives whose affine coefficients are less than the affine coefficient threshold are fused according to the affine coefficients between multiple 3D superprimitives to obtain multiple preliminary 3D instance segmentation representations. The affine coefficient threshold can be set according to actual needs. The initial 3D instance segmentation representation includes multiple 3D superprimitives with similar distances.

[0092] The preliminary 3D instance segmentation representations are used as vertices in the new graph structure, and the affine coefficients between the preliminary 3D instance segmentation representations are calculated by referring to the method for calculating the affine coefficients between 3D superprimitives. The affine coefficients between the preliminary 3D instance segmentation representations are used as edges in the new graph structure, and multiple preliminary 3D instance segmentation representations whose affine coefficients are less than the affine coefficient threshold are fused.

[0093] Repeat the fusion process multiple times to achieve gradual iterative fusion from local to global, and finally obtain the 3D instance segmentation representation of the scene, complete the 3D instance segmentation, and realize 3D scene understanding.

[0094] The technical solution of the embodiment of the present application is adopted, based on the graph structure modeling mechanism, the three-dimensional super primitives are gradually fused, and the gradual fusion process is based on the affine coefficients, which are determined according to the distribution of the labels of the instance segmentation mask of the projection of the three-dimensional super primitive under the multi-frame depth image, so the affine coefficients use semantic information. In general, the fusion process utilizes both graph structure information and semantic information, integrates the interactive relationship between semantics and geometric features in three-dimensional space, realizes accurate fusion, and realizes instance-level segmentation of scenes using semantic consistency and geometric relationships.

[0095] On the basis of the above technical solution, as an embodiment, the determining of the affine coefficients between the multiple three-dimensional superprimitives according to the three-dimensional Gaussian projections of the multiple three-dimensional superprimitives and the instance segmentation masks of the multiple frames of the depth images may include: projecting the three-dimensional superprimitive to obtain multiple two-dimensional images corresponding to the three-dimensional superprimitives of the multiple frames of the depth images; determining the distribution of the instance segmentation masks in the multiple two-dimensional images according to the two-dimensional images and the instance segmentation masks of the multiple frames of the depth images; determining the affine coefficients between the multiple three-dimensional superprimitives in the depth image of the same frame according to the distribution of the instance segmentation masks of the two-dimensional images of the depth image of the same frame corresponding to the multiple three-dimensional superprimitives; obtaining the visible ratio of the two-dimensional image in the corresponding depth image; determining the affine coefficients between the multiple three-dimensional superprimitives according to the affine coefficients between the multiple three-dimensional superprimitives in the depth images of the same frame and the visible ratio of the two-dimensional images in the depth images of the same frame.

[0096] When calculating the affine coefficients between multiple three-dimensional superprimitives, the three-dimensional Gaussian points in the multiple three-dimensional superprimitives are projected into a multi-frame depth image. Based on the projections of the three-dimensional Gaussian points in the three-dimensional superprimitives in the multi-frame depth image and the instance segmentation masks of the multi-frame depth image, the distribution of the labels of the instance segmentation masks of the projections of the three-dimensional superprimitives in the multi-frame depth image can be determined.

[0097] According to the distribution of labels of instance segmentation masks of multiple 3D superprimitives in the kth frame, the affine coefficients between multiple 3D superprimitives in the kth frame can be obtained by calculating the JS divergence of different distributions. Optionally, the affine coefficients between 3D superprimitives in the kth frame can be calculated by the following formula: ; in, represents the affine coefficient between the i-th 3D superprimitive and the j-th 3D superprimitive in the k-frame, Characterizes the distribution of the i-th 3D superprimitive in the k-frame, Characterizes the distribution of the jth 3D superprimitive in the k frame, =( + ) / 2, Characterization and The average distribution of Characterization label.

[0098] Since the instance segmentation of a single frame may not be accurate, resulting in noise in the calculation of the affine coefficient, the information of multiple frames is fused according to the visible ratio of the three-dimensional superprimitive in different frames to obtain the final multi-frame fusion affine coefficient. The visible ratio can be determined by the method for determining the visible relationship described above, and the number of three-dimensional Gaussian points visible in the k-th frame depth image is divided by the number of three-dimensional Gaussian points visible in the three-dimensional superprimitive in the k-th frame depth image, and the visible ratio of the two-dimensional image projected by the three-dimensional superprimitive in the depth image is obtained.

[0099] According to the affine coefficients between the multiple 3D superprimitives in the depth images of the same frames and the visible ratio of the 2D image in the depth images of the same frames, the affine coefficients between the multiple 3D superprimitives are determined according to the following formula: ; in, represents the affine coefficient between the i-th 3D superprimitive and the j-th 3D superprimitive, represents the affine coefficient between the i-th 3D super primitive and the j-th 3D super primitive in the k-frame, where k represents the number of frames. represents the visible ratio of the i-th 3D superprimitive in the k-th frame depth image, Characterizes the visible proportion of the j-th 3D superprimitive in the k-th depth image.

[0100] By adopting the technical solution of the embodiment of the present application, the final multi-frame fusion affine coefficient is obtained according to the affine coefficient of the three-dimensional superprimitive in multiple frames. Whether to fuse the three-dimensional superprimitive is determined based on the multi-frame fusion affine coefficient, so as to obtain a more accurate fusion result.

[0101] By adopting the technical solution of the embodiment of the present application, there is no need to explicitly store the semantic features of the three-dimensional space. The language feature information of the scene is modeled by three planes with multiple resolutions, which can effectively reduce the number of parameters of the model and segmentation. In addition, three-dimensional super primitives are constructed by reconstructing geometric features and high-dimensional language features, and the segmentation model is used to construct the affine relationship between the three-dimensional super primitives, so as to perform clustering and obtain three-dimensional instance segmentation results. It is proved through mathematical derivation and experimental comparison that the scene representation scheme based on three-dimensional super primitives has better progressive segmentation convergence, and the affine coefficient as the edge calculation method effectively reduces the influence of noise and improves the stability and accuracy of segmentation.

[0102] The embodiments of the present application introduce the joint optimization of semantics and geometry. In multiple open vocabulary segmentation tasks, the three-dimensional semantic segmentation and instance segmentation of the embodiments of the present application are effectively improved, which is significantly better than the method with fixed label sets, especially in the unseen categories. In addition, the embodiments of the present application also support open vocabulary query and instance segmentation. Therefore, the technical solution proposed in the embodiments of the present application can effectively solve the current storage overhead, open vocabulary adaptability and segmentation accuracy problems in the field of three-dimensional panoramic segmentation, and has a wide range of practical application value.

[0103] In order to better implement the scene understanding method of the present application, the present application also provides a scene understanding device based on the above scene understanding method. The meanings of the terms are the same as those in the above scene understanding method, and the specific implementation details can refer to the description in the method embodiment.

[0104] See also Figure 2 , Figure 2 : is a schematic diagram of the structure of a scene understanding device provided in an embodiment of the present application, and the scene understanding device includes: An acquisition module 201 is used to acquire multiple depth images of a scene; A construction module 202 is used to construct a plurality of three-dimensional Gaussian points of the scene according to the multiple frames of the depth image; the three-dimensional Gaussian points include: high-dimensional language features and geometric features; A clustering module 203, configured to cluster the plurality of three-dimensional Gaussian points according to the similarity of the high-dimensional linguistic features and the similarity of the geometric features of the plurality of three-dimensional Gaussian points to obtain a plurality of three-dimensional super-primitives; The fusion module 204 is used to fuse the multiple 3D superprimitives to obtain a 3D instance segmentation representation of the scene.

[0105] In one embodiment, the building block 202 includes: A construction unit, configured to construct a plurality of initial three-dimensional Gaussian points of the scene according to pixel points in the multiple frames of the depth image; the initial three-dimensional Gaussian points include: the geometric features and semantic coding; the semantic coding of the initial three-dimensional Gaussian points is learned from the three-plane features of the latent space pyramid; An input unit, used for inputting the semantic coding of the initial three-dimensional Gaussian point into a three-dimensional decoder to obtain the high-dimensional language feature; A combining unit is used to combine the high-dimensional language features and the initial three-dimensional Gaussian points to obtain the three-dimensional Gaussian points of the scene.

[0106] In one embodiment, the step of training the 3D decoder at least comprises: Obtain semantic coding samples of initial three-dimensional Gaussian point samples; Determining, according to the semantic coding samples of the initial three-dimensional Gaussian point samples, high-dimensional language feature samples corresponding to the initial three-dimensional Gaussian point samples and the confidence of the high-dimensional language feature samples; Inputting the initial three-dimensional Gaussian point samples into an initial three-dimensional decoder to obtain predicted high-dimensional language features of the initial three-dimensional Gaussian point samples; Constructing a loss function according to the confidence of the high-dimensional language feature sample and the cosine similarity between the predicted high-dimensional language feature corresponding to the initial three-dimensional Gaussian point sample and the high-dimensional language feature sample; Based on the loss function, the initial 3D decoder is trained to obtain the trained 3D decoder.

[0107] In one embodiment, determining the high-dimensional language feature samples corresponding to the initial three-dimensional Gaussian point samples according to the semantic coding samples of the initial three-dimensional Gaussian point samples includes: Acquire multiple frames of depth image samples of scene samples corresponding to the initial three-dimensional Gaussian point samples; Extracting language feature maps of multiple frames of depth image samples; Performing multi-angle projection on the initial three-dimensional Gaussian point samples to obtain two-dimensional pixel position information corresponding to the initial three-dimensional Gaussian point samples in multiple frames of the depth image samples; Acquire the visual relationship between the initial three-dimensional Gaussian point samples in multiple frames of depth image samples; Determining, according to the visible relationship, a target depth image sample corresponding to the initial three-dimensional Gaussian point sample; the initial three-dimensional Gaussian point sample is visible in the target depth image sample; Determine the two-dimensional language features corresponding to the initial three-dimensional Gaussian point samples in the target depth image samples according to the language feature graphs of the multiple frames of the depth image samples and the two-dimensional pixel position information corresponding to the initial three-dimensional Gaussian point samples in the target depth image samples; Average pooling is performed on the two-dimensional language features corresponding to the initial three-dimensional Gaussian point samples in the target depth image samples to obtain high-dimensional language feature samples corresponding to the initial three-dimensional Gaussian point samples.

[0108] In one embodiment, determining the confidence of the high-dimensional language feature sample corresponding to the initial three-dimensional Gaussian point sample according to the semantic coding sample of the initial three-dimensional Gaussian point sample includes: Obtaining the variance of the two-dimensional language features corresponding to the initial three-dimensional Gaussian point samples in the target depth image samples; The confidence of the high-dimensional language feature sample corresponding to the initial three-dimensional Gaussian point sample is determined according to the visual relationship and the variance of the two-dimensional language feature corresponding to the initial three-dimensional Gaussian point sample.

[0109] In one embodiment, the fusion module 204 includes: A determining unit, configured to determine affine coefficients between the plurality of the three-dimensional superprimitives according to projections of the three-dimensional Gaussian points of the plurality of the three-dimensional superprimitives and instance segmentation masks of the plurality of frames of the depth images; a fusion unit, configured to fuse, according to the affine coefficients between the plurality of the three-dimensional superprimitives, the plurality of three-dimensional superprimitives whose affine coefficients are less than an affine coefficient threshold, to obtain a plurality of preliminary three-dimensional instance segmentation representations; An iterative unit is used to iteratively fuse the multiple preliminary 3D instance segmentation representations to obtain a 3D instance segmentation representation of the scene.

[0110] In one embodiment, the determining unit includes: A projection subunit, configured to project the three-dimensional superprimitive to obtain a plurality of two-dimensional images of the three-dimensional superprimitive corresponding to a plurality of frames of the depth image; a distribution determination subunit, configured to determine the distribution of instance segmentation masks in a plurality of the two-dimensional images according to the instance segmentation masks of the two-dimensional image and the plurality of frames of the depth image; a coefficient determination subunit, configured to determine affine coefficients between a plurality of the three-dimensional superprimitives in the depth image of the same frame according to distribution of instance segmentation masks of the two-dimensional images of the depth image of the same frame corresponding to each of the plurality of the three-dimensional superprimitives; A ratio acquisition subunit, used to acquire a visible ratio of the two-dimensional image in the corresponding depth image; The affine coefficient determination subunit is used to determine the affine coefficients between the multiple three-dimensional superprimitives according to the affine coefficients between the multiple three-dimensional superprimitives in the depth images of the multiple same frames and the visible ratio of the two-dimensional image in the depth images of the multiple same frames.

[0111] By adopting the technical solution of the embodiment of the present application, multiple three-dimensional Gaussian points of the scene can be directly constructed according to the multi-frame depth image of the scene, and the three-dimensional Gaussian points include high-dimensional language features and geometric features, and then the multiple three-dimensional Gaussian points can be clustered according to the similarity of the high-dimensional language features and the similarity of the geometric features of the three-dimensional Gaussian points to obtain multiple three-dimensional super primitives, and the multiple three-dimensional super primitives are fused to obtain a three-dimensional instance segmentation representation of the scene. In this way, three-dimensional scene understanding is realized directly based on three-dimensional Gaussian points, and the problem of domain differences between two-dimensional feature space and three-dimensional feature space is solved. For the specific definition of the scene understanding device, please refer to the definition of the scene understanding method above, which will not be repeated here. The various modules in the above-mentioned scene understanding device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0112] In addition, the present application also provides an electronic device, such as Figure 3 As shown, it shows a schematic diagram of the structure of the electronic device involved in this application, specifically: The electronic device may include one or more processors 301 of processing cores and one or more computer-readable storage media memories 302 and other components. Those skilled in the art will appreciate that Figure 3 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. The processor 301 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, it executes various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 301.

[0113] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and data processing by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.

[0114] In one embodiment, the electronic device further includes a power supply 303 for supplying power to various components. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 303 can also include any components such as one or more DC or AC power supplies, recharging systems, power supply device debugging circuits, power converters or inverters, and power status indicators.

[0115] In one embodiment, the electronic device may further include an input unit 304, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0116] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 301 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application programs stored in the memory 302, thereby implementing the steps in any of the scene understanding methods provided in the embodiments of the present application.

[0117] Those skilled in the art will understand that Figure 3 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0118] In one embodiment, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method described in any embodiment of the present application is implemented.

[0119] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in any embodiment of the present application is implemented.

[0120] In some embodiments, a computer program product is also proposed, including a computer program or instructions, which implement the method described in any embodiment of the present application when executed by a processor.

[0121] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0122] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0123] To this end, the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program can be loaded by a processor to execute the steps in any scene understanding method provided in the present application.

[0124] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0125] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0126] Since the instructions stored in the computer-readable storage medium can execute the steps in any scene understanding method provided in the present application, the beneficial effects that can be achieved by any scene understanding method provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0127] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0128] The above is a detailed introduction to a scene understanding method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A scene understanding method, characterized in that: include: Acquire multiple frames of depth images of the scene; Constructing a plurality of three-dimensional Gaussian points of the scene according to the plurality of frames of the depth image; The three-dimensional Gaussian points include: high-dimensional language features and geometric features; Clustering the plurality of three-dimensional Gaussian points according to the similarity of the high-dimensional linguistic features and the similarity of the geometric features of the plurality of three-dimensional Gaussian points to obtain a plurality of three-dimensional super-primitives; A plurality of the three-dimensional superprimitives are fused to obtain a three-dimensional instance segmentation representation of the scene.

2. A scene understanding method according to claim 1, characterized in that: The step of constructing a plurality of three-dimensional Gaussian points of the scene according to the plurality of frames of the depth image comprises: Constructing a plurality of initial three-dimensional Gaussian points of the scene according to the pixel points in the multiple frames of the depth image; the initial three-dimensional Gaussian points include: the geometric features and the semantic coding; the semantic coding of the initial three-dimensional Gaussian points is learned from the three-plane features of the latent space pyramid; Inputting the semantic coding of the initial three-dimensional Gaussian points into a three-dimensional decoder to obtain the high-dimensional language features; The high-dimensional language features and the initial three-dimensional Gaussian points are combined to obtain the three-dimensional Gaussian points of the scene.

3. A scene understanding method according to claim 2, characterized in that: The training step of the three-dimensional decoder at least includes: Obtain semantic coding samples of initial three-dimensional Gaussian point samples; Determining, according to the semantic coding samples of the initial three-dimensional Gaussian point samples, high-dimensional language feature samples corresponding to the initial three-dimensional Gaussian point samples and the confidence of the high-dimensional language feature samples; Inputting the initial three-dimensional Gaussian point samples into an initial three-dimensional decoder to obtain predicted high-dimensional language features of the initial three-dimensional Gaussian point samples; Constructing a loss function according to the confidence of the high-dimensional language feature sample and the cosine similarity between the predicted high-dimensional language feature corresponding to the initial three-dimensional Gaussian point sample and the high-dimensional language feature sample; Based on the loss function, the initial 3D decoder is trained to obtain the trained 3D decoder.

4. A scene understanding method according to claim 3, characterized in that: The step of determining the high-dimensional language feature samples corresponding to the initial three-dimensional Gaussian point samples according to the semantically encoded samples of the initial three-dimensional Gaussian point samples comprises: Acquire multiple frames of depth image samples of scene samples corresponding to the initial three-dimensional Gaussian point samples; Extracting language feature maps of multiple frames of depth image samples; Performing multi-angle projection on the initial three-dimensional Gaussian point samples to obtain two-dimensional pixel position information corresponding to the initial three-dimensional Gaussian point samples in multiple frames of the depth image samples; Acquire the visual relationship between the initial three-dimensional Gaussian point samples in multiple frames of depth image samples; Determining, according to the visible relationship, a target depth image sample corresponding to the initial three-dimensional Gaussian point sample; the initial three-dimensional Gaussian point sample is visible in the target depth image sample; Determine the two-dimensional language features corresponding to the initial three-dimensional Gaussian point samples in the target depth image samples according to the language feature graphs of the multiple frames of the depth image samples and the two-dimensional pixel position information corresponding to the initial three-dimensional Gaussian point samples in the target depth image samples; Average pooling is performed on the two-dimensional language features corresponding to the initial three-dimensional Gaussian point samples in the target depth image samples to obtain high-dimensional language feature samples corresponding to the initial three-dimensional Gaussian point samples.

5. A scene understanding method according to claim 4, characterized in that: Determining the confidence of the high-dimensional language feature sample corresponding to the initial three-dimensional Gaussian point sample according to the semantic coding sample of the initial three-dimensional Gaussian point sample includes: Obtaining the variance of the two-dimensional language features corresponding to the initial three-dimensional Gaussian point samples in the target depth image samples; The confidence of the high-dimensional language feature sample corresponding to the initial three-dimensional Gaussian point sample is determined according to the visual relationship and the variance of the two-dimensional language feature corresponding to the initial three-dimensional Gaussian point sample.

6. A scene understanding method according to claim 1, characterized in that: The step of fusing the plurality of three-dimensional superprimitives to obtain a three-dimensional instance segmentation representation of the scene includes: Determine affine coefficients between the plurality of the three-dimensional superprimitives according to projections of the three-dimensional Gaussian points of the plurality of the three-dimensional superprimitives and instance segmentation masks of the plurality of frames of the depth images; According to the affine coefficients between the plurality of the three-dimensional superprimitives, a plurality of the three-dimensional superprimitives whose affine coefficients are less than an affine coefficient threshold are merged to obtain a plurality of preliminary three-dimensional instance segmentation representations; Iteratively fuse the multiple preliminary 3D instance segmentation representations to obtain a 3D instance segmentation representation of the scene.

7. A scene understanding method according to claim 6, characterized in that: The step of determining affine coefficients between the plurality of the three-dimensional superprimitives according to the three-dimensional Gaussian projections of the plurality of the three-dimensional superprimitives and the instance segmentation masks of the plurality of frames of the depth images comprises: Projecting the three-dimensional superprimitive to obtain a plurality of two-dimensional images corresponding to a plurality of frames of the depth image of the three-dimensional superprimitive; Determine, according to the instance segmentation masks of the two-dimensional image and the multiple frames of the depth image, a distribution of the instance segmentation masks in the multiple two-dimensional images; Determine affine coefficients between the plurality of three-dimensional superprimitives in the depth image of the same frame according to distribution of instance segmentation masks of the two-dimensional images of the depth image of the same frame corresponding to the plurality of three-dimensional superprimitives; Obtaining a visible ratio of the two-dimensional image in the corresponding depth image; Affine coefficients between the plurality of three-dimensional superprimitives are determined according to affine coefficients between the plurality of three-dimensional superprimitives in the depth images of the same frames and visible ratios of the two-dimensional images in the depth images of the same frames.

8. A scene understanding device, characterized in that: include: An acquisition module, used to acquire multiple frames of depth images of a scene; A construction module, used to construct a plurality of three-dimensional Gaussian points of the scene according to a plurality of frames of the depth image; The three-dimensional Gaussian points include: high-dimensional language features and geometric features; A clustering module, used for clustering the plurality of three-dimensional Gaussian points according to the similarity of the high-dimensional linguistic features and the similarity of the geometric features of the plurality of three-dimensional Gaussian points to obtain a plurality of three-dimensional super-primitives; A fusion module is used to fuse the multiple three-dimensional super primitives to obtain a three-dimensional instance segmentation representation of the scene.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in a scene understanding method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the scene understanding method according to any one of claims 1 to 7 are implemented.

Images