A method and apparatus for representing a three-dimensional scene based on hyperquadrics

Abstract

The application provides a three-dimensional scene representation method and device based on a hyperquadric surface, the application proposes the hyperquadric surface with stronger geometric representation capability as a basic unit of a scene, efficiently models a complex structure by using rich shapes of the hyperquadric surface, constructs a probability mixed model based on the hyperquadric surface, models each hyperquadric surface as a local occupancy probability distribution with a geometric prior, and realizes three-dimensional occupancy prediction by using a probability mixed superposition mode, proposes a three-dimensional semantic occupancy prediction framework based on the hyperquadric surface, designs a clipping and splitting module to make the hyperquadric surface be concentratedly distributed in an occupancy area, improves modeling efficiency, and verifies performance of a three-dimensional semantic occupancy prediction task on a data set.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and machine learning technology, specifically relating to a method and apparatus for representing three-dimensional scenes based on hyperquadratic surfaces. Background Technology

[0002] Vision-centric 3D semantic occupancy prediction has garnered significant attention in recent years due to its crucial applications in autonomous driving. The goal of this task is to predict the occupancy state and semantic label of each voxel in a 3D environment based on visual input, thereby achieving comprehensive perception of the surrounding environment and providing fine-grained structural and semantic information.

[0003] While 3D semantic occupancy prediction has shown great potential in various applications, it faces significant computational efficiency challenges due to the need for dense 3D prediction. This has prompted researchers to urgently design a 3D scene representation method that is both efficient and highly expressive. Currently, the mainstream voxel method models scene details using dense 3D meshes, but this approach fails to fully utilize the sparsity features of autonomous driving scenarios, resulting in high computational costs. Therefore, recent research has begun to explore object-centric representation methods based on Gaussian distributions. This method models the scene using a set of learnable 3D Gaussians, each describing the occupancy probability distribution of a local region with parameters such as position, covariance matrix, opacity, and semantic attributes. This representation can sparsely model the scene, alleviating the computational bottleneck of voxel methods to some extent. However, due to the constraints of its mathematical structure, Gaussians can only model objects in space using ellipsoidal shapes, limiting their ability to express complex geometric structures. In real-world driving scenarios, object shapes are diverse, and a small number of Gaussians is insufficient to accurately reconstruct the target geometry. To improve reconstruction accuracy, a large number of high-density Gaussians must be stacked, severely impacting representation efficiency. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] Therefore, the first objective of this invention is to propose a method for representing three-dimensional scenes based on hyperquadratic surfaces.

[0006] This invention introduces hyperquadratic surfaces as the basic unit for scene modeling, employing a probabilistic superposition approach for modeling. A pruning and splitting module is also incorporated into the model to improve the utilization rate of these modeling primitives. This invention proposes a 3D semantic occupancy prediction framework based on hyperquadratic surfaces. It utilizes the geometric properties of hyperquadratic surfaces to accurately and efficiently model complex 3D scenes. Furthermore, a probabilistic fusion method based on hyperquadratic surfaces is proposed, and its effectiveness has been verified on 3D semantic occupancy prediction tasks.

[0007] The second objective of this invention is to propose a three-dimensional scene representation device based on a hyperquadratic surface.

[0008] To achieve the above objectives, a first aspect of the present invention proposes a method for representing a three-dimensional scene based on a hyperquadratic surface, comprising:

[0009] S1, using an image backbone network to extract multi-scale image features of multi-view images of a 3D scene, and based on the multi-scale image features, initialize a set of hyperquadratic surfaces, and iteratively update the attribute parameters of the hyperquadratic surfaces through coding blocks; S2, model each hyperquadric surface of the probabilistic hyperquadric surface hybrid model constructed based on attribute parameters as the occupancy probability distribution of a local region, and superimpose the occupancy probabilities of all related hyperquadric surfaces through hybrid weights to obtain the final occupancy probability of the target point in the 3D scene; S3, based on the final occupancy probability of the target point, perform a weighted average of the semantic probabilities of all relevant hyperquadratic surfaces to obtain the semantic prediction result of the target point; S4. Based on the semantic prediction results, small-scale hyperquadratic surfaces are pruned and large-scale hyperquadratic surfaces are divided to optimize their distribution density in three-dimensional space.

[0010] In one embodiment of the present invention, S1 includes: The geometric properties of the current hyperquadratic surface are converted into feature vectors by quadratic surface coding blocks. The feature vectors include embedded representations of position, scale, rotation angle and shape index. The feature vector is locally feature-encoded using 3D sparse convolution, and multi-scale image features and hyperquadratic surface features are fused through a deformable attention mechanism to update the attribute parameters of the hyperquadratic surface.

[0011] In one embodiment of the present invention, S2 includes: The target points within the 3D scene are transformed to the local coordinate system of each hyperquadratic surface, wherein the transformation is based on a rigid transformation of the position and rotation of the hyperquadratic surface; The occupancy probability of a target point in a 3D scene in the local coordinate system of a hyperquadratic surface is calculated. The probability is determined by the scale, shape index, and opacity of the hyperquadratic surface. The final occupancy probability is obtained by superimposing the normalized mixed weights, where the mixed weights are the occupancy probabilities of each hyperquadratic surface at the target point.

[0012] In one embodiment of the present invention, S4 further includes: Based on the semantic prediction results, hyperquadratic surfaces with a scale smaller than the preset pruning threshold are removed from the three-dimensional space. The pruning threshold is dynamically calculated based on the primitive distribution density in the early stage of training. A splitting operation is performed on the hyperquadratic surface with a scale greater than the pruning threshold to obtain multiple sub-surfaces. The splitting operation generates multiple sub-surfaces based on the semantic uncertainty of the covered region and retains the residual update of the original semantic and geometric properties.

[0013] In one embodiment of the present invention, the method further includes: The attribute parameters of the hyperquadratic surface are jointly optimized by cross-entropy loss and lovasz softmax loss in the 3D semantic occupancy prediction task. The cross-entropy loss is used to measure the accuracy of semantic prediction, and the lovasz softmax loss is used to alleviate the class imbalance problem and improve the overall prediction performance.

[0014] To achieve the above objectives, a second aspect of the present invention provides a three-dimensional scene representation device based on a hyperquadratic surface, comprising: The initialization and parameter update module is used to extract multi-scale image features of multi-view images of a 3D scene using an image backbone network, initialize a set of hyperquadratic surfaces based on the multi-scale image features, and iteratively update the attribute parameters of the hyperquadratic surfaces through coding blocks. The probabilistic hybrid modeling module is used to model each hyperquad surface of the probabilistic hyperquad surface hybrid model constructed based on attribute parameters as the occupancy probability distribution of a local region, and to superimpose the occupancy probabilities of all related hyperquad surfaces through hybrid weights to obtain the final occupancy probability of the target point in the 3D scene. The semantic probability weighted prediction module is used to perform a weighted average of the semantic probabilities of all relevant hyperquadratic surfaces based on the final occupancy probability of the target point, so as to obtain the semantic prediction result of the target point. The surface density optimization module is used to prune small-scale hyperquadratic surfaces and divide large-scale hyperquadratic surfaces according to the semantic prediction results, so as to optimize their distribution density in three-dimensional space.

[0015] This invention discloses a method and apparatus for representing 3D scenes based on hyperquadratic surfaces. Modeling is performed through probabilistic superposition, and a pruning and splitting module is added to the model to improve the utilization rate of modeling primitives. This invention proposes a 3D semantic occupancy prediction framework based on hyperquadratic surfaces, which accurately and efficiently models complex 3D scenes using the geometric properties of hyperquadratic surfaces. Furthermore, a probabilistic fusion method based on hyperquadratic surfaces is proposed, and its effectiveness has been verified in 3D semantic occupancy prediction tasks.

[0016] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0017] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a three-dimensional scene representation method based on hyperquadratic surfaces according to an embodiment of the present invention; Figure 2 This is an architecture diagram of a three-dimensional scene representation method based on hyperquadratic surfaces according to an embodiment of the present invention; Figure 3 This is a visual comparison of the method according to an embodiment of the present invention with other methods; Figure 4 This is a schematic diagram comparing the method according to an embodiment of the present invention with other methods; Figure 5 This is a structural diagram of a three-dimensional scene representation device based on a hyperquadratic surface according to an embodiment of the present invention. Detailed Implementation

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0020] The following description, with reference to the accompanying drawings, describes a method and apparatus for representing a three-dimensional scene based on a hyperquadratic surface, according to an embodiment of the present invention.

[0021] Example 1 Figure 1 This is a flowchart of a three-dimensional scene representation method based on hyperquadratic surfaces according to an embodiment of the present invention, as shown below. Figure 1 As shown, it includes: S1. Obtain multi-view image input and extract multi-scale image features using an image backbone network; based on the multi-scale image features, initialize a set of hyperquadratic surfaces and iteratively update the attribute parameters of the hyperquadratic surfaces through coding blocks.

[0022] Specifically, this step first involves acquiring image sequences from different perspectives using a multi-camera system, typically including front-view, rear-view, left-view, right-view, and top-view perspectives, to cover the multidimensional information of the 3D scene. The image input resolution is usually set to 1600×900 or higher to ensure sufficient spatial detail. At the same time, a unified image preprocessing workflow is adopted, including color space normalization (such as RGB→sRGB), geometric correction (such as distortion correction for fisheye lenses), and time synchronization mechanisms to ensure the consistency of multi-view images in the spatiotemporal dimensions.

[0023] In terms of image feature extraction, this invention employs an Image Backbone Network to encode multi-scale features of the input image. The backbone network can optionally be based on ResNet-50, ResNet-101, or a more advanced Transformer architecture (such as the Swin Transformer) to achieve efficient extraction of semantic and geometric information from the image. Multi-scale features typically include feature maps from shallow to deep layers, such as features at 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image resolution, corresponding to different levels of semantic abstraction and spatial awareness. The feature dimension output by the backbone network is typically 256 or 512 channels, and cross-scale fusion is performed through an FPN (Feature Pyramid Network) structure, thereby enhancing the model's robustness to perceiving objects at different scales.

[0024] Furthermore, the core of this step lies in using multi-scale image features as input to initialize a set of superquadric surfaces with geometric and semantic attributes, and iteratively optimizing their attribute parameters through coding blocks, thereby gradually constructing a sparse yet accurate representation of the 3D scene.

[0025] In some implementations, this step first extracts multi-scale features from the input image using an image backbone network (such as ResNet, Swin Transformer, etc.), covering multiple levels from low-level edges and textures to high-level semantic information. Based on these features, the model randomly initializes several hyperquadratic surfaces in three-dimensional space. Each surface is defined by six key attribute parameters: position (x, 3D coordinates, in meters), scale (s, scaling coefficients of the three dimensions, in meters), rotation angle (r, represented by Euler angles or quaternions, in radians), opacity (a, ranging from [0,1]), semantic label (c, representing the object category, such as vehicle, pedestrian, road, etc.), and shape index (…). Two parameters control the convexity and angularity of the surface, respectively, with values ​​typically ranging from [0.1, 2.0]. These parameters collectively determine the geometric shape of the hyperquadratic surface in three-dimensional space and its semantic contribution to the scene.

[0026] Specifically, the number of initial hyperquadratic surfaces is typically set to N = 1000~5000, depending on the resolution of the input image and the complexity of the scene. The initial range of the position parameters usually covers a ±50-meter cube space centered on the camera to accommodate the perception requirements of autonomous driving scenarios. The initial value of the scale parameter is set to [0.5, 1.0] meters to avoid excessively large or small initial surfaces interfering with the training process. The initial value of the shape index is set to [1.0, 1.0], corresponding to a standard ellipsoidal shape, which facilitates gradient stabilization during subsequent optimization.

[0027] Further, S1 includes: S11, the geometric properties of the current hyperquadratic surface are converted into feature vectors through quadratic surface coding blocks. The feature vectors include embedded representations of position, scale, rotation angle and shape index.

[0028] In some implementations, the hyperquad surface encoding block employs a multilayer perceptron (MLP) structure. The input consists of geometric attribute parameters of the hyperquad surface, specifically: position vector, scale vector, rotation angle (usually represented as Euler angles or quaternions), and shape index. These parameters collectively define the geometry of the hyperquad surface, with the shape index controlling the surface's convexity and concavity, giving it a richer geometric expressiveness than a traditional Gaussian surface. The encoding block maps these parameters into a high-dimensional feature vector to meet the input requirements of subsequent semantic and probabilistic prediction modules.

[0029] Furthermore, the encoding process can incorporate normalization techniques, such as logarithmic transformation of the scale parameter to enhance numerical stability; and sine-cosine encoding of the rotation angle to avoid periodic ambiguity. In addition, to improve the model's adaptability to different shapes, nonlinear activation functions with shape indices (such as ReLU or GELU) can be optionally embedded in the encoding blocks to enhance its ability to model complex geometric structures.

[0030] S12, local feature encoding of the feature vector is performed using three-dimensional sparse convolution, and image features and hyperquadratic surface features are fused through deformable attention mechanism to enhance the interactive expression of geometry and semantics.

[0031] In some implementations, 3D sparse convolution is computationally efficient based on sparse tensor structures, performing convolution operations only on non-empty voxel regions, significantly reducing computational complexity. Specifically, the input hyperquadratic surface features are organized into a sparse point cloud, where each point corresponds to a hyperquadratic surface element carrying attributes such as scale, rotation, shape exponent, opacity, and semantic probability. Sparse convolution aggregates the features of these points within their local neighborhoods, generating local feature encodings with spatial contextual information. This encoding process typically employs multiple layers of stacked sparse convolutions, each containing a kernel size of [missing information]. The sparse convolution operation is performed, and nonlinear enhancement and normalization are carried out through ReLU activation function and Batch Normalization.

[0032] Furthermore, to achieve the fusion of image features and hyperquadrant features, this invention introduces a deformable attention mechanism. This mechanism learns the spatial alignment relationship between image features and hyperquadrant features, dynamically adjusting the attention sampling position to achieve efficient cross-modal feature interaction. Specifically, image features are obtained through a multi-view feature extraction network (such as ResNet-50 or Swin Transformer) and projected into three-dimensional space to align with the hyperquadrant features. The deformable attention module generates sampling offsets through an offset prediction network; the offset is typically [value missing]. or The vector, with a sampling kernel size of or The sampling step size is And feature fusion is performed using bilinear interpolation.

[0033] S2. Construct a probabilistic hyperquadratic surface hybrid model, modeling each hyperquadratic surface as a local region occupancy probability distribution. Based on the conditional independence assumption, the occupancy probabilities of all relevant hyperquadratic surfaces are superimposed through hybrid weights to obtain the final occupancy probability of the target point.

[0034] Specifically, each hyperquadratic surface Q is defined by parameters such as its position m, rotation r, scale s, and shape exponents α and β. Its geometry is highly adjustable, capable of fitting various complex structures ranging from convex to concave, and from prisms to cylinders. When calculating the occupancy probability of point x, its coordinates are first transformed to the local coordinate system of Q, i.e., x is mapped to the center-aligned coordinate system of Q through rigid body transformation (translation + rotation). Then, according to the definition function of the hyperquadratic surface, the occupancy probability of this point within Q is calculated. Typically, the Sigmoid function is used to perform a nonlinear mapping on the surface interpolation results, obtaining a probability value between 0 and 1.

[0035] The blending weights are typically determined by the semantic probability *c* and the opacity parameter, where *c* represents the semantic influence of the hyperquadratic surface on the target point *x*, and the opacity reflects the confidence level of its geometric coverage. In practical modeling, the blending weights can be calculated by weighting the semantic probabilities of all relevant surfaces using a normalized exponential function (Softmax) to ensure the rationality and interpretability of the probability distribution. Furthermore, the shape indices α and β are usually set to a range of [0.1, 2.0] to guarantee the diversity and stability of surface shapes.

[0036] Further, S2 includes: S21, the target point is transformed to the local coordinate system of each hyperquadratic surface, the transformation being a rigid transformation based on the position and rotation of the hyperquadratic surface.

[0037] Specifically, this rigid transformation includes two operations: translation and rotation. First, the target point... Translate to the hyperquadratic surface from the global coordinate system. central position That is, execution Subsequently, using a rotation matrix Rotate the translated point to align it with the principal axes of the hyperquadratic surface. Rotation matrix. Typically composed of quaternions It may be derived from Euler angles. In specific implementation, the rotation parameters can be analyzed using the Rodrigues formula or the exponential mapping on the SO(3) group.

[0038] Specifically, position represents the center coordinates of the hyperquadratic surface in three-dimensional space, and rotation typically represents the rotation angle (in radians) about the three coordinate axes, with a range generally ranging from... Rotation matrix The construction must satisfy orthogonality constraints to ensure the rigidity of the transformation. Furthermore, this transformation process can be efficiently parallelized on GPUs, making it particularly suitable for processing large-scale point clouds or voxel spaces.

[0039] S22, calculate the occupancy probability of the target point in the local coordinate system of the hyperquadratic surface. The probability is determined by the scale, shape index and opacity of the hyperquadratic surface, and is superimposed by normalized mixed weights.

[0040] Specifically, hyperquadratic surfaces The local coordinate system is determined by its position and rotation angle Definition. First, define the point. Points mapped to this local coordinate system through rigid body transformations (translation and rotation), i.e. ,in To be determined by rotation angle The rotation matrix is ​​constructed. Then, the occupancy probability of this point is calculated using the parametric expression of the hyperquadratic surface. The equiprobability surface of the hyperquadratic surface is determined by both the scale parameter and the shape exponent, and its probability density function can be expressed as: .

[0041] Furthermore, to achieve joint probabilistic modeling of multiple surfaces, this invention assumes that each hyperquadratic surface has points... The possession events are probabilistically superimposed under the premise of conditional independence. Specifically, the point... The final occupancy probability is for all relevant surfaces The normalized mixed weighted sum of the occupancy probabilities: .

[0042] S3. Based on the occupancy probability of the target point, perform a weighted average of the semantic probabilities of all relevant hyperquadratic surfaces to obtain the semantic prediction result of the target point.

[0043] Specifically, each hyperquadratic surface not only defines its geometric distribution in three-dimensional space (through parameters such as position, scale, rotation, and shape exponent), but also carries a semantic probability distribution associated with that region. ,in This represents the total number of semantic categories. For any target point x, it is first mapped to each hyperquadratic surface through geometric transformation. In the local coordinate system, calculate the point being The probability of occupancy. Semantic prediction is obtained by weighted averaging of the semantic probabilities of all relevant hyperquadratic surfaces: .

[0044] S4 applies the pruning-partitioning module to prune small-scale hyperquadratic surfaces and partition large-scale hyperquadratic surfaces to optimize their distribution density in three-dimensional space, thereby improving modeling efficiency and prediction accuracy.

[0045] In some implementations, pruning operations identify and remove hyperquadratic surfaces that are too small and contribute little to scene modeling by setting a scale threshold. The scale parameter is typically composed of the principal axis lengths of the hyperquadratic surface, representing the volume it covers in 3D space. If the scale of a hyperquadratic surface is smaller than the scale threshold, it is considered to be mainly located in empty regions or has weak expressive power for scene structure, and therefore it is removed from the current representation set. Pruning operations can effectively reduce the number of redundant primitives, reduce computational load, and improve the model's focus on key regions.

[0046] Furthermore, the splitting operation is applied to hyperquadratic surfaces with larger scales. During training, if the scale of a hyperquadratic surface exceeds a set splitting threshold, it indicates that it may cover multiple semantic regions or geometric structures. In this case, it is split into two or more sub-surfaces to achieve more refined local modeling. The splitting strategy is usually based on gradient information or the local rate of change of the occupancy probability distribution, using a splitting point selection algorithm (such as K-means clustering or gradient descent direction) to determine the initial position and parameters of the sub-surfaces. The new surface after splitting inherits some attributes of the original surface (such as semantic labels, opacity, etc.) and is independently optimized in subsequent steps.

[0047] Optionally, pruning and splitting operations can be performed after the initial training phase to ensure that the model has a preliminary ability to represent scenes. In each iteration, the module keeps the total number of hyperquadratic surfaces constant and dynamically adjusts their distribution to make the model more focused on the actually occupied regions. The introduction of this module significantly improves the utilization rate of hyperquadratic surfaces and reduces the interference of invalid primitives on the prediction results, thereby achieving higher prediction accuracy and lower computational resource consumption.

[0048] Further, S4 includes: S41, according to a preset pruning threshold, remove hyperquadratic surfaces in three-dimensional space with a scale smaller than the threshold, the threshold being dynamically calculated based on the primitive distribution density in the early stage of training.

[0049] Specifically, this step begins with a density analysis of the initialized hyperquadratic surface set during the initial training phase. Specifically, by statistically analyzing the number and distribution density of primitives in each region of 3D space, and combining this with the actual proportion of voxels occupied in the scene, a pruning threshold is dynamically calculated. This threshold is typically defined as the lower limit of the hyperquadratic surface scale parameters (such as principal axis length).

[0050] The main parameters involved in this step include the scale, position, and rotation parameters of the hyperquadratic surface, as well as a dynamically calculated pruning threshold. The scale parameter is typically represented as a three-dimensional vector, corresponding to the extent of the surface's expansion along its three principal axes. The pruning threshold is a scalar, with units consistent with the scale parameter, usually standardized to meters. In the early stages of training, due to the lack of structural priors, the hyperquadratic surface may be initialized as small-scale primitives covering empty regions. These primitives contribute very little to the scene in subsequent optimizations and therefore need to be removed through a pruning mechanism.

[0051] S42, perform a splitting operation on the hyperquadratic surface with a scale greater than the threshold. The splitting operation generates two sub-surfaces based on the semantic uncertainty of the covered area and retains the residual update of the original semantic and geometric properties.

[0052] The splitting operation is implemented through the following steps: First, the geometric parameters (such as position, rotation, scale, and shape index) of the original hyperquadratic surface are decomposed, and two sub-regions with significant semantic differences are selected as splitting targets within its coverage area. Then, based on the geometric centers of these two sub-regions, two new hyperquadratic surfaces are initialized, with their initial positions corresponding to the centroids of the sub-regions. The scale parameters are scaled down proportionally to the original surface scale (e.g., by a factor of 0.6 to 0.8), while the shape index inherits the parameters of the original surface to maintain geometric continuity. Regarding semantic attributes, the semantic probability distributions of the two sub-surfaces are updated through local semantic features to ensure that their semantic labels better match the actual scene.

[0053] Furthermore, the splitting operation preserves the residual updates of the semantic and geometric properties of the original surface. That is, the newly generated sub-surface still shares some parameters with the original surface during training, and property optimization is performed through a residual connection mechanism. This mechanism helps maintain model stability and avoids the parameter explosion problem caused by splitting.

[0054] Furthermore, the method also includes: S5, the attribute parameters of the hyperquadratic surface are jointly optimized by the cross-entropy loss and lovasz softmax loss of the 3D semantic occupancy prediction task, where the cross-entropy loss is used to measure the accuracy of semantic prediction, and the lovasz softmax loss is used to alleviate the class imbalance problem and improve the overall prediction performance.

[0055] An embodiment of the present invention provides a three-dimensional scene representation method based on hyperquadratic surfaces. By introducing hyperquadratic surfaces for modeling, the number of modeling primitives is reduced while the richness of modeling shapes is enhanced, thereby improving the efficiency and effectiveness of three-dimensional occupancy prediction of the model.

[0056] Example 2 The following describes in detail, with reference to the accompanying drawings, a three-dimensional scene representation method based on hyperquadratic surfaces according to an embodiment of the present invention.

[0057] This invention proposes a three-dimensional scene representation framework based on hyperquadratic surfaces. Figure 2 and Figure 3 The figures provided are the overall flowchart of this invention and a visual comparison with other works. This invention proposes an efficient and highly expressive object-oriented 3D representation method that uses hyperquadratic surfaces as the basic unit of the scene to achieve sparse modeling of complex geometric structures. This invention also proposes a time-series-aware framework based on the Gaussian world model. Figure 2 and Figure 3The diagrams show the overall flowchart and the unified evolution layer module of this invention. This invention proposes explicitly modeling the evolution of a scene in a 3D Gaussian space, learning the inherent dynamic logic and temporal consistency of scene evolution, thereby promoting temporal awareness tasks. This invention utilizes a set of learnable hyperquadratic surfaces to model the scene, each hyperquadratic surface being defined by attributes such as position, scale, rotation, transparency, semantic label, and shape index. To achieve 3D semantic occupancy prediction, this invention further constructs a probabilistic hyperquadratic surface hybrid model, interpreting each hyperquadratic surface as a local occupancy probability distribution with geometric priors, and completing semantic inference through a probabilistic hybrid mechanism. This invention designs pruning and splitting modules to concentrate the hyperquadratic surfaces in the actual occupancy area, further improving modeling efficiency and inference accuracy. Finally, this invention uses all optimized hyperquadratic surfaces to complete 3D semantic occupancy prediction, and constrains the model using cross-entropy loss and Lovasz loss for the 3D semantic occupancy prediction task. Figure 4 This diagram illustrates a comparison between the present invention and other methods. Previous methods typically consume significant amounts of video memory or employ a large number of modeling primitives. In contrast, the present invention improves the predictive performance and inference efficiency of the model by using hyperquadratic surfaces as a modeling primitive.

[0058] The specific steps are as follows: S10, Representation of hyperquadratic surfaces: Vision-centric 3D semantic occupancy prediction aims to estimate the occupancy state and semantic label of each voxel in 3D space based on visual input. Formally, given visual input... The model takes an input image from multiple viewpoints and aims to predict voxel-level semantic labels for the 3D scene. ,in Indicates semantic category, Representing the spatial shape of occupancy. To achieve this goal, this invention introduces an object-oriented scene representation method based on hyperquadratic surface elements, leveraging its efficiency and expressive power to realize 3D semantic occupancy prediction. Hyperquadratic surfaces are a family of parametric shapes with strong geometric expressiveness, defined as follows:

[0059] in Indicates the position of the point. This represents the hyperparameters of the surface family. Compared to the ellipsoid in Gaussian geometry, the hyperquadratic surface introduces only two additional shape indices. However, it can represent a wider variety of shapes. This inherent parametric efficiency and geometric expressiveness allows hyperquadratic surfaces to model diverse shapes without dense stacking. Therefore, complex scene structures can be represented with only a small number of hyperquadratic surfaces, thus achieving efficient and powerful scene representation. In this invention, a set of... A parameterized hyperquadratic surface is used to represent the 3D scene. Each hyperquadratic surface is defined by its scale. and shape index Define its geometric properties. To extend to the global coordinate system, each element is also assigned a position. and rotation In addition to geometric properties, each hyperquadratic surface also includes opacity. The semantic probability c is incorporated to integrate semantic information. In summary, the hyperquadratic surface representation of this invention can be formalized as:

[0060] Next, we will discuss how to obtain 3D occupancy prediction from hyperquadular surface representation. In order to achieve robust 3D semantic occupancy prediction, this invention designs a probabilistic modeling mechanism that converts hyperquadular surfaces into occupancy probabilities. Inspired by GaussianFormer-2[7], this invention adopts a probabilistic hyperquadular surface hybrid model, in which each hyperquadular surface defines the occupancy probability distribution in its local region. To calculate the probability that a point x is occupied by a hyperquadular surface Q, this invention first transforms x to the local coordinate system of Q, which is defined by its position m and rotation r:

[0061] The probability that point is occupied by Q is:

[0062] Assuming the occupancy events of different hyperquadratic surfaces are conditionally independent, the final occupancy probability of point x is:

[0063] Semantic prediction is obtained by weighted averaging of the semantic probabilities of all relevant hyperquadratic surfaces, where the weights are determined by their occupancy at point x:

[0064] The core of this probabilistic modeling mechanism lies in incorporating hyperquadratic surface geometry as a shape prior into the probability distribution. Its equiprobable surfaces precisely conform to the geometric shapes defined in hyperquadratic surfaces. Leveraging the geometric expressive power of hyperquadratic surfaces, the model can efficiently model complex 3D structures with a sparse set of elements, avoiding dense stacking. Furthermore, this probabilistic framework effectively models structural uncertainties arising from visual uncertainty, significantly improving the model's robustness and generalization ability.

[0065] S20. Three-dimensional semantic occupancy prediction model based on hyperquadratic surface: For image input from multiple perspectives First, an image backbone network is used. Extracting multi-scale image features :

[0066] Since the initial scene lacks structural priors, this invention randomly initializes several hyperquadratic surfaces in three-dimensional space. The final hyperquadratic surface is predicted from the image using B coding blocks. Within each block, the invention first encodes the current hyperquadratic surface as features using a quadratic surface coding block:

[0067] Subsequently, this invention utilizes 3D sparse convolution to autoencode hyperquadratic surface features and employs deformable attention to achieve interaction between the image and the hyperquadratic surface features:

[0068] Finally, the hyperquadratic surface decoding module is used to predict the property updates of the hyperquadratic surface, and the residuals are combined with the original properties through residual addition:

[0069] After updating B coding blocks, this invention obtains the final hyperquadratic surface prediction and obtains the three-dimensional semantic occupancy prediction through a probabilistic modeling mechanism:

[0070] During training, this invention employs cross-entropy loss and Lovasz softmax loss for optimization. Due to the lack of structural priors, hyperquadrants are uniformly initialized in 3D space. Therefore, some hyperquadrants located in empty regions are optimized to a very small scale during training, contributing little to scene modeling and resulting in inefficiency. To address this, this invention introduces a pruning-partitioning module after initial training. This module prunes small-scale hyperquadrants (potentially in empty regions) and partitions large-scale hyperquadrants (potentially in occupied regions) to achieve more refined modeling. This invention maintains the total number of hyperquadrants unchanged and uses two additional encoding blocks to further optimize their properties. Notably, this lightweight module improves the utilization of hyperquadrants, achieving more efficient scene representation without significantly increasing computational overhead.

[0071] Example 3 To achieve the above embodiments, such as Figure 5 As shown, this embodiment also provides a three-dimensional scene representation device 10 based on super quadric surfaces. The device 10 includes an initialization and parameter update module 100, a probabilistic hybrid modeling module 200, a semantic probability weighted prediction module 300, and a surface density optimization module 400.

[0072] The initialization and parameter update module 100 is used to acquire multi-view image input and extract multi-scale image features using an image backbone network; based on the multi-scale image features, a set of hyperquadratic surfaces is initialized, and the attribute parameters of the hyperquadratic surfaces are iteratively updated through coding blocks; The probabilistic hybrid modeling module 200 is used to construct a probabilistic hyperquadratic surface hybrid model. Each hyperquadratic surface is modeled as the occupancy probability distribution of a local region. Based on the assumption of conditional independence, the occupancy probabilities of all relevant hyperquadratic surfaces are superimposed through hybrid weights to obtain the final occupancy probability of the target point. The semantic probability weighted prediction module 300 is used to perform a weighted average of the semantic probabilities of all relevant hyperquadratic surfaces based on the occupancy probability of the target point, so as to obtain the semantic prediction result of the target point. The surface density optimization module 400 is used to apply the pruning-partitioning module to prune small-scale hyperquadratic surfaces and partition large-scale hyperquadratic surfaces in order to optimize their distribution density in three-dimensional space, thereby improving modeling efficiency and prediction accuracy.

[0073] Furthermore, the initialization and parameter update module 100 described above is also used for: The geometric properties of the current hyperquadratic surface are converted into feature vectors by quadratic surface coding blocks. The feature vectors include embedded representations of position, scale, rotation angle and shape index. The feature vector is locally feature-encoded using 3D sparse convolution, and image features and hyperquadratic surface features are fused through a deformable attention mechanism to enhance the interactive expression of geometry and semantics.

[0074] Furthermore, the aforementioned probabilistic hybrid modeling module 200 is also used for: The target point is transformed to the local coordinate system of each hyperquadratic surface, the transformation being a rigid transformation based on the position and rotation of the hyperquadratic surface; The occupancy probability of the target point in the local coordinate system of the hyperquadratic surface is calculated. The probability is determined by the scale, shape index and opacity of the hyperquadratic surface, and is superimposed by normalized mixed weights.

[0075] Furthermore, the aforementioned surface density optimization module 400 is also used for: According to a preset pruning threshold, hyperquadratic surfaces with a scale smaller than the threshold are removed in three-dimensional space. The threshold is dynamically calculated based on the primitive distribution density in the early stage of training. A splitting operation is performed on hyperquadratic surfaces with a scale greater than the threshold. The splitting operation generates two sub-surfaces based on the semantic uncertainty of the covered region, and retains the residual update of the original semantic and geometric properties.

[0076] Furthermore, the device 10 also includes: The loss calculation module is used to jointly optimize the attribute parameters of the hyperquadratic surface by using the cross-entropy loss and lovasz softmax loss of the 3D semantic occupancy prediction task. The cross-entropy loss is used to measure the accuracy of semantic prediction, and the lovasz softmax loss is used to alleviate the class imbalance problem and improve the overall prediction performance.

[0077] An embodiment of the present invention provides a three-dimensional scene representation device based on hyperquadratic surfaces. By introducing hyperquadratic surfaces for modeling, the number of modeling primitives is reduced while the richness of modeling shapes is enhanced, thereby improving the efficiency and effectiveness of three-dimensional occupancy prediction of the model.

[0078] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0079] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for representing a 3D scene based on a hyperquadratic surface, characterized in that, include: S1, using an image backbone network to extract multi-scale image features of multi-view images of a 3D scene, and based on the multi-scale image features, initialize a set of hyperquadratic surfaces, and iteratively update the attribute parameters of the hyperquadratic surfaces through coding blocks; S2, model each hyperquadric surface of the probabilistic hyperquadric surface hybrid model constructed based on attribute parameters as the occupancy probability distribution of a local region, and superimpose the occupancy probabilities of all related hyperquadric surfaces through hybrid weights to obtain the final occupancy probability of the target point in the 3D scene; S3, based on the final occupancy probability of the target point, perform a weighted average of the semantic probabilities of all relevant hyperquadratic surfaces to obtain the semantic prediction result of the target point; S4. Based on the semantic prediction results, small-scale hyperquadratic surfaces are pruned and large-scale hyperquadratic surfaces are divided to optimize their distribution density in three-dimensional space.

2. The method as described in claim 1, characterized in that, S1 includes: The geometric properties of the current hyperquadratic surface are converted into feature vectors by quadratic surface coding blocks. The feature vectors include embedded representations of position, scale, rotation angle and shape index. The feature vector is locally feature-encoded using 3D sparse convolution, and multi-scale image features and hyperquadratic surface features are fused through a deformable attention mechanism to update the attribute parameters of the hyperquadratic surface.

3. The method as described in claim 1, characterized in that, S2 includes: The target points within the 3D scene are transformed to the local coordinate system of each hyperquadratic surface, wherein the transformation is based on a rigid transformation of the position and rotation of the hyperquadratic surface; The occupancy probability of a target point in a 3D scene in the local coordinate system of a hyperquadratic surface is calculated. The probability is determined by the scale, shape index, and opacity of the hyperquadratic surface. The final occupancy probability is obtained by superimposing the normalized mixed weights, where the mixed weights are the occupancy probabilities of each hyperquadratic surface at the target point.

4. The method as described in claim 1, characterized in that, The S4 further includes: Based on the semantic prediction results, hyperquadratic surfaces with a scale smaller than the preset pruning threshold are removed from the three-dimensional space. The pruning threshold is dynamically calculated based on the primitive distribution density in the early stage of training. A splitting operation is performed on the hyperquadratic surface with a scale greater than the pruning threshold to obtain multiple sub-surfaces. The splitting operation generates multiple sub-surfaces based on the semantic uncertainty of the covered region and retains the residual update of the original semantic and geometric properties.

5. The method as described in claim 1, characterized in that, The method further includes: The attribute parameters of the hyperquadratic surface are jointly optimized by cross-entropy loss and lovasz softmax loss in the 3D semantic occupancy prediction task. The cross-entropy loss is used to measure the accuracy of semantic prediction, and the lovasz softmax loss is used to alleviate the class imbalance problem and improve the overall prediction performance.

6. A three-dimensional scene representation device based on hyperquadratic surfaces, characterized in that, include: The initialization and parameter update module is used to extract multi-scale image features of multi-view images of a 3D scene using an image backbone network, initialize a set of hyperquadratic surfaces based on the multi-scale image features, and iteratively update the attribute parameters of the hyperquadratic surfaces through coding blocks. The probabilistic hybrid modeling module is used to model each hyperquad surface of the probabilistic hyperquad surface hybrid model constructed based on attribute parameters as the occupancy probability distribution of a local region, and to superimpose the occupancy probabilities of all related hyperquad surfaces through hybrid weights to obtain the final occupancy probability of the target point in the 3D scene. The semantic probability weighted prediction module is used to perform a weighted average of the semantic probabilities of all relevant hyperquadratic surfaces based on the final occupancy probability of the target point, so as to obtain the semantic prediction result of the target point. The surface density optimization module is used to prune small-scale hyperquadratic surfaces and divide large-scale hyperquadratic surfaces according to the semantic prediction results, so as to optimize their distribution density in three-dimensional space.

7. The apparatus as claimed in claim 6, characterized in that, The initialization and parameter update module is also used for: The geometric properties of the current hyperquadratic surface are converted into feature vectors by quadratic surface coding blocks. The feature vectors include embedded representations of position, scale, rotation angle and shape index. The feature vector is locally feature-encoded using 3D sparse convolution, and multi-scale image features and hyperquadratic surface features are fused through a deformable attention mechanism to update the attribute parameters of the hyperquadratic surface.

8. The apparatus as claimed in claim 6, characterized in that, The probabilistic hybrid modeling module is also used for: The target points within the 3D scene are transformed to the local coordinate system of each hyperquadratic surface, wherein the transformation is based on a rigid transformation of the position and rotation of the hyperquadratic surface; The occupancy probability of a target point in a 3D scene in the local coordinate system of a hyperquadratic surface is calculated. The probability is determined by the scale, shape index, and opacity of the hyperquadratic surface. The final occupancy probability is obtained by superimposing the normalized mixed weights, where the mixed weights are the occupancy probabilities of each hyperquadratic surface at the target point.

9. The apparatus as claimed in claim 6, characterized in that, The surface density optimization module is also used for: Based on the semantic prediction results, hyperquadratic surfaces with a scale smaller than the preset pruning threshold are removed from the three-dimensional space. The pruning threshold is dynamically calculated based on the primitive distribution density in the early stage of training. A splitting operation is performed on the hyperquadratic surface with a scale greater than the pruning threshold to obtain multiple sub-surfaces. The splitting operation generates multiple sub-surfaces based on the semantic uncertainty of the covered region and retains the residual update of the original semantic and geometric properties.

10. The apparatus as claimed in claim 6, characterized in that, Also includes: The loss calculation module is used to jointly optimize the attribute parameters of the hyperquadratic surface by using the cross-entropy loss and lovasz softmax loss of the 3D semantic occupancy prediction task. The cross-entropy loss is used to measure the accuracy of semantic prediction, and the lovasz softmax loss is used to alleviate the class imbalance problem and improve the overall prediction performance.

Images