Four-dimensional dynamic scene representation and reconstruction method, system and device based on structural equivalence prior

Abstract

The present application belongs to the field of computer vision and artificial intelligence, and particularly relates to a four-dimensional dynamic scene representation and reconstruction method, system and device based on structural equivalence prior, aiming at solving the problems of poor spatio-temporal consistency, large calculation cost and low long sequence reconstruction accuracy in dynamic scene reconstruction. The present application comprises: decomposing the input four-dimensional voxel sequence into multiple two-dimensional feature planes containing spatial projection and spatio-temporal coupling; introducing structural equivalence prior to fuse the plane features, generating unified and continuous spatio-temporal latent space representation by establishing the internal correlation between spatial geometric gradient and time evolution; finally, using a decoding module containing residual connection to decode the latent space representation, and recovering the high-fidelity four-dimensional dynamic scene layer by layer. The present application effectively improves the consistency and accuracy of dynamic scene reconstruction by establishing an explicit spatio-temporal coupling relationship, while taking into account the calculation efficiency, and has broad application prospects in the fields of autonomous driving, augmented reality, etc.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and artificial intelligence, and specifically relates to a method, system and device for four-dimensional dynamic scene representation and reconstruction based on structural equivalence prior. Background Technology

[0002] With the rapid development of applications such as autonomous driving, augmented reality (AR / VR), and robotics, the perception and modeling of dynamic 3D scenes (3D+T, i.e., four-dimensional scenes) has gradually become an important research direction in the field of computer vision. Compared with traditional static 3D modeling, four-dimensional dynamic scenes need to simultaneously express spatial structure and temporal evolution characteristics, which places higher demands on both computation and storage.

[0003] In existing technologies, Neural Radiation Fields (NeRF) and its variants have achieved significant results in static 3D reconstruction, but directly extending them to dynamic scenes leads to high computational overhead, low training efficiency, and difficulty in guaranteeing spatiotemporal consistency. To alleviate these problems, researchers have proposed decomposition-based representation methods, such as K-Plane and HexPlane, which compress the representation dimension by decomposing high-dimensional four-dimensional data into several two-dimensional planes, thereby improving efficiency to some extent.

[0004] However, these methods typically rely solely on simple planar stitching or frame-by-frame modeling, neglecting the intrinsic connection between spatial structure and temporal evolution, making it difficult to maintain spatiotemporal consistency in long sequences or complex scenes. Meanwhile, while discretization-based autoregressive methods can utilize temporal recursion, they are prone to error accumulation and inference delays, limiting their practicality.

[0005] Therefore, how to achieve high-precision modeling of dynamic 3D scenes with limited computing resources, while ensuring the physical consistency of spatial structure and temporal evolution, has become a key problem that urgently needs to be solved in the field of 4D dynamic scene reconstruction. Summary of the Invention

[0006] To address the aforementioned problems in existing technologies, namely poor spatiotemporal consistency, high computational cost, and low accuracy in long sequence reconstruction of dynamic scenes, a four-dimensional dynamic scene representation and reconstruction method based on structural equivalence priors is proposed. This method includes:

[0007] Acquire multi-frame 3D voxel sequence data and corresponding semantic labels as raw dynamic scene data, and standardize them into a four-dimensional input tensor; The standardized four-dimensional input tensor is decomposed into orthogonal two-dimensional feature planes, which include a spatial projection plane and a spatiotemporal coupling plane. Establish a corresponding latent probability distribution for each two-dimensional feature plane, and sample from the established latent probability distribution to obtain a differentiable latent plane representation; Based on structural equivalence priors, all differentiable latent plane representations are fused to generate a unified spatiotemporal latent space representation. The structural equivalence priors establish a mapping relationship where the rate of change of time is driven by the spatial gradient. A unified spatiotemporal latent space representation is decoded in multiple layers to generate an upsampled feature map. The decoding process utilizes a residual connection structure, which adds the input features of the decoding layer to the output features of the decoding layer. Scene reconstruction is performed based on upsampled feature maps to obtain a reconstructed four-dimensional dynamic scene, and semantic category predictions are generated for each voxel in the voxel sequence.

[0008] Further steps in preprocessing the raw dynamic scene data include: Unify spatial resolution, align temporal frame rates, normalize, and filter noise; It also includes spatiotemporal consistency enhancement processing of voxel sequence data, which is applied to the geometric structure and temporal evolution characteristics of dynamic scenes.

[0009] Furthermore, the steps for decomposing the normalized four-dimensional input tensor include: The standardized four-dimensional input tensor is projected onto three orthogonal spatial dimension pairs to generate a spatial projection plane. The standardized four-dimensional input tensor is projected onto three spatiotemporal pairs consisting of the time dimension and each spatial dimension to generate a spatiotemporal coupling plane.

[0010] Further steps in establishing the potential probability distribution and sampling include: An encoding process is applied to each two-dimensional feature plane to map the two-dimensional feature plane to the mean and variance parameters of the latent probability distribution; By employing a reparameterization method, combining the obtained mean and variance parameters with random sampled values ​​obtained from the standard normal distribution, a differentiable latent plane representation is calculated.

[0011] Furthermore, the steps for fusing differentiable latent plane representations include: The application incorporates a fusion function that combines geometric perception weights and attention mechanisms to process a set of differentiable latent plane representations; Under the constraint of structural equivalence priors, the fusion function aggregates features from different spatial projection planes and spatiotemporal coupling planes to generate a unified spatiotemporal latent space representation.

[0012] Furthermore, the mapping relationship established by structural equivalence priors is defined as follows: The temporal rate of change of the dynamic scene representation is a function of the spatial gradient of the dynamic scene representation; the prior correlates temporal evolution with spatial geometric deformation in the latent space.

[0013] Furthermore, the steps for multi-layer decoding of the unified spatiotemporal latent space representation further include: Reorganize and unify the spatiotemporal latent space representation; The reconstructed representation is then processed sequentially using local contextual processing and channel interaction processing. The processed features are progressively upsampled by applying multi-level deconvolution operations to generate upsampled feature maps.

[0014] Furthermore, the steps for scene reconstruction based on upsampled feature maps include: Perform a convolution operation on the upsampled feature map to map the high-dimensional features of the upsampled feature map to the probability of each voxel on a preset semantic category set; The obtained probabilities are processed using a normalization function, and the final semantic category prediction result is output for each voxel.

[0015] In a second aspect, the present invention proposes a four-dimensional dynamic scene representation and reconstruction system based on structural equivalence priors, for executing a four-dimensional dynamic scene representation and reconstruction method based on structural equivalence priors. The system includes: The data processing module is configured to acquire multi-frame three-dimensional voxel sequence data and corresponding semantic labels as raw dynamic scene data, and standardize them into four-dimensional input tensors. The decomposition module is configured to decompose a standardized four-dimensional input tensor into orthogonal two-dimensional feature planes, wherein the two-dimensional feature planes include a spatial projection plane and a spatiotemporal coupling plane. The characterization generation module is configured to establish a corresponding latent probability distribution for each two-dimensional feature plane and to sample from the established latent probability distribution to obtain a differentiable latent plane characterization. The fusion module is configured to fuse all differentiable latent plane representations based on structural equivalence priors to generate a unified spatiotemporal latent space representation. The structural equivalence priors establish a mapping relationship in which the rate of change of time is driven by the spatial gradient. The decoding module is configured to perform multi-layer decoding on a unified spatiotemporal latent space representation to generate an upsampled feature map. The decoding process utilizes a residual connection structure, which adds the input features of the decoding layer to the output features of the decoding layer. The reconstruction module is configured to perform scene reconstruction based on upsampled feature maps to obtain a reconstructed four-dimensional dynamic scene and generate semantic category predictions for each voxel in the voxel sequence.

[0016] In a third aspect, the present invention provides an apparatus comprising: At least one processor; and a memory communicatively connected to at least one of the processors; The memory stores instructions that can be executed by the processor to implement a four-dimensional dynamic scene representation and reconstruction method based on structural equivalence priors.

[0017] The beneficial effects of this invention are: This invention introduces structural equivalence priors, establishing explicit mathematical constraints on the temporal evolution and spatial geometric deformation of a scene within the latent space. This ensures that the model's learning process follows the physical laws of temporal change driven by spatial deformation, thereby effectively guaranteeing the continuity of the reconstructed dynamic scene in the temporal dimension and the rationality of its spatial structure.

[0018] This invention employs a planar decomposition and spatiotemporal fusion mechanism to decompose high-dimensional four-dimensional spatiotemporal data into multiple low-dimensional two-dimensional feature planes for processing, significantly reducing the computational burden and storage requirements of the model. Furthermore, its fusion mechanism is not a simple feature concatenation, but rather uses structural equivalence priors for constraint, effectively suppressing the error accumulation problem commonly encountered when processing long sequence data, thus improving computational efficiency while ensuring high reconstruction accuracy.

[0019] This invention incorporates a residual decoding module in the decoding stage. Through the residual connection structure, this module effectively prevents feature collapse or gradient vanishing problems that may occur in deep networks during the layer-by-layer upsampling process to restore scene resolution, thus preserving fine geometric and texture features. Therefore, this invention significantly improves the reconstruction accuracy of sparse, small-sized targets in a scene (such as distant pedestrians or vehicles). Attached Figure Description

[0020] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating a four-dimensional dynamic scene representation and reconstruction method based on structural equivalence priors according to the present invention. Figure 2 This is a schematic diagram of the encoder process structure of a four-dimensional dynamic scene representation and reconstruction method based on structural equivalence prior according to the present invention. Figure 3 This is a schematic diagram of the decoder process structure of a four-dimensional dynamic scene representation and reconstruction method based on structural equivalence prior according to the present invention. Figure 4This is a reconstruction effect diagram of a four-dimensional dynamic scene representation and reconstruction method based on structural equivalence prior according to the present invention; wherein (a) is the reconstruction effect diagram of frames 1, 8, and 16; and (b) is the reconstruction detail effect diagram. Figure 5 This is a structural diagram of a four-dimensional dynamic scene representation and reconstruction system based on structural equivalence priors according to the present invention. Figure 6 This is a schematic diagram of the structure of a computer system used to implement the methods, systems, and electronic devices of this application. Detailed Implementation

[0021] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0022] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] The first embodiment of the present invention provides a method for four-dimensional dynamic scene representation and reconstruction based on structural equivalence priors, the method comprising: Step S10: Obtain multi-frame 3D voxel sequence data and corresponding semantic labels as raw dynamic scene data, and standardize them into a four-dimensional input tensor. Step S20: Decompose the standardized four-dimensional input tensor into orthogonal two-dimensional feature planes, wherein the two-dimensional feature planes include a spatial projection plane and a spatiotemporal coupling plane; Step S30: Establish a corresponding latent probability distribution for each two-dimensional feature plane, and sample from the established latent probability distribution to obtain a differentiable latent plane representation; Step S40: Based on the structural equivalence prior, all differentiable latent plane representations are fused to generate a unified spatiotemporal latent space representation. The structural equivalence prior establishes a mapping relationship where the rate of change of time is driven by the spatial gradient. Step S50: Perform multi-layer decoding on the unified spatiotemporal latent space representation to generate an upsampled feature map. The decoding process utilizes a residual connection structure, which adds the input features of the decoding layer to the output features of the decoding layer. Step S60: Reconstruct the scene based on the upsampled feature map to obtain the reconstructed four-dimensional dynamic scene, and generate a semantic category prediction for each voxel in the voxel sequence.

[0024] To more clearly explain the four-dimensional dynamic scene representation and reconstruction method based on structural equivalence prior of this invention, the following will be combined with... Figure 1 The steps in the embodiments of the present invention are described in detail below, including steps S10-S60. Step S10: Obtain multi-frame 3D voxel sequence data and corresponding semantic labels as raw dynamic scene data, and standardize them into a four-dimensional input tensor. In this embodiment, the steps for preprocessing the original dynamic scene data include: Unify spatial resolution, align temporal frame rates, normalize, and filter noise; It also includes spatiotemporal consistency enhancement processing of voxel sequence data, which is applied to the geometric structure and temporal evolution characteristics of dynamic scenes.

[0025] The specific implementation of step S10 is as follows: First, acquire the original dynamic scene data. This data comes from an autonomous driving simulation platform, a robot perception system, or an augmented reality environment. Specifically, it can use publicly available or self-built datasets, such as CarlaSC and Occ3D-Waymo. The acquired data consists of a series of three-dimensional voxel sequences and semantic labels corresponding to each frame. The voxel sequences can express the spatiotemporal changes of the scene, and the semantic labels cover static and dynamic object categories such as roads, buildings, vehicles, and pedestrians. Each data sequence contains voxel labels for multiple time frames to ensure the spatiotemporal continuity of the scene. The acquired raw dataset is divided into training, testing, and validation sets in a 6:2:2 ratio to ensure the generalization ability of the subsequent model. The training set data undergoes standardization preprocessing, which includes four core operations: First, the spatial resolution is unified, that is, the spatial dimensions of all sequences are uniformly adjusted to the specified height, width and depth. In this embodiment, the height is preferably set to 128, the width is set to 128 and the depth is set to 8. Second, align the time frame rate. Since there may be frame rate differences between different data sources, this step aligns the time frame number of all sequences to a fixed 16 consecutive frames by frame interpolation and resampling. Third, voxel normalization is performed to normalize the values ​​in the voxel grid to a specific range in order to stabilize the training process. Fourth, noise filtering is performed to eliminate outliers or irrelevant information that may be introduced during the data acquisition process; In addition, the preprocessing includes a spatiotemporal consistency enhancement process. This process acts on the geometric structure and temporal evolution features of the dynamic scene, enhancing the contrast of the input data to preserve key features of the geometric structure and temporal evolution in the dynamic scene to the greatest extent, thereby improving the robustness of subsequent model learning. After the above preprocessing, the original dynamic scene data is transformed into a standardized four-dimensional input tensor, which is in the form of... Where T represents the number of time frames, which is 16 frames in this embodiment. H, W, and D represent the unified spatial height, width, and depth, respectively, which are set to 128, 128, and 8 in this embodiment. This tensor is used as the input for subsequent modules.

[0026] Step S20: Decompose the standardized four-dimensional input tensor into orthogonal two-dimensional feature planes, wherein the two-dimensional feature planes include a spatial projection plane and a spatiotemporal coupling plane; Specifically, step S20 includes the following steps: Step S21: Project the standardized four-dimensional input tensor onto the three orthogonal spatial dimension pairs respectively to generate a spatial projection plane; Step S22: Project the standardized four-dimensional input tensor onto the three spatiotemporal pairs consisting of the time dimension and each spatial dimension to generate a spatiotemporal coupling plane.

[0027] The normalized four-dimensional input tensor obtained after step S10 The input is fed into the planar decomposition and encoding module, which first uses a three-dimensional convolutional neural network to perform deep feature extraction on the input tensor, resulting in a high-dimensional four-dimensional feature tensor. F Subsequently, the module decomposes the four-dimensional feature tensor into six orthogonal two-dimensional feature planes, the set of which is defined as follows: The specific decomposition process includes the following two consecutive steps: First, step S21 is executed, which involves retrieving the four-dimensional feature tensor. F Projections are made onto three pairs of orthogonal planes consisting of pure spatial dimensions to generate spatial projection planes, where... Indicates height in spatial dimensions H With width W The projection on the plane formed Indicates the height H in the spatial dimension H With depth D D The projection on the plane formed Indicates the width in the spatial dimension W With depth D The projection on the plane formed; Next, step S22 is executed, which involves the four-dimensional feature tensor. F To three directions respectively, based on the time dimension TProjecting the planes coupled with each spatial dimension onto each other generates a spatiotemporal coupling plane, where... In the time dimension T With spatial height H The projection on the plane formed In the time dimension T With space width W The projection on the plane formed In the time dimension T With spatial depth D The projection onto the plane formed; this projection operation is mathematically implemented through a learnable projection function that aggregates and transforms the input feature tensor in two specified dimensions.

[0028] Step S30: Establish a corresponding latent probability distribution for each two-dimensional feature plane, and sample from the established latent probability distribution to obtain a differentiable latent plane representation; In this embodiment, the steps of establishing the potential probability distribution and sampling include: Step S31: Apply the encoding process to each two-dimensional feature plane to map the two-dimensional feature plane to the mean and variance parameters of the potential probability distribution; Step S32: Using the reparameterization method, the obtained mean and variance parameters are combined with random sampled values ​​obtained from the standard normal distribution to calculate the differentiable latent plane representation.

[0029] The set of six two-dimensional feature planes obtained in step S20 This step aims to provide each of the planes with [the necessary parameters]. ,in Establish its corresponding latent probability distribution, and obtain a continuous and differentiable latent plane representation through sampling. .

[0030] See Figure 2 The process first executes step S31, which involves independently applying a variational autoencoder (VAE) to each two-dimensional feature plane; specifically, each plane... It is fed into a corresponding encoder network Ek The encoder network typically consists of several convolutional and fully connected layers. Its function is to map and compress the input two-dimensional feature plane into a parameter vector representing its potential probability distribution. The encoder network Ek The output consists of two independent parameters: the mean of the Gaussian distribution. With variance This fully defines the latent probability distribution of the planar feature as a Gaussian distribution. Its mathematical expression is .

[0031] Then, step S32 is executed, which involves sampling from the established probability distribution using a reparameterization method. Direct sampling is non-differentiable and would block gradient propagation; therefore, a reparameterization technique is used to achieve a differentiable sampling process. This technique first uses a standard normal distribution... Sampling a random noise vector Then, using the distribution parameters obtained in step S31 and The final latent plane representation is calculated through a deterministic linear transformation. Its calculation formula is In this way, the gradient can be obtained through parameters. and The data successfully propagates backward from the decoder to the encoder. Ek This allows the entire planar decomposition coding module to undergo stable end-to-end training using stochastic gradient descent. At this point, all six two-dimensional feature planes have been transformed into their corresponding differentiable latent representations. This provides input for subsequent spatiotemporal fusion.

[0032] Step S40: Based on the structural equivalence prior, all differentiable latent plane representations are fused to generate a unified spatiotemporal latent space representation. The structural equivalence prior establishes a mapping relationship where the rate of change of time is driven by the spatial gradient. In this embodiment, the steps for fusing differentiable latent plane representations include: Step S41: Apply a fusion function containing geometric perception weights and attention mechanisms to process a set of differentiable latent plane representations. In step S42, the fusion function, under the constraint of structural equivalence prior, aggregates features from different spatial projection planes and spatiotemporal coupling planes to generate a unified spatiotemporal latent space representation.

[0033] The mapping relationship established by structural equivalence priors is defined as follows: The temporal rate of change of the dynamic scene representation is a function of the spatial gradient of the dynamic scene representation; the prior correlates temporal evolution with spatial geometric deformation in the latent space.

[0034] In obtaining the set of six differentiable latent plane representations Subsequently, the core objective of this step is to integrate these representations into a unified and coherent spatiotemporal latent space representation under the constraint of structural equivalence a priori knowledge. .

[0035] The process first executes step S41, which involves applying a specific fusion function. All potential plane representations are processed. This fusion function... It is a parameter of The Multilayer Perceptron (MLP) incorporates geometric perception weights and attention mechanisms to adaptively weight and interact with the importance of features from different planes. Specifically, the function divides the six input planes into two groups: the first group is a set of spatial projection planes. Sspatial ={ xy , xz , yz The second group is a set of spatiotemporal coupling planes. Sspatio temporal ={ tx , ty , tz For a set of spatial planes, the function employs a convolution operation. (Specifically, two-dimensional convolution) for each plane Apply its corresponding geometric perception weights This weight is used to capture and emphasize the geometric characteristics of spatial structure; for the spatiotemporal plane group, element-wise multiplication (Hadamard product) is used. Combining attention weights This process is performed to model the fine-grained coupling relationship between the temporal and spatial dimensions. The specific calculation process of the fusion function is defined as follows: First, the two sets of planes are weighted and aggregated to obtain two aggregated feature vectors. Then, these two vectors are concatenated and fed into a multilayer perceptron for nonlinear fusion, ultimately outputting a unified latent space representation. , .

[0036] The aforementioned fusion process is strictly constrained by the structural equivalence prior defined in step S42. This prior is the core of the invention, establishing a physical constraint that correlates temporal evolution with spatial geometric deformation. Its mathematical form is defined as: In this equation, Represents dynamic scene representation S Over time t rate of change, This represents the representation in space. gradient on the function This represents a mapping relationship where temporal evolution is driven by spatial geometric deformation. The physical significance of this a priori is that it forces temporal changes in the latent space to be driven by changes in the spatial structure of the scene (i.e., geometric deformation), thereby fundamentally ensuring the consistency of the generated spatiotemporal representation in terms of physical rationality and avoiding the decoupling or contradiction between temporal and spatial features.

[0037] In practical network implementations, this prior can be reflected by designing specific loss function terms, such as constraining the correlation between the temporal differences of latent space features and their spatial gradients; alternatively, it can be achieved through network architecture design, making the fusion function... The learning process implicitly satisfies this mapping relationship. Finally, through the fusion calculation in step S41 and the prior constraints in step S42, the system outputs a unified latent space representation that contains rich spatiotemporal information while maintaining internal physical consistency. This provides input for the subsequent scene reconstruction module.

[0038] See Figure 3 Step S50: Perform multi-layer decoding on the unified spatiotemporal latent space representation to generate an upsampled feature map. The decoding process utilizes a residual connection structure, which adds the input features of the decoding layer to the output features of the decoding layer. The steps for multi-layer decoding of the unified spatiotemporal latent space representation further include: Step S51: Reorganize the unified spatiotemporal latent space representation; Step S52: Apply local context processing and channel interaction processing to the recombined representation in sequence; Step S53: Apply multi-level deconvolution operations to the processed features to perform progressive upsampling, thereby generating an upsampled feature map.

[0039] The unified spatiotemporal latent space representation generated in step S40 This step aims to gradually restore the feature map to a high-resolution upsampled feature map through a hierarchical decoding process, preparing for the final reconstruction. The core of this decoding process lies in using a residual connection structure to avoid gradient vanishing and feature collapse, ensuring that spatial and temporal details are preserved.

[0040] First, step S51 is executed, which involves reorganizing the unified latent space representation. Specifically, the latent space reorganization unit receives the input latent space tensor. ,Right now The set representation of the unit divides and reorganizes features along the time dimension T. Through tensor deformation and block operation, the global latent space features are mapped into a series of spatiotemporally continuous local feature sub-blocks. This process aims to adapt the feature organization structure to the subsequent decoder processing flow and maintain the intrinsic consistency of features in the spatial and temporal dimensions.

[0041] Next, step S52 is executed, where the recombined feature tensor undergoes local context processing and channel interaction processing sequentially. First, local context processing is performed: the features are input into a local context block, the structure of which is “InstanceNorm→InstanceNorm→3×3×3 convolution”, that is, “instance normalization→instance normalization→3×3×3 convolution”.

[0042] The first instance normalization layer normalizes the features across batches to reduce internal covariate bias; the second instance normalization layer further stabilizes the feature distribution; this is followed by a 3×3×3 convolutional layer to extract detailed contextual information within the local neighborhood of the features (covering both spatial and temporal dimensions). The output of this process is added to the block's input features via a residual connection, calculated using the following formula: and This allows for the enhancement of local features while preserving the original information.

[0043] Then, channel interaction processing is performed: the features enhanced by local context are... The input channel interaction block has a structure of "InstanceNorm→InstanceNorm→1×1×1 convolution", which is "instance normalization→instance normalization→1×1×1 convolution". The two layers of instance normalization serve a similar purpose as described above, while the 1×1×1 convolution is used to model the dependencies between different feature channels, enabling cross-channel information interaction and feature compression.

[0044] Similarly, the output of this block is added to its input via a residual connection, calculated using the following formula: This emphasizes important channel features and enhances feature representation capabilities. Finally, step S53 is executed, which involves applying multi-level deconvolution operations to progressively upsample the refined features.

[0045] Specifically, features The input is a deconvolutional unit consisting of multiple 3D transposed convolutional layers. Each deconvolutional layer (i.e., ConvTranspose3D) is responsible for upsampling in the spatial dimensions (height, width, depth) and / or the temporal dimension, progressively increasing the resolution of the feature map. This process can be formally represented as... ,in Indicates the first The feature maps of the multi-level decoding layer are also used, and residual connections are applied between each level. Through this multi-level, progressive upsampling method, the resolution of the feature maps is gradually restored to a size close to that of the original input dynamic scene voxel sequence, ultimately generating upsampled feature maps containing rich spatiotemporal details. This provides direct input for subsequent scene semantic reconstruction.

[0046] Step S60: Reconstruct the scene based on the upsampled feature map to obtain the reconstructed four-dimensional dynamic scene, and generate a semantic category prediction for each voxel in the voxel sequence.

[0047] In this embodiment, the steps for scene reconstruction based on the upsampled feature map include: Step S61: Perform a convolution operation on the upsampled feature map to map the high-dimensional features of the upsampled feature map to the probability of each voxel on the preset semantic category set; Step S62: Apply a normalization function to process the obtained probabilities and output the final semantic category prediction result for each voxel.

[0048] The upsampled feature map output from step S50 This step aims to complete the final reconstruction of the scene, generate a four-dimensional dynamic scene voxel sequence corresponding to the original input, and assign a semantic category label to each voxel point in the sequence.

[0049] The process first executes step S61, which inputs the upsampled feature map into a classification head module. This classification head consists of a 1×1×1 three-dimensional convolutional layer, and its core function is to perform final feature transformation and dimensionality reduction on the feature map. Specifically, let the preset total number of semantic categories be... C (For example, including categories such as roads, buildings, vehicles, and pedestrians), this convolutional layer will The number of channels is mapped from the number of high-dimensional feature channels to C Output a tensor with one dimension, and the last dimension... C Each value in the table represents a corresponding voxel point. Belongs to the c The unnormalized scores for each category. This operation completes a crucial step in mapping high-dimensional, abstract visual features into spatial probability distributions for specific semantic classification tasks.

[0050] Then, step S62 is executed, which involves applying a normalization function to process the obtained category scores to transform them into a normalized probability distribution. In this embodiment, the normalization function is preferably the Softmax function. The Softmax function operates along the category dimension. C For each voxel C Each score is calculated to ensure that the sum of the probabilities of all categories is 1.

[0051] Its calculation process is formally represented as follows: ,in, Indicates a location in time and space. The voxel at that location belongs to the first cThe final predicted probability of the class. By taking the index of the class with the highest probability, a definite semantic class prediction label can be output for each voxel, thus obtaining a complete four-dimensional dynamic scene reconstruction result with semantic information, i.e., a discrete label tensor.

[0052] Finally, the performance of the reconstruction system is evaluated by comparing the predicted voxel labels with the ground truth semantic labels. Specifically, the Intersection over Union (IoU) and the class-average IoU (mIoU) are used as the core evaluation metrics. For each semantic category... c By statistically analyzing true positive results False positives and false negatives Calculate the IoU based on the number of voxels: .

[0053] By all C The average IoU across all categories is used to obtain the overall performance metric mIoU: The system aims to achieve accurate semantic understanding while outputting high-fidelity four-dimensional scene geometry through the aforementioned reconstruction and classification process.

[0054] See Figure 4 Experimental results show that, under the same computing resource conditions, the present invention achieves a significant improvement in mIoU compared to existing methods (such as DynamicCity and OccSora) in various temporal settings of 4–64 frames, and has a faster training convergence speed, demonstrating good prospects for engineering applications. As can be seen from the figure, SEP-4D can accurately restore the spatial structure of the scene (such as the outline of the bus station and the position of the vehicle) and the temporal evolution (vehicle movement trajectory) in each temporal frame, while the baseline model has problems such as blurry outline and target position offset, verifying the spatiotemporal consistency advantage of SEP-4D.

[0055] This embodiment also includes, after completing the forward process construction of the four-dimensional dynamic scene representation and reconstruction network based on structural equivalence priors, optimizing the network parameters through end-to-end training. The training process of this invention involves the following key designs: The first is the loss function. The overall loss function of the model consists of three parts: reconstruction loss, semantic classification loss, and representation regularization loss. The specific definitions are as follows: ; This is an L1-norm-based reconstruction loss used to constrain the similarity between the reconstructed scene and the real scene at the voxel level. The weighted cross-entropy loss is used to optimize the accuracy of semantic category prediction, with hyperparameter λ=0.05. β =0.1 is the weight of the KL divergence.

[0056] Specifically, the training loss of the encoder part Focusing on reconstruction and regularization: Training loss of the decoder part This focuses on both reconstruction and semantic accuracy: The network is jointly optimized using the total loss L.

[0057] The second aspect is the optimizer and training strategy. In this embodiment, the AdamW optimizer is used to optimize all parameters of the model. θ The update is performed using the following formula: , ,in, For a moment gradient and These are first-order and second-order momentum estimates, respectively. The optimizer hyperparameters are set as follows: initial learning rate η = 0.001, momentum parameter... =0.9, =0.999, weight decay coefficient λ=0.05, It is the numerical stability constant; The learning rate scheduling employs a Multi-Step Decay strategy, scaling the learning rate to 0.70 times its original value at the 30th, 40th, and 50th epochs. This strategy can be formally expressed as: ; in, The initial learning rate, This is an indicator function that takes the value 1 when the number of cycles meets the condition, and 0 otherwise. This strategy helps to fine-tune the model parameters in the later stages of training, improving convergence stability and final performance.

[0058] Although the steps in the above embodiments are described in the above order, those skilled in the art will understand that in order to achieve the effect of this embodiment, different steps do not need to be executed in such an order. They can be executed simultaneously (in parallel) or in a reverse order. These simple variations are all within the protection scope of this invention.

[0059] See Figure 5 The second embodiment of the present invention proposes a four-dimensional dynamic scene representation and reconstruction system based on structural equivalence priors, used to execute a four-dimensional dynamic scene representation and reconstruction method based on structural equivalence priors. The system includes: The data processing module is configured to acquire multi-frame three-dimensional voxel sequence data and corresponding semantic labels as raw dynamic scene data, and standardize them into four-dimensional input tensors. The decomposition module is configured to decompose a standardized four-dimensional input tensor into orthogonal two-dimensional feature planes, wherein the two-dimensional feature planes include a spatial projection plane and a spatiotemporal coupling plane. The characterization generation module is configured to establish a corresponding latent probability distribution for each two-dimensional feature plane and to sample from the established latent probability distribution to obtain a differentiable latent plane characterization. The fusion module is configured to fuse all differentiable latent plane representations based on structural equivalence priors to generate a unified spatiotemporal latent space representation. The structural equivalence priors establish a mapping relationship in which the rate of change of time is driven by the spatial gradient. The decoding module is configured to perform multi-layer decoding on a unified spatiotemporal latent space representation to generate an upsampled feature map. The decoding process utilizes a residual connection structure, which adds the input features of the decoding layer to the output features of the decoding layer. The reconstruction module is configured to perform scene reconstruction based on upsampled feature maps to obtain a reconstructed four-dimensional dynamic scene and generate semantic category predictions for each voxel in the voxel sequence.

[0060] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related explanations of the methods described above can be found in the corresponding processes in the foregoing system embodiments, and will not be repeated here.

[0061] It should be noted that the four-dimensional dynamic scene representation and reconstruction system based on structural equivalence priors provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiments can be merged into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the various modules or steps and are not considered as an improper limitation of the present invention.

[0062] A device according to a third embodiment of the present invention includes: At least one processor; and a memory communicatively connected to at least one of the processors; The memory stores instructions that can be executed by the processor to implement the above-described method for four-dimensional dynamic scene representation and reconstruction based on structural equivalence priors.

[0063] A computer-readable storage medium according to a fourth embodiment of the present invention stores computer instructions, which are executed by the computer to implement the above-described method for four-dimensional dynamic scene representation and reconstruction based on structural equivalence priors.

[0064] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process and related descriptions of the storage device and processing device described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0065] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a computer system for implementing embodiments of the systems, methods, and electronic devices of this application. Figure 6 The server shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0066] like Figure 6 As shown, the computer system includes a Central Processing Unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in Read Only Memory (ROM) 602 or programs loaded from storage section 608 into Random Access Memory (RAM) 603. The RAM 603 also stores various programs and data required for system operation. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An Input / Output (I / O) interface 605 is also connected to the bus 604.

[0067] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0068] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined in the methods of this application. It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0069] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0070] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0071] The terms “first”, “second”, etc., are used to distinguish similar objects, not to describe or indicate a specific order or sequence.

[0072] The term "comprising" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus / device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent in such process, method, article, or apparatus / device.

[0073] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A method for representing and reconstructing four-dimensional dynamic scenes based on structural equivalence priors, characterized in that, The method includes: Acquire multi-frame 3D voxel sequence data and corresponding semantic labels as raw dynamic scene data, and standardize them into a four-dimensional input tensor; The standardized four-dimensional input tensor is decomposed into orthogonal two-dimensional feature planes, which include a spatial projection plane and a spatiotemporal coupling plane. Establish a corresponding latent probability distribution for each two-dimensional feature plane, and sample from the established latent probability distribution to obtain a differentiable latent plane representation; Based on structural equivalence priors, all differentiable latent plane representations are fused to generate a unified spatiotemporal latent space representation. The structural equivalence priors establish a mapping relationship where the rate of change of time is driven by the spatial gradient. A unified spatiotemporal latent space representation is decoded in multiple layers to generate an upsampled feature map. The decoding process utilizes a residual connection structure, which adds the input features of the decoding layer to the output features of the decoding layer. Scene reconstruction is performed based on upsampled feature maps to obtain a reconstructed four-dimensional dynamic scene, and semantic category predictions are generated for each voxel in the voxel sequence.

2. The four-dimensional dynamic scene representation and reconstruction method based on structural equivalence priors as described in claim 1, characterized in that, The steps for preprocessing raw dynamic scene data include: Unify spatial resolution, align temporal frame rates, normalize, and filter noise; It also includes spatiotemporal consistency enhancement processing of voxel sequence data, which is applied to the geometric structure and temporal evolution characteristics of dynamic scenes.

3. The four-dimensional dynamic scene representation and reconstruction method based on structural equivalence priors as described in claim 1, characterized in that, The steps for decomposing a normalized four-dimensional input tensor include: The standardized four-dimensional input tensor is projected onto three orthogonal spatial dimension pairs to generate a spatial projection plane. The standardized four-dimensional input tensor is projected onto three spatiotemporal pairs consisting of the time dimension and each spatial dimension to generate a spatiotemporal coupling plane.

4. The four-dimensional dynamic scene representation and reconstruction method based on structural equivalence priors as described in claim 1, characterized in that, The steps for establishing the potential probability distribution and sampling include: An encoding process is applied to each two-dimensional feature plane to map the two-dimensional feature plane to the mean and variance parameters of the latent probability distribution; By employing a reparameterization method, combining the obtained mean and variance parameters with random sampled values ​​obtained from the standard normal distribution, a differentiable latent plane representation is calculated.

5. The four-dimensional dynamic scene representation and reconstruction method based on structural equivalence priors according to claim 1, characterized in that, The steps for fusing differentiable latent plane representations include: The application incorporates a fusion function that combines geometric perception weights and attention mechanisms to process a set of differentiable latent plane representations; Under the constraint of structural equivalence priors, the fusion function aggregates features from different spatial projection planes and spatiotemporal coupling planes to generate a unified spatiotemporal latent space representation.

6. The four-dimensional dynamic scene representation and reconstruction method based on structural equivalence priors according to claim 1, characterized in that, The mapping relationship established by structural equivalence priors is defined as follows: The temporal rate of change of the dynamic scene representation is a function of the spatial gradient of the dynamic scene representation; the prior correlates temporal evolution with spatial geometric deformation in the latent space.

7. The four-dimensional dynamic scene representation and reconstruction method based on structural equivalence priors according to claim 1, characterized in that, The steps for multi-layer decoding of the unified spatiotemporal latent space representation further include: Reorganize and unify the spatiotemporal latent space representation; The reconstructed representation is then processed using local contextualization and channel interaction methods in sequence. The processed features are progressively upsampled by applying multi-level deconvolution operations to generate upsampled feature maps.

8. The four-dimensional dynamic scene representation and reconstruction method based on structural equivalence priors according to claim 1, characterized in that, The steps for scene reconstruction based on upsampled feature maps include: Perform a convolution operation on the upsampled feature map to map the high-dimensional features of the upsampled feature map to the probability of each voxel on a preset semantic category set; The obtained probabilities are processed using a normalization function, and the final semantic category prediction result is output for each voxel.

9. A four-dimensional dynamic scene representation and reconstruction system based on structural equivalence priors, used to execute the four-dimensional dynamic scene representation and reconstruction method based on structural equivalence priors as described in any one of claims 1-8, characterized in that, The system includes: The data processing module is configured to acquire multi-frame three-dimensional voxel sequence data and corresponding semantic labels as raw dynamic scene data, and standardize them into four-dimensional input tensors. The decomposition module is configured to decompose a standardized four-dimensional input tensor into orthogonal two-dimensional feature planes, wherein the two-dimensional feature planes include a spatial projection plane and a spatiotemporal coupling plane. The characterization generation module is configured to establish a corresponding latent probability distribution for each two-dimensional feature plane and to sample from the established latent probability distribution to obtain a differentiable latent plane characterization. The fusion module is configured to fuse all differentiable latent plane representations based on structural equivalence priors to generate a unified spatiotemporal latent space representation. The structural equivalence priors establish a mapping relationship in which the rate of change of time is driven by the spatial gradient. The decoding module is configured to perform multi-layer decoding on a unified spatiotemporal latent space representation to generate an upsampled feature map. The decoding process utilizes a residual connection structure, which adds the input features of the decoding layer to the output features of the decoding layer. The reconstruction module is configured to perform scene reconstruction based on upsampled feature maps to obtain a reconstructed four-dimensional dynamic scene and generate semantic category predictions for each voxel in the voxel sequence.

10. A device, characterized in that, include: At least one processor; and a memory communicatively connected to at least one of the processors; The memory stores instructions that can be executed by the processor to implement the four-dimensional dynamic scene representation and reconstruction method based on structural equivalence prior as described in any one of claims 1-8.

Images