Three-dimensional dynamic scene reconstruction method based on adaptive key frame and gear attention

Abstract

The invention relates to the technical field of neural radiation field reconstruction, in particular to a three-dimensional dynamic scene reconstruction method based on adaptive key frames and gear attention, which comprises the following steps: S1, acquiring a multi-view video sequence and camera internal and external parameters and timestamps; s2, generating a sampling point set; s3, generating multi-level gear weight distribution of the space-time coordinates of each sampling point; s4, an SAM model and an optical flow algorithm are combined with an iteration strategy to obtain a self-adaptive key frame set corresponding to the training view angle; s5, calculating the spatial-temporal feature representation of the sampling points based on the multi-stage gear weight distribution of the spatial-temporal coordinates; s6, generating a rendering image of a training visual angle based on the spatial-temporal feature representation of the sampling points and the visual angle direction prediction density value and color value; s7, calculating a loss function based on the rendered image and the corresponding real image, and optimizing parameters of the scene reconstruction model; and S8, iteratively training the scene reconstruction model until convergence or reaching a preset number of iterations. According to the invention, the efficiency and quality of three-dimensional dynamic scene reconstruction can be improved.

Description

Technical Field

[0001] This invention relates to the field of neural radiation field reconstruction technology, specifically to a method for reconstructing three-dimensional dynamic scenes based on adaptive keyframes and gear attention. Background Technology

[0002] 3D scene reconstruction is of great significance for promoting the development of industries such as VR / AR, cultural heritage protection, film and television, and games. In 2020, the introduction of Neural Radiance Field (NeRF) triggered a revolutionary change in the field of computer vision and graphics. NeRF takes multi-view images as input and uses deep neural networks to learn the color and density distribution of light in a scene, thereby implicitly representing 3D geometric and texture information in continuous space. During rendering, the model queries the color and density of each sampling point through the neural network, and then integrates this information using volume rendering technology to finally obtain the color of each pixel and generate a new perspective image. This method based on implicit neural representation shows extremely high accuracy and realism in static scene reconstruction. However, the challenges increase significantly when extending to NeRF dynamic scenes. Dynamic scenes in the real world often contain non-rigid motion, lighting changes, occlusion, and complex spatiotemporal dependencies. Under the condition of limited computing resources, how to efficiently and accurately model these complex changes has become a major bottleneck restricting the practical application of NeRF scene reconstruction.

[0003] To address this issue, scholars have proposed various improvement directions in recent years, driving the development of NeRF dynamic scene reconstruction technology. Current mainstream methods can be broadly categorized into three types: The first type typically introduces a deformation field, mapping the radiation field of each time frame to a shared canonical space, thereby achieving time-consistent reconstruction and rendering. The second type directly establishes a four-dimensional representation in the spatiotemporal domain (such as spatiotemporal voxel grids, tensor decomposition, or hash coding structures), explicitly modeling the dynamic process, thus achieving more efficient modeling. The third type employs a dynamic-static decomposition strategy, decomposing the scene into a static background and a dynamic foreground, and modeling and optimizing them separately to reduce the interference of dynamic changes on the overall reconstruction accuracy.

[0004] While the aforementioned NeRF-based 3D reconstruction methods significantly improve the reconstruction performance of dynamic scenes, achieving high-quality and efficient modeling in complex real-world scenes still faces numerous challenges. Real-world dynamic scenes exhibit high non-uniform time-varying characteristics—some areas show dramatic motion or significant appearance changes, while others remain almost static. Furthermore, due to limited viewing angles or insufficient temporal sampling by the acquisition devices, some video sequences may only record a portion of the motion state of dynamic objects. In such cases, uniform sampling or fixed keyframe strategies struggle to capture the subtle differences in motion, resulting in a significant consumption of computational resources in low-dynamic regions while the representation of high-dynamic regions remains insufficient, leading to blurry rendering results and noticeable motion artifacts. Therefore, achieving adaptive allocation of computational resources while maintaining reconstruction accuracy has become a major research direction for NeRF-based dynamic scene reconstruction.

[0005] Gear-NeRF, an improved version of NeRF, is a technology specifically designed for efficient rendering and tracking of dynamic scenes. The core of Gear-NeRF is dividing the dynamic scene into multiple "gear" levels, each corresponding to a different spatiotemporal sampling density, to achieve a balance between rendering quality and computational efficiency. However, the gear level division process in existing technologies is computationally expensive and prone to errors. For example, objects with simple textures and fast movement may not exhibit significant changes in luminosity but could be incorrectly assigned to a low-level gear; conversely, static areas with complex reflections or detailed textures may be incorrectly classified into higher-level gears due to changes in viewing angle.

[0006] In summary, how to adaptively allocate computing resources in complex dynamic scenes and fully explore the spatiotemporal structure information of the scene in order to achieve efficient and high-fidelity 3D dynamic scene reconstruction has become the core issue of current research. Summary of the Invention

[0007] To address the shortcomings of the existing technologies, the technical problem to be solved by this invention is: how to provide a 3D dynamic scene reconstruction method based on adaptive keyframes and gear attention, which optimizes the temporal sampling of keyframes through an adaptive keyframe selection strategy, and simultaneously generates gear weights adaptively based on spatial and temporal coordinates through a gear attention mechanism, intelligently fusing the optimal feature representation, thereby improving the efficiency and quality of 3D dynamic scene reconstruction.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0009] A method for reconstructing 3D dynamic scenes based on adaptive keyframes and gear attention includes:

[0010] S1: Obtain the multi-view video sequence and the corresponding camera intrinsic and extrinsic parameters and timestamps;

[0011] S2: For each video frame in the video sequence, emit rays based on the camera's intrinsic and extrinsic parameters and sample multiple points along each ray in three-dimensional space. Record the spatial position, timestamp, and viewing direction of each sampling point to generate a set of sampling points.

[0012] S3: Extract the location and time features of the spatiotemporal coordinates of each sampling point in the sampling point set through spatial encoder and temporal encoder, and combine the gating layer and self-attention mechanism to perform feature fusion to generate a multi-level gear weight distribution of the spatiotemporal coordinates of each sampling point;

[0013] S4: For the training viewpoint, the importance score of each video frame is calculated using the SAM model and optical flow algorithm, and the corresponding training viewpoint adaptive keyframe set is obtained through an iterative strategy; the sampling points and rays corresponding to each keyframe in the keyframe set of the corresponding training viewpoint are used as the input of the scene reconstruction model.

[0014] S5: For each sampling point in the keyframe, calculate the spatiotemporal feature representation based on the multi-level gear weight distribution of its corresponding spatiotemporal coordinates;

[0015] S6: For each sampling point in the keyframe, predict the density and color values ​​based on its corresponding spatiotemporal feature representation and viewpoint direction, and generate the rendering color of each ray based on the density and color values ​​of each sampling point, and finally generate the rendering image corresponding to the training viewpoint.

[0016] S7: Calculate the loss function based on the rendered image and the corresponding real image, and optimize the parameters of the scene reconstruction model;

[0017] S8: Repeat steps S4 to S7 to iteratively train the scene reconstruction model until the model converges or reaches the preset number of iterations to obtain the trained scene reconstruction model.

[0018] Preferably, in step S1, time synchronization and scene calibration are performed on the video frames in the multi-view video sequence.

[0019] Preferably, in step S3, the multi-level gear weight distribution of the spatiotemporal coordinates of each sampling point is generated through the following steps:

[0020] S301: Select a sampling point from the set of sampling points and obtain its spatiotemporal coordinates (x,t), where x represents the spatial location and t represents the timestamp;

[0021] S302: Via space encoder MLP s Extracting the positional features f of spatiotemporal coordinates (x, t) s ;

[0022] The formula is expressed as:

[0023] f s =MLP s (x);

[0024] S303: Via time encoder MLP t Extracting the temporal features f of spatiotemporal coordinates (x, t) t ;

[0025] The formula is expressed as:

[0026] f t =MLP t (t);

[0027] S304: The positional features of the spatiotemporal coordinates (x,t) f s and time features f t Connect them into a unified spatiotemporal feature vector f st =[f s ,f t The gate vector g is obtained by adaptive fusion through a gating layer.

[0028] The formula is expressed as:

[0029] g=σ(W g f st +b g );

[0030] In the formula: [·, ·] denotes the feature join operation; W g and b g The parameters represent the learnable parameters of the gated layer; σ(·) is the sigmoid activation function;

[0031] S305: Using a self-attention mechanism, the spatiotemporal feature vector f of the spatiotemporal coordinates (x,t) is... st Projected onto projection matrix W Q W K W V Generate the query matrix Q, the key matrix K, and the value matrix V;

[0032] The formula is expressed as:

[0033] Q = W Q f st K = W K f st V = W V f st ;

[0034] S306: Element-wise multiplication and softmax normalization are performed on the query matrix Q and key matrix K of the spatiotemporal coordinates (x,t) to obtain the self-attention weight α. Then, element-wise multiplication is performed on the self-attention weight α, the value matrix V, and the gate vector g to obtain the attention enhancement feature f. g ;

[0035] The formula is expressed as:

[0036] α = softmax(Q⊙K);

[0037] f g =g⊙α⊙V;

[0038] S307: Attention-enhanced features f based on spatiotemporal coordinates (x, t) through multi-layer nonlinear transformation g Assign gear weights to obtain the spatiotemporal coordinates (x, t) in Attention weights

[0039] The formula is expressed as:

[0040]

[0041] In the formula: Let represent a multilayer nonlinear transform network; λ is a learnable residual weight parameter; u is a uniformly distributed vector; β is a dynamic weight balancing parameter vector; ψ(γ) is an adaptive temperature function.

[0042] S308: Repeat S301 to S307 to obtain the spatiotemporal coordinates of each sampling point and the attention weights of each level of gear as the multi-level gear weight distribution.

[0043] Preferably, in step S4, the importance score of the video frame is calculated through the following steps:

[0044] S401: Extract semantic features of video frames using a pre-trained SAM model.

[0045] S402: Semantic features based on video frames Calculate the semantic importance score S based on SAM. t ;

[0046] The formula is expressed as:

[0047] S t =α V ·V t +α A ·A t +α E ·E t ;

[0048]

[0049] In the formula: α V α A α E V represents the weighting coefficient; t Representing semantic features The variance; A t Indicates the use of semantic features The L2 norm is used to quantify visual saliency, E t Indicates edge density;

[0050] S403: Calculate the optical flow-based motion dynamics score M of video frames using an optical flow algorithm. t ;

[0051] The formula is expressed as:

[0052]

[0053] In the formula: H and W represent the height and width of the video frame, respectively; (i, j) represent the pixel coordinates in the video frame; u(i, j) and v(i, j) represent the horizontal and vertical components of the optical flow field of the video frame.

[0054] S404: Semantic importance score based on SAM for video frames. t and the motion dynamics fraction M based on optical flow t Calculate its importance score I t ;

[0055] The formula is expressed as:

[0056] I t =α·M t +(1-α)·S t ;

[0057] In the formula: α∈(0,1) represents the balance factor.

[0058] Preferably, in step S4, the keyframe set is generated through the following steps:

[0059] S411: For a video sequence containing T frames, select the first and last video frames of the video sequence as keyframes and add them to the keyframe set.

[0060] S412: Select the remaining K-2 keyframes using an iterative strategy. The specific steps include:

[0061] S4121: Construct the initial keyframe set Initialize the importance score of each video frame to I′. t =I t ;

[0062] S4122: Update the importance scores of the remaining video frames based on the keyframes already selected in the keyframe set;

[0063] The formula is expressed as:

[0064]

[0065] In the formula: θ=max(1,T / (2K)) is the time window parameter; τ j It is the time index of the j-th selected keyframe in the keyframe set; Π represents the product; t is the timestamp;

[0066] S4123: Select the most important video frame τ from the remaining video frames. k Add it to the keyframe set as a keyframe;

[0067] The formula is expressed as:

[0068]

[0069] S4124: Repeat steps S4122 and S4123, iteratively selecting K-2 keyframes to add to the keyframe set.

[0070] Preferably, in step S5, the spatiotemporal feature representation of the sampling points is calculated through the following steps;

[0071] S501: For sampling points in keyframes, obtain their spatiotemporal coordinates (x, t) and the attention weights of each level of gear, i.e., the multi-level gear weight distribution.

[0072] S502: Calculate the eigenvectors of the spatiotemporal coordinates (x, t) at each gear level;

[0073] The formula is expressed as:

[0074]

[0075]

[0076] In the formula: Represents the spacetime coordinates (x, t) in eigenvectors; This represents the spatiotemporal coordinates (x, t) calculated by the gear attention mechanism in [the context of the image / image]. Attention weights; B j (·) indicates that h j (·,·)and The product of is mapped to a linear transformation of the eigenvectors; h j (·,·) is a vector-valued function with an output dimension of M; It is each Unique spatiotemporal embedding; x = (x, y, z) represents the three-dimensional coordinates of the sampling point in space;

[0077] S503: The spatiotemporal coordinates (x, t) are weighted and summed in the feature vectors of each gear level to obtain the spatiotemporal feature representation f(x, t) of the corresponding sampling point;

[0078] The formula is expressed as:

[0079]

[0080] Where: N gear This indicates the number of gear levels.

[0081] Preferably, in step S6, the fusion feature f(x, t) of the sampling point and the view direction are input into the MLP together to predict the density value and color value of the sampling point.

[0082] Preferably, in step S7, the loss function is calculated using the following formula:

[0083]

[0084] In the formula: This represents the total variation regularization loss; Indicates the L1 regularization loss; λ represents the reconstruction loss consisting of two parts; TV and λ L1 It's a hyperparameter;

[0085] in:

[0086]

[0087] In the formula: r represents a ray emitted from the center of the camera along the pixel direction; C represents the set of all rays used during training; C(r) represents the color values ​​of the rendered image; C gt (r) represents the color value of the real image; F SAM (r) represents the semantic features extracted from the rendered image by the pre-trained SAM model; This represents the semantic features extracted from real images using a pre-trained SAM model.

[0088] Preferably, in step S7, the total variation regularization loss is calculated using the following formula:

[0089]

[0090] In the formula: The set of all spatial feature planes; H and W represent the height and width of the video frame, respectively; Pu，v This represents the eigenvalue of the characteristic plane at coordinates (u, v).

[0091] Preferably, in step S7, the L1 regularization loss is calculated using the following formula:

[0092]

[0093] Where: Ω σ T represents the set of all learnable parameters that constitute density σ; T represents a specific feature plane or vector in the set; T i N represents the value of the i-th element in the tensor. σ This indicates the total number of parameters.

[0094] Compared with existing technologies, the 3D dynamic scene reconstruction method based on adaptive keyframes and gear attention in this invention has the following advantages:

[0095] Existing NeRF-based dynamic scene reconstruction methods typically use keyframes at fixed intervals, leading to the neglect of dramatic motion present in some non-keyframes. To address this issue, this invention introduces an adaptive keyframe selection strategy to optimize keyframe temporal sampling. It adaptively selects the distribution of keyframes along the time axis using semantic and optical flow information, enabling the model to concentrate its representational capabilities on the most dynamic moments of the scene. This allows the keyframe distribution to be adaptively determined based on scene complexity, thereby improving the accuracy of keyframe selection and the efficiency and quality of 3D dynamic scene reconstruction.

[0096] Existing NeRF-based dynamic scene reconstruction methods mostly apply the same weights to all motion region features, resulting in poor reconstruction performance for complex motion regions. To address this issue, this invention proposes a gear attention mechanism. This mechanism adaptively generates gear weights based on spatial and temporal coordinates, intelligently fusing optimal feature representations to achieve adaptive feature fusion across different motion scales. This allows the model to dynamically adjust the weight distribution of different gear levels according to scene content, better handling continuous motion scales and mixed motion modes, thereby further improving the efficiency and quality of 3D dynamic scene reconstruction. Simultaneously, by utilizing low-rank tensor decomposition and lightweight network design, it effectively captures complex dynamic scene changes. Attached Figure Description

[0097] To make the objectives, technical solutions, and advantages of the invention clearer, the invention will now be described in further detail with reference to the accompanying drawings, wherein:

[0098] Figure 1 This is a logic diagram of a 3D dynamic scene reconstruction method based on adaptive keyframes and gear attention.

[0099] Figure 2 Diagram of the adaptive keyframe iteration selection process. Detailed Implementation

[0100] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but only to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0101] The following detailed explanation illustrates the specific implementation methods:

[0102] Example:

[0103] This embodiment discloses a method for reconstructing a 3D dynamic scene based on adaptive keyframes and gear attention.

[0104] It should be noted that in this invention, the term "gear" does not refer to a mechanical transmission component, but rather, through its metaphor of "multi-stage speed change and on-demand power transmission," it is used to represent a spatiotemporal sampling level division unit in the dynamic scene modeling and rendering process. Specifically, a gear is a basic unit for hierarchical management of different regions in a scene based on their motion amplitude, geometric complexity, and semantic importance. Each gear level corresponds to an independent spatiotemporal sampling configuration, thereby achieving differentiated processing for highly dynamic regions and lightweight processing for low-dynamic regions. By dynamically allocating and updating gear levels, the system can significantly reduce redundant computational overhead while maintaining visual fidelity.

[0105] like Figure 1 As shown, the 3D dynamic scene reconstruction method based on adaptive keyframes and gear attention includes:

[0106] S1: Obtain the multi-view video sequence and the corresponding camera intrinsic and extrinsic parameters and timestamps;

[0107] S2: For each video frame in the video sequence, emit rays based on the camera's intrinsic and extrinsic parameters and sample multiple points along each ray in three-dimensional space. Record the spatial position, timestamp, and viewing direction of each sampling point to generate a set of sampling points.

[0108] S3: Extract the location and time features of the spatiotemporal coordinates of each sampling point in the sampling point set through spatial encoder and temporal encoder, and combine the gating layer and self-attention mechanism to perform feature fusion to generate a multi-level gear weight distribution of the spatiotemporal coordinates of each sampling point;

[0109] S4: For the training viewpoint, the importance score of each video frame is calculated using the SAM (SegmentAnythingModel) model and optical flow algorithm, and the corresponding training viewpoint adaptive keyframe set is obtained through an iterative strategy; the sampling points and rays corresponding to each keyframe in the keyframe set of the corresponding training viewpoint are used as input to the scene reconstruction model.

[0110] S5: For each sampling point in the keyframe, calculate the spatiotemporal feature representation based on the multi-level gear weight distribution of its corresponding spatiotemporal coordinates;

[0111] S6: For each sampling point in the keyframe, predict the density and color values ​​based on its corresponding spatiotemporal feature representation and viewpoint direction, and generate the rendering color of each ray based on the density and color values ​​of each sampling point, and finally generate the rendering image corresponding to the training viewpoint.

[0112] S7: Calculate the loss function based on the rendered image and the corresponding real image, and optimize the parameters of the scene reconstruction model;

[0113] S8: Repeat steps S4 to S7 to iteratively train the scene reconstruction model until the model converges or reaches the preset number of iterations to obtain the trained scene reconstruction model.

[0114] Existing NeRF-based dynamic scene reconstruction methods typically use keyframes at fixed intervals, leading to the neglect of dramatic motion present in some non-keyframes. To address this issue, this invention introduces an adaptive keyframe selection strategy to optimize keyframe temporal sampling. It adaptively selects the distribution of keyframes along the time axis using semantic and optical flow information, enabling the model to concentrate its representational capabilities on the most dynamic moments of the scene. This allows the keyframe distribution to be adaptively determined based on scene complexity, thereby improving the accuracy of keyframe selection and the efficiency and quality of 3D dynamic scene reconstruction.

[0115] Existing NeRF-based dynamic scene reconstruction methods mostly apply the same weights to all motion region features, resulting in poor reconstruction performance for complex motion regions. To address this issue, this invention proposes a gear attention mechanism. This mechanism adaptively generates gear weights based on spatial and temporal coordinates, intelligently fusing optimal feature representations to achieve adaptive feature fusion across different motion scales. This allows the model to dynamically adjust the weight distribution of different gear levels according to scene content, better handling continuous motion scales and mixed motion modes, thereby further improving the efficiency and quality of 3D dynamic scene reconstruction. Simultaneously, by utilizing low-rank tensor decomposition and lightweight network design, it effectively captures complex dynamic scene changes.

[0116] To better illustrate the technical solution of the present invention, this embodiment is described in the following parts.

[0117] I. Multi-view video sequence

[0118] In the specific implementation process, firstly, multi-view video sequences from different perspectives and their corresponding camera pose parameters are input to ensure that images from different perspectives can be accurately aligned in a unified three-dimensional coordinate system. Subsequently, the video sequences are time-synchronized and scene-calibrated, and samples are taken from them to generate the light samples required for training.

[0119] II. Keyframe Set

[0120] In practical implementation, the key to NeRF dynamic scene reconstruction lies in constructing a 4D scene representation that accurately captures time-varying details while maintaining memory efficiency. Unlike traditional uniform keyframe selection, this invention proposes an adaptive keyframe selection strategy to optimize the dynamic representation of video content and effectively capture scene changes. Adaptive keyframe selection optimizes the distribution of keyframes along the time axis, enabling the model to concentrate its representational capabilities on the most dynamic and complex moments of the scene.

[0121] The following section will detail how to use semantic and optical flow information to dynamically adjust keyframe selection, and how to use iterative optimization algorithms to determine the optimal keyframe set.

[0122] To accurately identify keyframes, this invention designs an importance evaluation framework that integrates optical flow motion information and semantic information.

[0123] Specifically, the importance score of a video frame is calculated through the following steps:

[0124] S401: Extract semantic features of video frames using a pre-trained SAM model.

[0125] S402: Semantic features based on video frames Calculate the semantic importance score S based on SAM. t ;

[0126] This invention utilizes a pre-trained SAM model to extract semantic features and construct a multi-dimensional semantic importance assessment. The formula is expressed as:

[0127] S t =α V ·V t +α A ·A t +α E ·E t ;

[0128]

[0129] In the formula: α V α A α E V represents the weighting coefficient; t Representing semantic features The variance; A t Indicates the use of semantic features The L2 norm is used to quantify visual saliency, E t Indicates edge density;

[0130] S403: Calculate the optical flow-based motion dynamics score M of video frames using an optical flow algorithm. t ;

[0131] This invention uses optical flow analysis to quantify dynamic changes in a scene. Specifically, for a multi-view image at time t, the spatial average of the optical flow vector magnitude is calculated to obtain the motion score of a single viewpoint at that time. The formula is expressed as:

[0132]

[0133] In the formula: H and W represent the height and width of the video frame, respectively; (i,j) represents the pixel coordinates in the video frame; v(i,j) and v(i,j) represent the horizontal and vertical components of the optical flow field calculated using the Farneback algorithm; then the scores of all viewpoints in the entire 3D scene at that moment are averaged. To suppress noise, this invention also applies a one-dimensional Gaussian filter to smooth the average motion score of the entire time series.

[0134] S404: Semantic importance score based on SAM for video frames. t and the motion dynamics fraction M based on optical flow t Calculate its importance score I t ;

[0135] The formula is expressed as:

[0136] I t =α·M t +(1-α)·S t ;

[0137] In the formula: α∈(0,1) represents the balance factor.

[0138] This invention employs an iterative selection strategy and spatiotemporal distribution constraints to prevent keyframes from being overly concentrated in the time dimension. Figure 2 The iterative process of keyframe selection is demonstrated.

[0139] Specifically, the keyframe set is generated through the following steps:

[0140] S411: For a video sequence containing T frames, select the first and last video frames of the video sequence as keyframes and add them to the keyframe set to ensure complete coverage of the time span.

[0141] S412: Select the remaining K-2 keyframes using an iterative strategy. The specific steps include:

[0142] S4121: Construct the initial keyframe set Initialize the importance score of each video frame to I′. t =I t ;

[0143] S4122: Update the importance scores of the remaining video frames based on the keyframes already selected in the keyframe set;

[0144] In each iteration, a Gaussian decay penalty is applied around the selected keyframes. This reduces the likelihood of selecting redundant frames. The importance score after the penalty is updated as follows:

[0145]

[0146] In the formula: θ=max(1,T / (2K)) is the time window parameter, a larger θ indicates that adjacent frames will be more strongly affected; τ j It is the time index of the j-th selected keyframe in the keyframe set; Π represents the product; t is the timestamp;

[0147] S4123: Select the most important video frame τ from the remaining video frames. k Add it to the keyframe set as a keyframe;

[0148] The formula is expressed as:

[0149]

[0150] S4124: Repeat steps S4122 and S4123, iteratively selecting K-2 keyframes to add to the keyframe set. Expand the selected keyframe set by merging the newly selected frames. This iterative update continues until a predefined number of keyframes is reached.

[0151] III. Multi-level gear weight distribution

[0152] In its implementation, Gear-NeRF classifies different regions of a dynamic scene into multiple "gear" levels, each corresponding to a different sampling density. However, this classification process is computationally expensive and prone to errors. For example, objects with simple textures and fast movement may not exhibit significant changes in luminosity, yet they might be incorrectly assigned to a low-gear level; conversely, static regions with complex reflections or detailed textures may be incorrectly classified into high-gear levels due to significant errors caused by changes in viewing angle.

[0153] Therefore, this invention proposes a gear attention mechanism that adaptively generates gear weights based on spatial and temporal coordinates and fuses optimal feature representations. This invention effectively captures complex dynamic scene changes by utilizing low-rank tensor decomposition and lightweight network design.

[0154] Specifically, the multi-level gear weight distribution of the spatiotemporal coordinates of each sampling point is generated through the following steps:

[0155] S301: Select a sampling point from the set of sampling points and obtain its spatiotemporal coordinates (x,t), where x represents the spatial location and t represents the timestamp;

[0156] S302: Via space encoder MLP s Extracting the positional features f of spatiotemporal coordinates (x, t) s ;

[0157] The formula is expressed as:

[0158] f s =MLP s (x);

[0159] S303: Via time encoder MLP t Extracting the temporal features f of spatiotemporal coordinates (x, t) t ;

[0160] The formula is expressed as:

[0161] f t =MLP t (t);

[0162] In this embodiment, two independent shallow MLPs are used to encode any spatiotemporal coordinate (x, t) to extract multi-scale features. The spatial encoder MLP is one of them. s The 3D coordinates are processed to extract geometric patterns, while the time encoder focuses on time variations and motion features.

[0163] S304: The positional features of the spatiotemporal coordinates (x, t) f s and time features f t Connect them into a unified spatiotemporal feature vector f st =[f s f t The gate vector g is obtained by adaptive fusion through a gating layer.

[0164] The formula is expressed as:

[0165] g=σ(W g f st +b g );

[0166] In the formula: [·, ·] denotes the feature join operation; W g and b g σ represents the learnable parameters of the gating layer; σ(·) is the sigmoid activation function; the gating vector g is used to control the weight distribution of spatiotemporal feature fusion.

[0167] S305: The spatiotemporal feature vector f of spatiotemporal coordinates (x, t) is obtained through a self-attention mechanism with low-rank decomposition. st Projected onto projection matrix W Q W K W V Generate the query matrix Q, the key matrix K, and the value matrix V;

[0168] The formula is expressed as:

[0169] Q = W Q f s K = W K f st V = W v f st ;

[0170] Where Q, K, and V represent the query, key, and value matrices, respectively, these matrices are derived from the input features f. st Projected onto projection matrix W Q W K W V To calculate.

[0171] S306: Element-wise multiplication and softmax normalization are performed on the query matrix Q and key matrix K of the spatiotemporal coordinates (x, t) to obtain the self-attention weight α. Then, element-wise multiplication is performed on the self-attention weight α, the value matrix V, and the gating vector g to obtain the attention enhancement feature f. g ;

[0172] The formula is expressed as:

[0173] α = softmax(Q⊙K);

[0174] f g =g⊙α⊙V;

[0175] In this embodiment, the attention score is calculated by summing the element-wise products of Q and K, avoiding large-scale matrix multiplication. The self-attention weights α obtained through softmax normalization are then used to weight the value vector V, thereby obtaining the attention enhancement feature f. g .

[0176] S307: Attention-enhanced features f based on spatiotemporal coordinates (x, t) through multi-layer nonlinear transformation g Assign gear weights to obtain the spatiotemporal coordinates (x, t) in Attention weights

[0177] The formula is expressed as:

[0178]

[0179] In the formula: This represents a multi-layer nonlinear transformation network responsible for applying attention-enhanced features f. g Mapped to the original gear weights; λ is a learnable residual weight parameter; u is a uniformly distributed vector; β is a dynamic weight balancing parameter vector that allows the model to learn prior preferences for different gears; the denominator ψ(γ) is an adaptive temperature function used to dynamically control the sharpness of the attention distribution.

[0180] In this embodiment, gear weights are assigned through multi-layer nonlinear transformation. By integrating residual connections, dynamic balancing parameters, and adaptive temperature parameters, the final attention weight distribution is achieved within a unified framework.

[0181] S308: Repeat S301 to S307 to obtain the spatiotemporal coordinates of each sampling point and the attention weights of each level of gear as the multi-level gear weight distribution.

[0182] IV. Spatiotemporal Feature Representation

[0183] This invention uses feature volumes to represent scenes, where each feature volume corresponds to a gear level and weight. Specifically, this representation uses a set of gears to dynamically weight multiple spatiotemporal planes to handle dynamic changes between keyframes. This multi-gear design aims to decouple scene dynamics from different motion velocities to different layers, enabling the model to learn and represent complex dynamic processes more effectively.

[0184] Specifically, the spatiotemporal feature representation of the sampling points is calculated through the following steps;

[0185] S501: For sampling points in keyframes, obtain their spatiotemporal coordinates (x, t) and the attention weights of each level of gear, i.e., the multi-level gear weight distribution.

[0186] S502: Calculate the eigenvectors of the spatiotemporal coordinates (x, t) at each gear level;

[0187] The formula is expressed as:

[0188]

[0189] In the formula: Represents the spacetime coordinates (x, t) in eigenvectors; This represents the spatiotemporal coordinates (x, t) calculated by the gear attention mechanism in [the context of the image / image]. Attention weights; B j (·) indicates that h j (·,·)and The product of is mapped to a linear transformation of the eigenvectors; h j (·,·) is a vector-valued function with an output dimension of M; It is each Unique spatiotemporal embedding is used to learn dynamic patterns at specific velocity levels; x = (x, y, z) represents the three-dimensional coordinates of the sampling point in space;

[0190] S503: The spatiotemporal coordinates (x, t) are weighted and summed in the feature vectors of each gear level to obtain the spatiotemporal feature representation f(x, t) of the corresponding sampling point;

[0191] The final feature vector f(x, t) no longer comes from a single gear level, but is generated by applying these attention weights to the feature vectors of all gear levels. We obtain the result by weighted summation. The formula is expressed as:

[0192]

[0193] Where: N gear This indicates the number of gear levels.

[0194] In this way, the present invention can intelligently fuse features from different motion scales. This makes the model more robust and smooth when dealing with complex dynamics and transition regions between gears.

[0195] V. NeRF Reconstruction and Rendering

[0196] In practice, the fused features f(x, t) of the sampling points and the viewpoint direction are input into a small MLP to predict the density and color values ​​of the sampling points. Then, based on the density and color values ​​of each sampling point, the rendering colors of each ray are generated, and finally, the rendering image corresponding to the training viewpoint is generated.

[0197] VI. Loss Function

[0198] In practice, the training process is guided by multiple loss functions, which collectively optimize the geometry and appearance of the scene representation. The total loss function... It can be represented as:

[0199]

[0200] In the formula: This indicates total variation regularization; Indicates L1 regularization; λ represents the reconstruction loss consisting of two parts; TV and λ L1 These are hyperparameters used to balance the weights of various losses in the total loss.

[0201] in:

[0202]

[0203] In the formula: r represents a ray emitted from the center of the camera along the pixel direction; C represents the set of all rays used during training; C(r) represents the color values ​​of the rendered image; C gt (r) represents the color value of the real image; F SAM (r) represents the semantic features extracted from the rendered image by the pre-trained SAM model; This represents the semantic features extracted from real images using a pre-trained SAM model; the first term in the loss function is the color loss, used to quantize the color value C(r) of the rendered image compared to the color value C of the real RGB image. gt The difference between (r) is the second term, which is the semantic feature loss, achieved by minimizing the F extracted from the rendered view. SAM (r) and the true features obtained from the SAM encoder The mean square error between them is used to monitor the process.

[0204]

[0205] In the formula: The set of all spatial feature planes; H and W represent the height and width of the video frame, respectively; P u,v This represents the eigenvalue of the characteristic plane at coordinates (u, v).

[0206]

[0207] Where: Ω σ T represents the set of all learnable parameters that constitute density σ; T represents a specific feature plane or vector in the set; T i N represents the value of the i-th element in the tensor. σ This represents the total number of parameters and is used for normalization.

[0208] VII. Experimental Instructions

[0209] 1. Experimental Data Set and Evaluation Metrics

[0210] This experiment validated the algorithm's effectiveness on the Google Immersive dataset, which contains videos captured simultaneously by an array of 46 fisheye cameras. These scenes are highly challenging, featuring a variety of complex visual elements such as flames, reflective surfaces, and translucent objects.

[0211] This experiment uses PSNR, SSIM, LPIPS, and training time as evaluation metrics: PSNR is Peak Signal-to-Noise Ratio, mainly used to evaluate the effect of the reconstruction algorithm. The higher the PSNR value, the more similar the two images are and the smaller the quality loss; SSIM is Structural Similarity Index, used to measure the similarity between two images in brightness, contrast, and structure; a high SSIM value indicates that the reconstructed image is more similar to the original image in structure and detail.

[0212] 2. Experimental Implementation Details

[0213] This experiment was implemented using PyTorch, and all experiments were performed on a single NVIDIA RTX 4090 GPU with 24GB of memory. During processing, each input video was segmented into 50-frame segments. The gear attention mechanism has 256 hidden dimensions, 6 depths, and skip connections in layer 3. Training was performed using the Adam optimizer with an initial learning rate of 0.02.

[0214] 3. Experimental Comparison

[0215] On the Google Immersive dataset, this invention was evaluated against HyperReel and Gear-NeRF, and experiments were conducted on six sequences: Flames, Truck, Horse, Car, Welder, and Exhibit. The viewpoint of the central camera was preserved for testing in these sequences. Quantitative comparison results are shown in Tables 1 and 2. This invention outperforms almost all other methods in both individual scenes and on average, achieving a 1.25 dB improvement in PSNR compared to GearNeRF.

[0216] Table 1 shows a quantitative comparison of different methods for each scenario on the Google Immersive dataset. The best results are marked in bold, and the second-best results are marked in underline.

[0217]

[0218] Table 2 shows the average quantitative results of different methods on the Google Immersive dataset. The best results are marked in bold, and the second-best results are marked in underline.

[0219]

[0220] 4. Ablation test

[0221] To verify the effectiveness of the innovation proposed in this invention, ablation experiments were conducted on the Google Immersive dataset, and the results are summarized in Table 3.

[0222] The best results of the ablation studies for the proposed modules in Table 3 are marked in bold, and the second-best results are marked in underline.

[0223]

[0224] As shown in Table 3, the complete model including the gear attention mechanism outperforms the model without this module, achieving a PSNR of 29.88 dB, compared to 28.77 dB for the model without the gear attention mechanism. The gear attention mechanism enables the model to intelligently fuse features at different motion scales, thereby improving rendering quality. Ablation studies comparing models with adaptive keyframe selection to those without this mechanism demonstrate the advantages of this invention. The optimal results are achieved when the adaptive keyframe selection mechanism is combined with the gear attention mechanism, achieving a PSNR of 29.88 dB, an SSIM of 0.888, and an LPIPS of 0.100. This indicates that the strategy of selecting keyframes based on scene dynamics and semantic dynamics is more effective because it focuses the model's capabilities on the most challenging moments in the video.

[0225] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit the technical solutions. Those skilled in the art should understand that any modifications or equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention should be covered within the scope of the claims of the present invention.

Claims

1. A 3D dynamic scene reconstruction method based on adaptive keyframes and gear attention, characterized in that, include: S1: Obtain the multi-view video sequence and the corresponding camera intrinsic and extrinsic parameters and timestamps; S2: For each video frame in the video sequence, emit rays based on the camera's intrinsic and extrinsic parameters and sample multiple points along each ray in three-dimensional space. Record the spatial position, timestamp, and viewing direction of each sampling point to generate a set of sampling points. S3: Extract the location and time features of the spatiotemporal coordinates of each sampling point in the sampling point set through spatial encoder and temporal encoder, and combine the gating layer and self-attention mechanism to perform feature fusion to generate a multi-level gear weight distribution of the spatiotemporal coordinates of each sampling point; S4: For the training viewpoint, the importance score of each video frame is calculated using the SAM model and optical flow algorithm, and the corresponding training viewpoint adaptive keyframe set is obtained through an iterative strategy; the sampling points and rays corresponding to each keyframe in the keyframe set of the corresponding training viewpoint are used as the input of the scene reconstruction model. S5: For each sampling point in the keyframe, calculate the spatiotemporal feature representation based on the multi-level gear weight distribution of its corresponding spatiotemporal coordinates; S6: For each sampling point in the keyframe, predict the density and color values ​​based on its corresponding spatiotemporal feature representation and viewpoint direction, and generate the rendering color of each ray based on the density and color values ​​of each sampling point, and finally generate the rendering image corresponding to the training viewpoint. S7: Calculate the loss function based on the rendered image and the corresponding real image, and optimize the parameters of the scene reconstruction model; S8: Repeat steps S4 to S7 to iteratively train the scene reconstruction model until the model converges or reaches the preset number of iterations to obtain the trained scene reconstruction model.

2. The 3D dynamic scene reconstruction method based on adaptive keyframes and gear attention as described in claim 1, characterized in that: In step S1, time synchronization and scene calibration are performed on the video frames in the multi-view video sequence.

3. The 3D dynamic scene reconstruction method based on adaptive keyframes and gear attention as described in claim 1, characterized in that: In step S3, the multi-level gear weight distribution of the spatiotemporal coordinates of each sampling point is generated through the following steps: S301: Select a sampling point from the set of sampling points and obtain its spatiotemporal coordinates (x,t), where x represents the spatial location and t represents the timestamp; S302: Via space encoder MLP s Extracting the positional features f of spatiotemporal coordinates (x,t) s ; The formula is expressed as: f s =MLP s (x); S303: Via time encoder MLP t Extracting the temporal features f of spatiotemporal coordinates (x,t) t ; The formula is expressed as: f t =MLP t (t); S304: The positional features of the spatiotemporal coordinates (x, t) f s and time features f t Connect them into a unified spatiotemporal feature vector f st =[f s f t The gate vector g is obtained by adaptive fusion through a gating layer. The formula is expressed as: g=σ(W g f st +b g ); In the formula: [·, ·] denotes the feature connection operation; W g and b g The parameters represent the learnable parameters of the gated layer; σ(·) is the sigmoid activation function; S305: Using a self-attention mechanism, the spatiotemporal feature vector f of the spatiotemporal coordinates (x,t) is... st Projected onto projection matrix W Q W K W V Generate the query matrix Q, the key matrix K, and the value matrix V; The formula is expressed as: Q=W Q f st ,K=W K f st ,V=W V f st ; S306: Element-wise multiplication and softmax normalization are performed on the query matrix Q and key matrix K of the spatiotemporal coordinates (x, t) to obtain the self-attention weight α. Then, element-wise multiplication is performed on the self-attention weight α, the value matrix V, and the gating vector g to obtain the attention enhancement feature f. g ; The formula is expressed as: α = softmax(Q⊙K); f g =g⊙α⊙V; S307: Attention-enhanced features f based on spatiotemporal coordinates (x, t) through multi-layer nonlinear transformation g Assign gear weights to obtain the spatiotemporal coordinates (x, t) in Attention weights The formula is expressed as: In the formula: Let represent a multilayer nonlinear transform network; λ is a learnable residual weight parameter; u is a uniformly distributed vector; β is a dynamic weight balancing parameter vector; ψ(γ) is an adaptive temperature function. S308: Repeat S301 to S307 to obtain the spatiotemporal coordinates of each sampling point and the attention weights of each level of gear as the multi-level gear weight distribution.

4. The 3D dynamic scene reconstruction method based on adaptive keyframes and gear attention as described in claim 1, characterized in that: In step S4, the importance score of the video frame is calculated through the following steps: S401: Extract semantic features of video frames using a pre-trained SAM model. S402: Semantic features based on video frames Calculate the semantic importance score S based on SAM. t ; The formula is expressed as: S t =a V ·V t +a A ·A t +a E ·E t ; In the formula: α V α A α E V represents the weighting coefficient; t Representing semantic features The variance; A t Indicates the use of semantic features The L2 norm is used to quantify visual saliency, E t Indicates edge density; S403: Calculate the optical flow-based motion dynamics score M of video frames using an optical flow algorithm. t ; The formula is expressed as: In the formula: H and W represent the height and width of the video frame, respectively; (i, j) represent the pixel coordinates in the video frame; u(i, j) and v(i, j) represent the horizontal and vertical components of the optical flow field of the video frame. S404: Semantic importance score based on SAM for video frames. t and the motion dynamics fraction M based on optical flow t Calculate its importance score I t ; The formula is expressed as: I t =α·M t +(1-α)·S t ; In the formula: α∈(0,1) represents the balance factor.

5. The 3D dynamic scene reconstruction method based on adaptive keyframes and gear attention as described in claim 4, characterized in that: In step S4, a set of keyframes is generated through the following steps: S411: For a video sequence containing T frames, select the first and last video frames of the video sequence as keyframes and add them to the keyframe set. S412: Select the remaining K-2 keyframes using an iterative strategy. The specific steps include: S4121: Construct the initial keyframe set Initialize the importance score of each video frame to I′. t =I t ; S4122: Update the importance scores of the remaining video frames based on the keyframes already selected in the keyframe set; The formula is expressed as: In the formula: θ=max(1,T / (2K)) is the time window parameter; τ j It is the time index of the j-th selected keyframe in the keyframe set; Π represents the product; t is the timestamp; S4123: Select the most important video frame τ from the remaining video frames. k Add it to the keyframe set as a keyframe; The formula is expressed as: S4124: Repeat steps S4122 and S4123, iteratively selecting K-2 keyframes to add to the keyframe set.

6. The 3D dynamic scene reconstruction method based on adaptive keyframes and gear attention as described in claim 1, characterized in that: In step S5, the spatiotemporal feature representation of the sampling points is calculated through the following steps; S501: For sampling points in keyframes, obtain their spatiotemporal coordinates (x, t) and the attention weights of each level of gear, i.e., the multi-level gear weight distribution. S502: Calculate the eigenvectors of the spatiotemporal coordinates (x, t) at each gear level; The formula is expressed as: In the formula: Represents the spacetime coordinates (x, t) in eigenvectors; This represents the spatiotemporal coordinates (x, t) calculated by the gear attention mechanism in [the context of the image / image]. Attention weights; B j (·) indicates that h j (·,·)and The product of is mapped to a linear transformation of the eigenvectors; h j (·,·) is a vector-valued function with an output dimension of M; It is each Unique spatiotemporal embedding; x = (x, y, z) represents the three-dimensional coordinates of the sampling point in space; S503: The spatiotemporal coordinates (x, t) are weighted and summed in the feature vectors of each gear level to obtain the spatiotemporal feature representation f(x, t) of the corresponding sampling point; The formula is expressed as: Where: N gear This indicates the number of gear levels.

7. The 3D dynamic scene reconstruction method based on adaptive keyframes and gear attention as described in claim 1, characterized in that: In step S6, the fusion feature f(x, t) of the sampling point and the view direction are input into the MLP together to predict the density value and color value of the sampling point.

8. The 3D dynamic scene reconstruction method based on adaptive keyframes and gear attention as described in claim 1, characterized in that: In step S7, the loss function is calculated using the following formula: In the formula: This represents the total variation regularization loss; Indicates the L1 regularization loss; λ represents the reconstruction loss consisting of two parts; TV and λ L1 It's a hyperparameter; in: In the formula: r represents a ray emitted from the center of the camera along the pixel direction; C represents the set of all rays used during training; C(r) represents the color values ​​of the rendered image; C gt (r) represents the color value of the real image; F SAM (r) represents the semantic features extracted from the rendered image by the pre-trained SAM model; This represents the semantic features extracted from real images using a pre-trained SAM model.

9. The 3D dynamic scene reconstruction method based on adaptive keyframes and gear attention as described in claim 8, characterized in that: In step S7, the total variation regularization loss is calculated using the following formula: In the formula: The set of all spatial feature planes; H and W represent the height and width of the video frame, respectively; P u,v This represents the eigenvalue of the characteristic plane at coordinates (u,v).

10. The 3D dynamic scene reconstruction method based on adaptive keyframes and gear attention as described in claim 8, characterized in that: In step S7, the L1 regularization loss is calculated using the following formula: Where: Ω σ T represents the set of all learnable parameters that constitute density σ; T represents a specific feature plane or vector in the set; T i N represents the value of the i-th element in the tensor. σ This indicates the total number of parameters.

Images