A device and method for synthesizing new perspectives of asynchronous videos based on motion consistency

Abstract

This invention provides a device and method for synthesizing new perspectives of asynchronous videos based on motion consistency. The device comprises a neural scene modeling module for constructing a neural representation of the dynamic scene; a camera time offset estimation module for introducing and optimizing learnable time offset parameters for each camera; and a global and local motion consistency supervision module for calculating the three-dimensional scene flow of the neural scene from each camera's perspective and projecting it into two dimensions. This flow is then compared with the pre-computed two-dimensional optical flow to construct a temporal alignment supervision signal. This invention achieves precise temporal alignment of asynchronous videos and high-quality dynamic new perspective image synthesis without the need for external synchronization equipment.

Description

Technical Field

[0001] The present invention belongs to the technical field of visual signal processing, and in particular relates to a device and method for synthesizing new perspectives of asynchronous videos based on motion consistency. Background Art

[0002] With the development of neural representation technology, new perspective synthesis methods based on neural networks have become a research hotspot in computer vision and graphics. Implicit neural representations such as NeRF achieve high-quality volume rendering by inputting 3D spatial position and viewing direction into a multi-layer perceptron. They are widely used for 3D reconstruction and new perspective synthesis of static and dynamic scenes. In recent years, explicit visual signal representations such as 3D Gaussian Splatting have been proposed. By explicitly modeling the position and properties of Gaussian points in the scene, they further improve rendering efficiency and editability.

[0003] However, most existing methods assume that multi-view cameras capture video sequences synchronously, an assumption often difficult to meet in the real world. Due to equipment cost, deployment difficulties, or limited acquisition conditions, asynchronous multi-view videos are a common data source for dynamic scene modeling. This temporal misalignment leads to distortions such as blurring and artifacts in spatiotemporal modeling, severely impacting the quality of the final reconstruction.

[0004] Some work has attempted to alleviate this problem by introducing temporal interpolation, pose-based alignment, or directly learning temporal offset parameters. However, these approaches generally rely on scene priors or labeled data, making them difficult to generalize to general scenarios. Furthermore, existing methods focus primarily on local temporal consistency and lack the ability to model global dynamic consistency in multi-view scenes. Summary of the Invention

[0005] Purpose of the Invention: This invention addresses the shortcomings of existing technologies by providing a device and method for synthesizing new perspectives in asynchronous videos based on motion consistency. This method achieves spatiotemporally consistent dynamic reconstruction and new perspective synthesis without the need for external synchronization equipment or manual annotation. Another purpose of the invention is to provide methods and applications of the aforementioned device.

[0006] The present invention first provides a new perspective synthesis device for asynchronous videos based on motion consistency, which includes a neural scene modeling module, a camera time offset estimation module, and a global and local motion consistency supervision module;

[0007] The neural scene modeling module is used to construct a neural representation of dynamic scenes;

[0008] The camera time offset estimation module is used to introduce a learnable time offset parameter for each camera and jointly optimize it with the scene representation;

[0009] The global and local motion consistency supervision module is used to calculate the three-dimensional scene flow of the neural scene and project it into two dimensions, compare it with the pre-calculated two-dimensional optical flow, and construct a time alignment supervision signal.

[0010] The expression of the neural representation is:

[0011] ,

[0012] Among them, R is the dynamic scene reconstruction operator, which is used to generate the neural representation of the dynamic scene; the dynamic scene reconstruction operator The input includes multi-camera image sequences and corresponding camera projection matrix sets; Indicates the i-th camera at timestamp Captured image frames, Represents the projection matrix corresponding to the i-th camera, Represents the total number of cameras; the output of the dynamic scene reconstruction operator R is the neural representation of the dynamic scene at any time stamp t .

[0013] The camera time offset estimation module is used to set an independent learnable time offset parameter for each camera , Indicates the time offset of the i-th camera. During the training process, it participates in back-propagation and is jointly optimized with the scene representation through the loss function to correct the time asynchrony between multi-camera input videos.

[0014] The neural representation is obtained by combining the learnable time offset parameters The joint optimization expression is:

[0015] ,

[0016] in, represents the neural representation of the optimal dynamic scene optimized at time t, represents the optimal time offset parameter set obtained by optimization; is the projection operator, Represents the neural representation of the dynamic scene at time t Through the projection matrix of the corresponding i-th camera The projected image frame, is the image frame corresponding to the i-th camera after time offset.

[0017] The global and local motion consistency supervision module is used to construct the alignment loss term between the two-dimensional projected scene flow of the neural scene and the pre-computed optical flow, and introduces the flow loss function , the expression is:

[0018] ,

[0019] in, Representation flow consistency supervision loss, which is used to measure the difference between the two-dimensional projected scene flow obtained based on the dynamic scene representation and the pre-computed optical flow, represents the optical flow precomputed from the image frame, Representing neural representations of dynamic scenes over time The two-dimensional projected scene flow obtained by back-projection.

[0020] The two-dimensional projected scene flow and the optical flow precomputed from the image frames Obtained through the following operations:

[0021] ,

[0022] ,

[0023] in, Indicates timestamp Neural representation of dynamic scenes Apply the 3D scene flow extraction operator, Represents the image frame through the pre-calculated optical flow estimation network The calculated 2D optical flow.

[0024] The global and local motion consistency supervision module is also used to introduce a binary confidence mask to perform weighted supervision on the difference between the two-dimensional projected scene flow and the pre-calculated optical flow, and introduce a flow loss function of the binary confidence mask The expression is:

[0025] ,

[0026] in, represents pixel-wise element-wise multiplication, is the binary confidence mask constructed under the i-th camera view.

[0027] The joint optimization objective of the neural representation of the dynamic scene and the learnable time shift parameter includes the image reconstruction loss term, the optical flow consistency supervision term, and the time shift regularization term. The expression of the optimization objective is:

[0028] ,

[0029] in, Represents the image reconstruction loss, which is used to measure the pixel difference between the rendered image and the real image; The L1 regularization term representing the time offset parameter is used to prevent the offset from being too large; 、 They represent the weight hyperparameters of the flow loss function and the weight hyperparameters of the time offset parameter L1 regularization term, respectively, and control the influence weight of each loss term on the overall optimization objective.

[0030] The present invention also provides a method for synthesizing new perspectives of asynchronous videos based on motion consistency using the device, comprising: constructing a neural representation of a dynamic scene; , the input is a multi-camera image sequence and the corresponding camera projection matrix set, and the output is a neural representation of the dynamic scene at any time stamp; a learnable time offset parameter is set for each camera , and jointly optimized with the scene representation during training; Calculate the 3D scene flow from the scene representation and project it into a 2D projected scene flow , and precomputed optical flow In contrast, a binary confidence mask is constructed based on the optical flow amplitude. , calculate the motion consistency supervision loss; jointly optimize the image reconstruction loss, optical flow consistency loss and time regularization term to achieve spatiotemporal consistent dynamic reconstruction and new perspective synthesis.

[0031] The method is applicable to six-dimensional dynamic neural scene modeling tasks; the neural representation of the dynamic scene is an implicit neural representation (such as Kplanes) or an explicit representation composed of three-dimensional Gaussians (such as 4DGS, EDGS). The method jointly optimizes the neural representation of the dynamic scene and the learnable time offset parameters to achieve spatiotemporally consistent dynamic reconstruction and new perspective synthesis.

[0032] Beneficial effects: (1) The present invention proposes a new perspective synthesis device for asynchronous videos based on motion consistency, which achieves efficient and automatic cross-camera time alignment and unified modeling of dynamic scenes without the need for external synchronization hardware or manual labeling.

[0033] (2) The present invention proposes a global and local motion consistency supervision method that combines three-dimensional scene flow and two-dimensional optical flow. The alignment between the three-dimensional scene flow and the two-dimensional optical flow is introduced as a supervision signal, which effectively alleviates the spatial reconstruction error caused by temporal inconsistency and improves the spatiotemporal consistency of new perspective rendering.

[0034] (3) Our method is compatible with a variety of neural representations, including implicit neural representations (e.g., Kplanes) and explicit representations composed of three-dimensional Gaussians (e.g., 4DGS and EDGS). It exhibits excellent versatility and scalability, achieving excellent new perspective synthesis results on multiple representative datasets, surpassing previous methods that only optimized temporal offset parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more apparent.

[0036] Figure 1 Schematic diagram of a device for implementing the method of the present invention;

[0037] Figure 2 This is a comparison chart of the results of the 4DGS representation framework in the six-dimensional dynamic neural radiation field reconstruction task of the present invention;

[0038] Figure 3 This is a comparison chart of the results of the EDGS representation framework in the six-dimensional dynamic neural radiation field reconstruction task of the present invention;

[0039] Figure 4 This is a comparison chart of the results of the Kplanes representation framework in the six-dimensional dynamic neural radiation field reconstruction task of the present invention. DETAILED DESCRIPTION

[0040] Reference Figure 1 As shown, an embodiment of the present invention provides a new perspective synthesis device for asynchronous videos based on motion consistency, including a neural scene modeling module, a camera time offset estimation module, and a global and local motion consistency supervision module. Among them, the neural scene modeling module is used to construct a neural representation of the dynamic scene; the camera time offset estimation module is used to introduce a learnable time offset parameter for each camera and jointly optimize it with the scene representation; the global and local motion consistency supervision module is used to calculate the three-dimensional scene flow of the neural scene and project it into two dimensions, compare it with the pre-calculated two-dimensional optical flow, and construct a time alignment supervision signal.

[0041] Among them, the mathematical expressions used by the global and local motion consistency supervision modules are:

[0042] ,

[0043] The temporal alignment of multi-view cameras is achieved by minimizing the difference between the pre-computed optical flow and the projected estimated optical flow. This method ensures that the supervisory signal originates primarily from high-confidence regions with sufficient motion intensity, effectively suppressing supervisory error interference from low-motion or noisy regions and significantly improving the stability and robustness of dynamic scene reconstruction. Compared to traditional methods that rely solely on image reconstruction errors, this method effectively reduces geometric blur and motion artifacts in asynchronous data by introducing global and local motion consistency losses, achieving a more accurate spatiotemporally consistent representation of dynamic scenes with complex motion characteristics.

[0044] This embodiment also provides a method for synthesizing new perspectives of asynchronous multi-view videos using the device, specifically comprising the following steps: constructing a neural scene modeling function , the input is a multi-camera image sequence and the corresponding camera projection matrix set, and the output is a neural representation of the dynamic scene at any time stamp; a learnable time offset parameter is set for each camera , and jointly optimized with the scene representation during training; Calculate the 3D scene flow from the scene representation and project it into a 2D projected scene flow , and precomputed optical flow Contrast, build confidence mask based on optical flow amplitude , calculate the motion consistency supervision loss; jointly optimize the image reconstruction loss, optical flow consistency loss and time regularization term to achieve spatiotemporal consistent dynamic reconstruction and new perspective synthesis.

[0045] The specific implementation steps of the neural scene modeling module are as follows:

[0046] The neural scene modeling module is used to construct a neural scene representation, and the expression of the neural scene representation is:

[0047] ,

[0048] Among them, R is a dynamic scene reconstruction operator, and the input is a multi-camera image sequence and the corresponding camera projection matrix The output is a collection of Neural representation of dynamic scenes . In this embodiment, the dynamic scene representation can be specifically implemented as an implicit neural representation or an explicit representation composed of three-dimensional Gaussians. The implicit neural representation is implemented using the neural radiation field Kplanes based on multi-plane decomposition, and high-dimensional mapping of the scene space position and observation direction is performed through multi-plane decomposition and multi-layer perceptron; the three-dimensional Gaussian explicit representation is implemented using 4DGS and EDGS, among which 4DGS uses a learnable deformation field to model the time change of the Gaussian center, and EDGS directly expresses the Gaussian center as a function of time, thereby realizing explicit modeling of the dynamic process.

[0049] The specific implementation steps of the camera time offset estimation module are as follows:

[0050] For each camera perspective Set an independent, learnable time offset parameter , according to the number of cameras M of the multi-view input video, initialize a length of A list of time offset parameters The initial values ​​of the list are randomly generated using a normal distribution with a mean of -0.001 and a standard deviation of 0.001 to ensure the stability of training and the effectiveness of model parameter optimization. During the training process, the neural representation parameters and the time offset parameters are jointly optimized through gradient backpropagation. The expression of the optimization process is:

[0051] ,

[0052] in, is the projection operator, used to project the neural representation into the camera image space, is the time offset for each camera, is the image frame corresponding to the i-th camera after time offset. The essence of this optimization goal is to achieve accurate time offset calibration by minimizing the difference between the neural representation projection image and the real image.

[0053] The specific implementation steps of the global and local motion consistency supervision module are as follows:

[0054] Step 1: Use the pre-selected two-dimensional optical flow estimation algorithm to calculate the pre-calculated two-dimensional optical flow between adjacent frames of each camera view image sequence :

[0055] ,

[0056] in, To pre-calculate the optical flow algorithm calculation operator, represents the image captured by the i-th camera at time t, Indicates the i-th camera at adjacent time frames Captured image.

[0057] Step 2: Dynamic Neural Scene Representation , obtain the 3D scene flow through the 3D scene flow extraction operator , representing the spatial point in the scene at time to Three-dimensional motion information between;

[0058] Step 3: For the i-th camera, use the projection matrix 3D scene flow Projected onto the two-dimensional image plane to obtain the estimated two-dimensional projected scene flow :

[0059] ,

[0060] Step 4: Based on the pre-calculated optical flow Construct optical flow confidence mask. Calculate precomputed optical flow The amplitude of , using the set threshold to construct a binary mask :

[0061] ,

[0062] in, is the set optical flow amplitude threshold, the amplitude is lower than the threshold Pixels with are considered as areas with insufficient or unreliable motion information and are therefore excluded from the calculation of the supervision loss to avoid the impact of noise on training;

[0063] Step 5: Use confidence mask Construct a weighted motion consistency supervision loss function to calculate the consistency loss between the pre-computed optical flow and the estimated optical flow:

[0064] ,

[0065] in, represents pixel-wise element-wise multiplication, and the L1 norm ensures that fine errors in edges and non-rigid regions are preserved.

[0066] By minimizing the above motion consistency loss function The method described in the present invention can significantly improve the temporal alignment accuracy between multi-view input video sequences. Specifically, the proposed global and local motion consistency supervision mechanism can calculate the 3D scene flow from the scene representation and project it into a 2D projected scene flow during each round of iterative optimization of the model. This is then combined with the 2D optical flow calculation loss independently calculated from each camera's single-view image sequence to perform motion consistency supervision.

[0067] The supervision strategy is based on the three-dimensional scene flow extracted from the neural representation, combines the camera projection matrix to obtain the two-dimensional projected scene flow, and compares it with the pre-calculated optical flow pixel by pixel, so as to have spatial-temporal features that are consistent across perspectives. Compared with traditional methods that only rely on image reconstruction error loss or methods that only optimize time offsets, the present invention introduces a clear motion consistency supervision term, so that the model can more effectively capture the slight time offset differences caused by time-asynchronous sampling in multi-camera inputs, and thus correct them more accurately. In addition, the introduced optical flow confidence mask mechanism filters high-confidence areas based on the amplitude of the pre-calculated optical flow, thereby further ensuring the quality and reliability of the supervision signal. It ensures that the supervision signal mainly comes from pixel areas with real motion characteristics, thereby effectively preventing the model from being interfered with by erroneous signals.

[0068] The extraction and projection process of the 3D scene flow in the global and local motion consistency supervision module can be divided into the following two implementation paths according to the different forms of the selected neural scene representation:

[0069] When the implicit neural representation is implemented using the multi-plane encoding neural radiance field Kplanes, the extraction and projection steps of the 3D scene flow are:

[0070] First, the feature vector obtained by six-plane encoding is input into the multi-layer perceptron (MLP) of the scene flow decoder, and the three-dimensional motion vector is decoded as the three-dimensional scene flow under implicit representation; then, combined with the camera projection matrix , project the 3D motion vector onto the 2D image plane to obtain the estimated 2D projected scene flow.

[0071] When using 3D Gaussian display representation 4DGS, EDGS implementation, the extraction and projection steps of the 3D scene flow are as follows: for 4DGS representation, the center position difference of the Gaussian basis element of the 2D image plane of the temporally adjacent frames is obtained by Gaussian splash rendering as the estimated 2D projected scene flow; for EDGS representation, the center position difference of the Gaussian basis element in the adjacent time frames is obtained by modeling the time function of the Gaussian center position, and then, the camera projection matrix is ​​used to calculate the center position difference of the Gaussian basis element. Project it onto the two-dimensional image plane to obtain the estimated two-dimensional projected scene flow.

[0072] The device and method of the present invention are applicable to the task of reconstructing a six-dimensional dynamic neural radiation field. In this embodiment, the device and method are specifically applied to the task of reconstructing a six-dimensional dynamic neural radiation field.

[0073] This embodiment describes a device for synthesizing new perspectives from asynchronous videos based on motion consistency. In the six-dimensional dynamic neural radiance field reconstruction task, the baseline methods used for comparison are: the original baseline model and a dynamic new perspective synthesis method that only incorporates temporal offset. Specific implementation details vary depending on the neural radiance field representation framework. Therefore, the following describes specific implementations for different neural radiance field representation frameworks, including 4DGS, EDGS, and Kplanes.

[0074] For the six-dimensional dynamic neural radiation field reconstruction task under the 4DGS representation framework, all comparison methods use default settings. This representation framework uses a deformation field to model the three-dimensional position center of the Gaussian primitive, and uses the current timestamp as input to predict the center offset to achieve dynamic scene representation. In this embodiment, a dynamic scene representation is first constructed through a neural scene modeling module. The input is an image sequence under multiple camera perspectives and the corresponding camera projection matrix, and the output is a dynamic scene neural representation at an arbitrary timestamp. An independent learnable time offset parameter is introduced for each camera perspective. Its initial value obeys a normal distribution with a mean of -0.001 and a standard deviation of 0.001. It is used to correct the time alignment error between multi-view camera video inputs and is fixed after 6k rounds of training optimization. For the real scene dataset, the number of pre-training iterations is 3k and the total number of training iterations is 15k; for the virtual synthetic scene dataset, the number of pre-training iterations is 3k and the total number of training iterations is 20k. During training, the center position differences of Gaussian primitives in the 2D image plane of temporally adjacent frames are obtained through Gaussian splatter rendering. This is used as the estimated 2D projected scene flow. This is then compared with the pre-computed optical flow obtained using the VideoFlow optical flow algorithm. A motion consistency supervision loss is then calculated using a confidence mask constructed based on the pre-computed optical flow amplitudes. Finally, by combining the image reconstruction loss with the motion consistency supervision loss, spatiotemporally consistent dynamic reconstruction and novel perspective synthesis are achieved. Training and testing are performed on the Dynamic Blender Datasets, a virtual synthetic dataset, and the Plenoptic Video Datasets, a real-world dataset. Figure 2 Comparative results are presented. The method of the present invention demonstrates advantages in reconstructing rapidly moving areas, such as the ears and mouth of a deer, the reflective spray gun, and flames. Compared to the original baseline model based on the 4DGS representation framework and the dynamic new perspective synthesis method that only incorporates a time offset, significant blurring and artifacts are observed in moving areas due to the asynchrony of multiple perspectives. The method of the present invention effectively reduces the blurring and artifacts caused by time asynchrony, achieving more realistic reconstruction of dynamic scenes.

[0075] For the six-dimensional dynamic neural radiation field reconstruction task under the EDGS representation framework, all comparison methods use the default settings. This representation framework realizes dynamic scene representation by modeling the center position of each Gaussian primitive as a function of time. In this embodiment, the dynamic scene representation is first constructed by the neural scene modeling module. The input is an image sequence under multiple camera perspectives and its corresponding camera projection matrix, and the output is a dynamic scene neural representation at an arbitrary timestamp. A learnable time offset parameter is introduced for each camera, and its initial value obeys a normal distribution with a mean of -0.001 and a standard deviation of 0.001. It is used to correct the time alignment error between multi-view camera video inputs and is fixed after 6k rounds of training optimization. For the real scene dataset, the number of pre-training iterations is 3k and the total number of training iterations is 20k; for the virtual synthetic scene dataset, the number of pre-training iterations is 3k and the total number of training iterations is 30k. During the training process, the center position difference of the Gaussian primitives in adjacent time frames is obtained by modeling the time function of the Gaussian center position, and then the camera projection matrix is ​​used to obtain the center position difference of the Gaussian primitives in adjacent time frames. This is projected onto a 2D image plane to obtain an estimated 2D projected scene flow. This is then compared with the pre-computed optical flow obtained using the RAFT optical flow algorithm. A motion consistency supervision loss is then calculated using a confidence mask constructed based on the pre-computed optical flow magnitude. Finally, by combining the image reconstruction loss with the motion consistency supervision loss, a temporally and spatially consistent dynamic reconstruction and novel perspective synthesis are achieved. Training and testing are performed on the Dynamic Blender Datasets, a virtual synthetic dataset, and the Plenoptic Video Datasets, a real-world dataset. Figure 3 Comparison results are presented. The present invention demonstrates its advantages in reconstruction of moving and weakly textured areas. Examples include the curved texture and contour edges of a pendulum, glass surface areas, and spring surface structures and textures. The original baseline model of the EDGS representation framework and the dynamic new perspective synthesis method that only adds time offset exhibit blurring or artifacts. However, the present invention significantly reduces these artifacts and blurring by introducing motion consistency supervision, thereby improving the spatial consistency of dynamic scene reconstruction.

[0076] For the six-dimensional dynamic neural radiation field reconstruction task under the Kplanes representation framework, all comparison methods use the default settings. The framework adopts an implicit neural representation method of multi-plane decomposition. By projecting the three-dimensional spatial position and time information onto six two-dimensional planes in orthogonal directions, the voxel features are obtained and then input into the multi-layer perceptron (MLP) for decoding to realize dynamic scene representation. In this embodiment, the decoding network used is a one-layer fully connected neural network containing 64 hidden units. First, a dynamic scene representation is constructed through the neural scene modeling module. A learnable time offset parameter is introduced for each input camera. Its initial value obeys a normal distribution with a mean of -0.001 and a standard deviation of 0.001. It is used to correct the time alignment error between multi-view camera video inputs and is fixed after 50k rounds of optimization. For the real scene dataset and the virtual synthetic scene dataset, the total number of training iterations is 90k. During the training process, the feature vector obtained by the six-plane encoding is input as the multi-layer perceptron (MLP) of the scene stream decoder, and the three-dimensional motion vector is decoded as the three-dimensional scene stream under the implicit representation; then, combined with the camera projection matrix The 3D motion vectors are projected onto the 2D image plane to obtain an estimated 2D projected scene flow. This is then compared with the pre-computed optical flow obtained using the VideoFlow optical flow algorithm. A motion consistency supervision loss is then calculated using a confidence mask constructed based on the pre-computed optical flow magnitudes. Finally, by combining the image reconstruction loss with the motion consistency supervision loss, spatiotemporally consistent dynamic reconstruction and novel perspective synthesis are achieved. Training and testing are performed on the Dynamic Blender dataset and the PlenopticVideo dataset. Figure 4 The results are compared. This method demonstrates its advantages in reconstructing moving and weakly textured areas. For example, the dog's head and the highlight of the clip exhibit noticeable blurring and artifacts compared to the original baseline model based on the Kplanes representation framework and the dynamic new perspective synthesis method that only adds time offset. However, the global and local motion consistency supervision mechanism introduced in this method effectively improves the accuracy of multi-camera temporal alignment, ensures spatiotemporal consistency in the reconstruction results, and achieves a more realistic reconstruction of dynamic scenes.

[0077] The present invention provides a device and method for synthesizing new perspectives of asynchronous videos based on motion consistency. There are numerous methods and approaches for implementing this technical solution. The above is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art may make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications are also within the scope of protection of the present invention. Any components not specified in this embodiment may be implemented using existing technologies.

Claims

1. A device for synthesizing new perspectives of asynchronous videos based on motion consistency, characterized in that: It includes a neural scene modeling module, a camera time offset estimation module, and a global and local motion consistency supervision module; The neural scene modeling module is used to construct a neural representation of dynamic scenes; The camera time offset estimation module is used to introduce a learnable time offset parameter for each camera and jointly optimize it with the scene representation; The global and local motion consistency supervision module is used to calculate the three-dimensional scene flow of the neural scene and project it into two dimensions, compare it with the pre-calculated two-dimensional optical flow, and construct a time alignment supervision signal; The expression of the neural representation is: ， Among them, R is a dynamic scene reconstruction operator, which is used to generate a neural representation of the dynamic scene; the input of the dynamic scene reconstruction operator R includes a multi-camera image sequence and a corresponding camera projection matrix set; Indicates the i-th camera at timestamp Captured image frames, The output of the dynamic scene reconstruction operator R is the neural representation of the dynamic scene at any time stamp t. ; The camera time offset estimation module is used to set an independent learnable time offset parameter for each camera , Indicates the time offset of the i-th camera. Participate in backpropagation during training and jointly optimize the loss function with the scene representation to correct the time asynchrony between multi-camera input videos; The neural representation is obtained by combining the learnable time offset parameters The joint optimization expression is: ， in, Indicates at time The neural representation of the optimal dynamic scene obtained by optimization, represents the optimal time offset parameter set obtained by optimization; is the projection operator, Indicates the moment Neural representation of dynamic scenes Through the projection matrix of the corresponding i-th camera The projected image frame, is the image frame corresponding to the i-th camera after time offset; The global and local motion consistency supervision module is used to construct the alignment loss term between the two-dimensional projected scene flow of the neural scene and the pre-computed optical flow, and introduces the flow loss function , the expression is: ， in, represents the flow consistency supervision loss, represents the optical flow precomputed from the image frame, Representing neural representations of dynamic scenes over time The two-dimensional projection scene flow obtained by post-projection; The two-dimensional projected scene flow and the optical flow precomputed from the image frames Obtained through the following operations: ， ， in, Represents the neural representation of the dynamic scene at time stamp t Apply the 3D scene flow extraction operator, Represents the image frame through the pre-calculated optical flow estimation network The calculated two-dimensional optical flow; The global and local motion consistency supervision module is also used to introduce a binary confidence mask to perform weighted supervision on the difference between the two-dimensional projected scene flow and the pre-calculated optical flow, and introduce a flow loss function of the binary confidence mask The expression is: ， in, represents pixel-wise element-wise multiplication, is the binary confidence mask constructed under the i-th camera view; The joint optimization objective of the neural representation of the dynamic scene and the learnable time shift parameter includes the image reconstruction loss term, the optical flow consistency supervision term, and the time shift regularization term. The expression of the optimization objective is: ， in, Represents the image reconstruction loss, which is used to measure the pixel difference between the rendered image and the real image; The L1 regularization term representing the time offset parameter is used to prevent the offset from being too large; 、 They represent the weight hyperparameters of the flow loss function and the weight hyperparameters of the time offset parameter L1 regularization term, respectively, and control the influence weight of each loss term on the overall optimization objective.

2. A method for synthesizing new perspectives of asynchronous videos based on motion consistency implemented by the device according to claim 1, characterized in that: include: Building Neural Representations of Dynamic Scenes , the input is a multi-camera image sequence and the corresponding camera projection matrix set, and the output is a neural representation of the dynamic scene at any time stamp; a learnable time offset parameter is set for each camera , and jointly optimized with the scene representation during training; Calculate the 3D scene flow from the scene representation and project it into a 2D projected scene flow , and precomputed optical flow In contrast, a binary confidence mask is constructed based on the optical flow amplitude. , calculate the motion consistency supervision loss; jointly optimize the image reconstruction loss, optical flow consistency loss and time regularization term to achieve spatiotemporal consistent dynamic reconstruction and new perspective synthesis.

3. The method according to claim 2, characterized in that The method is applicable to six-dimensional dynamic neural scene modeling tasks; the neural representation of the dynamic scene is an implicit neural representation, or an explicit representation composed of three-dimensional Gaussians. The method jointly optimizes the neural representation of the dynamic scene and a learnable time offset parameter to achieve spatiotemporally consistent dynamic reconstruction and new perspective synthesis.

Images