Asynchronous video new view angle synthesis device and method based on motion consistency

Abstract

The invention provides an asynchronous video new view angle synthesis device and method based on motion consistency, and the device comprises a neural scene modeling module which is used for constructing neural representation of a dynamic scene; the camera time offset estimation module is used for introducing a learnable time offset parameter for each camera and optimizing the learnable time offset parameter; and the global and local motion consistency supervision module is used for calculating a three-dimensional scene flow of the neural scene under each camera visual angle, projecting the three-dimensional scene flow into a two-dimensional scene flow, comparing the three-dimensional scene flow with a pre-calculated two-dimensional light flow, and constructing a time alignment supervision signal. According to the invention, accurate time alignment and high-quality dynamic new view angle image synthesis of asynchronous videos can be realized under the condition that external synchronization equipment is not needed.

Description

Technical Field

[0001] The present invention belongs to the technical field of visual signal processing, and particularly relates to a new view synthesis device and method for asynchronous videos based on motion consistency. Background Art

[0002] With the development of neural representation technology, new view synthesis methods based on neural networks have become a research hotspot in computer vision and graphics. Implicit neural representations such as NeRF have achieved high-quality volume rendering by inputting three-dimensional spatial positions and viewing directions into a multi-layer perceptron, and are widely used in 3D reconstruction and new view synthesis of static and dynamic scenes. In recent years, explicit visual signal representations such as 3D Gaussian Splatting have been proposed, which further improve the rendering efficiency and editability by explicitly modeling the positions and attributes of Gaussian points in the scene.

[0003] However, most existing methods assume that multi-view cameras synchronously acquire video sequences, which is often difficult to meet in the real world. Due to the cost of shooting equipment, the difficulty of layout, or the limitations of acquisition conditions, asynchronous multi-view videos have become a common data source for dynamic scene modeling. This problem of temporal misalignment leads to distortion phenomena such as blurring and artifacts in spatio-temporal modeling, seriously affecting the final reconstruction quality.

[0004] Some existing works have tried to alleviate this problem by introducing time interpolation, human pose-based alignment, or directly learning time offset parameters, etc., but they generally rely on scene priors or labeled data and are difficult to generalize to general scenes. At the same time, existing methods mostly focus on local temporal consistency and lack the modeling of global dynamic consistency in multi-view scenes. Summary of the Invention

[0005] Object of the Invention: The technical problem to be solved by the present invention is to provide a new view synthesis device and method for asynchronous videos based on motion consistency, which can achieve spatio-temporally consistent dynamic reconstruction and new view synthesis without external synchronization devices or manual annotations. Another object of the present invention is to provide the method and application of the above device.

[0006] The present invention first provides a new view synthesis device for asynchronous videos based on motion consistency, including a neural scene modeling module, a camera time offset estimation module, and a global and local motion consistency supervision module; The neural scene modeling module is used to construct a neural representation of the dynamic scene; The camera time offset estimation module is used to introduce learnable time offset parameters for each camera and jointly optimize them with the scene representation; The global and local motion consistency supervision module is used to calculate the three-dimensional scene flow of the neural scene and project it into two dimensions, compare it with the pre-computed two-dimensional optical flow, and construct a temporal alignment supervision signal.

[0007] The expression of the neural representation is: , where R is the dynamic scene reconstruction operator for generating the neural representation of the dynamic scene; the input of the dynamic scene reconstruction operator includes the image sequences of multiple cameras and the corresponding set of camera projection matrices; represents the image frame captured by the i-th camera at the timestamp , represents the projection matrix corresponding to the i-th camera, represents the total number of cameras; the output of the dynamic scene reconstruction operator R is the neural representation of the dynamic scene at any timestamp t .

[0008] The camera time offset estimation module is used to set an independent learnable time offset parameter for each camera , represents the time offset of the i-th camera, and this parameter participates in backpropagation during training and is jointly optimized with the scene representation through the loss function to correct the temporal asynchrony between the multi-camera input videos.

[0009] The neural representation is obtained through joint optimization with the learnable time offset parameter , and the expression of the joint optimization is: , where, represents the neural representation of the optimal dynamic scene optimized at time t, represents the set of optimal time offset parameters obtained through optimization; is the projection operator, represents projecting the neural representation of the dynamic scene at time t through the projection matrix of the corresponding i-th camera to obtain the image frame, is the image frame corresponding to the i-th camera after time offset.

[0010] The global and local motion consistency supervision module is used to construct an alignment loss term between the two-dimensional projected scene flow and the pre-computed optical flow of the neural scene, and introduce the flow loss function , and the expression is: , where, represents the flow consistency supervision loss, which is used to measure the difference between the two-dimensional projected scene flow obtained based on the dynamic scene representation and the pre-computed optical flow. represents the optical flow pre-computed from the image frames. represents the two-dimensional projected scene flow obtained by projecting the neural representation of the dynamic scene after a time offset The two-dimensional projected scene flow

[0011] and the optical flow pre-computed from the image frames are obtained through the following operations respectively: , , where represents applying a three-dimensional scene flow extraction operator to the neural representation of the dynamic scene at time stamp , represents the two-dimensional optical flow calculated by the pre-computed optical flow estimation network for the image frame

[0012] The global and local motion consistency supervision module is also used to introduce a binary confidence mask to perform weighted supervision on the difference between the two-dimensional projected scene flow and the pre-computed optical flow. The flow loss function with the binary confidence mask introduced has the following expression: , where represents element-wise multiplication, and

[0013] is the binary confidence mask constructed at the i-th camera view. , represents the image reconstruction loss, which is used to measure the pixel difference between the rendered image and the real image; represents the L1 regularization term of the time offset parameter, which is used to prevent excessive offset; ,

[0014] The present invention also provides a new view synthesis method for asynchronous videos based on motion consistency implemented by using the said device, including: constructing a neural representation of the dynamic scene​​​​​​​​​ The input is the image sequences of multiple cameras and the corresponding set of camera projection matrices, and the output is the neural representation of the dynamic scene at any timestamp; a learnable time offset parameter is set for each camera and jointly optimized with the scene representation during training; the three-dimensional scene flow is calculated from the scene representation and projected into a two-dimensional projected scene flow and compared with the pre-computed optical flow to construct a binary confidence mask based on the optical flow magnitude and calculate the motion consistency supervision loss; jointly optimize the image reconstruction loss, the optical flow consistency loss, and the temporal regularization term to achieve spatio-temporally consistent dynamic reconstruction and novel view synthesis

[0015] The method is applicable to the six-dimensional dynamic neural scene modeling task; the neural representation of the dynamic scene is an implicit neural representation (such as Kplanes), or an explicit representation composed of three-dimensional Gaussians (such as 4DGS, EDGS). The method jointly optimizes the neural representation of the dynamic scene and the learnable time offset parameter to achieve spatio-temporally consistent dynamic reconstruction and novel view synthesis

[0016] Advantages: (1) The present invention proposes a novel view synthesis device for asynchronous videos based on motion consistency, which realizes efficient and automatic cross-camera time alignment and unified modeling of dynamic scenes without external synchronization hardware or manual annotation

[0017] (2) The present invention proposes a global and local motion consistency supervision method combining three-dimensional scene flow and two-dimensional optical flow, introducing the alignment between three-dimensional scene flow and two-dimensional optical flow as a supervision signal, effectively alleviating the spatial reconstruction error caused by temporal inconsistency, and improving the spatio-temporal consistency of novel view rendering

[0018] (3) The method of the present invention is compatible with multiple neural representations, including implicit neural representations (such as Kplanes) and explicit representations composed of three-dimensional Gaussians (such as 4DGS, EDGS). It has good generality and scalability, and achieves excellent novel view synthesis results on multiple representative datasets, outperforming previous methods that only optimize time offset parameters BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The following further specifically describes the present invention in conjunction with the drawings and specific embodiments, and the above and / or other advantages of the present invention will become clearer

[0020] Figure 1 Schematic diagram of the device for implementing the method of the present invention

[0021] Figure 2 Comparison diagram of the results of the 4DGS representation framework for the six-dimensional dynamic neural radiance field reconstruction task of the present invention

[0022] Figure 3 This is the result comparison diagram of the present invention in the EDGS representation framework for the six - dimensional dynamic neural radiance field reconstruction task;

[0023] Figure 4 This is the result comparison diagram of the present invention in the Kplanes representation framework for the six - dimensional dynamic neural radiance field reconstruction task. Detailed implementation manners

[0024] Referring to Figure 1 As shown, an asynchronous video novel view synthesis device based on motion consistency provided by an embodiment of the present invention includes a neural scene modeling module, a camera time offset estimation module, and a global and local motion consistency supervision module. Among them, the neural scene modeling module is used to construct a neural representation of a dynamic scene; the camera time offset estimation module is used to introduce learnable time offset parameters for each camera and jointly optimize them with the scene representation; the global and local motion consistency supervision module is used to calculate the three - dimensional scene flow of the neural scene and project it into two - dimensions, compare it with the pre - computed two - dimensional optical flow, and construct a time alignment supervision signal.

[0025] Among them, the mathematical expression adopted by the global and local motion consistency supervision module is: , By minimizing the difference between the pre - computed optical flow and the projected estimated optical flow, the time alignment of multi - view cameras is achieved. The binary confidence mask ensures that the supervision signal mainly comes from high - confidence regions with sufficient motion intensity, effectively suppressing the interference of supervision errors brought by low - motion or noise regions, and significantly improving the stability and robustness of dynamic scene reconstruction. Compared with traditional methods that only rely on image reconstruction errors, the present invention effectively reduces geometric blur and motion artifacts under asynchronous data by introducing global and local motion consistency losses, and realizes a more accurate spatio - temporal consistent representation of dynamic scenes with complex motion characteristics.

[0026] This embodiment also provides a method for asynchronous multi - view video novel view synthesis using the above - mentioned device. The specific steps include: constructing a neural scene modeling function , with the input being the image sequences of multiple cameras and the corresponding set of camera projection matrices, and the output being the neural representation of the dynamic scene at any timestamp; setting a learnable time offset parameter for each camera, and jointly optimizing it with the scene representation during the training process; calculating the three - dimensional scene flow from the scene representation and projecting it into a two - dimensional projected scene flow , comparing it with the pre - computed optical flow , and constructing a confidence mask according to the optical flow amplitude , calculate the motion consistency supervision loss; jointly optimize the image reconstruction loss, optical flow consistency loss and temporal regularization term to achieve spatiotemporal consistent dynamic reconstruction and new perspective synthesis.

[0027] The specific implementation steps of the neural scene modeling module are as follows: The neural scene modeling module is used to construct a neural scene representation, and the expression of the neural scene representation is: , Among them, R is a dynamic scene reconstruction operator, and the input is a multi-camera image sequence and the corresponding camera projection matrix The output is a collection of Neural representation of dynamic scenes . In this embodiment, the dynamic scene representation can be specifically implemented as an implicit neural representation or an explicit representation composed of three-dimensional Gaussians. The implicit neural representation is implemented using the neural radiation field Kplanes based on multi-plane decomposition, and high-dimensional mapping of the scene space position and observation direction is performed through multi-plane decomposition and multi-layer perceptron; the three-dimensional Gaussian explicit representation is implemented using 4DGS and EDGS, among which 4DGS uses a learnable deformation field to model the time change of the Gaussian center, and EDGS directly expresses the Gaussian center as a function of time, thereby realizing explicit modeling of the dynamic process.

[0028] The specific implementation steps of the camera time offset estimation module are as follows: For each camera perspective Set an independent, learnable time offset parameter , according to the number of cameras M of the multi-view input video, initialize a length List of time offset parameters The initial value of the list is randomly generated using a normal distribution with a mean of -0.001 and a standard deviation of 0.001 to ensure the stability of the training and the effectiveness of the model parameter optimization. During the training process, the neural representation parameters and the time offset parameters are jointly optimized through gradient back propagation. The expression of the optimization process is: , in, is the projection operator, used to project the neural representation into the camera image space, is the time offset for each camera, is the image frame corresponding to the i-th camera after time offset. The essence of this optimization goal is to achieve accurate time offset calibration by minimizing the difference between the neural representation projection image and the real image.

[0029] The specific implementation steps of the global and local motion consistency supervision module are as follows: Step 1: Use a pre-selected two-dimensional optical flow estimation algorithm to calculate the pre-computed two-dimensional optical flow between adjacent frames of the image sequences of each camera view : , where is the calculation operator of the pre-computed optical flow algorithm, represents the image captured by the i-th camera at time t, represents the adjacent time frame of the i-th camera captured image

[0030] Step 2: Based on the dynamic neural scene representation , obtain the three-dimensional scene flow through the three-dimensional scene flow extraction operator , representing the three-dimensional motion information of the spatial points in the scene between time and ; Step 3: For the i-th camera, use the projection matrix to project the three-dimensional scene flow onto the two-dimensional image plane to obtain the estimated two-dimensional projected scene flow : , Step 4: Construct an optical flow confidence mask according to the pre-computed optical flow . Calculate the magnitude of the pre-computed optical flow , and use the set threshold to construct a binary mask : , where is the set optical flow magnitude threshold, and pixels with magnitudes lower than the threshold are regarded as regions with insufficient or unreliable motion information, and thus are excluded when calculating the supervision loss to avoid the influence of noise on training; Step 5: Use the confidence mask to construct a weighted motion consistency supervision loss function, and calculate the consistency loss between the pre-computed optical flow and the estimated optical flow: , where represents element-wise multiplication, and the L1 norm ensures that the fine errors in the edges and non-rigid regions are retained.

[0031] By minimizing the above motion consistency loss function , the method of the present invention can significantly improve the alignment accuracy of multi-view input video sequences in the temporal dimension. Specifically, the proposed global and local motion consistency supervision mechanism can calculate the three-dimensional scene flow from the scene representation in each iteration optimization process of the model, project it into a two-dimensional projected scene flow, and calculate the loss with the two-dimensional optical flow independently calculated from each camera's single-view image sequence for motion consistency supervision.

[0032] The supervision strategy is based on the three-dimensional scene flow extracted from the neural representation, combines the camera projection matrix to obtain the two-dimensional projected scene flow, and compares it with the pre-computed optical flow pixel by pixel, thus possessing the spatio-temporal characteristics of cross-view consistency. Compared with the traditional method that only relies on the image reconstruction error loss or the method that only optimizes the time offset, the present invention enables the model to more effectively capture the small time offset differences caused by asynchronous sampling in multi-camera input by introducing an explicit motion consistency supervision term, and then corrects more accurately. In addition, the introduced optical flow confidence mask mechanism screens the high-confidence regions according to the amplitude of the pre-computed optical flow, further ensuring the quality and reliability of the supervision signal. Ensure that the supervision signal mainly comes from the pixel regions with real motion characteristics, thus effectively avoiding the model being interfered by false signals.

[0033] The extraction and projection process of the three-dimensional scene flow in the global and local motion consistency supervision module can be divided into the following two implementation paths according to different forms of the selected neural scene representation: When implemented using the Neural Radiance Field Kplanes based on multi-plane encoding of implicit neural representation, the extraction and projection steps of the three-dimensional scene flow are as follows: First, input the feature vector encoded by six planes into a multi-layer perceptron (MLP) serving as the scene flow decoder to decode the three-dimensional motion vector as the three-dimensional scene flow under the implicit representation; then, combine the camera projection matrix , project the three-dimensional motion vector onto the two-dimensional image plane to obtain the estimated two-dimensional projected scene flow.

[0034] When implemented using the three-dimensional Gaussian explicit representation 4DGS, EDGS, the extraction and projection steps of the three-dimensional scene flow are as follows: For the 4DGS representation, obtain the center position difference of the Gaussian basis elements on the two-dimensional image plane of adjacent frames in time by Gaussian splash rendering as the estimated two-dimensional projected scene flow; for the EDGS representation, obtain the center position difference of the Gaussian basis elements in adjacent time frames by modeling the time function of the Gaussian center position, and then use the camera projection matrix to project it onto the two-dimensional image plane to obtain the estimated two-dimensional projected scene flow.

[0035] The device and method of the present invention are applicable to the six - dimensional dynamic neural radiance field reconstruction task. In this embodiment, the device and method are specifically applied to the six - dimensional dynamic neural radiance field reconstruction task.

[0036] This embodiment is denoted as an asynchronous video novel view synthesis device based on motion consistency. In the six - dimensional dynamic neural radiance field reconstruction task, the benchmark methods for comparison are: the original baseline model, and the dynamic novel view synthesis method with only time offset added. The specific implementation details vary due to different neural radiance field representation frameworks. Therefore, the specific implementation methods under different neural radiance field representation frameworks will be described below, including 4DGS representation, EDGS representation, and Kplanes representation.

[0037] For the six - dimensional dynamic neural radiance field reconstruction task under the 4DGS representation framework, all comparison methods adopt default settings. This representation framework models the three - dimensional position center of Gaussian basis elements using a deformation field, predicts the center offset with the current timestamp as the input, and realizes the dynamic scene representation. In this embodiment, first, a dynamic scene representation is constructed through a neural scene modeling module. The input is an image sequence under multiple camera views and the corresponding camera projection matrices, and the output is the dynamic scene neural representation at any timestamp. An independent learnable time - offset parameter is introduced for each camera view, and its initial value follows a normal distribution with a mean of - 0.001 and a standard deviation of 0.001, which is used to correct the time - alignment error between multi - view camera video inputs and is fixed after 6k training optimization rounds. For the real - world scene dataset, the number of pre - training iterations is 3k, and the total number of training iterations is 15k; for the virtual synthetic scene dataset, the number of pre - training iterations is 3k, and the total number of training iterations is 20k. During the training process, the center - position difference of Gaussian basis elements on the two - dimensional image plane of temporally adjacent frames is obtained through Gaussian splatter rendering as the estimated two - dimensional projected scene flow, which is compared with the pre - computed optical flow obtained through the VideoFlow optical flow calculation algorithm. The motion - consistency supervision loss is calculated by combining the confidence mask constructed according to the pre - computed optical flow amplitude. Finally, through the joint image reconstruction loss and motion - consistency supervision loss, spatio - temporally consistent dynamic reconstruction and novel view synthesis are achieved. Training and testing are carried out on the virtual synthetic dataset Dynamic Blender Datasets and the real - world scene dataset Plenoptic Video Datasets. Figure 2 The result comparison is shown. The method of the present invention shows the reconstruction advantage in fast - moving regions, such as the ear and mouth parts of the deer, the spray gun part with reflection, and the flame part. The original baseline model and the dynamic novel view synthesis method with only time offset added based on the 4DGS representation framework have obvious blurring and artifacts in the moving regions due to the asynchrony of multiple views. The present invention can effectively reduce the blurring and artifacts caused by time asynchrony and achieve a more realistic dynamic scene reconstruction.

[0038] For the six-dimensional dynamic neural radiance field reconstruction task under the EDGS representation framework, all comparison methods adopt default settings. This representation framework realizes dynamic scene representation by modeling the central position of each Gaussian basis element as a function of time. In this embodiment, a dynamic scene representation is first constructed through a neural scene modeling module. The input is an image sequence under multiple camera views and their corresponding camera projection matrices, and the output is a dynamic scene neural representation at any timestamp. A learnable time offset parameter is introduced for each camera, and its initial value follows a normal distribution with a mean of -0.001 and a standard deviation of 0.001, which is used to correct the time alignment error between multi-view camera video inputs and is fixed after 6k rounds of training optimization. For the real scene dataset, the number of pre-training iterations is 3k, and the total number of training iterations is 20k; for the virtual synthetic scene dataset, the number of pre-training iterations is 3k, and the total number of training iterations is 30k. During the training process, the difference in the central positions of Gaussian basis elements in adjacent time frames is obtained by modeling the time function of the Gaussian center position. Then, using the camera projection matrix project it onto the two-dimensional image plane to obtain the estimated two-dimensional projected scene flow, compare it with the pre-computed optical flow obtained by the RAFT optical flow calculation algorithm, and calculate the motion consistency supervision loss in combination with the confidence mask constructed according to the pre-computed optical flow amplitude. Finally, through the joint image reconstruction loss and the motion consistency supervision loss, spatio-temporally consistent dynamic reconstruction and novel view synthesis are achieved. Training and testing are carried out on the virtual synthetic dataset Dynamic Blender Datasets and the real scene dataset Plenoptic Video Datasets. Figure 3 The result comparison is shown. The present invention demonstrates the reconstruction advantages in the motion area and the weak texture area. For example, the curved surface texture and contour edge parts of the pendulum, the glass surface area, and the surface structure and texture of the spring. There are blurring or artifact phenomena in the original baseline model of the compared baseline EDGS representation framework and the dynamic novel view synthesis method with only time offset added. However, the present invention significantly reduces these artifacts and blurring phenomena by introducing motion consistency supervision, improving the spatial consistency of dynamic scene reconstruction.

[0039] For the six - dimensional dynamic neural radiance field reconstruction task under the Kplanes representation framework, all comparison methods adopt default settings. This framework uses an implicit neural representation method based on multi - plane decomposition. By projecting the three - dimensional spatial position and time information onto two - dimensional planes in six orthogonal directions respectively, voxel features are obtained and then input into a multi - layer perceptron (MLP) for decoding to achieve dynamic scene representation. In this embodiment, the decoding network used is a one - layer fully - connected neural network, which contains 64 hidden units. First, a dynamic scene representation is constructed through a neural scene modeling module. A learnable time - offset parameter is introduced for each input camera, and its initial value follows a normal distribution with a mean of - 0.001 and a standard deviation of 0.001, which is used to correct the time - alignment error between multi - view camera video inputs and is fixed after 50k optimization rounds. For the real - world scene dataset and the virtual synthetic scene dataset, the total number of training iterations is 90k. During the training process, the feature vector obtained by six - plane encoding is input into the multi - layer perceptron (MLP) serving as the scene - flow decoder, and the decoded three - dimensional motion vector is used as the three - dimensional scene flow under the implicit representation. Then, combined with the camera projection matrix , the three - dimensional motion vector is projected onto the two - dimensional image plane to obtain the estimated two - dimensional projected scene flow, which is compared with the pre - computed optical flow obtained by the VideoFlow optical - flow calculation algorithm. The motion - consistency supervision loss is calculated by combining the confidence mask constructed based on the magnitude of the pre - computed optical flow. Finally, through the joint image - reconstruction loss and the motion - consistency supervision loss, spatio - temporally consistent dynamic reconstruction and novel - view synthesis are achieved. Training and testing are carried out on the Dynamic Blender Datasets and PlenopticVideo Datasets. Figure 4 The result comparison is shown. The present invention demonstrates the reconstruction advantages in the motion area and the weak - texture area. For example, in the head part of the dog and the highlighted part of the clip, obvious blurring and artifacts appear in the original baseline model based on the Kplanes representation framework and the dynamic novel - view synthesis method with only time - offset added for comparison. However, the global and local motion - consistency supervision mechanism introduced by the present invention effectively improves the multi - camera time - alignment accuracy and the spatio - temporal consistency of the reconstruction results, achieving a more realistic dynamic scene reconstruction.

[0040] The present invention provides a device and method for novel - view synthesis of asynchronous videos based on motion consistency. There are many specific methods and ways to implement this technical solution. The above - mentioned is only the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention. Each component not clearly defined in this embodiment can be implemented using existing technologies.

Claims

1. An asynchronous video new perspective synthesis device based on motion consistency, characterized in that, It includes a neural scene modeling module, a camera time offset estimation module, and a global and local motion consistency supervision module; The neural scene modeling module is used to construct a neural representation of a dynamic scene; The camera time offset estimation module is used to introduce learnable time offset parameters for each camera and jointly optimize them with the scene representation; The global and local motion consistency supervision module is used to calculate the three-dimensional scene flow of the neural scene, project it into two dimensions, compare it with the pre-computed two-dimensional optical flow, and construct a time alignment supervision signal.

2. The device according to claim 1, wherein The expression of the neural representation is: ， Among them, R is a dynamic scene reconstruction operator for generating a neural representation of a dynamic scene; the input of the dynamic scene reconstruction operator R includes an image sequence of multiple cameras and a corresponding set of camera projection matrices; denotes the image frame captured by the i-th camera at timestamp , denotes the projection matrix corresponding to the i-th camera, and M represents the total number of cameras; the output of the dynamic scene reconstruction operator R is the neural representation of the dynamic scene at any timestamp t .

3. The device according to claim 2, wherein The camera time offset estimation module is used to set an independent learnable time offset parameter for each camera , denotes the time offset of the i-th camera, and this parameter participates in backpropagation during training and is jointly optimized with the scene representation through the loss function, and is used to correct the time asynchrony between multi-camera input videos.

4. The device according to claim 3, wherein The neural representation is obtained by jointly optimizing with learnable time shift parameters The expression for the joint optimization is: ， Among them, represents the neural representation of the optimal dynamic scene optimized at time ; represents the set of optimal time offset parameters obtained by optimization; is a projection operator, which represents the neural representation of the dynamic scene at time projected through the projection matrix of the corresponding i-th camera to obtain the image frame , and is the image frame corresponding to the i-th camera after time offset.

5. The device according to claim 4, characterized in that, The global and local motion consistency supervision module is used to construct an alignment loss term between the two-dimensional projected scene flow of the neural scene and the pre-computed optical flow, and introduce a flow loss function , and the expression is: ， Among them, represents the flow consistency supervision loss, represents the optical flow pre-computed from the image frames, represents the two-dimensional projected scene flow obtained by projecting the neural representation based on the dynamic scene after a time offset ​ 6. The device according to claim 5, characterized in that, The two-dimensional projected scene flow and the optical flow pre-computed from the image frames are obtained through the following operations respectively: ， ， Among them, represents the neural representation of the dynamic scene for the timestamp t Apply a three-dimensional scene flow extraction operator, represents the two-dimensional optical flow calculated through the pre-computed optical flow estimation network for the image frame Calculated two-dimensional optical flow.

7. The device according to claim 6, characterized in that, The global and local motion consistency supervision module is also used to introduce a binary confidence mask to perform weighted supervision on the difference between the two-dimensional projected scene flow and the pre-computed optical flow, and a flow loss function with a binary confidence mask is expressed as: ， Among them, represents per-pixel element multiplication, is the binary confidence mask constructed under the i-th camera view.

8. The device according to claim 7, wherein The joint optimization objective of the neural representation of the dynamic scene and the learnable time offset parameters includes an image reconstruction loss term, an optical flow consistency supervision term, and a time offset regularization term. The expression of the optimization objective is: ， Among them, represents the image reconstruction loss, which is used to measure the pixel difference between the rendered image and the real image; represents the L1 regularization term of the time offset parameter, which is used to prevent excessive offset; and respectively represent the weight hyperparameter of the flow loss function and the weight hyperparameter of the L1 regularization term of the time offset parameter, which control the influence weight of each loss term on the total optimization objective.

9. A novel perspective synthesis method for asynchronous videos based on motion consistency implemented by using the device according to any one of claims 1 to 8, characterized in that, Including: Constructing Neural Representations of Dynamic Scenes , with the input being the image sequences of multiple cameras and the corresponding set of camera projection matrices, and the output being the neural representation of the dynamic scene at any timestamp; a learnable time offset parameter is set for each camera , and jointly optimized with the scene representation during training; calculate the 3D scene flow from the scene representation and project it into a 2D projected scene flow , compared with the pre-computed optical flow , construct a binary confidence mask according to the optical flow magnitude , calculate the motion consistency supervision loss; jointly optimize the image reconstruction loss, the optical flow consistency loss, and the temporal regularization term to achieve spatio-temporally consistent dynamic reconstruction and novel view synthesis.

10. The method according to claim 9, wherein The method is applicable to six-dimensional dynamic neural scene modeling tasks; the neural representation of the dynamic scene is an implicit neural representation or an explicit representation composed of three-dimensional Gaussians. The method jointly optimizes the neural representation of the dynamic scene and the learnable time offset parameters to achieve spatio-temporally consistent dynamic reconstruction and novel view synthesis.

Images