A deblurring view synthesis method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application disclose a deblurring view synthesis method and device, electronic equipment and storage medium, relating to the technical field of computer vision and three-dimensional reconstruction, wherein the method comprises: obtaining an initial camera pose through three-dimensional reconstruction software, and modeling a camera motion trajectory using a cubic Bezier curve; dynamically predicting three-plane resolution according to the trajectory length using a feedforward neural network; performing adaptive three-plane down-sampling based on the predicted resolution, and generating a clear image through volume rendering technology; and finally obtaining a full-resolution three-plane field through loss optimization and gradient descent algorithm, and generating a clear image corresponding to a target camera pose according to learnable parameters and the three-plane field. The present application solves the problems of low sampling efficiency, long training time and degraded rendering quality under long trajectories when the prior art faces different lengths of camera motion trajectories, significantly improves the training efficiency, reduces the memory occupation, and enhances the generalization ability and reconstruction quality of the model.

Description

Technical Field

[0001] This invention relates to the field of computer vision and 3D reconstruction technology, and in particular to a method, apparatus, electronic device and storage medium for deblurred view synthesis. Background Technology

[0002] With the rapid development of computer vision technology, new view synthesis, as a key task for generating new view images from known view images, has shown broad application prospects in fields such as virtual reality, augmented reality, and 3D reconstruction.

[0003] However, in practical applications, the movement of the camera or object within the exposure time often leads to image blur (motion blur), which seriously affects the quality and accuracy of the synthesized new view. Traditional deblurring methods face problems such as low sampling efficiency, lengthy training time, and severe degradation of rendering quality under long motion trajectories when processing camera motion trajectories of different lengths.

[0004] Specifically, these methods struggle to effectively balance sampling accuracy and computational cost when modeling non-uniform blur effects caused by camera motion, especially when dealing with complex or extremely long-exposure blur scenes, where their performance is poor. Furthermore, traditional methods often use a fixed number of light samples, which cannot be dynamically adjusted based on the camera trajectory length, resulting in insufficient generalization ability in diverse real-world application scenarios.

[0005] Therefore, there is an urgent need for a deblurred view synthesis method that is efficient, flexible, and adaptable to camera motion trajectories of different lengths, and that can improve the quality and efficiency of 3D scene reconstruction. Summary of the Invention

[0006] The embodiments of the present invention provide a method for deblurring view synthesis to address the problems of low sampling efficiency, long training time, severe rendering quality degradation under long trajectories, and inability to dynamically adjust sampling strategies according to the scene in existing technologies when dealing with camera motion trajectories of different lengths. The technical solution is as follows:

[0007] According to one aspect of the present invention, a method for deblurring view synthesis includes: acquiring multi-view data of a scene containing dynamic blur; estimating an initial camera pose based on the multi-view data using 3D reconstruction software; modeling the motion trajectory of a camera based on the initial camera pose using cubic Bézier curves, and obtaining learnable parameters of the Bézier curves; uniformly selecting multiple camera positions as sampling points on the motion trajectory of the camera using Bézier interpolation; for each sampling point, obtaining a three-plane resolution prediction level of the sampling point using a feedforward neural network based on the learnable parameters and the camera pose of the sampling point; downsampling the full-resolution three planes to a set resolution based on the three-plane resolution prediction level to generate three-plane data of the sampling points; generating a corresponding clear image based on the three-plane data and the camera pose using volume rendering technology; averaging the clear images of all the sampling points to obtain a synthesized blurred image, comparing it with a real blurred image in the multi-view data to obtain a loss value; optimizing the learnable parameters and the three-plane field based on the loss value to obtain a full-resolution three-plane field, and then generating a clear image corresponding to the target camera pose.

[0008] In one embodiment, the camera's motion trajectory is modeled based on the initial camera pose using a cubic Bézier curve, and the learnable parameters of the Bézier curve are obtained through the following steps: The camera's motion trajectory is mathematically modeled using a cubic Bézier curve based on the initial camera pose; the control points of the Bézier curve are set as learnable parameters; the control points include a start point, a control point, and an end point; the learnable parameters are automatically optimized using a backpropagation algorithm; the start point and end point correspond to the camera pose at the start and end points of the motion trajectory, respectively; the control points are used to adjust the curvature of the Bézier curve to fit the nonlinear motion pattern of the camera during the exposure time.

[0009] In one embodiment, for each sampling point, the prediction level of the three-plane resolution of the sampling point is obtained by a feedforward neural network based on the learnable parameters and the camera pose of the sampling point through the following steps: For each sampling point, the learnable parameters and the camera pose of the sampling point are input into a three-layer feedforward neural network. The mapping relationship between trajectory length and optimal resolution is automatically learned through supervised learning to obtain the prediction level of the three-plane resolution; the hidden layer dimension of the feedforward neural network is 128, the activation function is ReLU, and the output activation function is Sigmoid; the prediction level of 0 corresponds to the lowest resolution of 128×128, and 1 corresponds to the highest resolution of 512×512; the camera pose includes a rotation matrix and a translation vector.

[0010] In one embodiment, the full-resolution three planes are downsampled to a set resolution according to the three-plane resolution prediction level to generate the three-plane data of the sampling points through the following steps: according to the three-plane resolution prediction level, the full-resolution three planes are downsampled by bilinear interpolation to obtain the downsampled three-plane data; the bilinear interpolation downsampling is used to ensure that sampling points of different trajectory lengths can participate in training at an appropriate resolution.

[0011] In one embodiment, generating a clear image based on the three-plane data and the camera pose using volume rendering technology is achieved through the following steps: employing a layered sampling strategy, multiple rendering points are uniformly sampled in three-dimensional space based on the camera pose of the sampling points using volume rendering technology; the density and color values ​​of the rendering points are obtained based on the three-plane data; the final RGB pixel values ​​are calculated by transmissivity weighted integration of the density and color values ​​of the rendering points; and a clear image without dynamic blur is generated based on the RGB pixel values.

[0012] In one embodiment, the process of averaging the clear images of all the sampling points to obtain a synthesized blurred image and comparing it with the real blurred image in the multi-view data to obtain a loss value is achieved through the following steps: averaging the clear images of all the sampling points to obtain a synthesized blurred image; calculating the mean absolute error based on the synthesized blurred image and the real blurred image in the multi-view data; obtaining the clear images of the sampling points by downsampling the motion velocity within the exposure time to obtain three planes of different resolutions, and then performing volume rendering; using the mean absolute error as the loss value, while introducing a three-plane field smoothness regularization term and camera motion trajectory rationality constraints; and using the loss value for backpropagation to update the Bézier curve parameters and the three-plane field.

[0013] In one embodiment, the generation of a clear image corresponding to the target camera pose after optimizing the learnable parameters and the three-plane field according to the loss value to obtain a full-resolution three-plane field is achieved through the following steps: continuously optimizing the learnable parameters and the three-plane field of the Bézier curve using a gradient descent algorithm until the loss value converges to the obtained full-resolution three-plane field; after obtaining the target camera pose, performing volume rendering based on the full-resolution three-plane field and the target camera pose to generate a clear image in real time.

[0014] According to one aspect of the present invention, a deblurred view synthesis apparatus includes: a pose estimation and trajectory modeling module, configured to acquire scene multi-view data containing dynamic blur, estimate an initial camera pose based on the multi-view data using 3D reconstruction software, model the motion trajectory of the camera using a cubic Bézier curve based on the initial camera pose, and obtain learnable parameters of the Bézier curve; and a trajectory sampling and resolution prediction module, configured to uniformly select multiple camera positions as sampling points on the camera's motion trajectory using Bézier interpolation, and for each sampling point, obtain the desired image based on the learnable parameters and the camera pose of the sampling point using a feedforward neural network. The sampling points have a three-plane resolution prediction level; an adaptive downsampling and volume rendering module is used to downsample the full-resolution three planes to a set resolution according to the three-plane resolution prediction level, generate the three-plane data of the sampling points, and generate a corresponding clear image based on the three-plane data and the camera pose using volume rendering technology; a loss optimization and view synthesis module is used to average the clear images of all the sampling points to obtain a synthesized blurred image, compare it with the real blurred image in the multi-view data to obtain a loss value, optimize the learnable parameters and the three-plane field according to the loss value to obtain a full-resolution three-plane field, and then generate a clear image corresponding to the target camera pose.

[0015] According to one aspect of the present invention, an electronic device includes at least one processor and at least one memory, wherein computer-readable instructions are stored on the memory; the computer-readable instructions are executed by one or more of the processors to cause the electronic device to implement the deblurred view synthesis method as described above.

[0016] According to one aspect of the present invention, a storage medium has computer-readable instructions stored thereon, which are executed by one or more processors to implement the deblurred view synthesis method as described above.

[0017] The beneficial effects of the technical solution provided by this invention are:

[0018] In the above technical solution, this invention first acquires multi-view scene data containing dynamic blur using 3D reconstruction software and estimates the initial camera pose. It then models the camera trajectory using cubic Bézier curves and sets learnable parameters. Subsequently, it uniformly samples along the trajectory and uses a feedforward neural network to predict the three-plane resolution of each sampling point, generating a clear image through volume rendering. Further, the clear image is averaged to obtain a synthesized blurred image, which is compared with the real blurred image to calculate the L1 loss. A regularization term is introduced to constrain the smoothness of the three-plane field and the rationality of the camera trajectory. Finally, the learnable parameters and the three-plane field are optimized using a gradient descent algorithm to obtain a full-resolution three-plane field, and a clear image corresponding to the target camera pose is generated in real time. Through adaptive resolution control and spatial awareness mechanisms, the contradiction between sampling efficiency and rendering quality under camera trajectories of different lengths is effectively resolved, significantly improving training efficiency, memory usage optimization, and reconstruction quality, and enhancing generalization ability. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a method for compositing deblurred views according to an exemplary embodiment;

[0021] Figure 2 This is a flowchart illustrating a deblurred view synthesis method in an exemplary embodiment;

[0022] Figure 3 This is a block diagram illustrating a deblurred view synthesis apparatus according to an exemplary embodiment;

[0023] Figure 4 This is a hardware structure diagram of an electronic device according to an exemplary embodiment;

[0024] Figure 5 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0025] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0026] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this disclosure means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0027] This invention provides a method for deblurring view synthesis. By modeling camera motion trajectories using Bézier curves and combining adaptive resolution prediction and volume rendering techniques, it achieves high-quality, clear image synthesis in dynamically blurred scenes. This effectively solves the problems of reconstruction distortion and low computational efficiency in traditional methods under complex motion modes. This deblurring view synthesis method is applicable to deblurring view synthesis devices, which can be electronic devices. The deblurring view synthesis method in this invention can be applied to various scenarios, such as multi-view 3D reconstruction.

[0028] Please see Figure 1 This invention provides a method for compositing deblurred views, which is applicable to electronic devices.

[0029] In the following method embodiments, for ease of description, the execution subject of each step of the method is an electronic device, but this does not constitute a specific limitation.

[0030] like Figure 1 As shown, the method may include the following steps:

[0031] Step 110: Obtain multi-view data of the scene containing dynamic blur, estimate the initial camera pose based on the multi-view data using 3D reconstruction software, model the camera's motion trajectory based on the initial camera pose using cubic Bézier curves, and obtain the learnable parameters of the Bézier curves.

[0032] In one possible implementation, a cubic Bézier curve is used to mathematically model the camera's motion trajectory based on the initial camera pose. The control points of the Bézier curve are set as learnable parameters, and the learnable parameters are automatically optimized through the backpropagation algorithm.

[0033] The control points include a start point, a control point, and an end point. The start point and the end point correspond to the camera pose at the beginning and end of the motion trajectory, respectively. The control points are used to adjust the curvature of the Bézier curve and fit the nonlinear motion pattern of the camera during the exposure time.

[0034] Specifically, the scene multi-view data containing dynamic blur is processed using 3D reconstruction software (such as COLMAP) to obtain the initial camera pose of each image. This step provides a basic spatial reference for subsequent trajectory modeling.

[0035] A cubic Bézier curve is used to mathematically model the camera motion trajectory, and the starting point, control point, and ending point of the curve are set as learnable parameters.

[0036] Furthermore, these parameters are automatically optimized using a backpropagation algorithm, enabling the trajectory to fit the nonlinear motion pattern of the camera within the exposure time. Multiple camera positions are uniformly selected as sampling points from the optimized Bézier curve through interpolation, with each sampling point corresponding to a camera pose (rotation matrix R and translation vector T).

[0037] In the above process, the embodiments of the present invention, through Bézier curve modeling and learnable parameter optimization, enable trajectory modeling to adapt to complex nonlinear motion modes, thereby achieving accurate modeling of dynamic fuzzy scenes.

[0038] Step 120: Multiple camera positions are uniformly selected as sampling points on the camera's motion trajectory using Bezier interpolation. For each sampling point, the three-plane resolution prediction level of the sampling point is obtained by using a feedforward neural network based on the learnable parameters and the camera pose of the sampling point.

[0039] In one possible implementation, for each sampling point, the learnable parameters and the camera pose of the sampling point are input into a three-layer feedforward neural network. Through supervised learning, the mapping relationship between the trajectory length and the optimal resolution is automatically learned to obtain the prediction level of the three-plane resolution.

[0040] Among them, the three planes are tri-plane feature fields, which refer to the use of three mutually perpendicular feature planes (xy, yz, zx) to represent the three-dimensional scene; the hidden layer dimension of the feedforward neural network is 128, the activation function is ReLU, and the output activation function is Sigmoid; the prediction level 0 corresponds to the lowest resolution of 128×128, 1 corresponds to the highest resolution of 512×512, and the prediction level 0~1 maps to the intermediate resolution; the camera pose includes rotation matrix and translation vector.

[0041] Specifically, the learnable parameters of the Bézier curve and the camera pose (R and T) of the sampling points are used as input data. A three-layer feedforward neural network (128 hidden layer dimensions, ReLU activation function) is used to automatically learn the mapping relationship between the trajectory length and the optimal three-plane resolution, and output the predicted three-plane resolution level (0 corresponds to 128×128, 1 corresponds to 512×512, and 0~1 maps to the intermediate resolution).

[0042] In the above process, the embodiments of the present invention utilize the spatial perception capability of the feedforward neural network to enable resolution prediction to dynamically adapt to camera trajectories of different lengths, providing an adaptive resolution adjustment mechanism and achieving efficient resource utilization.

[0043] Step 130: Based on the three-plane resolution prediction level, downsample the full-resolution three planes to the set resolution to generate three-plane data of the sampling points. Then, use volume rendering technology to generate a corresponding clear image based on the three-plane data and camera pose.

[0044] In one possible implementation, the full-resolution three planes are downsampled by bilinear interpolation based on the three-plane resolution prediction level to obtain downsampled three-plane data.

[0045] Among them, bilinear interpolation downsampling is used to ensure that sampling points of different trajectory lengths can participate in training at an appropriate resolution.

[0046] The formula for calculating the resolution R is: s represents the three-plane resolution prediction level, ensuring that the resolution is dynamically adjusted between 128×128 and 512×512.

[0047] In one possible implementation, a layered sampling strategy is used to uniformly sample multiple rendering points in three-dimensional space based on the camera pose of the sampling points using volume rendering technology. The density and color values ​​of the rendering points are obtained based on the three-plane data. The final RGB pixel values ​​are calculated by weighted integral of the transmittance of the density and color values ​​of the rendering points. A clear image without dynamic blur is generated based on the RGB pixel values.

[0048] Specifically, based on the predicted three-plane resolution level, bilinear interpolation downsampling is performed on the full-resolution three planes (512×512) to obtain three-plane data adapted to the current trajectory length. A layered sampling strategy is employed to uniformly sample multiple rendering points in 3D space, and the density and color values ​​of these rendering points are obtained from the downsampled three-plane data. RGB pixel values ​​are calculated through transmittance-weighted integral to generate a clear image without dynamic blur.

[0049] In the above process, the embodiments of the present invention use adaptive downsampling and volume rendering technology to enable clear image generation to balance accuracy and efficiency, providing high-quality rendering results and achieving effective deblurring of dynamically blurred scenes.

[0050] Step 140: Average the clear images of all sampling points to obtain a synthesized blurred image and compare it with the real blurred image in the multi-view data to obtain the loss value. Optimize the learnable parameters and the three-plane field based on the loss value to obtain a full-resolution three-plane field and then generate a clear image corresponding to the target camera pose.

[0051] In one possible implementation, the clear images of all sampling points are averaged to obtain a synthetic blurred image. The mean absolute error is calculated based on the synthetic blurred image and the real blurred image in the multi-view data. The mean absolute error is used as the loss value. At the same time, a three-plane field smoothness regularization term and a camera motion trajectory rationality constraint are introduced.

[0052] Among them, the clear image of the sampling point is obtained by downsampling the motion speed during the exposure time to obtain three planes of different resolutions, and then performing volume rendering; the real blurred image refers to the unclear image generated by the camera due to motion or shaking during the imaging process; the loss value is used for backpropagation to update the Bézier curve parameters and the three-plane field.

[0053] In one possible implementation, the learnable parameters of the Bézier curve and the three-plane field are continuously optimized using a gradient descent algorithm until the loss value converges to the full-resolution three-plane field. After obtaining the target camera pose, volume rendering is performed based on the full-resolution three-plane field and the target camera pose to generate a clear image in real time.

[0054] In the above process, the embodiments of the present invention, through loss optimization and gradient descent algorithms, enable continuous improvement of model parameters, providing accurate and clear image generation capabilities and achieving high-quality reconstruction of dynamically blurred scenes. Through the above process, the present invention proposes a set of efficient deblurring new view synthesis algorithms by employing camera pose estimation and trajectory modeling, spatially aware three-plane resolution prediction, adaptive three-plane downsampling and volume rendering, loss optimization, and clear image generation. This solves the problems of low sampling efficiency, lengthy training time, and severe rendering quality degradation under long motion trajectories in existing methods, significantly improving training efficiency, optimizing memory usage, and enhancing reconstruction quality, as well as generalization ability.

[0055] In one exemplary embodiment, the process of performing deblurred view synthesis by the deblurred view synthesis method of the present invention is illustrated.

[0056] like Figure 2 As shown, the following steps may be included:

[0057] Part One: The Training Phase

[0058] Step S1: Acquire camera motion trajectory and select camera position on the trajectory.

[0059] Specifically, multi-view data of the scene containing motion blur is acquired, and this data is processed using 3D reconstruction software (such as COLMAP) to estimate the initial camera pose, thereby obtaining the camera's motion trajectory. Further, camera positions along this trajectory are selected; these positions will serve as sampling points for subsequent processing.

[0060] In the above process, the embodiments of the present invention accurately obtain the camera motion trajectory and initial pose through three-dimensional reconstruction software, providing basic data for subsequent trajectory modeling and sampling.

[0061] Step S2, spatial perception module processing.

[0062] Specifically, the camera position on the trajectory selected in step S1 is input to the spatial perception module (consisting of a three-layer MLP with a hidden layer dimension of 128, an activation function of ReLU, and an output activation function of Sigmoid). This module combines the learnable parameters of the cubic Bézier curve (including the start point, control point, and end point, where the start point and end point correspond to the camera pose at the start and end points of the motion trajectory, respectively, and the control point is used to adjust the curvature of the Bézier curve to fit the nonlinear motion pattern of the camera during the exposure time) and calculates the expected resolution internally.

[0063] Furthermore, the learnable parameters of the cubic Bézier curve are automatically optimized during training using a backpropagation algorithm.

[0064] In the above process, the embodiments of the present invention achieve accurate representation of camera motion trajectory and dynamic resolution prediction through spatial perception module and Bézier curve modeling, thereby improving the adaptability and accuracy of the model.

[0065] Step S3: Obtain the expected resolution, and adjust the full resolution three-plane field to the expected resolution three-plane field according to the expected resolution.

[0066] Specifically, based on the expected resolution output by the spatial perception module in step S2, bilinear interpolation downsampling is performed on the full-resolution three-plane field (512×512).

[0067] Furthermore, the resolution of the downsampled three-plane field is determined according to the expected resolution level, which is output by the feedforward neural network (0 corresponds to the lowest resolution of 128×128, and 1 corresponds to the highest resolution of 512×512) to ensure that sampling points of different trajectory lengths can participate in training with an appropriate resolution.

[0068] In the above process, the embodiments of the present invention optimize the allocation of computing resources and improve training efficiency through adaptive resolution adjustment.

[0069] Step S4: Perform volume rendering based on the camera position on the trajectory to generate multiple clear images.

[0070] Specifically, the camera position is selected again on the trajectory, and combined with the downsampled expected resolution three-plane field data from step S3, a corresponding clear image is generated using volumetric rendering technology. During volumetric rendering, a layered sampling strategy is adopted to uniformly sample multiple rendering points in three-dimensional space, and the density and color values ​​of the rendering points are obtained based on the three-plane field data.

[0071] Furthermore, the final RGB pixel values ​​are calculated by transmissivity weighted integral, thereby generating a clear image without motion blur.

[0072] In the above process, the embodiments of the present invention realize the conversion from three-plane field data to a clear image through volume rendering technology, which provides a foundation for subsequent blur synthesis and loss calculation.

[0073] Step S5: The multiple clear images are summed to obtain a dynamic blurred image.

[0074] Specifically, the sharp images of all sampling points generated in step S4 are averaged to obtain a motion-blurred image. The weight of the average is inversely proportional to the motion speed during the exposure time.

[0075] Furthermore, the dynamic blurred image is compared with the real blurred image in the multi-view data to calculate the mean absolute error (L1 loss), while introducing a three-plane field smoothness regularization term and camera motion trajectory rationality constraints.

[0076] In the above process, the embodiments of the present invention provide direction for model optimization by calculating the loss value and introducing regularization terms and constraints, which helps to improve the deblurring effect of the model.

[0077] Step S6: Iterative optimization.

[0078] Specifically, steps S1 to S5 are repeated n times to continuously optimize the learnable parameters of the Bézier curve and the three-plane field using the gradient descent algorithm until the loss value converges, resulting in an optimized full-resolution three-plane field, thereby continuously improving the model's performance and deblurring effect.

[0079] Part Two: Reasoning Stage.

[0080] Step S7: Target camera pose acquisition and clear image generation.

[0081] Specifically, the target camera pose is obtained, which represents the viewpoint from which a clear image needs to be generated. Based on the trained full-resolution three-plane field obtained during the training phase, volume rendering is performed on the target camera pose to generate a clear image in real time.

[0082] In the above process, the embodiments of the present invention achieve rapid generation of clear images under the target camera pose by applying a trained model, thus solving the problem of image sharpening in dynamic blurry scenes.

[0083] Through the above process, this embodiment of the invention first acquires the camera trajectory and selects sampling points during training, then predicts the resolution using the spatial perception module, adjusts the three-plane field, and generates a clear image through volume rendering. A blurred image is then synthesized, and the loss is calculated to optimize the model. During inference, a clear image is generated in real time based on the target pose, effectively solving the image sharpening problem in dynamically blurred scenes.

[0084] The following are embodiments of the apparatus of the present invention, which can be used to execute the deblurred view synthesis method involved in the present invention. For details not disclosed in the embodiments of the apparatus of the present invention, please refer to the method embodiments of the deblurred view synthesis method involved in the present invention.

[0085] Please see Figure 3 This invention provides a deblurred view synthesis device 800.

[0086] The deblurred view synthesis device 800 includes, but is not limited to: a pose estimation and trajectory modeling module 810, a trajectory sampling and resolution prediction module 830, an adaptive downsampling and volume rendering module 850, and a loss optimization and view synthesis module 870.

[0087] Among them, the pose estimation and trajectory modeling module 810 is used to acquire scene multi-view data containing dynamic blur, estimate the initial camera pose based on the multi-view data through 3D reconstruction software, model the camera's motion trajectory based on the initial camera pose using cubic Bézier curves, and obtain the learnable parameters of the Bézier curves.

[0088] The trajectory sampling and resolution prediction module 830 is used to uniformly select multiple camera positions as sampling points on the camera's motion trajectory through Bezier interpolation. For each sampling point, the three-plane resolution prediction level of the sampling point is obtained through a feedforward neural network based on learnable parameters and the camera pose of the sampling point.

[0089] The adaptive downsampling and volume rendering module 850 is used to downsample the full-resolution three planes to a set resolution based on the three-plane resolution prediction level, generate three-plane data of the sampling points, and generate a corresponding clear image based on the three-plane data and camera pose through volume rendering technology.

[0090] The loss optimization and view synthesis module 870 is used to average the clear images of all sampling points to obtain a synthesized blurred image, and compare it with the real blurred image in the multi-view data to obtain the loss value. Based on the loss value, the learnable parameters and the three-plane field are optimized to obtain the full-resolution three-plane field and then generate a clear image corresponding to the target camera pose.

[0091] It should be noted that the above embodiments of deblurred view synthesis are only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed. That is, the internal structure of the deblurred view synthesis device will be divided into different functional modules to complete all or part of the functions described above.

[0092] Furthermore, the embodiments of the deblurred view synthesis apparatus and the deblurred view synthesis method provided in the above embodiments belong to the same concept, and the specific way in which each module performs its operation has been described in detail in the method embodiments, and will not be repeated here.

[0093] Figure 4 A schematic diagram of the structure of an electronic device according to an exemplary embodiment is shown.

[0094] It should be noted that this electronic device is merely an example adapted to the present invention and should not be construed as providing any limitation on the scope of use of the present invention. Furthermore, this electronic device should not be interpreted as requiring or depending on having... Figure 4 One or more components of the exemplary electronic device 2000 shown.

[0095] The hardware structure of electronic devices 2000 can vary significantly due to differences in configuration or performance, such as... Figure 4 As shown, the electronic device 2000 includes: a power supply 210, an interface 230, at least one memory 250, and at least one central processing unit (CPU) 270.

[0096] Specifically, power supply 210 is used to provide operating voltage for various hardware devices on electronic device 2000.

[0097] Interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. Of course, in other examples adapted to this invention, interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input / output interface 235, and at least one USB interface 237, etc. Figure 4 As shown, this does not constitute a specific limitation.

[0098] The memory 250 serves as a carrier for resource storage and can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it include the operating system 251, application programs 253, and data 255, etc., and the storage method can be temporary storage or permanent storage.

[0099] The operating system 251 is used to manage and control the various hardware devices and application programs 253 on the electronic device 2000, so as to enable the central processing unit 270 to perform calculations and processing on the massive data 255 in the memory 250. It can be Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0100] Application 253 is a computer-readable instruction based on operating system 251 that performs at least one specific task, and may include at least one module ( Figure 4 (Not shown), each module may contain computer-readable instructions for the electronic device 2000. For example, the deblurred view synthesis apparatus can be considered as application 253 deployed on the electronic device 2000.

[0101] Data 255 may be signal information, etc., and is stored in memory 250.

[0102] The central processing unit 270 may include one or more processors and is configured to communicate with the memory 250 via at least one communication bus to read computer-readable instructions stored in the memory 250, thereby performing operations and processing on the massive amount of data 255 stored in the memory 250. For example, a deblurred view synthesis method may be performed by the central processing unit 270 reading a series of computer-readable instructions stored in the memory 250.

[0103] Furthermore, the present invention can also be implemented through hardware circuits or a combination of hardware circuits and software. Therefore, the implementation of the present invention is not limited to any specific hardware circuit, software, or combination thereof.

[0104] Please see Figure 5 This invention provides an electronic device 4000, which may include: a desktop computer, a laptop computer, a server, etc., with sensor recognition capabilities.

[0105] exist Figure 5 In this context, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0106] The data interaction between the processor 4001 and the memory 4003 can be achieved through at least one communication bus 4002. This communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0107] Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0108] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0109] The memory 4003 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program instructions or code in the form of instructions or data structures and accessible by the electronic device 4000, but not limited thereto.

[0110] The memory 4003 stores computer-readable instructions, and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002.

[0111] The computer-readable instructions are executed by one or more processors 4001 to implement the deblurred view synthesis method in the above embodiments.

[0112] Furthermore, this embodiment of the invention provides a storage medium storing computer-readable instructions, which are executed by one or more processors to implement the deblurred view synthesis method described above.

[0113] This invention provides a computer program product, which includes computer-readable instructions stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, thereby enabling the electronic device to implement the deblurred view synthesis method as described above.

[0114] Compared with related technologies, the beneficial effects of the present invention are:

[0115] 1. This invention can improve training efficiency; by introducing a spatial perception module and an adaptive three-plane resolution mechanism, compared with the existing methods that require 32-64 sampling poses for camera trajectory, this scheme only requires 8-16 sampling points to achieve comparable reconstruction quality, reducing training time by 50-70%.

[0116] 2. This invention has the effect of optimizing memory usage; by dynamically adjusting the resolution, the average memory usage is reduced by 30-40% compared with the fixed high resolution method, effectively reducing the consumption of computing resources.

[0117] 3. This invention can improve reconstruction quality; in long-trajectory motion blur scenes, the PSNR index is improved by 1-2dB compared with the best existing methods, and the SSIM index is improved by 0.05-0.08, which significantly improves the clarity of image reconstruction.

[0118] 4. This invention features enhanced generalization ability; for unseen scene types and motion patterns, this solution exhibits stronger adaptability and stability, and can effectively handle dynamic blur problems in various complex scenes.

[0119] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0120] The above description is only a partial embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A deblurring view synthesis method characterized by, The method comprises: acquiring scene multi-view data containing dynamic blur, estimating initial camera poses from the multi-view data by three-dimensional reconstruction software, mathematically modeling a motion trajectory of a camera according to the initial camera poses by using a cubic Bezier curve, and setting control points of the Bezier curve as learnable parameters; the control points include a start point, a control point, and an end point; automatically optimizing the learnable parameters by a back propagation algorithm; the start point and the end point correspond to camera poses at a start point and an end point of the motion trajectory respectively, and the control point is used to adjust a bending degree of the Bezier curve to fit a non-linear motion mode of the camera within an exposure time; uniformly selecting a plurality of camera positions on the motion trajectory of the camera as sampling points by Bezier interpolation, inputting the learnable parameters and camera poses of the sampling points into a three-layer feedforward neural network to automatically learn a mapping relationship between a trajectory length and an optimal resolution by supervised learning, and obtaining a prediction level of three-plane resolution; a dimension of a hidden layer of the feedforward neural network is 128, an activation function is ReLU, and an output activation function is Sigmoid; 0 of the prediction level corresponds to a lowest resolution of 128x128, and 1 corresponds to a highest resolution of 512x512; the camera poses include a rotation matrix and a translation vector; down-sampling a full-resolution three-plane to a set resolution according to the prediction level of the three-plane resolution, generating three-plane data of the sampling points, uniformly sampling a plurality of rendering points in three-dimensional space according to the camera poses of the sampling points by a layered sampling strategy through volume rendering technology, and obtaining density and color values of the rendering points according to the three-plane data; calculating the density and the color values of the rendering points by transmittance weighted integration to obtain final RGB pixel values, and generating a clear image without dynamic blur according to the RGB pixel values; averaging clear images of all the sampling points to obtain a synthetic blur image, comparing the synthetic blur image with a real blur image in the multi-view data to obtain a loss value, and optimizing the learnable parameters and a three-plane field to obtain a clear image corresponding to a target camera pose after generating a full-resolution three-plane field.

2. The deblurring view synthesis method of claim 1, wherein, The down-sampling of the full-resolution three-plane to the set resolution according to the prediction level of the three-plane resolution, and the generation of the three-plane data of the sampling points, comprise: performing bilinear interpolation down-sampling on the full-resolution three-plane according to the prediction level of the three-plane resolution to obtain down-sampled three-plane data; the bilinear interpolation down-sampling is used to ensure that sampling points with different trajectory lengths can participate in training at adaptive resolutions.

3. The deblurring view synthesis method of claim 1, wherein, The averaging of the clear images of all the sampling points to obtain the synthetic blur image, and the comparison of the synthetic blur image with the real blur image in the multi-view data to obtain the loss value, comprise: The clear images of all the sampling points are averaged to obtain a synthetic blurred image, and an average absolute error is calculated according to the synthetic blurred image and a real blurred image in the multi-view data; the clear images of the sampling points are obtained by downsampling the motion speed in the exposure time to obtain three planes of different resolutions, and then performing volume rendering to obtain the three planes; The average absolute error is taken as a loss value, and a three-plane field smoothness regularization term and a camera motion trajectory rationality constraint are introduced at the same time; the loss value is used to update the Bezier curve parameters and the three-plane field through back propagation.

4. The deblurring view synthesis method of claim 1, wherein, After the loss value is optimized to obtain the full-resolution three-plane field, a clear image corresponding to a target camera pose is generated, including: The learnable parameters of the Bezier curve and the three-plane field are continuously optimized through a gradient descent algorithm until the full-resolution three-plane field is obtained when the loss value converges; After the target camera pose is obtained, a clear image is generated in real time based on the full-resolution three-plane field according to the target camera pose.

5. A deblurring view synthesis apparatus characterized by comprising: The device is applied to the deblurring view synthesis method of claim 1, and the device includes: A pose estimation and trajectory modeling module is configured to obtain multi-view data of a scene containing dynamic blur, estimate an initial camera pose from the multi-view data by using three-dimensional reconstruction software, model a motion trajectory of the camera based on the initial camera pose by using a cubic Bezier curve, and obtain learnable parameters of the Bezier curve; A trajectory sampling and resolution prediction module is configured to uniformly select a plurality of camera positions as sampling points on the motion trajectory of the camera by using Bezier interpolation, and for each sampling point, obtain a three-plane resolution prediction level of the sampling point from the learnable parameters and a camera pose of the sampling point by using a feedforward neural network; An adaptive downsampling and volume rendering module is configured to downsample a full-resolution three-plane to a set resolution according to the three-plane resolution prediction level, generate three-plane data of the sampling points, and generate a corresponding clear image from the three-plane data and the camera pose by using volume rendering technology; A loss optimization and view synthesis module is configured to average the clear images of all the sampling points to obtain a synthetic blurred image, compare the synthetic blurred image with a real blurred image in the multi-view data to obtain a loss value, and generate a clear image corresponding to a target camera pose from the full-resolution three-plane field obtained by optimizing the learnable parameters and the three-plane field according to the loss value.

6. An electronic device, comprising: including: At least one processor and at least one memory, The memory has computer readable instructions stored thereon; The computer readable instructions are executed by one or more processors to implement the deblurring view synthesis method of any one of claims 1 to 4.

7. A storage medium having stored thereon computer readable instructions, characterized in that, The computer readable instructions are executed by one or more processors to implement the deblurring view synthesis method of any one of claims 1 to 4.

Images