Scene surface reconstruction method, device and equipment based on image motion blur removal and two-dimensional Gaussian point shading and medium

Abstract

The invention relates to a scene surface reconstruction method and device based on image motion blur removal and two-dimensional Gaussian point shading, equipment and a medium. The method comprises the following steps: carrying out gradient back propagation according to luminosity loss, smoothness loss, normal consistency loss and depth consistency loss so as to adjust a camera pose and Gaussian point cloud parameters of a real motion blurred image; based on ray tracing, transparency accumulation and depth regularization constraint, carrying out surface estimation on a clear image corresponding to the real motion blurred image to restore a two-dimensional Gaussian point-share scene of the clear image corresponding to the real motion blurred image, the two-dimensional Gaussian point-share scene comprising an object surface depth and an object surface shape; and performing scene surface reconstruction on a clear image corresponding to the real motion blurred image by adopting a preset truncated signed distance function according to the object surface depth and the object surface shape. According to the method, the authenticity of scene reconstruction can be enhanced, and accurate two-dimensional Gaussian point-share scene reconstruction is realized.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a scene surface reconstruction method based on image motion deblurring and two-dimensional Gaussian point rendering, a corresponding device, an electronic device and a computer-readable storage medium. Background Art

[0002] With the widespread popularity of shooting devices (such as smartphones, drones, self-driving car cameras, etc.), the demand for cameras to capture clear images in dynamic scenes is increasing. However, during the shooting process, the relative motion between the camera and the scene will cause image blur, which is especially obvious in low-light environments, fast-moving scenes or high-resolution video shooting. This blur affects the image quality and makes subsequent computer vision tasks such as image recognition and 3D reconstruction difficult.

[0003] Motion blur is usually caused by the relative motion between the camera and the scene during the exposure process, when the blur kernel (i.e., point spread function, PSF) is spatially non-uniform. In the blind deblurring problem, the goal is to simultaneously estimate the blur kernel and the corresponding sharp image at the pixel level. Early research on deblurring mainly focused on spatially uniform blur in images, assuming that the blur is uniformly distributed throughout the image. However, in real scenes, due to depth of field changes and complex camera motions (such as rolling or shaking), the assumption of uniform blur often does not hold, making deblurring more difficult.

[0004] Recently, many methods have been proposed to deal with the problem of non-uniform deblurring, especially using multi-view information to improve the accuracy of depth estimation. However, existing methods still have shortcomings in dealing with non-uniform blur caused by depth variation, especially when the camera motion trajectory is complex and the scene depth varies significantly, traditional methods find it difficult to accurately reconstruct the scene surface and depth information.

[0005] In summary, there are still deficiencies in adapting to the non-uniform blur caused by depth of field changes in the existing technology. In particular, when the camera motion trajectory is complex and the scene depth changes significantly, traditional methods are difficult to accurately reconstruct the scene surface and depth information. The applicant has made corresponding explorations in order to solve this problem. Summary of the invention

[0006] The purpose of the present application is to solve the above-mentioned problem and to provide a scene surface reconstruction method based on image motion deblurring and two-dimensional Gaussian point rendering, a corresponding device, an electronic device and a computer-readable storage medium.

[0007] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0008] A scene surface reconstruction method based on image motion blur removal and two-dimensional Gaussian point rendering is proposed to meet one of the purposes of this application, including:

[0009] In response to a scene surface reconstruction event, a camera pose and Gaussian point cloud parameters corresponding to a real motion blurred image are obtained, and a preset Bezier curve model is used to fit the camera motion trajectory of the real motion blurred image according to the camera pose corresponding to the real motion blurred image;

[0010] Sampling clear images at different time points on the camera motion trajectory to construct a clear image sequence, and constructing a synthetic motion blurred image according to the camera poses corresponding to the clear images at different time points and the exposure time of the camera motion trajectory, wherein the clear image is a differentiable rendering of a two-dimensional Gaussian point rendering scene;

[0011] Calculate and determine the photometric loss between the synthesized motion blurred image and the real motion blurred image, the smoothness loss, the normal consistency loss, and the depth consistency loss between the clear images of adjacent frames in the clear image sequence, and perform gradient back propagation according to the photometric loss, the smoothness loss, the normal consistency loss, and the depth consistency loss to adjust the camera pose and Gaussian point cloud parameters of the real motion blurred image to determine the clear image corresponding to the real motion blurred image;

[0012] Performing surface estimation on the clear image corresponding to the real motion blurred image based on ray tracing, transparency accumulation, and depth regularization constraints to restore a two-dimensional Gaussian point rendering scene of the clear image corresponding to the real motion blurred image, wherein the two-dimensional Gaussian point rendering scene includes the surface depth and the surface shape of the object;

[0013] A preset truncated signed distance function is used to reconstruct the scene surface of the clear image corresponding to the real motion blurred image according to the surface depth and surface shape of the object, so as to complete the scene surface reconstruction based on image de-motion blurring and two-dimensional Gaussian point rendering.

[0014] Optionally, the luminance loss between the synthesized motion blurred image and the real motion blurred image is expressed as:

[0015] L image =(1-λ dssim )|I blur -I gt |1+λ dssim (1-SSIM(I blur ,I gt )),

[0016] Among them, L imagerepresents the photometric loss λ between the synthetic motion blurred image and the real motion blurred image dssim is the similarity loss weight value, I blur represents the synthetic motion blurred image, I gt Represents a real motion blurred image, SSIM(I blur ,I gt ) is used to measure the similarity between the synthetic motion blurred image and the real motion blurred image. blur -I gt |1 represents the pixel difference between the synthetic motion blurred image and the real motion blurred image, (1-λ dssim )|I blur -I gt |1 represents the pixel-level loss term, λ dssim (1-SSIM(I blur ,I gt )) represents the structural similarity loss term;

[0017] The smoothness loss between the clear images of adjacent frames in the clear image sequence is expressed as:

[0018]

[0019] Among them, L smooth represents the smoothing loss between the clear images of adjacent frames in the clear image sequence, N represents the total number of pixels in the image sequence, i is a pixel index, which represents the position of each pixel in the image, j represents a frame index, which represents a subframe in the clear image sequence, represents the value of the i-th pixel in the j-th frame;

[0020] The normal consistency loss between the clear images of adjacent frames in the clear image sequence is expressed as:

[0021]

[0022] Among them, L normal It is expressed as the normal consistency loss between the clear images of adjacent frames in the clear image sequence, represents the normal rendered at point j of the i-th clear image sequence, represents the normal calculated at point j of the i-th clear image sequence, ω i represents the weight of the i-th clear image sequence;

[0023] The depth consistency loss between the clear images of adjacent frames in the clear image sequence is expressed as:

[0024]

[0025] Among them, L dist Indicates the depth consistency loss between clear images of adjacent frames in the clear image sequence; represents the depth value of the i-th clear image sequence at point j, represents the depth value of the i-th clear image sequence at point k, ω i represents the weight of the i-th clear image sequence, ω j The weight of the i-th clear image sequence at point j, ω k Represents the weight of the i-th clear image sequence at point k.

[0026] Optionally, the step of performing gradient back propagation according to the photometric loss, smoothness loss, normal consistency loss, and depth consistency loss to adjust the camera pose and Gaussian point cloud parameters of the real motion blurred image to determine a clear image corresponding to the real motion blurred image includes:

[0027] Obtaining a first weight corresponding to the photometric loss between the synthesized motion blurred image and the real motion blurred image, a second weight corresponding to the smoothness loss between the clear images of adjacent frames in the clear image sequence, a third weight corresponding to the normal consistency loss between the clear images of adjacent frames in the clear image sequence, and a fourth weight corresponding to the depth consistency loss between the clear images of adjacent frames in the clear image sequence;

[0028] Calculate and determine a first product between the photometric loss and the first weight, a second product between the smoothness loss and the second weight, a third product between the normal consistency loss and the third weight, and a fourth product between the depth consistency loss and the fourth weight;

[0029] Calculating and determining a sum of the first product, the second product, the third product, and the fourth product to determine a total loss between the synthesized motion blurred image and the clear image;

[0030] Gradient back propagation is performed according to the total loss between the synthesized motion blurred image and the clear image to adjust the camera pose and Gaussian point cloud parameters of the real motion blurred image to determine the clear image corresponding to the real motion blurred image.

[0031] Optionally, the total loss between the synthesized motion blurred image and the clear image is expressed as:

[0032] L total =λ image L image +λ smooth(t) L smooth +λnormal(t) L normal +

[0033] λ dist(t) L dist ,

[0034] Among them, λ image represents the first weight corresponding to the photometric loss between the synthesized motion blurred image and the real motion blurred image, λ smooth(t) represents the second weight corresponding to the smoothness loss between the clear images of adjacent frames in the clear image sequence, λ normal(t) represents the third weight corresponding to the normal consistency loss between the clear images of adjacent frames in the clear image sequence, λ dist(t) represents the fourth weight corresponding to the depth consistency loss between the clear images of adjacent frames in the clear image sequence, λ smooth(t) , normal(t) With λ dist(t) It changes with the number of iterations t.

[0035] Optionally, the step of using a preset Bezier curve model to fit the camera motion trajectory of the real motion blurred image according to the camera pose corresponding to the real motion blurred image comprises:

[0036] The expression of the Bezier curve model is:

[0037]

[0038] Among them, T t represents the camera motion trajectory, T j represents the control points of the Bezier curve, represents the basis function of the Bezier curve, which represents the combination coefficient and defines the weight of each control point in the curve, log(T j ) represents the Lie algebra logarithmic mapping of the camera pose, exp() represents the exponential mapping of the Lie group, which means mapping the elements in the Lie algebra back to the elements in the group, and u represents the sampling parameter, which is used to interpolate at different positions on the Bezier curve.

[0039] Optionally, the step of sampling clear images at different time points on the camera motion trajectory to construct a clear image sequence, and constructing a synthetic motion blurred image according to the camera poses corresponding to the clear images at different time points and the exposure time of the camera motion trajectory includes:

[0040] The clear image is fitted into the synthesized motion blurred image, and its expression is:

[0041]

[0042] Among them, B is the synthetic motion blurred image, I is the clear image, and P τ is the camera pose, I(P τ ) represents the camera pose P τ A clear image, where the exposure time τ = [τ o ,τ c ],τ o Indicates the start time of the exposure time interval for motion blur, τ c Indicates the end time of the exposure time interval for motion blur.

[0043] Optionally, the step of performing surface estimation on the clear image corresponding to the real motion blurred image based on ray tracing, transparency accumulation and depth regularization constraint to restore a two-dimensional Gaussian point rendering scene of the clear image corresponding to the real motion blurred image includes:

[0044] The camera emits a ray from each pixel and performs depth estimation along the path of the ray. Each ray intersects with multiple elliptical Gaussian patches, where the Gaussian patches represent the local area of ​​the object. The intersection of the ray and the Gaussian patch will be used as a potential candidate point for surface estimation.

[0045] When the ray intersects with the Gaussian patch, each intersection point has a transparency value. The transparency values ​​of all intersection points are gradually accumulated along the ray direction. When the accumulated transparency reaches 0.5, the intersection point is regarded as the surface of the object to determine the depth value corresponding to the ray, wherein the depth value represents the distance from the object surface to the camera, and the transparency value ranges from 0 to 1;

[0046] A deep regularization constraint is used to constrain the compactness of the distribution between Gaussian patches on the ray and the normals of adjacent Gaussian patches, so that the distribution of Gaussian points tends to be closer to the surface of the object.

[0047] Based on the ray tracing, transparency accumulation and depth regularization constraint, a two-dimensional Gaussian point rendering scene of a clear image corresponding to the real motion blurred image is restored.

[0048] A scene surface reconstruction device based on image motion blur removal and two-dimensional Gaussian point rendering is provided to adapt to another purpose of the present application, comprising:

[0049] A camera trajectory fitting module is configured to respond to a scene surface reconstruction event, obtain a camera pose and Gaussian point cloud parameters corresponding to a real motion blurred image, and use a preset Bezier curve model to fit the camera motion trajectory of the real motion blurred image according to the camera pose corresponding to the real motion blurred image;

[0050] a blurred image synthesis module, configured to sample clear images at different time points on the camera motion trajectory to construct a clear image sequence, and to construct a synthesized motion blurred image according to the camera poses corresponding to the clear images at different time points and the exposure time of the camera motion trajectory, wherein the clear image is a differentiable rendering of a two-dimensional Gaussian point rendering scene;

[0051] A motion deblurring module is configured to calculate and determine the photometric loss between the synthesized motion blurred image and the real motion blurred image, the smoothness loss, the normal consistency loss, and the depth consistency loss between the clear images of adjacent frames in the clear image sequence, and perform gradient back propagation according to the photometric loss, the smoothness loss, the normal consistency loss, and the depth consistency loss to adjust the camera pose and Gaussian point cloud parameters of the real motion blurred image to determine a clear image corresponding to the real motion blurred image;

[0052] a surface estimation module, configured to perform surface estimation on the clear image corresponding to the real motion blurred image based on ray tracing, transparency accumulation and depth regularization constraint, so as to restore a two-dimensional Gaussian point rendering scene of the clear image corresponding to the real motion blurred image, wherein the two-dimensional Gaussian point rendering scene includes the surface depth and the surface shape of the object;

[0053] The scene surface reconstruction module is configured to use a preset truncated signed distance function to reconstruct the scene surface of the clear image corresponding to the real motion blurred image according to the surface depth and surface shape of the object, so as to complete the scene surface reconstruction based on image de-motion blurring and two-dimensional Gaussian point rendering.

[0054] An electronic device provided to meet another purpose of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the scene surface reconstruction method based on image de-motion blur and two-dimensional Gaussian point rendering described in the present application.

[0055] A computer-readable storage medium is provided to meet another purpose of the present application, which stores a computer program implemented according to the scene surface reconstruction method based on image de-motion blur and two-dimensional Gaussian point rendering in the form of computer-readable instructions. When the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

[0056] Compared with the prior art, the present application aims to solve the problem that the prior art still has deficiencies in terms of non-uniform blur caused by depth of field changes, especially when the camera motion trajectory is complex and the scene depth changes significantly, the traditional method is difficult to accurately reconstruct the scene surface and depth information. The present application includes but is not limited to the following beneficial effects:

[0057] This application can accurately reconstruct the scene surface and extract the camera pose and clear image corresponding to the real motion blurred image by combining the Bezier curve model and Gaussian point cloud parameters, thereby improving the image clarity and reconstruction accuracy. The joint optimization of photometric loss, smoothness loss, normal consistency loss and depth consistency loss can effectively eliminate motion blur and accurately restore the clear image sequence. In addition, based on ray tracing and transparency accumulation, combined with depth regularization constraints, the realism of scene reconstruction is enhanced, and finally high-quality deblurring effect and accurate two-dimensional Gaussian point rendering scene reconstruction are achieved, which is suitable for surface reconstruction of dynamic scenes and augmented reality applications, in order to solve the non-uniform blur caused by depth of field changes in traditional technologies. There are still deficiencies, especially when the camera motion trajectory is complex and the scene depth changes significantly. It is difficult for traditional methods to accurately reconstruct the scene surface and depth information. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0059] Figure 1 It is a flow chart of a scene surface reconstruction method based on image motion blur removal and two-dimensional Gaussian point rendering in an embodiment of the present application;

[0060] Figure 2 A schematic diagram of a process for determining a clear image corresponding to a real motion blurred image according to the total loss in an embodiment of the present application;

[0061] Figure 3 A schematic diagram of a process of restoring a two-dimensional Gaussian point rendering scene of a clear image corresponding to a real motion blurred image in an embodiment of the present application;

[0062] Figure 4 It is a principle block diagram of a scene surface reconstruction device based on image motion blur removal and two-dimensional Gaussian point rendering in an embodiment of the present application;

[0063] Figure 5 It is a schematic diagram of the structure of the computer device in the embodiment of the present application. DETAILED DESCRIPTION

[0064] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0065] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0066] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0067] It will be understood by those skilled in the art that the "client", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; conventional laptop and / or palmtop computers or other devices, which have and / or include a conventional laptop and / or palmtop computer or other device with and / or including a radio frequency receiver. The "client", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or adapted and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "client", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, such as a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or may be a smart TV, a set-top box and other devices.

[0068] The hardware referred to by the names such as "server", "client", and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0069] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.

[0070] Unless expressly specified, one or more technical features of the present application can be deployed on a server for implementation and accessed by a client through a remote call to obtain an online service interface provided by the server, or can be directly deployed and run on a client for access.

[0071] The neural network models referenced or may be referenced in this application, unless expressly specified, can be deployed on a remote server and remotely called on the client, or can be deployed and directly called on a client with sufficient device capabilities. In some embodiments, when it runs on the client, its corresponding intelligence can be obtained through transfer learning to reduce the requirements for the client's hardware operating resources and avoid excessive occupation of the client's hardware operating resources.

[0072] Unless explicitly specified, the various data involved in this application can be stored remotely on a server or on a local terminal device, as long as it is suitable for being called by the technical solution of this application.

[0073] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.

[0074] Unless the mutually exclusive relationship between the embodiments to be disclosed in this application is explicitly stated, the relevant technical features involved in each embodiment can be cross-combined to flexibly construct a new embodiment, as long as such combination does not deviate from the creative spirit of this application and can meet the needs of the prior art or solve certain deficiencies in the prior art. Those skilled in the art should be aware of this flexibility.

[0075] See also Figure 1 In one embodiment, the scene surface reconstruction method based on image de-motion blurring and two-dimensional Gaussian point rendering of the present application includes:

[0076] Step S10, in response to a scene surface reconstruction event, obtaining a camera pose and Gaussian point cloud parameters corresponding to the real motion blurred image, and using a preset Bezier curve model to fit the camera motion trajectory of the real motion blurred image according to the camera pose corresponding to the real motion blurred image;

[0077] The image scene surface reconstruction system in the terminal device can respond to the scene surface reconstruction event, obtain the camera pose and Gaussian point cloud parameters corresponding to the real motion blurred image, and use a preset Bezier curve model to fit the camera motion trajectory of the real motion blurred image according to the camera pose corresponding to the real motion blurred image, wherein motion blur is usually due to the relative movement of the camera during the shooting process, resulting in image blur.

[0078] In some embodiments, before using a preset Bezier curve model to fit the camera motion trajectory of the real motion blurred image according to the camera pose corresponding to the real motion blurred image, the motion blurred image is preprocessed using the COLMAP tool to establish a sparse point cloud and camera pose corresponding to the real motion blurred image. COLMAP is a powerful open source image reconstruction and 3D reconstruction tool that is widely used in computer vision, robotics, photogrammetry, and computer graphics. It focuses on reconstructing the camera pose, dense and sparse 3D structure of the scene from a set of images, and is one of the most popular and advanced multi-view stereo vision (MVS) reconstruction tools. COLMAP supports the recovery of camera poses and sparse 3D point clouds of scenes from multiple input images.

[0079] In some embodiments, the step of using a preset Bezier curve model to fit the camera motion trajectory of the real motion blurred image according to the camera pose corresponding to the real motion blurred image includes:

[0080] The expression of the Bezier curve model is:

[0081]

[0082] Among them, T t represents the camera motion trajectory, T j represents the control points of the Bezier curve, represents the basis function of the Bezier curve, which represents the combination coefficient and defines the weight of each control point in the curve, log(T j) represents the Lie algebra logarithmic mapping of the camera pose, exp() represents the exponential mapping of the Lie group, which represents mapping the elements in the Lie algebra back to the elements in the group, and u represents the sampling parameter, which is used to interpolate at different positions on the Bezier curve. Specifically, in the Bezier curve, the parameter u varies between [0,1], indicating the smooth transition of the camera from the initial position to the final position. For each sampling position u, according to the basis function of the Bezier curve Calculate the corresponding camera pose. Then, use the Lie algebra logarithmic mapping to convert these poses into elements in the Lie algebra space, and then convert them back to the actual camera pose through the Lie group exponential mapping. In this way, the system can gradually fit the complete motion trajectory of the camera. After completing the Bezier curve fitting, the system will obtain a smooth camera motion trajectory T t , this trajectory represents the path of the camera's movement during the whole process. This motion trajectory can be used to compensate the original motion blurred image, thereby restoring the clarity of the image for further tasks such as image scene surface reconstruction.

[0083] Step S20, sampling clear images at different time points on the camera motion trajectory to construct a clear image sequence, and constructing a synthetic motion blurred image according to the camera poses corresponding to the clear images at different time points and the exposure time of the camera motion trajectory, wherein the clear image is a differentiable rendering image of a two-dimensional Gaussian point rendering scene;

[0084] A preset Bezier curve model is used to fit the camera motion trajectory of the real motion blurred image according to the camera pose corresponding to the real motion blurred image, and then clear images at different time points are sampled on the camera motion trajectory to construct a clear image sequence, and a synthetic motion blurred image is constructed according to the camera poses corresponding to the clear images at different time points and the exposure time of the camera motion trajectory, wherein the clear image is a differentiable rendering of a two-dimensional Gaussian point rendering scene;

[0085] In some embodiments, the step of sampling clear images at different time points on the camera motion trajectory to construct a clear image sequence, and constructing a synthetic motion blurred image according to the camera poses corresponding to the clear images at different time points and the exposure time of the camera motion trajectory includes:

[0086] The clear image is fitted into the synthesized motion blurred image, and its expression is:

[0087]

[0088] Among them, B is the synthetic motion blurred image, I is the clear image, and P τ is the camera pose, I(P τ) represents the camera pose P τ A clear image, where the exposure time τ = [τ o ,τ c ],τ o Indicates the start time of the exposure time interval for motion blur, τ c Indicates the end time of the exposure time interval for motion blur.

[0089] Specifically, after fitting the camera motion trajectory of the real motion blurred image using a preset Bezier curve model according to the camera pose corresponding to the real motion blurred image, the corresponding camera pose can be determined by different time points on the trajectory. Based on the fitted camera motion trajectory, clear images at different time points are sampled on the motion trajectory, and these time points represent clear images at different exposure times. The camera pose corresponding to each clear image sequence is based on the sampling point of the motion trajectory. Since motion blur is caused by the rapid movement of the camera, multiple clear images are superimposed to generate a motion blurred image. It should be noted that these clear images are modeled using two-dimensional Gaussian point rendering scene rendering to ensure that they are differentiable renderings.

[0090] Step S30, calculating and determining the photometric loss between the synthesized motion blurred image and the real motion blurred image, the smoothness loss, the normal consistency loss, and the depth consistency loss between the clear images of adjacent frames in the clear image sequence, and performing gradient back propagation according to the photometric loss, the smoothness loss, the normal consistency loss, and the depth consistency loss to adjust the camera pose and Gaussian point cloud parameters of the real motion blurred image to determine the clear image corresponding to the real motion blurred image;

[0091] Sampling clear images at different time points on the camera motion trajectory to construct a clear image sequence, and after constructing a synthetic motion blurred image according to the camera poses corresponding to the clear images at different time points and the exposure time of the camera motion trajectory, calculating and determining the photometric loss between the synthetic motion blurred image and the real motion blurred image, the smoothness loss, the normal consistency loss, and the depth consistency loss between the clear images of adjacent frames in the clear image sequence, performing gradient back propagation according to the photometric loss, the smoothness loss, the normal consistency loss, and the depth consistency loss to adjust the camera pose and Gaussian point cloud parameters of the real motion blurred image, so as to determine the clear image corresponding to the real motion blurred image;

[0092] In some embodiments, the luminance loss between the synthesized motion blurred image and the real motion blurred image is expressed as:

[0093] L image =(1-λdssim )|I blur -I gt |1+λ dssim (1-SSIM(I blur ,I gt )),

[0094] Among them, L image represents the photometric loss between the synthetic motion blurred image and the real motion blurred image, λ dssim is the similarity loss weight value, I blur represents the synthetic motion blurred image, I gt Represents a real motion blurred image, SSIM(I blur ,I gt ) is used to measure the similarity between the synthetic motion blurred image and the real motion blurred image. blur -I gt |1 represents the pixel difference between the synthetic motion blurred image and the real motion blurred image, (1-λ dssim )|I blur -I gt |1 represents the pixel-level loss term, λ dssim (1-SSIM(I blur ,I gt )) represents the structural similarity loss term;

[0095] The smoothness loss between the clear images of adjacent frames in the clear image sequence is expressed as:

[0096]

[0097] Among them, L smooth represents the smoothing loss between the clear images of adjacent frames in the clear image sequence, N represents the total number of pixels in the image sequence, i is a pixel index, which represents the position of each pixel in the image, j represents a frame index, which represents a subframe in the clear image sequence, represents the value of the i-th pixel in the j-th frame;

[0098] The normal consistency loss between the clear images of adjacent frames in the clear image sequence is expressed as:

[0099]

[0100] Among them, L normal It is expressed as the normal consistency loss between the clear images of adjacent frames in the clear image sequence, represents the normal rendered at point j of the i-th clear image sequence, represents the normal calculated at point j of the i-th clear image sequence, ωi represents the weight of the i-th clear image sequence;

[0101] The depth consistency loss between the clear images of adjacent frames in the clear image sequence is expressed as:

[0102]

[0103] Among them, L dist Indicates the depth consistency loss between clear images of adjacent frames in the clear image sequence; represents the depth value of the i-th clear image sequence at point j, represents the depth value of the i-th clear image sequence at point k, ω i represents the weight of the i-th clear image sequence, ω j The weight of the i-th clear image sequence at point j, ω k Represents the weight of the i-th clear image sequence at point k.

[0104] For further examples, see Figure 2 , performing gradient back propagation according to the photometric loss, smoothness loss, normal consistency loss, and depth consistency loss to adjust the camera pose and Gaussian point cloud parameters of the real motion blurred image to determine a clear image corresponding to the real motion blurred image, comprising:

[0105] Step S301, obtaining a first weight corresponding to the photometric loss between the synthesized motion blurred image and the real motion blurred image, a second weight corresponding to the smoothness loss between the clear images of adjacent frames in the clear image sequence, a third weight corresponding to the normal consistency loss between the clear images of adjacent frames in the clear image sequence, and a fourth weight corresponding to the depth consistency loss between the clear images of adjacent frames in the clear image sequence;

[0106] Step S302, calculating and determining a first product between the photometric loss and the first weight, a second product between the smoothness loss and the second weight, a third product between the normal consistency loss and the third weight, and a fourth product between the depth consistency loss and the fourth weight;

[0107] Step S303: Calculate and determine the sum of the first product, the second product, the third product and the fourth product to determine the total loss between the synthesized motion blurred image and the clear image;

[0108] Step S304: performing gradient back propagation according to the total loss between the synthesized motion blurred image and the clear image to adjust the camera pose and Gaussian point cloud parameters of the real motion blurred image to determine the clear image corresponding to the real motion blurred image.

[0109] Specifically, the total loss between the synthesized motion blurred image and the clear image is expressed as:

[0110] L total =λ image L image +λ smooth(t) L smooth +λ normal(t) L normal +

[0111] λ dist(t) L dist ,

[0112] Among them, λ image represents the first weight corresponding to the photometric loss between the synthesized motion blurred image and the real motion blurred image, λ smooth(t) represents the second weight corresponding to the smoothness loss between the clear images of adjacent frames in the clear image sequence, λ normal(t) represents the third weight corresponding to the normal consistency loss between the clear images of adjacent frames in the clear image sequence, λ dist(t) represents the fourth weight corresponding to the depth consistency loss between the clear images of adjacent frames in the clear image sequence, λ smooth(t) , normal(t) With λ dist(t) It changes with the number of iterations t.

[0113] More specifically, once the total loss L is calculated total , gradient back propagation can be performed. Through the back propagation algorithm, the camera pose and Gaussian point cloud parameters in the network are optimized according to the total loss. The specific process is as follows: Calculate the gradient of the camera pose and point cloud parameters in the model according to the total loss. Update the camera pose and point cloud parameters through optimization algorithms (such as SGD, Adam, etc.) to reduce the value of the loss function. Repeat the gradient back propagation multiple times, update the parameters each iteration, and gradually reduce the motion blur. Through the iterative optimization of the above process, the best match between the synthesized motion blurred image and the clear image can be determined, and the corresponding clear image can be restored. This process simultaneously optimizes the parameters of the camera pose and the three-dimensional point cloud, so that the quality of the clear image is gradually improved. After several optimization iterations, a clear image with motion blur removed is finally obtained, and the result is output. At this point, all loss terms have been effectively optimized.

[0114] Step S40, performing surface estimation on the clear image corresponding to the real motion blurred image based on ray tracing, transparency accumulation and depth regularization constraint to restore a two-dimensional Gaussian point rendering scene of the clear image corresponding to the real motion blurred image, wherein the two-dimensional Gaussian point rendering scene includes the surface depth and the surface shape of the object;

[0115] Performing gradient back propagation according to the photometric loss, smoothness loss, normal consistency loss, and depth consistency loss to adjust the camera pose and Gaussian point cloud parameters of the real motion blurred image to determine a clear image corresponding to the real motion blurred image, and then performing surface estimation on the clear image corresponding to the real motion blurred image based on ray tracing, transparency accumulation, and depth regularization constraints to restore a two-dimensional Gaussian point rendering scene of the clear image corresponding to the real motion blurred image, wherein the two-dimensional Gaussian point rendering scene includes object surface depth and object surface shape;

[0116] In some embodiments, see Figure 3 The step of performing surface estimation on the clear image corresponding to the real motion blurred image based on ray tracing, transparency accumulation and depth regularization constraint to restore a two-dimensional Gaussian point rendering scene of the clear image corresponding to the real motion blurred image comprises:

[0117] Step S401: The camera emits a ray from each pixel and performs depth estimation along the path of the ray. Each ray intersects with a plurality of elliptical Gaussian patches, wherein the Gaussian patches represent the local area of ​​the object, and the intersection points of the ray and the Gaussian patches are used as potential candidate points for surface estimation.

[0118] Step S402: When the ray intersects with the Gaussian patch, each intersection point has a transparency value. The transparency values ​​of all intersection points are gradually accumulated along the ray direction. When the accumulated transparency reaches 0.5, the intersection point is regarded as the surface of the object to determine the depth value corresponding to the ray, wherein the depth value represents the distance from the object surface to the camera, and the transparency value ranges from 0 to 1.

[0119] Step S403: using a depth regularization constraint to constrain the compactness of the distribution of Gaussian patches on the ray and to constrain the normals of adjacent Gaussian patches, so that the distribution of Gaussian points tends to be closer to the surface of the object;

[0120] Step S404: Based on the ray tracing, transparency accumulation and depth regularization constraint, a two-dimensional Gaussian point rendering scene of a clear image corresponding to the real motion blurred image is restored.

[0121] This embodiment effectively extracts the geometric information of the object surface from the real motion blurred image through ray tracing, transparency accumulation and depth regularization constraints, and restores the two-dimensional Gaussian point rendering scene of the clear image. This method combines the accuracy of ray tracing, the feasibility of transparency accumulation and the optimization ability of depth regularization, which can significantly enhance the accuracy and efficiency of scene surface reconstruction.

[0122] In some embodiments, a Gaussian patch generally refers to a Gaussian function that performs weight distribution on a given area to achieve the purpose of smoothing, filtering or approximating the area.

[0123] Step S50, using a preset truncated signed distance function to reconstruct the scene surface of the clear image corresponding to the real motion blurred image according to the surface depth and surface shape of the object, so as to complete the scene surface reconstruction based on image de-motion blurring and two-dimensional Gaussian point rendering.

[0124] Based on ray tracing, transparency accumulation and depth regularization constraints, the surface of the clear image corresponding to the real motion blurred image is estimated to restore the two-dimensional Gaussian point rendering scene of the clear image corresponding to the real motion blurred image. Then, a preset truncated signed distance function is used to reconstruct the scene surface of the clear image corresponding to the real motion blurred image according to the object surface depth and the object surface shape, so as to complete the scene surface reconstruction based on image de-motion blurring and two-dimensional Gaussian point rendering. Among them, the object surface depth refers to the spatial distance between each point on the object surface and the camera. The object surface shape refers to the geometric characteristics of each point on the object surface, such as curvature, shape contour, etc.

[0125] Specifically, the truncated signed distance function (TSDF) in the Open3D library is used to reconstruct the surface of the scene. TSDF is widely used in 3D reconstruction. Its main function is to reconstruct the complete three-dimensional object surface by calculating the distance to the nearest surface for each point in the scene. The truncated signed distance function (TSDF) avoids excessive noise and unnecessary calculations by limiting the distance of each point within a certain threshold, thereby improving the efficiency and accuracy of the reconstruction process. The surface depth and surface shape of the object will be used as input, and the truncated signed distance function (TSDF) will calculate the three-dimensional reconstruction data of the scene based on the surface depth and surface shape of the object. Finally, an accurate three-dimensional model can be obtained, which represents the clear scene after removing motion blur.

[0126] It can be seen from the above embodiments that, compared with the prior art, the present application still has deficiencies in the prior art in terms of non-uniform blur caused by depth of field changes, especially when the camera motion trajectory is complex and the scene depth changes significantly, the traditional method is difficult to accurately reconstruct the scene surface and depth information, and the present application includes but is not limited to the following beneficial effects:

[0127] This application can accurately reconstruct the scene surface and extract the camera pose and clear image corresponding to the real motion blurred image by combining the Bezier curve model and Gaussian point cloud parameters, thereby improving the image clarity and reconstruction accuracy. The joint optimization of photometric loss, smoothness loss, normal consistency loss and depth consistency loss can effectively eliminate motion blur and accurately restore the clear image sequence. In addition, based on ray tracing and transparency accumulation, combined with depth regularization constraints, the realism of scene reconstruction is enhanced, and finally high-quality deblurring effect and accurate two-dimensional Gaussian point rendering scene reconstruction are achieved, which is suitable for surface reconstruction of dynamic scenes and augmented reality applications, in order to solve the non-uniform blur caused by depth of field changes in traditional technologies. There are still deficiencies, especially when the camera motion trajectory is complex and the scene depth changes significantly. It is difficult for traditional methods to accurately reconstruct the scene surface and depth information.

[0128] See also Figure 4A scene surface reconstruction device based on image motion deblurring and two-dimensional Gaussian point rendering is provided to meet one of the purposes of the present application, including a camera trajectory fitting module 1100, a blurred image synthesis module 1200, a motion deblurring module 1300, a surface estimation module 1400 and a scene surface reconstruction module 1500. Among them, the camera trajectory fitting module 1100 is configured to respond to the scene surface reconstruction event, obtain the camera pose and Gaussian point cloud parameters corresponding to the real motion blurred image, and use a preset Bezier curve model to fit the camera motion trajectory of the real motion blurred image according to the camera pose corresponding to the real motion blurred image; the blurred image synthesis module 1200 is configured to sample clear images at different time points on the camera motion trajectory to construct a clear image sequence, and construct a synthesized motion blurred image according to the camera poses corresponding to the clear images at different time points and the exposure time of the camera motion trajectory, wherein the clear image is a differentiable rendering of a two-dimensional Gaussian point rendering scene; the motion deblurring module 1300 is configured to calculate and determine the photometric loss between the synthesized motion blurred image and the real motion blurred image, the smoothness loss, the normal consistency loss and the depth loss between the clear images of adjacent frames in the clear image sequence. A luminosity consistency loss is used, and gradient back propagation is performed according to the luminosity loss, smoothness loss, normal consistency loss, and depth consistency loss to adjust the camera pose and Gaussian point cloud parameters of the real motion blurred image to determine a clear image corresponding to the real motion blurred image; a surface estimation module 1400 is configured to perform surface estimation on the clear image corresponding to the real motion blurred image based on ray tracing, transparency accumulation, and depth regularization constraints to restore a two-dimensional Gaussian point rendering scene of the clear image corresponding to the real motion blurred image, wherein the two-dimensional Gaussian point rendering scene includes an object surface depth and an object surface shape; a scene surface reconstruction module 1500 is configured to perform scene surface reconstruction on the clear image corresponding to the real motion blurred image according to the object surface depth and the object surface shape using a preset truncated signed distance function to complete scene surface reconstruction based on image de-motion blurring and two-dimensional Gaussian point rendering.

[0129] Based on any embodiment of this application, please refer to Figure 5 Another embodiment of the present application further provides an electronic device, which can be implemented by a computer device, such as Figure 5As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a scene surface reconstruction method based on image de-motion blurring and two-dimensional Gaussian point rendering. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the scene surface reconstruction method based on image de-motion blurring and two-dimensional Gaussian point rendering of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0130] In this embodiment, the processor is used to execute the specific functions of each module and its submodule in 4, and the memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the scene surface reconstruction device based on image de-motion blur and two-dimensional Gaussian point rendering of this application, and the server can call the program code and data of the server to execute the functions of all submodules.

[0131] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the scene surface reconstruction method based on image de-motion blur and two-dimensional Gaussian point rendering described in any embodiment of the present application.

[0132] The present application also provides a computer program product, including a computer program / instruction, which, when executed by one or more processors, implements the steps of the scene surface reconstruction method based on image de-motion blurring and two-dimensional Gaussian point rendering described in any embodiment of the present application.

[0133] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0134] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

[0135] In summary, this application can accurately reconstruct the scene surface and extract the camera pose and clear image corresponding to the real motion blurred image by combining the Bezier curve model and Gaussian point cloud parameters, thereby improving the image clarity and reconstruction accuracy. The joint optimization of photometric loss, smoothness loss, normal consistency loss and depth consistency loss can effectively eliminate motion blur and accurately restore the clear image sequence. In addition, based on ray tracing and transparency accumulation, combined with depth regularization constraints, the realism of scene reconstruction is enhanced, and finally high-quality deblurring effect and accurate two-dimensional Gaussian point rendering scene reconstruction are achieved, which is suitable for surface reconstruction of dynamic scenes and augmented reality applications, in order to solve the non-uniform blur caused by depth of field changes in traditional technologies. There are still deficiencies, especially when the camera motion trajectory is complex and the scene depth changes significantly. It is difficult for traditional methods to accurately reconstruct the scene surface and depth information.

Claims

1. A scene surface reconstruction method based on image motion blur removal and two-dimensional Gaussian point rendering, characterized in that: include: In response to a scene surface reconstruction event, a camera pose and Gaussian point cloud parameters corresponding to a real motion blurred image are obtained, and a preset Bezier curve model is used to fit the camera motion trajectory of the real motion blurred image according to the camera pose corresponding to the real motion blurred image; Sampling clear images at different time points on the camera motion trajectory to construct a clear image sequence, and constructing a synthetic motion blurred image according to the camera poses corresponding to the clear images at different time points and the exposure time of the camera motion trajectory, wherein the clear image is a differentiable rendering of a two-dimensional Gaussian point rendering scene; Calculate and determine the photometric loss between the synthesized motion blurred image and the real motion blurred image, the smoothness loss, the normal consistency loss, and the depth consistency loss between the clear images of adjacent frames in the clear image sequence, and perform gradient back propagation according to the photometric loss, the smoothness loss, the normal consistency loss, and the depth consistency loss to adjust the camera pose and Gaussian point cloud parameters of the real motion blurred image to determine the clear image corresponding to the real motion blurred image; Performing surface estimation on the clear image corresponding to the real motion blurred image based on ray tracing, transparency accumulation, and depth regularization constraints to restore a two-dimensional Gaussian point rendering scene of the clear image corresponding to the real motion blurred image, wherein the two-dimensional Gaussian point rendering scene includes the surface depth and the surface shape of the object; A preset truncated signed distance function is used to reconstruct the scene surface of the clear image corresponding to the real motion blurred image according to the surface depth and surface shape of the object, so as to complete the scene surface reconstruction based on image de-motion blurring and two-dimensional Gaussian point rendering.

2. The scene surface reconstruction method based on image motion deblurring and two-dimensional Gaussian point rendering according to claim 1, characterized in that: The luminance loss between the synthesized motion blurred image and the real motion blurred image is expressed as: L image =(1-λ dssim )|I blur -I gt |1+λ dssim (1-SSIM(I blur ,I gt )), Among them, L image represents the photometric loss between the synthetic motion blurred image and the real motion blurred image, λ dssim is the similarity loss weight value, I blur represents the synthetic motion blurred image, I gt Represents a real motion blurred image, SSIM(I blur ,I gt ) is used to measure the similarity between the synthesized motion blurred image and the real motion blurred image. blur -I gt |1 represents the pixel difference between the synthetic motion blurred image and the real motion blurred image, (1-λ dssim )|I blur -I gt |1 represents the pixel-level loss term, λ dssim (1-SSIM(I blur ,I gt )) represents the structural similarity loss term; The smoothness loss between the clear images of adjacent frames in the clear image sequence is expressed as: Among them, L smooth represents the smoothing loss between the clear images of adjacent frames in the clear image sequence, N represents the total number of pixels in the image sequence, i is a pixel index, which represents the position of each pixel in the image, j represents a frame index, which represents a subframe in the clear image sequence, represents the value of the i-th pixel in the j-th frame; The normal consistency loss between the clear images of adjacent frames in the clear image sequence is expressed as: Among them, L normal It is expressed as the normal consistency loss between the clear images of adjacent frames in the clear image sequence, represents the normal rendered at point j of the i-th clear image sequence, represents the normal calculated at point j of the i-th clear image sequence, ω i represents the weight of the i-th clear image sequence; The depth consistency loss between the clear images of adjacent frames in the clear image sequence is expressed as: Among them, L dist Indicates the depth consistency loss between clear images of adjacent frames in the clear image sequence; represents the depth value of the i-th clear image sequence at point j, represents the depth value of the i-th clear image sequence at point k, ω i represents the weight of the i-th clear image sequence, ω j The weight of the i-th clear image sequence at point j, ω k Represents the weight of the i-th clear image sequence at point k.

3. The scene surface reconstruction method based on image motion blur removal and two-dimensional Gaussian point rendering according to claim 1, characterized in that: The step of performing gradient back propagation according to the photometric loss, smoothness loss, normal consistency loss and depth consistency loss to adjust the camera pose and Gaussian point cloud parameters of the real motion blurred image to determine a clear image corresponding to the real motion blurred image includes: Obtaining a first weight corresponding to the photometric loss between the synthesized motion blurred image and the real motion blurred image, a second weight corresponding to the smoothness loss between the clear images of adjacent frames in the clear image sequence, a third weight corresponding to the normal consistency loss between the clear images of adjacent frames in the clear image sequence, and a fourth weight corresponding to the depth consistency loss between the clear images of adjacent frames in the clear image sequence; Calculate and determine a first product between the photometric loss and the first weight, a second product between the smoothness loss and the second weight, a third product between the normal consistency loss and the third weight, and a fourth product between the depth consistency loss and the fourth weight; Calculating and determining a sum of the first product, the second product, the third product, and the fourth product to determine a total loss between the synthesized motion blurred image and the clear image; Gradient back propagation is performed according to the total loss between the synthesized motion blurred image and the clear image to adjust the camera pose and Gaussian point cloud parameters of the real motion blurred image to determine the clear image corresponding to the real motion blurred image.

4. The scene surface reconstruction method based on image motion blur removal and two-dimensional Gaussian point rendering according to claim 3, characterized in that: The total loss between the synthesized motion blurred image and the sharp image is expressed as: L total =λ image L image +λ smooth(t) L smooth +λ normal(t) L normal + l dist(t) L dist , Among them, λ image represents the first weight corresponding to the photometric loss between the synthesized motion blurred image and the real motion blurred image, λ smooth(t) represents the second weight corresponding to the smoothness loss between the clear images of adjacent frames in the clear image sequence, λ normal(t) represents the third weight corresponding to the normal consistency loss between the clear images of adjacent frames in the clear image sequence, λ dist(t) represents the fourth weight corresponding to the depth consistency loss between the clear images of adjacent frames in the clear image sequence, λ smooth(t) , normal(t) With λ dist(t) It changes with the number of iterations t.

5. The scene surface reconstruction method based on image motion blur removal and two-dimensional Gaussian point rendering according to claim 1, characterized in that: The step of using a preset Bezier curve model to fit the camera motion trajectory of the real motion blurred image according to the camera pose corresponding to the real motion blurred image comprises: The expression of the Bezier curve model is: Among them, T t represents the camera motion trajectory, T j represents the control points of the Bezier curve, represents the basis function of the Bezier curve, which represents the combination coefficient and defines the weight of each control point in the curve, log(T j ) represents the Lie algebra logarithmic mapping of the camera pose, exp() represents the exponential mapping of the Lie group, which means mapping the elements in the Lie algebra back to the elements in the group, and u represents the sampling parameter, which is used to interpolate at different positions on the Bezier curve.

6. The scene surface reconstruction method based on image motion blur removal and two-dimensional Gaussian point rendering according to claim 1, characterized in that: The steps of sampling clear images at different time points on the camera motion trajectory to construct a clear image sequence, and constructing a synthetic motion blurred image according to the camera poses corresponding to the clear images at different time points and the exposure time of the camera motion trajectory include: The clear image is fitted into the synthesized motion blurred image, and its expression is: Among them, B is the synthetic motion blurred image, I is the clear image, and P τ is the camera pose, I(P τ ) represents the camera pose P τ A clear image, where the exposure time τ = [τ o ,τ c ],τ o Indicates the start time of the exposure time interval for motion blur, τ c Indicates the end time of the exposure time interval for motion blur.

7. The scene surface reconstruction method based on image motion blur removal and two-dimensional Gaussian point rendering according to any one of claims 1 to 6, characterized in that: The step of performing surface estimation on the clear image corresponding to the real motion blurred image based on ray tracing, transparency accumulation and depth regularization constraint to restore a two-dimensional Gaussian point rendering scene of the clear image corresponding to the real motion blurred image includes: The camera emits a ray from each pixel and performs depth estimation along the path of the ray. Each ray intersects with multiple elliptical Gaussian patches, where the Gaussian patches represent the local area of ​​the object. The intersection of the ray and the Gaussian patch will be used as a potential candidate point for surface estimation. When the ray intersects with the Gaussian patch, each intersection point has a transparency value. The transparency values ​​of all intersection points are gradually accumulated along the ray direction. When the accumulated transparency reaches 0.5, the intersection point is regarded as the surface of the object to determine the depth value corresponding to the ray, wherein the depth value represents the distance from the object surface to the camera, and the transparency value ranges from 0 to 1; A deep regularization constraint is used to constrain the compactness of the distribution between Gaussian patches on the ray and the normals of adjacent Gaussian patches, so that the distribution of Gaussian points tends to be closer to the surface of the object. Based on the ray tracing, transparency accumulation and depth regularization constraint, a two-dimensional Gaussian point rendering scene of a clear image corresponding to the real motion blurred image is restored.

8. A scene surface reconstruction device based on image motion blur removal and two-dimensional Gaussian point rendering, characterized in that: include: A camera trajectory fitting module is configured to respond to a scene surface reconstruction event, obtain a camera pose and Gaussian point cloud parameters corresponding to a real motion blurred image, and use a preset Bezier curve model to fit the camera motion trajectory of the real motion blurred image according to the camera pose corresponding to the real motion blurred image; a blurred image synthesis module, configured to sample clear images at different time points on the camera motion trajectory to construct a clear image sequence, and to construct a synthesized motion blurred image according to the camera poses corresponding to the clear images at different time points and the exposure time of the camera motion trajectory, wherein the clear image is a differentiable rendering of a two-dimensional Gaussian point rendering scene; A motion deblurring module is configured to calculate and determine the photometric loss between the synthesized motion blurred image and the real motion blurred image, the smoothness loss, the normal consistency loss, and the depth consistency loss between the clear images of adjacent frames in the clear image sequence, and perform gradient back propagation according to the photometric loss, the smoothness loss, the normal consistency loss, and the depth consistency loss to adjust the camera pose and Gaussian point cloud parameters of the real motion blurred image to determine a clear image corresponding to the real motion blurred image; a surface estimation module, configured to perform surface estimation on the clear image corresponding to the real motion blurred image based on ray tracing, transparency accumulation and depth regularization constraint, so as to restore a two-dimensional Gaussian point rendering scene of the clear image corresponding to the real motion blurred image, wherein the two-dimensional Gaussian point rendering scene includes the surface depth and the surface shape of the object; The scene surface reconstruction module is configured to use a preset truncated signed distance function to reconstruct the scene surface of the clear image corresponding to the real motion blurred image according to the surface depth and surface shape of the object, so as to complete the scene surface reconstruction based on image de-motion blurring and two-dimensional Gaussian point rendering.

9. An electronic device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images