A multi-view image deblurring method, apparatus, system, and electronic medium

Abstract

Embodiments of the present disclosure disclose a multi-view image deblurring method, device, system and electronic medium. A specific embodiment of the method comprises: first, constructing a camera pose estimation framework through event stream guidance, thereby obtaining an accurate shooting pose of a blurred image as a NeRF input; then, further simulating a real blurring process through a blurring rendering loss and an event rendering loss, thereby better realizing three-dimensional representation learning of an object or a scene. This embodiment effectively utilizes the internal relationship between events and images, not only realizing image deblurring, but also realizing high-quality new view image rendering.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the field of three-dimensional reconstruction of neural radiance fields and event cameras, and in particular to a multi-view image deblurring method, device, system and electronic medium. BACKGROUND

[0002] With the advent of neural radiance fields (NeRF), neural 3D representation and novel view image synthesis tasks have made great progress in the past few years. NeRF takes 3D position and 2D view direction as input, and uses multi-view images of an object or scene as supervision to learn a neural 3D representation, which is parameterized as a multi-layer perceptron (MLP). In order to generate new view images, NeRF uses voxel rendering technology to render each pixel using network output (color and density), so that high-fidelity and high-restoration images are rendered.

[0003] The premise for NeRF to achieve good results is that the input image quality is high enough (no blur, sufficient light), but in many real-world scenarios, it is difficult to obtain high-quality images. For example, handheld camera shaking will produce motion blur in the captured photo, especially in low-light scenes, and it is necessary to extend the exposure time to obtain a photo that meets the viewing habits of the human eye, which makes it more likely for the captured image to be blurred. Deblur NeRF locates this problem and proposes Deformable Sparse Kernel to simulate the image blurring process, but when the camera appears consistent blur in all views or the blur is very serious, this method will fail.

[0004] Compared with methods that only rely on blurred images, it is more effective to combine additional information to guide the neural radiance field learning process. Event cameras are a new type of bionic visual sensor that asynchronously measures the luminance change of each pixel. Compared with traditional frame-based cameras, event cameras can record scene information at high temporal resolution, which is important for modeling the blurring process. Therefore, event-based image deblurring has become a popular research topic in recent years.

[0005] This invention introduces event streaming into the learning process of neural radiation fields, proposing ENeRF to learn sharp 3D representations of objects and scenes using blurred images and their corresponding event data. Based on the NeRF framework, firstly, the pose sequence of the blurred image is estimated using events. During training, a synthesized blurred image can be rendered using the aforementioned pose sequence and compared with the input blurred image to obtain a rendering loss. Furthermore, event generation is calculated as pose changes, and an event loss is constructed using actual event data as supervision to optimize the learning of the neural radiation field. Due to the enhancement of event data, some embodiments of this disclosure can learn a sharp NeRF even with a blurred input image, achieving not only deblurring of the input image but also high-quality generation of new perspective images when the input image quality is degraded. Summary of the Invention

[0006] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0007] Some embodiments of this disclosure provide multi-view image deblurring methods, apparatuses, systems, and electronic media to address one or more of the technical problems mentioned in the background section above.

[0008] Based on the aforementioned practical needs and key issues, the purpose of some embodiments of this disclosure is to propose a multi-viewpoint image deblurring method, which inputs a blurred image and event stream from multiple perspectives of an object or scene, performs initial pose estimation with event assistance, and uses events and blurred images for joint reinforcement supervision, ultimately learning a clear NeRF, thereby achieving input image deblurring and high-quality new perspective image generation.

[0009] In a first aspect, some embodiments of this disclosure provide a multi-viewpoint image deblurring method, which includes: Step S100, for a scene or object, given multiple blurred image inputs from various viewpoints, and event data captured within the exposure time of the blurred image. For the blurred image and event data at each viewpoint, the event data is grouped equally by time, and then an initial deblurred image is recovered using an EDI algorithm. Step S200, all the obtained initial deblurred images are input into a COLMAP tool to obtain the shooting pose of the initial deblurred image at each viewpoint. Step S300, for each viewpoint, the obtained pose is input into a NeRF network to render multiple clear images at the aforementioned viewpoints, and a blurred image is synthesized using the multiple clear images, with the difference from the original input blurred image used as the blur loss function; simultaneously, using the multiple clear images, the same pixel point at different times is randomly selected, the theoretical value of the event data is calculated, and the difference from the real event data is used as the event loss function, and the NeRF network parameters are adjusted through backpropagation to learn the three-dimensional implicit representation of the scene or object. In step S400, given the camera pose, input the trained NeRF network to render a clear image of the scene or object in that pose.

[0010] Secondly, some embodiments of this disclosure provide a multi-viewpoint image deblurring method apparatus, the apparatus comprising: a first input unit configured to, for a scene or object, input multiple blurred images from multiple viewpoints, and event data within the exposure time of capturing the blurred images; for each blurred image and event data from each viewpoint, grouping the event data equally by time, and then using an EDI algorithm to recover an initial deblurred image; a second input unit configured to input all the obtained initial deblurred images into a COLMAP tool to obtain the shooting pose of the initial deblurred image from each viewpoint; and a third input unit configured to, for each... From a given viewpoint, the obtained pose is input into the NeRF network to render multiple clear images from that viewpoint. A blurred image is synthesized from these multiple clear images, and the difference between the blurred image and the original input blurred image is used as the blur loss function. Simultaneously, using the multiple clear images, the same pixel at different times is randomly selected to calculate the theoretical value of the event data, and the difference between the theoretical value and the actual event data is used as the event loss function. Backpropagation is used to adjust the parameters of the NeRF network to learn the 3D implicit representation of the scene or object. The fourth input unit is configured to be given a camera pose and input into the trained NeRF network to render a clear image of the scene or object in that pose.

[0011] Thirdly, some embodiments of this disclosure provide an electronic system, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0012] Fourthly, some embodiments of this disclosure provide a computer electronic medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above. Attached Figure Description

[0013] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0014] Figure 1 This is a flowchart of some embodiments of the multi-view image deblurring method according to the present disclosure;

[0015] Figure 2 This is a general flowchart of some embodiments of the multi-view image deblurring method according to the present disclosure;

[0016] Figure 3 This is a schematic diagram of a NeRF network module according to some embodiments of the multi-view image deblurring method of this disclosure;

[0017] Figure 4 This is a flowchart of step S300 of some embodiments of the multi-view image deblurring method according to the present disclosure.

[0018] Figure 5 This is a schematic diagram of the structure of some embodiments of the multi-view image deblurring apparatus according to the present disclosure;

[0019] Figure 6 This is a schematic diagram of the structure of an electronic system suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0020] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0021] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0022] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0023] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0024] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0025] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0026] Figure 1 A flow 100 of some embodiments of a multi-view image deblurring method according to the present disclosure is shown. The multi-view image deblurring method includes the following steps:

[0027] Step S100: For a scene or object, given multiple blurred image inputs from different viewpoints, and event data within the exposure time of capturing the blurred image. For each blurred image I from different viewpoints... blurry And the event data, divide the event data into b groups equally according to time. Then, the sharp image I0 corresponding to the exposure start time of the blurred image is recovered using the EDI algorithm:

[0028]

[0029] The clear image I at any moment during the exposure process k for:

[0030]

[0031] Where b is the number of event groups, C is the event camera threshold, e represents the natural constant, and B i This indicates the i-th event group.

[0032] Step S200: The image sequence obtained in step S100 under each viewpoint By merging the input COLMAP, the pose matrix corresponding to the image can be output.

[0033]

[0034] Figure 4The flowchart of step S300 of some embodiments of the multi-view image deblurring method of this disclosure is shown:

[0035] Step S310: Assuming a ray of light emitted from a camera pixel is rendered by the NeRF network, the estimated color value of the camera pixel is obtained. A schematic diagram of the NeRF network module is shown below. Figure 3 As shown. Using 3D position information O and 2D orientation information d as inputs to the NeRF network, the outputs are color information C and density σ:

[0036] (c, σ) = F θ (γ o (o), γ d (d)).

[0037] Among them, F θ This represents the multilayer perceptron in the NeRF network, where γ represents the position information encoder.

[0038]

[0039] Where M represents the hyperparameter, sin() represents the sine function, cos() represents the cosine function, and π represents pi.

[0040] Suppose a ray of light emitted from the camera is r(t) = o + ld, l n and l f These represent the closest and farthest boundaries from the camera, respectively. [l] n , l f If the fiber is divided into N independent blocks, then each block through which the fiber passes has color information C and density σ output by the NeRF network. Therefore, the final pixel color of the fiber on the camera is:

[0041]

[0042] Where, δ i =l i+1 -l i T represents the length of each small block. i Indicates from the nearest boundary l n The transmittance of all particles to this sampling point, where exp() represents an exponential function with base e.

[0043] Step S320: Calculate the mean of the color estimates under each pose, and use the mean squared error between this value and the input blurred image as the blur loss function. There are b corresponding poses for each viewpoint. Based on these b poses, b color estimates can be obtained through NeRF network rendering. The blur loss function is the mean square error between the mean of the color sequence at the corresponding pixel and the input blurred image.

[0044]

[0045]

[0046] in, express The square of the 2-norm, r k Let represent a ray of light emitted from the camera, and k represent the summation sequence index, k∈[0, b].

[0047] Step S330: Randomly select the color prediction values ​​of two different times at the same pixel point, convert them to grayscale values ​​to calculate the event estimate, and calculate the mean squared error between this estimate and the actual number of events at that pixel point as the event loss function. The event loss function is randomly selected from... Select two predicted values Then convert the two selected predicted values ​​to grayscale values. If θ is the simulated threshold, then the estimated value of the event at pixel x is:

[0048]

[0049] The event loss is the mean squared error between the actual number of events and the estimated number of events:

[0050]

[0051]

[0052] Where x represents any pixel among all the pixels of the camera.

[0053] Step S340: Optimize the NeRF network by backpropagating the calculation results of the two loss functions mentioned above.

[0054] In step S400, given the camera pose, input the NeRF network trained in step S300 to render a clear image of the scene or object in that pose.

[0055] The relevant content of steps S100-S400 above is as follows: Figure 2 As shown.

[0056] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a multi-viewpoint image deblurring apparatus, which are similar to... Figure 1 Corresponding to the method embodiments shown, this multi-view image deblurring device can be specifically applied to various electronic systems.

[0057] like Figure 5As shown, the multi-view image deblurring apparatus 500 in some embodiments includes: a first input unit 501, a second input unit 502, a third input unit 503, and a fourth input unit 504. The first input unit 501 is configured to, for a scene or object, input blurred images from multiple viewpoints, and event data within the exposure time of capturing the blurred images. For each viewpoint, the event data is grouped equally by time, and then an EDI algorithm is used to recover the initial deblurred image. The second input unit 502 is configured to input all the obtained initial deblurred images into the COLMAP tool to obtain the shooting pose of the initial deblurred image at each viewpoint. The third input unit 503 is configured to, for each viewpoint, input the obtained pose into the NeRF network to render multiple clear images at the aforementioned viewpoints. The blurred image is synthesized using the multiple clear images, and the difference between the blurred image and the original input blurred image is used as the blur loss function. Simultaneously, using the multiple clear images, the same pixel point at different times is randomly selected to calculate the theoretical value of the event data, and the difference between the theoretical value and the real event data is used as the event loss function. The NeRF network parameters are adjusted by backpropagation to learn the three-dimensional implicit representation of the scene or object. The fourth input unit 504 is configured to, given the camera pose, input the trained NeRF network to render a clear image of the scene or object at that pose.

[0058] It is understandable that the units and references described in the multi-view image deblurring apparatus 500 Figure 1 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the multi-view image deblurring apparatus 500 and the units contained therein, and will not be repeated here.

[0059] The following is for reference. Figure 6 The diagram illustrates a structural schematic of an electronic system 600 suitable for implementing some embodiments of the present disclosure. The electronic systems in some embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 3 The electronic system shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0060] like Figure 3As shown, the electronic system 600 may include a processing device 601 (e.g., a central processing unit, a graphics processor, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the electronic system 600. The processing device 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0061] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic system 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic system 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0062] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer electronic medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of some embodiments of this disclosure.

[0063] It should be noted that the computer electronic medium described in some embodiments of this disclosure may be a computer-readable signal medium or a computer-readable electronic medium, or any combination thereof. A computer-readable electronic medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable electronic medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable electronic medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer electronic medium other than computer-readable electronic media, which can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer electronic medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0064] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0065] The aforementioned computer electronic medium may be included in the aforementioned electronic system; or it may exist independently and not assembled into the electronic system. The aforementioned computer electronic medium carries one or more programs, which, when executed by the electronic system, cause the electronic system to: Step S Step S100: For a scene or object, given multiple blurred image inputs from different viewpoints, and event data within the exposure time of capturing the blurred image, for each viewpoint, the event data is grouped equally by time, and then the initial deblurred image is recovered using an EDI algorithm; Step S200: Input all the obtained initial deblurred images into the COLMAP tool to obtain the shooting pose of the initial deblurred image for each viewpoint; Step S300: For each viewpoint, input the obtained pose into the NeRF network to render multiple clear images from the above viewpoints, synthesize a blurred image using the multiple clear images, and use the difference from the original input blurred image as the blur loss function; simultaneously, using the multiple clear images, randomly select the same pixel at different times to calculate the theoretical value of the event data, and use the difference from the real event data as the event loss function, backpropagate to adjust the NeRF network parameters, and learn the three-dimensional implicit representation of the scene or object; Step S400: Given the camera pose, input the trained NeRF network to render a clear image of the scene or object at that pose.

[0066] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0067] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0068] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a first input unit, a second input unit, a third input unit, and a fourth input unit. The names of these units do not necessarily limit the specific unit; for example, the fourth input unit may be described as "a unit that, given a camera pose, inputs a trained NeRF network to render a clear image of the scene or object in that pose."

[0069] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0070] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A multi-view image deblurring method, comprising: Step S100: For a scene or object, given multiple blurred image inputs from different viewpoints, and event data within the exposure time of capturing the blurred image, for each blurred image and event data from each viewpoint, the event data is grouped equally according to time, and then the initial deblurred image is recovered using the EDI algorithm. Step S200: Input all the obtained initial deblurred images into the COLMAP tool to obtain the shooting pose of the initial deblurred images from each viewpoint; Step S300: For each viewpoint, the obtained pose is input into the NeRF network to render multiple clear images under the viewpoint. A blurred image is synthesized using the multiple clear images, and the difference between the blurred image and the original input blurred image is used as the blur loss function. At the same time, using the multiple clear images, the same pixel point at different times is randomly selected to calculate the theoretical value of the event data, and the difference between the theoretical value and the real event data is used as the event loss function. The NeRF network parameters are adjusted by backpropagation to learn the three-dimensional implicit representation of the scene or object. In step S400, given the camera pose, input the trained NeRF network to render a clear image of the scene or object in that pose.

2. The method according to claim 1, wherein, The EDI algorithm has the following form: For event data from each perspective, it is divided into equal parts based on time. Group At the same time, using blurred images This yields the sharp image corresponding to the start of exposure of the blurred image. for: ， A clear image at any point during the exposure process. for: ， in, The number of event groups, For the event camera threshold, Represents the natural constant. Indicates the first Grouping events.

3. The method according to claim 2, wherein, The method further includes: The obtained image sequence from each viewpoint By merging the input COLMAP, an image sequence can be output. Corresponding pose matrix : 。 4. The method according to claim 1, wherein, The NeRF network uses three-dimensional location information and two-dimensional directional information As input, the output is color information. and density : ， in, This represents a multilayer perceptron in a NeRF network. Encoder representing location information: ， in, Indicates hyperparameters, Represents the sine function. Represents the cosine function. It represents pi (π).

5. The method according to claim 1, wherein, The rendering process assumes a ray of light emitted from the camera. , and These represent the closest and farthest boundaries from the camera, respectively. Divide into equal parts Each individual small block through which the light passes contains color information output by the NeRF network. and density Then the final color of the pixel on the camera for that ray of light is: ， in, This indicates the length of each small block. Indicates from the nearest boundary To the The transmittance of all particles at each sampling point. Indicates An exponential function with base 0.

6. The method according to claim 5, wherein, Each perspective has Each corresponding pose According to this Each pose is obtained through rendering. Color estimates The blur loss function is the mean square error between the mean of the color sequence at the corresponding pixel and the input blurred image. ， ， in, express The square of the 2-norm, This represents a ray of light emitted from the camera. Indicates the summation sequence indicator number, .

7. The method according to claim 6, wherein, The event loss function is randomly generated from... Select two predicted values , Then, the two selected predicted values ​​are converted into grayscale values. , Then the pixel The estimated value of the event is: ， in, The simulated threshold is represented by the event loss, which is the mean squared error between the actual number of events and the estimated number of events. ， ， in, For pixels, Indicates the summation sequence indicator number, , Represents pixels The Event data for each group.

8. A multi-viewpoint image deblurring apparatus, comprising: The first input unit is configured to take a scene or object as a given blurred image input from multiple viewpoints, as well as event data within the exposure time of capturing the blurred image. For the blurred image and event data from each viewpoint, the event data is grouped equally according to time, and then the initial deblurred image is recovered using the EDI algorithm. The second input unit is configured to input all the obtained initial deblurred images into the COLMAP tool to obtain the shooting pose of the initial deblurred images at each viewpoint; The third input unit is configured to input the obtained pose into the NeRF network for each viewpoint, render multiple clear images under the viewpoint, synthesize a blurred image using the multiple clear images, and use the difference with the original input blurred image as the blur loss function; at the same time, using the multiple clear images, randomly select the same pixel at different times, calculate the theoretical value of the event data, and use the difference with the real event data as the event loss function, backpropagate to adjust the NeRF network parameters, and learn the three-dimensional implicit representation of the scene or object; The fourth input unit is configured to be input into the trained NeRF network at a given camera pose, which can then render a clear image of the scene or object at that pose.

9. An electronic system comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 7.

10. A computer electronic medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

Images