Extremely high dynamic range imaging method and device based on event camera assistance, equipment and medium

Abstract

The invention relates to an event camera assistance-based extremely high dynamic range imaging method, device and equipment and a medium, and the method comprises the following steps: collecting an image by using a dual-camera imaging system based on an event camera and an RGB camera, collecting a high dynamic range event signal by the event camera, and transmitting the high dynamic range event signal to the RGB camera; the RGB camera collects low-dynamic-range RGB images of different exposure times; inputting the event signal and the RGB image into an event camera-assisted alignment module, and assisting alignment of the RGB images with different exposure times by using the event signal; and inputting the aligned RGB images with different exposure times into a fusion module based on a diffusion model for fusion to obtain a high-dynamic image generation result. Compared with the prior art, the method has the advantages that imaging in an extremely high dynamic range can be realized, and the method is suitable for large-motion and large-exposure scenes.

Description

Technical Field

[0001] The present invention relates to the field of dynamic range imaging technology, and in particular to an event camera-assisted extremely high dynamic range imaging method, device, equipment and medium. Background Art

[0002] In the real world, the dynamic range of ordinary cameras is much lower than that of the human visual system, and it is impossible to capture the details of extremely bright and extremely dark areas in the scene at the same time. However, high dynamic range (HDR) scenes have important application value in many fields, such as capturing clear road conditions ahead in night driving assistance, monitoring systems identifying facial details under strong backlight conditions, and accurately recording lesion areas with strong light-dark contrast in medical images. These scenes are characterized by extremely high dynamic range, usually reaching 12 stops or even higher, far exceeding the dynamic range of traditional cameras.

[0003] In order to achieve HDR imaging, existing technologies usually use multi-exposure image fusion methods to synthesize low dynamic range (LDR) images taken at different exposure times into HDR images. However, when there is fast motion or large exposure differences in the scene, these methods have difficulty in accurately aligning images, resulting in problems such as ghosting artifacts, limiting their application in dynamic scenes. At the same time, existing HDR imaging algorithms are mostly limited to 6 stops of dynamic range, which cannot meet the needs of detail recovery in high-contrast scenes.

[0004] Traditional high-dynamic imaging algorithms that use multi-exposure fusion with ordinary cameras cannot handle the alignment problem between images with large exposure differences due to the lack of inter-frame motion information. Therefore, they cannot be expanded to high-dynamic algorithms with 12 exposure levels, and it is difficult to obtain good results for complex motion scenes. At the same time, the high-dynamic algorithms that used to use event cameras in the past still cannot handle the fusion of 12 exposure levels due to the implicit alignment module designed and the simple use of neural networks for fusion.

[0005] In summary, there are two problems in high dynamic imaging of dynamic scenes with 12 exposure levels: (1) Image alignment problem. As the difference in exposure values ​​increases, alignment between images becomes very difficult. At the same time, even if event camera signals are used, the gap between event camera simulation and reality makes it difficult for previous methods to obtain good alignment results. (2) The fusion problem of 12 exposure levels is a difficult problem, especially since motion occlusion and errors in alignment may greatly affect exposure fusion. The above two problems will lead to difficulties in high dynamic imaging and insufficient accuracy of the final imaging results. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide an extremely high dynamic range imaging method, device, equipment and medium based on event camera assistance. By using event camera assistance for alignment, and utilizing a diffusion module and prior information of natural images, the effect of 12-speed exposure fusion is improved to perform 12-speed high dynamic range imaging of dynamic scenes.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] In view of the two problems existing in the prior art, the present invention proposes: (1) an alignment module assisted by an event camera, and further trains the alignment module using real event camera interpolation data, so that a good alignment effect can also be obtained on the real event camera. (2) It proposes to use the natural image prior information of the diffusion model to fuse 12 exposure levels, which can effectively solve the problems of motion occlusion and alignment errors.

[0009] According to a first aspect of the present invention, there is provided an event camera-assisted very high dynamic range imaging method, the method comprising the following steps:

[0010] Using a dual-camera imaging system based on an event camera and an RGB camera to collect images, wherein the event camera collects event signals with a high dynamic range, and the RGB camera collects RGB images with a low dynamic range at different exposure times;

[0011] The event signal and the RGB image are input into the event camera-assisted alignment module, and the event signal is used to assist in the alignment of the RGB images with different exposure times;

[0012] The aligned RGB images with different exposure times are input into the fusion module based on the diffusion model for fusion to obtain a high-dynamic image generation result.

[0013] As a preferred technical solution, the dual-camera imaging system uses a beam splitter to split the input light into two beams, which are input into the event camera and the RGB camera respectively, so that the fields of view of the two cameras are the same.

[0014] As a preferred technical solution, the event camera-assisted alignment module takes an RGB image at a reference exposure time, an RGB image at a non-reference exposure time, and an event signal as input, and uses the time-dense motion information provided by the event signal as an aid to align each RGB image at the non-reference exposure time with the RGB image at the reference exposure to obtain multiple aligned images.

[0015] As a preferred technical solution, the event camera-assisted alignment module performs the following steps on the input:

[0016] Inputting the RGB image at the reference exposure time and another non-reference exposure time into the corresponding image encoder respectively to obtain the reference image depth feature and the exposure image depth feature;

[0017] The event signal, the reference image depth feature and the exposure image depth feature are input into the optical flow estimation module to estimate the enhanced optical flow information;

[0018] Based on the enhanced optical flow, the depth features of the reference image and the depth features of the exposure image are aligned at the feature level to obtain aligned image features;

[0019] The aligned image features are input into the image decoder and the aligned image is output.

[0020] As a preferred technical solution, the diffusion model-based fusion module utilizes the prior information of the natural image of the diffusion model, adaptively processes the alignment error of the alignment module and the occluded parts between different images, and fuses the aligned images with different exposures.

[0021] As a preferred technical solution, the diffusion model-based fusion module performs the following steps:

[0022] Input multiple aligned images processed by the event camera-assisted alignment module into the variational autoencoder to obtain image features;

[0023] The image features and the Gaussian noise of the current iteration are input into the control network to obtain the control signal, which acts on the diffusion model decoder;

[0024] The Gaussian noise of the current iteration passes through the diffusion model encoder and the diffusion model decoder in sequence to obtain the output of the diffusion model as the Gaussian noise of the next iteration;

[0025] After a preset number of iterations, the output of the final diffusion model is processed by a variational self-decoder to obtain an image after fusion of multiple aligned images.

[0026] As a preferred technical solution, the diffusion model is pre-trained on a large-scale data set so that it has prior information of the image.

[0027] According to a second aspect of the present invention, there is provided an event camera-assisted ultra-high dynamic range imaging device, the device comprising:

[0028] Image acquisition module: uses a dual-camera imaging system based on an event camera and an RGB camera to acquire images, wherein the event camera acquires event signals with a high dynamic range, and the RGB camera acquires RGB images with a low dynamic range at different exposure times;

[0029] Event camera-assisted alignment module: takes event signals and RGB images as input, and uses event signals to assist in the alignment of RGB images with different exposure times;

[0030] Fusion module based on diffusion model: takes aligned RGB images with different exposure times as input, fuses them and obtains high-dynamic image generation results.

[0031] According to a third aspect of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the method described above is implemented when the processor executes the program.

[0032] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program implements the method described when executed by a processor.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] 2. Propose an event signal-based alignment module for display.

[0035] 3. A fusion module based on diffusion model is proposed for better processing.

[0036] 1. The present invention proposes a dual-camera imaging system for motion magnification, which is composed of an event camera and an RGB camera. It uses a time signal with dense time domain to perform imaging of motion scenes with extremely high dynamic range.

[0037] 2. The present invention can capture synchronized event signals and multi-exposure RGB images in real time in dynamic scenes, and supports imaging with a 12-stop dynamic range from -6EV to +6EV.

[0038] 3. In view of the characteristics of large dynamic range and fast motion in the scene, the present invention integrates a motion alignment module based on event signals and a fusion module based on diffusion models, which can better handle large motion and alignment error accumulation, and better fuse large exposures.

[0039] 4. This paper designs an alignment module based on event signals, which can better handle large motions. The alignment module combines the asynchronous characteristics of event signals with the global information of RGB frames, significantly reducing the computational cost and artifact risk of traditional image-based alignment methods, and can be widely used in a variety of dynamic scenes, including image alignment in extreme cases with many moving objects or large exposure differences.

[0040] 5. The present invention designs a fusion module based on a diffusion model. By introducing the pre-trained image prior in the diffusion model, the cumulative effect of the alignment error in HDR fusion is reduced, and the fusion of ultra-high dynamic range can be better performed. In addition, the fusion module can handle scenes with highly compressed details in bright and dark areas, reduce artifact generation, and improve the texture restoration effect of bright and dark areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of the method of the present invention;

[0042] Figure 2 It is a schematic diagram of a dual-camera imaging system of the present invention;

[0043] Figure 3 This is a comparison chart of high dynamic imaging results of the present invention and other methods. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0045] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0046] The technical problem to be solved by the present invention is to achieve 12-stop high dynamic range (HDR) imaging in dynamic scenes, thereby breaking through the limitations of dynamic range and alignment accuracy of traditional methods. Traditional HDR imaging technology mainly relies on the fusion of multi-exposure low dynamic range (LDR) images, but when processing dynamic scenes, image alignment faces significant challenges due to object motion or large exposure differences, especially in high-contrast scenes where ghosting artifacts may occur. Therefore, existing HDR fusion algorithms can usually only process images with a 6-stop range (such as -3EV to +3EV), and are insufficient in recovering bright and dark details in high dynamic range scenes.

[0047] At the same time, the present invention designs a fusion module based on a diffusion model, which can effectively fuse LDR images with large exposure differences, and use pre-trained image priors to reduce alignment error accumulation and artifacts during the fusion process. Compared with the previous high-dynamic algorithm combined with event cameras, the present invention improves the alignment accuracy by designing a displayed alignment module, and uses a diffusion module to use the prior information of natural images to improve the effect of 12-speed exposure fusion.

[0048] Through the above method, the present invention can achieve higher dynamic range imaging in dynamic scenes, improve the ability to restore details of bright and dark areas, and provide a new technical means for the application of HDR imaging in complex scenes.

[0049] This embodiment provides an extremely high dynamic range imaging method based on event camera assistance, such as Figure 1 As shown, the method comprises the following steps:

[0050] S1, uses a dual-camera imaging system based on an event camera and an RGB camera to collect images, wherein the event camera collects event signals with a high dynamic range, and the RGB camera collects RGB images with a low dynamic range with different exposure times.

[0051] The present invention proposes a dual-camera HDR imaging system combining an event camera and a traditional RGB camera to solve the above technical problems. The event camera has high temporal resolution and high dynamic range (usually greater than 100dB), which can capture dense time series event streams for precise alignment of motion between bright and dark areas.

[0052] The core of this technical solution is to use a dual-camera imaging system to perform ultra-high dynamic range imaging, and to collect dense event signals through the event camera for frame alignment. Figure 2 As shown in Figure 1, the dual-camera imaging system uses a beam splitter to split the input light into two beams, which are input to the event camera and the RGB camera respectively, so that the fields of view of the two cameras are the same. The dual-camera imaging system collects RGB images and event signals with different exposures as shown in Figure 1. Figure 1 as shown in .

[0053] S2, inputting the event signal and the RGB image into the event camera assisted alignment module, and using the event signal to assist the alignment of RGB images with different exposure times.

[0054] like Figure 1 As shown in (a) in the figure, the event camera-assisted alignment module takes an RGB image at a reference exposure time (0EV), an RGB image at a non-reference exposure time (e.g., -6EV), and an event signal as input, and uses the time-dense motion information provided by the event signal as an aid to align each RGB image at the non-reference exposure time with the RGB image at the reference exposure time to obtain multiple aligned images.

[0055] The event-camera-assisted alignment module performs the following steps on the input:

[0056] S201, inputting the RGB image at a reference exposure time and another non-reference exposure time into a corresponding image encoder respectively to obtain a reference image depth feature and an exposure image depth feature;

[0057] S202, inputting the event signal, the reference image depth feature and the exposure image depth feature into an optical flow estimation module to estimate and obtain enhanced optical flow information;

[0058] S203, aligning the reference image depth features and the exposure image depth features at a feature level based on the enhanced optical flow to obtain aligned image features;

[0059] S204, inputting the aligned image features into an image decoder to obtain an aligned image for output.

[0060] In this embodiment, an alignment module is used to align each RGB image at a non-reference exposure time with the RGB image at a reference exposure time to obtain a plurality of aligned images, which are used as inputs of a subsequent fusion module.

[0061] Based on the hardware structure of dual cameras, this method proposes an image alignment module assisted by displayed event signals, which has solved the problem of ghosting in high-dynamic fusion results. Furthermore, this method proposes a strategy of using real event camera interpolation datasets for further training, which improves the generalization performance of the alignment module in real scenes.

[0062] S3, inputs the aligned RGB images with different exposure times into the fusion module based on the diffusion model for fusion, and obtains a high-dynamic image generation result.

[0063] This method designs a fusion module based on the diffusion model, and solves the problem of ultra-high dynamic range fusion of 12 exposure levels with the help of the prior information of natural images in the diffusion model.

[0064] The fusion module based on the diffusion model utilizes the prior information of the natural image of the diffusion model, adaptively handles the alignment error of the alignment module and the occluded parts between different images, and fuses the aligned images with different exposures.

[0065] The diffusion model is pre-trained on a large-scale dataset to enable it to have prior information about the image.

[0066] The diffusion model-based fusion module performs the following steps:

[0067] S301, inputting a plurality of aligned images processed by the event camera-assisted alignment module into a variational autoencoder to obtain image features;

[0068] S302, image features and the Gaussian noise Z of the current iteration are combined. t Input the control network to obtain the control signal, which acts on the diffusion model decoder;

[0069] S303, Gaussian noise Z of the current iteration t The output of the diffusion model is obtained by passing through the diffusion model encoder and the diffusion model decoder in sequence, and is used as the Gaussian noise Z for the next iteration. t-1 ;

[0070] S304, after T iterations, the output Z0 of the final diffusion model is considered as the image fusion result with the noise eliminated, which is processed by the variational self-decoder to obtain an image after the fusion of multiple aligned images.

[0071] Figure 3 This is a comparison chart of high dynamic imaging results between the method of the present invention and other prior arts. It can be found that the present invention can perform 12-level large exposure fusion, which can well solve the problems of motion occlusion and alignment errors, and the final imaging effect is excellent.

[0072] The above is an introduction to a method embodiment. The following is a further explanation of the solution of the present invention through an apparatus embodiment.

[0073] An extremely high dynamic range imaging device based on event camera assistance, the device comprising:

[0074] Image acquisition module: uses a dual-camera imaging system based on an event camera and an RGB camera to acquire images, wherein the event camera acquires event signals with a high dynamic range, and the RGB camera acquires RGB images with a low dynamic range at different exposure times;

[0075] Event camera-assisted alignment module: takes event signals and RGB images as input, and uses event signals to assist in the alignment of RGB images with different exposure times;

[0076] Fusion module based on diffusion model: takes aligned RGB images with different exposure times as input, fuses them and obtains high-dynamic image generation results.

[0077] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0078] In one embodiment, the electronic device includes a computing unit that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) or a computer program loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for the operation of the device can also be stored. The computing unit, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0079] Many components in an electronic device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0080] The computing unit may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of computing units include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit performs the various methods and processes described above, such as methods S1 to S3. For example, in some embodiments, methods S1 to S3 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into RAM and executed by the computing unit, one or more steps of methods S1 to S3 described above may be performed. Alternatively, in other embodiments, the computing unit may be configured to perform methods S1 to S3 in any other appropriate manner (e.g., by means of firmware).

[0081] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0082] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0083] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0084] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An event camera-assisted extremely high dynamic range imaging method, characterized in that: The method comprises the following steps: Using a dual-camera imaging system based on an event camera and an RGB camera to collect images, wherein the event camera collects event signals with a high dynamic range, and the RGB camera collects RGB images with a low dynamic range at different exposure times; The event signal and the RGB image are input into the event camera-assisted alignment module, and the event signal is used to assist in the alignment of the RGB images with different exposure times; The aligned RGB images with different exposure times are input into the fusion module based on the diffusion model for fusion to obtain a high-dynamic image generation result.

2. The event camera-assisted ultra-high dynamic range imaging method according to claim 1, characterized in that: The dual-camera imaging system uses a beam splitter to split the input light into two beams, which are input into the event camera and the RGB camera respectively, so that the fields of view of the two cameras are the same.

3. The event camera-assisted ultra-high dynamic range imaging method according to claim 1, characterized in that: The event camera-assisted alignment module takes an RGB image at a reference exposure time, an RGB image at a non-reference exposure time, and an event signal as input, and uses the time-dense motion information provided by the event signal as an aid to align each RGB image at the non-reference exposure time with the RGB image at the reference exposure to obtain multiple aligned images.

4. The event camera-assisted ultra-high dynamic range imaging method according to claim 1, characterized in that: The event-camera-assisted alignment module performs the following steps on the input: Inputting the RGB image at the reference exposure time and another non-reference exposure time into the corresponding image encoder respectively to obtain the reference image depth feature and the exposure image depth feature; The event signal, the reference image depth feature and the exposure image depth feature are input into the optical flow estimation module to estimate the enhanced optical flow information; Based on the enhanced optical flow, the depth features of the reference image and the depth features of the exposure image are aligned at the feature level to obtain aligned image features; The aligned image features are input into the image decoder and the aligned image is output.

5. The event camera-assisted ultra-high dynamic range imaging method according to claim 1, characterized in that: The diffusion model-based fusion module utilizes the prior information of the natural image of the diffusion model, adaptively processes the alignment error of the alignment module and the occluded parts between different images, and fuses the aligned images with different exposures.

6. The event camera-assisted ultra-high dynamic range imaging method according to claim 1, characterized in that: The diffusion model-based fusion module performs the following steps: Input multiple aligned images processed by the event camera-assisted alignment module into the variational autoencoder to obtain image features; The image features and the Gaussian noise of the current iteration are input into the control network to obtain the control signal, which acts on the diffusion model decoder; The Gaussian noise of the current iteration passes through the diffusion model encoder and the diffusion model decoder in sequence to obtain the output of the diffusion model as the Gaussian noise of the next iteration; After a preset number of iterations, the output of the final diffusion model is processed by a variational self-decoder to obtain an image after fusion of multiple aligned images.

7. The event camera-assisted ultra-high dynamic range imaging method according to claim 1, characterized in that: The diffusion model is pre-trained on a large-scale dataset so that it has prior information of the image.

8. An event camera-assisted ultra-high dynamic range imaging device, characterized in that: The device includes: Image acquisition module: uses a dual-camera imaging system based on an event camera and an RGB camera to acquire images, wherein the event camera acquires event signals with a high dynamic range, and the RGB camera acquires RGB images with a low dynamic range at different exposure times; Event camera-assisted alignment module: takes event signals and RGB images as input, and uses event signals to assist in the alignment of RGB images with different exposure times; Fusion module based on diffusion model: takes aligned RGB images with different exposure times as input, fuses them and obtains high-dynamic image generation results.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images