Imaging module for realizing DVS and RGB fusion, camera and electronic equipment

Abstract

The embodiment of the utility model provides an imaging module for realizing DVS and RGB fusion. The imaging module for realizing DVS and RGB fusion comprises a DVS imaging device, an RGB imaging device, a fusion device and a main control device. The DVS imaging device is used for generating an event image; the RGB imaging device is used for generating an RGB image; the fusion device is used for fusing the event image and the RGB image to obtain a corresponding target image and outputting the target image; the main control device is electrically connected with the DVS imaging device, the RGB imaging device and the fusion device and used for controlling the DVS imaging device, the RGB imaging device and the fusion device to operate according to a preset sequence. In addition, the utility model also provides a camera and electronic equipment. According to the technical scheme, the problems that the image of the moving object is blurred and a clear image cannot be output when the object moves at a high speed are effectively solved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an imaging module, a camera, and an electronic device for realizing DVS and RGB fusion. Background Art

[0002] Images have gradually become an indispensable part of our daily life and work, and people have a growing demand for images and higher requirements. Especially for high-speed motion scenes, people often use sports cameras to record wonderful moments. However, due to the characteristics of camera image acquisition, shooting sports scenes faces the problem of poor image quality and blurred images due to smearing.

[0003] The existing technology reduces motion blur by increasing the shutter speed, image restoration algorithms, and image post-processing technology. However, the existing technologies have certain limitations and shortcomings in the process of reducing motion blur, and cannot truly output clear images when the object is moving at high speed. For example, the disadvantage of increasing the shutter speed is that it has high requirements for lighting. When the light is reduced, the gain of the photosensitive device will be increased first, and there will be obvious noise in the picture. The disadvantage of the image restoration algorithm is that in some cases, it may not be able to fully restore the clear image, the computational complexity is high, and it may require more processing time. The disadvantage of image post-processing technology is that it may introduce additional noise, and excessive enhancement of details may cause image artifacts or distortion. Summary of the invention

[0004] The present application provides an imaging module, a camera, and an electronic device that realize DVS and RGB fusion, and can output clear images when an object moves at high speed.

[0005] In the first aspect, the embodiment of the present application provides an imaging module for realizing DVS and RGB fusion, and the imaging module for realizing DVS and RGB fusion includes a DVS imaging device, an RGB imaging device, a fusion device, and a main control device. The DVS imaging device is used to generate an event image; the RGB imaging device is used to generate an RGB image; the fusion device is used to fuse the event image and the RGB image to obtain a corresponding target image and output it; the main control device is electrically connected to the DVS imaging device, the RGB imaging device, and the fusion device, and is used to control the DVS imaging device, the RGB imaging device, and the fusion device to operate in a preset order.

[0006] In the second aspect, an embodiment of the present application provides a camera that realizes DVS and RGB fusion. The camera includes a body, an optical element disposed on the body, and an imaging module that realizes DVS and RGB fusion disposed in the body, wherein the optical element guides external ambient light to the imaging module that realizes DVS and RGB fusion; the imaging module that realizes DVS and RGB fusion forms images based on the ambient light. The imaging module that realizes DVS and RGB fusion includes a DVS imaging device, an RGB imaging device, a fusion device, and a main control device. The DVS imaging device is used to generate an event image; the RGB imaging device is used to generate an RGB image; the fusion device is used to fuse the event image and the RGB image to obtain a corresponding target image and output it; the main control device is electrically connected to the DVS imaging device, the RGB imaging device, and the fusion device, and is used to control the DVS imaging device, the RGB imaging device, and the fusion device to operate in a preset order.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a main body and the above-mentioned camera, and the above-mentioned camera is arranged on the main body.

[0008] The imaging module, camera and electronic device that realize DVS and RGB fusion integrate the DVS imaging device and the RGB imaging device, and can use the DVS imaging device and the RGB imaging device for synchronous imaging and fusion, which can reduce motion blur and output clear images of moving objects in high-speed motion. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A schematic diagram of the structural framework of an imaging module for realizing DVS and RGB fusion provided in an embodiment of the present application.

[0010] Figure 2 A schematic diagram of the image processing process of the DVS and RGB fused imaging module provided in an embodiment of the present application.

[0011] Figure 3 A camera with an imaging module that uses DVS and RGB fusion is provided in an embodiment of the present application.

[0012] Figure 4 A schematic diagram of an electronic device using an imaging module that incorporates DVS and RGB according to an embodiment of the present application.

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0014] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0016] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar planning objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate. In other words, the described embodiments are implemented according to an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any of their variations may also include other content. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to only those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0017] It should be noted that the descriptions involving "first", "second", etc. in this application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0018] Please see Figure 1, which is an imaging module 100 for realizing DVS and RGB fusion provided by the first embodiment of the present application. The imaging module 100 includes a DVS imaging device 1, an RGB imaging device 2, a fusion device 3, and a main control device 4. Among them, the main control device 4 is respectively connected to the DVS imaging device 1, the RGB imaging device 2, and the fusion device 3 for controlling the imaging device 1, the RGB imaging device 2, and the fusion device 3 to operate in a preset order. In this embodiment, the DVS imaging device 1, the RGB imaging device 2, and the fusion device 3 are respectively chips that perform specific functions or modules composed of a chip and a circuit board. In this embodiment, the main control device 4, the DVS imaging device 1, the RGB imaging device 2, and the fusion device 3 are respectively integrated on a circuit board. It can be understood that in some feasible embodiments, the main control device 4, the DVS imaging device 1, the RGB imaging device 2, and the fusion device 3 can also be a plurality of independent modules that are connected to each other for communication.

[0019] The DVS imaging device 1 is used to generate event images. The DVS imaging device 1 can be understood as a sensor that can sense moving objects. Specifically, the DVS imaging device 1 includes a photoelectric sensing module corresponding to each pixel, and when the brightness change at each pixel exceeds a set threshold, event data is generated and output. The DVS imaging device 1 can use an existing DVS imaging device, which is not limited here.

[0020] RGB imaging device 2 is used to generate RGB images. RGB imaging device 2 can be a visible light imaging device, which is used to convert light signals into electrical signals, obtain digital image signals through analog-to-digital conversion, and then obtain RGB format image signals through a familiar signal processing chip (DSP). RGB imaging device 2 can use existing RGB imaging devices, which are not limited here.

[0021] The fusion device 3 is used to fuse the event image and the RGB image to obtain the target image and output it. It can be understood that the fusion device 3 is a device that can fuse two image signals from different imaging devices. In this embodiment, the existing image signal fusion device can be used, and it is not limited here. The imaging module 100 also includes a memory (not shown). The memory is used to store the event image and the RGB image.

[0022] When an object moves at high speed, the DVS imaging device 1 and the RGB imaging device 2 synchronously shoot and expose the high-speed moving object, fuse the time-synchronized and exposure-synchronized event image and the RGB image, and finally output a clear image of the high-speed moving object. For example, during the driving of a vehicle, due to the excessive speed of the vehicle, the edge of the vehicle image is easily blurred when the RGB imaging device 2 is imaging. The DVS imaging device 1 has the characteristics of high dynamic range and low latency, so the image imaged by the DVS imaging device 1 can better reflect the edge changes of the vehicle movement. The RGB image generated by the RGB imaging device 2 is fused with the event image generated by the DVS imaging device 1, which can reduce motion blur, thereby outputting a clear image of the vehicle when it is driving.

[0023] In this embodiment, the main control device 4 is used to control the synchronous imaging of the DVS imaging device 1 and the RGB imaging device 2. For example, the main control device 4 synchronizes the images generated by the DVS imaging device 1 and the RGB imaging device 2 through a trigger and a synchronization signal. For example, the master-slave camera is confirmed, and the master camera can be the DVS imaging device 1 or the RGB imaging device 2, wherein the master camera is responsible for sending a synchronization signal, connecting the synchronization input ports of the DVS imaging device 1 and the RGB imaging device 2 through a synchronization signal line to ensure that the signal remains smooth, and performing corresponding settings on the DVS imaging device 1 and the RGB imaging device 2, specifying the reception of the synchronization signal, and synchronizing the image acquisition with the signal as a reference point. No further details will be given here.

[0024] The DVS imaging device 1 is a special type of imaging device. Unlike an RGB camera, it can output a DVS image based on the brightness change in the picture. Based on the principle of global light change, the DVS imaging device 1 uses asynchronous acquisition and processing to generate events only when brightness changes occur in the image. Compared with traditional RGB cameras, event cameras have the following characteristics:

[0025] 1. High dynamic range: It can capture changes in light intensity very quickly, has a high dynamic range, and can adapt to scenes with strong light and weak light.

[0026] 2. Low latency: Output in the form of events can respond to fast-moving scenes in real time, avoiding the motion blur problem caused by exposure time and frame rate limitations in traditional cameras.

[0027] 3. Low energy consumption: Since events are generated only when brightness changes occur in the image, energy consumption is lower than that of traditional RGB cameras.

[0028] However, the DVS imaging device 1 cannot provide continuous images but can only provide discrete event data, and is not suitable for application scenarios that require precise color information.

[0029] However, the dynamic range of the RGB imaging device 2 is relatively narrow, which is limited for scenes with large brightness changes, and may cause motion blur for fast-moving scenes. The RGB imaging device 2 also has the problem of high energy consumption, especially when continuously collecting and processing images, which requires high computing power and strong processors and storage resources.

[0030] Please see Figure 2 , Figure 2 Image 10 is an event image generated by the DVS imaging device 1, and image 20 is an RGB image 20 generated by the RGB imaging device 2, wherein the generation time of the event image 10 is the same as the generation time of the RGB image. Image 30 is the event image 10 and the RGB image 20. That is, image 30 is an image after the event image 10 and the RGB image 20 are fused. Image 30 is a clear image of a high-speed moving object in motion.

[0031] Please see Figure 3 , which is a schematic diagram of a camera provided in an embodiment of the present application.

[0032] The embodiment of the present application also provides a camera 200, which includes a body 201, an optical element 202 arranged on the body, and an imaging module 100 arranged on the body 201. The optical element 201 guides the external ambient light to the imaging module 100. The imaging module 100 is based on ambient light imaging. The structure and imaging principle of the imaging module 100 are described above and will not be repeated here. The camera 200 can be installed in a vehicle and a portable electronic device such as a mobile phone, a tablet computer, etc.

[0033] Please see Figure 4 , which is a schematic diagram of an electronic device provided in an embodiment of the present application.

[0034] The embodiment of the present application also provides an electronic device 300. In this embodiment, the electronic device 300 can be a camera, a laptop computer, a mobile phone, etc. The following description takes a camera as an example. The camera 300 includes a main body 301 and a camera 200 on which the main body 301 is set. The structure and imaging principle of the camera 200 are described above and will not be repeated here.

[0035] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0036] The above examples are only preferred embodiments of the present application, and certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application still fall within the scope covered by the present application.

Claims

1. An imaging module for realizing DVS and RGB fusion, characterized in that: The imaging module includes: A DVS imaging device for generating event images; An RGB imaging device, used for generating an RGB image; A fusion device, used for fusing the event image with the RGB image to obtain a corresponding target image and output it; The main control device is electrically connected to the DVS imaging device, the RGB imaging device, and the fusion device, and is used to control the DVS imaging device, the RGB imaging device, and the fusion device to operate in a preset sequence.

2. The imaging module for realizing DVS and RGB fusion as claimed in claim 1, characterized in that: The main control device is used to control the DVS imaging device and the RGB imaging device to synchronously generate corresponding images.

3. The imaging module for realizing DVS and RGB fusion as claimed in claim 2, characterized in that: The DVS imaging device, the RGB imaging device, and the main control device are all arranged on the same circuit board.

4. The imaging module according to claim 2, characterized in that: The main control device is also used to control the exposure synchronization of the DVS imaging device and the RGB imaging device.

5. The imaging module for realizing DVS and RGB fusion as claimed in claim 2, characterized in that: The imaging module also includes a memory for storing the event image and the RGB image.

6. A camera, comprising a body, an optical element disposed in the body, and an imaging module for realizing DVS and RGB fusion disposed in the body, wherein the optical element guides external ambient light to the imaging module for realizing DVS and RGB fusion; The imaging module for realizing DVS and RGB fusion is based on the ambient light imaging, and is characterized in that: The imaging module for realizing DVS and RGB fusion includes: A DVS imaging device for generating event images; An RGB imaging device for generating an RGB image; and The fusion device is used to fuse the event image and the RGB image to obtain a corresponding target image and output it.

7. The camera according to claim 6, wherein: The imaging module also includes a main control device, which is used to control the DVS imaging device and the RGB imaging device to synchronously generate corresponding images.

8. The camera according to claim 7, wherein: The DVS imaging device, the RGB imaging device, and the main control device are all arranged on the same circuit board.

9. The camera according to claim 7, wherein: The main control device is also used to control the exposure synchronization of the imaging device and the RGB imaging device.

10. An electronic device, comprising a main body, characterized in that: The electronic device also includes a camera as described in any one of claims 6 to 9, which is arranged on the main body.

Images