Imaging apparatus

Abstract

To solve the problem in which there is a risk of missing a subject when performing frame-based sensor imaging on the basis of event detection by an event-based sensor and activating the frame-based sensor each time an event is detected results in high power consumption.SOLUTION: The event-based sensor and frame-based sensor are positioned so their imaging ranges do not overlap. When a trajectory of an event signal detected by the event-based sensor moves toward an imaging range of the frame-based sensor, predetermined control is performed on the frame-based sensor.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an imaging device. [Background technology]

[0002] Asynchronous image sensors (hereinafter referred to as event-based sensors) that detect luminance changes in each pixel as events are known. Because event-based sensors operate when an event is detected, they are characterized by lower power consumption than synchronous image sensors (hereinafter referred to as frame-based sensors) that capture image data in synchronization with a synchronization signal generally used in imaging devices.

[0003] Therefore, a method has been proposed in which both an event-based sensor and a frame-based sensor are installed and the frame-based sensor is controlled based on the detection of an event by the event-based sensor, thereby reducing the power consumption of the frame-based sensor.

[0004] For example, Patent Document 1 discloses a method in which, when an event is detected by an event-based sensor, a frame-based sensor is transitioned to an operating state to start capturing images. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2020-161987 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in Patent Document 1, the imaging ranges of the event-based sensor and the frame-based sensor overlap, and if the frame-based sensor is activated after the event-based sensor detects an event, it may not be able to capture the image in time. Because there is a time lag between the event detection and the start of imaging by the frame-based sensor, in the case of a fast-moving subject, the subject may have already passed the imaging range of the frame-based sensor by the time the frame-based sensor is activated.

[0007] Furthermore, because the frame-based sensor starts capturing images every time an event is detected by the event-based sensor, the frame-based sensor is activated even when capturing images is not necessary, resulting in increased power consumption. [Means for solving the problem]

[0008] In order to solve the above problem, the imaging device of the present invention comprises an event-based sensor that outputs event information indicating the pixel position and time at which an event occurrence is detected, and a frame-based sensor that outputs image data frame by frame, and a control means that controls the frame-based sensor to start imaging when the trajectory of the event occurrence position detected by the event-based sensor heads toward the imaging range of the frame-based sensor, and is characterized in that the event-based sensor and the frame-based sensor are positioned so that their respective imaging ranges do not overlap. [Effects of the Invention]

[0009] According to the present invention, it is possible to reliably capture an image of a subject moving at high speed with a frame-based sensor, and further to reduce the power consumption of the frame-based sensor. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 2 is a diagram showing an example of the internal configuration of an imaging apparatus according to an embodiment. [Figure 2] 10A and 10B are diagrams illustrating the operation of determining the trajectory of an event by an event-based sensor in the embodiment. [Figure 3] FIG. 3 is a diagram showing a control flow of the imaging apparatus according to the first embodiment. [Figure 4] 4 is an imaging timing chart for a frame-based sensor in the embodiment. [Figure 5] FIG. 10 is a diagram showing a control flow of the imaging apparatus according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0012] Example 1 FIG. 1 is a diagram showing an example of the internal configuration of an image capturing apparatus 100 according to the first embodiment.

[0013] As shown in FIG. 1, the imaging device 100 includes a lens 101, an event-based sensor 102, a lens 103, a frame-based sensor 104, an image processing unit 105, a recording unit 106, a synchronization signal generating unit 107, a lens control unit 108, and a CPU 109.

[0014] Lens 101 is positioned to adjust the light incident on event-based sensor 102. Event-based sensor 102 has multiple pixels, detects changes in the luminance of each pixel as the occurrence of an event, and outputs an event signal indicating the pixel position and time at which the event occurred. Each pixel in event-based sensor 102 consists of a light-receiving section and a luminance change detection section. In the light-receiving section, a photodiode converts incident light into an electric charge, which is then converted into a voltage corresponding to the generated electric charge by a charge-voltage conversion circuit. The converted voltage is compared with a reference voltage in the luminance change detection section, and an event signal is output if the difference exceeds a threshold.

[0015] The lens 103 is arranged to adjust the light incident on the frame-based sensor 104. The lens 103 may have an optical zoom mechanism. The frame-based sensor 104 is a sensor that outputs image data on a frame-by-frame basis. The frame-based sensor 104 includes a plurality of pixels.

[0016] Specific examples of frame-based sensors include color sensors and monochrome sensors for capturing images. CMOS sensors and CCD sensors can be suitably used as the imaging elements of color sensors and monochrome sensors. Each pixel of a color sensor or monochrome sensor outputs a brightness value (pixel signal) on a frame-by-frame basis. The frame-based sensor 104 can capture images at multiple frame rates, and captures images based on the frame rate value set by the CPU 109.

[0017] Image processing unit 105 generates an image from the output signals of event-based sensor 102 and frame-based sensor 104. The event signal output from event-based sensor 102 indicates the position and time of a pixel where a change in brightness occurred, so by limiting the output time to within a certain period, it is possible to extract the position of the pixel where a change in brightness occurred.

[0018] The image processing unit 105 is capable of identifying the location where an event occurs at a certain time from the event signal, determining whether the event occurrence location is moving over time, and detecting the trajectory of the event (details will be described later). The image processing unit 105 also generates one image for each frame from the signal output from the frame-based sensor 104.

[0019] The recording unit 106 records each image data transmitted from the image processing unit 105. The recording unit 106 also records a program related to processing by the CPU 109, operating parameters of each functional block of the imaging device 100, and the like.

[0020] The synchronization signal generation unit 107 generates a horizontal synchronization signal (HD) and a vertical synchronization signal (VD) for the frame-based sensor 104 and the image processing unit 105 to process image data based on a reference clock input from an oscillator (not shown) and each mode of the imaging device.

[0021] The lens control unit 108 is responsible for controlling the driving of the lens 103, such as focusing, panning, and zooming. The CPU 109 is electrically connected to each functional block of the imaging device 100 via an internal bus (not shown). The CPU 109 controls each process of the imaging device 100.

[0022] Figure 2 shows the operation of determining the trajectory of an event occurrence position by event-based sensor 102. Figure 2 shows imaging range 200 of event-based sensor 102 and imaging range 201 of frame-based sensor 104. Imaging range 200 of event-based sensor 102 and imaging range 201 of frame-based sensor 104 are set so that they do not overlap. Also shown are events 210 to 213 detected at time t1, events 220 and 221 detected at time t2, and events 230 to 232 detected at time t3. The order of time is t1, t2, t3.

[0023] Among these, events 210, 220, and 230 are not sporadic, unlike the other events, and it can be seen that they are heading towards the imaging range 201 of the frame-based sensor 104, as indicated by arrow 240. In this way, the image processing unit 105 can detect the trajectory of the event occurrence position from the event information received from the event-based sensor 102, based on the event occurrence time and occurrence position.

[0024] In addition, the image processing unit 105 can calculate the amount of movement from the position where the event 210 occurs to the position where the event 230 occurs, and the movement speed of the arrow 240 from the time transition from time t1 to time t3, from multiple images of the event sensor 102.

[0025] In this embodiment, the addresses of the three occurrence positions of events 210, 220, and 230 are used to detect the trajectory of an event, but the number of addresses measured for detecting the trajectory of an event can be changed depending on the situation. For example, in the case of a subject moving at high speed, the trajectory can be determined by detecting the event at three occurrence positions as described above, but for a subject moving at low speed, the number of event occurrence positions to be measured can be increased. This makes it possible to accurately detect whether the trajectory of the event occurrence positions is moving toward the imaging range 201 of the frame-based sensor 104.

[0026] Although Fig. 2 illustrates a configuration in which one event-based sensor and one frame-based sensor are provided, it is also effective to place multiple event-based sensors for one frame-based sensor. For example, by placing multiple event-based sensors around a frame-based sensor, it is possible to detect the trajectory of the event occurrence position from the periphery of the frame-based sensor toward the imaging range of the frame-based sensor.

[0027] In addition, a drive unit (not shown) may be provided to move the event-based sensor, allowing it to move around the frame-based sensor. While the above configuration shows the event-based sensor and frame-based sensor being arranged horizontally, the event-based sensor may also be arranged perpendicular to the frame-based sensor.

[0028] 3 is a diagram showing a control flow of the imaging device 100 in the first embodiment, and shows the flow up to the start of imaging by the frame-based sensor 104. The processing of each step is performed by the CPU 109 executing a program stored in the recording unit 106.

[0029] In S301, the event-based sensor 102 is activated to transition to a state in which a change in luminance of each pixel can be detected.

[0030] In S302, image processing unit 105 detects the location where the event occurred from the event signal received from event-based sensor 102. If an event is detected (Yes in S302), the process proceeds to S303. If an event is not detected (No in S302), the process continues to wait for the detection of an event.

[0031] In S303, if a locus of event occurrence positions is detected from events detected within a predetermined period (Yes in S303), the process proceeds to S304. If a locus of event occurrence positions is not detected (No in S303), the process returns to S302. Furthermore, if a large number of events are detected within a predetermined period and a locus of event occurrence positions cannot be detected, the event information detected within the predetermined period may be discarded and the process may return to S302.

[0032] In S304, it is determined whether the trajectory of the event occurrence position is heading towards the imaging range of the frame-based sensor 104. If the trajectory of the event occurrence position is heading towards the imaging range of the frame-based sensor 104 (Yes in S304), the process proceeds to S305. If the trajectory of the event occurrence position is not heading towards the imaging range of the frame-based sensor 104 (No in S304), the process returns to S302.

[0033] In S305, the frame-based sensor 104 is activated to transition to a state where it can capture images, and the process proceeds to S306.

[0034] In S306, the frame-based sensor 104 starts capturing an image of a subject approaching the image capturing range of the frame-based sensor 104.

[0035] The above-described processing allows frame-based sensor 104 to transition to a state where it can capture an image before the subject enters the imaging range of frame-based sensor 104, thereby preventing the subject from being missed. Furthermore, if the subject to be captured does not enter the imaging range of frame-based sensor 104, frame-based sensor 104 is not activated, thereby reducing power consumption.

[0036] 4 is a timing chart showing the start of imaging by the frame-based sensor 104 in this embodiment. Based on an event signal output from the event-based sensor 102 at time A in the diagram, the frame-based sensor 104 transitions from a standby state to an imaging-enabled state. The frame-based sensor 104 starts capturing the first frame in synchronization with the next vertical synchronization signal VD (which occurs at time B in the diagram) after it has entered the imaging-enabled state.

[0037] Example 2 5 is a diagram showing a control flow of the imaging device 100 in the second embodiment, and shows the flow up to the end of imaging by the frame-based sensor 104. The processing of each step is performed by the CPU 109 executing a program stored in the record 106.

[0038] The processing from S501 to S504 is the same as the processing from S301 to S304 described with reference to FIG. 3, and therefore a description thereof will be omitted.

[0039] In S505, the moving speed of the subject is calculated based on the trajectory of the event occurrence position.

[0040] In S506, if the moving speed of the subject is equal to or greater than the predetermined speed (Yes in S506), the process proceeds to S507. If the moving speed of the subject is less than the predetermined speed (No in S506), the process proceeds to S508.

[0041] In S507, the frame-based sensor 104 is set to a mode for capturing images at a high frame rate and activated, the frame-based sensor 104 is transitioned to a state in which it is capable of capturing images, and the process proceeds to S509.

[0042] In addition, in S508, the frame-based sensor 104 is set to a mode for capturing images at a low frame rate and activated, transitioning the frame-based sensor 104 to a state in which it is capable of capturing images, and the process proceeds to S509.

[0043] Here, the process of S509 is the same as the process of S306 in FIG. 3, and the processes from S510 to S511 are the same as the processes from S302 to S303 in FIG. 3, so a description thereof will be omitted.

[0044] In S512, it is determined whether the trajectory of the event occurrence position is moving away from the imaging range of the frame-based sensor 104. If the trajectory of the event occurrence position is moving away from the imaging range of the frame-based sensor 104 (Yes in S512), the process proceeds to S513. If the trajectory of the event occurrence position is not moving away from the imaging range of the frame-based sensor 104 (No in S512), the process returns to S510.

[0045] In S513, the image capturing by the frame-based sensor 104 is terminated, and the frame-based sensor 104 is transitioned to a standby state.

[0046] By performing the above-described processing, it becomes possible to capture an image by setting the image capturing frame rate of frame-based sensor 104 according to the moving speed of the subject. Furthermore, by transitioning frame-based sensor 104 from an image capturing enabled state to a standby state when the subject moves away from the image capturing range of frame-based sensor 104, it is possible to reduce the power consumption of frame-based sensor 104.

[0047] Furthermore, in the process of S507, the lens control unit 108 may zoom out the lens 103 to control the frame-based sensor 104 to capture an image at a wider angle of view than usual. This control allows the frame-based sensor 104 to start capturing an image of the subject earlier.

[0048] Furthermore, in the process of S508, the lens control unit 108 may control the lens 103 to zoom in so that the frame-based sensor 104 captures an image at a narrower angle of view than usual. By such control, the predicted arrival point of the trajectory of the event occurrence position can be enlarged to obtain a detailed captured image.

[0049] Furthermore, if both a trajectory of the event occurrence position moving toward the imaging range of the frame-based sensor 104 and a trajectory of the event occurrence position moving away from the imaging range of the frame-based sensor 104 are detected, control may be performed to continue imaging.

[0050] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0051] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0052] The disclosure of this embodiment includes the following configuration.

[0053] (Configuration 1) an event-based sensor that outputs event information indicating information on a pixel position and time at which an event occurrence is detected; a frame-based sensor that outputs image data frame by frame; a control means for controlling the frame-based sensor to start capturing an image when a trajectory of an event occurrence position detected by the event-based sensor is heading toward an image capturing range of the frame-based sensor; The imaging device is characterized in that the event-based sensor and the frame-based sensor are arranged so that their imaging ranges do not overlap.

[0054] (Configuration 2) The imaging device according to configuration 1, wherein the control means calculates a moving speed of the subject based on a trajectory of the event occurrence position, and changes an imaging frame rate of the frame-based sensor in accordance with the calculated moving speed of the subject.

[0055] (Configuration 3) 3. The imaging device according to configuration 2, wherein the control means controls the imaging angle of view to be changed in accordance with the imaging frame rate of the frame-based sensor.

[0056] (Configuration 4) 4. The imaging device according to any one of configurations 1 to 3, wherein the control means controls the frame-based sensor to terminate imaging when a trajectory of an event occurrence position detected by the event-based sensor moves away from an imaging range of the frame-based sensor.

[0057] (Configuration 5) The imaging device according to configuration 4, wherein the control means controls the frame-based sensor to capture an image when the event occurrence position detected by the event-based sensor detects both a trajectory toward the imaging range of the frame-based sensor and a trajectory away from the imaging range of the frame-based sensor.

[0058] (Configuration 6) The imaging device according to any one of configurations 1 to 5, wherein the control means discards the event signal when the positions of occurrence of multiple events detected by the event-based sensor cannot be determined as a trajectory.

[0059] (Configuration 7) 7. The imaging device according to any one of configurations 1 to 6, wherein the imaging device has a plurality of the event-based sensors for one of the frame-based sensors.

[0060] (Configuration 8) Further, a moving means for moving the event-based sensor is provided, 8. The imaging device according to any one of configurations 1 to 7, wherein the event-based sensor is moved around the frame-based sensor by the moving means to detect the occurrence of an event. [Explanation of symbols]

[0061] 100 Imaging device 102 Event-based sensors 104 Frame-based Sensor 105 Image processing section

Claims

1. an event-based sensor that outputs event information indicating information on a pixel position and time at which an event occurrence is detected; a frame-based sensor that outputs image data frame by frame; a control means for controlling the frame-based sensor to start capturing an image when a trajectory of an event occurrence position detected by the event-based sensor is heading toward an image capturing range of the frame-based sensor; The imaging device is characterized in that the event-based sensor and the frame-based sensor are arranged so that their imaging ranges do not overlap.

2. 2. The imaging device according to claim 1, wherein the control means calculates a moving speed of the subject based on the trajectory of the event occurrence position, and changes an imaging frame rate of the frame-based sensor in accordance with the calculated moving speed of the subject.

3. 3. The imaging device according to claim 2, wherein the control means controls the imaging angle of view to be changed in accordance with the imaging frame rate of the frame-based sensor.

4. 2. The imaging device according to claim 1, wherein the control means controls the frame-based sensor to terminate imaging when a trajectory of an event occurrence position detected by the event-based sensor moves away from an imaging range of the frame-based sensor.

5. 5. The imaging device according to claim 4, wherein the control means controls the frame-based sensor to capture an image when the event occurrence position detected by the event-based sensor detects both a trajectory toward the imaging range of the frame-based sensor and a trajectory away from the imaging range of the frame-based sensor.

6. 2. The imaging device according to claim 1, wherein the control means discards the event signal when the positions at which a plurality of events have occurred detected by the event-based sensor cannot be determined as a locus.

7. 2. The imaging device according to claim 1, wherein the imaging device has a plurality of the event-based sensors for one of the frame-based sensors.

8. Further, a moving means for moving the event-based sensor is provided, 2. The imaging device according to claim 1, wherein the event-based sensor is moved around the frame-based sensor by the moving means to detect the occurrence of an event.

Images