Image processing apparatus, imaging apparatus, image processing method, and program

Abstract

To provide an image processing apparatus that allows a user to check an image at desired timing.SOLUTION: An image processing apparatus (100) comprises: an imaging sensor (111) that, when a change in the brightness of every pixel exceeds a predetermined threshold, detects the occurrence of an event, and outputs an event signal including information on the time of occurrence of the event and a pixel position where the event occurs; driving units (106, 107) that drive at least either one of an optical member (108) constituting at least a part of an imaging optical system (109) or the imaging sensor so that the change in brightness exceeds the predetermined threshold; and a processing unit (112) that processes the event signal output from the imaging sensor while the driving units drive at least either one of the optical member or the imaging sensor.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, an imaging device, an image processing method, and a program. [Background technology]

[0002] Conventionally, imaging devices equipped with an event-driven vision sensor (event-based sensor) are known. The event-based sensor detects the occurrence of an event based on a luminance change in each pixel, and asynchronously outputs an event signal including the time when the event occurred and the pixel position. In this way, the event-based sensor can detect the occurrence of an event when the luminance change exceeds a predetermined threshold, and has low latency and low calculation cost compared to reading out pixel signals of all pixels.

[0003] In addition, because event-based sensors convert brightness into voltage logarithmically, they can detect slight brightness differences in low brightness conditions, while reacting to large brightness differences in high brightness conditions, preventing events from becoming saturated and providing a wide dynamic range.Event-based sensors also have a high time resolution for event information, ranging from several ns to several μs, and subject blurring for moving subjects is minimal.

[0004] Patent Document 1 discloses an event camera that generates an image (frame data) from an event signal output from an event-based sensor. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2020-182122 A Summary of the Invention [Problem to be solved by the invention]

[0006] In the event camera disclosed in Patent Document 1, when no change in luminance occurs, the event-based sensor does not output an event signal, and therefore the user cannot check the image.

[0007] SUMMARY OF THE PRESENT EMBODIMENT An object of the present invention is to provide an image processing device that allows a user to check an image at a desired timing. [Means for solving the problem]

[0008] An image processing device as one aspect of the present invention has an imaging sensor that detects the occurrence of an event when a change in luminance for each pixel exceeds a predetermined threshold, and outputs an event signal including information about the time the event occurred and the pixel position at which the event occurred, a drive unit that drives at least one of an optical element that constitutes at least a part of an imaging optical system or the imaging sensor so that the change in luminance exceeds the predetermined threshold, and a processing unit that processes the event signal output from the imaging sensor while the drive unit drives at least one of the optical element or the imaging sensor.

[0009] Other objects and features of the present invention will be described in the following embodiments. Effect of the Invention

[0010] According to the present invention, it is possible to provide an image processing device that allows a user to check an image at a desired timing. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram of an imaging device according to a first embodiment. [Diagram 2] FIG. 2 is an explanatory diagram of an event generated by an event-based sensor in the first embodiment. [Diagram 3] 4 is an example of an event image in the first embodiment. [Figure 4] FIG. 4 is an explanatory diagram of the movement amount of the motion compensation lens in the first embodiment. [Diagram 5] FIG. 4 is an explanatory diagram of an event image in the first embodiment. [Figure 6] FIG. 11 is a block diagram of an imaging device according to a second embodiment. [Figure 7] FIG. 11 is an explanatory diagram of a focus determination method in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the embodiments of the present invention will be described in detail with reference to the drawings. Note that, although multiple features are described in each embodiment, not all of these multiple features are necessarily essential to the present invention, and multiple features may be combined in any manner. In addition, in the drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0013] (First embodiment) First, an imaging device (image processing device) 100 according to a first embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a block diagram of the imaging device 100. The imaging device 100 has an event-based sensor (imaging sensor) 111. The imaging device 100 also has an optical shake correction mechanism.

[0014] Imaging device 100 has at least two modes for switching the use of the optical blur correction mechanism. One of the two modes is a blur correction mode (second mode) in which the blur correction mechanism is driven based on vibrations applied to imaging device 100. The other is an event issuing mode (first mode) in which blur correction lens (optical member) 108 is driven to cause event-based sensor 111 to issue an event (output an event signal). Note that, as will be described in detail later, in the event issuing mode, blur correction lens 108 is minutely driven regardless of vibrations applied to imaging device 100. This changes the imaging position of the subject image on event-based sensor 111, forcibly causing a change in luminance for each pixel in event-based sensor 111. Note that in each embodiment, vibrations applied to the imaging device are referred to as "shake", and a shift in the subject position between frames of an imaged image caused by a shake applied to the imaging device or a blur of the subject image is referred to as "shake".

[0015] The event-based sensor 111 detects a luminance change for each pixel within an imaging range and asynchronously outputs an event signal. The event-based sensor 111 is configured, for example, by arranging a plurality of pixels in an array, and generates a trigger signal and outputs it as an event signal when a change in a voltage signal (luminance change) that is the logarithm of the intensity of light incident on each pixel exceeds a predetermined threshold. The event signal is a signal associated with an event. For example, it includes the time when the occurrence of the event is detected (occurrence time) and the pixel position where the event occurs. The time when the occurrence of the event is detected may be measured based on an internal clock of the event-based sensor 111 (the time of the event-based sensor 111) or may be reset as necessary.

[0016] The event signal includes information about the time when the event occurred and the pixel position where the event occurred, and may further include information about a change in luminance. The information about the change in luminance may be the amount of change in luminance itself, or information indicating whether the luminance change is positive or negative. The event-based sensor 111 asynchronously outputs the event signal only when a change in luminance occurs (when the change in luminance exceeds a predetermined threshold). Here, "outputting the event signal asynchronously" means that the event signal is output independently in time on a pixel-by-pixel basis, without synchronization among all pixels of the event-based sensor 111.

[0017] Here, events generated by event-based sensor 111 will be described with reference to Fig. 2. Fig. 2 is an explanatory diagram of events generated by event-based sensor 111. In Fig. 2, the horizontal axis is time t, and the vertical axis is voltage V, which is the logarithm of the intensity of light incident on event-based sensor 111. P In addition, in FIG. 2, the voltage V P The horizontal dotted line drawn from indicates the threshold value (predetermined threshold value) of the voltage signal at which the event-based sensor 111 generates a trigger signal, and is set in units of the amount of change in voltage (threshold value Θ). The lower diagram in Fig. 2 shows the detection timing of an event. That is, when the voltage signal (amount of change in voltage) increases beyond the threshold value Θ (predetermined threshold value), it is represented by an upward arrow ("+ event"), and conversely, when the voltage signal decreases beyond the threshold value Θ, it is represented by a downward arrow ("- event").

[0018] 1, data processing unit (processing unit) 112 receives an event signal output from event-based sensor 111, processes the received event signal, and generates image data (event image) from the event signal. In this embodiment, data processing unit 112 processes the event signal output from event-based sensor 111 while a driving unit drives blur correction lens 108, as described below.

[0019] Here, an example of an event image will be described with reference to Fig. 3. Fig. 3 shows an example of an event image generated based on an event signal output from event-based sensor 111. Image 30 is an image captured by a general imaging device equipped with an imaging element such as a CMOS image sensor or a CCD image sensor. Image 30 includes detailed data even for static background parts with no change in luminance.

[0020] On the other hand, image 31 is an event image (framed event image) generated as one frame from a plurality of events that occurred during a period equivalent to the period during which a general imaging element such as a CMOS image sensor accumulated light to generate image 30. In image 31, black areas (black pixels) represent "- events" and white areas (white pixels) represent "+ events". Gray areas are pixel areas where no events have occurred. The outline of the area where a person is moving (moving from right to left in image 31) has black or white pixels, and the movement of the person can be recognized by detecting the change in luminance. On the other hand, a static background part (such as a pedestrian crossing) is a gray area because there is no change in luminance. Note that the method of expressing each of the black areas, white areas, and gray areas is not limited to the above example, and other colors may be used, and the pixel values ​​may be changed according to the intensity level of the luminance change to generate the image.

[0021] 3, image 31 contains much less data per given time period than image 30, making post-processing easier and more efficient for tracking or recognizing changes in a scene. Image 31 generated by data processing unit 112 can be displayed on an output device (not shown) for a user to view, or image 31 may be recorded in a recording medium (not shown) in conjunction with the occurrence of an event.

[0022] Next, the shake correction function will be described. The shake detection sensor 101 is a sensor that detects shake applied to the imaging device 100, and is, for example, an angular velocity sensor that detects angular velocity occurring in the imaging device 100. The shake correction amount calculation unit 102 calculates a target position (movement target position) of the shake correction lens 108 based on the output signal of the shake detection sensor 101. The shake correction amount calculation unit 102 has, for example, an HPF (high pass filter) for removing unnecessary offset output from the output signal of the angular velocity sensor. The shake correction amount calculation unit 102 also has, for example, an integrator for converting angular velocity shake data into angles, a unit conversion unit for converting angle data into units of position information of the shake correction lens 108, and a phase compensation filter for compensating for phase delay of the shake detection sensor 101 itself. The shake correction lens movement amount selection unit 103 selects an appropriate target position of the shake correction lens 108 according to the current mode, based on the target position calculated by the shake correction amount calculation unit 102 and a vibration target position described later. The mode is one of at least two modes: the blur correction mode and the event issuing mode described above.

[0023] Next, a description will be given of the control for moving (driving) the motion compensation lens 108 to the target position output from the motion compensation lens movement amount selection unit 103. The motion compensation lens 108 is driven by feedback control based on the difference data between a signal indicating the target position and the output signal of the position detection sensor 110. The difference data obtained by subtracting the output signal of the position detection sensor 110 from the signal indicating the target position is output to the control filter 104. The control filter 104 performs signal processing such as amplification and phase compensation on the difference data. The pulse width modulation unit 105 modulates the output data of the control filter 104 into a waveform that changes the duty ratio of the pulse wave, i.e., a PWM waveform.

[0024] The motor driving unit 106 is a circuit that applies a driving signal to the motor 107. For example, the motor driving unit 106 is an H-bridge circuit, and the motor 107 is a voice coil type motor, but is not limited to these. In this embodiment, the motor driving unit 106 and the motor 107 constitute a driving unit that drives the blur correction lens 108 so that the luminance change exceeds a predetermined threshold value.

[0025] Imaging lens unit (imaging optical system) 109 has a blur correction lens 108 and forms a subject image on event-based sensor 111. Imaging lens unit 109 includes, for example, at least one of a zoom lens or a focus lens, but is not limited to these.

[0026] The motion compensation lens 108 is, for example, a shift lens, and is capable of deflecting the optical axis OA by moving in a direction different from the direction along the optical axis OA (the optical axis direction). However, the motion compensation lens 108 is not limited to a shift lens, and other optical members may be used as long as they are capable of deflecting the optical axis OA, such as a mechanism (variable angle prism) that injects liquid between the lenses to change the shape of the lenses themselves to deflect the optical axis OA. By moving the motion compensation lens 108 in accordance with the shake of the imaging device 100, the change in the imaging position of the subject caused by the shake of the imaging device 100 can be canceled by deflecting the optical axis OA, and the imaging position of the subject image can be kept at a predetermined position.

[0027] The position detection sensor 110 has a magnet and a Hall sensor. The positional relationship between the magnet and the Hall sensor changes as the motion compensation lens 108 moves, and the magnetic flux density received by the Hall sensor changes, causing the output of the Hall sensor to change.

[0028] The event issuing mode setting unit (setting unit) 113 notifies the fixed movement amount calculation unit 114 that the current mode is the event issuing mode (first mode). The event issuing mode can be selected and set by the user performing a menu operation. The event issuing mode setting unit 113 can set, for example, the event issuing mode (first mode) or the blur correction mode (second mode). Note that the event issuing mode setting unit 113 may be configured to automatically set the event issuing mode when the imaging device 100 is started up or initialized. If the event mode is not selected, as described above, the blur correction mode (second mode) in which the blur correction lens 108 is moved based on the shake of the imaging device 100 is selected. However, even if the event issuing mode is not selected, the user can select to enable or disable the blur correction.

[0029] Fixed movement amount calculation section 114 outputs a fixed movement amount unrelated to shake of imaging device 100 as the movement amount of motion compensation lens 108. Movement of motion compensation lens 108 by the fixed movement amount causes the imaging position of the subject image to change by a greater amount than the distance between pixels (pixel pitch) of event-based sensor 111. When the imaging position of the subject image changes by an amount equal to or greater than the pixel pitch of event-based sensor 111, each pixel detects a luminance change, and an event is issued (detected). The movement target position of motion compensation lens 108 at this time is set as the vibration target position.

[0030] Next, specific examples of vibration target positions will be described with reference to Figures 4(A) to 4(C). Figures 4(A) to 4(C) are explanatory diagrams of the amount of movement of the motion compensation lens 108.

[0031] In FIG. 4(A), the grid indicates pixels arranged in an array in the event-based sensor 111. The pixels are arranged at positions where horizontal and vertical lines intersect, and the distance P corresponds to the pixel pitch. FIG. 4(B) is an explanatory diagram of the movement amount (vibration target position) of the motion compensation lens 108. In FIG. 4(B), the horizontal axis indicates time, and the vertical axis indicates the target position. The movement amount of the motion compensation lens 108 is defined as the amount of deflection of the optical axis OA, that is, an angle, and is expressed by deg_X as shown in FIG. 4(B). Frame_T in FIG. 4(B) is the exposure time of one frame in the imaging device 100. FIG. 4(B) shows that the motion compensation lens 108 is moved by an angle deg_X within the exposure time of one frame. However, the purpose here is to generate an image in which luminance changes are accumulated for one frame, as in the image 31. For this reason, the movement of the motion compensation lens 108 can achieve its purpose as long as it is within the time of Frame_T, and the movement may be completed in a time shorter than Frame_T.

[0032] With reference to FIG. 4(C), the relationship between the distance P between pixels (pixel pitch) and the amplitude (angle deg_X) of the vibration target position will be described. Note that imaging lens unit 109 includes motion compensation lens 108, but is illustrated in a simplified manner for ease of explanation. The optical axis OA passing through imaging lens unit 109 passes through approximately the center of event-based sensor 111 when motion compensation lens 108 is not moving. On the other hand, by moving motion compensation lens 108, the optical axis OA becomes tilted by angle deg_X. When motion compensation lens 108 moves from a stationary state to a vibration target position, the imaging position of the subject image on event-based sensor 111 changes. Here, the amount of change in the imaging position is assumed to be D.

[0033] The focal length f of imaging lens unit 109 is in the same unit as the amount of change D, for example, millimeters. Here, angle deg_X can generally be expressed by the formula arctan(D / f). To cause a luminance change in each pixel in event-based sensor 111, the amount of change D of the imaging position on event-based sensor 111 needs to be greater than the distance P between pixels (pixel pitch). For example, D=P may be satisfied, and in this case, angle deg_x, which is the vibration target position, is expressed by arctan(P / f). Note that conversion between radians and degrees is omitted here to simplify the formula.

[0034] Next, with reference to Figs. 5(A) to (F), an event image resulting from the movement of the motion compensation lens 108 will be described. Here, an example will be described in which a chart is photographed with a white color on the right side and a black color on the left side. Fig. 5(A) shows the imaging range before the motion compensation lens 108 is moved. When the motion compensation lens 108 is moved by a predetermined amount, the black area within the imaging range becomes larger. If an image acquired when photographing with a general imaging device equipped with a CMOS image sensor is described as a frame image, the frame image photographed in the state of Fig. 5(A) is the image in Fig. 5(C). Moreover, the frame image photographed in the state of Fig. 5(B) is the image in Fig. 5(D).

[0035] Here, it is assumed that the motion compensation lens 108 has not moved before the image of FIG. 5(A) is captured, and if there is no change in the brightness on the chart, the event-based sensor 111 does not issue any event, and the image shown in FIG. 5(E) is obtained. As described above, this is an image in which the areas where no events have occurred are represented in gray, and other color schemes may be used. Meanwhile, among the pixels in which the white part of the chart was captured before the motion compensation lens 108 was moved, there are pixels in which the black part of the chart becomes the captured area after the motion compensation lens 108 is moved. As for this area, the brightness decreases due to the movement of the motion compensation lens 108, so it becomes black, indicating a negative event, as shown in FIG. 5(F). Note that, although a simple two-color chart has been described as an example in FIGS. 5(A) to (F), by moving the motion compensation lens 108, a fixed subject present within the captured area can be viewed as an event image in which the outline is extracted, as shown in FIG. 5(F) by using a similar principle.

[0036] According to this embodiment, in the image capturing device 100 having the event-based sensor 111, even if no change in luminance occurs within the image capturing range, the image capturing position of the subject image is slightly shifted using the image blur correction lens 108, thereby forcibly issuing an event.

[0037] Below, variations in the method of causing the event-based sensor 111 to issue an event will be described.

[0038] First, an example of using a shake correction mechanism will be described. A mechanism may be used in which an actuator is mounted on event-based sensor 111 (a stage that holds event-based sensor 111) to move the stage itself. Alternatively, a pan-tilt mechanism may be used that can rotate a camera section (imaging section) that integrates imaging lens unit 109 and event-based sensor 111 up, down, left, and right. In event issuance mode, the image position of the subject image is changed by driving these shake correction mechanisms by a fixed amount to issue an event.

[0039] Alternatively, instead of the imaging position of the subject image, an event may be issued by adjusting the amount of light passing through imaging lens unit 109 to change the luminance value detected by each pixel of event-based sensor 111. Specifically, a mechanism (light amount adjustment unit) for changing the amount of light (brightness), such as a diaphragm (aperture diaphragm) or a neutral density filter in imaging lens unit 109, may be used.

[0040] Alternatively, an event may be issued by changing the angle of view by moving the zoom lens or focus lens constituting the imaging lens unit 109 by a small amount in the optical axis direction.

[0041] Although the above-mentioned configurations are event issuing methods that utilize mechanisms for implementing functions that are normally included in image capture devices, such as image blur correction or exposure adjustment, a mechanism solely for issuing an event may be provided. For example, a vibration generating member may be provided between an image capture unit that integrates image capture lens unit 109 and event-based sensor 111, and a fixed unit on which the image capture device is installed. In this case, when the event issuing mode is selected, an event may be issued by changing the positional relationship between the image capture unit and the subject using the vibration generating member.

[0042] In this way, the driving unit can drive at least one of the optical members constituting at least a part of the imaging lens unit 109 and the event-based sensor 111 so that the luminance change exceeds a predetermined threshold. Therefore, according to this embodiment, the user can generate an event image at a desired timing, and can perform work such as setting up the imaging device (image processing device) while checking the event image.

[0043] Second embodiment Next, an imaging device (image processing device) 200 according to a second embodiment of the present invention will be described with reference to Fig. 6. Fig. 6 is a block diagram of the imaging device 200. The imaging device 200 has an event-based sensor 111, similar to the imaging device 100 described with reference to Fig. 1. Furthermore, in addition to the components of the imaging device 100, the imaging device 200 has an autofocus function.

[0044] The imaging lens 201 has at least the motion compensation lens 108 and a focus lens 202. The focus lens 202 is a focus compensator lens that moves in the optical axis direction.

[0045] The focus signal processing unit 203 generates a focus signal based on the event image output from the data processing unit 112. The focus signal is a value indicating the sharpness (contrast state) of an image, and represents the focus state of the imaging optical system. When the focus state is in focus, the sharpness is high. On the other hand, when the focus state is blurred (out of focus), the sharpness is low. In this way, the focus signal can be used as a value indicating the focus state of the imaging optical system. In addition to the focus signal, the focus signal processing unit 203 generates signals such as a luminance difference signal (the difference between the maximum and minimum values ​​of the luminance level of the area used for focus detection). Regarding the detection of the luminance level, since the event-based sensor 111 does not have a mechanism for detecting absolute luminance information, it is preferable to provide a sensor capable of detecting luminance (not shown) separately from the event-based sensor 111. The focus lens control unit 204 drives the focus lens 202 based on the output signal of the focus signal processing unit 203.

[0046] Next, a change in focus signal (focus determination method) according to the position (focus lens position) of focus lens 202 will be described with reference to Figures 7(A) and (B). Figures 7(A) and (B) are explanatory diagrams of a change in focus signal (focus determination method) according to the focus lens position for a specified subject.

[0047] 7A and 7B, the closer the focus lens position is to the in-focus position, the higher the sharpness becomes, and the larger the focus signal value becomes. On the other hand, the farther the focus lens position is from the in-focus position, the smaller the focus signal value becomes. The autofocus function is a function that moves the focus lens position while checking the value of the focus signal, and searches for the position where the focus signal value is the largest.

[0048] Also, an index called a simple focusing degree may be used to switch the control of the focus lens 202. For example, when the determination using the simple focusing degree indicates that the focus lens 202 is not in focus, the moving speed of the focus lens 202 is increased to quickly reach the vicinity of the in-focus point. On the other hand, in an area where the simple focusing degree indicates that the focus lens 202 is in focus, the moving speed of the focus lens 202 is decreased to perform a detailed search. As described above, the focus signal TEP is a value obtained by extracting high-frequency components from the video signal. Using the difference MMP between the maximum and minimum luminance levels of the area used for focus detection, the simple focusing degree can be calculated by dividing the focus signal TEP by the difference MMP.

[0049] The dotted line 702 in FIG. 7(A) and (B) shows the concept of how the differential MMP changes depending on the focus lens position. As shown in FIG. 7(A) and (B), the dotted line 702 has a smaller increase or decrease in value depending on the focus lens position compared to the focus signal described above. This is because the maximum and minimum values ​​of the luminance level are substantially the same regardless of the blur state as long as the subject is the same, and the fluctuation of the focus signal due to the subject can be suppressed to some extent. Therefore, in this embodiment, when the value of the simple focusing degree (TEP / MM) is 55% or more (area 703), it is determined that the focus state is in focus. On the other hand, when the simple focusing degree is 40% or more and less than 55% (area 704), it is determined that the small blur state is present. Also, when the simple focusing degree is less than 40% (area 705), it is determined that the large blur state is present. However, the ratio (determination value) of the simple focusing degree is not limited to the above ratio.

[0050] Here, when the shake correction lens 108 is moved to issue an event from the event-based sensor 111, the sharpness of the event image decreases. As described in the first embodiment, since the shake correction lens 108 is moved so that the imaging position changes by the pixel pitch, that is, by one pixel, a subject blur of one pixel occurs. Also, as described above, if the subject is the same, even if the blur state changes, the maximum and minimum values of the luminance level can be regarded as substantially the same. Therefore, the maximum and minimum values of the luminance level do not change even when the shake correction lens 108 is moved. This is as shown in the graphs of the solid line 701 and the dotted line 702 in FIGS. 7(A) and 7(B). That is, the simple focus degree calculated from the image obtained when the shake correction lens 108 is moved has a lower numerical value than the simple focus degree calculated from the image obtained when the shake correction lens 108 is not moved.

[0051] Therefore, considering that the simple focus degree in the event issuance mode for issuing an event image by moving the shake correction lens 108 becomes lower than the original numerical value, it is preferable to change the determination threshold value. As shown in FIG. 7(B), in the forced issuance mode involving the movement of the shake correction lens 108, if the simple focus degree is X1% or more (region 703), it is determined to be in the in-focus state, and if it is less than X2% (region 705), it is determined to be in the out-of-focus state, satisfying 55 < X1 and 40 < X2.

[0052] The judgment value (ratio) merely indicates that the threshold value used for judgment based on the simple focusing degree in the event issuing mode is lower than the threshold value used in an imaging device equipped with a normal CMOS imaging sensor, and is not limited to the value given in this embodiment. As an example, X1 and X2 may be measured by measuring the degree of decrease in the simple focusing degree relative to the amount of movement of the motion compensation lens 108 and storing the values ​​in advance as table data. The focus signal processing unit 203 obtains the amount of movement of the motion compensation lens 108 from the fixed movement amount calculation unit 114, and obtains X1 and X2 corresponding to the obtained amount of movement from the table. The focus signal processing unit 203 uses the obtained X1 and X2 for judgment based on the simple focusing degree, thereby making it possible to make an appropriate judgment even when the motion compensation lens 108 is moving.

[0053] In this embodiment, it is possible to achieve autofocus performance equivalent to that of an imaging device equipped with a CMOS image sensor by performing an autofocus operation that takes into consideration the reduction in sharpness of the subject image caused by moving the motion compensation lens 108. Note that in this embodiment, the object to be driven by the drive unit is not limited to the motion compensation lens 108, as in the first embodiment, but may be a mechanism for moving a stage on which the event-based sensor 111 is mounted, a variable angle prism, a pan-tilt mechanism, or the like.

[0054] In each embodiment, the imaging device is described as a digital camera, but is not limited thereto. Each embodiment can be applied to other devices with an event-driven vision sensor. That is, each embodiment can be applied to a mobile phone terminal, a portable image viewer, a television equipped with a camera, a digital photo frame, a music player, a game machine, an electronic book reader, an industrial device, a measuring device, and the like. Furthermore, each embodiment is not limited to the imaging device, and can be applied to an image processing device that does not have an imaging function but has a function of playing back moving images. (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0055] According to each embodiment, it is possible to provide an image processing device, an imaging device, an image processing method, and a program that allow a user to check an image at a desired timing.

[0056] The disclosure of each embodiment includes the following configurations and methods.

[0057] (Configuration 1) an image sensor that detects the occurrence of an event when a luminance change for each pixel exceeds a predetermined threshold, and outputs an event signal including information about a time when the event occurred and a pixel position where the event occurred; a drive unit that drives at least one of an optical member that constitutes at least a part of an imaging optical system or the imaging sensor so that the luminance change exceeds the predetermined threshold value; and a processing unit that processes the event signal output from the image sensor while the driving unit drives at least one of the optical member and the image sensor. (Configuration 2) 2. The image processing device according to configuration 1, wherein the drive section drives the optical member or the image sensor to change an image forming position on the image sensor. (Configuration 3) The driving unit drives the optical member, 3. The image processing device according to configuration 2, wherein the optical member is a shift lens or a variable angle prism that deflects the optical axis of the imaging optical system. (Configuration 4) the driving unit drives the image sensor; 3. The image processing device according to configuration 2, wherein the imaging sensor is movable in a direction including a component perpendicular to an optical axis of the imaging optical system. (Configuration 5) 3. The image processing device according to configuration 2, wherein the drive unit rotates the imaging optical system and the imaging sensor in a pan direction and a tilt direction. (Configuration 6) The driving unit drives the optical member, 3. The image processing device according to configuration 2, wherein the optical member is a light amount adjustment unit that adjusts the amount of light passing through the imaging optical system. (Configuration 7) 7. The image processing device according to configuration 6, wherein the light amount adjustment unit is an aperture or a neutral density filter. (Configuration 8) The driving unit drives the optical member, the optical member is a zoom lens, 3. The image processing device according to claim 2, wherein the image sensor outputs the event signal when the zoom lens moves in the optical axis direction of the imaging optical system to change the focal length. (Configuration 9) Further comprising a setting unit capable of setting the first mode or the second mode, The drive unit is When the first mode is set, at least one of the optical member and the image sensor is driven so that the luminance change exceeds the predetermined threshold value; 9. The image processing device according to any one of configurations 1 to 8, wherein when the second mode is set, the driving is not performed. (Configuration 10) A shake detection unit that detects shake of the image processing device; A setting unit capable of setting the first mode or the second mode, The drive unit is When the first mode is set, at least one of the optical member and the image sensor is driven so that the luminance change exceeds the predetermined threshold value; The image processing device according to any one of configurations 1 to 8, characterized in that when the second mode is set, at least one of the optical member and the imaging sensor is driven to a target position generated based on the output signal of the shake detection unit. (Configuration 11) a focus lens control unit that controls a focus lens of the imaging optical system; a determination unit that determines a focus state based on the image processed by the processing unit, 11. The image processing device according to any one of configurations 1 to 10, wherein the determination unit changes a threshold value at which the in-focus state is determined based on an amount of driving of the optical member or the image sensor by the drive unit. (Configuration 12) 12. An imaging device comprising: an image processing device according to any one of configurations 1 to 11; and an imaging optical system. (Method 1) driving an optical member constituting at least a part of an imaging optical system or an imaging sensor; detecting an occurrence of an event when a luminance change for each pixel of the image sensor exceeds a predetermined threshold while the optical member or the image sensor is being driven, and outputting an event signal including information on a time when the event occurred and a pixel position where the event occurred; and processing the event signal output from the image sensor. (Configuration 13) A program for causing a computer to execute the image processing method according to Method 1.

[0058] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0059] 100, 200 Imaging device (image processing device) 106 Motor drive unit (drive unit) 107 Motor (drive unit) 108 Image stabilization lens (optical component) 109 Imaging lens unit (imaging optical system) 111 Event-based sensor (imaging sensor) 112 Data processing unit (processing unit)

Claims

1. an image sensor that detects the occurrence of an event when a change in luminance of each pixel exceeds a predetermined threshold, and outputs an event signal that includes information about the time when the event occurred and the pixel position where the event occurred; a driving unit that drives at least one of an optical member that constitutes at least a part of an imaging optical system or the imaging sensor so that the change in luminance exceeds the predetermined threshold; a processing unit that processes the event signal output from the image sensor while the driving unit drives at least one of the optical member and the image sensor; a control unit that controls a focus lens of the imaging optical system based on the image processed by the processing unit.

2. 2. The image processing device according to claim 1, wherein the driving section drives the optical member or the image sensor to change the image forming position on the image sensor.

3. the driving unit drives the optical member, 3. The image processing device according to claim 2, wherein the optical member is a shift lens or a variable angle prism that deflects the optical axis of the imaging optical system.

4. the driving unit drives the image sensor; 3. The image processing apparatus according to claim 2, wherein the image sensor is movable in a direction including a component perpendicular to the optical axis of the imaging optical system.

5. 3. The image processing device according to claim 2, wherein the drive unit rotates the imaging optical system and the imaging sensor in a pan direction and a tilt direction.

6. the driving unit drives the optical member, 3. The image processing apparatus according to claim 2, wherein the optical member is a light amount adjusting section that adjusts the amount of light passing through the imaging optical system.

7. 7. The image processing device according to claim 6, wherein the light amount adjusting unit is an aperture or a neutral density filter.

8. the driving unit drives the optical member, the optical member is a zoom lens, 3. The image processing device according to claim 2, wherein the image sensor outputs the event signal when the zoom lens moves in the direction of the optical axis of the imaging optical system to change the focal length.

9. Further comprising a setting unit that can set the first mode or the second mode, The drive unit is When the first mode is set, at least one of the optical member and the image sensor is driven so that the change in luminance exceeds the predetermined threshold; 2. The image processing apparatus according to claim 1, wherein the driving is not performed when the second mode is set.

10. a shake detection unit that detects shake of the image processing device; a setting unit that can set the first mode or the second mode, The drive unit is When the first mode is set, driving at least one of the optical member and the image sensor so that the change in luminance exceeds the predetermined threshold; 2. The image processing apparatus according to claim 1, wherein when the second mode is set, at least one of the optical member and the image sensor is driven to a target position generated based on the output signal of the shake detection unit.

11. The image processing device described in Claim 1, characterized in that the control unit controls the focus lens of the imaging optical system based on the image processed by the processing unit and the driving amount of the optical element or the imaging sensor by the driving unit.

12. Further comprising a determination unit that determines a focus state based on the image processed by the processing unit, 12. The image processing device according to claim 11, wherein the determining section changes the threshold value for determining that the image is in focus based on the amount of driving of the optical member or the image sensor by the driving section.

13. An imaging device comprising: the image processing device according to claim 1; and an imaging optical system.

14. driving an optical member constituting at least a part of an imaging optical system or an imaging sensor; detecting the occurrence of an event when a change in luminance of each pixel of the image sensor exceeds a predetermined threshold while the optical member or the image sensor is being driven, and outputting an event signal including information on the time of the event occurrence and the pixel position where the event occurred; processing the event signal output from the imaging sensor; and a control step of controlling a focus lens of the imaging optical system based on the image processed by the processing unit.

15. A program causing a computer to execute the image processing method according to claim 14.

Images