Imaging device

Abstract

To provide an imaging apparatus which has the superiority of an event camera such as time-variant high resolution, high dynamic range, data saving and power saving while suppressing the output saturation state of event data.SOLUTION: Output restriction processing by a control unit 11 can restrict output of event data generated by an imaging element 13a by outputting event data of a pixel which has a luminance change for a limited area where a prescribed imaging area S is limited to a narrow area, and sequentially switches the limited area so as to cover the prescribed imaging area S completely. Image generation processing by the control unit 11 generates a photographed image of the prescribed imaging area S by aggregating the event data output every time the limited area is switched in the time of output restriction of the event data by the output restriction processing.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an imaging device. [Background technology]

[0002] In recent years, the event camera (event-based camera) disclosed in Patent Document 1 listed below has become known as a technology for generating images of a captured object faster. The event camera is a brightness value differential output camera, developed inspired by the retinal structure of living organisms, and is configured to sense changes in brightness for each pixel and output the coordinates, time, and polarity of the brightness change. This configuration allows the event camera to not output pixel information without brightness changes, i.e., redundant data, as in conventional cameras. This not only enables the event camera to generate images of the captured object faster, but also provides advantages such as high temporal resolution, a high dynamic range, reduced data consumption, and reduced power consumption. In particular, Patent Document 1 listed below reconstructs an image of an information code by estimating the brightness of each cell constituting the information code based on events output in response to pixels where brightness changes occur when the event camera and the information code are moved relative to each other. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-082273 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, when an event camera captures an image of a QR code (registered trademark) passing by at close range, the two-dimensional black-and-white changes result in the instantaneous output of a large amount of event data due to the high temporal resolution of the event camera. This creates a problem in that inexpensive event cameras that cannot instantaneously output a large amount of event data reach their limit and become saturated with the event data that should be output, resulting in some data being lost without being output. This problem is not limited to the above-mentioned case of capturing an image of a QR code, but also occurs, for example, when capturing an image in connection with SLAM (Simultaneous Localization and Mapping: simultaneous execution of self-localization and environmental map creation).

[0005] The present invention has been made to solve the above-mentioned problems, and its purpose is to provide an imaging device that has the advantages of an event camera, such as high temporal resolution, high dynamic range, reduced data consumption, and reduced power consumption, while suppressing output saturation of event data. [Means for solving the problem]

[0006] In order to achieve the above object, the imaging device (10) according to claim 1 of the claims comprises: an imaging element (13a) that generates event data including two-dimensional point data that identifies the position of a pixel corresponding to a pixel that has experienced a luminance change; an image generating unit (11) that generates a captured image of a predetermined imaging area (S) using the event data output from the imaging element; Outputting event data of pixels in which luminance changes have occurred only in a limited area (Sa to Sf) obtained by limiting the predetermined imaging area to a narrow area. , the event data of pixels in the area excluding the limited area in the predetermined imaging area is not output regardless of whether or not there is a change in luminance. an output limiting unit (11) that can limit the output of event data generated by the imaging element; Equipped with the output limiting unit sequentially switches the limited area so as to cover the predetermined imaging area; The image generation unit is characterized in that, when the output restriction unit restricts the output of event data, it generates an image of the specified imaging area by aggregating the event data that is output each time the limited area is switched. The symbols in parentheses above indicate the correspondence with the specific means described in the embodiments to be described later. [Effects of the Invention]

[0007] In the invention of claim 1, the output limiting unit limits the predetermined imaging area to a narrow area and outputs event data of pixels where a luminance change has occurred only in the limited area. , pixel event data is not output regardless of whether there is a change in luminance in an area other than the limited area of ​​the predetermined imaging area. By doing so, it is possible to limit the output of event data generated by the image sensor, and the limited area is sequentially switched so as to cover the predetermined imaging area. When the output limiting unit limits the output of event data, the image generating unit generates a captured image of the predetermined imaging area by aggregating the event data output each time the limited area is switched.

[0008] As a result, even if a QR code crosses the specified imaging area at close range, for example, event data is output for the part of the QR code included in the limited area and not for the remaining part of the QR code that is not included in the limited area, thereby preventing saturation of the event data output. Furthermore, by aggregating the event data output each time the limited area is switched, a captured image of the specified imaging area can be generated, thereby preventing saturation of the event data output and realizing an imaging device that has the advantages of an event camera, such as high temporal resolution, high dynamic range, low data consumption, and low power consumption.

[0009] In the invention of claim 2, the output limiting unit temporarily disables the functions of pixels that do not correspond to the limited area, thereby outputting event data of pixels that have experienced luminance changes only in the limited area, thereby enabling accurate and fast switching of the limited area. In particular, the range of each limited area within a predetermined imaging area can be easily adjusted, such as by dividing it into four parts or by spacing it out.

[0010] In the invention of claim 3, the output limiting unit has a shielding unit whose surface facing the image sensor is formed in a single color and which covers pixels different from the pixels corresponding to the limited area, and by moving the shielding unit in response to sequential switching of the limited area, event data of pixels whose luminance has changed is output only in the limited area. In this way, the limited area can be switched in response to movement such as rotation or sliding of the shielding unit.

[0011] In the invention of claim 4, the output limiting unit has an infrared transmission unit that covers the light receiving side of the image sensor, and an infrared irradiation unit that irradiates infrared rays from the image sensor side onto a part of the surface of the infrared transmission unit that corresponds to a predetermined image capturing area, and by moving the irradiation range of infrared rays irradiated from the infrared irradiation unit in accordance with the sequential switching of the limited area, event data of pixels that have experienced a luminance change in only the limited area is output. In this way, the limited area can be switched in accordance with the movement of the infrared irradiation range.

[0012] In the invention of claim 5, when the output limiting unit limits the output of event data, the image generating unit generates a captured image of the predetermined imaging area by aggregating the event data output each time the limited area is switched, taking into account the moving speed of the imaging target estimated by the estimation unit. As a result, even if the imaging target, such as a QR code, is moving at high speed and the position of the imaging target in the predetermined imaging area shifts each time the limited area is switched, the event data can be aggregated to correct the positional shift, and a captured image of the predetermined imaging area can be generated with high accuracy.

[0013] In the invention of claim 6, the output limiting unit limits the output of event data when the amount of event data output from the imaging element per unit time exceeds a predetermined amount, so the output of event data is not limited even when only a small amount of event data is output, making it possible to suppress output restrictions on unnecessary event data. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a block diagram showing a schematic configuration of an imaging device according to a first embodiment. [Figure 2] FIG. 2 is an explanatory diagram illustrating the relationship between an imaging area and each limited area in the first embodiment. [Figure 3] FIG. 10 is an explanatory diagram illustrating a state in which a QR code enters an imaging area. [Figure 4] Figure 4(A) is an explanatory diagram illustrating a portion of the QR code in Figure 3 that is the target of event output in the limited area shown in Figure 2(A), Figure 4(B) is an explanatory diagram illustrating a portion of the QR code in Figure 3 that is the target of event output in the limited area shown in Figure 2(B), Figure 4(C) is an explanatory diagram illustrating a portion of the QR code in Figure 3 that is the target of event output in the limited area shown in Figure 2(C), and Figure 4(D) is an explanatory diagram illustrating a portion of the QR code in Figure 3 that is the target of event output in the limited area shown in Figure 2(D). [Figure 5] 10 is an explanatory diagram illustrating a method of switching a limited area relative to an imaging area in the second embodiment. FIG. [Figure 6] FIG. 11 is an explanatory diagram illustrating a method of switching a limited area relative to an imaging area in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] [First embodiment] Hereinafter, a first embodiment of an imaging device according to the present invention will be described with reference to the drawings. The imaging device 10 according to this embodiment functions as a so-called event camera. The imaging device 10 outputs event data containing two-dimensional point data for identifying the position of a pixel that has experienced a brightness change, as well as time and the polarity of the brightness change, and generates image data of the imaging target by plotting the two-dimensional point data of multiple event data output within a certain period of time as points on a predetermined plane.

[0016] As shown in FIG. 1, the imaging device 10 includes a control unit 11 consisting of a CPU or the like and a storage unit 12 consisting of a semiconductor memory or the like, as well as an imaging unit 13 and a display unit 14 controlled by the control unit 11, an operation unit 15 that outputs an operation signal to the control unit 11 in response to an input operation, and a communication unit 16 for communicating with external devices or the like.

[0017] The imaging unit 13 is configured to include an imaging element 13a, a light-receiving lens, etc., and the imaging element 13a is configured to output event data including two-dimensional point data that identifies the position of a pixel corresponding to a pixel that has undergone a luminance change when light is received through the light-receiving lens to the control unit 11. That is, the imaging element 13a functions to output event data (two-dimensional point data, time, polarity of luminance change) corresponding to a pixel that has undergone a luminance change to the control unit 11, and not to output data for pixels that have not undergone a luminance change.

[0018] In particular, in this embodiment, the imaging element 13a of the imaging unit 13 is configured to be switchable on a pixel-by-pixel basis in response to output restriction processing by the control unit 11 between pixels that output event data when there is a change in brightness (hereinafter also referred to as output-enabled pixels) and pixels that do not output event data regardless of whether there is a change in brightness by temporarily disabling the pixel function (hereinafter also referred to as output-restricted pixels).

[0019] Therefore, for example, in a predetermined imaging area S using all pixels of the image sensor 13a, by setting only pixels corresponding to the limited area Sa shown in Fig. 2(A) as output-enabled pixels and the other pixels as output-restricted pixels, the amount of output event data can be reduced compared to when all pixels are output-enabled pixels. Also, the amount of output event data can be similarly reduced when only pixels corresponding to the limited area Sb shown in Fig. 2(B) are output-enabled pixels and the other pixels are output-restricted pixels, when only pixels corresponding to the limited area Sc shown in Fig. 2(C) are output-enabled pixels and the other pixels are output-restricted pixels, or when only pixels corresponding to the limited area Sd shown in Fig. 2(D) are output-enabled pixels and the other pixels are output-restricted pixels.

[0020] In this embodiment, the limited areas Sa to Sd are set so that the combined area corresponds to the imaging area S, and under predetermined imaging conditions, the limited areas Sa to Sd are sequentially switched to cover the imaging area S. Under such imaging conditions, the amount of event data output is reduced as described above, and the event data output each time the limited areas Sa to Sd are switched is aggregated to generate a captured image of the imaging area S. Note that the control unit 11 that performs the output restriction process can be an example of an "output restriction unit" that can restrict the output of event data generated by the image sensor 13a by outputting event data of pixels that have experienced a luminance change only for limited areas (Sa to Sd) that limit the imaging area S to a small area. Furthermore, the limited areas Sa to Sd may be set so that the combined area of ​​the limited areas Sa to Sd includes the imaging area S.

[0021] For example, under the above-described predetermined imaging conditions, when a QR code C is held over the imaging device 10 and the QR code C enters the imaging area S as shown in Fig. 3, first, when switching to limited area Sa, event data for generating the image shown in Fig. 4(A) is output from the imaging element 13a to the control unit 11. Next, when switching to limited area Sb, event data for generating the image shown in Fig. 4(B) is output from the imaging element 13a to the control unit 11, and then when switching to limited area Sc, event data for generating the image shown in Fig. 4(C) is output from the imaging element 13a to the control unit 11. Finally, when switching to limited area Sd, event data for generating the image shown in Fig. 4(D) is output from the imaging element 13a to the control unit 11.

[0022] The control unit 11 then performs an image generation process that aggregates the event data output each time the limited area is switched, thereby generating a captured image of the predetermined imaging area S. As a result, the event data obtained when switching to the limited area Sa, the event data obtained when switching to the limited area Sb, the event data obtained when switching to the limited area Sc, and the event data obtained when switching to the limited area Sd are aggregated, and an captured image corresponding to the QR code C is generated, as shown in Fig. 3. The control unit 11 that performs the image generation process can correspond to an example of an "image generation unit" that generates a captured image of the predetermined imaging area S by aggregating the event data output each time the limited area is switched.

[0023] As described above, in the imaging device 10 according to this embodiment, the output restriction process by the control unit 11 can restrict the output of event data generated by the imaging element 13a by outputting event data of pixels (output-enabled pixels) that have experienced a change in luminance only for limited areas (Sa to Sb) that limit the predetermined imaging area S to a small area, and sequentially switches the limited area so as to cover the predetermined imaging area S. When the output restriction process restricts the output of event data, the image generation process by the control unit 11 generates a captured image of the predetermined imaging area S by aggregating the event data that is output each time the limited area is switched.

[0024] As a result, even if, for example, a QR code C enters the specified imaging area S by crossing it at close range, event data is output for the part of the QR code C included in the limited area, and event data is not output for the remainder of the QR code C that is not included in the limited area, thereby preventing saturation of the event data output. Furthermore, even if, for example, the surrounding environment enters the specified imaging area S, event data is output for the part of the surrounding environment included in the limited area, and event data is not output for the remainder of the surrounding environment that is not included in the limited area, thereby preventing saturation of the event data output. Furthermore, by aggregating the event data output each time the limited area is switched, a captured image of the specified imaging area S can be generated, thereby preventing saturation of the event data output and realizing an imaging device that has the advantages of an event camera, such as high temporal resolution, high dynamic range, low data consumption, and low power consumption.

[0025] In particular, the output restriction process by the control unit 11 temporarily disables the functions of pixels that do not correspond to the limited area (output-restricted pixels), thereby outputting event data of pixels (output-enabled pixels) that have experienced a luminance change only in the limited area, thereby enabling accurate and fast switching of the limited area. Furthermore, the range of each limited area within the specified imaging area S can be easily adjusted, such as by dividing it into four parts or by spacing it out.

[0026] [Second embodiment] Next, an imaging device according to a second embodiment of the present invention will be described with reference to the drawings. The second embodiment differs from the first embodiment mainly in that the limited area is physically switched. Therefore, the same components as those in the first embodiment are denoted by the same reference numerals and their description will be omitted.

[0027] In the imaging device 10 according to this embodiment, as illustrated in Fig. 5, a shielding portion 20 is rotatably disposed on the light receiving side of the imaging unit 13 (imaging element 13a). The surface of this shielding portion 20 facing the imaging element is a single color (for example, gray, which is an intermediate color between white and black), has a sector-shaped notch 21 formed therein, and is disposed so that the center of rotation coincides with the imaging center of the imaging element 13a. When viewed from the imaging target side, the shielding portion 20 is formed so as to cover the imaging element 13a except for the notch 21 when rotated, and the central angle of the notch 21 is set to an angle (for example, 90°) according to the switching frequency.

[0028] The output restriction process performed by control unit 11 drives and controls a motor or the like that rotates shielding unit 20 to switch between the range of image sensor 13a covered by shielding unit 20 and the range of image sensor 13a exposed through notch 21. The pixels of image sensor 13a covered by shielding unit 20 become the output-restricted pixels because they receive light from the single-color surface of shielding unit 20 and do not experience a change in brightness, while the pixels of image sensor 13a exposed through notch 21 become the output-enabled pixels.

[0029] Therefore, the limited area can be switched by rotating the shielding unit 20. Then, the control unit 11 performs an image generation process in which the limited area is considered to have been switched each time the shielding unit 20 is rotated by the central angle of the cutout 21, and the control unit 11 aggregates the event data, thereby generating a captured image of a predetermined imaging area S. Note that in FIG. 5, the limited area switched by rotating the shielding unit 20 is indicated by the symbol Se. The control unit 11 and the shielding unit 20 that perform the output limiting process can correspond to an example of an "output limiting unit."

[0030] Note that the limited area may not be physically switched by rotating the shielding portion 20 in which the notch 21 is formed, but may also be physically switched by, for example, sliding the shielding portion in a predetermined direction (for example, up and down).

[0031] [Third embodiment] Next, an imaging device according to a third embodiment of the present invention will be described with reference to the drawings. The third embodiment differs from the first embodiment mainly in that the limited area is switched using infrared rays. Therefore, the same components as those in the first embodiment are denoted by the same reference numerals and their description will be omitted.

[0032] 6, the imaging device 10 according to this embodiment is provided with an infrared transmitting section 31 that covers the light receiving side of the imaging section 13 (imaging element 13a), and an infrared irradiating section 32 that irradiates infrared rays from the imaging element side onto a part of the surface of the infrared transmitting section 31 that corresponds to a predetermined imaging area. The infrared transmitting section 31 is, for example, an infrared transmitting filter, and functions to transmit infrared rays while not transmitting visible light.

[0033] Therefore, some pixels of the imaging element 13a that receive infrared light reflected from an imaging target or the like and transmitted through the infrared transmitting section 31 become output-enabled pixels. On the other hand, since light reflected from an imaging target or the like that is not irradiated with infrared light cannot pass through the infrared transmitting section 31, the remaining pixels of the imaging element 13a whose light receiving range is an area that is not irradiated with infrared light become output-limited pixels.

[0034] The infrared irradiator 32 is configured so that the irradiation range (irradiation direction) of the infrared rays can be controlled by the control unit 11. Therefore, in the output limiting process performed by the control unit 11, the limited area can be switched by moving the irradiation range of infrared rays by the infrared irradiator 32. Then, every time the irradiation range of infrared rays by the infrared irradiator 32 changes, the control unit 11 performs an image generation process in which the limited area is considered to have been switched and event data is aggregated to generate a captured image of a predetermined imaging area S. Note that in FIG. 6, the limited area switched by the irradiation of infrared rays from the infrared irradiator 32 is illustrated with the symbol Sf. The control unit 11, the infrared transmission unit 31, and the infrared irradiator 32 that perform the output limiting process can correspond to an example of an "output limiting unit."

[0035] In addition, the infrared irradiation unit 32 is not limited to being configured to change the irradiation range of the infrared transmission unit 31 by changing the irradiation direction, but may also be configured, for example, to have infrared lights corresponding to each limited area and to change the irradiation range of the infrared transmission unit 31 by switching the infrared lights that are turned on.

[0036] [Fourth embodiment] Next, an imaging device according to a fourth embodiment of the present invention will be described. The fourth embodiment is different from the first embodiment in that event data is aggregated taking into account the estimated moving speed of the imaging target. Therefore, the same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof will be omitted.

[0037] If the relative movement speed of the object to be imaged relative to the imaging device 10 is relatively fast or the speed at which the limited area is switched is relatively slow, the object to be imaged may move during the switching of the limited area, causing a positional shift, and an accurate image of the object to be imaged may not be generated.

[0038] For this reason, in this embodiment, the moving speed of the imaging target is estimated, and the event data output each time the limited area is switched is aggregated taking into account the estimated moving speed of the imaging target.

[0039] Specifically, first, a moving speed estimation process is performed by the control unit 11, where multiple images of the imaging target are generated using event data in the imaging area or one limited area, and the amount of positional deviation of the imaging target between each image is detected, thereby estimating the moving speed of the imaging target. Note that the control unit 11 that performs the moving speed estimation process can correspond to an example of an "estimation unit" that estimates the moving speed of the imaging target.

[0040] Next, in the output restriction process performed by the control unit 11, the limited area is switched as described above, and event data is acquired for each limited area. Then, in the image generation process performed by the control unit 11, the position of the acquired event data is corrected using the moving direction of the imaging target calculated from the moving speed estimated as described above and the moving distance at the limited area switching interval, and the position-corrected event data is aggregated to generate an image of the imaging target.

[0041] For example, when four limited areas Sa to Sd are employed as in the first embodiment, first, the event data output when the limited area Sa is switched is position-corrected using the movement distance, etc., and the event data output when the limited area Sb is switched are aggregated. Next, the event data aggregated for the two limited areas as described above is position-corrected using the movement distance, etc., and the event data output when the limited area Sc is switched are further aggregated. Then, the event data aggregated for the three limited areas as described above is position-corrected using the movement distance, etc., and the event data output when the limited area Sd is switched are further aggregated. In this way, a captured image of the predetermined imaging area S is generated based on the event data aggregated so that positional deviations are corrected.

[0042] As described above, in the imaging device 10 according to this embodiment, when the output of event data is restricted by the output restriction process, the image generation process performed by the control unit 11 generates an image of a specified imaging area S by aggregating the event data output each time the limited area is switched, taking into account the estimated moving speed of the imaging subject.

[0043] This allows event data to be aggregated to correct the positional shift even when the position of the imaging target in the specified imaging area S shifts each time the limited area is switched due to the imaging target such as a QR code moving at high speed, and enables the imaging image of the specified imaging area S to be generated with high accuracy.

[0044] The characteristic configuration of this embodiment, in which event data is aggregated in consideration of the estimated moving speed of the imaging target, can also be applied to other embodiments.

[0045] The present invention is not limited to the above-described embodiments, and may be embodied as follows, for example. (1) In the output limiting process performed by the control unit 11, for example, the output of event data may be limited when the amount of event data output from the image sensor 13a per unit time is equal to or greater than a predetermined amount as the predetermined imaging condition, and may not be limited when the amount of event data output is less than the predetermined amount. This prevents the output of event data from being limited even when only a small amount of event data is output, thereby preventing the output of unnecessary event data from being limited. Furthermore, the output of event data may be limited when, for example, a predetermined operation is performed on the operation unit 15 or a predetermined instruction is received from the outside via the communication unit 16.

[0046] (2) The image generation process performed by the control unit 11 is not limited to generating an image of a specified imaging area S by aggregating all event data output each time the limited area is switched, but may also generate, for example, an image of a portion of the imaging target or an image of a part of the specified imaging area S. [Explanation of symbols]

[0047] 10...imaging device 11...Control unit (output limiting unit, image generating unit, estimation unit) 13...imaging unit 13a...imaging element S...Image capture area Sa~Sf...Limited area

Claims

1. an image sensor that generates event data including two-dimensional point data that identifies the position of a pixel that has experienced a luminance change; an image generation unit that generates a captured image of a predetermined imaging area using the event data output from the imaging element; an output limiting unit that limits the output of event data generated by the image sensor by outputting event data of pixels that have undergone a luminance change only in a limited area that limits the predetermined image capture area to a narrow area, and not outputting event data of pixels that have undergone a luminance change in an area other than the limited area of ​​the predetermined image capture area, regardless of whether or not there is a luminance change; Equipped with the output limiting unit sequentially switches the limited area so as to cover the predetermined imaging area; The imaging device is characterized in that, when the output restriction unit restricts the output of event data, the image generation unit generates an image of the specified imaging area by aggregating the event data that is output each time the limited area is switched.

2. The imaging device according to claim 1, characterized in that the output limiting unit temporarily disables the functions of pixels that do not correspond to the limited area, thereby outputting event data of pixels that have experienced a luminance change only in the limited area.

3. The imaging device described in claim 1, characterized in that the output restriction unit has a shielding unit whose surface on the imaging element side is formed in a single color and covers pixels different from the pixels corresponding to the limited area, and by moving the shielding unit in accordance with the sequential switching of the limited area, event data of pixels that have experienced a change in brightness only in the limited area is output.

4. 2. The imaging device according to claim 1, wherein the output limiting unit includes an infrared-transmitting unit that covers the light-receiving side of the imaging element, and an infrared irradiating unit that irradiates infrared rays from the imaging element side onto a portion of a surface of the infrared-transmitting unit that corresponds to the specified imaging area, and by moving an irradiation range of infrared rays irradiated from the infrared irradiating unit in accordance with sequential switching of the limited area, event data of pixels that have experienced a change in luminance in only the limited area is output.

5. an estimation unit that estimates a moving speed of an imaging target; The imaging device according to any one of claims 1 to 4, characterized in that, when the output limiting unit limits the output of event data, the image generating unit generates an image of the specified imaging area by aggregating the event data output each time the limited area is switched, taking into account the movement speed of the imaging target estimated by the estimation unit.

6. The imaging device according to any one of claims 1 to 5, characterized in that the output limiting unit limits the output of the event data when the amount of event data output from the imaging element per unit time exceeds a predetermined amount.

Images