Imaging device

Abstract

The present invention provides an imaging device capable of suppressing the overall number of spikes by performing different processing on different pixels. [Solution] An imaging device that outputs image data derived by processing the pixel values ​​of multiple pixels using spatial filters 12, 12a, 13, 13a and temporal filters 14, 14a, wherein the multiple pixels are divided into multiple pixel groups, and different spatial filters 12, 12a, 13, 13a and different temporal filters 14, 14a are used to process the pixel values ​​of the pixels depending on the pixel group to which the pixel belongs.

Description

Technical Field

[0001] The present invention relates to an imaging device that performs predetermined processing on the pixel values of each pixel.

Background Art

[0002] Recently, an event-driven camera (event-based camera) designed based on the visual system of living organisms has attracted attention as being useful in the fields of computer vision and robotics. An event-driven camera detects only pixels whose luminance has changed by a predetermined amount or more and outputs the data asynchronously, which is different from a conventional camera that detects the luminance of each of all pixels and outputs it as a frame image.

[0003] Here, it is important that the visual system, like other systems, can be realized with limited hardware resources. For example, in the field of robotics, there are restrictions such as the placement of sensors, computational power, and communication wiring for connecting each component, and within these restrictions, the corresponding visual system must be designed. In this regard, when a conventional event-driven camera processes a specific visual event in real time, the number of spikes (events) increases, and a large amount of resources are allocated to the processing and transmission of those spikes.

[0004] The visual system of living organisms has central vision and peripheral vision that perform different processes. Specifically, central vision focuses on the central region within the entire visual field and processes visual information with high resolution, while peripheral vision targets the peripheral region around the central region and performs processing for quickly detecting the movement of objects and changes in light in that region. That is, in each region, a large amount of resources are used for necessary functions, and the resources allocated to other functions are restricted, thereby suppressing the overall resources while realizing the required visual functions.

Prior Art Documents

Patent Documents

[0005] [Patent Document 1] Japanese Patent Publication No. 2024-016542 [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] However, while conventional imaging devices, including event-driven cameras, exist that focus on reproducing the central visual acuity of biological visual systems or that only detect changes in brightness, there were no devices that performed different processing on different pixels using a single imaging method. This invention has been made in view of the above circumstances, and aims to provide an imaging device that can suppress the total number of spikes by performing different processing on different pixels. [Means for solving the problem]

[0007] An imaging device according to the present invention, which is in line with the above objective, is an imaging device that outputs image data derived by processing the pixel value of each of a plurality of pixels using a spatial filter and a temporal filter, wherein the plurality of pixels are divided into a plurality of pixel groups, and different spatial filters and different temporal filters are used to process the pixel value of each pixel depending on the pixel group to which the pixel belongs. [Effects of the Invention]

[0008] The imaging device according to the present invention divides multiple pixels into multiple pixel groups, and uses different spatial filters and different temporal filters to process the pixel values ​​of pixels depending on the pixel group to which the pixel belongs. This allows different processing to be performed on different pixels, thereby suppressing the overall number of spikes. [Brief explanation of the drawing]

[0009] [Figure 1] This is an explanatory diagram of an imaging device according to one embodiment of the present invention. [Figure 2] This is an explanatory diagram of pixel groups. [Figure 3] This is a diagram illustrating the individual processing circuits. [Modes for carrying out the invention]

[0010] Next, with reference to the attached drawings, embodiments of the present invention will be described to facilitate understanding of the invention. As shown in Figures 1, 2, and 3, the imaging device 10 according to one embodiment of the present invention is a device that outputs image data derived by processing the pixel values ​​of each of a plurality of pixels 11 using spatial filters 12, 12a, 13, 13a and temporal filters 14, 14a.

[0011] In this embodiment, the imaging device 10 includes an optical lens 15, an image sensor 16, and a signal processing means 17, as shown in Figure 1. The image sensor 16 has a plurality of light-receiving elements that each receive light that has passed through the optical lens 15, and each light-receiving element converts the received light into an electrical signal (i.e., a pixel value). As shown in Figure 2, each light-receiving element corresponds to each pixel 11, and in this embodiment, each pixel 11 (light-receiving element) is arranged so that the plurality of pixels 11 together form a rectangle.

[0012] The pixel values ​​derived at each pixel 11 are converted into digital data by an A / D conversion circuit and then provided to the signal processing means 17. The signal processing means 17 can be configured, for example, by an FPGA (field programmable gate array). The signal processing means 17 may also receive analog pixel values ​​and convert them into digital data internally. Furthermore, it goes without saying that the signal processing means 17 can be configured using something other than an FPGA.

[0013] The signal processing means 17 comprises multiple individual processing circuits 20, each of which is configured to include spatial filters 12, 12a, 13, 13a, time filters 14, 14a, an ON path section 18, and a spike conversion section 19, as shown in Figure 3. There is a one-to-one relationship between the individual processing circuit 20 and the pixels 11, and the digitized pixel value corresponding to the nth pixel 11 is given to the nth individual processing circuit 20, where predetermined processing is performed and an output is produced.

[0014] Spatial filters 12, 12a, 13, and 13a are low-pass filters that perform spatial filtering on a given value, smoothing the pixel value of the pixel 11 based on the pixel values ​​of the pixel 11 in question and the multiple pixels 11 arranged around it in spatial coordinates. Temporal filters 14 and 14a are filters that perform temporal filtering on a given value. Specifically, temporal filter 14 is a low-pass filter that smooths the pixel value of the pixel 11 in question based on its temporally changing pixel value, while temporal filter 14a is a band-pass filter that enhances contrast.

[0015] The ON pathway 18 corresponds to the ON pathway in the visual system of living organisms. It goes without saying that an OFF pathway, corresponding to the OFF pathway in the visual system of living organisms, may be provided instead of, or in addition to, the ON pathway 18. The spike conversion unit 19 converts the value output from the ON path unit 18 (or the OFF path unit if an OFF path unit is provided) into event pixel (spike) data. The event pixel data is output from the individual processing circuit 20.

[0016] Furthermore, as shown in Figure 3, the individual processing circuit 20 includes a switching unit 22 that switches whether to use spatial filters 12 or 12a for processing a given pixel value, a switching unit 23 that switches whether to use spatial filters 13 or 13a for the same processing, and a switching unit 24 that switches whether to use temporal filters 14 or 14a for the same processing.

[0017] If the individual processing circuit 20 is using the spatial filters 12, 13 and the time filter 14, the following processes are sequentially performed on the pixel values ​​given to the individual processing circuit 20.

[0018] 1) Filtering is performed by the spatial filter 12. 2) The value output from the spatial filter 12 is sent to two paths at the branch point, and the value sent to one path is further filtered by the spatial filter 13 (the value sent to the other path is not filtered by the spatial filter 13). 3) A subtraction process is performed to subtract the value derived by only the process of the spatial filter 12 from the value derived by the processes of both the spatial filters 12 and 13.

[0019] 4) The value calculated by the subtraction process is filtered by the temporal filter 14. 5) The value derived by the filtering process of the temporal filter 14 passes through the ON path section 18 and the spike conversion section 19 in sequence and is output from the individual processing circuit 20.

[0020] The value corresponding to each pixel 11 output from the individual processing circuit 20 is output from the signal processing means 17 together with the information on which pixel 11 the value corresponds to, and is given to the information processing terminal 25 (see FIG. 1) connected to the signal processing means 17. The information processing terminal 25 generates an image by performing a predetermined calculation based on the value given from the signal processing means 17.

[0021] Here, in the present embodiment, as shown in FIG. 2, a plurality of pixels 11 are divided into a plurality (two in the present embodiment) of pixel groups 27 and 28. That is, the plurality of pixels 11 are divided so that each belongs to one of the plurality of pixel groups 27 and 28. The individual processing circuit 20 corresponding to each pixel 11 belonging to the pixel group 27 is set to a state in which the spatial filters 12 and 13 and the temporal filter 14 are used. On the other hand, the individual processing circuit 20 corresponding to each pixel 11 belonging to the pixel group 28 is set to a state in which the spatial filters 12a and 13a and the temporal filter 14a are used.

[0022] Therefore, in this embodiment, different spatial filters 12, 12a, different spatial filters 13, 13a, and different temporal filters 14, 14a are used to process the pixel value of pixel 11 depending on the pixel groups 27 and 28 to which pixel 11 belongs. In this embodiment, spatial filters 12 and 13 are filters that perform processing to increase spatial sensitivity compared to spatial filters 12a and 13a, respectively, while temporal filter 14 is a filter that performs processing to decrease temporal sensitivity compared to temporal filter 14a.

[0023] Therefore, the signal processing means 17 performs processing on the pixel values ​​of each pixel 11 belonging to pixel group 27, comparing them with the pixel values ​​of each pixel 11 belonging to pixel group 28, to increase spatial sensitivity and decrease temporal sensitivity. In other words, in a single image corresponding to multiple pixels 11, by providing regions where the spatial resolution is sparse and the temporal resolution is dense, and regions where the spatial resolution is dense and the temporal resolution is sparse, it is possible to derive the necessary information for each region, suppress the overall number of spikes (number of events), and reduce the computational load of post-processing (processing by the information processing terminal 25 in this embodiment) and the amount of communication when transferring data to post-processing.

[0024] Furthermore, this embodiment includes a switching unit 22 for switching the spatial filters 12 and 12a used for each of the multiple pixels 11, a switching unit 23 for switching the spatial filters 13 and 13a, and a switching unit 24 for switching the temporal filters 14 and 14a. Therefore, by switching each of the switching units 22, 23, and 24, it is possible to group the pixels 11 into groups, and for example, it is possible to create three or more pixel groups.

[0025] Furthermore, the switching by each switching unit 22, 23, and 24 can be performed statically (pre-set) or dynamically (even while the signal processing means 17 is processing data).

[0026] Although embodiments of the present invention have been described above, the present invention is not limited to the above-described forms, and any changes to the conditions, etc., that do not depart from the gist of the invention are all within the scope of application of the present invention. For example, it is not necessary to provide a switching unit to switch between spatial filters or temporal filters. [Explanation of Symbols]

[0027] 10: Imaging device, 11: Pixel, 12, 12a, 13, 13a: Spatial filter, 14, 14a: Time filter, 15: Optical lens, 16: Image sensor, 17: Signal processing means, 18: ON path unit, 19: Spike conversion unit, 20: Individual processing circuit, 22, 23, 24: Switching unit, 25: Information processing terminal, 27, 28: Pixel group

Claims

1. An imaging device that outputs image data derived by processing the pixel values ​​of multiple pixels using spatial filters and temporal filters, The imaging apparatus is characterized in that the plurality of pixels are divided into a plurality of pixel groups, and different spatial filters and different temporal filters are used to process the pixel values ​​of the pixels depending on the pixel group to which the pixels belong.

2. The imaging apparatus according to claim 1, further comprising a switching unit for switching between the spatial filter and the temporal filter used for each of the plurality of pixels.

Images