Imaging apparatus

Abstract

To provide an imaging apparatus capable of efficiently transmitting image data from a frame-based sensor to an image processing unit.SOLUTION: When transmitting image data from a frame-based sensor to an image processing unit on the basis of event information output by an event-based sensor, pixel signals with luminance changes are retained in memory and transmitted collectively as one frame of image data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an imaging device. [Background technology]

[0002] In recent years, an event vision sensor has been developed, as described in Patent Document 1. In an event vision sensor, a detection pixel compares the amount of change in the amount of incident light (change in brightness) with a predetermined threshold to detect whether an event has occurred in the subject, and the sensor is controlled to read out an image signal from a normal pixel that detects the occurrence of an event. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2020-96347 Summary of the Invention [Problem to be solved by the invention]

[0004] The event vision sensor described above can reduce power consumption by controlling the reading of image signals only from normal pixels in areas where brightness changes are detected, and not from normal pixels in areas where brightness changes are not detected. However, if brightness changes are detected in multiple locations within a frame, the image signals from the event vision sensor must be read multiple times, which complicates read control and increases power consumption. [Means for solving the problem]

[0005] In order to solve the above problem, the imaging device of the present invention is an imaging device comprising an event-based sensor that outputs event information indicating the coordinates and time information of a pixel where a luminance change has been detected, a frame-based sensor that outputs image data on a frame-by-frame basis, and an image processing unit, wherein the frame-based sensor stores pixel signals where a luminance change has occurred based on the event information in a memory and then transmits the pixel signals as image data for one frame to the image processing unit, and the image processing unit receives the image data for one frame transmitted from the frame-based sensor during a predetermined period that includes the pixel signals where the luminance change has occurred based on the event information, and performs predetermined signal processing on the received image data. [Effects of the Invention]

[0006] According to the present invention, even if areas in which brightness changes are detected are scattered across multiple locations within a single frame, image data can be efficiently transmitted and received from a frame-based sensor to an image processing unit. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram illustrating a configuration of an imaging apparatus according to an embodiment. [Figure 2] 4 is a timing chart illustrating processing of the imaging device in the embodiment. [Figure 3] 10 is a flowchart illustrating processing of a frame-based sensor in an embodiment. [Figure 4] 10 is a flowchart showing the processing of an image processing unit in the embodiment. [Figure 5] 10A and 10B are diagrams illustrating image synthesis by a signal processing unit in the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] An embodiment of the present invention will now be described below. Fig. 1 is a block diagram showing the configuration of an imaging device according to the embodiment.

[0009] 1, the imaging device 101 includes an event-based sensor 102, a frame-based sensor 103, and an image processing unit 104. The event-based sensor 102 detects a change in luminance of each pixel, and outputs event information indicating the "coordinates" and "time information" of the pixel where the luminance change was detected to the frame-based sensor 103 and the image processing unit 104. The frame-based sensor 103 outputs image data on a frame-by-frame basis.

[0010] The event-based sensor 102 includes multiple pixels, each consisting of a light-receiving unit and a brightness change detection unit. In the light-receiving unit of the event-based sensor, a photoelectric conversion unit such as a photodiode generates an electric charge in response to incident light and outputs a voltage corresponding to the generated electric charge. The brightness change detection unit compares the output voltage of the light-receiving unit with a reference voltage, and outputs event information when the difference between the output voltage and the reference voltage exceeds a threshold.

[0011] The frame-based sensor 103 includes a pixel unit 105, a transmission control unit 106, and a data transmission unit 107, and is an imaging element such as a CCD or CMOS. The pixel unit 105 includes a photoelectric conversion unit such as a photodiode, an amplifier that amplifies a pixel signal output from the photoelectric conversion unit, an A / D converter that converts the pixel signal into a digital signal, and the like.

[0012] Transmission control unit 106 receives event information output from event-based sensor 102, and outputs a transmission control signal indicating whether or not to output image data from frame-based sensor 102 based on the received event information to data transmission unit 107. Data transmission unit 107 outputs pixel signals from pixel unit 105 to image processing unit 104 based on the transmission control signal output from transmission control unit 106. When pixel signals from pixel unit 105 are output to image processing unit 104, they are held in a memory for at least one frame and output together as image data.

[0013] Here, the transmission control signal includes not only information indicating whether or not to output image data from frame-based sensor 102, but also information about the amount of data to be output as image data. For example, if the period for outputting image data is one horizontal period, the information indicates that the amount of data to be output as image data is 10 horizontal periods in total. Based on this data amount, data transmitter 107 sets the timing for starting to transmit image data within one frame, and outputs the data collectively at the beginning or end of the frame.

[0014] Image processing unit 104 includes a reception control unit 108, a data receiving unit 109, and a signal processing unit 110. Reception control unit 108 receives event information output from event-based sensor 102, and outputs a reception control signal to data receiving unit 109 indicating whether or not to receive image data from frame-based sensor 102 based on the received event information. Data receiving unit 109 receives image data from frame-based sensor 103 based on the reception control signal output from reception control unit 108.

[0015] Here, the reception control signal includes not only information indicating whether or not to receive image data from the frame-based sensor 102, but also information regarding the amount of image data to be received. For example, if the period for receiving image data is one horizontal period, the information indicates that the amount of image data to be received is a total of 10 horizontal periods. Based on this data amount, the data receiving unit 109 sets the timing for starting to receive image data within one frame, and receives the image data all at once at the beginning or end of one frame.

[0016] The signal processing unit 110 performs predetermined signal processing on the image data received by the data receiving unit 109. For example, as will be described later, the signal processing unit 110 combines the image data of the portion where the luminance has changed received by the data receiving unit 109 with the image data of the previous frame. Alternatively, the signal processing unit 110 may perform processing to display only the portion where the luminance has changed.

[0017] FIG. 2 is a timing chart for explaining the processing of the imaging device in this embodiment.

[0018] In FIG. 2, periods 201 to 205 each represent an interval of one vertical period, that is, one frame period.

[0019] The top shows output 206 of pixel unit 105 of frame-based sensor 103. Pixel signals 207, 208, and 211 are pixel signals with luminance changes, and pixel signals 209, 210, and 212 are pixel signals with no luminance changes.

[0020] The middle shows output 213 of data transmission unit 107 of frame-based sensor 103. Image data 214 to 219 is image data for one frame that is output collectively after pixel signals output from pixel unit 105 are stored in the memory of data transmission unit 107.

[0021] Since all pixel signals in one frame of pixel signal 207 output from pixel unit 105 have had a luminance change, the entire frame is stored in memory before being output as image data 214. Furthermore, image data 215 corresponding to pixel signal 209, which had no luminance change, is not output, and image data 216 corresponding to pixel signal 208, which had a luminance change, is stored in memory of data transmission unit 107 before being output.

[0022] Image data 217 corresponding to pixel signals 210 that did not change in luminance within one frame is not output. If pixel signals 211 that did change in luminance are scattered among pixel signals 212 that did not change in luminance, image data 219 corresponding to pixel signals 211 that did change in luminance are held in the memory of data transmission unit 107 and output collectively for one frame. Also, image data 218 corresponding to pixel signals 212 that did not change in luminance is not output.

[0023] The bottom shows transmission control 220 of data transmission unit 107 of frame-based sensor 103. Transmission controls 221, 223, and 226 are controls for transmitting image data from data transmission unit 107. Transmission controls 222, 224, and 225 are controls for not transmitting image data from data transmission unit 107.

[0024] FIG. 3 is a flowchart showing the processing of the frame-based sensor 103 in this embodiment.

[0025] In S301, the pixel unit 105 outputs a pixel signal.

[0026] In S302, event information is received from the event-based sensor 102.

[0027] In S303, it is determined whether a luminance change has occurred within one frame based on the received event information. If it is determined that a luminance change has not occurred within one frame, the process ends, and if it is determined that a luminance change has occurred within one frame, the process proceeds to S304.

[0028] In S304, it is determined whether or not there is a luminance change in the pixel signal output from each pixel based on the received event information. If it is determined that there is no luminance change in the pixel signal output from each pixel, the process proceeds to S306, and if it is determined that there is a luminance change, the process proceeds to S305.

[0029] In S305, the pixel signals of the pixels where the luminance has changed are stored in the memory of the data transmission unit 107.

[0030] In S306, it is determined whether processing has been completed for all pixel signals in one frame. If it is determined that processing has been completed for all pixel signals in one frame, the process proceeds to S307, and if it is determined that processing has not been completed for all pixel signals in one frame, the process returns to S304.

[0031] In S307, pixel signals in which luminance changes have occurred and which are held in the memory of the data transmission unit 107 are output together as image data for one frame.

[0032] FIG. 4 is a flowchart showing the processing of the image processing unit 104 in this embodiment.

[0033] In S401, event information is received from the event-based sensor 102.

[0034] In S402, it is determined whether a luminance change has occurred within one frame based on the received event information. If it is determined that a luminance change has not occurred within one frame, the process ends, and if it is determined that a luminance change has occurred within one frame, the process proceeds to S403.

[0035] In S403, image data is received from the frame-based sensor 103 for a predetermined period within one frame that includes a pixel signal in which a luminance change has occurred.

[0036] In S404, it is determined whether processing has been completed for all image data in one frame. If it is determined that processing has been completed for all image data in one frame, the process proceeds to S405, and if it is determined that processing has not been completed for all image data in one frame, the process returns to S402.

[0037] In S405, newly received image data is used for areas where there is a luminance change in the pixel signal within one frame, and image data from the previous frame is used for areas where there is no luminance change in the pixel signal within one frame to perform synthesis processing in the signal processing unit 110.

[0038] FIG. 5 is a diagram for explaining an example of image synthesis performed by the signal processing unit 110 in this embodiment under control in units of one horizontal period.

[0039] Image 501 is a portion where there is a change in luminance within one frame. Image data where there is a change in luminance within one frame is input from data receiving unit 109.

[0040] Image 502 is a portion where there was no change in brightness within one frame. Since no image data is input from data receiving unit 109 for this portion, image data from the previous frame is used.

[0041] Image 503 is a one-frame image generated by combining the image data of image 501 and image 502 .

[0042] Here, the reception control signal output from the reception control unit 108 may include information on the position of a pixel signal where a luminance change occurred in one frame. If the period for receiving image data is one horizontal period, the information indicates that the position of a pixel signal where a luminance change occurred in one frame is from the 20th horizontal period to the 30th horizontal period. Based on this information, the received image data from the 20th horizontal period to the 30th horizontal period can be combined with image data from the previous frame in any other horizontal period.

[0043] It should be noted that the control is not limited to the control in units of one horizontal period as in this embodiment, but may be control in units of one pixel, for example.

[0044] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0045] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0046] The disclosure of this embodiment includes the following configuration.

[0047] (Configuration 1) An imaging device comprising: an event-based sensor that outputs event information indicating coordinates and time information of a pixel where a luminance change is detected; a frame-based sensor that outputs image data on a frame-by-frame basis; and an image processing unit, the frame-based sensor stores pixel signals whose luminance has changed based on the event information in a memory, and then transmits the pixel signals as one frame of image data to the image processing unit; The image processing unit receives the image data of one frame transmitted from the frame-based sensor during a predetermined period that includes a pixel signal in which the luminance change occurred based on the event information, and performs predetermined signal processing on the received image data.

[0048] (Configuration 2) The imaging device according to configuration 1, wherein the image processing unit performs synthesis processing by using newly received image data for areas where there is a change in brightness within one frame, and by using image data of the previous frame for areas where there is no change in brightness within one frame.

[0049] (Configuration 3) 3. The imaging device according to configuration 1 or 2, wherein the frame-based sensor stores pixel signals, which have undergone a luminance change based on the event information, in a memory in units of one horizontal period.

[0050] (Configuration 4) 4. The imaging device according to configuration 3, wherein the image processing unit switches the predetermined period for receiving the image data for one frame in units of one horizontal period based on the event information.

[0051] (Configuration 5) 3. The imaging device according to claim 1, wherein the frame-based sensor stores pixel signals, which have undergone a luminance change based on the event information, in a memory on a pixel-by-pixel basis.

[0052] (Configuration 6) 6. The imaging device according to configuration 5, wherein the image processing unit switches the predetermined period for receiving the image data of one frame in units of one pixel based on the event information.

[0053] (Configuration 7) The imaging device according to any one of the first to sixth configurations, wherein the frame-based sensor controls the timing of transmitting the image data of one frame based on the amount of data of pixel signals that have changed in luminance within one frame.

[0054] (Configuration 8) The imaging device according to any one of configurations 1 to 7, wherein the image processing controls the timing of receiving the image data for one frame based on the amount of data of pixel signals that have changed in luminance within one frame. [Explanation of symbols]

[0055] 101 Imaging device 102 Event-based sensors 103 Frame-based Sensor 104 Image processing section

Claims

1. An imaging device comprising: an event-based sensor that outputs event information indicating coordinates and time information of a pixel where a luminance change is detected; a frame-based sensor that outputs image data on a frame-by-frame basis; and an image processing unit, the frame-based sensor stores pixel signals whose luminance has changed based on the event information in a memory, and then transmits the pixel signals as one frame of image data to the image processing unit; The image processing unit receives the image data of one frame transmitted from the frame-based sensor during a predetermined period that includes a pixel signal in which the luminance change occurred based on the event information, and performs predetermined signal processing on the received image data.

2. The imaging device according to claim 1, characterized in that the image processing unit performs synthesis processing by using newly received image data for areas where there is a change in brightness within one frame, and by using image data from the previous frame for areas where there is no change in brightness within one frame.

3. 2. The imaging device according to claim 1, wherein the frame-based sensor stores pixel signals, which have undergone a luminance change based on the event information, in a memory in units of one horizontal period.

4. 4. The imaging device according to claim 3, wherein the image processing unit switches the predetermined period for receiving the image data for one frame in units of one horizontal period based on the event information.

5. 2. The imaging device according to claim 1, wherein the frame-based sensor stores pixel signals in a memory for each pixel where a luminance change has occurred based on the event information.

6. 6. The imaging device according to claim 5, wherein the image processing unit switches the predetermined period for receiving the image data of one frame in units of one pixel based on the event information.

7. 2. The imaging device according to claim 1, wherein the frame-based sensor controls the timing of transmitting the image data of one frame based on the amount of data of pixel signals that have changed in luminance within one frame.

8. 2. The imaging device according to claim 1, wherein the image processing controls the timing of receiving the image data for one frame based on the amount of data of pixel signals whose luminance has changed within one frame.

Images