Imaging apparatus, control method, and computer program

Abstract

To provide an imaging apparatus for reducing a data processing load.SOLUTION: The imaging apparatus includes: an imaging element for an image for acquiring image data for image display; an event-based imaging element for acquiring data of an image area in which a predetermined change occurs; and a subject information processing part for generating subject information by processing the data acquired by the event-based imaging element. The subject information processing part associates the subject information with the image data of a storage period in the imaging element for the image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an imaging device, a control method, a computer program, and the like. [Background technology]

[0002] In recent years, mixed reality (MR) technology has become known as a technology that seamlessly combines the real world and the virtual world in real time, and augmented reality (AR) technology that overlays augmented information on real-world scenery.

[0003] That is, a real-world scene is videotaped in real time, and the user views the image with both eyes using a head-mounted display (HMD), and a computer graphics (CG) image generated according to the position and orientation of the imaging device is superimposed on it, which can then be viewed through a display device such as the HMD.

[0004] To achieve the above operation, images captured by an imaging element such as a CMOS image sensor are used to perform computer vision processing such as the display image that the user sees, position and orientation, surrounding distance information, and subject detection such as hands and fingers.

[0005] In conventional computer vision processing, as in Patent Document 1, update processing is performed where there is a change in the image obtained frame by frame, but redundant processing is also performed, such as updating parts that do not change.

[0006] Furthermore, Patent Document 2 describes an event-based sensor that outputs a change in luminance of each pixel of an imaging element in real time as an address event signal. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2007-219082 [Patent Document 2] Japanese Patent Application Publication No. 2019-134271 Summary of the Invention [Problem to be solved by the invention]

[0008] This increases the system load, which is one of the reasons for the huge increase in hardware processing capacity and power consumption. Furthermore, because CMOS image sensors capture images on a frame-by-frame basis, image information is lost during the non-exposure periods between frames. Therefore, in computer vision processing, which requires real-time processing, the sensor is unable to capture information that changes between frames, resulting in a lack of processing accuracy.

[0009] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an imaging device that reduces the data processing load. [Means for solving the problem]

[0010] In order to achieve the above object, the present invention provides an imaging device comprising: an image pickup element for acquiring image data for image display; an event-based imaging device that acquires data of an image area where a predetermined change occurs; a subject information processing unit that processes the data acquired by the event-based image sensor to generate subject information, The subject information processing section associates the subject information with image data during an accumulation period in the image pickup device. [Effects of the Invention]

[0011] According to the present invention, it is possible to provide an imaging device with a reduced data processing load. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a functional block diagram showing an example of the configuration of an imaging device according to a first embodiment of the present invention. [Figure 2] 1A and 1B are functional block diagrams showing an example of the configuration of an event-based imaging device according to a first embodiment. [Figure 3] FIG. 1 is a timing diagram showing an example of associating a computer vision processing result with a see-through video image according to the first embodiment. [Figure 4A] FIG. 10 is a timing diagram showing an example of associating a computer vision processing result with a see-through video image according to the second embodiment. [Figure 4B] FIG. 4B is a timing diagram showing a continuation of the example of FIG. 4A. [Figure 5] FIG. 10 is a diagram showing an example of a photographing angle of view according to the third embodiment. [Figure 6] FIG. 10 is a timing diagram showing an example of associating a computer vision processing result with a see-through video image according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. In each drawing, the same members or elements are designated by the same reference numerals, and duplicate descriptions will be omitted or simplified.

[0014] (Embodiment 1) Fig. 1 is a functional block diagram showing an example of the configuration of an imaging device according to embodiment 1 of the present invention. Note that some of the functional blocks shown in Fig. 1 are realized by causing a CPU or the like serving as a computer (not shown) included in the imaging device to execute a computer program stored in a memory serving as a storage medium (not shown).

[0015] However, some or all of these functions may be implemented by hardware. Examples of hardware that can be used include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs). Furthermore, the functional blocks shown in Figure 1 do not have to be built into the same housing, and may be configured as separate devices connected to each other via signal paths.

[0016] The imaging device 1 according to the first embodiment includes see-through video imaging units 100-1 and 100-2, event-based imaging units 108-1 and 108-2, a control unit 107, and display circuits 106-1 and 106-2, and constitutes an HMD. Note that the imaging device according to the first embodiment is not limited to an HDM, and can also be applied to, for example, a surveillance camera.

[0017] The see-through video imaging units 100-1 and 100-2 have see-through video imaging lenses 101-1 and 101-2 and see-through video imaging elements 102-1 and 102-2. The see-through video imaging lenses 101-1 and 101-2 collect light from a subject onto the see-through video imaging elements 102-1 and 102-2, respectively.

[0018] The see-through video imaging units 100-1 and 100-2 are collectively referred to as the see-through video imaging unit 100, and the see-through video imaging elements 102-1 and 102-2 are collectively referred to as the see-through video imaging element 102. The see-through video imaging lenses 101-1 and 101-2 are collectively referred to as the see-through video imaging lens 101. The see-through video imaging element 102 functions as an image imaging element that acquires image data for image display.

[0019] The see-through video imaging element 102 is, for example, a CMOS image sensor, and photoelectrically converts light incident through the see-through video imaging lens 101 through a Bayer array color filter consisting of red, blue, or green for each pixel. The see-through video imaging element 102 outputs a color image signal frame by frame based on a vertical synchronization signal from the control circuit 104 of the control unit 107.

[0020] Note that CMOS image sensors include a rolling shutter type imaging element that performs readout scanning on a row-by-row basis to perform a sequential readout operation, for example, from top to bottom of the screen, and a global shutter type imaging element that performs a collective accumulation operation on the entire screen of the image sensor. In this embodiment, an example using a rolling shutter type imaging element will be described, but a global shutter type imaging element may also be used.

[0021] The see-through video imaging elements 102-1 and 102-2 capture images for the left and right eyes, respectively, as images through which the HMD user views the surrounding environment. The see-through video imaging elements 102-1 and 102-2 are positioned within the imaging device 1 in consideration of human parallax so that the user does not feel uncomfortable when viewing with both eyes.

[0022] The event-based imaging units 108-1 and 108-2 have event-based imaging lenses 109-1 and 109-2 and event-based imaging elements 110-1 and 110-2, respectively. The event-based imaging lenses 109-1 and 109-2 collect light from the subject onto the event-based imaging elements 110-1 and 110-2.

[0023] The event-based imaging units 108-1 and 108-2 are collectively referred to as the event-based imaging unit 108, and the event-based imaging elements 110-1 and 110-2 are collectively referred to as the event-based imaging element 110.

[0024] The event-based imaging lenses 109-1 and 109-2 are collectively referred to as the event-based imaging lens 109. The event-based imaging element 110 acquires data of an image area where a predetermined change (such as a change in brightness or color of a predetermined amount or more) has occurred.

[0025] The event-based image sensor 110 outputs an event signal, such as a pixel address corresponding to incident light received through the event-based imaging lens 109 and the time when an event occurred. That is, the event-based image sensor outputs an event signal indicating the position of an image area where a predetermined change has occurred.

[0026] In this embodiment, for the sake of simplicity, an example will be described in which the see-through video imaging units 100-1 and 100-2 and the event-based imaging units 108-1 and 108-2 have the same field of view.

[0027] However, the field of view of event-based imaging unit 108 may be set to be wider so as to encompass the field of view of see-through video imaging unit 100. In other words, event-based imaging units 108-1 and 108-2 may have a field of view and a field of view that encompass the field of view and the field of view of see-through video imaging units 100-1 and 100-2. This may enable changes in the subject within the range being captured by see-through video imaging unit 100 to be captured.

[0028] The control unit 107 includes an image processing circuit 103, a control circuit 104, a memory circuit 105, and a computer vision processing circuit 111. The image processing circuit 103 performs digital image processing such as various corrections such as filtering and compression on the image signals output from the see-through video imaging elements 102-1 and 102-2.

[0029] Furthermore, the image processing circuit 103 can perform dynamic range expansion processing and the like by combining the image signals output from the see-through video imaging elements 102-1 and 102-2.

[0030] The control circuit 104 has a built-in CPU and the like as a computer, and functions as a control means that controls the operation of each part of the entire image pickup apparatus based on a computer program stored in a memory as a storage medium.

[0031] That is, the control circuit 104 controls the drive timing of the see-through video imaging element 102 and the event-based imaging element 110, and performs overall drive and control of the entire imaging device, including the image processing circuit 103, display circuit 106, and computer vision processing circuit 111. The memory circuit 105 is a recording medium such as a non-volatile memory or memory card that records and holds the image signals output from the image processing circuit 103.

[0032] The computer vision processing circuit 111 generates an event image based on the event signal from the event-based image sensor 110, and performs signal processing to generate subject information such as position information, posture information, distance information, and image recognition information of the subject's hands and fingers, etc. As described above, the subject information includes at least one of the position information, posture information, distance information, and image recognition information of the subject's hands and fingers, etc.

[0033] Here, the computer vision processing circuit 111 functions as a subject information processing unit that processes data acquired by the event-based image sensor to generate subject information. The subject information processing unit also performs signal processing to generate the position of an image area where a predetermined change has occurred, based on an event signal from the event-based image sensor.

[0034] As described above, the event-based image sensor 110 generates event signals asynchronously. In response to the asynchronously transmitted event signals, the computer vision processing circuit 111 generates a time-series image that indicates the position of at least one pixel where a change in luminance of, for example, a predetermined amount or more has occurred within a predetermined time range, as well as the direction of the change in luminance.

[0035] Specifically, the information on brightness changes detected by the event-based image sensor 110 is rearranged into a raster scan format to convert address event signals into a coordinate space consisting of spatial X and Y axes, thereby generating an XY coordinate image of only the area where brightness changes occurred.

[0036] This event image signal can be used to perform the computer vision processing described above, and only areas where there is a change in brightness can be processed, which can reduce the system load and shorten the processing latency.

[0037] The display circuit 106 (display circuits 106-1 and 106-2) is composed of a display panel such as an LCD (Liquid Crystal Display) or an OLED (Organic Light Emitting Diode). The display circuit 106 displays an image captured by the see-through video imaging element 102.

[0038] At this time, display circuit 106-1 displays to the user an image for display to the left eye captured by see-through video imaging element 102-1, and display circuit 106-2 displays to the user an image for display to the right eye captured by see-through video imaging element 102-2, thereby enabling the images from see-through video imaging elements 102-1 and 102-2, which have parallax, to be viewed as pseudo-see-through images in the HMD.

[0039] Next, the configuration of the event-based imaging device 110 will be described with reference to Fig. 2. Figs. 2(A) and 2(B) are functional block diagrams showing an example configuration of the event-based imaging device according to the first embodiment.

[0040] Next, the configuration of the event-based image sensor 110 will be described. Fig. 2(A) is a diagram showing an example configuration of the event-based image sensor 110. The event-based image sensor 110 is composed of an event-based pixel unit 201 and an event-based peripheral circuit 202. The event-based peripheral circuit 202 includes a vertical arbitration circuit 2021 and a horizontal output circuit 2022.

[0041] 2(B) is a diagram showing an example of the configuration of each pixel of each event-based pixel unit 201 that constitutes the event-based image sensor 110. Each pixel of the event-based pixel unit 201 includes a photoelectric conversion unit 2011, a pixel counter 2012, a time counter 2013, a first determination circuit 2014, a memory 2015, a comparator 2016, a second determination circuit 2017, a response circuit 2018, and a selection circuit 2019.

[0042] The photoelectric conversion unit 2011 includes an avalanche photodiode (SPAD) that operates in Geiger mode, and a pixel counter 2012 counts the number of photons incident on the photoelectric conversion unit 2011. A time counter 2013 counts the time at which a photon enters the photoelectric conversion unit 2011.

[0043] By configuring an event-based sensor using a SPAD, it is possible to detect luminance changes at the level of a single photon. By detecting luminance changes at the level of a single photon, it is possible to obtain an address event signal even in night vision conditions such as at night.

[0044] When the number of photons counted by the pixel counter 2012 reaches a first threshold, a first decision circuit 2014 stops the time counter 2013 from counting time. Past count values of the time counter 2013 are stored in a memory 2015, and a comparator 2016 is used to calculate the difference between the current count value of the time counter 2013 and the past count value of the time counter 2013.

[0045] If the difference count value is equal to or greater than the second threshold, the second determination circuit 2017 sends a request signal to the vertical arbitration circuit 2021 via the response circuit 2018. The response circuit 2018 also receives a response from the vertical arbitration circuit 2021 indicating whether or not the output of address event data is permitted. If the difference count value is less than the second threshold, the response circuit 2018 does not send a request signal.

[0046] When a response indicating permission to output address event data is received via the response circuit 2018, the count value of the time counter 2013 is output via the selection circuit 2019 to the horizontal output circuit 2022. The horizontal output circuit 2022 outputs the received count value as an output signal from the event-based image sensor 110 to the image processing circuit 103.

[0047] Because the differential count value calculated by the comparator 2016 corresponds to the reciprocal of the frequency of incident photons, the event-based image sensor 110 according to this embodiment has the function of measuring "changes in the frequency of incident photons," i.e., changes in luminance. Furthermore, the second decision circuit 2017 is used to output an address event only if the difference in the intervals at which the number of incident photons reaches the first threshold is equal to or greater than the second threshold.

[0048] That is, the photoelectric conversion element outputs the incidence frequency when the difference in incidence frequency is equal to or greater than the second threshold, and does not output the incidence frequency when the difference is less than the threshold. By adopting the above configuration, it is possible to realize an asynchronous photoelectric conversion element that detects changes in luminance as address events for each pixel address in real time.

[0049] In this way, the event-based image sensor counts the number of incident photons and determines the timing when the counted number of photons exceeds a predetermined threshold. The event-based image sensor also measures the time (number of clocks) required for the number of photons to reach or exceed a first threshold, and detects changes in brightness by comparing the time required.

[0050] That is, for example, if the previously measured required time is T0 and the latest required time is T, if the difference T-T0 is equal to or greater than the second threshold, a change in brightness in the negative direction is detected. If the difference T0-T is equal to or greater than the second threshold, a change in brightness in the positive direction is detected.

[0051] If the difference between T and T0 is less than a second threshold, no change in brightness is detected. The second threshold is a value equal to or greater than zero, and is set in accordance with a preset value or other parameters.

[0052] In this embodiment, a photoelectric conversion element is used in which a SPAD is used as the photoelectric conversion unit, and a change in the frequency of photon incidence is detected by measuring the time at which a photon is incident. However, the configuration shown in Fig. 2 is not necessary as long as the photoelectric conversion element detects a change in luminance as an address event in real time. For example, as described in Patent Document 2, a photoelectric conversion element that detects a change in luminance as a voltage change may be used.

[0053] 3 is a timing diagram showing an example of associating a computer vision processing result with a see-through video image according to embodiment 1. Using FIG. 3, an example of associating the display image data obtained by the see-through video imaging unit 100 with the results of processing by the computer vision processing circuit 111 based on the event signal obtained by the event-based imaging unit 108 will be described.

[0054] The see-through video imaging element 102 starts shutter scanning 301 at time t31 based on the vertical synchronization signal VD from the control circuit 104, in accordance with, for example, exposure conditions determined by the control unit 107. Note that shutter scanning here refers to the operation of resetting the values of the pixel counters and the like of each pixel, for each pixel row, in order from top to bottom of the screen.

[0055] Also, readout scanning 302 starts at time t32 in synchronization with the falling edge of the vertical synchronization signal VD, and readout scanning is performed row by row from the top row to the bottom row of the screen, and accumulation operation is performed until time t33.

[0056] Here, the readout scan refers to the operation of reading out the count value of the pixel counter of each pixel, for each pixel row, sequentially from top to bottom of the screen. Note that the inclinations of the shutter scan 301 and the readout scan 302 in Figure 3 are assumed to be the same.

[0057] At this time, the see-through video accumulation period 1 by the see-through video imaging element 102 is the period from time t31 to t33, and the user will view the image data after this accumulation period.

[0058] That is, the see-through video accumulation period 1 corresponds to the period from the start to the end of capturing the image visually recognized by the user. Note that the see-through video accumulation period 1 is the period during which any pixel in the see-through video imaging element 102, which serves as an image imaging element, photoelectrically converts image data for image display.

[0059] On the other hand, the event-based image capture device 110 performs an image capture operation asynchronously with the image capture operation of the see-through video image capture device 102, and outputs an event signal on a pixel-by-pixel basis when a change in luminance occurs.

[0060] At this time, the computer vision processing circuit 111 performs signal processing such as generating an event image and recognizing objects such as hands, fingers, and position and orientation information and surrounding distance information based on the event image, based on the event signal from the event-based image sensor 110. However, because the event-based image sensor 110 outputs asynchronously, the image capturing operation will not stop during its operation unless instructed to do so by the control circuit 104.

[0061] Meanwhile, the computer vision processing circuit 111 performs an operation for a predetermined period to associate the subject of the image obtained by the see-through video imaging element 102 with the event detection information. That is, the computer vision processing circuit 111 performs a process to associate only the results of computer vision processing result association period 1 from time t31 to t33 as information on image data of the see-through video imaging element 102 obtained during the accumulation period from time t31 to t33.

[0062] Next, from times t34 to t36, the see-through video imaging element 102 performs imaging operation for the next frame, and enters operation in see-through video accumulation period 2. At this time, the results of computer vision processing result association period 1 associated with see-through video accumulation period 1, which is the accumulation period for the previous frame of the see-through video imaging element 102, are reset.

[0063] The see-through video accumulation period 2 is also a period during which any pixel photoelectrically converts image data for image display in the see-through video imaging element 102 serving as an image imaging element.

[0064] Then, the computer vision processing circuit 111 performs processing to associate the image data from the see-through video accumulation period 2 with the computer vision processing results from the computer vision processing result association period 2 from time t34 to time t36. That is, the computer vision processing circuit 111 as a subject information processing unit associates the subject information with the image data from the accumulation period in the image pickup element.

[0065] In this manner, in this embodiment, only the area where a change in the subject's situation has occurred is imaged by the event-based image capture device 110, and the event signal is used to perform computer vision processing only on the area where the subject has changed.

[0066] This reduces the processing load on the computer vision processing circuit 111, and by linking it to see-through video imaging data, it is possible to provide an imaging device that improves the real-time processing performance of computer vision processing for images viewed by the user.

[0067] In this embodiment, processing is performed asynchronously with the see-through video imaging in the computer vision processing circuit 111. Then, among the processing results, the processing results during the accumulation period of the see-through video imaging element 102 are associated only with the image data during the accumulation period of the see-through video imaging element 102.

[0068] However, the computer vision processing circuit 111 may process only the event-based signals obtained by the event-based imager 110 during the accumulation period of the see-through video imager 102. The results of this processing may then be associated with the image data of the see-through video imager 102 during the same period.

[0069] In this way, the processing load can be reduced by associating subject information based on an event signal generated during the accumulation period of the image pickup element, or subject information based on an event signal processed during the accumulation period, with image data generated during the accumulation period.

[0070] (Embodiment 2) FIG. 4A is a timing diagram showing an example of associating a computer vision processing result with a see-through video image according to the second embodiment, and FIG. 4B is a timing diagram showing a continuation of the example shown in FIG. 4A.

[0071] 4A and 4B show an example of association between the see-through video image capture device 102 according to the second embodiment of the present invention and the computer vision processing results using the event signals of the event-based image capture device 110. Note that 401 to 404 in FIG. 4 correspond to 301 to 304 in FIG. 3, and times t41 to t46 correspond to times t31 to t36 in FIG. 3, and their explanations will be omitted.

[0072] In the second embodiment, during the see-through video accumulation period 1 from time t41 to time t43, the computer vision processing circuit 111 associates the computer vision processing results from time t41 to time t43 with the image data obtained during the see-through video accumulation period 1.

[0073] On the other hand, during the see-through video accumulation period 2 from time t44 to t46, the computer vision processing circuit 111 associates the computer vision processing results from time t44 to t46 with the image data obtained during the see-through video accumulation period 2.

[0074] Note that the period from time t43 to t44 is the inter-frame (blanking period between the first and second frames) between see-through video accumulation period 1 and see-through video accumulation period 2. The computer vision processing results during this period (computer vision processing results between the first and second frames) are not associated with see-through video accumulation period 1 and see-through video accumulation period 2.

[0075] This is because the change in image data during see-through video accumulation period 1 or see-through video accumulation period 2 at times t43 to t44 does not necessarily match the computer vision processing result between the first and second frames.

[0076] That is, this is to prevent the position information detection result obtained by computer vision processing between the first and second frames from not matching the subject position information in the image data of the see-through video imaging element 102.

[0077] On the other hand, if the subject moves or the situation changes between times t43 and t44, the results of computer vision processing result association period 2, which are associated with see-through video accumulation period 2 processed between times t44 and t46, may suddenly change. In other words, the results of computer vision processing result association period 2 may become more discrete than the results of computer vision processing result association period 1 obtained between times t41 and t43.

[0078] Therefore, if the position of the subject suddenly changes, the subject recognition information from time t44 to t46 cannot keep up with the change, resulting in a missed subject detection. Alternatively, an error may occur, such as associating the computer vision processing results with a completely different subject.

[0079] Therefore, in the second embodiment, the computer vision processing circuit 111 receives and processes the event signal from the event-based image sensor 110 even between times t43 and t44, and continues to capture changes in the subject during that period.

[0080] The detection result is then associated with subject information from the computer vision processing results from time t44 to t46, and subject movement information, etc. is reflected in the computer vision processing results from time t44 onwards, preventing subject detection loss, etc. In this way, in the second embodiment, the event signal associated with image data generated during the accumulation period reflects an event signal that occurred before the start of the accumulation period.

[0081] At this time, the computer vision processing results obtained from time t41 to t43 may be reflected in the computer vision processing results processed from time t43 to t44, or the computer vision processing results obtained from time t41 to t43 may not be reflected.

[0082] The period for calculating the results of computer vision processing between frames is from the end of readout scans 402, 404, and 406 in the see-through video accumulation period until the start of shutter scans 403, 405, etc. of the next frame.

[0083] Therefore, if the accumulation operation period differs between frames, the computer processing results between frames will vary. That is, the subject information processing unit varies the processing period for the processing data to be handed over to the next frame depending on the period from the end of the accumulation period of a given frame to the start of the accumulation period of the next frame.

[0084] To give a specific example, the see-through video accumulation period 3 from time t47 to t49 is a shorter accumulation period than the see-through video accumulation period 2. Note that the see-through video accumulation period 3 is also a period during which any pixel in the see-through video imaging element 102, which serves as an image imaging element, photoelectrically converts image data for image display.

[0085] Therefore, the period from time t46 when readout scan 404 in see-through video accumulation period 2 is completed to time t47 when shutter scan 405 in see-through video accumulation period 3 starts is longer than the period from time t43 to t44.

[0086] Therefore, the computer vision processing result between the second and third frames from time t46 to time t47 is controlled to be longer than the processing period for the computer vision processing result between the first and second frames from time t43 to t44.

[0087] By performing such operations, it is possible to suppress the data processing rate while improving the tracking and real-time performance of computer vision processing between frames of see-through video imaging.

[0088] (Embodiment 3) 5 and 6, the association between the readout rows of the see-through video image capture device 102 according to the third embodiment of the present invention and the results of computer vision processing using the event signals of the event-based image capture device 110 will be described.

[0089] FIG. 5 is a diagram showing an example of the imaging angle of view according to the third embodiment, and is used to explain the range of the angle of view captured by the see-through video imaging unit 100 and the event-based imaging unit 108 according to the third embodiment.

[0090] In this embodiment, the imaging angle of view 501 is set to be approximately the same for the see-through video imaging section 100 and the event-based imaging section 108 of the imaging device 1. The imaging angle of view 501 is determined by the see-through video imaging lens 101, the event-based imaging lens 109, the see-through video imaging element 102, and the event-based imaging element 110.

[0091] Specifically, the shooting angle of view 501 of the see-through video imaging unit 100 is determined by the focal length of the see-through video imaging lens 101 and the light receiving range of the see-through video imaging element 102. The shooting angle of view 501 of the event-based imaging unit 108 is determined by the focal length of the event-based imaging lens 109 and the light receiving range of the event-based imaging element 110.

[0092] Furthermore, the shooting angle of view 501, which is determined by the focal length of each lens and the light receiving range of each image sensor, is determined by the horizontal angle of view 502 in the horizontal direction and the vertical angle of view 503 in the vertical direction. The shooting angle of view 501 is set so that the shooting angle of view 501 is the same for the subject by setting the positional configuration of the see-through video imaging unit 100 and the event-based imaging unit 108 within the imaging device 1, the focal length of each lens, and the light receiving range of each image sensor.

[0093] In this embodiment, for ease of explanation, the horizontal angle of view 502 is assumed to be the same imaging range, but the horizontal viewpoints of see-through video imaging unit 100-1 and event-based imaging unit 108-1 are set to a common first viewpoint. The horizontal viewpoints of see-through video imaging unit 100-2 and event-based imaging unit 108-2 are set to a common second viewpoint. The first viewpoint and the second viewpoint are positioned with a horizontal offset of a predetermined base line length.

[0094] The see-through video imaging element 102 and the event-based imaging element 110 perform imaging operations within a range determined by the vertical angle of view 503, and the see-through video imaging element 102 is a CMOS image sensor that performs sequential readout scanning in the row and column directions. On the other hand, the event-based imaging element captures images asynchronously on a pixel-by-pixel basis, so it outputs imaging signals only from areas where there is a change in brightness, such as subject movement, within the range of the imaging angle of view 501.

[0095] Furthermore, the timing of the readout scanning of the see-through video imaging element 102 is different between the subject position on the readout start row 504 of the see-through video imaging element 102 and the subject position on the n-th readout row 505 .

[0096] Therefore, if the computer vision processing results based on the event output of the event-based image capture device 110 are directly associated with the image data of the see-through video image capture device 102 between times t31 and t33 in FIG. 3, a timing discrepancy may occur.

[0097] For example, the read timing of the nth read row 505 of the see-through video imager 102 may differ from the read timing when the object changes at the nth row position in the event-based imager 110. In this case, for example, the detected object position as a result of computer vision processing may be associated with an incorrect position in the see-through video image.

[0098] Therefore, in the third embodiment, the readout row position and readout time of the see-through video imaging element 102 are associated with the computer vision processing results of the same row position and time.

[0099] Fig. 6 is a timing diagram showing an example of associating a computer vision processing result with a see-through video image according to embodiment 3. Note that 601 and 602 in Fig. 6 correspond to 301 and 302 in Fig. 3, and times t61, t63, and t66 correspond to times t31, t32, and t33 in Fig. 3, and their explanations will be omitted.

[0100] At time t61, the see-through video imaging element 102 starts shutter scanning 601 from the start row at the top of the screen in synchronization with the falling edge of the see-through video imaging element horizontal synchronization signal HD from the control circuit 104. This causes photon counting to occur from time t61 to time t63. In other words, the period from time t61 to time t63 corresponds to the exposure period or photoelectric conversion period of the see-through video imaging element 102.

[0101] On the other hand, the event-based image sensor 110 outputs an event signal for a subject that has undergone a change within the photographic angle of view 501, and the computer vision processing circuit 111 processes the event signal.

[0102] However, the computer vision processing circuit 111 processes only the event signal at the same row position as the readout start row 504 of the see-through video imaging element 102. In addition, only the event signal in the period from time t61 to t63 is associated with the image data of the start row of the see-through video imaging element 102.

[0103] Next, for example, at time t62, shutter scanning 301 is performed on the nth row of the see-through video imaging element 102, and the period until readout scanning 302 is performed at time t64 is the exposure period or photoelectric conversion period for the nth row.

[0104] Meanwhile, the computer vision processing circuit 111 processes only the event signals at the row position corresponding to the readout position of the nth row of the see-through video imaging element of the event-based imaging element 110. Furthermore, only the event signals during the period from time t62 to t64 are associated with the image data of the nth row of the see-through video imaging element 102.

[0105] That is, the event signal of the nth row of the event-based imaging element 110 at a time other than the exposure period (or photoelectric conversion period) of the nth row of the see-through video imaging element 102 is not associated with the image data of the nth row of the see-through video imaging element 102.

[0106] This is repeated for each row, and similar operations are performed until the end row of the see-through video image sensor 102 at time t66. By performing such operations, for example, after time t63, which is outside the accumulation operation period of the start row, the computer vision processing results for the same readout position as the corresponding row are no longer associated, thereby improving subject tracking ability.

[0107] In this embodiment, the computer vision processing results are associated with the readout rows of the see-through video image sensor 102 on a row-by-row basis, but this is not limited to this. For example, multiple readout rows of the event-based image sensor 110 may be set for multiple readout rows of the see-through video image sensor 102, and event signals for each of the multiple rows may be received and processed by the computer vision processing circuit 111 to associate the results.

[0108] That is, it is sufficient if the subject information is associated with the image data during the accumulation period of the image pickup element in units of one or more rows. Note that in this embodiment, the row position of the read start row 504 of the see-through video pickup element 102 and the read position address of the event-based pickup element are assumed to coincide in spatial coordinates.

[0109] However, the image data of the see-through video image sensor 102 may be associated with the computer vision processing results so that a combination of multiple rows of the see-through video image sensor 102 and a combination of multiple rows of the event-based image sensor 110 coincide on the same spatial coordinates. That is, subject information based on an event signal at a position of a specific row that occurs during the accumulation period of the specific row of the image image sensor may be associated with the image data of the specific row.

[0110] Furthermore, subject information based on an event signal generated during the accumulation period of a specified row of the image pickup element, or subject information based on an event signal processed during the accumulation period of the specified row, may be associated with image data generated during the accumulation period.

[0111] Although the present invention has been described in detail above based on the preferred embodiments, the present invention is not limited to the above embodiments, and various modifications and combinations of the above embodiments are possible based on the spirit of the present invention, and are not excluded from the scope of the present invention. The present invention also includes the following combinations.

[0112] (Configuration 1) An imaging device comprising: an image imaging element that acquires image data for image display; an event-based imaging element that acquires data for an image area where a predetermined change has occurred; and a subject information processing unit that processes the data acquired by the event-based imaging element to generate subject information, wherein the subject information processing unit associates the subject information with image data for an accumulation period in the image imaging element.

[0113] (Configuration 2) The imaging device according to configuration 1, wherein the predetermined change includes a change in luminance of a predetermined amount or more or a change in color of a predetermined amount or more.

[0114] (Configuration 3) The imaging device according to configuration 1 or 2, wherein the subject information includes at least one of position information, posture information, distance information, and image recognition information of the subject, such as hands and fingers.

[0115] (Configuration 4) The imaging device according to any one of configurations 1 to 3, wherein the accumulation period is a period during which the image data for image display is photoelectrically converted in the image imaging element.

[0116] (Configuration 5) The imaging device according to any one of configurations 1 to 4, wherein the image imaging element is a rolling shutter type imaging element or a global shutter type imaging element.

[0117] (Configuration 6) The imaging device according to any one of configurations 1 to 5, wherein the event-based imaging device outputs an event signal indicating the position of the image area where the predetermined change has occurred.

[0118] (Configuration 7) The imaging device according to configuration 6, wherein the subject information processing unit performs signal processing to generate the position based on the event signal from the event-based imaging device.

[0119] (Configuration 8) The imaging device described in Configuration 6 or 7, characterized in that the subject information processing unit associates the subject information based on the event signal generated during the accumulation period of the image imaging element, or the subject information based on the event signal processed during the accumulation period, with image data generated during the accumulation period.

[0120] (Configuration 9) An imaging device according to any one of configurations 6 to 8, characterized in that the event signal associated with the image data generated during the accumulation period reflects the event signal that occurred before the start of the accumulation period.

[0121] (Configuration 10) The imaging device described in Configuration 9, characterized in that the subject information processing unit varies the processing period of the processing data to be handed over to the next frame depending on the period from the end of the accumulation period of a specified frame to the start of the accumulation period of the next frame.

[0122] (Configuration 11) An imaging device described in any one of configurations 6 to 10, characterized in that the subject information processing unit associates the subject information based on the event signal generated during the accumulation period of a specified row of the image imaging element, or the subject information based on the event signal processed during the accumulation period of the specified row, with image data generated during the accumulation period.

[0123] (Configuration 12) The imaging device described in Configuration 11, characterized in that the subject information processing unit associates the subject information based on the event signal at the position of the specified row that occurred during the accumulation period of the specified row of the image imaging element with the image data of the specified row.

[0124] (Configuration 13) The imaging device according to any one of configurations 1 to 12, wherein the subject information processing section associates the subject information with image data during the accumulation period of the image imaging element on a row-by-row basis.

[0125] (Method) A control method for controlling an imaging device having an image imaging element that acquires image data for image display and an event-based imaging element that acquires data of an image area where a predetermined change has occurred, characterized in that the control method processes the data acquired by the event-based imaging element to calculate subject information and associates the subject information with image data during an accumulation period in the image imaging element.

[0126] (Program) A computer program for controlling each unit of the imaging device according to any one of configurations 1 to 13 by a computer.

[0127] In order to realize some or all of the control in the above-described embodiments, a computer program that realizes the functions of the above-described embodiments may be supplied to an imaging device or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the imaging device or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0128] 1: Imaging device 100: See-through video imaging unit 101: See-through video imaging lens 102: See-through video imaging element 103: Image processing circuit 104: Control circuit 105: Memory circuit 106:Display circuit 107: Control unit 108: Event-based imaging unit 109: Event-based imaging lens 110: Event-based image sensor 111: Computer vision processing circuit 201: Event-based pixel unit 2011: Photoelectric conversion section 2012: Pixel Counter 2013: Time Counter 2014: First decision circuit 2015: Memory 2016: Comparator 2017: Second decision circuit 2018: Response Circuit 2019: Selection circuit 202: Event-based peripheral circuit 2021: Vertical arbitration circuit 2022: Horizontal readout circuit 301, 401, 403, 405: Shutter scanning 302, 402, 404, 406: Readout scan 501: Shooting angle 502: Horizontal angle of view 503:Vertical angle of view

Claims

1. an image pickup element for acquiring image data for image display; an event-based imaging device that acquires data of an image area where a predetermined change occurs; a subject information processing unit that processes the data acquired by the event-based image sensor to generate subject information, The imaging device, wherein the subject information processing section associates the subject information with image data during an accumulation period in the image pickup element.

2. 2. The imaging device according to claim 1, wherein the predetermined change includes a change in luminance or a change in color of a predetermined amount or more.

3. 2. The imaging device according to claim 1, wherein the subject information includes at least one of position information, posture information, distance information, and image recognition information of a hand or finger of the subject.

4. 2. The imaging device according to claim 1, wherein the accumulation period is a period during which the image data for image display is photoelectrically converted in the image imaging element.

5. 2. The imaging device according to claim 1, wherein the image pickup element is a rolling shutter type image pickup element or a global shutter type image pickup element.

6. 2. The imaging device according to claim 1, wherein the event-based imaging device outputs an event signal indicating a position of the image area where the predetermined change has occurred.

7. 7. The imaging apparatus according to claim 6, wherein the subject information processing unit performs signal processing to generate the position based on the event signal from the event-based imaging device.

8. The imaging device according to claim 6, characterized in that the subject information processing unit associates the subject information based on the event signal generated during the accumulation period of the image imaging element, or the subject information based on the event signal processed during the accumulation period, with image data generated during the accumulation period.

9. 7. The imaging device according to claim 6, wherein the event signal associated with the image data generated during the accumulation period reflects the event signal that occurred before the start of the accumulation period.

10. 10. The imaging device according to claim 9, wherein the subject information processing section varies a processing period for processing data to be handed over to the next frame depending on a period from the end of an accumulation period of a predetermined frame to the start of an accumulation period of the next frame.

11. The imaging device described in claim 6, characterized in that the subject information processing unit associates the subject information based on the event signal generated during the accumulation period of a specified row of the image imaging element, or the subject information based on the event signal processed during the accumulation period of the specified row, with image data generated during the accumulation period.

12. The imaging device according to claim 11, characterized in that the subject information processing unit associates the subject information based on the event signal at the position of the specified row that occurred during the accumulation period of the specified row of the image imaging element with the image data of the specified row.

13. 2. The imaging device according to claim 1, wherein the subject information processing section associates the subject information with the image data during the accumulation period of the image pickup element in units of rows.

14. an image pickup element for acquiring image data for image display; an event-based imaging element that acquires data of an image area where a predetermined change occurs, A control method comprising: processing the data acquired by the event-based image capture device to calculate subject information; and associating the subject information with image data during an accumulation period in the image capture device.

15. A computer program for controlling each unit of the imaging device according to any one of claims 1 to 13 by a computer.

Images