Subject tracking device, control method therefor, imaging system, and program

Abstract

To provide a technique capable of continuously tracking a subject while suppressing a processing load and a power consumption.SOLUTION: An imaging system 10 comprises: an event detection device 11; and an imaging device 12. The event detection device 11 detects a fact that an illumination change of a pixel for performing a photoelectric conversion of an incident radiation exceeds a predetermined threshold. The imaging device 12 images a subject with a fixed frame rate. A system control part 13 performs a control of the event detection device 11 and the imaging device 12. An event detection control part 14 performs a control for setting an event detection condition of the event detection device 11 on the basis of an imaging state of the imaging device 12. A data processing part 15 generates an imaging data from each output signal of the event detection device 11 and the imaging device 12. A tracking control part 16 performs a detection of the subject and a tracking control by using the image data generated by the data processing part 15.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing technique using an imaging device and an event detection device.

Background Art

[0002] Image sensors mounted on imaging devices include asynchronous sensors and synchronous sensors. An event-driven vision sensor (hereinafter referred to as an "event-based sensor") is an asynchronous sensor that detects a luminance change for each pixel as an event and outputs an event signal including the time when the event occurred and the pixel position asynchronously. A synchronous sensor (hereinafter also referred to as a "frame-based sensor") performs imaging in synchronization with a vertical synchronization signal and outputs frame data, which is image data for one frame (screen) at the period of the vertical synchronization signal.

[0003] An event-based sensor can detect, for example, that the amount of luminance change exceeds a predetermined threshold as an event, and thus has the advantages of low latency and low power consumption compared to a frame-based sensor that reads out all pixels. In addition, the pixels of an event-based sensor perform logarithmic conversion of the luminance of incident light into voltage. Even in a low-luminance state, a slight luminance difference can be detected, and in a high-luminance state, conversely, when a large luminance difference occurs, the sensor reacts to prevent saturation of the event signal and obtain a wide dynamic range. Furthermore, the temporal resolution of event information is as high as several ns (nanoseconds) to several μs (microseconds), and there is a feature that there is no image blur (subject blur) for a moving subject.

[0004] Imaging systems that utilize the respective characteristics of the above sensors have been proposed. Patent Document 1 discloses means for dynamically changing the threshold value of an event detection device based on external information in an imaging system including an imaging device and an event detection device. Patent Document 2 also discloses means for designating a detection range of an event according to a subject recognition result of a tone image based on a tone signal in a sensor capable of simultaneously outputting a tone signal and event data.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] In an imaging device having a subject tracking function, it is necessary to perform imaging while tracking a specific subject. In the case of severe exposure conditions or a fast-moving subject, the imaging device may lose sight of the subject. Severe exposure conditions are conditions corresponding to scenes with a sudden change in brightness or scenes with low contrast (saturation, low illuminance), etc. Here, it is assumed that an event detection device is used assistively. If the event detection device is frequently operated together with the imaging device, the processing load and power consumption of the entire imaging system may increase.

[0007] Patent Document 1 discloses a means for dynamically changing an event detection threshold, but only mentions changing it according to external vehicle information such as a sunlight sensor or a raindrop sensor. Further, Patent Document 2 does not mention a means for changing the size of an event detection area according to the moving speed and moving direction of a target object in the setting of the event detection area based on the target object in the gradation image. An object of the present invention is to provide a technique that enables continuous tracking of a subject while suppressing the processing load and power consumption.

Means for Solving the Problems

[0008] The subject tracking device according to an embodiment of the present invention includes: acquisition means for acquiring the output of an event detection device that detects an event from the luminance change of pixels and the output of an imaging device that images a subject at a predetermined frame rate; first control means for controlling the event detection device and the imaging device; second control means for controlling the detection conditions for the event detection device to detect an event; data processing means for generating image data from the output of the imaging device and the output of the event detection device; and third control means for performing subject detection and tracking control using the image data generated by the data processing means. The second control means performs control to set the detection conditions of the event detection device corresponding to the imaging state of the imaging device.

Advantages of the Invention

[0009] According to the present invention, it is possible to provide a technique that enables continuous tracking of a subject while suppressing the processing load and power consumption.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the embodiments, as an application example of the subject tracking device according to the present invention, an imaging system using an event-based sensor and a frame-based sensor will be described.

[0012] FIG. 1 is a block diagram showing a configuration example of an imaging system 10. The imaging system 10 includes an event detection device 11 and an imaging device 12. The imaging element included in the event detection device 11 is an event-based sensor. The imaging element included in the imaging device 12 is a frame-based sensor.

[0013] The event detection device 11 detects a change in luminance within the imaging range of the event-based sensor and outputs an event signal asynchronously. As an example of an asynchronous type event-based sensor, there is a configuration in which a plurality of pixels are arranged in a two-dimensional array, and when a voltage signal that is the logarithm of the intensity of light incident on each pixel exceeds a threshold value, a trigger signal is generated and an event signal is output. The event signal is a signal associated with an event and includes, for example, each piece of information on the time when the event is detected and the pixel position where the event occurs. The time when the event is detected may be measured based on the time of the internal clock of the event detection device 11 (event detection device time), or may be reset as necessary. Further, the event signal may include one or more pieces of information shown below. · Information representing a luminance change value. · Information representing the sign (positive or negative) of the luminance change value.

[0014] The event detection device 11 outputs an event signal asynchronously only when a change in the luminance of the image occurs. Note that "output asynchronously" means performing an output that is temporally independent in pixel units without synchronizing all pixels.

[0015] The imaging device 12 captures images at a fixed frame rate synchronized with the vertical synchronization signal and outputs image data in a frame format. Examples of synchronous frame-based sensors include a CMOS (Complementary Metal Oxide Semiconductor) type image sensor, a CCD (Charge Coupled Device) type image sensor, and the like.

[0016] The system control unit 13 oversees the control of the imaging system 10. The system control unit 13 includes a CPU (Central Processing Unit) and the like, and performs control of the entire system and various calculations.

[0017] The data processing unit 15 acquires the event signal from the event detection device 11 and the image data including the imaging information from the imaging device 12. Further, the data processing unit 15 processes the event signal and the image data and outputs them to the event detection control unit 14 and the tracking control unit 16. The data processing unit 15 acquires the tracking subject information, which is the tracking result of the subject, from the tracking control unit 16.

[0018] The event detection control unit 14 calculates the conditions (hereinafter referred to as detection conditions) for the event detection device 11 to detect an event based on the tracking subject information acquired from the data processing unit 15, and instructs the system control unit 13 to set or change the detection conditions.

[0019] The tracking control unit 16 acquires the image data and the event signal from the data processing unit 15, or acquires the data of the framed image (hereinafter referred to as the event image) generated from the event signal. The tracking control unit 16 performs subject detection processing and tracking control based on the acquired information. For example, the tracking control unit 16 performs pattern matching processing to identify whether the subject is a person or an animal, or an object (such as a vehicle), and when the subject is a person, detects the position of the face and the position of the pupils. Note that the identification process may be executed by other methods without being limited to the pattern matching process. The tracking control unit 16 outputs the control information (tracking control information) of the tracking subject to the system control unit 13. Subject tracking is performed by a known method, and imaging is performed with appropriate adjustment of focusing, brightness, color, etc. for the tracking subject.

[0020] The event detection device 11, the imaging device 12, each of the control units 13, 14, 16, and the data processing unit 15 have storage areas necessary for control and processing, and read and write data at necessary timings.

[0021] Referring to FIG. 2, the events generated by the event detection device 11 will be described. In the graph shown in the upper diagram of FIG. 2, the horizontal axis is the time axis t, and the vertical axis represents the voltage (Vp), which is the logarithmic function of the intensity of the incident light. A plurality of dotted lines shown horizontally from the vertical axis represent the threshold value (denoted as Θ) of the voltage signal when the event detection device 11 generates a trigger signal. That is, the vertical axis is set with the voltage change amount Θ as the unit. The lower diagram of FIG. 2 shows the state of event detection, and the horizontal axis is the time axis t. When the voltage Vp increases beyond the threshold value Θ, it is represented as a “+ event” by an upward arrow, and when the voltage Vp decreases beyond the threshold value Θ, it is represented as a “− event” by a downward arrow.

[0022] Referring to FIG. 3, an example of an event image will be described. FIG. 3 shows an output example from the event detection device 11. The image 30 is an image output by an image sensor (frame-based sensor), and details are shown not only for the subject within the imaging range but also for the static background portion. The image 31 is an event image, which is an image generated by taking a plurality of events that occurred during a period equivalent to the period in which the image sensor accumulated light to generate the image 30 as one frame.

[0023] In the image 31 of FIG. 3, the - events correspond to black pixels, and the + events correspond to white pixels. The gray area corresponds to pixels where no event has occurred. The contour of the area where the subject person is moving from the right to the left of the screen corresponds to black or white pixels, and the movement of the person can be recognized by detecting the luminance change. On the other hand, the static background part such as the crosswalk is gray because there is no or little luminance change. Compared with the image 30, the image 31 has a significantly smaller amount of data per predetermined period, and post - processing for tracking or recognizing changes within the scene is easy, so efficient processing is possible. In the example of FIG. 3, a representation method using black pixels, white pixels, and gray areas is shown, but it is not limited to this example, and other colors may be used, or an event image generated by changing pixel values according to the intensity level of the luminance change may be used.

[0024] Next, with reference to FIG. 4, the problems to be solved in the present disclosure will be described. FIGS. 4(A) to (C) show captured images captured by the imaging device 12 of the subject, and are a plurality of images arranged in time series. The moving direction of the subject to be tracked (tracking subject) is from the left to the right of the screen. FIG. 4(A) shows an image in which the subject is reflected on the left side, FIG. 4(B) shows an image in which the subject is reflected in the center, and FIG. 4(C) shows an image in which the subject is reflected on the right side. FIGS. 4(D) to (E) are event images generated by the event detection device 11 corresponding to FIGS. 4(A) to (C), respectively.

[0025] The subject tracking operation starts from the state of FIG. 4(A). In FIG. 4(B), the imaging state is close to saturation (low - contrast state), and there is a possibility that the imaging device 12 may lose sight of the tracking subject. Control is performed to continuously track the subject detected from the event image shown in FIG. 4(E). In FIG. 4(C), it is assumed that the imaging device 12 takes over the tracking of the subject detected by the event detection device 11. At this time, if the event detection device 11 is performing event detection, events will always occur. Therefore, there is a possibility that the processing load and power consumption of the entire imaging system 10 will increase. In the present disclosure, means for solving this problem will be described.

[0026] [First Embodiment] Referring to FIG. 5, the control performed by the imaging system 10 of this embodiment will be described. The following processes are realized according to a predetermined program executed by the system control unit 13. At S501, the system control unit 13 starts the processes related to imaging system control and proceeds to the process of S502.

[0027] At S502, the system control unit 13 performs initialization processing. The system control unit 13 sets imaging conditions for the imaging device 12 according to requests from the data processing unit 15 or the tracking control unit 16. The imaging conditions to be set are determined by the system control unit 13 based on the imaging information from the data processing unit 15 or the tracking subject information by the tracking control unit 16. However, at the start point of imaging system control, since there is no imaging information or tracking subject information, the imaging conditions stored by the system control unit 13 are set. Subsequently, the system control unit 13 sets event detection conditions for the event detection device 11 according to requests from the event detection control unit 14. The event detection conditions to be set are determined by the event detection control unit 14 based on the imaging information acquired from the data processing unit 15 or the tracking subject information acquired from the tracking control unit 16 via the data processing unit 15. However, at the start point of imaging system control, since there is no imaging information or tracking subject information, the event detection conditions stored by the event detection control unit 14 are set. For example, detection conditions associated with the imaging conditions set for the imaging device 12 are set.

[0028] Next to S502, the first process (S503, S504) by the imaging device 12 and the second process (S505, S506) by the event detection device 11 are executed in parallel (parallel processing). At S503, the imaging device 12 performs imaging processing and an imaging image is generated. At the next S504, the imaging device 12 performs subject detection processing on the imaging image generated at S503, and first subject detection information is acquired.

[0029] In S505, the event detection device 11 performs event detection processing, and a framed event image corresponding to the detected event is generated. In the next S506, the event detection device 11 performs subject detection processing on the framed event image generated in S505, and second subject detection information is obtained.

[0030] After S504 or S506, the process proceeds to S507. In the imaging device 12, the process shifts from S504 to S507 for each imaging cycle, and in the event detection device 11, the process shifts from S506 to S507 for each event framing cycle.

[0031] In S507, the tracking control unit 16 performs control including determination processing of the tracking subject. Based on the first subject detection information obtained in S504 and the second subject detection information obtained in S506, a determination is made as to which subject should be the main subject and which subject should be the tracking subject. Also, the tracking subject information, which is the determination result, is stored in the storage area within the tracking control unit 16. Next, the process proceeds to S508.

[0032] In S508, the event detection control unit 14 updates the event detection conditions for the event detection device 11 to detect an event based on the tracking subject information determined in S507. According to the updated event detection conditions, the system control unit 13 sets conditions for the event detection device 11. Details of the determination process of the event detection conditions will be described later. Next, the process proceeds to S509.

[0033] In S509, the system control unit 13 determines whether to end the control of the imaging system 10. For example, the system control unit 13 determines whether a user operation has been performed on an operation member (such as a power on / off switch) equipped in the imaging system 10. When the operation member has not been operated (the power is on), the process proceeds to S503 and S505, and the imaging process by the imaging device 12 and the event detection device 11 continues. Also, when an operation to end the imaging system control has been performed (a power off operation), in S510, the system control unit 13 ends the control of the imaging system 10.

[0034] Next, the subject tracking control shown in S507 of FIG. 5 will be described in detail. As described above, the imaging process by the imaging device 12 and the imaging process by the event detection device 11 are executed in parallel, and after imaging, a subject detection process is performed. The outputs from the imaging device 12 and the event detection device 11 are each passed to the data processing unit 15 asynchronously. The data processing unit 15 performs data processing in accordance with the outputs of the respective devices.

[0035] The tracking control unit 16 acquires the data of the captured image generated by the imaging device 12 and the data of the event image generated by the event detection device 11, which have been processed by the data processing unit 15, and performs a subject detection process and a subject determination process for subject tracking. These processes are performed in accordance with the input timing for the input captured image and event image. Also, the tracking control unit 16 performs a process of holding, in a temporary storage device or the like, data indicating the subject detection result based on the captured image and the event image. The following three cases will be described. (1) When subject detection is performed for both the captured image and the event image. (2) When subject detection is performed based only on the event image. (3) When subject detection is performed based only on the captured image.

[0036] First, (1) will be described. Although it is premised on prioritizing the subject detected by the imaging device 12, for example, a state where subject detection is difficult, such as a low contrast state, is assumed. Therefore, the tracking control unit 16 calculates an index such as the reliability of the detected subject and uses the index to determine which subject to track. For example, when the reliability of subject detection based on the captured image and the reliability of subject detection based on the event image can be obtained, the tracking control unit 16 determines which subject detection result of which image to use according to the reliability of each subject detection. The subject detection result with the higher reliability is selected.

[0037] Subsequently, (2) will be described. In actual operation, the periods of the image signals respectively generated from the imaging device 12 and the event detection device 11 are different. For example, assume that the event detection device 11 has a higher output frequency than the imaging device 12. When the imaging device 12 is in the imaging operation and in the state after the subject detection process of the event detection device 11, the subject detection results of the event image and the subject detection results of the immediately preceding captured image are held in the storage unit. In this case, the tracking control unit 16 compares the reliability of subject detection based on the event image with the reliability of subject detection based on the immediately preceding captured image to determine the tracking subject. When the reliability of subject detection based on the event image is higher than the reliability of subject detection based on the immediately preceding captured image, the subject detected from the event image is selected and determined as the tracking target. Also, when the reliability of subject detection based on the immediately preceding captured image is higher than the reliability of subject detection based on the event image, the subject detected from the captured image is continuously determined as the tracking target.

[0038] Next, (3) will be described. In this case, a pattern opposite to (2) is assumed. In the state after the subject detection process in the imaging device 12, the subject detection result based on the captured image and the subject detection result based on the immediately previous event image are held in the storage unit. In this case, the tracking control unit 16 compares the reliability of the subject detection based on the captured image with a predetermined threshold to determine the tracking subject. When the reliability of the subject detection based on the captured image exceeds the predetermined threshold, the subject detected from the captured image is determined as the tracking target. When the reliability of the subject detection based on the captured image is equal to or lower than the predetermined threshold, the tracking control unit 16 compares the reliability of the subject detection based on the captured image with the reliability of the subject detection based on the held event image. When the reliability of the subject detection based on the captured image is higher than the reliability of the subject detection based on the event image, the subject detected from the captured image is determined as the tracking target. Also, when the reliability of the subject detection based on the event image is higher than the reliability of the subject detection based on the captured image, the subject detected from the event image is determined as the tracking target. Information about the determined tracking target subject (tracking subject information) is held in the temporary storage unit of the tracking control unit 16.

[0039] Next, a process of determining the event detection conditions of the event detection device 11 based on the imaging state of the imaging device 12 and the tracking subject information determined by the tracking control unit 16 will be described. From the tracking subject information determined by the tracking control unit 16, the event detection device 11 sets an area (event detection area) in which to perform event detection. The event detection area in the image is one of the event detection conditions, and by limiting the detection area based on the tracking subject information to reduce the occurrence frequency of events, it is possible to reduce the processing load and power consumption of the entire imaging system 10.

[0040] A process of changing event detection conditions according to the imaging state of the imaging device 12 will be described. The threshold value for event detection is set as a fixed value or a variable value. In the example described with reference to FIG. 2, the threshold value Θ is set to a uniform value. In imaging in a low illuminance state or a saturation state where it is difficult to track a subject, the contrast is often low even for the same subject. In this case, as with the imaging device 12, there is a possibility that the subject cannot be detected from the event image generated by the event detection device 11. Therefore, the event detection device 11 is set to a state where an event is likely to occur, and the event detection conditions are changed so that an event image capable of subject detection can be generated.

[0041] FIG. 6 is a graph for explaining a process in which the event detection control unit 14 changes the event detection threshold according to the contrast of the captured image. The horizontal axis represents the contrast of the captured image, and the vertical axis represents the event detection threshold Θ. In the graph line (a), the event detection threshold Θ is a constant value regardless of the contrast of the captured image. In the graph line (b), as the contrast decreases, the event detection threshold Θ decreases, and as the contrast increases, the event detection threshold Θ increases. When the event detection threshold Θ is decreased, event detection becomes possible even when the change per pixel is small under low contrast in a low illuminance state or a saturation state. Also, under high contrast such as in an appropriate exposure state, since subject detection and subject tracking by the imaging device 12 are functioning sufficiently, event detection conditions that minimize event occurrence are required. Therefore, the event detection threshold Θ is set large to control the event occurrence frequency to decrease.

[0042] As described above, when subject detection is performed from the captured image acquired by the imaging device 12 and subject tracking is possible, the event detection conditions are set so that the event occurrence frequency in the event detection device 11 is reduced. In imaging conditions where subject tracking by the imaging device 12 is difficult, event detection conditions are set in the event detection device 11 such that an event is likely to occur. By doing so, it becomes possible to generate an event image at the necessary timing and continuously perform subject tracking.

[0043] Figs. 7(A) to 7(C) show output examples of the imaging device when the event detection conditions are changed, and Fig. 7(D) shows an output example of the event detection device when the event detection conditions are changed. Similar to Figs. 4(A) to 4(C), Figs. 7(A) to 7(C) show a plurality of imaging images arranged in time series for the subject imaged by the imaging device 12. Fig. 7(D) schematically shows the event image corresponding to Fig. 7(B), and it is an image generated by the event detection device 11. In Fig. 7(D), it is assumed that the imaging images in Fig. 7(B) are a plurality of images in a low contrast state. When the imaging state changes from Fig. 7(A) to (B) and from Fig. 7(B) to (C), the event detection device 11 performs event detection to generate an event image, and subject detection is performed.

[0044] In this embodiment, control is performed to dynamically change the event detection threshold of the event detection device 11 according to the contrast of the imaging image. In addition, there is a method of changing the event detection threshold based on the brightness of the imaging image and the exposure conditions (aperture value, shutter speed, sensor gain, etc.) of the imaging device 12. Further, there is a method of changing the event detection threshold according to the inter-frame difference in luminance of the image region including the tracking subject in the imaging image or the inter-frame difference in luminance of the entire imaging image.

[0045] According to this embodiment, by dynamically changing the event detection conditions based on the subject information recognized from the imaging image, event detection by the event detection device is possible even when the exposure conditions of the imaging device are severe, and the subject can be continuously tracked.

[0046] [Second Embodiment] Next, a second embodiment will be described. In this embodiment, the setting and change of event detection conditions are dynamically performed according to the moving direction and moving speed of the following subject. For example, in the control of event detection conditions, the event detection control unit 14 adaptively sets the size or shape of the event detection area in the captured image. By doing so, it is possible to continuously track the subject even in a situation where subject tracking by the imaging device is difficult. Note that in this embodiment, the description of matters similar to those in the first embodiment will be omitted, and the differences from the first embodiment will be described. The processing of this embodiment is the same as the processing of FIG. 5 described in the first embodiment, but since the processing of S508 (event detection condition update) is different, the content of the processing will be described in detail.

[0047] With reference to FIG. 8, the process of dynamically changing the event detection conditions of the event detection device 11 according to the moving speed of the following subject will be described. FIGS. 8(A) to (C) show examples of captured images generated by the imaging device 12. The moving direction of the following subject is from left to right on the screen. FIGS. 8(A) to (C) show that the moving speed of the subject is increasing. FIGS. 8(D) to (F) are event images corresponding to FIGS. 8(A) to (C) respectively, and are generated by the event detection device 11. Similar to FIGS. 8(A) to (C), the moving speed of the following subject is increasing from FIGS. 8(D) to (F). The rectangular dotted frames 801 to 803 in FIGS. 8(D) to (F) represent the boundaries of the event detection area in the image, and the relationship is "the size of dotted frame 801 < the size of dotted frame 802 < the size of dotted frame 803".

[0048] In the first embodiment, the method of setting the event detection area based on the following subject information determined by the following control unit 16 was described, but the moving speed of the subject was not considered. For example, when the moving speed of the following subject is high, it is necessary to set the event detection area assuming the moving range of the subject.

[0049] Figs. 8(A) and 8(D) show the case where the moving speed of the subject is relatively small. In this case, as shown by the dotted frame 801 in Fig. 8(D), the event detection region can be limited only to the periphery of the tracked subject (the size of the event detection region in the moving direction is small). Also, since the difference between pixels hardly occurs, the frequency of event occurrence becomes low. Therefore, when the number of events is small when generating an event image, there is a possibility that the tracking control unit 16 at the subsequent stage cannot acquire a subject image that can be detected. Thus, when the moving speed of the subject is smaller than the threshold value, the data processing unit 15 performs control to increase the frame period for generating the event image. By doing so, it becomes possible to generate a subject image that can be detected by the tracking control unit 16. On the other hand, when it is not desired to extremely increase the frame period for generating the event image by the event detection device 11, the event detection threshold value may be decreased so that a slight change in pixel output can be detected.

[0050] Figs. 8(C) and 8(F) show the case where the moving speed of the subject is relatively large. In this case, as shown by the dotted frame 803 in Fig. 8(F), not only the periphery of the tracked subject but also an event detection region considering the moving speed is set (the size of the event detection region in the moving direction is large). Although it is also possible to set the entire region as the event detection region from the size and moving direction of the subject image in the captured image, as described above, when the event detection region is enlarged, the frequency of event occurrence increases, and accordingly, the processing load of the entire imaging system 10 may increase. Therefore, it is desirable to set the minimum event detection region that does not lose sight of the tracked subject. Regarding how much event detection region to set according to the moving speed of the subject, for example, there is a method of using the reference table data for setting held by the imaging system 10. Also, there is a method of estimating the next moving position from the moving speed of the subject and setting the region including the moving position as the event detection region.

[0051] When the moving speed of the subject is high, differences between pixels are likely to occur, and the frequency of event occurrence increases. Therefore, when the event detection device 11 generates an event image, if the number of events is extremely large, there is a possibility that the subject image that can be detected by the subsequent tracking control unit 16 cannot be obtained. For example, the subject image may be crushed. When the moving speed of the subject is greater than the threshold value, the data processing unit 15 performs control to shorten the frame period for generating the event image. By doing so, it becomes possible to generate a subject image that can be detected by the tracking control unit 16. On the other hand, there may be cases where it is not desired to extremely shorten the frame period for generating the event image by the event detection device 11 (such as when the processing load becomes high). In that case, the event detection threshold may be increased so that an event can be detected when the amount of change in pixel output is equal to or greater than a predetermined threshold value.

[0052] In this embodiment, an event detection area can be set according to the moving speed (speed and direction) of the tracking subject, and further, the frame period and the event detection period can be determined based on the processing load and power consumption of the imaging system 10. By dynamically setting the event detection conditions of the event detection device according to the moving speed of the tracking subject based on the captured image acquired by the imaging device, it is possible to continuously track the subject. Also, the imaging system 10 can set or change the event detection area according to the moving direction of the tracking subject. Further, instead of the moving speed of the tracking subject, the event detection conditions may be dynamically changed according to the relative moving speed between the imaging system 10 and the subject.

[0053] According to the above embodiment, subject tracking under severe exposure conditions such as a scene with a sudden change in brightness or a low-contrast scene, and tracking control according to the moving speed and moving direction of the subject can be continuously performed. By setting or changing the event detection conditions of the event detection device based on the imaging state of the imaging device, subject tracking control is possible while suppressing the processing load and power consumption of the imaging system.

[0054] [Other Embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and causing one or more processors in a computer of the system or apparatus to read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0055] As described above, the present invention has been described in detail based on its preferred embodiments. However, the present invention is not limited to the above-described embodiments, and various forms within the scope not departing from the gist of the present invention are also included in the technical scope of the present invention. A part of the above-described embodiments may be appropriately combined.

[0056] Embodiments of the present disclosure include the following configurations, methods, and programs. [Configuration 1] Acquisition means for acquiring an output of an event detection device that detects an event from a luminance change of pixels and an output of an imaging device that images a subject at a predetermined frame rate, First control means for controlling the event detection device and the imaging device, Second control means for controlling detection conditions for the event detection device to detect an event, Data processing means for generating image data from the output of the imaging device and the output of the event detection device, Third control means for performing subject detection and tracking control using the image data generated by the data processing means, and the second control means performs control to set detection conditions of the event detection device corresponding to an imaging state of the imaging device A subject tracking device characterized by the above. [Configuration 2] The second control means performs control to set detection conditions of the event detection device by determining brightness or contrast of an imaging image acquired by the imaging device or a subject detection result. The subject tracking device according to Configuration 1, characterized by the above. [Configuration 3] The second control means acquires information on the tracking subject determined by the third control means and performs control to set a detection area for an event in an image generated by the event detection device. The subject tracking device according to Configuration 1 or Configuration 2, characterized in that. [Configuration 4] The second control means performs control to change a detection threshold for an event in the event detection device. The subject tracking device according to any one of Configurations 1 to 3, characterized in that. [Configuration 5] The third control means calculates the reliability of a subject detection result based on a captured image acquired by the imaging device and the reliability of a subject detection result based on an image acquired by the event detection device, and performs tracking control using the subject detection result with the higher reliability. The subject tracking device according to any one of Configurations 1 to 4, characterized in that. [Configuration 6] The second control means performs control to change a detection area for an event in an image acquired by the event detection device according to the moving speed or moving direction of the tracking subject determined by the third control means. The subject tracking device according to any one of Configurations 1 to 5, characterized in that. [Configuration 7] The data processing means changes a framing period for generating a framed image from the output of the event detection device according to the moving speed of the subject detected by the third control means. The subject tracking device according to any one of Configurations 1 to 6, characterized in that. [Configuration 8] The second control means performs control to change the size or shape of a detection area for an event in an image generated by the event detection device by controlling the detection conditions. The subject tracking device according to any one of Configurations 1 to 7, characterized in that. [Configuration 9] The event detection device detects, as the event, a case where a signal based on the intensity of light incident on the imaging element included in the event detection device increases beyond a threshold value or a case where the signal decreases beyond the threshold value. The subject tracking device according to any one of Configurations 1 to 8, characterized by the above. [Configuration 10] The subject tracking device according to any one of Configurations 1 to 9, The event detection device having an asynchronous imaging element, and the imaging device having a synchronous imaging element. An imaging system characterized by the above. [Configuration 11] The second control means performs control to set detection conditions of the event detection device based on the moving speed of the tracking subject or the relative moving speed between the imaging system and the tracking subject. The imaging system according to Configuration 10, characterized by the above. [Method 1] An acquisition step of acquiring an output of an event detection device that detects an event from a luminance change of a pixel and an output of an imaging device that images a subject at a predetermined frame rate, A first control step of controlling the event detection device and the imaging device, A second control step of controlling detection conditions for the event detection device to detect an event, A data processing step of generating image data from the output of the imaging device and the output of the event detection device, and a third control step of performing subject detection and tracking control using the image data generated in the data processing step. In the second control step, control is performed to set detection conditions of the event detection device corresponding to an imaging state of the imaging device. A method for controlling a subject tracking device, characterized by the above. [Program] A program for causing a computer of a subject tracking device to execute each step described in Method 1.

Explanation of Signs

[0057] 10... Imaging system, 11... Event detection device, 12... Imaging device, 13... System control unit, 14... Event detection control unit, 15... Data processing unit, 16... Tracking control unit

Claims

1. An acquisition means for acquiring the output of an event detection device that detects an event from the luminance change of pixels and the output of an imaging device that images a subject at a predetermined frame rate; A first control means for controlling the event detection device and the imaging device; A second control means for controlling the detection conditions for the event detection device to detect an event; Data processing means for generating image data from the output of the imaging device and the output of the event detection device; A third control means for performing subject detection and tracking control using the image data generated by the data processing means, wherein the second control means performs control to set the detection conditions of the event detection device corresponding to the imaging state of the imaging device A subject tracking device characterized by this.

2. The second control means performs control to determine the brightness or contrast of the captured image acquired by the imaging device or the detection result of the subject and set the detection conditions of the event detection device. The subject tracking device according to claim 1, characterized by this.

3. The second control means acquires the information of the tracking subject determined by the third control means and performs control to set the detection area of the event in the image generated by the event detection device. The subject tracking device according to claim 1, characterized by this.

4. The second control means performs control to change the detection threshold of the event in the event detection device. The subject tracking device according to claim 1, characterized by this.

5. The third control means calculates the reliability of the subject detection result based on the captured image acquired by the imaging device and the reliability of the subject detection result based on the image acquired by the event detection device, and performs tracking control using the subject detection result with the higher reliability. The subject tracking device according to claim 1, characterized by this.

6. The second control means performs control to change the detection area of the event in the image acquired by the event detection device according to the moving speed or moving direction of the tracking subject determined by the third control means. The subject tracking device according to claim 1, characterized by this.

7. The data processing means changes the framing period for generating a framed image from the output of the event detection device according to the moving speed of the subject detected by the third control means. The subject tracking device according to claim 1, characterized by this.

8. The second control means performs control to change the size or shape of a detection area of an event in an image generated by the event detection device by controlling the detection conditions. The subject tracking device according to claim 1, characterized in that.

9. The event detection device detects, as the event, a case where a signal based on the intensity of light incident on an image sensor included in the event detection device increases beyond a threshold value or a case where the signal decreases beyond the threshold value. The subject tracking device according to claim 1, characterized in that.

10. The subject tracking device according to any one of claims 1 to 9, The event detection device having a non-synchronous image sensor, And the imaging device having a synchronous image sensor. An imaging system characterized by that.

11. The second control means performs control to set detection conditions of the event detection device based on the moving speed of a tracking subject or the relative moving speed between the imaging system and the tracking subject. The imaging system according to claim 10, characterized in that.

12. An acquisition step of acquiring an output of an event detection device that detects an event from a change in luminance of a pixel and an output of an imaging device that images a subject at a predetermined frame rate, A first control step of controlling the event detection device and the imaging device, A second control step of controlling detection conditions for the event detection device to detect an event, A data processing step of generating image data from the output of the imaging device and the output of the event detection device, And a third control step of performing subject detection and tracking control using the image data generated by the data processing step, In the second control step, control is performed to set detection conditions of the event detection device corresponding to an imaging state of the imaging device. A control method for a subject tracking device, characterized in that.

13. A program for causing a computer of a subject tracking device to execute each step according to claim 12.

Images