Information processing device, information processing method, and program

Abstract

To track a subject when an event-based sensor is used.SOLUTION: An information processing device includes detection means for detecting a tracking target included in an imaging range on the basis of an address event signal indicating the position and time of a pixel at which a change in luminance occurs, estimation for estimating the motion of the detected tracking target, and determination means for determining a control value for changing the imaging range on the basis of the estimated motion of the tracking target.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to subject tracking using an event-based sensor. [Background technology]

[0002] An event-based sensor is known that outputs a change in luminance of each pixel as an address event signal in real time (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-72317 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to track a subject when an event-based sensor is used. [Means for solving the problem]

[0005] The information processing device according to the present invention for solving the above-mentioned problems includes a detection unit that detects a tracking target included in a shooting range based on an address event signal that indicates the position and time of a pixel where a change in luminance has occurred, an estimation unit that estimates the movement of the detected tracking target, and a shooting unit that performs a tracking operation based on the estimated movement of the tracking target. direction a determining means for determining a control value for changing the The determining means determines the control value so that the tracking target is tracked at a speed different from that of the tracking target. do. [Effects of the Invention]

[0006] According to the present invention, when an event-based sensor is used, it is possible to track a subject. [Brief explanation of the drawings]

[0007] [Figure 1]A block diagram showing an example of the hardware configuration of an information processing device. [Figure 2] A diagram showing an example of the configuration of an event-based sensor [Figure 3] A block diagram showing an example of the functional configuration of an information processing device. [Figure 4] Schematic diagram illustrating an example of a method for tracking a subject. [Figure 5] 10 is a flowchart illustrating a process executed by an information processing device. [Figure 6] Schematic diagram illustrating an example of a method for tracking a subject. [Figure 7] 10 is a flowchart illustrating a process executed by an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0008] Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

[0009] <Embodiment 1> Conventionally, synchronous photoelectric conversion elements that capture image data (frames) in synchronization with a synchronization signal such as a vertical synchronization signal have been used in imaging devices. These general synchronous photoelectric conversion elements can only acquire image data at synchronization signal intervals (e.g., 1 / 60 seconds), making it difficult to meet high-speed processing requirements. To address this issue, asynchronous photoelectric conversion elements that detect luminance changes as address events for each pixel address in real time have been proposed. Such photoelectric conversion elements that detect events for each pixel address are called dynamic vision sensors (DVSs, hereafter referred to as event-based sensors). Because imaging devices using event-based sensors can detect subject movement with high sensitivity, they can be used to track detected moving objects.

[0010] However, when tracking a moving object using an image capture device that uses an event-based sensor to detect changes in luminance of the moving object, there is a problem that the object is tracked at the same speed as the moving object, making it difficult for changes in luminance to occur. In this embodiment, an image capture device that uses an event-based sensor controls the shooting range according to the movement of the object, thereby generating a situation in which changes in luminance occur, thereby enabling accurate tracking of the object.

[0011] <Hardware configuration of the imaging device 100: Figure 1> FIG. 1 is a block diagram showing an example of the hardware configuration of an imaging device (information processing device) 100. As shown in FIG.

[0012] The imaging device 100 includes an imaging unit 101, which includes an imaging optical system 1010 and a photoelectric conversion element 1011, a CPU 102, a memory 103, a display unit 104, and an operation unit 105. The photoelectric conversion element 1011 is an event-based sensor that outputs an address event signal in response to received incident light. The event-based sensor detects a change in luminance for each pixel as an event, and the address event signal indicates the pixel position and time at which the luminance change occurred. The imaging optical system 1010 is specifically a light-receiving lens that receives incident light and forms an image on the photoelectric conversion element 1011. The CPU 102 reads and executes an OS and other programs stored in the memory 103, controls each connected component, and performs various processing operations and logical judgments. The processing performed by the CPU 102 includes information processing according to this embodiment. The CPU 102 also controls the focus and aperture of the imaging optical system 1010, the photoelectric conversion element 1011, and other functions. The memory 103 is, for example, a hard disk drive or an external storage device, and stores programs and various data related to the information processing of the embodiment. The display unit 104 is, for example, a display device that displays the calculation results of the imaging device 100 in accordance with instructions from the CPU 102. The display device may be of any type, such as a liquid crystal display device, a projector, or an LED indicator. The operation unit 105 is, for example, a touch panel, a keyboard, a mouse, or a robot controller, and is a user interface that accepts input instructions from a user. The imaging device 100 may have mechanisms other than the hardware configurations listed here, and the function of performing image processing and the function of performing image analysis (tracking) may be provided in separate devices.

[0013] <Photoelectric conversion element 1011: Figure 2> An example of a photoelectric conversion element (event-based sensor) according to this embodiment will be described. The event-based sensor counts the number of incident photons and determines the timing when the counted number of photons exceeds a predetermined threshold. The event-based sensor also measures the time (clock count) required for each pixel until the number of photons reaches or exceeds a first threshold, and detects a change in luminance by comparing the required times. Specifically, when the previously measured required time is T0 and the latest required time is T, if the difference T-T0 is equal to or greater than a second threshold, a negative change in luminance is detected. If the difference T0-T is equal to or greater than the second threshold, a positive change in luminance is detected. If the difference between T and T0 is less than the second threshold, no change in luminance is detected. The second threshold is a value greater than or equal to zero, and is set in accordance with a preset value or other parameters.

[0014] The detailed configuration will be described below. Fig. 2(A) is a diagram showing an example of the configuration of a photoelectric conversion element 1011. The photoelectric conversion element 1011 is composed of a pixel unit 110 and a peripheral circuit 120. The peripheral circuit 120 includes a vertical arbitration circuit 121 and a horizontal readout circuit 122.

[0015] FIG. 2B illustrates an example configuration of each pixel unit constituting an event-based sensor. The pixel unit 110 includes a photoelectric conversion unit 111, a pixel counter 112, a time counter 113, a first decision circuit 114, a memory 115, a comparator 116, a second decision circuit 117, a response circuit 118, and a selection circuit 119. The photoelectric conversion unit 111 includes an avalanche photodiode (SPAD) operating in Geiger mode, and the pixel counter 112 counts the number of photons incident on the photoelectric conversion unit 111. The time counter 113 counts the time during which a photon is incident on the photoelectric conversion unit 111. Using a SPAD to configure an event-based sensor enables detection of luminance changes at the level of a single photon. By detecting luminance changes at the level of a single photon, an address event signal can be acquired even in night vision conditions, such as at night.

[0016] When the number of photons counted by pixel counter 112 reaches a first threshold, first decision circuit 114 stops counting time by time counter 113. Past count values ​​of time counter 113 are stored in memory 115, and comparator 116 is used to determine the difference between the current count value of time counter 113 and the past count value of time counter 113.

[0017] If the difference count value is equal to or greater than the second threshold, the second determination circuit 117 sends a request signal to the vertical arbitration circuit 121 via the response circuit 118. The response circuit 118 receives a response from the vertical arbitration circuit 121 indicating whether or not the output of address event data is permitted. If the difference count value is less than the second threshold, the response circuit 118 does not send a request signal.

[0018] When the response circuit 118 receives a response indicating permission to output, the selection circuit 119 outputs the count value of the time counter circuit 113 to the horizontal output circuit 122. The horizontal output circuit 122 outputs the received count value as an output signal from the photoelectric conversion element 1011 to the detection unit 201.

[0019] Because the differential count value calculated by the comparator 116 corresponds to the reciprocal of the incident frequency of photons, the photoelectric conversion element 1011 according to this embodiment has the function of measuring "changes in the incident frequency of photons," i.e., changes in luminance. Furthermore, using the second determination circuit 117, an address event is output only when the difference in the intervals at which the number of incident photons reaches the first threshold is equal to or greater than the second threshold. That is, the photoelectric conversion element outputs the incident frequency when the difference in the incident frequency is equal to or greater than the second threshold, and does not output the incident frequency when the difference is less than the threshold. This configuration realizes an asynchronous photoelectric conversion element that detects changes in luminance as address events in real time for each pixel address.

[0020] <Variations of photoelectric conversion elements> The above describes a case where a photoelectric conversion element is used that uses a SPAD as the photoelectric conversion unit and measures the time at which a photon is incident to detect changes in the frequency of photon incidence. However, the configuration shown in Figure 2 is not necessary as long as the photoelectric conversion element is an asynchronous type that detects changes in luminance as address events in real time. For example, as described in Patent Document 1, a photoelectric conversion element that detects changes in luminance as voltage changes may be used.

[0021] <Example of functional configuration of imaging device 100: FIG. 3> 3 is a block diagram showing an example of the functional configuration of an imaging device (information processing device) 100. The imaging device 100 has an imaging unit 101 consisting of an imaging optical system 1010 and a photoelectric conversion element 1011, an image acquisition unit 301, a control unit 302, a drive unit 305, a detection unit 303, and a calculation unit 304.

[0022] The imaging unit 101 includes an imaging optical system 1010 and a photoelectric conversion element 1011. Light passing through the imaging optical system 1010 is focused by the solid-state imaging element 103 and converted into an electrical signal, which is then output as an image signal by the image acquisition unit 301. In other words, the image acquisition unit 301 acquires an address event signal indicating the position and time of a pixel where a change in luminance occurred. The imaging unit 101 particularly refers to a lens housing. The imaging optical system 1010 is composed of multiple lenses and holding members, has a mechanical structure for aperture, zoom, and focus control, and may be configured to be controlled by an imaging optical system control unit (not shown). The photoelectric conversion element 1011 detects changes in the luminance value of the subject and outputs a signal only when a change is detected. The image acquisition unit 301 performs correction and development processes on the electrical signal converted by the photoelectric conversion element 1011, integrates the output signal for a certain period of time, converts it into frame data, and outputs an image signal. That is, the image acquisition unit 301 acquires an image (frame data indicating pixels where a luminance change occurred during a predetermined period) indicating the positions of pixels where a luminance change occurred during a predetermined period based on an address event signal indicating the position and time of the pixel where a luminance change occurred. The detection unit 303 detects a tracking target using this image signal. That is, the detection unit 303 detects a tracking target included in the shooting range based on an address event signal indicating the position and time of the pixel where a luminance change occurred. The tracking target may be a subject determined to be a moving object, or may be a subject determined to be a moving object that matches the outline of a tracking target previously stored. Specifically, a moving object that moves by itself, such as a person or a vehicle, may be detected as a subject to be tracked. The detection method used by the detection unit 303 will be described later. When the detection unit 303 detects a tracking target, the calculation unit 304 estimates the motion vector (information indicating the speed and moving direction) of the tracking target. The control unit 302 determines a drive angle for controlling the drive direction of the imaging device 100 based on the estimated motion vector of the tracking target. The control value for changing the shooting range may be determined by the calculation unit 304. The method for calculating the motion vector and the drive angle will be described later. The control unit 302 transmits a control signal to the drive unit 306 based on the drive angle received from the calculation unit 304.In other words, the control unit 302 determines a control value for changing the imaging range based on the movement of the tracking target. Here, the control signal (control value) is a control parameter for changing the imaging range (angle of view) of the imaging device 100, and specifically, is a numerical value of the drive angle or speed. Therefore, the following description will focus on control for changing the position (amount of movement) and attitude (drive direction) of the drive unit in order to change the imaging range (angle of view) of the imaging device. The drive unit 306 has at least one-axis drive mechanism that can change the imaging direction (imaging range) of the imaging device 100 based on the control signal (control value) received from the control unit 302. The drive mechanism of the imaging device is, for example, a mechanism that can rotate within a predetermined range of 0 to 360 degrees around an axis that is a perpendicular line perpendicular to the ground. The drive mechanism is not limited to this example, and may have a PTZ mechanism, or the imaging device itself may be movable.

[0023] <Detection unit 303> First, a method for detecting whether or not a subject is moving using the detection unit 303 will be described in detail.

[0024] First, when pixel output is detected, the motion information is compared to see if the connectivity number is equal to or greater than the first threshold N1. If it is equal to or greater than the first threshold N1, it is determined not to be noise, and the process proceeds to matching with feature point data. Here, the connectivity number refers to the number of pixels with pixel output at the same timestamp (or any period) that are connected (adjacent) on a two-dimensional plane. Using the above method, the detected moving object can be used as a tracking target.

[0025] In addition, among the detected moving objects, those that match the characteristics (e.g., outer shape) of the tracking target in advance may be selected. In this case, the imaging device 100 has a storage unit that stores in advance the characteristics of the tracking target. For the change in the luminance value of the image signal acquired by the image acquisition unit 301 when a moving object is detected, features indicating the outer shape of the subject are extracted from the image signal by existing edge analysis processing or the like, and compared with the tracking target stored in the storage unit by image recognition processing such as machine learning. Those with a similarity degree equal to or higher than the threshold value are recognized as the tracking target. The image used for the analysis is an integral image converted into frame data by the image acquisition unit 301.

[0026] <Driving method> FIG. 4 shows a schematic diagram illustrating a method for determining the driving angle of the driving unit 306 of the imaging device in Embodiment 1. An imaging device using an event-based sensor acquires and outputs a luminance change due to the movement of a subject. Therefore, if the imaging device is driven with the same movement as the subject for tracking, the difference in movement between the imaging device and the tracking target disappears, no luminance change occurs, and the tracking target cannot be acquired. Therefore, in Embodiment 1, referring to FIG. 4, a method of tracking such that at least one of the driving speed or the driving angle of the driving unit 306 is different from that of the tracking target will be described.

[0027] FIG. 4 shows a captured image 200 captured by the imaging device 100 centered on point P. For the captured image 200, an x-axis and a y-axis are defined with point O as the origin. That is, it shows the imaging range of the imaging device 100 in two dimensions of x and y. Here, let the number of pixels in the x-axis direction of the captured image 200 be X, and the number of pixels in the y-axis direction be Y. Point α and point β represent one of the feature points of the tracking target 201 detected by the detection unit 303 at times tα and tβ (tα < tβ), respectively. Point α and point β can be represented as point α(xα, yα) and point β(xβ, yβ) using the number of pixels. That is, FIG. 4 shows the state in which the tracking target 201 has moved from point α to point β during the time tβ - tα while the imaging device 100 captures images centered on point P.

[0028] <Variation of driving method: Tracking in the same direction as the tracking target but at a different speed> First, a method for improving tracking accuracy by driving the driving unit 306 at a driving speed different from the moving speed of the tracking target 201 and causing a change in brightness will be described. If the imaging device 100 captures an image at an angle of view θX in the x-axis direction and an angle of view θY in the y-axis direction, the angles per pixel are θX / X and θY / Y, respectively, using the number of pixels X and Y. Here, the motion vector of the tracking target 201 is

[0029]

number

[0030] can be expressed by equation (1).

[0031]

number

[0032] The angles per pixel are θX / X and θY / Y, so if equation (1) is converted to angles φx and φy, it can be expressed as equation (2).

[0033]

number

[0034] The drive unit 306

[0035]

number

[0036] In order to track at a speed different from that, it is sufficient to drive the object at drive angles θx and θy that can be expressed using Ax and Ay (Ax=Ay≠0 and Ax=Ay≠1) shown in equation (3) between tβ and tα.

[0037]

number

[0038] <Variations in driving method: Driving in a different direction from the target to track it [1]> The control unit 302 may determine a drive angle (control value) so that the image capturing device 100 tracks the tracking target 201 in a direction different from the traveling direction of the tracking target 201. Compared to the above-mentioned method in which the tracking speed is different from that of the tracking target 201 but the traveling direction is the same, the movement of the driving unit 306 and the tracking target 201 is different, which causes more brightness changes and makes it possible to improve tracking accuracy. In equation (3), by setting Ax≠Ay, the tracking target 201 can be driven in a direction different from the traveling direction of the tracking target 201 to be tracked.

[0039] <Variations in driving method: Driving in a different direction from the target to track it [2]> The control unit 302 may determine a control value so as to change the shooting range in a direction different from the moving direction of the tracking target. In the processing method where Ax≠Ay in the above-mentioned equation (3), if the tracking target 201 moves parallel to the x-axis or parallel to the y-axis, it is not possible to change the moving direction and track it. Therefore, it is sufficient to drive the tracking target 201 at drive angles θ'x and θ'y that can be expressed using angles Bx and By (Bx≠0, By≠0) shown in equation (4) between tβ and tα.

[0040]

number

[0041] <Variations in driving methods: How variables are determined> The control unit 302 may determine the amount (control value) by which the drive unit 306 is moved depending on the magnitude (speed) of the velocity of the tracking target. When the velocity of the tracking target 201 is fast, a small difference in the drive angle of the drive unit 306 results in a relatively small change in luminance. Therefore, in order to improve tracking accuracy, it is preferable to increase the drive angle, i.e., to make |Log(A)| greater than 1. Furthermore, when the velocity of the tracking target 201 is slow, leaving the drive angle large makes it easier for the tracking target 201 to become distant, increasing the frequency of losing sight of the tracking target 201. To alleviate this, it is preferable to increase the drive angle, i.e., to make |Log(A)| closer to 1. The image capture device 100 has a table in advance that indicates the relationship between the velocity of the tracking target 201 and the drive angle, and Ax and Ay may be selected from the table, or may be calculated from the ratio of the velocities of the tracking target 201. Bx and By can also be determined in a similar manner.

[0042] <Variations in drive method: moving only one axis> The control unit 302 may determine a control value for a selected control direction among a plurality of control directions. That is, the control unit 302 determines a control value so as to change the shooting range in a specific direction based on the movement of the tracking target. In the above method, mainly the case where the driving unit 306 having a plurality of driving shafts drives the plurality of driving shafts simultaneously has been described, but the calculation unit 304 may select and drive one shaft. In this case, in order to increase the luminance change, it is desirable to select the direction with less movement as the shaft to be driven. For example, as shown in FIG. 4, when the tracking target 201 is moving parallel to the x-axis, there is little movement in the y-axis direction, so it is difficult for a luminance change to occur. In this case, the y-axis direction is selected to drive the driving unit 306. That is, when xβ - xα < yβ - yα, it is selected to drive the x-axis, and when xβ - xα > yβ - yα, it is selected to drive the y-axis. When xβ - xα = yβ - yα, either axis may be selected, but it is preferable to select the one with a wider viewing angle (the x-axis in FIG. 4) where the frequency of losing sight of the tracking target 201 is low and drive it. Also, the driving angle at this time is arbitrary, but in order to reduce the frequency of losing sight of the tracking target 201, it is desirable to make the driving angle smaller when selecting the y-axis with a narrower viewing angle than when selecting the x-axis with a wider viewing angle.

[0043] <Method for not losing the tracking target out of the frame> Based on the movement of the tracking target, when the tracking target moves outside a predetermined range in the current shooting range, the control unit 302 determines a control value so as to change the shooting range so that the tracking target is reflected. In Embodiment 1, since the imaging device 100 is driven at a speed different from that of the tracking target, after a certain period of time, the tracking target goes out of the viewing angle of the imaging device 100 and tracking becomes impossible. Therefore, a threshold value is provided inside the viewing angle that can be obtained by the imaging device 100, and when the feature point of the tracking target 201 exceeds the threshold value at the viewing angle end on the traveling direction side represented by Equation (1), it is desirable to change the shooting direction so that the feature point appears at the viewing angle end on the opposite side of the traveling direction of the tracking target 201. This will be described using FIG. 4(b). For simplicity, it is assumed that the tracking target 201 is moving parallel to the x-axis.

[0044] In the captured image 200, the dashed line 210 is set as the threshold value that defines the corner of the shooting angle. The threshold value 210 can be defined by the number of pixels, and in FIG. 4(b), it is set as Cx and Cy. The captured image 211 shows the captured image after driving the imaging device 100 in the direction of the driving direction 212 so as not to frame out. After driving, the coordinates of the point γ indicating one of the feature points of the tracking target 201 are set as (xγ, yγ) for the threshold value, and when the tracking target 201 moves such that xγ>X - Cx, the driving angle θ’’x of the driving unit 306 can be expressed by Equation (5).

[0045]

Number

[0046] When the tracking target 201 moves such that xγ<Cx, it may be driven as shown in Equation (6).

[0047]

Number

[0048] When the speed of the tracking target 201 is high, it is preferable to increase the threshold value in order to make it difficult to lose sight.

[0049] <Predict the movement of the tracking target and perform tracking> Based on the movement of the tracking target, the control unit 302 determines the control value so that the tracking target is reflected in the next frame image. While determining the driving angle and driving the driving unit 306, it takes 2(tβ - tα) time, and since the tracking target 201 moves after the driving angle is determined, it is desirable to predict the movement of the tracking target 201 and determine the driving angle of the driving unit 306. Since it can be considered that the movement amount of the tracking target 201 is doubled, it is advisable to substitute 2φx and 2φy into φx and φy in (3) and (4) above, respectively. That is, it is expressed by the following Equations (7) and (8).

[0050]

Number

[0051] <Flowchart> Fig. 5 shows an example of a flowchart for realizing embodiment 1. The processing shown in the flowchart in Fig. 5 is executed by CPU 101 in Fig. 1, which is a computer, in accordance with a computer program stored in memory 103. In the following description, each process (step) is represented by adding an S to the beginning, and the notation of the process (step) is omitted.

[0052] First, in S501, the detection unit 303 detects the tracking target from the image acquired by the image acquisition unit 301. Repeat (S501) until the tracking target is detected, and if the tracking target is detected, proceed to (S502).

[0053] In S502, if a tracking target is detected, the calculation unit 304 estimates the motion vector of the tracking target. The calculation unit 304 estimates the motion of the tracking target from the history of the tracking target's positions up to now.

[0054] In S503, the control unit 302 determines, from the estimated motion vector, a control value for the driving unit 306. The control value is determined using the method described above.

[0055] In S504, the control unit 302 transmits the determined control value to the driving unit 306 as a driving signal, and the driving unit 306 drives in accordance with the control signal.

[0056] In S505, the detection unit 303 checks the position of the tracking target within the angle of view after driving the drive unit 306. If it is determined that the tracking target is not at the edge of the angle of view in the motion vector direction, the process returns to S501. If it is determined that the tracking target is at the edge of the angle of view in the motion vector direction, the process proceeds to S506.

[0057] In S506, if it is determined that the tracking target is at the edge of the angle of view in the direction of the motion vector, the control unit 302 determines a drive angle so that the subject is captured at the edge of the angle of view in the opposite direction to the motion vector. This drive angle is sent to the control unit 302, and the control unit 302 drives the drive unit 306 based on that drive angle. After that, the process returns to (S506) and repeats the same process thereafter.

[0058] As a result, it is possible to drive the driving unit 306 at a speed different from that of the tracking target, thereby improving the tracking accuracy. In other words, it is possible to improve the tracking accuracy in an imaging device using an event-based sensor.

[0059] <Embodiment 2> In the first embodiment, a tracking method was described in which at least one of the magnitude or direction of the velocity was changed. In the second embodiment, a tracking method is described in which the driving unit is constantly shaken. Shaking refers to increasing the driving frequency of the driving unit 306 compared to the first embodiment. In other words, the control unit 302 determines a control value that changes the shooting range by a predetermined amplitude in the direction of travel of the tracking target. In particular, it refers to driving at a frequency greater than that required to create a single image. Shaking makes it possible to cause brightness changes more frequently than the method of the first embodiment, which is expected to further improve the tracking accuracy of the tracking target.

[0060] 6 is a schematic diagram showing a tracking method in the second embodiment of the present invention. For simplicity, the case where the tracking target 201 moves parallel to the x-axis will be described. The motion vector of the tracking target 201

[0061]

number

[0062] 4 until the calculation is completed, but the dotted line 400 in FIG. 6(a) indicates the drive unit 306.

[0063]

number

[0064] 6(b) and 6(c) show the directions in which the driving unit 306 is shaken. First, as shown in FIG. 6(b), the driving unit 306 is rotated in the direction of travel of the tracking target 201. One way to track an object is to shake it back and forth.

[0065]

number

[0066] Equations (9) and (10) are repeated between tβ and tα using D (D≠0) and time t.

[0067]

number

[0068] In order to increase the difference in movement and to produce a large change in luminance, it is preferable that D in equation (9) for driving in the traveling direction be made larger than that in equation (10).

[0069] Next, as shown in Figure 6(c), there is a method of tracking the target 201 by shaking the driver 306 in a direction perpendicular to the direction of travel of the target 201. Since the target 201 is tracked by shaking it in a direction in which it is not moving, the luminance change is more pronounced, and tracking accuracy can be improved. The drive angle of the driver 306 can be expressed by equation (11) using amplitude E1 and variable E2.

[0070]

number

[0071] It is desirable to set the amplitude E1 so that it does not exceed the threshold shown in Figure 6(b). Also, it is desirable to set the variable E2 as large as possible, specifically, to more than twice the value of 1 / (tβ-tα). The same applies when the parallel direction is not the x direction.

[0072] FIG. 7 shows an example of a flowchart for implementing the second embodiment. The process shown in the flowchart in FIG. 7 is executed by the CPU 101 of FIG. 1, which is a computer, in accordance with a computer program stored in the memory 103. In the following description, each process (step) is denoted by prefixing it with "S," and the process (step) is omitted. Unlike the first embodiment, because there is no risk of the camera going out of frame, the flow can be executed by moving from (S701) to (S704) and repeating this process. In S701, similar to S501, the detection unit 303 detects the tracking target from the image acquired by the image acquisition unit 301, repeating (S701) until the tracking target is detected. If the tracking target is detected, the process proceeds to (S702). In S702, similar to S502, if the tracking target is detected, the calculation unit 304 estimates the motion vector of the tracking target. In S703, the control unit 302 determines a control value for the drive unit 306 from the estimated motion vector. Here, the control value is determined so as to vibrate the shooting range at a predetermined amplitude for a certain period of time. In S704, similarly to S504, the control unit 302 transmits the determined control value as a drive signal to the drive unit 306, and the drive unit 306 drives in accordance with the control signal. As described above, by performing tracking while shaking the drive unit 306, it is possible to improve the tracking accuracy.

[0073] <Embodiment 3> In the third embodiment, a solution method for when the tracking target 201 is lost will be described.

[0074] If the tracking target 201 goes out of frame, it is highly likely that the tracking target 201 is still in the direction of travel, so it is desirable to predict the direction and control the driver 306. For example, if the driver 306 was driving at a slower speed than the tracking target 201, the driver 306 is driven at a faster speed in the direction of travel than the tracking target 201. If the driver 306 was driving at a faster speed than the tracking target 201, it is advisable to drive the driver at a slower speed in the direction of travel or drive the driver in the opposite direction to the direction of travel. With this method, even if the tracking target 201 goes out of frame, the tracking target can be quickly captured, improving tracking accuracy.

[0075] When the tracking target 201 stops, the luminance change stops and the tracking target is lost. When the detection unit 303 detects that the tracking target has stopped, it is desirable to control the driving unit 306 so that it shakes.

[0076] The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a data communication network or various storage media. Then, a computer (or CPU, MPU, etc.) of the system or device reads and executes the programs. The programs may also be provided by recording them on a computer-readable storage medium. [Explanation of symbols]

[0077] 100 Imaging device 101 Imaging unit 1010 Imaging optical system 1011 Photoelectric conversion element 301 Image Acquisition Unit 302 Control Unit 306 Drive Unit 303 Detection unit 304 Arithmetic section

Claims

1. a detection means for detecting a tracking target included in a photographing range based on an address event signal indicating the position and time of a pixel where a change in luminance has occurred; an estimation means for estimating the movement of the detected tracking target; a determination means for determining a control value for changing the imaging direction based on the estimated movement of the tracking target, The information processing device is characterized in that the determination means determines the control value so that the target is tracked at a speed different from that of the target.

2. A detection means for detecting a tracking target included in a shooting range based on an address event signal indicating the position and time of a pixel where a change in luminance has occurred; an estimation means for estimating the movement of the detected tracking target; a determination means for determining a control value for changing the imaging direction based on the estimated movement of the tracking target, The information processing device is characterized in that the determination means determines the control value such that the shooting direction is changed to a direction different from a traveling direction of the tracking target.

3. 3. The information processing device according to claim 1, wherein the address event signal is output by a sensor that detects a change in luminance per unit time for each pixel, and when no change in luminance is detected, the address event signal is not output or indicates that there is no change in luminance.

4. The method further includes acquiring means for acquiring frame data indicating pixels in which a luminance change occurs during a predetermined period based on the address event signal, The information processing device according to claim 1 , wherein the detection means detects the tracking target when a number of connections of pixels in which a change in luminance occurs in the frame data is equal to or greater than a threshold value.

5. Further comprising a storage means for storing features indicative of the tracking target, 5. The information processing device according to claim 1, wherein the detection means detects the tracking target by comparing the feature acquired based on the address event signal with the stored feature.

6. The information processing apparatus according to claim 1 , wherein the determining means determines the control value so that the target object is tracked at a speed that is faster or slower than the speed of the target object.

7. The information processing device according to claim 1 , wherein the determining means determines the magnitude of the control value in accordance with the magnitude of the speed of the tracking target.

8. The information processing device according to claim 1 , wherein the determining means determines the control value based on the movement of the tracking target so as to change the shooting range in a specific direction.

9. 8. The information processing device according to claim 1, wherein the determining means determines the control value based on the movement of the tracking target so that, when the tracking target moves outside a predetermined range in the current shooting range, the shooting range is changed so that the tracking target is captured.

10. 5. The information processing device according to claim 4, wherein the determining means determines the control value based on the movement of the target to be tracked so that the target to be tracked appears in frame data generated after the frame data.

11. The information processing device according to claim 1 , wherein the determining means determines the control value such that the imaging range is changed by a predetermined amplitude in the traveling direction of the tracking target.

12. The information processing device according to claim 1 , wherein the determining means determines the control value so as to change the shooting range at a faster speed when the tracking target is no longer detected.

13. a detection step of detecting a tracking target included in a photographing range based on an address event signal indicating the position and time of a pixel where a change in luminance has occurred; an estimation step of estimating the movement of the detected tracking target; a determination step of determining a control value for changing the imaging direction based on the estimated movement of the tracking target, In the determining step, the control value is determined so that the tracking target is tracked at a speed different from that of the tracking target.

14. A detection step of detecting a tracking target included in a shooting range based on an address event signal indicating the position and time of a pixel where a change in luminance has occurred; an estimation step of estimating the movement of the detected tracking target; a determination step of determining a control value for changing the imaging direction based on the estimated movement of the tracking target, The information processing method, wherein the determining step determines the control value such that the shooting direction is changed to a direction different from a traveling direction of the tracking target.

15. A program for causing a computer to function as each of the means included in the information processing device according to any one of claims 1 to 12.

Images