Information processing device, information processing method, and program

Abstract

To easily adjust the angle of view in an imaging device using an event-based sensor.SOLUTION: An information processing device includes control means for executing a predetermined operation for operating an imaging device in at least one direction, and detection means for detecting a subject from an address event signal indicating the position and time of a pixel at which a change in luminance occurs according to the operation of the imaging device. When executing a first operation that operates the imaging device in a predetermined direction, the control means executes a second operation that operates the imaging device in a direction different from the predetermined direction.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to imaging using event-based sensors. [Background technology]

[0002] An event-based sensor is known that outputs a change in luminance of each pixel as an address event signal in real time (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-134271 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to easily adjust the angle of view in an imaging device using an event-based sensor. [Means for solving the problem]

[0005] The information processing device of the present invention, which solves the above problem, comprises a control means for executing a predetermined operation to move an imaging device in at least one direction, and a generation means for generating an image from an address event signal indicating the position and time of a pixel where a change in brightness occurred in accordance with the operation of the imaging device, wherein the control means, when executing a first operation to move the imaging device in a predetermined direction, executes a second operation to vibrate the imaging device in a direction different from the predetermined direction. [Effects of the Invention]

[0006] According to the present invention, the angle of view can be easily adjusted in an imaging device using an event-based sensor. [Brief explanation of the drawings]

[0007] [Figure 1] A block diagram showing an example of the hardware configuration of an information processing device. [Figure 2] A diagram showing an example of the configuration of an event-based sensor [Figure 3] A block diagram showing an example of the functional configuration of an information processing device. [Figure 4] Schematic diagram illustrating an example of an imaging method [Figure 5] FIG. 10 is a schematic diagram illustrating an example of a captured image. [Figure 6] 10 is a flowchart illustrating a process executed by an information processing device. [Figure 7] FIG. 10 is a schematic diagram illustrating an example of a captured image. [Figure 8] 10 is a flowchart illustrating a process executed by an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

[0009] Conventionally, synchronous photoelectric conversion elements that capture image data (frames) in synchronization with synchronization signals such as vertical synchronization signals have been used in imaging devices. These general synchronous photoelectric conversion elements can only acquire image data at synchronization signal intervals (e.g., 1 / 60 seconds), making it difficult to meet high-speed processing requirements. Therefore, an asynchronous photoelectric conversion element called a Dynamic Vision Sensor (DVS, hereafter referred to as an event-based sensor) has been proposed, which detects luminance changes for each pixel address in real time as address events.

[0010] However, imaging devices using DVS have the problem that when a subject stops moving, the brightness no longer changes and the contour of the subject cannot be detected. Therefore, the object of this embodiment is to adjust the angle of view while detecting the contour of the subject, regardless of the subject's movement state and the camera's driving state, in an imaging device using an event-based sensor.

[0011] <Hardware configuration of the imaging device 100: Figure 1> FIG. 1 is a block diagram showing an example of the hardware configuration of an imaging device (information processing device) 100. In FIG. 1, the imaging device 100 includes an imaging unit 101, which is composed of an imaging optical system 1010 and a photoelectric conversion element 1011, a CPU 102, a memory 103, a display unit 104, and an operation unit 105. The imaging optical system 1010 is specifically a light-receiving lens that receives incident light and forms an image on the photoelectric conversion element 1011. The imaging optical system 1010 may include, but is not limited to, a zoom lens, a focus lens, a blur correction lens, an aperture, a shutter, an optical low-pass filter, an iR cut filter, and other components. The photoelectric conversion element 1011 is an event-based sensor that outputs an address event signal corresponding to the received incident light. The event-based sensor detects a change in luminance for each pixel as an event, and the address event signal indicates the position and time of the pixel where the change in luminance occurred. The photoelectric conversion element 1011 includes a photodiode that converts light into an electrical signal, as well as a comparator for detecting the change in luminance. The luminance changes detected in each pixel are read out using a row drive circuit that drives each row to request the transmission of event detection, and a column drive circuit that sequentially sends the event detection results detected in each column to the image generation unit 302. The CPU 102 reads and executes an OS and other programs stored in the memory 103, controls each connected component, and performs various processing operations and logical judgments. The processing performed by the CPU 102 includes information processing according to this embodiment. The CPU 102 also controls the focus drive and aperture drive of the imaging optical system 1010, the drive of the photoelectric conversion element 1011, and the like. The memory 103 is, for example, a hard disk drive or an external storage device, and stores programs and various data related to the information processing according to this embodiment. The display unit 104 is, for example, a display device that displays the calculation results of the imaging device 100 according to instructions from the CPU 102. The display device may be any type, such as a liquid crystal display device, a projector, or an LED indicator. The operation unit 105 is, for example, a touch panel, a keyboard, a mouse, or a robot controller, and is a user interface that accepts input instructions from the user.Note that the imaging device 100 may have a mechanism other than the hardware configuration described here, and the function of performing image processing and the function of performing analysis processing (tracking) on ​​images may be provided in separate devices.

[0012] <Photoelectric conversion element 1011: Figure 2> An example of a photoelectric conversion element (event-based sensor) according to this embodiment will be described. The event-based sensor counts the number of incident photons and determines the timing when the counted number of photons exceeds a predetermined threshold. The event-based sensor also measures the time (clock count) required for each pixel until the number of photons reaches or exceeds a first threshold, and detects a change in luminance by comparing the required times. Specifically, when the previously measured required time is T0 and the latest required time is T, if the difference T-T0 is equal to or greater than a second threshold, a negative change in luminance is detected. If the difference T0-T is equal to or greater than the second threshold, a positive change in luminance is detected. If the difference between T and T0 is less than the second threshold, no change in luminance is detected. The second threshold is a value greater than or equal to zero, and is set in accordance with a preset value or other parameters.

[0013] 2a is a diagram showing an example of the configuration of a photoelectric conversion element 1011. The photoelectric conversion element 1011 is made up of a pixel unit 110 and a peripheral circuit 120. The peripheral circuit 120 includes a vertical arbitration circuit 121 and a horizontal readout circuit 122.

[0014] FIG. 2b illustrates an example configuration of each pixel unit constituting an event-based sensor. The pixel unit 110 includes a photoelectric conversion unit 111, a pixel counter 112, a time counter 113, a first decision circuit 114, a memory 115, a comparator 116, a second decision circuit 117, a response circuit 118, and a selection circuit 119. The photoelectric conversion unit 111 includes an avalanche photodiode (SPAD) operating in Geiger mode, and the pixel counter 112 counts the number of photons incident on the photoelectric conversion unit 111. The time counter 113 counts the time it takes for a photon to enter the photoelectric conversion unit 111. Using a SPAD to configure an event-based sensor enables detection of luminance changes at the level of a single photon. By detecting luminance changes at the level of a single photon, an address event signal can be acquired even in night vision conditions, such as at night.

[0015] When the number of photons counted by pixel counter 112 reaches a first threshold, first decision circuit 114 stops counting time by time counter 113. Past count values ​​of time counter 113 are stored in memory 115, and comparator 116 is used to determine the difference between the current count value of time counter 113 and the past count value of time counter 113.

[0016] If the difference count value is equal to or greater than the second threshold, the second determination circuit 117 sends a request signal to the vertical arbitration circuit 121 via the response circuit 118. The response circuit 118 receives a response from the vertical arbitration circuit 121 indicating whether or not the output of address event data is permitted. If the difference count value is less than the second threshold, the response circuit 118 does not send a request signal.

[0017] When the response circuit 118 receives a response indicating permission to output, the selection circuit 119 outputs the count value of the time counter circuit 113 to the horizontal output circuit 122. The horizontal output circuit 122 outputs the received count value as an output signal from the photoelectric conversion element 1011 to the detection unit 201.

[0018] Because the differential count value calculated by the comparator 116 corresponds to the reciprocal of the incident frequency of photons, the photoelectric conversion element 1011 according to this embodiment has the function of measuring "changes in the incident frequency of photons," i.e., changes in luminance. Furthermore, using the second determination circuit 117, an address event is output only when the difference in the intervals at which the number of incident photons reaches the first threshold is equal to or greater than the second threshold. That is, the photoelectric conversion element outputs the incident frequency when the difference in the incident frequency is equal to or greater than the second threshold, and does not output the incident frequency when the difference is less than the threshold. This configuration realizes an asynchronous photoelectric conversion element that detects changes in luminance as address events in real time for each pixel address.

[0019] <Variations of photoelectric conversion elements> The above describes a case where a photoelectric conversion element is used that uses a SPAD as the photoelectric conversion unit and measures the time at which a photon is incident to detect changes in the frequency of photon incidence. However, the configuration shown in Figure 2 is not necessary as long as the photoelectric conversion element is an asynchronous type that detects changes in luminance as address events in real time. For example, as described in Patent Document 1, a photoelectric conversion element that detects changes in luminance as voltage changes may be used.

[0020] <Example of functional configuration of imaging device 100: FIG. 3> 3 is a block diagram showing an example of the functional configuration of the imaging device (information processing device) 100. The imaging device 100 includes an imaging unit 301 including an imaging optical system 1010 and a photoelectric conversion element 1011, an image generation unit 302, a drive unit 303, a control unit 304, and an output unit 305.

[0021] The imaging unit 301 acquires an address event signal indicating the position and time of a pixel where a change in luminance occurred. The image generation unit 302 maps the temporal luminance change information obtained from the photoelectric conversion element and creates an image. By moving the imaging device itself in the PT direction or changing the zoom magnification, luminance changes occur in the outline of a stationary subject, allowing the stationary subject to be captured. On the other hand, when the imaging device is operating, the image generation unit 302 can cancel out any unnecessary shooting range caused by the operation of the imaging device through image stabilization processing. In other words, the image generation unit 302 generates an image from the address event signal indicating the position and time of a pixel where a change in luminance occurred, depending on the operation of the imaging device.

[0022] The detection unit 303 detects the subject from an address event signal that indicates the position and time of a pixel where a change in luminance has occurred.

[0023] The driving unit 300 has at least one-axis driving mechanism that can change the shooting direction of the imaging device 100. Here, there is a pan / tilt control method, and for example, when driving the imaging device in the pan direction, the outer shape of the subject can be captured more clearly by vibrating it in the tilt direction.

[0024] The control unit 304 executes a predetermined operation to move the imaging device in at least one direction. Furthermore, when the control unit 304 is in a detection mode for adjusting the imaging range of the imaging device, in addition to a first operation to move the imaging device in a predetermined direction (pan or tilt), the control unit 304 executes a second operation to vibrate the imaging device in a direction different from the predetermined direction. Specifically, the control unit 304 controls the pan direction or tilt direction. The control unit 304 also has the function of controlling each function and data exchange in the imaging device 100 and controlling overall communication such as parameter setting. The output unit 305 outputs the generated image to an external device, etc. For example, the image is stored or displayed by outputting it to a PC or other media (e.g., a hard disk, SD card, USB memory, etc.). Note that the imaging device may include various other components in addition to those described above.

[0025] <Embodiment 1> <Drive method> FIG. 4 is a schematic diagram illustrating a typical imaging method using an event-based sensor and the advantages of the first embodiment over conventional methods. In images 202-1 and 202-2, areas with higher brightness than the previous frame are shown in white, areas with lower brightness are shown in black, and areas with no change in brightness are shown in diagonal lines. Images 201-1, 201-2, and 201-3 are shown, as are images 202-1, 202-2, and 202-3, tree-like object 1 is shown in 203-1, 203-2, and 203-3, and car-like objects 204-1, 204-2, and 204-3. Images 205, 206, 207, and 208 are shown with electronic image stabilization. While this embodiment describes an outdoor surveillance application, this method is also effective for indoor surveillance where there is little change in light intensity. In this embodiment, each object is assumed to be stationary.

[0026] 4A shows the movement and image of the image capture device 201-1 when the image capture device is stationary. In an image capture device using an event-based sensor, under a constant amount of light, the luminance does not change unless the relative position between the image capture device and the object changes, except when the luminance of the image capture device itself changes. This means that the object cannot be detected. Therefore, as shown by the dashed lines within 202-1, even if objects 203-1 and 204-1 are present within 202-1, the edges of the objects cannot be detected. In this case, in the first embodiment, the image capture device is driven in multiple directions to detect the edges of the objects.

[0027] 4(b) shows the movement of the imaging device 201-2 and the image when the imaging device 201-2 is driven in the main imaging angle of view adjustment direction. As the imaging device 201-2 drives, the relative positions of the imaging device 201-2 and the objects 203-2 and 204-2 change, and the brightness changes with respect to the drive direction. Therefore, the edges of the objects corresponding to the drive direction of the imaging device 201-2 are reflected in the image 202-2. However, simply detecting the edges in the drive direction as in FIG. 4(b) results in low visibility of the objects.

[0028] FIG. 4(c) shows the movement and image when the imaging device is driven (first operation) to adjust the main imaging angle of view and then driven (second operation) perpendicularly to the main imaging angle of view. In FIG. 4(c), imaging device 201-3 is driven in the pan direction to adjust the main imaging angle of view, and imaging device 201-2 is driven in the tilt direction perpendicular to the pan direction. This enables edge detection in multiple directions for subjects 203-3 and 204-3 in image 202-3, improving visibility. Hereinafter, driving the imaging device to adjust the main imaging angle of view will be referred to as driving (first operation), and driving the imaging device for the purpose of subject edge detection will be referred to as vibration (second operation). The vibration direction does not need to be perpendicular to the drive direction, but it is desirable that the vibration direction be perpendicular to the drive direction because the larger the angle between the drive direction and the vibration direction, the wider the range in which the subject's edges can be detected.

[0029] Figure 4(d) shows an image obtained by performing electronic stabilization processing when the imaging device is driven in a direction perpendicular to the pan direction in addition to the driving for adjusting the shooting angle of view. As with Figure 4(c), the image is also vibrated in the tilt direction, assuming driving in the pan direction. Image 205 shows the image obtained when the imaging device is vibrated, but the imaging device is positioned at the center of the vibration. Vibrating the imaging device not only makes it possible to detect subject edges in the tilt direction, but also causes blurring of the captured image in the tilt direction. To solve this problem, blurring is suppressed when the imaging device is vibrated. Therefore, as shown in Figure 4(d), electronic stabilization processing is performed, in which the common area between image 206 at the top end of the vibration and image 207 at the bottom end of the vibration is used as a crop area 208, and the captured image is displayed on the user interface. To simplify the calculations of the electronic stabilization processing, it is desirable to define the vibration direction of the imaging device at a fixed angle relative to the driving direction and to maintain a constant vibration period.

[0030] FIG. 5(a) shows the positional relationship of the subject before and after an imaging frame, and FIG. 5(b) shows an image captured by the imaging device 100 under the conditions of FIG. 5(a). 31 indicates the edge of the subject in the frame before driving, 32 indicates the edge of the subject after driving, and 33 indicates the amount of movement of the subject due to driving. 34 indicates an image captured by the imaging device 100 under the conditions of FIG. 5(a). 35 indicates an area where the brightness is lower than in the previous frame, and 36 indicates an area where the brightness is higher than in the previous frame. If the driving speed is too fast for the frame rate of the imaging device, as shown in 34 and 35, the number of pixels where brightness changes compared to the previous frame increases, narrowing the distance between the subject edges and reducing visibility. Therefore, it is desirable to set an upper limit on the driving speed so as not to reduce visibility. When there are at least two or more subjects within the imaging range that serve as the basis for adjusting the angle of view, the upper limit of the driving speed of the imaging device is calculated based on the distance between the subjects. As will be described later, in the first embodiment, it is assumed that the angle of view adjustment is performed from the wide end, so the maximum drive speed is also determined in the wide end state. The maximum drive speed at the wide end depends on the frame rate of the imaging device and the distance to each subject, so it is desirable to adjust it while viewing the displayed image. If the camera is driven at the maximum drive speed at the wide end after zooming, the amount of movement of the subject due to drive within the imaging area will increase by the amount of zooming. Therefore, it is desirable to calculate the maximum drive speed according to the zoom magnification, using the maximum drive speed at the wide end as a reference. The maximum drive speed at the wide end is set to ω L [rad / s], zoom magnification x [%], the maximum drive speed ω at zoom magnification x x can be calculated using Equation 01.

[0031] ω x =ω L / x···Formula 01 Finally, behavior during zooming will be described. During optical zooming, edges can be detected in various directions on the subject, so there is little need to vibrate the imaging device. The imaging device is not vibrated during optical zooming. This is because vibration is not required to perform image stabilization processing to improve visibility, thereby reducing the processing load. In this embodiment, in order to perform angle of view adjustment, it is desirable to set the initial zoom position to the wide side. Thereafter, the process of determining the center position of the angle of view in detection mode and the process of zooming so that the shooting area is the desired size are repeated to adjust the angle of view to the desired size. By performing this process, the angle of view can be adjusted with the shortest processing time.

[0032] <Flowchart> Fig. 6 shows a flowchart for adjusting the angle of view when installing an imaging device 100 using an event-based sensor. In the following explanation, each process (step) is denoted by prefixing it with an S, and the process (step) is notated in detail. The processing shown in the flowchart in Fig. 6 is executed by the computer CPU 102 in Fig. 1, which is a computer, in accordance with a computer program stored in memory 103. The flowchart consists of the following steps S401 to S408.

[0033] S401: The control unit 304 determines whether or not the current mode is a detection mode for detecting the edges of a subject. If it is a detection mode, the process proceeds to S402. If not, the process proceeds to S406. The detection mode refers to a mode in which the camera is driven to detect the edges of a subject, separate from the drive for changing the shooting angle of view. The determination of whether or not the current mode is a detection mode is handled as an interrupt from S402 onward, and if the current mode is no longer a detection mode, the process proceeds to S408.

[0034] S402: The control unit 304 determines whether or not driving (first operation) for changing the photographing angle of view is being performed, and if driving is being performed, the process proceeds to S403. If driving is not being performed, the process proceeds to S405.

[0035] S403: The control unit 304 vibrates the imaging device in a direction different from the drive direction (second operation). As shown in Figures 5(b) and 5(c), subject edges can be easily detected in the drive direction without vibration. To clearly identify subject edges, it is desirable to be able to detect edges in all directions. The larger the angle between the drive direction and the vibration direction, the wider the range of subject edges that can be detected. Therefore, it is desirable for the vibration direction to be perpendicular to the drive direction. The drive direction can be determined from information input from the output unit 305, and the control unit 304 calculates the direction perpendicular to the PT drive direction and issues a control command to the drive unit 303 to vibrate. The vibration frequency is set to be faster than the frame rate of the imaging device 100 when no correction is performed. The vibration direction may be calculated based on the drive direction, or the vibration direction relative to the drive direction may be stored in advance in a table or the like.

[0036] S404: The generation unit 303 performs electronic image stabilization processing on the captured image blurred by the vibration of S403. That is, when the imaging device is vibrating, the generation unit 303 generates an image corrected by a predetermined width. As shown in FIG. 5(d), the control unit 304 calculates and displays the area common to the captured image at the upper and lower ends of the vibration. When the vibration stops, this step ends and the process proceeds to S401.

[0037] S405: The control unit 304 determines whether or not zoom driving is being performed. If zoom driving is being performed, the process proceeds to S406. If zoom driving is not being performed, the process proceeds to S407.

[0038] S406: The control unit 304 does not vibrate and proceeds to S401.

[0039] S407: The control unit 304 vibrates in multiple directions and proceeds to S404. The vibration directions may be set in advance or may move randomly, but it is desirable to set the directions so that edges in various directions can be detected. The vibration frequency is as described in S403.

[0040] S408: The control unit 304 ends the process without vibrating.

[0041] <Embodiment 2> In the first embodiment, a method for driving or vibrating the driving unit 303 was described, assuming that the subject is stationary. In the second embodiment, it is assumed that the subject may be moving. When the subject is moving, the output of the photoelectric conversion element changes due to a change in the luminance of the subject, making it possible to detect the contour of the subject without driving or vibrating. On the other hand, it is conceivable that the presence of the subject may make it difficult to adjust the angle of view. In the second embodiment, a method for detecting a moving subject (hereinafter referred to as a moving object) and variations in the operation of the imaging device 100 when a moving object is detected will be described.

[0042] <Motion detection method> When the imaging device 100 is stationary, the photoelectric conversion element 1011 detects only luminance changes resulting from the movement of a moving object, allowing the outer shape of the moving object to be detected. When the imaging device 100 is driving the driving unit 300, it is necessary to distinguish between luminance changes resulting from the driving and luminance changes resulting from the movement of the moving object. Referring to FIG. 7, a method for detecting a moving object when the imaging device 100 is driving will be described. Reference numeral 500 denotes an image, and three objects 501, 502, and 503 are shown in the image 500. In the case of FIG. 7, objects 501 and 502 are stationary objects, and object 503 is a moving object. Next, FIG. 504 is a schematic diagram showing the motion vectors of the objects shown in the image 500. The motion vectors corresponding to objects 501, 502, and 503 are motion vectors 505, 506, and 507. When the object is stationary, the direction and magnitude of the motion vectors resulting from the driving unit 300 are constant. On the other hand, for a moving object, the direction and magnitude of the motion vector depend on the direction and speed of the moving object. Except when the imaging area is occupied by moving objects, the motion vector of a stationary subject will dominate among the motion vectors, making it possible to identify moving objects by comparing their motion vectors.

[0043] <Flowchart> Fig. 8 shows a flowchart for detecting a moving object by image capture device 100 using an event-based sensor. The processing shown in the flowchart in Fig. 8 is executed by CPU 102 (Fig. 1), which is a computer, in accordance with a computer program stored in memory 103. The flowchart is composed of the following steps S601 to S605, and the processing content is as described above.

[0044] S601: The control unit 304 determines whether the imaging device is stationary, and if it is determined that it is stationary, the process proceeds to S602. If it is determined that it is not stationary, the process proceeds to S604.

[0045] S602: The detection unit 303 determines whether or not there is a change in luminance in the pixel of interest. If there is a change in luminance for a stationary object, the process proceeds to S603. If there is no change in luminance, the process proceeds to S605.

[0046] S603: The detection unit 303 determines that the subject is moving.

[0047] S604: If the detection unit 303 determines that the motion vector is in the minority (less than half or less than a predetermined reference value) when compared with all other motion vectors in the imaging area, it proceeds to S603. If it is determined that the motion vector is not in the minority, i.e., is in the majority, it proceeds to S605.

[0048] S605: The detection unit 303 determines that the subject is stationary.

[0049] <Operation of the imaging device when detecting a moving object> The following describes variations in the operation of the imaging device when a moving object is detected.

[0050] The first method is to wait for a moving object to be detected before adjusting the angle of view. For example, the output unit 306 displays a message on the user interface that a moving object has been detected, and disables drive control so that the angle of view can be changed after a certain time has passed or when the moving object is no longer detected. Alternatively, variations such as disabling drive control depending on the proportion of moving objects in the imaging area may be used.

[0051] The second method is to display moving objects distinctly from other subjects (still objects). This method is based on the premise that when adjusting the angle of view using a still subject as a landmark, if a moving object can be distinguished, it is possible to identify the still subject. When a high proportion of moving objects occupies the imaging area and it is difficult to determine the imaging angle of view from the still subject, it is desirable to use the first method described above for control. Examples of methods for distinguishing and displaying moving objects include adding color to the moving object or displaying the moving object in a circle. In this case, the imaging device is vibrated using the method described in embodiment 1 to adjust the angle of view.

[0052] As described above, in an imaging device using an event-based sensor, it is possible to adjust the angle of view so that the subject is captured more clearly.

[0053] (Other embodiments) The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a data communication network or various storage media. Then, a computer (or CPU, MPU, etc.) of the system or device reads and executes the programs. The programs may also be provided by recording them on a computer-readable storage medium. [Explanation of symbols]

[0054] 100 Imaging device 300 Drive Unit 301 Imaging unit 302 Image Generation Unit 303 Detection unit 304 Control Unit 305 Output section

Claims

1. a control means for performing a predetermined operation to move the imaging device in at least one direction; a detection means for detecting a subject from an address event signal indicating the position and time of a pixel where a change in luminance has occurred; The information processing device is characterized in that the control means, when performing a first operation of moving the imaging device in a predetermined direction, performs a second operation of vibrating the imaging device in a direction different from the predetermined direction.

2. a generating unit configured to generate an image from an address event signal indicating a position and time of a pixel where a change in luminance occurs in response to an operation of the imaging device; 2. The information processing apparatus according to claim 1, wherein the generating means generates the image corrected by a predetermined width when the second operation is being executed.

3. The information processing device according to claim 1 or 2, characterized in that when the control means is in a detection mode for detecting the edge of the subject and when the control means executes the first operation, it further executes the second operation of vibrating the imaging device in a direction different from the predetermined direction.

4. When the control means is in a detection mode for detecting an edge of the subject and the first operation is not being executed, If the zoom drive is not operating, the first operation is performed; 4. The information processing apparatus according to claim 3, wherein the first operation and the second operation are not executed when the zoom drive is in operation.

5. 5. The information processing apparatus according to claim 1, wherein the second operation vibrates the imaging device at a constant cycle.

6. 6. The information processing apparatus according to claim 1, wherein the second operation is a control of vibrating at a fixed angle in the predetermined direction.

7. The information processing device according to claim 1 , wherein the second operation is faster than a frame rate of the imaging device.

8. The information processing apparatus according to claim 1 , wherein the first operation sets an upper limit on a driving speed.

9. 9. The information processing apparatus according to claim 8, wherein the first operation sets an upper limit on a drive speed determined based on an image generated based on the address event signal.

10. 10. The information processing apparatus according to claim 8, wherein the first operation sets an upper limit on a drive speed determined according to a zoom magnification.

11. 11. The information processing device according to claim 1, wherein the control means does not execute the second operation when driving the lens of the imaging device while the first operation is not being executed.

12. 11. The information processing device according to claim 1, wherein when the control means adjusts the angle of view of the imaging device in executing the second operation, the control means sets the wide end setting as the initial state, and alternately performs a process of adjusting the center of shooting and a process of zooming in or out.

13. 13. The information processing apparatus according to claim 1, wherein, when a moving object is present within the angle of view of the imaging device, the control means does not execute the first operation until the moving object stops.

14. 14. The information processing device according to claim 1, wherein, when the second operation is performed, the detection means determines that a subject having a motion vector of a different magnitude or direction from the most common vector among all motion vectors of the subject within the angle of view of the imaging device is a moving object.

15. 15. The information processing device according to claim 1, wherein the address event signal is output by a sensor that detects a change in luminance per unit time for each pixel, and when no change in luminance is detected, the address event signal is not output or indicates that there is no change in luminance.

16. 16. The information processing device according to claim 1, wherein the address event signal is output by a photoelectric conversion element having a pixel that outputs a signal in response to an incidence of a photon.

17. A program for causing a computer to function as each of the means included in the information processing device according to any one of claims 1 to 16.

18. a control step of performing a predetermined operation of moving the imaging device in at least one direction; a detection step of detecting a subject from an address event signal indicating the position and time of a pixel where a change in luminance has occurred, An information processing method characterized in that the control step, when performing a first operation of moving the imaging device in a predetermined direction, performs a second operation of vibrating the imaging device in a direction different from the predetermined direction.

Images