Information processing device, information processing method, program, and imaging system

Abstract

An event detection unit 20 detects, as an event, that a luminance change of a pixel in an imaging unit which photoelectrically converts an optical image indicating an object has exceeded a preset threshold value. A determination unit 41a of an information processing unit 40-1 determines that the event is not detected on the basis of event detection information indicating an event detection result generated by the event detection unit 20. A movement control unit 42a moves, when the determination unit 41a determines that the event is not detected, the position of the optical image indicating the object in the imaging unit 22 of the event detection unit 20 so as to enable detection of the event in a non-event-detection region. An event information generation unit 43 generates event information including the event detection information and movement information relating to the movement of the position of the optical image. Thus, event information indicating the event detection result for a region wider than a conventional region can be acquired.

Description

[Technical Field]

[0001] This technology relates to an information processing device, an information processing method, a program, and an imaging system, and expands the area in which event information can be acquired. [Background technology]

[0002] As one of the conventional event-driven imaging devices, an asynchronous imaging device called a DVS (Dynamic Vision Sensor) is disclosed in Non-Patent Document 1. This asynchronous imaging device detects an event when a change in luminance of a pixel that photoelectrically converts incident light exceeds a predetermined threshold, and functions as an event detection device. Such an asynchronous imaging device (event detection device) has the advantages of low power consumption, small data volume, low latency, high frame rate imaging, and high dynamic range imaging.

[0003] Furthermore, as described in Non-Patent Document 2, it is possible to estimate a luminance image by accumulating event information over time. Furthermore, as described in Patent Document 1, a method has been proposed in which an event detection device and a normal imaging device (for example, an imaging device that captures images at a predetermined frame rate and generates captured images for each frame) are provided, and when an event is detected by the event detection device, an image is captured by the normal imaging device, thereby taking advantage of the benefits of low power consumption and reduced data volume. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2020-161992 [Non-patent literature]

[0005] [Non-Patent Document 1] Gallego, Guillermo, Tobi Delbruck, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, et al. 2019. "Event-Based Vision: A Survey," April. http: / / arxiv.org / abs / 1904.08405. [Non-patent document 2] Rebecq, Henri, Rene' Ren', Rene' Ranftl, Vladlen Koltun, and Davide Scaramuzza. 2019. "High Speed ​​and High Dynamic Range Video with an Event Camera MULTIMEDIA MATERIAL.". http: / / rpg.ifi.uzh.ch / e2vid. Summary of the Invention [Problem to be solved by the invention]

[0006] However, since the event detection device detects a positive event when a pixel value changes from a reference value to a larger value, and a negative event when a pixel value changes from a reference value to a smaller value, it is not possible to obtain event information for areas where there is little change in pixel value.

[0007] Furthermore, when estimating luminance information by accumulating event information over time, luminance information cannot be accurately estimated from the event information without background information. For example, if an event detection device is installed in a fixed location, such as a surveillance camera, event information can be obtained for areas of moving subjects, but event information cannot be obtained for stationary background areas. Furthermore, because the luminance estimated using event information for areas showing moving subjects indicates a relative difference with respect to the color of the background area (background color), the luminance of the subject may differ from the actual luminance due to the influence of the background color.

[0008] Furthermore, when an event detection device and a normal imaging device are used, it becomes difficult to reduce the size and cost.

[0009] Therefore, an object of this technology is to provide an information processing device, an information processing method, a program, and an imaging system that can expand the area from which event information can be acquired. [Means for solving the problem]

[0010] The first aspect of this technology is an event detection unit that detects, as an event, a change in luminance of a pixel in an imaging unit that photoelectrically converts an optical image of a subject, exceeding a preset threshold; and a determination unit that determines whether an event has been detected, based on event detection information that indicates a detection result of the event, the event detection information being generated by the event detection unit; a movement control unit that moves a position of an optical image representing the subject in the imaging unit when the determination unit determines that an event has not been detected; The information processing device includes:

[0011] In this technology, the determination unit determines whether an event has been detected based on event detection information indicating an event detection result generated by an event detection unit that detects, as an event, a change in brightness of pixels in an imaging unit that performs photoelectric conversion of an optical image of a subject, exceeding a preset brightness threshold. For example, the determination unit determines an event has not been detected when there is an event non-detection area where the number of events indicated in the event detection information is less than a preset threshold, there is a direction in which an edge is not detected, the determination unit determines the movement of an event and there is a direction in which there is no movement, there is a lack of capacity in the communication path for transmitting the event detection information, etc. The determination unit may also divide the area of ​​the optical image and determine whether each divided area is an event non-detection area, and determine an event non-detection if it is an event non-detection area.

[0012] When it is determined that an event has not been detected, the movement control unit moves the position of the optical image representing the subject in the imaging unit so that an event can be detected in an area where no event has been detected. The event detection unit has an imaging unit and an imaging optical system that forms an optical image representing the subject on an imaging surface of the imaging unit, and the movement control unit moves the position of the optical image representing the subject by moving the position of the optical axis of the imaging optical system in the imaging unit. The movement control unit may move the imaging unit in a direction perpendicular to the optical axis of the imaging optical system, or may move the optical axis of the imaging optical system relative to the imaging unit. Furthermore, the movement control unit may control the movement of a mobile object equipped with the event detection unit. Furthermore, when it is determined that an event has not been detected because insufficient capacity occurs in the communication path during transmission of event detection information, the movement control unit moves the imaging unit in a direction that reduces the number of events to be detected.

[0013] This technology further includes an event information generation unit that generates event information including the event detection information generated by the event detection unit and movement information related to the movement of the position of the optical image representing the subject. The event information generation unit uses information indicating the movement of the imaging unit as the movement information when moving the imaging unit in a direction perpendicular to the optical axis of the imaging optical system, and uses information indicating the movement of the optical axis as the movement information when moving the optical axis of the imaging optical system relative to the imaging unit. The event information generation unit also generates the event information using information generated by a motion sensor that detects the movement of a moving object equipped with the event detection unit.

[0014] The second aspect of this technology is an event detection unit that detects, as an event, a change in luminance of a pixel in an imaging unit that photoelectrically converts an optical image showing a subject, exceeding a preset threshold; and a determination unit that determines whether an event has been detected, based on event detection information that indicates a detection result of the event, the event detection information being generated by the event detection unit; When the determining unit determines that an event has not been detected, a movement control unit controls the image capturing unit to move a position of an optical image representing the subject. The present invention relates to an information processing method including the steps of:

[0015] The third aspect of this technology is An event detection unit detects, as an event, a change in luminance of a pixel in an imaging unit that photoelectrically converts an optical image of a subject, the change exceeding a preset threshold. The event detection unit generates a program that causes a computer to execute control in accordance with the detection result of the event, a step of determining whether an event has been detected based on the event detection information; a step of moving a position of an optical image showing the subject in the imaging unit; The program is executed by the computer.

[0016] The program of the present technology is, for example, a program that can be provided in a computer-readable format to a general-purpose computer capable of executing various program codes via a storage medium or communication medium, such as an optical disk, a magnetic disk, or a semiconductor memory, or a communication medium such as a network. By providing such a program in a computer-readable format, processing according to the program is realized on the computer.

[0017] The fourth aspect of this technology is an event detection unit that detects, as an event, a change in luminance of a pixel in an imaging unit that photoelectrically converts an optical image of a subject, exceeding a preset threshold; and a determination unit that determines whether an event has been detected, based on event detection information that indicates a detection result of the event, the event detection information being generated by the event detection unit; a movement control unit that moves a position of an optical image representing the subject in the imaging unit when the determination unit determines that an event has not been detected; an event information generation unit that generates event information including the event detection information generated by the event detection unit and movement information regarding movement of a position of an optical image representing the subject; an image generation unit that generates an image showing the subject based on the event information generated by the event information generation unit; The imaging system includes: [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a first embodiment. [Figure 2] 4 is a flowchart showing the operation of the first embodiment. [Figure 3] FIG. 10 is a diagram illustrating an example of the operation of an event detection unit. [Figure 4] 10A and 10B are diagrams illustrating an example of an operation when movement control is performed by an event detection unit. [Figure 5] FIG. 10 is a diagram illustrating an example in which a detection region is divided into small regions. [Figure 6] FIG. 10 is a diagram illustrating a configuration of a second embodiment. [Figure 7] 10 is a flowchart illustrating an operation of the second embodiment. [Figure 8] FIG. 10 is a diagram illustrating a case where an event detection unit is provided in a moving body (for example, a car). [Figure 9] FIG. 10 is a diagram illustrating a configuration of a third embodiment. [Figure 10] 10 is a flowchart illustrating an operation of the third embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example of an operation when the number of detected events is reduced. [Figure 12] FIG. 10 is a diagram illustrating an example of the number of events occurring when an edge with a different brightness step is moved by one pixel. [Figure 13] 10A and 10B are diagrams illustrating an example of generating a luminance image when the event information includes movement information of the event detection unit. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, embodiments of the present technology will be described in the following order. 1. First embodiment 1-1. Configuration of the first embodiment 1-2. Operation of the First Embodiment 2. Second embodiment 2-1. Configuration of the second embodiment 2-2. Operation of the Second Embodiment 3. Third Embodiment 3-1. Configuration of the third embodiment 3-2. Operation of the Third Embodiment 4. Image generation 5. Other Embodiments

[0020] <1. First embodiment> <1-1. Configuration of the First Embodiment> 1 illustrates a configuration of a first embodiment of an imaging system using an information processing device according to the present technology. The imaging system 10-1 includes an event detection unit 20 and an information processing unit 40-1. The event detection unit 20 and the information processing unit 40-1 may be provided independently or integrally.

[0021] The event detection unit 20 includes an imaging optical system 21, an imaging unit 22, an event detection processing unit 23, and a movement processing unit 24.

[0022] The imaging optical system 21 is configured using a focus lens, a zoom lens, etc. The imaging optical system 21 drives the focus lens, the zoom lens, etc. to form an optical image showing a subject (subject optical image) on an imaging surface of the imaging unit 22. The imaging optical system 21 may be integrated into and fixed to the imaging unit 22, or may be detachably fixed to the imaging unit 22.

[0023] The imaging unit 22 performs photoelectric conversion of the optical image of the subject for each pixel and generates a pixel signal according to the luminance. The imaging unit 22 is configured using an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor). The imaging unit 22 is configured using a pixel array unit and a pixel array drive unit, although not shown. The pixel array unit has a plurality of pixels arranged two-dimensionally in a matrix. Each pixel performs photoelectric conversion and generates a pixel signal of a captured image that represents the optical image of the subject. The pixel array drive unit drives the pixel array unit to output the pixel signal generated for each pixel to the event detection processing unit 23.

[0024] The event detection processing unit 23 is provided integrally with the imaging unit 22 and detects the presence or absence of an event based on whether a change (brightness change) has occurred in the pixel signal generated for each pixel in the imaging unit 22 that exceeds a preset threshold (brightness threshold). Events consist of, for example, an ON event, which indicates that the amount of change in the pixel signal has exceeded an upper brightness threshold, and an OFF event, which indicates that the amount of change has fallen below a lower brightness threshold. The event detection information consists of an event detection result including the polarity of the change in light intensity, such as one bit indicating the detection result of an ON event and one bit indicating the detection result of an OFF event. The event detection information also includes at least the event detection result and position information, such as coordinates indicating the position of the pixel where the event was detected.

[0025] Furthermore, the event detection information implicitly includes time information indicating the relative time at which the event occurred, as long as the output interval of the event detection information remains the same as when the event occurred. However, if the time interval of the event detection information can no longer be maintained as when the event occurred, due to, for example, the event detection information being recorded in memory, the implicitly included time information will be lost. Therefore, the event detection processing unit 222 may include time information, such as a timestamp, in the event detection information to enable determination of the timing at which the event occurred.

[0026] The event detection information includes position information of the pixel where the event was detected, time information indicating the time when the event occurred, and polarity information of the change in light intensity as the event. For example, a data format called AER (Address Event Representation) can be adopted. The event detection processing unit 23 outputs the generated event detection information to the information processing unit 40-1.

[0027] The movement processing unit 24 moves the position of the optical image of the subject on the imaging plane of the imaging unit 22 based on a control signal supplied from a movement control unit 42a of the information processing unit 40-1, which will be described later. For example, the event detection unit 20 drives the shake correction lens to move the position of the optical image of the subject on the imaging plane of the imaging unit 22, similar to when shake correction is performed by providing a shake correction lens in the imaging optical system 21. Furthermore, the event detection unit 20 may move the position of the optical image of the subject by moving the imaging unit 22 in a direction perpendicular to the optical axis of the imaging optical system 21, similar to when shake correction is performed by moving the imaging unit 22.

[0028] The information processing unit 40-1 includes a determination unit 41a, a movement control unit 42a, and an event information generation unit 43.

[0029] The determination unit 41a determines whether an event has been detected based on the event detection information that indicates the event detection result generated by the event detection unit 20, and outputs the determination result to the movement control unit 42a.

[0030] When the determination result of the determination unit 41a indicates that an event has not been detected, the movement control unit 42a performs movement control to move the position of the optical image of the subject in the imaging unit 22, and outputs a control signal to the movement processing unit 24 of the event detection unit 20. In addition, the movement control unit 42a generates movement information related to the movement of the position of the optical image of the subject and outputs the movement information to the event information generation unit 43.

[0031] The event information generating unit 43 generates event information including the event detection information generated by the event detecting unit 20 and the movement information generated by the movement control unit 42a, and outputs the event information to the image generating unit 50, an external device, or the like.

[0032] The image generating unit 50 generates an image of the subject captured by the event detecting unit 20 based on the event information.

[0033] <1-2. Operation of the First Embodiment> Next, the operation of the first embodiment will be described. The imaging system 10-1 determines whether there is an area where no event is detected based on the event detection information generated by the event detection unit 20, and if it determines that there is an area where no event is detected, moves the position of the optical image of the subject in the imaging unit and acquires event detection information about the area where no event is detected.

[0034] 2 is a flowchart showing the operation of the first embodiment. In step ST1, the imaging system 10-1 performs imaging using the event detection unit. The event detection unit 20 of the imaging system 10-1 performs photoelectric conversion of the optical image of the subject for each pixel to generate a pixel signal. Based on the pixel signal, the event detection unit 20 detects, as an event, that a change in pixel luminance exceeds a preset luminance threshold. The event detection unit 20 generates event detection information indicating the event detection result and proceeds to step ST2.

[0035] FIG. 3 illustrates the operation of the event detection unit, where (a) in FIG. 3 illustrates an image captured by the image sensor 221, and (b) in FIG. 3 illustrates the event detection results. In (a) in FIG. 3, the objects OBa and OBB are moving, while the backgrounds BG1 to BG4 are stationary. The background brightness increases in the order of background BG1, background BG2, background BG3, and background BG4. In this case, as shown in (b) in FIG. 3, an ON event (white portion) is detected at a pixel in an edge portion containing a component in a direction perpendicular to the direction of movement of the objects OBa and OBB, where the brightness change exceeds the upper brightness threshold, and an OFF event (black portion) is detected at a pixel where the brightness change falls below the lower brightness threshold. Areas where the brightness change is small are areas where no event is detected (gray portion). Note that (c) in FIG. 3 will be described later.

[0036] 2, in step ST2, the imaging system 10-1 records the event detection information. The information processing unit 40-1 of the imaging system 10-1 records the event detection information generated in step ST1 in the event information generating unit 43, and the process proceeds to step ST3.

[0037] In step ST3, the imaging system 10-1 determines whether it is in an event non-detection state. The information processing unit 40-1 of the imaging system 10-1 determines whether there is an area where no event has been detected based on the event detection information generated in step ST1 using the determination unit 41a. If there is an area where no event has been detected, the determination unit 41a determines that it is in an event non-detection state and proceeds to step ST4. If it has not determined that it is in an event non-detection state, the determination unit 41a proceeds to step ST8. The determination unit 41a determines that it is in an event non-detection state if there is an event non-detection area where the number of events indicated by the event detection information is less than a preset threshold (non-detection area determination threshold). For example, in the case of the captured image shown in FIG. 3(a), the backgrounds BG1 to BG4 are stationary, and therefore no events are detected in the backgrounds BG1 to BG4, as shown in FIG. 3(b), and therefore the number of events is less than the non-detection area determination threshold. Therefore, the determination unit 41a determines that it is in an event non-detection state.

[0038] Furthermore, when the movement control unit 42a moves the event detection unit 20 in the horizontal direction, the event detection unit 20 detects an event at an edge portion in the vertical direction, and when the event detection unit 20 is moved in the vertical direction, the event detection unit 20 detects an event at an edge portion in the horizontal direction. Therefore, the determination unit 41a may determine the direction of an edge in the optical image (captured image) of the subject, and determine that an event has not been detected if there is a direction in which an edge is not detected, so that an event at an edge portion in the direction in which an edge is not detected can be detected.

[0039] In step ST4, the imaging system 10-1 controls the movement of the event detection unit 20. The information processing unit 40-1 of the imaging system 10-1 controls the movement of the event detection unit 20 so that event detection information for areas where no events are detected can be acquired, and outputs a control signal to the event detection unit 20 to move the position of the optical image of the subject in the imaging unit 22 of the event detection unit 20, and then proceeds to step ST5.

[0040] In step ST5, the imaging system 10-1 records the movement information. The movement control unit 42a generates movement information (such as the amount of movement and the direction of movement) related to the movement of the position of the optical image performed by the event detection unit 20 based on the control signal. For example, when moving the imaging unit 22 in a direction perpendicular to the optical axis of the imaging optical system 21, the movement control unit 42a generates information indicating the amount of movement and the direction of movement of the imaging unit 22 as the movement information. Furthermore, when moving the optical axis of the imaging optical system 21 relative to the imaging unit 22, the movement control unit 42a generates information indicating the amount of drive and the direction of drive of the shake correction lens of the imaging optical system 21 as the movement information. The movement control unit 42a causes the event information generation unit 43 to record the generated movement information, and the process proceeds to step ST6.

[0041] In step ST6, the imaging system performs imaging using the event detection unit. The event detection unit 20 of the imaging system 10-1 performs photoelectric conversion of the optical image of the subject for each pixel to generate a pixel signal. Based on the pixel signal, the event detection unit 20 detects an event when a change in pixel luminance exceeds a preset luminance threshold. That is, the event detection unit 20 detects an event related to a stationary subject, generates event detection information indicating the detection result, and proceeds to step ST7.

[0042] Figure 4 illustrates an example of the operation when movement control is performed by the event detection unit, where (a) of Figure 4 illustrates an image captured by the imaging unit 22, and (b) of Figure 4 illustrates the event detection result. When movement control is performed by the event detection unit, the background position moves, so it becomes possible to detect an event related to a stationary area, for example, an event at the boundary between backgrounds BG1 to BG4. (c) of Figure 4 will be described later.

[0043] In step ST7, the imaging system 10-1 records the event detection information. The information processing unit 40-1 of the imaging system 10-1 records the event detection information generated in step ST6 in the event information generating unit 43, and the process proceeds to step ST8.

[0044] In step ST8, the imaging system 10-1 outputs the event information. The information processing unit 40-1 of the imaging system 10-1 generates event information including the event detection information recorded in the event information generation unit 43 and the movement information generated by the movement control unit 42a, and outputs the event information to the image generation unit 50, an external device, etc.

[0045] The event detection unit 20 may be asynchronous or synchronous. The event detection unit 20 may generate event detection information with matrix indexes, for example, and the event information generation unit 43 of the information processing unit 40-1 may store event data for each matrix index, or may store event detection information by dividing the matrix index into regions of a certain size. Furthermore, if the event detection unit 20 is synchronous, the event information generation unit 43 may store the event detection information in a state where the event detection information is added, for example, on a frame-by-frame basis.

[0046] The determination unit 41a may determine whether an area is one in which an event is not detected by dividing the detection area into small areas and determining whether the number of events added up for each small area is less than a threshold (non-detection area determination threshold). Fig. 5 illustrates an example in which the detection area is divided into small areas. In this case, the event detection unit 20 determines whether each divided area is one in which an event is not detected. Note that the black framed areas are areas determined to be event detection areas, and the white framed areas are areas determined to be event non-detection areas.

[0047] The movement control unit 42a may also be configured to perform movement control so that more boundaries can be detected. For example, as described above, since driving the event detection unit horizontally detects vertical edges, and moving it vertically detects horizontal edges, it is possible to first move it horizontally and then vertically. Boundaries may also be detected by moving it diagonally. Furthermore, in the event detection unit 20, if the movement processing unit 24 can move the position of the subject optical image in units smaller than one pixel (subpixel units) based on the control signal supplied from the movement control unit 42a, the movement may be configured to move it stepwise in subpixel units.

[0048] Alternatively, the event information generation unit 43 may generate event information indicating only the event non-detection region. For example, the event information generation unit 43 may generate event information indicating only the event non-detection region indicated by the white frame in Fig. 5. In this case, the information processing unit 40-1 can detect, for example, only the edges of the background.

[0049] In this way, in the first embodiment, when it is determined that an event is not detected, the position of the optical image of the subject in the imaging unit 22 of the event detection unit 20 is moved, so that an event can be detected even in a stationary area where no movement is occurring, such as the background shown in Fig. 4. Therefore, according to the first embodiment, the information processing unit 40-1 can widen the area from which event information can be acquired.

[0050] <2. Second Embodiment> Next, in the second embodiment, a case where an event detection unit is provided in a moving object will be described.

[0051] <2-1. Configuration of the second embodiment> 6 illustrates an example of the configuration of the second embodiment. The imaging system 10-2 has an event detection unit 20, a motion sensor 30, and an information processing unit 40-2. The event detection unit 20 and the information processing unit 40-2 may be provided independently or integrally. The motion sensor 30 is provided integrally with the event detection unit 20 so as to detect movement occurring in the event detection unit 20 provided on a moving object.

[0052] The event detection unit 20 includes an imaging optical system 21, an imaging unit 22, an event detection processing unit 23, and a movement processing unit 24, similar to the first embodiment.

[0053] The imaging optical system 21 is configured using a focus lens, a zoom lens, etc., and forms an optical image of a subject on the imaging surface of the imaging unit 22 .

[0054] The imaging unit 22 performs photoelectric conversion of the optical image of the subject for each pixel, generates a pixel signal according to the luminance, and outputs the pixel signal to the event detection processing unit 23. The event detection processing unit 23 detects the presence or absence of an event based on the pixel signal output from the imaging unit 22, and outputs event detection information indicating the detection result to the information processing unit 40-2.

[0055] The movement processing unit 24 moves the position of the subject optical image on the imaging surface of the imaging unit 22 based on a control signal supplied from a movement control unit 42b of the information processing unit 40-2, which will be described later.

[0056] The motion sensor 30 detects the movement of the event detection unit 20. The motion sensor 30 uses, for example, an IMU (Inertial Measurement Unit) or the like, detects the movement of the event detection unit 20 (such as a change in position or a change in posture) and outputs movement information indicating the detection result to the information processing unit 40-2.

[0057] The information processing unit 40-2 includes a determination unit 41b, a movement control unit 42b, and an event information generation unit 43.

[0058] The determination unit 41b determines whether an event has not been detected based on the event detection information generated by the event detection unit 20 and the movement information generated by the motion sensor 30. For example, the determination unit 41b determines the movement of the event based on the event detection information and the movement information, and determines that an event has not been detected if there is a direction in which there is no movement. The determination unit 41b outputs the determination result of whether an event has not been detected to the movement control unit 42b.

[0059] When the determination result of the determination unit 41b indicates that an event has not been detected, the movement control unit 42b performs movement control to move the position of the optical image of the subject in the imaging unit 22 and outputs a control signal to the movement processing unit 24 of the event detection unit 20. The movement control unit 42b also performs movement control based on movement information supplied from the motion sensor 30. For example, when the event detection unit 20 is provided on a moving body (e.g., a vehicle) and is moving only in a first direction (e.g., the forward direction of the vehicle), the movement control unit 42b determines that information on directions other than the first direction has not been obtained, and generates a control signal to output the control signal to the movement processing unit 24 of the event detection unit 20 so that information on the other directions can be obtained. The movement control unit 42b may also perform movement control of the moving body on which the event detection unit 20 is provided so that information on the other directions can be obtained. The movement control unit 42b also generates movement information regarding the movement of the position of the optical image of the subject and outputs the movement information to the event information generation unit 43.

[0060] The event information generation unit 43 generates event information including the event detection information generated by the event detection unit 20, the movement information generated by the motion sensor 30, and the movement information generated by the movement control unit 42b, and outputs the event information to the image generation unit 50, external devices, etc.

[0061] The image generating unit 50 generates an image of the subject captured by the event detecting unit 20 based on the event information.

[0062] <2-2. Operation of the Second Embodiment> Next, the operation of the second embodiment will be described. The imaging system 10-2 determines whether there is an area where no event is detected, based on the event detection information generated by the event detection unit 20 and the movement information generated by the motion sensor 30. If the imaging system 10-2 determines that there is an area where no event is detected, it moves the position of the optical image of the subject in the imaging unit based on the movement of the event detection unit 20, and acquires event detection information for the area where no event is detected.

[0063] 7 is a flowchart illustrating the operation of the second embodiment. In step ST11, the imaging system 10-2 performs imaging using the event detection unit. The event detection unit 20 of the imaging system 10-2 performs photoelectric conversion of the optical image of the subject for each pixel to generate a pixel signal. Furthermore, the event detection unit 20 detects, as an event, that a change in pixel luminance exceeds a preset luminance threshold based on the pixel signal. The event detection unit 20 generates event detection information indicating the event detection result and proceeds to step ST12.

[0064] In step ST12, the imaging system 10-2 records the motion information and the event detection information. The information processing unit 40-2 of the imaging system 10-2 records the motion information generated by the motion sensor 30 and the event detection information generated in step ST11 in the event information generation unit 43, and then proceeds to step ST13.

[0065] In step ST13, the imaging system 10-2 determines whether it is in an event non-detection state. The information processing unit 40-2 of the imaging system 10-2 determines, via the determination unit 41b, whether there is an area where no event has been detected, based on the motion information supplied from the motion sensor 30 and the event detection information generated in step ST11. For example, when moving in the motion direction indicated by the motion information generated by the motion sensor 30, the determination unit 41b determines that there is an area where no event has been detected, because no event is detected in the edge portion extending in the motion direction. If there is an area where no event has been detected, the determination unit 41b determines that it is in an event non-detection state and proceeds to step ST14, but if it has not determined that it is in an event non-detection state, it proceeds to step ST18.

[0066] In step ST14, the imaging system 10-2 controls the movement of the event detection unit 20. The information processing unit 40-2 of the imaging system 10-2 controls the movement of the event detection unit 20 using the movement control unit 42b so that event detection information for areas where no events are detected can be obtained. For example, the movement control unit 42b generates a control signal based on the movement information supplied from the motion sensor 30 so that the event detection unit 20 can obtain information in a direction different from the movement direction of the event detection unit 20. The movement control unit 42b outputs the generated control signal to the movement processing unit 24 of the event detection unit 20 to move the position of the optical image of the subject in the imaging unit 22, and the process proceeds to step ST15.

[0067] In step ST15, the imaging system 10-2 records the movement information. The movement control unit 42b generates movement information regarding the movement of the position of the optical image performed by the event detection unit 20 based on the control signal, records it in the event information generation unit 43, and proceeds to step ST16.

[0068] In step ST16, the imaging system performs imaging using the event detection unit. The event detection unit 20 of the imaging system 10-1 performs photoelectric conversion of the optical image of the subject for each pixel to generate a pixel signal. Based on the pixel signal, the event detection unit 20 detects, as an event, that a change in pixel luminance exceeds a preset luminance threshold. The event detection unit 20 generates event detection information indicating the event detection result and proceeds to step ST17.

[0069] In step ST17, the imaging system 10-2 records the motion information and the event detection information. The information processing unit 40-2 of the imaging system 10-2 records the motion information supplied from the motion sensor 30 and the event detection information generated in step ST16 in the event information generation unit 43, and then proceeds to step ST18.

[0070] In step ST18, the imaging system 10-2 outputs the event information. The information processing unit 40-2 of the imaging system 10-2 generates event information including the event detection information recorded in the event information generation unit 43, the movement information generated by the movement control unit 42b, and the movement information generated by the motion sensor 30, and outputs the event information to the image generation unit 50, an external device, etc.

[0071] Fig. 8 illustrates an example in which the event detection unit is provided in a moving body (e.g., a car). Fig. 8(a) illustrates an image captured by the image sensor 221 of the event detection unit 20 provided in the moving body, and Fig. 8(b) illustrates the event detection results of a conventional method. When the car moves, the boundaries extending radially from the center forward move little as the car moves, while the boundaries in other directions move more than the boundaries extending radially from the center forward. Therefore, while roadside trees RT can be detected by event detection, it is difficult to detect road boundaries RB and white lines WL extending in the direction of travel.

[0072] However, according to the second embodiment, the areas of the road boundaries RB and white lines WL extending in the traveling direction are determined to be event non-detection areas, and the event detection unit 20 moves the position of the optical image of the subject by the movement processing unit 24 based on a control signal supplied from the movement control unit 42b, so that the road boundaries RB and white lines WL extending in the traveling direction can be detected as events, as shown in (c) of Fig. 8. Note that (d) and (e) of Fig. 8 will be described later.

[0073] According to the second embodiment, it is possible to obtain not only the same effect as the first embodiment, but also to obtain event detection information for areas where event detection information cannot be obtained when movement occurs in the event detection unit 20. For example, even if event detection information indicating a vertical edge cannot be obtained because there is little horizontal movement due to the movement of the event detection unit 20, it is possible to obtain event detection information indicating a vertical edge because the position of the optical image is moved horizontally.

[0074] Furthermore, by including the motion information generated by the motion sensor 30 in the event detection information, not only the occurrence status of the event but also the accurate motion of the event detection unit 20 can be saved.

[0075] <3. Third Embodiment> However, as the number of events detected by the event detection unit increases, the amount of event detection information increases. Therefore, when transmitting event information including the event detection information to an external device, if the number of events increases and the capacity of the communication path becomes insufficient, the event information will be lost.

[0076] Therefore, in the third embodiment, when there are an excessive number of events detected by the event detection unit, the imaging unit is moved so that the number of events detected by the event detection unit decreases, thereby preventing the amount of event information from exceeding the communication capacity.

[0077] <3-1. Configuration of the third embodiment> 9 illustrates a configuration of the third embodiment. The imaging system 10-3 has an event detection unit 20, a motion sensor 30, and an information processing unit 40-3. The event detection unit 20 and the information processing unit 40-3 may be provided independently or integrally. The motion sensor 30 is provided integrally with the event detection unit 20 so as to detect the movement of the event detection unit 20.

[0078] The event detection unit 20 includes an imaging optical system 21, an imaging unit 22, an event detection processing unit 23, and a movement processing unit 24, similar to the first embodiment.

[0079] The imaging optical system 21 is configured using a focus lens, a zoom lens, etc., and forms an optical image of a subject on the imaging surface of the imaging unit 22 .

[0080] The imaging unit 22 performs photoelectric conversion of the optical image of the subject for each pixel, generates a pixel signal according to the luminance, and outputs the pixel signal to the event detection processing unit 23. The event detection processing unit 23 detects the presence or absence of an event based on the pixel signal output from the imaging unit 22, and outputs event detection information indicating the detection result to the information processing unit 40-3.

[0081] The movement processing unit 24 moves the position of the subject optical image on the imaging surface of the imaging unit 22 based on a control signal supplied from a movement control unit 42c of an information processing unit 40-3, which will be described later.

[0082] The motion sensor 30 detects the movement of the event detection unit 20. The motion sensor 30 uses, for example, an IMU (Inertial Measurement Unit) or the like, and detects the movement of the event detection unit 20 (such as a change in position or a change in posture) and outputs movement information indicating the detection result to the information processing unit 40-3.

[0083] The information processing unit 40-3 includes a determination unit 41c, a movement control unit 42c, and an event information generation unit 43.

[0084] The determination unit 41c determines whether or not an event excessive state exists. For example, based on the motion information generated by the motion sensor 30 and the event detection information generated by the event detection unit 20, the determination unit 41c determines that an event excessive state exists when the number of events detected by the event detection unit 20 during movement is greater than a preset threshold (excess state determination threshold). The determination unit 41c outputs the determination result of the event excessive state to the movement control unit 42c.

[0085] When the determination result of the determination unit 41c indicates an excessive event state, the movement control unit 42c performs movement control to move the position of the optical image of the subject in the imaging unit 22 in a direction in which fewer events are detected, based on the movement information supplied from the motion sensor 30 and the event detection information generated by the event detection unit 20, and outputs a control signal to the movement processing unit 24 of the event detection unit 20. In addition, the movement control unit 42c generates movement information related to the movement of the position of the optical image of the subject and outputs it to the event information generation unit 43.

[0086] The event information generation unit 43 generates event information including the event detection information generated by the event detection unit 20, the movement information generated by the motion sensor 30, and the movement information generated by the movement control unit 42c, and outputs the event information to the image generation unit 50, external devices, etc.

[0087] The image generating unit 50 generates an image of the subject captured by the event detecting unit 20 based on the event information.

[0088] <3-2. Operation of the Third Embodiment> 10 is a flowchart illustrating the operation of the third embodiment. In step ST21, the imaging system 10-3 performs imaging using the event detection unit. The event detection unit 20 of the imaging system 10-3 performs photoelectric conversion of the optical image of the subject for each pixel to generate a pixel signal. Based on the pixel signal, the event detection unit 20 detects, as an event, that a change in pixel luminance exceeds a preset luminance threshold. The event detection unit 20 generates event detection information indicating the event detection result and proceeds to step ST22.

[0089] In step ST22, the imaging system 10-3 records the motion information and the event detection information. The information processing unit 40-3 of the imaging system 10-3 records the motion information generated by the motion sensor 30 and the event detection information generated in step ST21 in the event information generating unit 43, and then proceeds to step ST23.

[0090] In step ST23, the imaging system 10-3 determines whether the imaging system 10-3 is in an event-excess state. The information processing unit 40-3 of the imaging system 10-3 determines, based on the movement information supplied from the motion sensor 30 and the event detection information generated in step ST21, whether the imaging system 10-3 is in an event-excess state, in which the number of events detected by the moving event detection unit 20 is greater than a preset event number threshold, using the determination unit 41c. If the determination unit 41c determines that the imaging system 10-3 is not in an event-excess state, it returns to step ST21, and if the determination unit 41c determines that the imaging system 10-3 is in an event-excess state, it proceeds to step ST24.

[0091] In step ST24, the imaging system 10-3 controls the movement of the event detection unit 20. In the information processing unit 40-3 of the imaging system 10-3, the movement control unit 42c controls the movement of the event detection unit 20 in order to reduce the number of detected events. For example, the movement control unit 42c generates a control signal to move the position of the optical image of the subject in the imaging unit 22 in a direction that reduces the movement of the subject on the imaging surface of the imaging unit 22 caused by the movement of the event detection unit 20. The movement control unit 42c outputs the generated control signal to the movement processing unit 24 of the event detection unit 20, causes the event detection unit 20 to move the position of the optical image of the subject, and then proceeds to step ST25.

[0092] In step ST25, the imaging system 10-3 records the movement information. The movement control unit 42c generates movement information related to the movement of the position of the subject optical image in the imaging unit 22 based on the control signal, records it in the event information generation unit 43, and then proceeds to step ST26.

[0093] In step ST26, the imaging system performs imaging using the event detection unit. The event detection unit 20 of the imaging system 10-3 performs photoelectric conversion of the optical image of the subject for each pixel to generate a pixel signal. Based on the pixel signal, the event detection unit 20 detects, as an event, that a change in pixel luminance exceeds a preset luminance threshold. The event detection unit 20 generates event detection information indicating the event detection result and proceeds to step ST27.

[0094] In step ST27, the imaging system 10-3 records the motion information and the event detection information. The information processing unit 40-3 of the imaging system 10-3 records the motion information supplied from the motion sensor 30 and the event detection information generated in step ST26 in the event information generation unit 43, and then proceeds to step ST28.

[0095] In step ST28, the imaging system 10-3 outputs the event information. The information processing unit 40-3 of the imaging system 10-3 generates event information including the event detection information recorded in the event information generation unit 43, the movement information generated by the movement control unit 42c, and the movement information generated by the motion sensor 30, and outputs the event information to the image generation unit 50, an external device, etc.

[0096] 11 illustrates an example of an operation for reducing the number of detected events. FIG. 11(a) illustrates an example of events detected when the event detection unit 20 is moving in the direction of the arrow MA. In the third embodiment, when there are an excessive number of events, the movement control unit 42c moves the position of the optical image of the subject in the imaging unit 22 so as to reduce the number of detected events. For example, as shown in FIG. 11(b), the movement control unit 42c moves the position of the optical image of the subject in the direction of the arrow MB, thereby reducing the number of detected events compared to FIG. 11(a).

[0097] According to the third embodiment, when there are many detected events, it is possible to reduce the number of detected events by moving the position of the optical image of the subject in the imaging unit 22. Furthermore, since the number of events can be reduced, it is possible to prevent the amount of event information from exceeding the communication capacity and resulting in the loss of event information.

[0098] <4. Image generation> Next, the image generation unit 50 that generates a captured image using the event information generated in the above-described embodiment will be described. The image generation unit 50 may be provided in the above-described imaging system 10-1 (10-2, 10-3), or may be provided separately from the imaging system 10-1 (10-2, 10-3). Furthermore, when the image generation unit 50 is provided in the imaging system 10-1 (10-2, 10-3), the image generation unit 50 may be provided separately from the event detection unit 20 and the information processing unit 40-1 (40-2, 40-3), or may be provided integrally with the information processing unit 40-1 (40-2, 40-3).

[0099] The image generating section 50 generates an image signal of an image (accumulated image) that indicates differences in luminance of the subject captured by the event detecting section 20 based on the event information, and outputs the image signal to a display device or the like.

[0100] FIG. 12 illustrates the number of events that occur when edges with different brightness steps are moved by one pixel.

[0101] The subject OB moves by one pixel during the period from time T1 to time T4. Note that Fig. 12(a) shows a case where the difference in luminance between the subject OB and the background BG is large, and Fig. 12(b) shows a case where the difference in luminance between the subject OB and the background BG is smaller than that in Fig. 12(a).

[0102] When the difference in luminance between the object OB and the background BG is large, for example, the luminance of pixel PT changes as shown in (c) of Figure 12, with three events occurring in the period from time T1 to time T4. When the difference in luminance between the object OB and the background BG is small, for example, the luminance of pixel PT changes as shown in (d) of Figure 12, with one event occurring in the period from time T1 to time T4. Therefore, it is possible to generate a luminance image based on event information by utilizing the number of events that have occurred.

[0103] When there is no motion information from the event detection unit 20, the image generation unit 50 generates a luminance image by using a technique such as estimating luminance using a DNN as described in Non-Patent Document 2 or a technique for estimating luminance by obtaining an optical flow. For example, if the position of the optical image of the subject is not moved as in the past, differences in background luminance cannot be reproduced, as shown in (c) of FIG. 3. However, by moving the position of the optical image of the subject when it is determined that an event has not been detected, as in the present technology, a luminance image showing differences in background luminance can be generated, as shown in (c) of FIG. 4.

[0104] In addition, when the event information includes motion information, the image generation unit 50 calculates the edge brightness step using the number of event occurrences and the brightness threshold, and generates a brightness image by adding it to the motion direction of the event detection unit 20.

[0105] Fig. 13 illustrates the generation of a luminance image when the event information includes movement information of the event detection unit. Fig. 13(a) illustrates an example of a subject captured by the event detection unit 20. The event detection unit 20 captures, for example, subject OBa and subject OBb, which has higher luminance than subject OBa. Fig. 13(b) shows an event detected when the event detection unit 20 moves one pixel to the right. The image generation unit 50 calculates luminance information line by line, from the first line L1 to the last line LH, as shown in Fig. 13(c).

[0106] Figure 13(d) illustrates the number of event occurrences EC on line Ln in Figure 13(c), where, for example, two ON events occur at the left edge of object OBa and two OFF events occur at the right edge. Also, for example, four ON events occur at the left edge of object OBa, which has a higher brightness than object OBa, and four OFF events occur at the right edge. Figure 13(e) illustrates the event accumulation result ET. The number of ON events is a positive value, and the number of OFF events is a negative value.

[0107] The image generation unit 50 performs the calculation shown in equation (1) to generate an image showing the brightness of the objects OBa and OBi. Equation (1) shows the event accumulation result ET(x, y) at position (x, y). In equation (1), if the number of pixels in one line is "W" and the number of lines is "H", the variables x and y are "x=1, 2,...W, y=1, 2,...H". In equation (1), "EC(x, y)" shows the number of events that occurred at position (x, y), "TB" shows a preset basic brightness level, and "MVP" shows the amount of movement (number of pixels) of the optical image of the object. ET(x,y)=ET(x-1,y)+EC(x,y)×TB / MVP ···(1)

[0108] Based on the position information of the event detection information generated by the event detection unit 20, the image generation unit 50 calculates the accumulation results of events for each line from the first line L1 to the last line LH, and generates an image showing the difference in brightness between the subjects OBa and OBB, as shown in (f) of Figure 13.

[0109] Furthermore, if the position of the optical image of the subject is not moved, the road boundaries RA, RB and white lines WL cannot be reproduced, as shown in (d) of Figure 8. However, if a non-event detection state is determined, as in the present technology, by moving the position of the optical image of the subject, a luminance image can be generated in which the road boundaries and white lines are reproduced, as shown in (e) of Figure 8.

[0110] In this way, the image generating unit 50 can estimate brightness with high accuracy and low cost using the event detection information and movement information contained in the event information, and generate an image that shows differences in brightness of the subject.

[0111] <5. Other embodiments> An embodiment of the present technology may have a configuration and operation that combines the above-described embodiments. Furthermore, the imaging unit 22 in the event detection unit 20 may have a color filter (e.g., a red, green, and blue mosaic filter) on the imaging surface. In this case, event information for each color component can be generated. Furthermore, the image generation unit 50 can generate a color image that shows the difference in luminance for each color component of the subject by using the event information for each color component.

[0112] Furthermore, when the event detection unit 20 receives infrared rays or millimeter waves for distance measurement as incident light, the information processing unit 40-1 (40-2, 40-3) can generate event information indicating not only changes in the distance to a moving subject but also changes in the distance to a non-moving subject. Furthermore, when the event detection unit 20 receives infrared rays for temperature measurement as incident light, the information processing unit 40-1 (40-2, 40-3) can generate event information indicating not only changes in the temperature of a moving subject but also changes in the temperature of a non-moving subject. Furthermore, if the event information includes movement information from the event detection unit, the image generation unit 50 can generate a distance image indicating the difference in the distance to a moving subject and the distance to a non-moving subject, or a temperature display image indicating the difference in temperature.

[0113] The series of processes described in this specification can be executed by hardware, software, or a combination of both. When executing processes by software, a program recording the processing sequence is installed in the memory of a computer incorporated in dedicated hardware and executed. Alternatively, the program can be installed and executed on a general-purpose computer capable of executing various processes.

[0114] For example, the program can be pre-recorded on a recording medium such as a hard disk, a solid state drive (SSD), or a read-only memory (ROM). Alternatively, the program can be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a compact disc read-only memory (CD-ROM), a magneto-optical (MO) disk, a digital versatile disc (DVD), a Blu-Ray Disc (BD (registered trademark)), a magnetic disk, or a semiconductor memory card. Such removable recording media can be provided as so-called packaged software.

[0115] In addition to being installed on a computer from a removable recording medium, the program may also be transferred wirelessly or by wire from a download site to a computer via a network such as a WAN (Wide Area Network) typified by cellular, a LAN (Local Area Network), or the Internet. The computer can receive the program transferred in this manner and install it on a recording medium such as a built-in hard disk.

[0116] Note that the effects described in this specification are merely examples and are not limiting, and additional effects not described may exist. Furthermore, the present technology should not be interpreted as being limited to the above-described embodiments of the technology. The embodiments of the technology disclose the present technology in the form of examples, and it is obvious that a person skilled in the art can modify or substitute the embodiments without departing from the gist of the present technology. In other words, the scope of the claims should be taken into consideration when determining the gist of the present technology.

[0117] Furthermore, the information processing device of the present technology can also have the following configuration. (1) a determination unit that determines whether an event has been detected based on event detection information indicating a detection result of an event generated by an event detection unit that detects, as an event, a change in luminance of a pixel in an imaging unit that photoelectrically converts an optical image showing a subject, the change exceeding a preset threshold; and a movement control unit that moves a position of an optical image representing the subject in the imaging unit when the determination unit determines that an event has not been detected; An information processing device comprising: (2) The information processing device according to (1), wherein the determination unit determines that an event has not been detected when there is an event non-detection area in which the number of events indicated in the event detection information is less than a preset threshold. (3) The information processing device according to (2), wherein the determination unit divides an area of ​​the optical image and determines whether each divided area is the event non-detection area. (4) The information processing device according to any one of (1) to (3), wherein the determination unit determines the direction of an edge in the optical image based on the event detection information, and determines that an event has not been detected if there is a direction in which no edge is detected. (5) The information processing device according to any one of (1) to (4), wherein the determination unit determines a movement of the event and determines that the event has not been detected if there is a direction in which there is no movement. (6) The information processing device according to any one of (1) to (5), wherein the determination unit determines that an event has not been detected when a capacity shortage occurs in a communication path during transmission of the event detection information. (7) The information processing device according to (6), wherein the movement control unit moves the imaging unit in a direction in which fewer events are detected when the capacity shortage occurs. (8) The event detection unit includes the imaging unit and an imaging optical system that forms an optical image representing the subject on an imaging surface of the imaging unit, The information processing device according to any one of (1) to (7), wherein the movement control unit moves the position of the optical axis of the imaging optical system in the imaging unit, thereby moving the position of the optical image representing the subject. (9) The information processing device according to (8), further comprising an event information generating unit that generates event information including the event detection information generated by the event detection unit and movement information related to movement of the position of the optical image representing the subject. (10) The information processing device according to (9), wherein the movement control unit moves the imaging unit in a direction perpendicular to an optical axis of the imaging optical system. (11) The information processing device according to (10), wherein the event information generating unit uses information indicating movement of the imaging unit as the movement information. (12) The information processing device according to (9), wherein the movement control unit moves an optical axis of the imaging optical system relative to the imaging unit. (13) The information processing device according to (12), wherein the event information generating unit uses information indicating a movement of the optical axis as the movement information. (14) The information processing device according to (9), wherein the event information generating unit includes motion information generated by a motion sensor that detects motion of a moving body that includes the imaging unit in the event information. [Explanation of symbols]

[0118] 10-1, 10-2, 10-3 Imaging system 20 Event detection unit 21 Imaging optical system 22 Imaging unit 23 Event detection processing unit 24. Movement processing section 30. Motion Sensor 40-1, 40-2, 40-3... Information processing section 41a, 41b, 41c...judgment section 42a, 42b, 42c...Movement control unit 43 Event information generation unit 50 Image generation unit 221 Image Sensor 222 Event detection processing unit

Claims

1. an event detection unit that includes an imaging unit that photoelectrically converts an optical image of a subject and a movement processing unit that moves the position of the optical image of the subject on an imaging surface of the imaging unit, and that generates event detection information indicating a result of detecting, as an event, that a change in luminance of a pixel in the imaging unit has exceeded a preset threshold value; and a determination unit that acquires event detection information output from the event detection unit and determines whether an event is not detected based on the event detection information; a movement control unit that outputs a control signal to the movement processing unit to move a position of an optical image representing the subject in the imaging unit when the determination unit determines that an event is not detected; An information processing device comprising:

2. The determination unit determines that the optical image is in an event non-detection state when there is an event non-detection region in which the number of pixels indicated as having an event in the event detection information is less than a preset threshold. The information processing device according to claim 1 .

3. The determination unit divides the area of ​​the optical image, determines whether each divided area is an event non-detection area, and determines that an event non-detection state exists if an event non-detection area is present. The information processing device according to claim 2 .

4. The determination unit determines the direction of an edge in the optical image based on the event detection information, and determines that an event has not been detected if there is a direction in which no edge is detected. The information processing device according to claim 1 .

5. The determination unit determines the movement of the event based on the event detection information and the movement information of the event detection unit, and determines that the event has not been detected if there is a direction in which there is no movement. The information processing device according to claim 1 .

6. The determination unit determines that an event has not been detected when a communication path capacity is insufficient during transmission of the event detection information. The information processing device according to claim 1 .

7. When the capacity shortage occurs, the movement control unit moves the imaging unit in a direction in which fewer events are detected, which is determined based on the event detection information and movement information of the event detection unit. The information processing device according to claim 6 .

8. the event detection unit further includes, together with the imaging unit, an imaging optical system that forms an optical image representing the subject on an imaging surface of the imaging unit; The movement processing unit moves the position of the optical axis of the imaging optical system in the imaging unit based on the control signal from the movement control unit, thereby moving the position of the optical image representing the subject. The information processing device according to claim 1 .

9. The apparatus further includes an event information generating unit that generates event information including the event detection information generated by the event detection unit and movement information regarding movement of the position of the optical image representing the subject. The information processing device according to claim 8 .

10. The movement control unit moves the imaging unit in a direction perpendicular to the optical axis of the imaging optical system. The information processing device according to claim 9 .

11. The event information generating unit uses information indicating the movement of the imaging unit as the movement information. The information processing device according to claim 10.

12. The movement control unit moves the optical axis of the imaging optical system relative to the imaging unit. The information processing device according to claim 9 .

13. The event information generating unit uses information indicating the movement of the optical axis as the movement information. The information processing device according to claim 12.

14. The event information generating unit includes motion information generated by a motion sensor that detects the motion of a moving object having the imaging unit in the event information. The information processing device according to claim 9 .

15. an event detection unit that includes an imaging unit that photoelectrically converts an optical image of a subject and a movement processing unit that moves the position of the optical image of the subject on an imaging surface of the imaging unit, and that generates event detection information indicating a result of detecting, as an event, that a change in luminance of a pixel in the imaging unit has exceeded a preset threshold value; acquiring the event detection information output from the event detection unit, and determining, by a determination unit, whether an event is not detected based on the event detection information; outputting a control signal to the movement control unit, the movement processing unit, for moving a position of an optical image representing the subject in the imaging unit when the determination unit determines that the event is in a non-detection state; An information processing method including:

16. An image pickup unit that photoelectrically converts an optical image showing a subject, and a movement processing unit that moves the position of the optical image of the subject on the image pickup surface of the image pickup unit, and a determination unit that acquires event detection information output from the event detection unit that generates event detection information indicating the result of detecting an event when a change in luminance of a pixel in the image pickup unit exceeds a predetermined threshold, and determines whether an event is not detected based on the event detection information; a movement control unit that outputs a control signal to the movement processing unit to move a position of an optical image representing the subject in the imaging unit when the determination unit determines that an event is not detected; A program written in a computer-readable format to cause a computer to function as a

17. an event detection unit that includes an imaging unit that photoelectrically converts an optical image of a subject and a movement processing unit that moves the position of the optical image of the subject on an imaging surface of the imaging unit, and generates event detection information that indicates a result of detecting, as an event, that a change in luminance of a pixel in the imaging unit exceeds a preset threshold; a determination unit that determines whether an event is not detected based on the event detection information output from the event detection unit; a movement control unit that outputs a control signal to the movement processing unit to move a position of an optical image representing the subject in the imaging unit when the determination unit determines that an event is not detected; an event information generation unit that generates event information including the event detection information generated by the event detection unit and movement information regarding movement of a position of an optical image representing the subject; an image generation unit that generates an image showing the subject based on the event information generated by the event information generation unit; An imaging system comprising:

Images