Information processing device, information processing method, and program

Abstract

To suppress a situation in which a subject is not detected, when an event-based sensor is used.SOLUTION: An information processing device acquires an address event signal indicating the position and time of a pixel at which a change in luminance occurs. The information processing device includes detection means for detecting a subject on the basis of the address event signal and parameter control means for controlling so as to change at least one parameter related to imaging, when the time period during which the subject is not detected is longer than a predetermined time period.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to object detection using an event-based sensor. [Background technology]

[0002] An event-based sensor is known that outputs a change in luminance of each pixel as an address event signal in real time (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-134271 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to suppress a situation in which a subject is not detected when an event-based sensor is used. [Means for solving the problem]

[0005] The information processing device according to the present invention for solving the above-mentioned problems is an information processing device that acquires an address event signal indicating the position and time of a pixel where a change in luminance has occurred, and includes: a detection means that detects a subject based on the address event signal; and a parameter control means that controls so as to change at least one parameter related to photography when the time period during which the subject is not detected is longer than a predetermined time period. and generating means for generating a frame image based on the address event signal, and when the shooting direction is controlled as the parameter, changing the position of the subject on the generated frame image in accordance with the shooting direction and outputting the frame image, and when the shooting angle of view is controlled as the parameter, outputting a part of the generated frame image. It is characterized by: [Effects of the Invention]

[0006] According to the present invention, when an event-based sensor is used, it is possible to reduce situations in which a subject is not detected. [Brief explanation of the drawings]

[0007] [Figure 1] A block diagram showing an example of the hardware configuration of an information processing device. [Figure 2] A block diagram showing an example of the functional configuration of an information processing device. [Figure 3] A diagram showing an example of the configuration of an event-based sensor [Figure 4] Figure 1 shows an example of an image captured by an event-based sensor. [Figure 5] 10 is a flowchart illustrating a process executed by an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0008] Conventional event-based sensors control the detection sensitivity for address events by controlling the time width of the dead band for brightness changes according to the detection frequency of address events. However, when used for applications requiring high-speed detection of specific objects, the following problems arise: If the detection sensitivity for address events is reduced (the width of the dead band is narrowed), brightness changes will not be detected, resulting in reduced object detection accuracy. On the other hand, if the detection sensitivity for address events is increased (the width of the dead band is widened), brightness changes caused by random noise such as photon shot noise will also be detected in addition to brightness changes in the object, resulting in reduced object detection accuracy.

[0009] Therefore, in the following embodiment, it will be described how the accuracy of detecting a subject is improved by controlling the event-based sensor so as to change at least one parameter related to image capture.

[0010] The following describes an image capture device (information processing device) that detects a subject based on the output of an event-based sensor according to an embodiment of the present invention, using the drawings. In this regard, parts having the same functions are assigned the same numbers throughout the drawings, and repeated explanations will be omitted.

[0011] <Embodiment 1> <Information processing device hardware: Figure 1> FIG. 1 shows an information processing device according to this embodiment. In FIG. 1, the information processing device 100 includes an imaging optical system 1010, an imaging unit 101 (event-based camera) including a photoelectric conversion element 1011, a CPU 102, a memory 103, a display unit 104, and an operation unit 105. The imaging unit 101 is a sensor using the photoelectric conversion element 1011, which outputs an address event signal in response to received incident light. The imaging unit 101 detects a change in luminance for each pixel as an event, and the address event signal indicates the position and time of the pixel where the luminance change occurred. The imaging optical system 1010 is specifically a light-receiving lens that receives incident light and forms an image on the photoelectric conversion element 1011. The CPU 102 reads and executes an OS and other programs stored in the memory 103, controls each connected component, and performs various processing operations and logical judgments. The processing executed by the CPU 102 includes the information processing according to this embodiment. The CPU 102 also controls the focus and aperture driving of the imaging optical system 1010, the photoelectric conversion element 1011, and the like. The memory 103 is, for example, a hard disk drive or an external storage device, and stores programs and various data related to the information processing of the embodiment. The display unit 104 outputs the calculation results of the information processing device 100 to a display device in accordance with instructions from the CPU 102. The display device may be of any type, such as a liquid crystal display device, a projector, or an LED indicator. The operation unit 105 is, for example, a touch panel, a keyboard, a mouse, or a robot controller, and is a user interface that accepts input instructions from a user. The information processing device 100 may have mechanisms other than the hardware configurations listed here.

[0012] <Example of functional configuration of information processing device: Figure 2> Next, an example of the functional configuration of an information processing device according to this embodiment will be described with reference to FIG. 2. The information processing device 100 includes an imaging unit 101, a detection unit 201, a parameter control unit 202, an image processing unit 203, a display control unit 204, and an operation reception unit 205. Note that the information processing device according to this embodiment is required to include at least the detection unit 201 and the parameter control unit 202, but may also include other functions. Here, an overview of each function will be described. The information processing device 100 performs various information processing based on the output of the imaging unit 101, which outputs an address event signal corresponding to the received incident light. The detection unit 201 detects a subject within the imaging angle of view based on the address event signal. As an example, assuming that the event-based sensor captures an image with a fixed angle of view, a moving object within the imaging angle of view is detected as the subject. Conversely, if the imaging parameters are fixed and there is no moving object, the address event signal decreases, and the detection unit 202 does not detect anything. A detailed description of the detection unit 201 will be given later. Parameter control unit 202 controls the event-based sensor to change at least one parameter related to image capture (hereinafter referred to as image capture parameter). Details of the image capture parameters and control will be described later. Based on the address event signal, image processing unit 203 generates an image in which a predetermined pixel value according to the direction of the luminance change is assigned to the position of a pixel where a luminance change has occurred. Display control unit 204 displays the generated image on display unit 104. Operation receiving unit 205 receives operations from the user. For example, when the user specifies a target parameter in parameter control unit 202, parameter control unit 202 determines the parameter to be changed based on the operation received by operation receiving unit 205.

[0013] <Event-based sensor: Figure 3> An example of an event-based sensor according to this embodiment will be described. The event-based sensor counts the number of incident photons and determines the timing when the counted number of photons exceeds a predetermined threshold. The event-based sensor also measures the time (clock count) required for the number of photons to reach or exceed a first threshold, and detects a change in luminance by comparing the required times. Specifically, when the previously measured required time is T0 and the latest required time is T, if the difference T-T0 is greater than or equal to a second threshold, a negative change in luminance is detected. If the difference T0-T is greater than or equal to the second threshold, a positive change in luminance is detected. If the difference between T and T0 is less than the second threshold, no change in luminance is detected. The second threshold is a value greater than or equal to zero, and is set in accordance with a preset value or other parameters.

[0014] 3a is a diagram showing an example of the configuration of a photoelectric conversion element 1011. The photoelectric conversion element 1011 is made up of a pixel unit 110 and a peripheral circuit 120. The peripheral circuit 120 includes a vertical arbitration circuit 121 and a horizontal readout circuit 122.

[0015] FIG. 3b shows an example of the configuration of each pixel unit constituting an event-based sensor. The pixel unit 110 includes a photoelectric conversion unit 111, a pixel counter 112, a time counter 113, a first decision circuit 114, a memory 115, a comparator 116, a second decision circuit 117, a response circuit 118, and a selection circuit 119. The photoelectric conversion unit 111 includes an avalanche photodiode (SPAD) operating in Geiger mode, and the pixel counter 112 counts the number of photons incident on the photoelectric conversion unit 111. The time counter 113 counts the time during which a photon is incident on the photoelectric conversion unit 111. Using a SPAD to configure an event-based sensor enables detection of luminance changes at the level of a single photon. By detecting luminance changes at the level of a single photon, an address event signal can be acquired even in night vision conditions, such as at night.

[0016] When the number of photons counted by pixel counter 112 reaches a first threshold, first decision circuit 114 stops counting time by time counter 113. Past count values ​​of time counter 113 are stored in memory 115, and comparator 116 is used to determine the difference between the current count value of time counter 113 and the past count value of time counter 113.

[0017] If the difference count value is equal to or greater than the second threshold, the second determination circuit 117 sends a request signal to the vertical arbitration circuit 121 via the response circuit 118. The response circuit 118 receives a response from the vertical arbitration circuit 121 indicating whether or not the output of address event data is permitted. If the difference count value is less than the second threshold, the response circuit 118 does not send a request signal.

[0018] When the response circuit 118 receives a response indicating permission to output, the selection circuit 119 outputs the count value of the time counter circuit 113 to the horizontal output circuit 122. The horizontal output circuit 122 outputs the received count value as an output signal from the photoelectric conversion element 1011 to the detection unit 201.

[0019] Since the differential count value calculated by the comparator 116 corresponds to the reciprocal of the incident frequency of photons, the photoelectric conversion element 1011 according to this embodiment has the function of measuring "changes in the incident frequency of photons," i.e., changes in luminance. Furthermore, using the second determination circuit 117, an address event is output only when the difference in the intervals at which the number of incident photons reaches the first threshold is equal to or greater than a second threshold (predetermined threshold). That is, the photoelectric conversion element outputs the incident frequency when the difference in the incident frequency is equal to or greater than the second threshold, and does not output the incident frequency when the difference is less than the threshold. With the above configuration, an asynchronous photoelectric conversion element that detects changes in luminance as address events in real time for each pixel address can be realized.

[0020] <Variations of photoelectric conversion elements> The above describes a case where a photoelectric conversion element is used that uses a SPAD as the photoelectric conversion unit and measures the time at which a photon is incident to detect changes in the frequency of photon incidence. However, the configuration shown in Figure 2 is not necessary as long as the photoelectric conversion element is an asynchronous type that detects changes in luminance as address events in real time. For example, as described in Patent Document 1, a photoelectric conversion element that detects changes in luminance as voltage changes may be used.

[0021] <Detection unit 201> The detection unit 201 detects objects based on address event signals. That is, it continuously detects specific objects (e.g., human eyes) that have been set as detection targets based on the brightness change information output by the event-based sensor. In one specific process, the image processing unit 203 first generates frame images by integrating the address event signals output by the event-based sensor for a certain period of time. The detection unit 201 detects feature points from the generated frame images and detects the location of the object (and the presence or absence of the specific object) by pattern matching with feature points detected in a template image of the specific object. The location of the object can be detected by a location with a predetermined similarity to the template. However, the object detection method may be other than pattern matching. For example, the object may be detected by machine learning the shape of the object in the generated frame images using a convolutional neural network (CNN). Alternatively, the address event signals (brightness change information) output by the event-based sensor may be directly machine-learned by a CNN without being converted into frame images. Alternatively, the object may be detected based on the number and range of responsive pixels from the address event signals without generating an image.

[0022] As described above, the event-based sensor outputs an address event only when the difference in the time intervals at which the number of incident photons reaches the first threshold is equal to or greater than a second threshold (predetermined threshold). For example, if the predetermined threshold is significantly large, an address event signal may not be output if the subject's movement is small. Conversely, if the predetermined threshold is small, any movement of the subject or objects included in the scene will be output as an address event signal, making it more difficult to detect the subject. On the other hand, the detection unit 201 cannot detect the subject unless an address event signal is output. Therefore, the predetermined threshold should be set appropriately depending on the situation, and setting the predetermined threshold appropriately may improve the detection accuracy of the detection unit 201.

[0023] If the angle of view of the event-based sensor is fixed and known, the location information (e.g., coordinates) of the detected subject is output to an external device via an output interface such as a network cable (not shown). The absolute position of the subject (position in world coordinates) can be determined from the position where the subject was detected within the angle of view and the position and orientation of the event-based sensor. Information processing device 100 may also be equipped with a display unit, which displays the location of the subject by superimposing it on a frame image obtained by integrating the luminance change output by the event-based sensor over a certain period of time. If the event-based sensor is movable, the position of the subject is determined taking into account the position of the event-based sensor.

[0024] <Parameter control unit 104> Parameter control unit 104 controls the event-based sensor so as to change at least one parameter related to the image capture. In other words, parameter control unit 104 controls the image capture conditions (image capture parameters) of the event-based sensor (or the image capture device having the event-based sensor). The image capture parameters include at least one of the position, shape, and brightness value of the subject image on the image plane. Specifically, these include the image capture direction of the image capture device, the image capture angle of view, the focus position, and the amount of light reaching the pixels of solid-state image sensor 102. Changing the image capture parameters changes the intensity of the brightness change to which the event-based sensor responds, which can cause changes in the quality and quantity of the address event signal output. As a result, the number of address event signals exceeding a predetermined threshold in detection unit 201 increases or decreases, making it easier to detect the subject even if the subject is stationary.

[0025] <Shooting parameter variation 1> The imaging parameters may indicate the imaging direction (angle of view) of an imaging means (event-based sensor) for acquiring an address event signal. For example, when controlling the imaging direction of an imaging device, information processing device 100 may be configured to include an imaging direction driving mechanism consisting of a motor and gears, and to rotate information processing device 100 around a specific rotation axis by controlling the power that drives the motor. In this case, a configuration having multiple rotation axes may be adopted by providing multiple motors.

[0026] <Shooting parameter variation 2> The photographing parameters may be parameters for changing the photographing angle of view of the photographing means. For example, when controlling the photographing angle of view of the photographing device, the information processing device 100 may be configured to have a zoom magnification driving mechanism consisting of a motor and gears, and to change the zoom ratio by moving some of the lenses in the imaging optical system 101 in the optical axis direction. Note that when the zoom is changed, the shape (size) of the subject previously detected changes, so it is preferable to perform processing such as inputting a correspondence between the previous and next images by the user.

[0027] <Shooting parameter variation 3> The photographing parameters may be parameters for changing the focus position of the photographing means. Similarly, when controlling the focus position of the information processing device 100, the information processing device 100 may be configured to include a focus position driving mechanism consisting of a motor and gears, and to change the focus position by moving some of the lenses in the imaging optical system 101 in the optical axis direction.

[0028] <Shooting parameter variation 4> The imaging parameters may be control values ​​for adjusting the amount of light reaching the event-based sensor (photoelectric conversion element) or the range of wavelengths of light received by the event-based sensor. For example, the amount of light reaching the pixels of the solid-state imaging element 102 can be controlled by controlling the aperture of the imaging optical system, the light-absorbing filter, or the illumination. When controlling the aperture of the imaging optical system, the information processing device 100 has an aperture control mechanism, and the amount of light reaching the pixels of the solid-state imaging element 102 can be controlled by opening and closing the aperture. When controlling the light-absorbing filter, the information processing device 100 has an insertion / removal mechanism for inserting and removing the light-absorbing filter into and from the optical path of the imaging optical system, and the amount of light reaching the pixels of the solid-state imaging element 102 can be controlled by inserting and removing the light-absorbing filter. Alternatively, the light-absorbing filter may be a variable transmittance filter made of liquid crystal or the like, and the amount of light reaching the pixels of the solid-state imaging element 102 can be controlled by controlling the voltage applied to the variable transmittance filter. When controlling the lighting, the information processing device 100 has an illumination unit that irradiates light to which the pixels of the solid-state imaging element 102 are sensitive, and by controlling the intensity of the light irradiated from the illumination unit, the amount of light reaching the pixels of the photoelectric conversion element 102 can be controlled.

[0029] <Effects> In the information processing device 100 of the present invention, when the detection unit 201 is unable to detect a subject for a predetermined period of time or longer, the parameter control unit 202 changes the subject image on the image plane. By adopting such a configuration, it is possible to suppress a decrease in subject detection accuracy. This will be explained below.

[0030] Consider a use case in which an information processing device using an event-based sensor continuously detects a specific subject. The event-based sensor outputs a luminance change as an address event signal only when there is a change in luminance equal to or greater than a predetermined threshold. Therefore, if the subject's movement is small and only a luminance change below the predetermined threshold occurs, no address event signal is output, making it impossible to detect the subject. Therefore, in the imaging device of the present invention, if the subject cannot be detected, the parameter control unit changes the subject image on the image plane to cause a luminance change equal to or greater than the threshold. If a luminance change equal to or greater than the threshold occurs, the event-based sensor outputs an address event signal, making it possible to detect the subject.

[0031] If the subject cannot be detected, it is possible to detect small changes in brightness by increasing the detection sensitivity for address events, as disclosed in Patent Document 1. However, if the detection sensitivity for address events is increased (i.e., the predetermined threshold is lowered), brightness changes caused by random noise such as photon shot noise will also be detected. As a result, brightness changes caused by minute movements of the subject will be buried in brightness changes caused by random noise, reducing the accuracy of subject detection.

[0032] On the other hand, in the information processing device of this embodiment, the detection sensitivity for address events is not increased, and luminance changes caused by changes in the subject image are output as address events. As a result, luminance changes caused by subject movement are not buried in luminance changes caused by random noise, making it possible to suppress a decrease in subject detection accuracy.

[0033] As mentioned above, the shooting parameters of the image capturing device controlled by the parameter control unit 202 may be any conditions that change at least one of the position, shape, and brightness value of the subject image on the image plane. However, it is better to change only the brightness value of the subject image rather than changing the brightness value of the entire image plane. Specifically, it is more preferable to control at least one of the position and shape of the subject image on the image plane by the shooting direction, zoom magnification, and focus of the image capturing device rather than controlling the amount of light reaching the pixels of the solid-state image sensor 102 by an aperture, a light-absorbing filter, lighting, etc. This is explained below.

[0034] When the amount of light reaching the pixels of the solid-state imaging device 102 is controlled by an aperture, a light-absorbing filter, lighting, etc., the luminance value of the entire image plane changes. In this case, if the physical property values ​​(absorption and reflection characteristics) at each position on the object plane are equal, the luminance value of each pixel changes at a uniform rate, and the same address event is detected at all pixels. As a result, the luminance change caused by the change in the subject image is buried in the luminance change of the background, making it impossible to suppress a decrease in the accuracy of subject detection.

[0035] In other words, when the amount of light reaching the pixels of the solid-state imaging device 102 is controlled, only the brightness change caused by the difference in the physical property values ​​at each position on the object surface can be detected as an address event. Therefore, the effect of suppressing the deterioration of the object detection accuracy depends on the difference in the physical property values ​​between the object image and the background.

[0036] On the other hand, changing the shooting direction of the image capture device changes the positional relationship between the subject and the image capture device, thereby changing the position of the subject image on the image plane. Furthermore, controlling the zoom magnification of the imaging optical system changes the size of the subject image on the image plane, and controlling the focus position changes the degree of blur of the subject image on the image plane, thereby changing the shape of the subject image in either case. In other words, by controlling the shooting direction, zoom magnification, and focus of the image capture device, it is possible to detect brightness changes due to the texture and contours of the subject image as address events. As a result, brightness changes caused by changes in the subject image are not obscured by brightness changes in the background, further reducing the deterioration of subject image detection accuracy.

[0037] <Variation 1> When a light-absorbing filter or lighting is used, the difference in physical properties between the subject image and the background can be effectively detected by controlling the spectral characteristics of the light-absorbing filter or lighting. Specifically, a filter or lighting that emphasizes the response to light that causes the physical properties of the subject image and the background to differ can be selected.

[0038] For example, consider a case where the subject is red, i.e., selectively reflects light with a wavelength around 650 nm, against a white background, i.e., a subject that reflects all visible light. If a band-stop filter that absorbs only red light (light with a wavelength around 650 nm) is used as the light-absorbing filter, the brightness value changes only in the subject area when the light-absorbing filter is inserted or removed, so the brightness change of only the subject image can be detected as an address event. The band-stop filter can be realized by a multilayer interference filter, etc.

[0039] Furthermore, if lighting that only emits red light (light with a wavelength of around 650 nm) is used, the brightness value changes only in the subject area due to changes in the illumination intensity, and therefore, the brightness change of only the subject image can be detected as an address event. Lighting that emits only red light can be achieved using an LED made of InGaAs, for example. In this way, by using at least one of a light-absorbing filter that selectively absorbs light of a specific wavelength, or lighting that emits light of a specific wavelength, differences in the physical properties of the subject image and the background can be effectively detected.

[0040] <Variation 2> If the subject cannot be detected even when the parameter control unit 202 changes the shooting parameters, it is preferable to change at least one of the amount and type of change in the shooting parameters controlled by the parameter control unit.

[0041] First, a case where the amount of change in the shooting parameters is controlled will be described. Here, a case where the shooting direction is controlled will be described as an example. If the subject cannot be detected even when the shooting direction is changed, for example, the subject may be located far away and the amount of movement of the subject on the image plane may be less than one pixel, so a luminance change of more than the threshold may not occur. Therefore, if the subject cannot be detected, the shooting direction may be significantly changed to make the amount of movement of the subject on the image plane greater than one pixel, thereby causing a luminance change of more than the threshold. In other words, if the amount of change in the shooting parameters controlled by the parameter control unit can be changed and the subject cannot be detected even when the shooting parameters are controlled, the amount of change in the shooting parameters controlled by the parameter control unit may be changed. This causes a luminance change of more than the threshold, making it possible to detect the subject.

[0042] Next, a case where the type of shooting parameter to be changed is changed will be described. If the subject cannot be detected even when the shooting direction is changed, it may be difficult to detect the brightness change caused by the changed shooting parameter. As described above, if the subject cannot be detected even when the shooting direction is changed, it may be because the subject is located far away and the amount of movement of the subject on the image plane is less than one pixel, so that a brightness change of more than the threshold value may not occur. In this case, the zoom magnification of the imaging optical system can be shifted to the telephoto side to increase the amount of movement of the subject on the image plane, thereby causing a brightness change of more than the threshold value. This is an example in which the shooting direction is controlled first, and then the zoom magnification is controlled.

[0043] In other cases, the amount of light is controlled first, and then the shooting direction, zoom magnification, or focus position is controlled. As mentioned above, when the amount of light is controlled by the aperture, light-absorbing filter, lighting, etc., if the difference in physical properties between the subject image and the background is small, the change in brightness caused by the change in the subject image may be obscured by the change in brightness of the background. In this case, the shooting direction of the imaging device may be changed, or the zoom or focus position may be controlled, so that the change in brightness caused by the texture or contour of the subject image can be detected as an address event.

[0044] Furthermore, if the subject cannot be detected for a certain period of time, the subject may move with a brightness change less than the threshold value and move out of the angle of view of the image capture device. Therefore, if the subject cannot be detected, the imaging direction can be changed or the imaging angle of view can be widened to bring the subject within the angle of view of the image capture device while causing a brightness change of more than the threshold value.

[0045] That is, the type of shooting parameter controlled by the parameter control unit can be changed, and if the subject cannot be detected even after controlling the shooting parameters, the type of shooting parameter controlled by the parameter control unit can be changed, thereby causing a change in brightness equal to or greater than a threshold value, making it possible to detect the subject.

[0046] <Variation 2> The detection unit 201 performs tracking processing by storing the position of the detected subject using the memory 103. At this time, if the subject has not moved from its original location as a result of detecting the subject by controlling the shooting parameters with the parameter control unit 202, it is preferable to reduce the frequency with which the parameter control unit 140 controls the shooting parameters. That is, if the subject is detected at the same position as the stored position of the subject after changing the shooting parameters, the parameter control unit 202 increases the predetermined threshold. Specifically, in a configuration in which shooting parameters are controlled when a period in which the subject cannot be detected continues for a predetermined time or longer, it is preferable to increase the predetermined threshold if the subject has not moved from its original location. Increasing the predetermined threshold reduces the frequency with which the parameter control unit 202 controls the shooting parameters, which leads to a reduction in power consumption for driving mechanism control, lighting control, and the like for controlling the predetermined threshold.

[0047] On the other hand, if a subject is detected at a position different from the stored subject position after changing the parameters, the parameter control unit 202 reduces the predetermined threshold. In this case, if the subject has moved from its original location, it is determined that the subject has moved with a brightness change less than the threshold. In such a case, it is preferable for the parameter control unit 202 to control the shooting parameters more frequently. Specifically, in a configuration in which shooting parameters are controlled when a period in which a subject cannot be detected continues for more than a predetermined threshold, it is preferable to reduce the predetermined threshold if the subject has not moved from its original location. By reducing the predetermined threshold, the parameter control unit 202 controls the shooting parameters more frequently, thereby reducing the possibility of overlooking subject movement with a brightness change less than the threshold. Whether the subject is moving may be determined, for example, by whether the movement of the subject's center of gravity position has moved by more than a specific number of pixels. The specific number of pixels is preferably one pixel or more and less than 1 / 10 of the angle of view.

[0048] Furthermore, when the shooting range is expanded by changing the parameters, the parameter control unit 202 may reduce the predetermined threshold value if a subject is detected. If the shooting direction is changed or the shooting angle of view is widened and the subject is detected, and the subject's position has moved outside the angle of view, it is determined that the subject has moved significantly with a brightness change less than the threshold value. In such a case, it is preferable to reduce the predetermined threshold value compared to when the subject has moved within the angle of view. By reducing the predetermined threshold value, the parameter control unit 202 controls the shooting parameters more frequently, further reducing the possibility of overlooking subject movement with a brightness change less than the threshold value.

[0049] <Variation 3> The display control unit 204 outputs the address event signal or the subject detection result to the display unit 104. The display control unit 204 may display a frame image that has been image processed based on the address event signal. Furthermore, if a subject is not detected, the display control unit 204 may perform control so as to notify the user of an error.

[0050] Furthermore, when the display control unit 204 is controlling the shooting conditions (shooting parameters), it may output a display that indicates that the shooting conditions (shooting parameters) are being controlled. For example, the information processing device 100 may have a display unit, and when the information processing device 100 is controlling the shooting parameters, it may display text information or the like that indicates that the shooting parameters are being controlled. With such a configuration, it is possible to distinguish whether the subject image is changing due to the control of the shooting parameters, or due to the movement of the subject image, a change in ambient light, or the like.

[0051] Furthermore, if the information processing device 100 has a processing unit that processes the output of the photoelectric conversion element 1011 to generate a frame image and is configured to display the frame image on the display unit, it is preferable to display the control of the shooting parameters superimposed on the image. A specific example will be described below.

[0052] When the shooting direction is controlled by the parameter control unit, the position of the subject on the frame image changes due to the shooting direction control, even if the subject is not moving. For this reason, it is preferable to use so-called electronic image stabilization, which shifts the position of the subject on the frame image in accordance with the shooting direction control. This makes it possible to distinguish between changes in the position of the subject due to the shooting direction control and changes in the position due to the subject's own movement when displayed.

[0053] Similarly, when the shooting angle of view is shifted to the wide-angle side using the parameter control unit, the position of the subject in the frame image changes. For this reason, it is preferable to use so-called electronic zoom, which cuts out a portion of the frame image in accordance with the control of the shooting angle of view. This makes it possible to distinguish between changes in the position of the subject due to control of the shooting angle of view and changes in position due to the movement of the subject itself. In both cases of electronic image stabilization and electronic zoom, only a portion of the frame image is displayed, so the entire frame image may also be displayed superimposed. In this case, it is even more preferable to display the entire frame images 200a, 200b by enclosing them in a frame or the like so that the difference between the entire frame images 200a, 200b and a portion of the frame image 201 can be seen (FIG. 4).

[0054] <Overall flowchart> FIG. 5 is a flowchart illustrating the operation of the information processing device 100 of the present invention. The processing shown in the flowchart in FIG. 5 is executed by the CPU 102 of FIG. 1, which is a computer, in accordance with a computer program stored in the memory 103. In the following description, each process (step) is denoted by prefixing it with an S, and the process (step) is notated accordingly. In S11, the detection unit 201 determines whether the period during which the subject cannot be detected is longer than a predetermined time, as a trigger for parameter control processing. If the determination result is that the period is equal to or longer than the specific predetermined time, the parameter control unit 202 controls the shooting parameters (S12). If the period is shorter than the specific predetermined time, the shooting parameters are not changed and subject detection continues (continue S11).

[0055] In S13, the detection unit 201 determines whether or not the subject has been detected after controlling the shooting parameters. If the subject has been detected, it further determines whether or not the position of the subject has moved (S18). On the other hand, if the subject has not been detected, the parameter control unit 202 changes the amount of change in the shooting parameters to be applied to the subject (S14).

[0056] In S15 following S14, the detection unit 201 determines whether or not a subject has been detected as a result of changing the amount of change in the shooting parameters. If a subject has been detected, the process returns to S11, the shooting parameters are restored to their original state, and detection of the subject continues. Note that the shooting parameters do not necessarily have to be restored to their original state. On the other hand, if a subject has not been detected, the parameter control unit 202 changes the type of shooting parameter to be controlled (S16).

[0057] In S17 following S16, the detection unit 201 determines whether the position of the subject has moved outside the angle of view. If the position of the subject has not moved outside the angle of view, the detection unit 201 proceeds to S18, where it further determines whether the position of the subject has moved within the angle of view.

[0058] If the detection unit 201 determines in S18 that the position of the subject has not moved, it increases the predetermined threshold (S19). On the other hand, if the detection unit 201 determines that the position of the subject has moved, it decreases the predetermined threshold (S20). Furthermore, if the position of the subject has moved outside the angle of view in S17, the detection unit 201 increases the predetermined threshold further than S20 (S21).

[0059] The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a data communication network or various storage media. Then, a computer (or CPU, MPU, etc.) of the system or device reads and executes the programs. The programs may also be provided by recording them on a computer-readable storage medium. [Explanation of symbols]

[0060] 100 Information processing device 101 Imaging unit 102 Display section 103 Operation section 201 Detection unit 202 Parameter control section 203 Image Processing Unit

Claims

1. An information processing device that acquires an address event signal indicating a position and time of a pixel where a change in luminance has occurred, a detection means for detecting a subject based on the address event signal; a parameter control means for controlling the camera so as to change at least one parameter related to photography when the time during which the subject is not detected is longer than a predetermined time; generating means for generating a frame image based on the address event signal; When the photographing direction is controlled as the parameter, the position of the subject on the generated frame image is changed in accordance with the photographing direction and output. When the parameter is a photographing angle of view, the information processing device outputs a part of the generated frame image.

2. 2. The information processing apparatus according to claim 1, wherein the parameters include at least one of a position, a shape, and a brightness value of the subject image on the image plane.

3. 3. The information processing apparatus according to claim 2, wherein the parameter indicates a photographing direction of a photographing means for acquiring the address event signal.

4. 4. The information processing apparatus according to claim 3, wherein the parameter is a parameter for changing the photographing angle of the photographing means.

5. 5. The information processing apparatus according to claim 3, wherein the parameter is a parameter for changing a focus position of the image capturing means.

6. An optical system having at least one of an aperture, a light absorbing filter that selectively absorbs light of a specific wavelength, and an illumination that irradiates light of a specific wavelength, An image capturing device that acquires an address event signal indicating a position and time of a pixel where a change in luminance occurs, a detection means for detecting a subject based on the address event signal; a parameter control means for controlling the camera so as to change at least one parameter related to photography when the time during which the subject is not detected is longer than a predetermined time; the parameter is a control value for adjusting the amount of light reaching a photoelectric conversion element for acquiring the address event signal, The information processing device, wherein the parameter is a control value for adjusting a range of wavelengths received by the photoelectric conversion element.

7. 7. The information processing device according to claim 1, wherein the parameter control means, when the subject cannot be detected even after changing the parameter, further changes the amount of change in the changed parameter or a different type of parameter among the parameters.

8. An information processing device that acquires an address event signal indicating the position and time of a pixel where a change in luminance has occurred, a detection means for detecting a subject based on the address event signal; a parameter control means for controlling the camera so as to change at least one parameter related to photography when the time during which the subject is not detected is longer than a predetermined time; The address event signal is output for a pixel whose luminance change exceeds a predetermined threshold value; a storage means for storing the position of the subject detected by the detection means, The information processing device is characterized in that the parameter control means increases the predetermined threshold value when the subject is detected at the same position as the position of the subject stored by the storage means after the parameter is changed.

9. 9. The information processing device according to claim 8, wherein the parameter control means reduces the predetermined threshold value when the subject is detected at a position different from the position of the subject stored by the storage means after the parameter is changed.

10. 10. The information processing device according to claim 8, wherein the parameter control means reduces the predetermined threshold value when the subject is detected in a case where the photographing range is widened by changing the parameter.

11. further comprising an output means for outputting the address event signal or the subject detection result; 11. The information processing apparatus according to claim 1, wherein the output means outputs a notification when the parameter is changed.

12. 12. The information processing apparatus according to claim 11, wherein said output means outputs a notice to a user when said subject cannot be detected even after said parameters are changed.

13. 13. The information processing apparatus according to claim 1, wherein the detection means detects the subject having a predetermined shape based on the address event signal.

14. 14. The information processing device according to claim 1, wherein the address event signal is output by a photoelectric conversion element having a pixel that outputs a signal in response to incidence of a photon.

15. A program for causing a computer to function as each of the means included in the information processing device according to any one of claims 1 to 14.

16. An information processing method for acquiring an address event signal indicating a position and time of a pixel where a change in luminance has occurred, comprising: a detecting step of detecting a subject based on the address event signal; a parameter control step of controlling the camera so as to change at least one parameter related to photography when the time during which the subject is not detected is longer than a predetermined time; a generating step of generating a frame image based on the address event signal, When the photographing direction is controlled as the parameter, the position of the subject on the generated frame image is changed in accordance with the photographing direction and output. An information processing method characterized in that, when the photographing angle of view is suppressed as the parameter, a part of the generated frame image is output.

Images